Compare commits
106
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f80e6cea2b | ||
|
|
2a0f2405b0 | ||
|
|
2d7d22776b | ||
|
|
a5eefee17d | ||
|
|
6e82c55371 | ||
|
|
cbeeda9210 | ||
|
|
c85149db13 | ||
|
|
cfb0f7c5d9 | ||
|
|
99282e739e | ||
|
|
d9b7052d5a | ||
|
|
533101033f | ||
|
|
ca2254d55b | ||
|
|
598c5a9d19 | ||
|
|
0a9a32b6e8 | ||
|
|
74b24fa8b1 | ||
|
|
cfbb2ba6cf | ||
|
|
928da17a29 | ||
|
|
bf1d7ebe15 | ||
|
|
23d215591e | ||
|
|
f61c8ad5ea | ||
|
|
b30f097c17 | ||
|
|
28bce9ebaf | ||
|
|
bd451e2fbc | ||
|
|
8284321510 | ||
|
|
b0a7f2d1b8 | ||
|
|
3fa3794bf3 | ||
|
|
1d70e5a4bc | ||
|
|
11d962406d | ||
|
|
785933ee51 | ||
|
|
c91fea7773 | ||
|
|
369d41f26b | ||
|
|
83188f70e6 | ||
|
|
e9cb936c81 | ||
|
|
0e75c70832 | ||
|
|
81d91adb49 | ||
|
|
63e1eaca8c | ||
|
|
1d328397ab | ||
|
|
682b2e25b1 | ||
|
|
28d5052abb | ||
|
|
b0a9f388cf | ||
|
|
ea0315b102 | ||
|
|
f0834ea1b5 | ||
|
|
481e4be1a1 | ||
|
|
276d98b5c5 | ||
|
|
8aba8e98de | ||
|
|
1dd2a538b1 | ||
|
|
64e7dc92e8 | ||
|
|
3d4978f89d | ||
|
|
1efcbbf631 | ||
|
|
7516559c54 | ||
|
|
9976993ba6 | ||
|
|
6fa3b5c9bb | ||
|
|
395beea104 | ||
|
|
7712a9803f | ||
|
|
60da688218 | ||
|
|
095c0e5112 | ||
|
|
f40c5629a5 | ||
|
|
4982b041c6 | ||
|
|
5679a2cfe7 | ||
|
|
711b94161a | ||
|
|
7b7d57c202 | ||
|
|
676c561372 | ||
|
|
3d2f926a82 | ||
|
|
4e21be2b1a | ||
|
|
6ac4efc864 | ||
|
|
67959c9013 | ||
|
|
50cb4f8a8d | ||
|
|
6b362fbb1a | ||
|
|
79873adc3c | ||
|
|
830422c6c6 | ||
|
|
67a8418809 | ||
|
|
75d5c53d36 | ||
|
|
8bd69a0f0e | ||
|
|
a350f58e44 | ||
|
|
c35c808da8 | ||
|
|
82724e99dc | ||
|
|
ce8a4790ae | ||
|
|
614a5bd095 | ||
|
|
215f7f5925 | ||
|
|
74399c5456 | ||
|
|
284a9fde3d | ||
|
|
fee4bf13cb | ||
|
|
be9c77320c | ||
|
|
1f0b2795bb | ||
|
|
6382722d2f | ||
|
|
4be16fc00e | ||
|
|
dcccafc00e | ||
|
|
3fa4fd5d18 | ||
|
|
ef6f616b37 | ||
|
|
0e0759a680 | ||
|
|
947c340e74 | ||
|
|
6f6eb92a90 | ||
|
|
0064e20805 | ||
|
|
59a956bf9f | ||
|
|
afd2fac50b | ||
|
|
edb6a263e9 | ||
|
|
1c00719f68 | ||
|
|
e30a4f2a1c | ||
|
|
ba198e000b | ||
|
|
24ed99a65d | ||
|
|
2cc0035a40 | ||
|
|
b41f12ee38 | ||
|
|
714caded9c | ||
|
|
200da8a167 | ||
|
|
9b02796049 | ||
|
|
062e4e12fd |
No files matched your search
+2
-1
@@ -68,4 +68,5 @@ Non-logged in users should be tightly limited to prevent attacks like SQL/shell
|
||||
|
||||
[`SERVER_SECURITY_MODE`](https://github.com/archiveBox/archiveBox/wiki/configuration#server_security_mode) + CSRF/XSS defenses were only added in the new 0.9.0 `dev` work and are planned to land in `main` in mid-2026.
|
||||
|
||||
Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs. The only URLs that are blocked are URLs hosted by the archivebox server itself.
|
||||
Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` accordingly if they need to limit what the server should archive.
|
||||
It does a basic check to try and avoid recursively archiving snapshots hosted by the archivebox server itself, but it's not a strict security check, it's just to prevent wasted CPU/disk from recursively archiving snapshots in a browser that has the ArchiveBox Browser Extension turned on.
|
||||
@@ -4,11 +4,6 @@ set -Eeuo pipefail
|
||||
repo_name="$1"
|
||||
target_dir="${2:-$repo_name}"
|
||||
|
||||
case "$repo_name" in
|
||||
abx-plugins)
|
||||
version="1.11.232"
|
||||
;;
|
||||
*)
|
||||
version="$(
|
||||
python3 - "$repo_name" <<'PY'
|
||||
import re
|
||||
@@ -27,8 +22,6 @@ if not match:
|
||||
print(match.group(1))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "Cloning ArchiveBox/${repo_name}@v${version} into ${target_dir}"
|
||||
git clone --depth=1 --branch "v${version}" "https://github.com/ArchiveBox/${repo_name}.git" "$target_dir"
|
||||
@@ -16,6 +16,7 @@ env:
|
||||
PYTHONLEGACYWINDOWSSTDIO: utf-8
|
||||
USE_COLOR: False
|
||||
UV_NO_SOURCES: "1"
|
||||
ABXPKG_MIN_RELEASE_AGE: "0"
|
||||
|
||||
jobs:
|
||||
discover-tests:
|
||||
@@ -374,7 +375,6 @@ jobs:
|
||||
|
||||
- name: Run plugin tests - ${{ matrix.plugin.name }}
|
||||
env:
|
||||
ABXPKG_NO_CACHE: "true"
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
TWOCAPTCHA_API_KEY: ${{ secrets.TWOCAPTCHA_API_KEY }}
|
||||
API_KEY_2CAPTCHA: ${{ secrets.TWOCAPTCHA_API_KEY }}
|
||||
|
||||
@@ -29,7 +29,7 @@ Without active preservation effort, everything on the internet eventually disapp
|
||||
*ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
|
||||
<br/>
|
||||
|
||||
> ➡️ Get ArchiveBox with `pip install archivebox>=0.9.5rc1` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
|
||||
> ➡️ Get ArchiveBox with `uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
|
||||
|
||||
*Once installed, you can interact with it through the: [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), [CLI](#usage), [self-hosted web interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python API](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [filesystem](#static-archive-exporting).*
|
||||
|
||||
@@ -52,7 +52,7 @@ It also detects any content featured *inside* pages & extracts it out into a fol
|
||||
- 💾 **Github**/**Gitlab**/etc. links ➡️ `clone of GIT source code`, `README`, `images`, ...
|
||||
- ✨ *and more, see [Output Formats](#output-formats) below...*
|
||||
|
||||
You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `pip`-installed CLI, Python API, and SQLite3 APIs.
|
||||
You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `uv`-installed CLI, Python API, and SQLite3 APIs.
|
||||
All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...
|
||||
|
||||
<br/>
|
||||
@@ -68,7 +68,7 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
|
||||
<br/>
|
||||
|
||||
|
||||
**📦 Install ArchiveBox using your preferred method: `docker` / `pip` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
|
||||
**📦 Install ArchiveBox using your preferred method: `docker` / `uv` / `brew` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
|
||||
|
||||
|
||||
<details>
|
||||
@@ -85,14 +85,14 @@ docker compose run archivebox init
|
||||
<br/>
|
||||
# Option B: Or use it as a plain Docker container:
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -it -v $PWD:/data archivebox/archivebox init
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox help
|
||||
# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev init
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
<br/>
|
||||
<br/>
|
||||
# Option C: Or install it with your preferred pkg manager (see Quickstart below for apt, brew, and more)
|
||||
pip install 'archivebox>=0.9.5rc1'
|
||||
# Option C: Or install it with uv (see Quickstart below for apt, brew, and more)
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
archivebox init
|
||||
archivebox install
|
||||
@@ -214,14 +214,14 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li>Create a new empty directory and initialize your collection (can be anywhere).
|
||||
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -v $PWD:/data -it archivebox/archivebox init
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev init
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
|
||||
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox
|
||||
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# docker run -v $PWD:/data -it [subcommand] [--help]
|
||||
docker run -v $PWD:/data -it archivebox/archivebox help
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev help
|
||||
</code></pre>
|
||||
<i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
|
||||
</li>
|
||||
@@ -256,13 +256,13 @@ See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as a
|
||||
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="Pip" height="28px" align="top"/> <code>pip</code></b> (macOS/Linux/BSD)</summary>
|
||||
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" height="28px" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
|
||||
<br/>
|
||||
<ol>
|
||||
|
||||
<li>Install <a href="https://realpython.com/installing-python/">Python >= v3.13</a> and <a href="https://nodejs.org/en/download/package-manager/">Node >= v22</a> on your system (if not already installed).</li>
|
||||
<li>Install the ArchiveBox package using <code>pip3</code> (or <a href="https://docs.astral.sh/uv/guides/tools/#running-tools"><code>uvx</code></a>).
|
||||
<pre lang="bash"><code style="white-space: pre-line">pip3 install --upgrade archivebox
|
||||
<li>Install <a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> on your system (if not already installed).</li>
|
||||
<li>Install the ArchiveBox package using <code>uv</code>.
|
||||
<pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
archivebox version
|
||||
# install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions
|
||||
# python@3.13 node curl wget git ripgrep ...
|
||||
@@ -305,7 +305,7 @@ archivebox version # make sure the package is installed
|
||||
<li>Create a new empty directory and initialize your collection (can be anywhere).
|
||||
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
archivebox init
|
||||
sudo archivebox install
|
||||
archivebox install
|
||||
archivebox add 'https://example.com'
|
||||
</code></pre>
|
||||
<br/>
|
||||
@@ -319,7 +319,7 @@ archivebox help
|
||||
</li>
|
||||
</ol>
|
||||
See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
|
||||
<sub>See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for more details about this distribution.</sub>
|
||||
<sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details, and run <code>sudo archivebox install</code> only if you want it to install missing system packages via apt.</sub>
|
||||
<br/><br/>
|
||||
</details>
|
||||
|
||||
@@ -503,8 +503,8 @@ archivebox help
|
||||
# equivalent: docker compose run archivebox [subcommand] [--help]
|
||||
docker compose run archivebox help
|
||||
|
||||
# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
|
||||
docker run -it -v $PWD:/data archivebox/archivebox help
|
||||
# equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
|
||||
# optional: import your browser cookies into a persona for logged-in archiving
|
||||
archivebox persona create --import=chrome personal
|
||||
@@ -525,12 +525,12 @@ archivebox persona create --import=chrome personal
|
||||
<summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
|
||||
<br/>
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# make sure you have pip-installed ArchiveBox and it's available in your $PATH first
|
||||
# make sure you have uv-installed ArchiveBox and it's available in your $PATH first
|
||||
<br/>
|
||||
# archivebox [subcommand] [--help]
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
archivebox init
|
||||
sudo archivebox install
|
||||
archivebox install
|
||||
archivebox add 'https://example.com'
|
||||
archivebox version # get archivebox version info + check dependencies
|
||||
archivebox help # get list of archivebox subcommands that can be run
|
||||
@@ -565,13 +565,13 @@ docker compose run archivebox add 'https://example.com'
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# make sure you create and cd into in a new empty directory first
|
||||
<br/>
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
|
||||
docker run -v $PWD:/data -it archivebox/archivebox init
|
||||
docker run -v $PWD:/data -it archivebox/archivebox install
|
||||
docker run -v $PWD:/data -it archivebox/archivebox version
|
||||
docker run -v $PWD:/data -it archivebox/archivebox help
|
||||
docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
|
||||
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev init
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev install
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev version
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev help
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev add 'https://example.com'
|
||||
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️</i>
|
||||
</details>
|
||||
@@ -603,8 +603,8 @@ nano docker-compose.yml # setup initial ADMIN_USERNAME &
|
||||
docker compose up # start the server
|
||||
<br/>
|
||||
# Or with a Docker container:
|
||||
docker run -v $PWD:/data -it archivebox/archivebox archivebox manage createsuperuser
|
||||
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
|
||||
docker run -v $PWD:/data -it archivebox/archivebox:dev archivebox manage createsuperuser
|
||||
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
|
||||
<sup>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sup><br/>
|
||||
@@ -630,15 +630,15 @@ docker compose run archivebox config --set ...
|
||||
|
||||
> [!TIP]
|
||||
> Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.
|
||||
> For example, you could run the Web UI in Docker Compose, and run one-off commands with `pip`-installed ArchiveBox.
|
||||
> For example, you could run the Web UI in Docker Compose, and run one-off commands with `uv`-installed ArchiveBox.
|
||||
|
||||
<details>
|
||||
<summary><i>Expand to show comparison...</i></summary><br/>
|
||||
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
archivebox add --depth=1 'https://example.com' # add a URL with pip-installed archivebox on the host
|
||||
archivebox add --depth=1 'https://example.com' # add a URL with uv-installed archivebox on the host
|
||||
docker compose run archivebox add --depth=1 'https://example.com' # or w/ Docker Compose
|
||||
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com' # or w/ Docker, all equivalent
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com' # or w/ Docker, all equivalent
|
||||
</code></pre>
|
||||
|
||||
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">Docker</a> wiki. ➡️</i>
|
||||
@@ -731,7 +731,7 @@ echo 'http://example.com' | archivebox add
|
||||
echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add
|
||||
|
||||
# if using Docker, add -i when piping stdin:
|
||||
# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox add
|
||||
# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add
|
||||
# if using Docker Compose, add -T when piping stdin / stdout:
|
||||
# echo 'https://example.com' | docker compose run -T archivebox add
|
||||
```
|
||||
@@ -882,10 +882,8 @@ You don't need to install every dependency to use ArchiveBox. ArchiveBox will au
|
||||
|
||||
If not using Docker, make sure to keep the dependencies up-to-date yourself and check that ArchiveBox isn't reporting any incompatibility with the versions you install.
|
||||
|
||||
<pre lang="bash"><code style="white-space: pre-line">#install python3 and archivebox with your system package manager
|
||||
# apt/brew/pip/etc install ... (see Quickstart instructions above)
|
||||
<pre lang="bash"><code style="white-space: pre-line"># install uv + archivebox first (see Quickstart instructions above)
|
||||
<br/>
|
||||
which -a archivebox # see where you have installed archivebox
|
||||
archivebox install # auto install all the extractors and extras
|
||||
archivebox --version # see info and check validity of installed dependencies
|
||||
</code></pre>
|
||||
@@ -1451,7 +1449,8 @@ archivebox server 0.0.0.0:8000
|
||||
# inside the container will reload and pick up your changes
|
||||
./bin/build_docker.sh dev
|
||||
|
||||
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init --install
|
||||
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init
|
||||
docker run -it -v $PWD/data:/data archivebox/archivebox:dev install
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
|
||||
|
||||
@@ -54,6 +54,7 @@ class AddCommandSchema(Schema):
|
||||
snapshot_max_size: int = 0
|
||||
parser: str = "auto"
|
||||
plugins: str = ""
|
||||
persona: str = "Default"
|
||||
only_new: bool | None = None
|
||||
update: bool = False
|
||||
overwrite: bool = False
|
||||
@@ -145,6 +146,7 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
|
||||
snapshot_max_size=args.snapshot_max_size,
|
||||
index_only=args.index_only,
|
||||
plugins=args.plugins,
|
||||
persona=args.persona,
|
||||
parser=args.parser,
|
||||
bg=True, # Always run in background for API calls
|
||||
created_by_id=request.user.pk,
|
||||
|
||||
@@ -4,6 +4,7 @@ import math
|
||||
import json
|
||||
import mimetypes
|
||||
import re
|
||||
from html import unescape
|
||||
from collections import defaultdict
|
||||
from pathlib import Path, PurePosixPath
|
||||
from uuid import UUID
|
||||
@@ -832,7 +833,7 @@ def _snapshots_rss_response(
|
||||
crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
|
||||
description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
|
||||
feed.add_item(
|
||||
title=snapshot.title or snapshot.url,
|
||||
title=unescape(snapshot.title or snapshot.url),
|
||||
link=archived_url or web_base_url,
|
||||
description=description,
|
||||
unique_id=str(snapshot.id),
|
||||
|
||||
@@ -228,6 +228,11 @@ def version(
|
||||
prnt()
|
||||
|
||||
in_data_dir = os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
|
||||
if isinstance(binaries, str):
|
||||
requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
|
||||
else:
|
||||
requested_names = {name for name in (binaries or ()) if name}
|
||||
|
||||
if not in_data_dir:
|
||||
PANEL_TEXT = "\n".join(
|
||||
(
|
||||
@@ -250,6 +255,22 @@ def version(
|
||||
)
|
||||
prnt()
|
||||
|
||||
if not in_data_dir and not requested_names:
|
||||
prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
|
||||
prnt("", "[grey53]Dependency checks require an initialized collection DATA_DIR.[/grey53]")
|
||||
prnt()
|
||||
prnt("[deep_sky_blue3][i] Code locations:[/deep_sky_blue3]")
|
||||
try:
|
||||
for name, path in get_code_locations().items():
|
||||
if isinstance(name, str) and isinstance(path, dict):
|
||||
prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
|
||||
except Exception as e:
|
||||
prnt(f" [red]Error getting code locations: {e}[/red]")
|
||||
prnt()
|
||||
prnt("[red][i] Data locations:[/red] (not in a data directory)")
|
||||
prnt()
|
||||
return []
|
||||
|
||||
prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
|
||||
failures = []
|
||||
seen_failures: set[str] = set()
|
||||
@@ -260,11 +281,6 @@ def version(
|
||||
from abx_dl.dependencies import load_binary
|
||||
from abx_dl.models import discover_plugins, filter_plugins
|
||||
|
||||
if isinstance(binaries, str):
|
||||
requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
|
||||
else:
|
||||
requested_names = {name for name in (binaries or ()) if name}
|
||||
|
||||
plugins = discover_plugins(runtime="archivebox")
|
||||
enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
|
||||
enabled_plugin_names = set(enabled_plugins)
|
||||
@@ -339,13 +355,19 @@ def version(
|
||||
):
|
||||
continue
|
||||
|
||||
any_rows = True
|
||||
installed = db_binaries.get(logical_name) if db_available else None
|
||||
if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
|
||||
abspath = installed.abspath
|
||||
version_str = (installed.version or "unknown")[:15]
|
||||
provider = (installed.binprovider or "env")[:8]
|
||||
valid = True
|
||||
elif not plugin_enabled and not requested_names:
|
||||
# `archivebox version` is expected to verify the active runtime, not
|
||||
# cold-load every optional plugin provider. Migration and status
|
||||
# checks often run with PLUGINS narrowed to a tiny set; resolving
|
||||
# disabled plugin binaries there can spend most of the command on
|
||||
# providers the current collection will never execute.
|
||||
continue
|
||||
else:
|
||||
loaded = load_binary(actual_record)
|
||||
abspath = str(loaded.loaded_abspath or "")
|
||||
@@ -353,6 +375,7 @@ def version(
|
||||
provider = (loaded.loaded_binprovider.name if loaded.loaded_binprovider else "env")[:8]
|
||||
valid = loaded.is_valid
|
||||
|
||||
any_rows = True
|
||||
if valid:
|
||||
display_path = (
|
||||
_format_binary_abspath(
|
||||
@@ -398,6 +421,50 @@ def version(
|
||||
"style": "" if plugin_enabled else "dim",
|
||||
},
|
||||
)
|
||||
|
||||
if db_available:
|
||||
for binary_name, installed in db_binaries.items():
|
||||
if requested_names and binary_name not in requested_names:
|
||||
continue
|
||||
abspath = installed.abspath
|
||||
version_str = (installed.version or "unknown")[:15]
|
||||
provider = (installed.binprovider or "env")[:8]
|
||||
row_key = _binary_row_dedupe_key(
|
||||
display_name=binary_name,
|
||||
valid=True,
|
||||
version=version_str,
|
||||
provider=provider,
|
||||
abspath=abspath,
|
||||
)
|
||||
if row_key in seen_rows:
|
||||
continue
|
||||
seen_rows.add(row_key)
|
||||
|
||||
display_path = (
|
||||
_format_binary_abspath(
|
||||
abspath,
|
||||
pwd=Path.cwd(),
|
||||
lib_dir=config.ABXPKG_LIB_DIR,
|
||||
personas_dir=CONSTANTS.PERSONAS_DIR,
|
||||
home=Path.home(),
|
||||
)
|
||||
if compact_paths
|
||||
else abspath
|
||||
)
|
||||
emit_row(
|
||||
{
|
||||
"plugin": "(database)",
|
||||
"state": "detected",
|
||||
"status": "[green]√[/green]",
|
||||
"binary": binary_name,
|
||||
"version": version_str,
|
||||
"provider": provider,
|
||||
"path": _render_binary_abspath(display_path) if compact_paths else display_path,
|
||||
"style": "dim",
|
||||
},
|
||||
)
|
||||
any_rows = True
|
||||
any_available = True
|
||||
finally:
|
||||
if live_cm is not None:
|
||||
live_cm.stop()
|
||||
|
||||
+79
-29
@@ -507,6 +507,32 @@ def _plugin_user_config(config: Mapping[str, object]) -> dict[str, str]:
|
||||
return {key: _plugin_user_config_value(value) for key, value in config.items()}
|
||||
|
||||
|
||||
def _plugin_input_config(config: Mapping[str, object]) -> dict[str, object]:
|
||||
"""Return raw config inputs understood by core or plugin schemas."""
|
||||
input_names = _archivebox_config_input_names()
|
||||
return {str(key): value for key, value in config.items() if str(key) in input_names}
|
||||
|
||||
|
||||
def _explicit_plugin_enabled_keys(config: Mapping[str, object]) -> set[str]:
|
||||
"""Return canonical ``*_ENABLED`` keys explicitly provided by the user.
|
||||
|
||||
``PLUGINS`` is a selector, not an override for concrete plugin enable flags.
|
||||
If a user passes ``PLUGINS=wget`` and ``SAVE_WGET=False`` together, the
|
||||
alias-resolved ``WGET_ENABLED=False`` must survive the model validator that
|
||||
expands ``PLUGINS`` into concrete booleans.
|
||||
"""
|
||||
provided = {str(key) for key in config}
|
||||
explicit: set[str] = set()
|
||||
for key, prop in _plugin_config_properties(PLUGIN_CONFIG_SCHEMAS).items():
|
||||
if not key.endswith("_ENABLED") or not isinstance(prop, Mapping):
|
||||
continue
|
||||
aliases = prop.get("x-aliases") if isinstance(prop, Mapping) else None
|
||||
input_names = {key, *(str(alias) for alias in aliases if str(alias).strip())} if isinstance(aliases, list) else {key}
|
||||
if provided & input_names:
|
||||
explicit.add(key)
|
||||
return explicit
|
||||
|
||||
|
||||
def _discover_plugin_config_schemas() -> PluginSchemaDocuments:
|
||||
from archivebox.plugins.discovery import discover_plugin_configs
|
||||
|
||||
@@ -664,7 +690,7 @@ class ArchiveBoxBaseConfig(
|
||||
def _scoped_config(self, *, include_execution: bool) -> dict[str, Any]:
|
||||
keys = type(self)._crawl_runtime_keys() if include_execution else type(self)._crawl_frozen_keys()
|
||||
payload = self.model_dump(mode="json")
|
||||
return {key: payload[key] for key in keys if payload.get(key) is not None}
|
||||
return {key: payload[key] for key in keys if payload.get(key, None) is not None}
|
||||
|
||||
def for_crawl(self) -> dict[str, Any]:
|
||||
"""Config scoped to crawl execution, without runtime object overlays."""
|
||||
@@ -752,15 +778,35 @@ class ArchiveBoxBaseConfig(
|
||||
|
||||
@model_validator(mode="after")
|
||||
def derive_plugin_enabled_config(self):
|
||||
self._derive_plugin_enabled_config(respect_current_enabled=False)
|
||||
return self
|
||||
|
||||
def _derive_plugin_enabled_config(self, *, respect_current_enabled: bool) -> None:
|
||||
plugin_names = _normalize_plugins_config_value(self.PLUGINS)
|
||||
selected_plugins = _plugins_with_required_plugins(plugin_names) if plugin_names else set()
|
||||
enabled_config_keys = _plugin_enabled_config_keys()
|
||||
disabled_plugins = []
|
||||
if respect_current_enabled:
|
||||
disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
|
||||
selected_plugin_roots = plugin_names
|
||||
if selected_plugin_roots:
|
||||
from abx_dl.models import discover_plugins, filter_plugins
|
||||
|
||||
selected_plugins = set(
|
||||
filter_plugins(
|
||||
discover_plugins(runtime="archivebox"),
|
||||
sorted(selected_plugin_roots),
|
||||
include_providers=True,
|
||||
disabled_names=disabled_plugins,
|
||||
),
|
||||
)
|
||||
else:
|
||||
selected_plugins = set()
|
||||
search_backend = self.SEARCH_BACKEND_ENGINE.strip().lower()
|
||||
if search_backend:
|
||||
selected_plugins.add(f"search_backend_{search_backend}")
|
||||
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
|
||||
for plugin_name, enabled_key in enabled_config_keys.items():
|
||||
if plugin_names or plugin_name in selected_plugins:
|
||||
setattr(self, enabled_key, plugin_name in selected_plugins)
|
||||
return self
|
||||
|
||||
|
||||
def _build_archivebox_config_model(plugin_schemas: PluginSchemaDocuments) -> type[ArchiveBoxBaseConfig]:
|
||||
@@ -817,23 +863,6 @@ def _plugin_enabled_config_keys() -> dict[str, str]:
|
||||
return enabled_keys
|
||||
|
||||
|
||||
def _plugins_with_required_plugins(plugin_names: set[str]) -> set[str]:
|
||||
selected = set(plugin_names)
|
||||
pending = list(selected)
|
||||
while pending:
|
||||
plugin_name = pending.pop()
|
||||
schema = PLUGIN_CONFIG_SCHEMAS.get(plugin_name, {})
|
||||
required_plugins = schema.get("required_plugins") if isinstance(schema, dict) else None
|
||||
if not isinstance(required_plugins, list):
|
||||
continue
|
||||
for required_plugin in required_plugins:
|
||||
required_plugin_name = str(required_plugin).strip().lower()
|
||||
if required_plugin_name and required_plugin_name not in selected:
|
||||
selected.add(required_plugin_name)
|
||||
pending.append(required_plugin_name)
|
||||
return selected
|
||||
|
||||
|
||||
def get_live_config_url(key: str) -> str:
|
||||
return f"{LIVE_CONFIG_BASE_URL}{quote(key)}/"
|
||||
|
||||
@@ -1007,9 +1036,11 @@ def get_config(
|
||||
config_data: ConfigPayload = dict(defaults or {})
|
||||
config_data["PERSONAS_DIR"] = str(CONSTANTS.PERSONAS_DIR)
|
||||
base_config_payload: ConfigPayload = {}
|
||||
base_config_model = ArchiveBoxConfig()
|
||||
|
||||
if crawl_config_base:
|
||||
config_data.update(
|
||||
normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
|
||||
normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
|
||||
)
|
||||
config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
|
||||
elif base_config is not None:
|
||||
@@ -1020,7 +1051,7 @@ def get_config(
|
||||
config_data.update(normalize_runtime_config(base_config_payload, exclude_runtime_derived=True, json_safe=False))
|
||||
else:
|
||||
config_data.update(
|
||||
normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
|
||||
normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
|
||||
)
|
||||
legacy_permissions = permissions_from_legacy_public_flags({**BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE), **os.environ})
|
||||
if legacy_permissions:
|
||||
@@ -1072,22 +1103,29 @@ def get_config(
|
||||
archivebox_scope_overrides = {key: value for key, value in scope_overrides.items() if key in _archivebox_config_input_names()}
|
||||
config_data.update(archivebox_scope_overrides)
|
||||
|
||||
explicit_plugin_enabled_keys: set[str] = set()
|
||||
if resolve_plugins:
|
||||
plugin_schemas = {plugin_name: schema for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items() if isinstance(schema, dict)}
|
||||
plugin_global_config = {key: str(value) if isinstance(value, Path) else value for key, value in config_data.items()}
|
||||
crawl_selected_plugins = crawl_config_base and bool(_normalize_plugins_config_value(dict(crawl.config or {}).get("PLUGINS")))
|
||||
# A frozen crawl-level PLUGINS selector is the exact extractor set for
|
||||
# this crawl. Env/file *_ENABLED values are ambient defaults and must
|
||||
# not be re-applied after validation, or a later process env can change
|
||||
# which hooks an already-queued crawl runs.
|
||||
if not crawl_selected_plugins:
|
||||
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(os.environ))
|
||||
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(scope_overrides))
|
||||
plugin_user_config = _plugin_user_config(
|
||||
{
|
||||
**normalize_runtime_config(config_data, only_crawl_execution=True, json_safe=False),
|
||||
**_plugin_input_config(os.environ),
|
||||
**scope_overrides,
|
||||
},
|
||||
)
|
||||
if not crawl_config_base:
|
||||
file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
|
||||
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(file_config))
|
||||
plugin_user_config = {
|
||||
**normalize_runtime_config(
|
||||
BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE),
|
||||
exclude_runtime_derived=True,
|
||||
json_safe=False,
|
||||
),
|
||||
**_plugin_user_config(_plugin_input_config(file_config)),
|
||||
**plugin_user_config,
|
||||
}
|
||||
plugin_sections = resolve_plugin_configs(
|
||||
@@ -1112,6 +1150,13 @@ def get_config(
|
||||
if crawl_config_base:
|
||||
config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
|
||||
config_data.update(archivebox_scope_overrides)
|
||||
if crawl_selected_plugins:
|
||||
# Crawl PLUGINS is a frozen selector. Drop ambient enable flags from
|
||||
# env/file/defaults before model validation so the selected plugin
|
||||
# set is derived from the crawl, not from the worker process that
|
||||
# happens to resume it.
|
||||
for enabled_key in _plugin_enabled_config_keys().values():
|
||||
config_data.pop(enabled_key, None)
|
||||
|
||||
# Decode JSON-encoded complex values (dict/list fields) that came from
|
||||
# string-only sources before validation. ``IniConfigSettingsSource`` does
|
||||
@@ -1137,6 +1182,11 @@ def get_config(
|
||||
)
|
||||
|
||||
config = ArchiveBoxConfig.model_validate(config_data)
|
||||
for key in explicit_plugin_enabled_keys:
|
||||
if key in config_data:
|
||||
setattr(config, key, config_data[key])
|
||||
if config.PLUGINS:
|
||||
config._derive_plugin_enabled_config(respect_current_enabled=True)
|
||||
if redact_sensitive:
|
||||
for key in type(config).model_fields:
|
||||
value = config[key]
|
||||
|
||||
@@ -260,13 +260,15 @@ class SnapshotChangeList(SearchResultsChangeList):
|
||||
rows = (
|
||||
ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
|
||||
.order_by("snapshot_id", "plugin")
|
||||
.values_list("snapshot_id", "plugin", "status", "output_size")
|
||||
.values_list("snapshot_id", "plugin", "status", "output_size", "output_files")
|
||||
)
|
||||
for snapshot_id, plugin, status, output_size in rows.iterator(chunk_size=1000):
|
||||
for snapshot_id, plugin, status, output_size, output_files in rows.iterator(chunk_size=1000):
|
||||
if plugin in seen_plugins[snapshot_id]:
|
||||
continue
|
||||
seen_plugins[snapshot_id].add(plugin)
|
||||
results_by_snapshot[snapshot_id].append(SimpleNamespace(plugin=plugin, status=status, output_size=output_size))
|
||||
results_by_snapshot[snapshot_id].append(
|
||||
SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
|
||||
)
|
||||
|
||||
for obj in self.result_list:
|
||||
obj.__dict__["_admin_archiveresults"] = results_by_snapshot[obj.pk]
|
||||
|
||||
@@ -93,7 +93,6 @@ from archivebox.core.routes_util import (
|
||||
from archivebox.core.forms import AddLinkForm
|
||||
from archivebox.plugins.forms import get_plugin_config_binary_urls
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.plugins.discovery import discover_plugin_configs
|
||||
from archivebox.plugins.views import get_config_definition_link
|
||||
from archivebox.progressmonitor.views import live_progress_view, progress_endpoint
|
||||
|
||||
@@ -1424,7 +1423,6 @@ class AddView(UserPassesTestMixin, FormView):
|
||||
request_config = get_request_config(self.request, resolve_plugins=True)
|
||||
required_search_plugin = f"search_backend_{request_config.SEARCH_BACKEND_ENGINE}".strip()
|
||||
can_override_crawl_config = self._can_override_crawl_config()
|
||||
plugin_configs = discover_plugin_configs() if can_override_crawl_config else {}
|
||||
public_persona_config_keys = {
|
||||
"CRAWL_MAX_CONCURRENT_SNAPSHOTS",
|
||||
"DELETE_AFTER",
|
||||
@@ -1453,17 +1451,6 @@ class AddView(UserPassesTestMixin, FormView):
|
||||
"binary_urls": binary_urls,
|
||||
}
|
||||
recent_personas = list(persona_queryset.order_by("-created_at", "name")[:5])
|
||||
plugin_dependency_map = {}
|
||||
if can_override_crawl_config:
|
||||
plugin_dependency_map = {
|
||||
plugin_name: [
|
||||
str(required_plugin).strip()
|
||||
for required_plugin in (schema.get("required_plugins") or [])
|
||||
if str(required_plugin).strip()
|
||||
]
|
||||
for plugin_name, schema in plugin_configs.items()
|
||||
if isinstance(schema.get("required_plugins"), list) and schema.get("required_plugins")
|
||||
}
|
||||
return {
|
||||
**context,
|
||||
"title": "Create Crawl",
|
||||
@@ -1473,7 +1460,6 @@ class AddView(UserPassesTestMixin, FormView):
|
||||
"VERSION": VERSION,
|
||||
"FOOTER_INFO": request_config.FOOTER_INFO,
|
||||
"required_search_plugin": required_search_plugin,
|
||||
"plugin_dependency_map_json": json.dumps(plugin_dependency_map, sort_keys=True),
|
||||
"persona_config_map_json": json.dumps(persona_config_map, sort_keys=True, default=str),
|
||||
"recent_personas": recent_personas,
|
||||
"can_override_crawl_config": can_override_crawl_config,
|
||||
|
||||
@@ -68,6 +68,55 @@ PERSONA_PROFILE_DIR_CANDIDATES = (
|
||||
)
|
||||
|
||||
|
||||
def _path_exists(path: Path) -> bool:
|
||||
try:
|
||||
return path.exists()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _path_is_dir(path: Path) -> bool:
|
||||
try:
|
||||
return path.is_dir()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _iter_children(path: Path) -> list[Path]:
|
||||
try:
|
||||
return list(path.iterdir())
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
|
||||
def _is_copyable_profile_entry(path: Path) -> bool:
|
||||
try:
|
||||
if path.is_symlink():
|
||||
return True
|
||||
if path.is_dir():
|
||||
return os.access(path, os.R_OK | os.X_OK)
|
||||
return os.access(path, os.R_OK)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def profile_copy_ignore(src: str, names: list[str]) -> set[str]:
|
||||
ignored = set(shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS)(src, names))
|
||||
src_path = Path(src)
|
||||
for name in names:
|
||||
if name in ignored:
|
||||
continue
|
||||
if not _is_copyable_profile_entry(src_path / name):
|
||||
# Chrome can leave generated profile components unreadable when a
|
||||
# previous run used a different runtime uid, a mounted profile came
|
||||
# from the host, or Chrome tightened permissions on cache/model
|
||||
# directories. Those entries are not required to launch a cloned
|
||||
# profile, and letting copytree abort here prevents the crawl from
|
||||
# even creating its first Snapshot.
|
||||
ignored.add(name)
|
||||
return ignored
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PersonaImportSource:
|
||||
kind: str
|
||||
@@ -394,13 +443,16 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
|
||||
continue
|
||||
seen_roots.add(resolved_root)
|
||||
|
||||
if not resolved_root.exists() or not resolved_root.is_dir():
|
||||
if not _path_is_dir(resolved_root):
|
||||
continue
|
||||
|
||||
for persona_dir in sorted((path for path in resolved_root.iterdir() if path.is_dir()), key=lambda path: path.name.lower()):
|
||||
for persona_dir in sorted(
|
||||
(path for path in _iter_children(resolved_root) if _path_is_dir(path)),
|
||||
key=lambda path: path.name.lower(),
|
||||
):
|
||||
for candidate_dir_name in PERSONA_PROFILE_DIR_CANDIDATES:
|
||||
user_data_dir = persona_dir / candidate_dir_name
|
||||
if not user_data_dir.exists() or not user_data_dir.is_dir():
|
||||
if not _path_is_dir(user_data_dir):
|
||||
continue
|
||||
|
||||
for profile_dir in _list_profile_names(user_data_dir):
|
||||
@@ -594,7 +646,7 @@ def copy_browser_user_data_dir(source_dir: Path, destination_dir: Path) -> None:
|
||||
source_dir,
|
||||
destination_dir,
|
||||
symlinks=True,
|
||||
ignore=shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS),
|
||||
ignore=profile_copy_ignore,
|
||||
)
|
||||
|
||||
|
||||
@@ -707,12 +759,12 @@ def export_browser_state(
|
||||
|
||||
|
||||
def _list_profile_names(user_data_dir: Path) -> list[str]:
|
||||
if not user_data_dir.exists() or not user_data_dir.is_dir():
|
||||
if not _path_is_dir(user_data_dir):
|
||||
return []
|
||||
|
||||
profiles: list[str] = []
|
||||
for child in sorted(user_data_dir.iterdir(), key=lambda path: path.name.lower()):
|
||||
if not child.is_dir():
|
||||
for child in sorted(_iter_children(user_data_dir), key=lambda path: path.name.lower()):
|
||||
if not _path_is_dir(child):
|
||||
continue
|
||||
if child.name == "System Profile":
|
||||
continue
|
||||
@@ -726,7 +778,7 @@ def _list_profile_names(user_data_dir: Path) -> list[str]:
|
||||
|
||||
|
||||
def _looks_like_profile_dir(path: Path) -> bool:
|
||||
if not path.exists() or not path.is_dir():
|
||||
if not _path_is_dir(path):
|
||||
return False
|
||||
|
||||
marker_paths = (
|
||||
@@ -738,7 +790,7 @@ def _looks_like_profile_dir(path: Path) -> bool:
|
||||
path / "Session Storage",
|
||||
)
|
||||
|
||||
if any(marker.exists() for marker in marker_paths):
|
||||
if any(_path_exists(marker) for marker in marker_paths):
|
||||
return True
|
||||
|
||||
return any(path.name == prefix or path.name.startswith(prefix) for prefix in BROWSER_PROFILE_DIR_NAMES)
|
||||
|
||||
@@ -11,6 +11,7 @@ Each persona has its own:
|
||||
__package__ = "archivebox.personas"
|
||||
|
||||
import shutil
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
@@ -179,7 +180,16 @@ class Persona(ModelWithConfig):
|
||||
if not profile_dir.exists():
|
||||
return False
|
||||
|
||||
for path in profile_dir.rglob("*"):
|
||||
def profile_paths():
|
||||
# Chrome profiles are user-controlled filesystem state. If a cache
|
||||
# or generated component is unreadable, cleanup should skip it and
|
||||
# continue pruning the rest instead of blocking the crawl runner.
|
||||
for dirpath, dirnames, filenames in os.walk(profile_dir, onerror=lambda _err: None):
|
||||
current_dir = Path(dirpath)
|
||||
for name in [*dirnames, *filenames]:
|
||||
yield current_dir / name
|
||||
|
||||
for path in profile_paths():
|
||||
if path.name in VOLATILE_PROFILE_FILE_NAMES:
|
||||
try:
|
||||
path.unlink()
|
||||
@@ -188,13 +198,21 @@ class Persona(ModelWithConfig):
|
||||
pass
|
||||
|
||||
for dirname in VOLATILE_PROFILE_DIR_NAMES:
|
||||
for path in profile_dir.rglob(dirname):
|
||||
if not path.is_dir():
|
||||
for path in profile_paths():
|
||||
if path.name != dirname:
|
||||
continue
|
||||
try:
|
||||
is_dir = path.is_dir()
|
||||
except OSError:
|
||||
is_dir = False
|
||||
if not is_dir:
|
||||
continue
|
||||
shutil.rmtree(path, ignore_errors=True)
|
||||
cleaned = True
|
||||
|
||||
for path in profile_dir.rglob("*.log"):
|
||||
for path in profile_paths():
|
||||
if not path.match("*.log"):
|
||||
continue
|
||||
try:
|
||||
path.unlink()
|
||||
cleaned = True
|
||||
@@ -252,7 +270,7 @@ class Persona(ModelWithConfig):
|
||||
return self.runtime_root_for_snapshot(snapshot) / "chrome_downloads"
|
||||
|
||||
def copy_chrome_profile(self, source_dir: Path, destination_dir: Path) -> None:
|
||||
from archivebox.personas.importers import VOLATILE_PROFILE_COPY_PATTERNS
|
||||
from archivebox.personas.importers import profile_copy_ignore
|
||||
|
||||
destination_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.rmtree(destination_dir, ignore_errors=True)
|
||||
@@ -260,7 +278,7 @@ class Persona(ModelWithConfig):
|
||||
source_dir,
|
||||
destination_dir,
|
||||
symlinks=True,
|
||||
ignore=shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS),
|
||||
ignore=profile_copy_ignore,
|
||||
)
|
||||
|
||||
def prepare_runtime_for_crawl(self, crawl, chrome_binary: str = "") -> dict[str, str]:
|
||||
|
||||
@@ -64,6 +64,15 @@ def get_plugins() -> list[str]:
|
||||
return sorted(set(plugins))
|
||||
|
||||
|
||||
def get_plugin_models():
|
||||
from abx_dl.models import discover_plugins
|
||||
|
||||
plugins = {}
|
||||
for base_dir in (BUILTIN_PLUGINS_DIR, USER_PLUGINS_DIR):
|
||||
plugins.update(discover_plugins(plugins_dir=base_dir, runtime="archivebox"))
|
||||
return plugins
|
||||
|
||||
|
||||
def get_plugin_name(plugin: str) -> str:
|
||||
"""
|
||||
Get the base plugin name without numeric prefix.
|
||||
@@ -90,36 +99,18 @@ def get_enabled_plugins(config: ConfigLookup | None = None, **config_kwargs: Any
|
||||
|
||||
config = get_config(**config_kwargs)
|
||||
|
||||
def normalize_enabled_plugins(value: Any) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
raw = value.strip()
|
||||
if not raw:
|
||||
return []
|
||||
if raw.startswith("["):
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
parsed = None
|
||||
if isinstance(parsed, list):
|
||||
return [str(plugin).strip() for plugin in parsed if str(plugin).strip()]
|
||||
return [plugin.strip() for plugin in raw.split(",") if plugin.strip()]
|
||||
if isinstance(value, (list, tuple, set)):
|
||||
return [str(plugin).strip() for plugin in value if str(plugin).strip()]
|
||||
return [str(value).strip()] if str(value).strip() else []
|
||||
|
||||
plugins_override = config.get("PLUGINS")
|
||||
if plugins_override:
|
||||
return normalize_enabled_plugins(plugins_override)
|
||||
|
||||
enabled = []
|
||||
disabled = []
|
||||
for plugin in get_plugins():
|
||||
plugin_config = get_plugin_special_config(plugin, config)
|
||||
if plugin_config["enabled"]:
|
||||
enabled.append(plugin)
|
||||
else:
|
||||
disabled.append(plugin)
|
||||
|
||||
return enabled
|
||||
from abx_dl.models import filter_plugins
|
||||
|
||||
return list(filter_plugins(get_plugin_models(), enabled, include_providers=True, disabled_names=disabled))
|
||||
|
||||
|
||||
def discover_plugins_that_provide_interface(
|
||||
@@ -245,61 +236,12 @@ def get_plugin_special_config(plugin_name: str, config: ConfigLookup, _visited:
|
||||
"""
|
||||
plugin_upper = plugin_name.upper()
|
||||
|
||||
plugins_whitelist = config.get("PLUGINS", "")
|
||||
if plugins_whitelist:
|
||||
plugin_configs = discover_plugin_configs()
|
||||
plugin_names = {p.strip().lower() for p in plugins_whitelist.split(",") if p.strip()}
|
||||
pending = list(plugin_names)
|
||||
|
||||
while pending:
|
||||
current = pending.pop()
|
||||
schema = plugin_configs.get(current, {})
|
||||
required_plugins = schema.get("required_plugins", [])
|
||||
if not isinstance(required_plugins, list):
|
||||
continue
|
||||
|
||||
for required_plugin in required_plugins:
|
||||
required_plugin_name = str(required_plugin).strip().lower()
|
||||
if not required_plugin_name or required_plugin_name in plugin_names:
|
||||
continue
|
||||
plugin_names.add(required_plugin_name)
|
||||
pending.append(required_plugin_name)
|
||||
|
||||
if plugin_name.lower() not in plugin_names:
|
||||
enabled = False
|
||||
else:
|
||||
enabled_key = f"{plugin_upper}_ENABLED"
|
||||
enabled = config.get(enabled_key)
|
||||
if enabled is None:
|
||||
enabled = True
|
||||
elif isinstance(enabled, str):
|
||||
enabled = enabled.lower() not in ("false", "0", "no", "")
|
||||
else:
|
||||
enabled_key = f"{plugin_upper}_ENABLED"
|
||||
enabled = config.get(enabled_key)
|
||||
if enabled is None:
|
||||
enabled = True
|
||||
elif isinstance(enabled, str):
|
||||
enabled = enabled.lower() not in ("false", "0", "no", "")
|
||||
|
||||
plugin_configs = discover_plugin_configs()
|
||||
plugin_name_lower = plugin_name.lower()
|
||||
|
||||
if enabled:
|
||||
visited = _visited or set()
|
||||
if plugin_name_lower not in visited:
|
||||
next_visited = visited | {plugin_name_lower}
|
||||
schema = plugin_configs.get(plugin_name_lower, {})
|
||||
required_plugins = schema.get("required_plugins", [])
|
||||
if isinstance(required_plugins, list):
|
||||
for required_plugin in required_plugins:
|
||||
required_plugin_name = str(required_plugin).strip()
|
||||
if not required_plugin_name:
|
||||
continue
|
||||
required_config = get_plugin_special_config(required_plugin_name, config, _visited=next_visited)
|
||||
if not required_config["enabled"]:
|
||||
enabled = False
|
||||
break
|
||||
enabled_key = f"{plugin_upper}_ENABLED"
|
||||
enabled = config.get(enabled_key)
|
||||
if enabled is None:
|
||||
enabled = True
|
||||
elif isinstance(enabled, str):
|
||||
enabled = enabled.lower() not in ("false", "0", "no", "")
|
||||
|
||||
timeout_key = f"{plugin_upper}_TIMEOUT"
|
||||
timeout = config.get(timeout_key) or config.get("TIMEOUT", 300)
|
||||
|
||||
@@ -55,6 +55,7 @@ from archivebox.plugins.discovery import (
|
||||
BUILTIN_PLUGINS_DIR,
|
||||
USER_PLUGINS_DIR,
|
||||
ConfigLookup,
|
||||
get_enabled_plugins,
|
||||
get_plugin_special_config,
|
||||
)
|
||||
|
||||
@@ -215,6 +216,7 @@ def discover_hooks(
|
||||
|
||||
config = get_config(**config_kwargs)
|
||||
|
||||
enabled_plugins = set(get_enabled_plugins(config=config))
|
||||
enabled_hooks = []
|
||||
|
||||
for hook in hooks:
|
||||
@@ -229,9 +231,7 @@ def discover_hooks(
|
||||
enabled_hooks.append(hook)
|
||||
continue
|
||||
|
||||
# Check if plugin is enabled
|
||||
plugin_config = get_plugin_special_config(plugin_name, config)
|
||||
if plugin_config["enabled"]:
|
||||
if plugin_name in enabled_plugins:
|
||||
enabled_hooks.append(hook)
|
||||
|
||||
hooks = enabled_hooks
|
||||
@@ -279,15 +279,27 @@ def run_hook(
|
||||
records = process.get_records() # Get parsed JSONL output
|
||||
"""
|
||||
from archivebox.machine.models import Process, Machine, NetworkInterface
|
||||
from archivebox.config.common import ArchiveBoxConfig, _archivebox_config_input_names, get_config, normalize_runtime_config
|
||||
from archivebox.config.common import (
|
||||
ArchiveBoxConfig,
|
||||
_archivebox_config_input_names,
|
||||
get_config,
|
||||
normalize_runtime_config,
|
||||
_plugin_enabled_config_keys,
|
||||
)
|
||||
|
||||
config_scope = {key.removeprefix("config_"): kwargs.pop(key) for key in list(kwargs) if key.startswith("config_")}
|
||||
config_overrides = _config_to_overrides(config)
|
||||
explicit_override_keys = set(config_overrides)
|
||||
resolved_config = get_config(overrides=config_overrides, **config_scope)
|
||||
hook_config = normalize_runtime_config(
|
||||
resolved_config.for_crawl_runtime(),
|
||||
resolved_config.for_crawl_runtime(runtime_overrides=config_overrides),
|
||||
json_safe=False,
|
||||
)
|
||||
hook_config.update(normalize_runtime_config(config_overrides, json_safe=False))
|
||||
plugin_enabled_keys = set(_plugin_enabled_config_keys().values())
|
||||
if plugin_enabled_keys.intersection(hook_config):
|
||||
for enabled_key in plugin_enabled_keys:
|
||||
hook_config.setdefault(enabled_key, False)
|
||||
|
||||
# Auto-detect timeout from plugin config if not explicitly provided
|
||||
if timeout is None:
|
||||
@@ -368,8 +380,10 @@ def run_hook(
|
||||
archivebox_config_input_names = _archivebox_config_input_names()
|
||||
for key in archivebox_config_input_names:
|
||||
env.pop(key, None)
|
||||
env.pop("PLUGINS", None)
|
||||
env["DATA_DIR"] = str(CONSTANTS.DATA_DIR)
|
||||
env["LIBRARY_VERSION"] = VERSION
|
||||
env.setdefault("ABXPKG_FAST_SCRIPT", "1")
|
||||
env.setdefault("MACHINE_ID", os.environ.get("MACHINE_ID", CONSTANTS.MACHINE_ID))
|
||||
snap_dir = hook_config.get("SNAP_DIR") or _model_output_dir_from_child_path(output_dir, CONSTANTS.SNAPSHOTS_DIR_NAME)
|
||||
crawl_dir = hook_config.get("CRAWL_DIR") or _model_output_dir_from_child_path(output_dir, CONSTANTS.CRAWLS_DIR_NAME)
|
||||
@@ -386,7 +400,9 @@ def run_hook(
|
||||
# Set Node.js module resolution paths.
|
||||
# NODE_PATH may be a path list, but NODE_MODULES_DIR is a single canonical directory.
|
||||
node_modules_dir = hook_config.get("NODE_MODULES_DIR")
|
||||
if not node_modules_dir and lib_dir:
|
||||
if lib_dir and "ABXPKG_LIB_DIR" in explicit_override_keys and "NODE_MODULES_DIR" not in explicit_override_keys:
|
||||
node_modules_dir = Path(lib_dir) / "pnpm" / "packages" / "chrome" / "node_modules"
|
||||
elif not node_modules_dir and lib_dir:
|
||||
node_modules_dir = Path(lib_dir) / "pnpm" / "packages" / "chrome" / "node_modules"
|
||||
|
||||
node_path_parts = [part for part in str(hook_config.get("NODE_PATH") or "").split(os.pathsep) if part]
|
||||
|
||||
@@ -4,6 +4,7 @@ import asyncio
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
@@ -240,6 +241,14 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh
|
||||
def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str:
|
||||
if event.exit_code == PROCESS_EXIT_SKIPPED:
|
||||
return "skipped"
|
||||
if event.exit_code in {128 + signal.SIGHUP, 128 + signal.SIGINT, 128 + signal.SIGTERM}:
|
||||
# This fallback only runs when a snapshot hook exited before emitting a
|
||||
# structured ArchiveResult. A polite shutdown signal means the runner
|
||||
# was interrupted during ownership transfer, not that the extractor
|
||||
# produced a durable negative result. Keep the hook queued so the next
|
||||
# runner can retry the exact work item instead of sealing in a transient
|
||||
# process-lifecycle failure.
|
||||
return "queued"
|
||||
if event.exit_code != 0:
|
||||
return "failed"
|
||||
return "noresults"
|
||||
|
||||
@@ -31,6 +31,27 @@ def current_network_interface_with_machine():
|
||||
return NetworkInterface.objects.select_related("machine").get(id=current_iface.id)
|
||||
|
||||
|
||||
def normalize_process_env(env: dict) -> dict:
|
||||
normalized = dict(env or {})
|
||||
raw_plugins = normalized.pop("PLUGINS", "")
|
||||
selected_plugins = {name.strip().lower() for name in str(raw_plugins).split(",") if name.strip()}
|
||||
from archivebox.config.common import ArchiveBoxConfig, _archivebox_config_input_names, is_sensitive_config_key
|
||||
|
||||
allowed_config_keys = ArchiveBoxConfig._crawl_runtime_keys()
|
||||
config_input_names = _archivebox_config_input_names()
|
||||
for key in list(normalized):
|
||||
if is_sensitive_config_key(key) or (key in config_input_names and key not in allowed_config_keys):
|
||||
normalized.pop(key, None)
|
||||
if selected_plugins:
|
||||
from abx_dl.models import discover_plugins, filter_plugins
|
||||
from archivebox.config.common import _plugin_enabled_config_keys
|
||||
|
||||
selected_plugins = set(filter_plugins(discover_plugins(runtime="archivebox"), sorted(selected_plugins), include_providers=True))
|
||||
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
|
||||
normalized.setdefault(enabled_key, "True" if plugin_name in selected_plugins else "False")
|
||||
return normalized
|
||||
|
||||
|
||||
class ProcessService(BaseService):
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
ProcessStartedEvent,
|
||||
@@ -74,6 +95,7 @@ class ProcessService(BaseService):
|
||||
started_at=started_at,
|
||||
)
|
||||
process = await process_query.order_by("-modified_at").afirst()
|
||||
process_env = normalize_process_env(event.env)
|
||||
if process is None:
|
||||
process = await Process.objects.acreate(
|
||||
machine=iface.machine,
|
||||
@@ -82,7 +104,7 @@ class ProcessService(BaseService):
|
||||
worker_type=worker_type,
|
||||
pwd=event.output_dir,
|
||||
cmd=[event.hook_path, *event.hook_args],
|
||||
env=event.env,
|
||||
env=process_env,
|
||||
timeout=event.timeout,
|
||||
pid=event.pid or None,
|
||||
url=event.url or None,
|
||||
@@ -97,7 +119,7 @@ class ProcessService(BaseService):
|
||||
|
||||
process.pwd = event.output_dir
|
||||
process.cmd = [event.hook_path, *event.hook_args]
|
||||
process.env = event.env
|
||||
process.env = process_env
|
||||
process.timeout = event.timeout
|
||||
process.pid = event.pid or None
|
||||
process.url = event.url or process.url
|
||||
@@ -172,6 +194,7 @@ class ProcessService(BaseService):
|
||||
started_at=started_at,
|
||||
)
|
||||
process = await process_query.order_by("-modified_at").afirst()
|
||||
process_env = normalize_process_env(event.env)
|
||||
if process is None:
|
||||
await Process.objects.acreate(
|
||||
machine=iface.machine,
|
||||
@@ -180,7 +203,7 @@ class ProcessService(BaseService):
|
||||
worker_type=worker_type,
|
||||
pwd=event.output_dir,
|
||||
cmd=[event.hook_path, *event.hook_args],
|
||||
env=event.env,
|
||||
env=process_env,
|
||||
timeout=event.timeout,
|
||||
pid=event.pid or None,
|
||||
url=event.url or None,
|
||||
@@ -197,6 +220,7 @@ class ProcessService(BaseService):
|
||||
"machine_id": iface.machine_id,
|
||||
"iface_id": iface.id,
|
||||
"pwd": event.output_dir,
|
||||
"env": process_env,
|
||||
"pid": event.pid or process.pid,
|
||||
"url": event.url or process.url,
|
||||
"process_type": process_type or process.process_type,
|
||||
|
||||
@@ -55,7 +55,11 @@ from abxbus import BaseEvent
|
||||
from abxbus.event_bus import EventBus, get_current_event, in_handler_context
|
||||
from abxbus.event_handler import EventHandlerAbortedError, EventHandlerCancelledError
|
||||
|
||||
from archivebox.config.common import ArchiveBoxBaseConfig, normalize_runtime_config
|
||||
from archivebox.config.common import (
|
||||
ArchiveBoxBaseConfig,
|
||||
normalize_runtime_config,
|
||||
_plugin_enabled_config_keys,
|
||||
)
|
||||
from archivebox.misc.db import run_db_analyze_batch
|
||||
from archivebox.core.shutdown_util import foreground_shutdown_signals, raise_if_shutdown_requested
|
||||
from archivebox.search.sonic_daemon import register_sonic_daemon_event_handler
|
||||
@@ -121,6 +125,10 @@ def _count_selected_hooks(plugins: dict[str, Plugin], selected_plugins: list[str
|
||||
return sum(1 for plugin in selected.values() for hook in plugin.hooks if "CrawlSetup" in hook.name or "Snapshot" in hook.name)
|
||||
|
||||
|
||||
def _is_nonfatal_setup_hook(plugin_name: str, hook_name: str) -> bool:
|
||||
return plugin_name == "chrome" and hook_name.endswith("_chrome_kill_zombies")
|
||||
|
||||
|
||||
def _discover_archivebox_plugins() -> dict[str, Plugin]:
|
||||
return discover_plugins(runtime="archivebox")
|
||||
|
||||
@@ -800,6 +808,11 @@ class CrawlRunner:
|
||||
},
|
||||
)
|
||||
normalized_config = normalize_runtime_config(config)
|
||||
configured_plugins = [name.strip().lower() for name in str(normalized_config.get("PLUGINS") or "").split(",") if name.strip()]
|
||||
if configured_plugins:
|
||||
selected_plugin_names = set(filter_plugins(self.plugins, configured_plugins, include_providers=True))
|
||||
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
|
||||
normalized_config.setdefault(enabled_key, plugin_name in selected_plugin_names)
|
||||
return {
|
||||
"id": str(snapshot.id),
|
||||
"url": snapshot.url,
|
||||
@@ -1081,6 +1094,8 @@ class CrawlRunner:
|
||||
await completed_process.wait(timeout=crawl_setup_phase_timeout)
|
||||
await completed_process.event_results_list()
|
||||
if completed_process.status == "failed":
|
||||
if _is_nonfatal_setup_hook(plugin.name, hook.name):
|
||||
continue
|
||||
raise RuntimeError(f"Crawl setup hook {plugin.name}:{hook.name} failed")
|
||||
|
||||
async def run_snapshot(self, snapshot_id: str, crawl_start_event: CrawlStartEvent | None = None) -> None:
|
||||
@@ -1130,7 +1145,14 @@ class CrawlRunner:
|
||||
)(snapshot["id"])
|
||||
if queued_plugins:
|
||||
if snapshot_selected_plugins:
|
||||
original_queued_plugins = queued_plugins
|
||||
queued_plugins = queued_plugins_selected_by_config(queued_plugins)
|
||||
disabled_queued_plugins = sorted(set(original_queued_plugins) - set(queued_plugins))
|
||||
if disabled_queued_plugins:
|
||||
await sync_to_async(skip_disabled_queued_plugins, thread_sensitive=True)(
|
||||
snapshot["id"],
|
||||
disabled_queued_plugins,
|
||||
)
|
||||
selected_hooks_by_plugin = {
|
||||
plugin: hooks for plugin, hooks in (selected_hooks_by_plugin or {}).items() if plugin in queued_plugins
|
||||
}
|
||||
@@ -1142,7 +1164,14 @@ class CrawlRunner:
|
||||
)(snapshot["id"])
|
||||
if queued_plugins:
|
||||
if snapshot_selected_plugins:
|
||||
original_queued_plugins = queued_plugins
|
||||
queued_plugins = queued_plugins_selected_by_config(queued_plugins)
|
||||
disabled_queued_plugins = sorted(set(original_queued_plugins) - set(queued_plugins))
|
||||
if disabled_queued_plugins:
|
||||
await sync_to_async(skip_disabled_queued_plugins, thread_sensitive=True)(
|
||||
snapshot["id"],
|
||||
disabled_queued_plugins,
|
||||
)
|
||||
selected_hooks_by_plugin = {
|
||||
plugin: hooks for plugin, hooks in (selected_hooks_by_plugin or {}).items() if plugin in queued_plugins
|
||||
}
|
||||
@@ -1173,7 +1202,13 @@ class CrawlRunner:
|
||||
if snapshot_selected_plugins and remaining_queued_plugins:
|
||||
remaining_queued_plugins = queued_plugins_selected_by_config(remaining_queued_plugins)
|
||||
if not remaining_queued_plugins:
|
||||
await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
|
||||
if snapshot["status"] in ("queued", "started"):
|
||||
await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(
|
||||
snapshot_id,
|
||||
output_dir=output_dir,
|
||||
)
|
||||
else:
|
||||
await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
|
||||
return
|
||||
snapshot_selected_plugins = remaining_queued_plugins
|
||||
plugins = filter_plugins(self.plugins, snapshot_selected_plugins, include_providers=True)
|
||||
@@ -1437,6 +1472,11 @@ def queued_plugins_for_snapshot(snapshot_id: str) -> list[str] | None:
|
||||
def config_overrides_for_queued_plugins(selected_plugins: list[str], **overrides: Any) -> dict[str, Any]:
|
||||
config_overrides = dict(overrides)
|
||||
config_overrides["PLUGINS"] = ",".join(selected_plugins)
|
||||
selected_plugin_names = set(
|
||||
filter_plugins(_discover_archivebox_plugins(), [plugin_name.lower() for plugin_name in selected_plugins], include_providers=True),
|
||||
)
|
||||
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
|
||||
config_overrides[enabled_key] = plugin_name in selected_plugin_names
|
||||
for plugin_name in selected_plugins:
|
||||
if plugin_name.startswith("search_backend_"):
|
||||
config_overrides[f"{plugin_name.upper()}_ENABLED"] = True
|
||||
@@ -1479,6 +1519,28 @@ def fail_unavailable_queued_hooks(
|
||||
)
|
||||
|
||||
|
||||
def skip_disabled_queued_plugins(snapshot_id: str, plugin_names: list[str]) -> None:
|
||||
from archivebox.core.models import ArchiveResult
|
||||
|
||||
if not plugin_names:
|
||||
return
|
||||
now = timezone.now()
|
||||
# Queued ArchiveResult rows are durable scheduler state and can outlive a
|
||||
# config change or deploy. If a plugin is no longer selected for this
|
||||
# Snapshot/Crawl, leaving its old row queued keeps the Snapshot STARTED
|
||||
# forever even though there is no runnable work left.
|
||||
ArchiveResult.objects.filter(
|
||||
snapshot_id=snapshot_id,
|
||||
plugin__in=plugin_names,
|
||||
status=ArchiveResult.StatusChoices.QUEUED,
|
||||
).update(
|
||||
status=ArchiveResult.StatusChoices.SKIPPED,
|
||||
start_ts=now,
|
||||
end_ts=now,
|
||||
output_str="Queued plugin is disabled by this Snapshot/Crawl config",
|
||||
)
|
||||
|
||||
|
||||
def include_background_prerequisite_hooks(
|
||||
selected_hooks_by_plugin: dict[str, set[str] | None],
|
||||
plugins: dict[str, Plugin],
|
||||
@@ -1511,12 +1573,13 @@ def include_background_prerequisite_hooks(
|
||||
def snapshot_hooks_for_pending_archiveresults(snapshot) -> list[tuple[str, str]]:
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.plugins.discovery import get_enabled_plugins
|
||||
|
||||
config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
|
||||
snapshot_plugin_names = [name.strip() for name in str((snapshot.config or {}).get("PLUGINS") or "").split(",") if name.strip()]
|
||||
crawl_plugin_names = [name.strip() for name in str((snapshot.crawl.config or {}).get("PLUGINS") or "").split(",") if name.strip()]
|
||||
config_plugin_names = [name.strip() for name in str(config.PLUGINS or "").split(",") if name.strip()]
|
||||
plugin_names = snapshot_plugin_names or crawl_plugin_names or config_plugin_names
|
||||
plugin_names = snapshot_plugin_names or crawl_plugin_names or config_plugin_names or get_enabled_plugins(config=config)
|
||||
plugins = (
|
||||
filter_plugins(_discover_archivebox_plugins(), plugin_names, include_providers=True)
|
||||
if plugin_names
|
||||
|
||||
@@ -48,7 +48,9 @@
|
||||
<div class="crawl-explanation">
|
||||
<p class="crawl-tip" style="user-select: none;">
|
||||
<span style="float: right">
|
||||
<a href="/admin/agent">💬 Crawl with AI</a> |
|
||||
{% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
|
||||
<a href="/admin/agent">💬 Crawl with AI</a> |
|
||||
{% endif %}
|
||||
<a href="https://github.com/ArchiveBox/archivebox-browser-extension"><img src="{% static 'chrome_extension_icon.png' %}" alt="Chrome Extension" style="height: 33px; margin-top: -5px;"> Get the extension</a>
|
||||
</span>
|
||||
💡 <strong>Tip:</strong> Instantly save a single URL by visiting:
|
||||
@@ -372,7 +374,6 @@
|
||||
if (sameDomainToggleSlot && shortcutToggles.length) {
|
||||
shortcutToggles.forEach((toggle) => sameDomainToggleSlot.appendChild(toggle));
|
||||
}
|
||||
const pluginDependencyMap = JSON.parse('{{ plugin_dependency_map_json|default:"{}"|escapejs }}');
|
||||
const personaConfigMap = JSON.parse('{{ persona_config_map_json|default:"{}"|escapejs }}');
|
||||
const personaSelect = document.querySelector('select[name="persona"]');
|
||||
|
||||
@@ -404,22 +405,6 @@
|
||||
return document.querySelector(`.plugin-section-toggle[value="${requiredSearchPlugin}"]`);
|
||||
}
|
||||
|
||||
function getPluginCheckbox(pluginName) {
|
||||
if (!pluginName) return null;
|
||||
return document.querySelector(`.plugin-section-toggle[value="${pluginName}"]`);
|
||||
}
|
||||
|
||||
function getRequiredPlugins(pluginName) {
|
||||
const requiredPlugins = pluginDependencyMap[pluginName];
|
||||
return Array.isArray(requiredPlugins) ? requiredPlugins : [];
|
||||
}
|
||||
|
||||
function getDependentPlugins(pluginName) {
|
||||
return Object.entries(pluginDependencyMap)
|
||||
.filter(([, requiredPlugins]) => Array.isArray(requiredPlugins) && requiredPlugins.includes(pluginName))
|
||||
.map(([dependentPlugin]) => dependentPlugin);
|
||||
}
|
||||
|
||||
function getConfigEditorRows() {
|
||||
return document.getElementById('id_config_rows');
|
||||
}
|
||||
@@ -555,47 +540,6 @@
|
||||
updateURLPreview();
|
||||
}
|
||||
|
||||
function ensureRequiredPluginsChecked(pluginName, visited = new Set()) {
|
||||
if (!pluginName || visited.has(pluginName)) {
|
||||
return;
|
||||
}
|
||||
visited.add(pluginName);
|
||||
|
||||
getRequiredPlugins(pluginName).forEach(requiredPlugin => {
|
||||
const requiredCheckbox = getPluginCheckbox(requiredPlugin);
|
||||
if (!requiredCheckbox) {
|
||||
return;
|
||||
}
|
||||
requiredCheckbox.checked = true;
|
||||
dispatchChange(requiredCheckbox);
|
||||
ensureRequiredPluginsChecked(requiredPlugin, visited);
|
||||
});
|
||||
}
|
||||
|
||||
function uncheckDependentPlugins(pluginName, visited = new Set()) {
|
||||
if (!pluginName || visited.has(pluginName)) {
|
||||
return;
|
||||
}
|
||||
visited.add(pluginName);
|
||||
|
||||
getDependentPlugins(pluginName).forEach(dependentPlugin => {
|
||||
const dependentCheckbox = getPluginCheckbox(dependentPlugin);
|
||||
if (!dependentCheckbox) {
|
||||
return;
|
||||
}
|
||||
dependentCheckbox.checked = false;
|
||||
dispatchChange(dependentCheckbox);
|
||||
uncheckDependentPlugins(dependentPlugin, visited);
|
||||
});
|
||||
}
|
||||
|
||||
function normalizePluginSelections() {
|
||||
const checkedPlugins = Array.from(document.querySelectorAll('.plugin-section-toggle:checked'))
|
||||
.map(cb => cb.value)
|
||||
.filter(Boolean);
|
||||
checkedPlugins.forEach(pluginName => ensureRequiredPluginsChecked(pluginName));
|
||||
}
|
||||
|
||||
function applyRequiredSearchPlugin(preferredValue = undefined) {
|
||||
const requiredCheckbox = getRequiredSearchCheckbox();
|
||||
if (!requiredCheckbox) return;
|
||||
@@ -1245,7 +1189,6 @@
|
||||
});
|
||||
}
|
||||
|
||||
normalizePluginSelections();
|
||||
applyRequiredSearchPlugin(requiredSearchPreference);
|
||||
syncEnabledPluginsConfig();
|
||||
updateChromeToggleButton();
|
||||
@@ -1279,7 +1222,6 @@
|
||||
dispatchChange(cb);
|
||||
});
|
||||
|
||||
normalizePluginSelections();
|
||||
applyRequiredSearchPlugin(requiredSearchPreference);
|
||||
syncEnabledPluginsConfig();
|
||||
updateChromeToggleButton();
|
||||
@@ -1289,11 +1231,6 @@
|
||||
|
||||
document.querySelectorAll('.plugin-section-toggle').forEach(checkbox => {
|
||||
checkbox.addEventListener('change', function() {
|
||||
if (this.checked) {
|
||||
ensureRequiredPluginsChecked(this.value);
|
||||
} else {
|
||||
uncheckDependentPlugins(this.value);
|
||||
}
|
||||
syncEnabledPluginsConfig();
|
||||
updateChromeToggleButton();
|
||||
saveFormState();
|
||||
@@ -1348,7 +1285,6 @@
|
||||
if (personaSelect) {
|
||||
applyPersonaConfig(personaSelect.value);
|
||||
}
|
||||
normalizePluginSelections();
|
||||
applyRequiredSearchPlugin();
|
||||
syncEnabledPluginsConfig();
|
||||
updateChromeToggleButton();
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
<a href="{% url 'Home' %}">Snapshots</a> |
|
||||
<a href="/admin/core/archiveresult/?o=-1">Log</a> |
|
||||
<a href="/admin/core/tag/">Tags</a>
|
||||
{% if user.is_authenticated and user.is_superuser %}
|
||||
{% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
|
||||
<a href="/admin/agent">💬 AI</a> |
|
||||
{% endif %}
|
||||
<a href="{% url 'Docs' %}" target="_blank" rel="noopener noreferrer">Docs</a> |
|
||||
|
||||
@@ -228,6 +228,8 @@ def run_archivebox_cmd(
|
||||
"SAVE_HTMLTOTEXT": "False",
|
||||
},
|
||||
)
|
||||
if args and args[0] == "version":
|
||||
run_env["PLUGINS"] = "__archivebox_test_no_plugins__"
|
||||
if env:
|
||||
run_env.update(env)
|
||||
_set_test_source_pythonpath(run_env)
|
||||
@@ -920,7 +922,7 @@ def _wait_for_archivebox_workers(cwd: Path, env: dict[str, str], names: tuple[st
|
||||
state: dict[str, Any] = {}
|
||||
while time.time() < deadline:
|
||||
state = _archivebox_worker_state(cwd, env)
|
||||
if all(state.get(name, {}).get("statename") == "RUNNING" for name in names):
|
||||
if all(isinstance(state.get(name), dict) and state[name].get("statename") == "RUNNING" for name in names):
|
||||
return state
|
||||
time.sleep(1)
|
||||
return state
|
||||
|
||||
@@ -475,11 +475,14 @@ def test_update_index_only_runs_paused_search_rows_and_resume_later_runs_crawl(t
|
||||
|
||||
if snapshot_finished_before_pause:
|
||||
indexed_state = get_crawl_runtime_state(tmp_path, crawl_id)
|
||||
assert indexed_state["crawl_status"] == "paused"
|
||||
assert indexed_state["crawl_retry_at"] == indexed_state["retry_at_max"]
|
||||
assert indexed_state["snapshots"][0]["status"] == "sealed"
|
||||
wget_results = [result for result in indexed_state["results"] if result["plugin"] == "wget"]
|
||||
assert any(result["status"] == "succeeded" and result["output_size"] > 0 for result in wget_results)
|
||||
if indexed_state["crawl_status"] == "paused":
|
||||
assert indexed_state["crawl_retry_at"] == indexed_state["retry_at_max"]
|
||||
else:
|
||||
assert indexed_state["crawl_status"] == "sealed"
|
||||
assert all(result["status"] not in {"queued", "started", "paused"} for result in indexed_state["results"])
|
||||
captured_text = wait_for_snapshot_capture(tmp_path, recursive_test_site["root_url"], timeout=60)
|
||||
assert "Root" in captured_text
|
||||
assert "About" in captured_text
|
||||
|
||||
@@ -162,13 +162,17 @@ def wait_for_import_processing(cwd: Path, expected_urls: set[str], *, timeout: f
|
||||
import time
|
||||
|
||||
deadline = time.time() + timeout
|
||||
counts = {url: 0 for url in expected_urls}
|
||||
while time.time() < deadline:
|
||||
with use_archivebox_db(cwd):
|
||||
snapshot_started = Snapshot.objects.filter(url__in=expected_urls).exists()
|
||||
if snapshot_started:
|
||||
rows = list(Snapshot.objects.filter(url__in=expected_urls).values_list("url", flat=True))
|
||||
counts = {url: 0 for url in expected_urls}
|
||||
for url in rows:
|
||||
counts[url] += 1
|
||||
if all(count >= 1 for count in counts.values()):
|
||||
return
|
||||
time.sleep(1)
|
||||
raise AssertionError("timed out waiting for import crawl processing to start")
|
||||
raise AssertionError(f"timed out waiting for import crawl processing to start, got counts={counts}")
|
||||
|
||||
|
||||
def wait_for_expected_import_snapshots(cwd: Path, expected_urls: set[str], *, timeout: float = 180.0) -> None:
|
||||
|
||||
@@ -228,7 +228,7 @@ SEARCH_BACKEND_PASSWORD = "SecretPassword"
|
||||
)
|
||||
|
||||
result = run_archivebox_cmd(
|
||||
["version"],
|
||||
["version", "--binaries=curl"],
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
@@ -991,6 +991,42 @@ class TestRecoverOrchestratorState:
|
||||
assert hook_names
|
||||
assert all(not hook_name.endswith((".py", ".js", ".sh")) for hook_name in hook_names)
|
||||
|
||||
def test_snapshot_hooks_for_pending_archiveresults_respects_disabled_plugins_when_plugins_empty(self):
|
||||
from django.utils import timezone
|
||||
|
||||
from archivebox.base_models.models import get_or_create_system_user_pk
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.services.runner import snapshot_hooks_for_pending_archiveresults
|
||||
|
||||
crawl = Crawl.objects.create(
|
||||
urls="https://example.com",
|
||||
created_by_id=get_or_create_system_user_pk(),
|
||||
config={
|
||||
"CLAUDECHROME_ENABLED": False,
|
||||
"CLAUDECODEEXTRACT_ENABLED": False,
|
||||
"CLAUDECODECLEANUP_ENABLED": False,
|
||||
"SEARCH_BACKEND_SQLITE_ENABLED": False,
|
||||
},
|
||||
status=Crawl.StatusChoices.STARTED,
|
||||
retry_at=timezone.now(),
|
||||
)
|
||||
snapshot = Snapshot.objects.create(
|
||||
url="https://example.com",
|
||||
crawl=crawl,
|
||||
status=Snapshot.StatusChoices.QUEUED,
|
||||
retry_at=timezone.now(),
|
||||
)
|
||||
|
||||
hooks = snapshot_hooks_for_pending_archiveresults(snapshot)
|
||||
queued_plugins = {plugin for plugin, _hook_name in hooks}
|
||||
|
||||
assert "title" in queued_plugins
|
||||
assert "claudechrome" not in queued_plugins
|
||||
assert "claudecodeextract" not in queued_plugins
|
||||
assert "claudecodecleanup" not in queued_plugins
|
||||
assert "search_backend_sqlite" not in queued_plugins
|
||||
|
||||
def test_run_due_snapshot_pauses_child_when_parent_is_paused(self):
|
||||
from django.utils import timezone
|
||||
|
||||
@@ -1467,6 +1503,73 @@ class TestRecoverOrchestratorState:
|
||||
assert result.status == ArchiveResult.StatusChoices.FAILED
|
||||
assert snapshot.retry_at is None
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
def test_run_due_snapshot_skips_disabled_queued_plugins_and_seals_started_snapshot(self):
|
||||
from django.utils import timezone
|
||||
|
||||
from archivebox.base_models.models import get_or_create_system_user_pk
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.services.runner import run_due_snapshot
|
||||
|
||||
crawl = Crawl.objects.create(
|
||||
urls="https://example.com",
|
||||
created_by_id=get_or_create_system_user_pk(),
|
||||
config={
|
||||
"CLAUDECHROME_ENABLED": False,
|
||||
"CLAUDECODEEXTRACT_ENABLED": False,
|
||||
"CLAUDECODECLEANUP_ENABLED": False,
|
||||
"SEARCH_BACKEND_SQLITE_ENABLED": False,
|
||||
},
|
||||
status=Crawl.StatusChoices.STARTED,
|
||||
retry_at=timezone.now(),
|
||||
)
|
||||
snapshot = Snapshot.objects.create(
|
||||
url="https://example.com",
|
||||
crawl=crawl,
|
||||
status=Snapshot.StatusChoices.STARTED,
|
||||
retry_at=timezone.now(),
|
||||
downloaded_at=timezone.now(),
|
||||
)
|
||||
ArchiveResult.objects.create(
|
||||
snapshot=snapshot,
|
||||
plugin="title",
|
||||
hook_name="on_Snapshot__01_title",
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
output_str="Example Domain",
|
||||
)
|
||||
stale_results = [
|
||||
ArchiveResult.objects.create(
|
||||
snapshot=snapshot,
|
||||
plugin=plugin,
|
||||
hook_name=hook_name,
|
||||
status=ArchiveResult.StatusChoices.QUEUED,
|
||||
)
|
||||
for plugin, hook_name in (
|
||||
("claudechrome", "on_Snapshot__47_claudechrome"),
|
||||
("claudecodeextract", "on_Snapshot__58_claudecodeextract"),
|
||||
("claudecodecleanup", "on_Snapshot__92_claudecodecleanup"),
|
||||
("search_backend_sqlite", "on_Snapshot__90_index_sqlite"),
|
||||
)
|
||||
]
|
||||
stale_result_ids = [result.id for result in stale_results]
|
||||
stale_plugins = [result.plugin for result in stale_results]
|
||||
|
||||
assert run_due_snapshot(snapshot, lock_seconds=60) is True
|
||||
|
||||
snapshot.refresh_from_db()
|
||||
assert snapshot.status == Snapshot.StatusChoices.SEALED
|
||||
assert snapshot.retry_at is None
|
||||
assert not snapshot.archiveresult_set.filter(
|
||||
plugin__in=stale_plugins,
|
||||
status=ArchiveResult.StatusChoices.QUEUED,
|
||||
).exists()
|
||||
for result in ArchiveResult.objects.filter(id__in=stale_result_ids):
|
||||
assert result.status == ArchiveResult.StatusChoices.SKIPPED
|
||||
assert result.start_ts is not None
|
||||
assert result.end_ts is not None
|
||||
assert "disabled by this Snapshot/Crawl config" in result.output_str
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
@pytest.mark.timeout(300)
|
||||
@pytest.mark.parametrize("chrome_isolation", ["crawl", "snapshot"])
|
||||
|
||||
@@ -10,7 +10,7 @@ import tempfile
|
||||
from pathlib import Path
|
||||
from archivebox.config.paths import tmp_dir_socket_path_is_short_enough
|
||||
from archivebox.cli.archivebox_version import _binary_row_dedupe_key
|
||||
from archivebox.tests.conftest import run_archivebox_cmd
|
||||
from archivebox.tests.conftest import cli_env, run_archivebox_cmd
|
||||
|
||||
|
||||
def _make_deep_collection_dir(tmp_path: Path) -> Path:
|
||||
@@ -126,6 +126,83 @@ def test_version_shows_binaries_after_init(tmp_path, initialized_archive):
|
||||
assert "Binary" in output or "Dependencies" in output
|
||||
|
||||
|
||||
def test_version_skips_disabled_plugin_binary_resolution(tmp_path):
|
||||
"""Disabled plugins should not trigger live binary detection during version."""
|
||||
data_dir = tmp_path / "no-plugins"
|
||||
data_dir.mkdir()
|
||||
env = cli_env(PLUGINS="__archivebox_test_no_plugins__", SEARCH_BACKEND_ENGINE="")
|
||||
|
||||
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env)
|
||||
assert init_result.returncode == 0, init_result.stderr
|
||||
|
||||
version_result = run_archivebox_cmd(["version"], cwd=data_dir, env=env)
|
||||
output = version_result.stdout + version_result.stderr
|
||||
|
||||
assert version_result.returncode == 0, output
|
||||
assert "No required binaries declared for discovered plugins" in output
|
||||
assert "not installed" not in output
|
||||
|
||||
|
||||
def test_version_honors_legacy_save_aliases_when_disabling_extractors(tmp_path):
|
||||
"""Legacy SAVE_* aliases should disable their canonical plugin flags."""
|
||||
data_dir = tmp_path / "legacy-save-disabled"
|
||||
data_dir.mkdir()
|
||||
env = {"PLUGINS": "wget,title", "SEARCH_BACKEND_ENGINE": ""}
|
||||
|
||||
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env, default_cli_env=True, disable_extractors=True, timeout=120)
|
||||
assert init_result.returncode == 0, init_result.stderr or init_result.stdout
|
||||
|
||||
config_result = run_archivebox_cmd(
|
||||
["config", "--get", "WGET_ENABLED", "TITLE_ENABLED", "CHROME_ENABLED"],
|
||||
cwd=data_dir,
|
||||
env=env,
|
||||
default_cli_env=True,
|
||||
disable_extractors=True,
|
||||
)
|
||||
config_output = config_result.stdout + config_result.stderr
|
||||
assert config_result.returncode == 0, config_output
|
||||
assert "WGET_ENABLED = false" in config_output
|
||||
assert "TITLE_ENABLED = false" in config_output
|
||||
assert "CHROME_ENABLED = false" in config_output
|
||||
|
||||
version_result = run_archivebox_cmd(
|
||||
["version"],
|
||||
cwd=data_dir,
|
||||
env=env,
|
||||
default_cli_env=True,
|
||||
disable_extractors=True,
|
||||
timeout=60,
|
||||
)
|
||||
output = version_result.stdout + version_result.stderr
|
||||
|
||||
assert version_result.returncode == 0, output
|
||||
assert "not installed" not in output
|
||||
|
||||
|
||||
def test_plugins_selection_includes_required_plugins_via_config_cli(tmp_path):
|
||||
"""PLUGINS selection should include transitive plugin dependencies exactly once."""
|
||||
data_dir = tmp_path / "plugin-dependencies"
|
||||
data_dir.mkdir()
|
||||
env = {"PLUGINS": "wget,screenshot"}
|
||||
|
||||
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env, default_cli_env=True, timeout=120)
|
||||
assert init_result.returncode == 0, init_result.stderr or init_result.stdout
|
||||
|
||||
config_result = run_archivebox_cmd(
|
||||
["config", "--get", "WGET_ENABLED", "SCREENSHOT_ENABLED", "CHROME_ENABLED", "YTDLP_ENABLED"],
|
||||
cwd=data_dir,
|
||||
env=env,
|
||||
default_cli_env=True,
|
||||
)
|
||||
config_output = config_result.stdout + config_result.stderr
|
||||
|
||||
assert config_result.returncode == 0, config_output
|
||||
assert "WGET_ENABLED = true" in config_output
|
||||
assert "SCREENSHOT_ENABLED = true" in config_output
|
||||
assert "CHROME_ENABLED = true" in config_output
|
||||
assert "YTDLP_ENABLED = false" in config_output
|
||||
|
||||
|
||||
def test_version_shows_data_locations(tmp_path, initialized_archive):
|
||||
"""Test that version shows data directory locations."""
|
||||
result = run_archivebox_cmd(["version"])
|
||||
|
||||
@@ -154,7 +154,7 @@ print(json.dumps({{
|
||||
retained = json.loads(stdout.strip().splitlines()[-1])
|
||||
assert retained["crawl_id"] == created["crawl_id"]
|
||||
|
||||
_cmd_result = run_archivebox_cmd(["run", "--crawl-id", retained["crawl_id"]], cwd=tmp_path, timeout=120, env=run_env)
|
||||
_cmd_result = run_archivebox_cmd(["run", "--maintenance-only"], cwd=tmp_path, timeout=120, env=run_env)
|
||||
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
|
||||
assert returncode == 0, f"archivebox run failed:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
|
||||
|
||||
|
||||
@@ -5,8 +5,8 @@ from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from urllib.parse import quote
|
||||
|
||||
import psutil
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from archivebox.tests.conftest import ADMIN_TEST_HOST, run_archivebox_cmd
|
||||
|
||||
@@ -54,7 +54,6 @@ def opencode_archive_config(initialized_archive):
|
||||
{
|
||||
"ABXPKG_INSTALL_TIMEOUT": "900",
|
||||
"ABXPKG_MIN_RELEASE_AGE": "0",
|
||||
"ABX_RUNTIME": "archivebox",
|
||||
"ARCHIVEBOX_ALLOW_NO_UNIX_SOCKETS": "true",
|
||||
"OPENCODE_ENABLED": "True",
|
||||
"OPENCODE_HOST": "127.0.0.1",
|
||||
@@ -175,10 +174,11 @@ def test_opencode_agent_superuser_gets_admin_wrapper(admin_client, live_opencode
|
||||
from abx_plugins.plugins.opencode import views
|
||||
|
||||
response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
|
||||
recent_session_id = views._recent_session_id(live_opencode.settings)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert f'<iframe src="{views._project_route(live_opencode.config.data_dir)}'.encode() in response.content
|
||||
assert b'/session"' in response.content
|
||||
assert recent_session_id
|
||||
assert f'<iframe src="{views._project_route(live_opencode.config.data_dir, recent_session_id)}"'.encode() in response.content
|
||||
assert b'id="header"' in response.content
|
||||
assert b'id="progress-monitor"' in response.content
|
||||
|
||||
@@ -225,20 +225,24 @@ def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
def test_opencode_starts_with_data_dir_cwd_and_isolated_state(live_opencode):
|
||||
process = psutil.Process(live_opencode.process.pid)
|
||||
env = process.environ()
|
||||
def test_opencode_starts_with_isolated_state(live_opencode):
|
||||
workdir = str(live_opencode.config.data_dir.resolve())
|
||||
state_dir = live_opencode.config.state_dir
|
||||
|
||||
assert Path(process.cwd()).resolve() == live_opencode.config.data_dir.resolve()
|
||||
assert env["BROWSER"] == "false"
|
||||
assert env["HOME"] == str(live_opencode.config.state_dir / "home")
|
||||
assert env["XDG_CONFIG_HOME"] == str(live_opencode.config.state_dir / "config")
|
||||
assert env["XDG_DATA_HOME"] == str(live_opencode.config.state_dir / "data")
|
||||
assert env["XDG_STATE_HOME"] == str(live_opencode.config.state_dir / "state")
|
||||
assert env["XDG_CACHE_HOME"] == str(live_opencode.config.state_dir / "cache")
|
||||
assert env["OPENCODE_DISABLE_PROJECT_CONFIG"] == "true"
|
||||
assert env["GIT_CEILING_DIRECTORIES"] == workdir
|
||||
project = requests.get(
|
||||
f"{live_opencode.settings['origin']}/project/current",
|
||||
params={"directory": workdir},
|
||||
timeout=live_opencode.settings["timeout"],
|
||||
)
|
||||
project.raise_for_status()
|
||||
|
||||
assert Path(live_opencode.settings["workdir"]).resolve() == Path(workdir)
|
||||
assert Path(str(project.json()["worktree"])).resolve() == Path(workdir)
|
||||
assert live_opencode.process.poll() is None
|
||||
assert (state_dir / "data" / "opencode" / "opencode.db").is_file()
|
||||
assert (state_dir / "SKILL.md").is_file()
|
||||
assert (state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md").resolve() == state_dir / "SKILL.md"
|
||||
assert not (live_opencode.config.data_dir / ".git" / "not-a-git").exists()
|
||||
|
||||
|
||||
def test_opencode_state_dir_is_separate_from_workdir(tmp_path):
|
||||
|
||||
@@ -484,9 +484,13 @@ def test_add_archivewebpage_installs_required_chrome_dependency(initialized_arch
|
||||
assert "archivewebpage" in binaries
|
||||
assert binaries["archivewebpage"]["status"] == Binary.StatusChoices.INSTALLED
|
||||
assert binaries["archivewebpage"]["binprovider"] == "chromewebstore"
|
||||
archivewebpage_manifest = Path(binaries["archivewebpage"]["abspath"])
|
||||
archivewebpage_metadata = Path(binaries["archivewebpage"]["abspath"])
|
||||
assert archivewebpage_metadata.exists()
|
||||
assert archivewebpage_metadata.name == "archivewebpage.extension.json"
|
||||
archivewebpage_extension = json.loads(archivewebpage_metadata.read_text(encoding="utf-8"))
|
||||
archivewebpage_manifest = Path(archivewebpage_extension["unpacked_path"]) / "manifest.json"
|
||||
assert archivewebpage_manifest.exists()
|
||||
assert archivewebpage_manifest.name == "manifest.json"
|
||||
assert json.loads(archivewebpage_manifest.read_text(encoding="utf-8"))["version"] == binaries["archivewebpage"]["version"]
|
||||
|
||||
plugins_seen = {plugin for plugin, _hook_name, _status, _output_str, _output_files in archive_results}
|
||||
assert {"chrome", "archivewebpage"}.issubset(plugins_seen)
|
||||
@@ -521,9 +525,7 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(initia
|
||||
from abx_dl.models import discover_plugins
|
||||
|
||||
root_url = "https://example.com/"
|
||||
plugin_selection = ",".join(
|
||||
sorted(plugin for plugin in discover_plugins().keys() if not plugin.startswith("claude")),
|
||||
)
|
||||
plugin_selection = ",".join(sorted(plugin for plugin in discover_plugins().keys() if not plugin.startswith("claude")))
|
||||
env = os.environ.copy()
|
||||
env.pop("CHROME_BINARY", None)
|
||||
env.update(
|
||||
|
||||
@@ -590,7 +590,7 @@ class TestPublicIndexSearch:
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
assert "/screenshot.png" in content
|
||||
assert "screenshot/screenshot.png" not in content
|
||||
assert "/screenshot/screenshot.png" not in content
|
||||
|
||||
@override_settings(PUBLIC_INDEX=True)
|
||||
def test_public_index_preview_respects_plugin_relative_screenshot_output(self, client, public_snapshot):
|
||||
@@ -634,7 +634,7 @@ class TestPublicIndexSearch:
|
||||
first = content.index("chrome_extension_screenshot/screenshot-1.png")
|
||||
second = content.index("chrome_extension_screenshot/screenshot.png")
|
||||
assert first < second
|
||||
assert "screenshot/screenshot.png" not in content
|
||||
assert "/screenshot/screenshot.png" not in content
|
||||
assert "chrome_extension_screenshot/screenshot-2.png" not in content
|
||||
|
||||
@override_settings(PUBLIC_INDEX=True)
|
||||
|
||||
@@ -99,6 +99,19 @@ def test_add_view_admin_renders_plugin_config_grid(client, admin_user, monkeypat
|
||||
assert b"plugin_config__wget__WGET_ENABLED" not in response.content
|
||||
|
||||
|
||||
def test_add_view_hides_agent_link_when_opencode_is_disabled(client, admin_user):
|
||||
from archivebox.machine.models import Machine
|
||||
|
||||
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
|
||||
client.force_login(admin_user)
|
||||
|
||||
response = client.get(reverse("add"), HTTP_HOST=ADMIN_HOST)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert b"/admin/agent" not in response.content
|
||||
assert b"Crawl with AI" not in response.content
|
||||
|
||||
|
||||
def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
|
||||
staff_user = User.objects.create_user(
|
||||
username="addviewstaff",
|
||||
|
||||
@@ -344,6 +344,97 @@ def _wait_for_worker_state(cwd, worker_name: str, statename: str, timeout: int =
|
||||
raise AssertionError(f"Timed out waiting for {worker_name}={statename}, last state={state}")
|
||||
|
||||
|
||||
@pytest.mark.timeout(240)
|
||||
def test_public_add_view_depth_one_crawl_skips_unreadable_persona_profile_entries(tmp_path, recursive_test_site):
|
||||
init_archive(tmp_path)
|
||||
|
||||
chrome_profile = tmp_path / "personas" / "Default" / "chrome_profile"
|
||||
default_profile = chrome_profile / "Default"
|
||||
default_profile.mkdir(parents=True, exist_ok=True)
|
||||
(default_profile / "Preferences").write_text('{"profile":{"name":"Default"}}', encoding="utf-8")
|
||||
unreadable_dir = chrome_profile / "ActorSafetyLists"
|
||||
unreadable_dir.mkdir(parents=True, exist_ok=True)
|
||||
(unreadable_dir / "Preferences").write_text("{}", encoding="utf-8")
|
||||
unreadable_dir.chmod(0)
|
||||
|
||||
port = get_free_port()
|
||||
env = cli_env(
|
||||
port=port,
|
||||
PLUGINS="wget,parse_html_urls",
|
||||
SAVE_WGET="True",
|
||||
PUBLIC_INDEX="True",
|
||||
PUBLIC_ADD_VIEW="True",
|
||||
URL_ALLOWLIST=r"127\.0\.0\.1[:/].*",
|
||||
)
|
||||
|
||||
try:
|
||||
start_archivebox_server(tmp_path, env=env, port=port)
|
||||
add_page = wait_for_http(port, host=f"web.archivebox.localhost:{port}", path="/add/")
|
||||
assert add_page.status_code == 200
|
||||
assert 'name="depth"' in add_page.text
|
||||
|
||||
response = requests.post(
|
||||
f"http://127.0.0.1:{port}/add/",
|
||||
headers={"Host": f"web.archivebox.localhost:{port}", "Referer": f"http://web.archivebox.localhost:{port}/add/"},
|
||||
data={
|
||||
"url": recursive_test_site["root_url"],
|
||||
"depth": "1",
|
||||
"max_urls": "10",
|
||||
"crawl_max_size": "0",
|
||||
"snapshot_max_size": "0",
|
||||
"tag": "public-depth-one-unreadable-profile",
|
||||
"url_filters_allowlist": r"127\.0\.0\.1[:/].*",
|
||||
"url_filters_denylist": "",
|
||||
"schedule": "",
|
||||
"notes": "public depth one with unreadable chrome profile entries",
|
||||
"persona": "Default",
|
||||
"permissions": "public",
|
||||
"start_paused": "",
|
||||
"config": "{}",
|
||||
},
|
||||
timeout=10,
|
||||
allow_redirects=False,
|
||||
)
|
||||
assert response.status_code in (302, 303), response.text
|
||||
|
||||
deadline = time.time() + 180
|
||||
while time.time() < deadline:
|
||||
with use_archivebox_db(tmp_path):
|
||||
crawl = Crawl.objects.order_by("-created_at").first()
|
||||
depth_counts = get_depth_counts(tmp_path)
|
||||
child_urls = set(Snapshot.objects.filter(depth=1).values_list("url", flat=True))
|
||||
failed_results = list(
|
||||
ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.FAILED).values_list("plugin", "output_str"),
|
||||
)
|
||||
if depth_counts.get(0, 0) >= 1 and set(recursive_test_site["child_urls"]).issubset(child_urls):
|
||||
break
|
||||
assert not failed_results
|
||||
time.sleep(2)
|
||||
else:
|
||||
raise AssertionError(f"timed out waiting for depth=1 crawl, got depth counts {get_depth_counts(tmp_path)}")
|
||||
|
||||
with use_archivebox_db(tmp_path):
|
||||
crawl = Crawl.objects.order_by("-created_at").first()
|
||||
snapshot_rows = list(Snapshot.objects.order_by("depth", "url").values_list("url", "depth", "status"))
|
||||
result_rows = list(ArchiveResult.objects.order_by("plugin", "status").values_list("plugin", "status", "output_size"))
|
||||
finally:
|
||||
unreadable_dir.chmod(0o700)
|
||||
stop_server(tmp_path)
|
||||
|
||||
assert crawl is not None
|
||||
assert crawl.max_depth == 1
|
||||
assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
|
||||
assert (recursive_test_site["root_url"], 0, Snapshot.StatusChoices.SEALED) in snapshot_rows
|
||||
assert set(recursive_test_site["child_urls"]).issubset({url for url, depth, _status in snapshot_rows if depth == 1})
|
||||
assert all(
|
||||
status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
|
||||
for _url, _depth, status in snapshot_rows
|
||||
)
|
||||
assert not [row for row in result_rows if row[1] == ArchiveResult.StatusChoices.FAILED]
|
||||
assert any(plugin == "wget" and status == ArchiveResult.StatusChoices.SUCCEEDED and size > 0 for plugin, status, size in result_rows)
|
||||
assert any(plugin == "parse_html_urls" and status == ArchiveResult.StatusChoices.SUCCEEDED for plugin, status, _size in result_rows)
|
||||
|
||||
|
||||
@pytest.mark.timeout(420)
|
||||
def test_public_add_view_import_text_formats_preserve_metadata_and_resume_without_duplicates(tmp_path):
|
||||
"""Public /add/ textarea should import rich text, survive runner restart, and preserve one row per URL."""
|
||||
@@ -516,9 +607,10 @@ def test_public_add_view_rejects_file_path_and_shell_injection_payloads(tmp_path
|
||||
with use_archivebox_db(tmp_path):
|
||||
snapshot = Snapshot.objects.get(url=safe_url)
|
||||
crawl = Crawl.objects.get()
|
||||
tag_names = set(snapshot.tags.values_list("name", flat=True))
|
||||
assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
|
||||
assert snapshot.status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
|
||||
assert "public-ui-security" in set(snapshot.tags.values_list("name", flat=True))
|
||||
assert "public-ui-security" in tag_names
|
||||
|
||||
|
||||
@pytest.mark.timeout(180)
|
||||
@@ -580,7 +672,10 @@ def test_add_view_post_creates_schedule_over_server(tmp_path, recursive_test_sit
|
||||
|
||||
with use_archivebox_db(tmp_path):
|
||||
schedule = CrawlSchedule.objects.select_related("template").order_by("-created_at").first()
|
||||
row = (schedule.schedule, schedule.template.urls, schedule.template.tags_str) if schedule else None
|
||||
row = None
|
||||
if schedule:
|
||||
template_url = json.loads(schedule.template.urls)["url"]
|
||||
row = (schedule.schedule, template_url, schedule.template.tags_str)
|
||||
|
||||
assert row == ("daily", recursive_test_site["root_url"], "web-ui")
|
||||
finally:
|
||||
|
||||
@@ -239,6 +239,19 @@ def test_snapshot_admin_zip_links():
|
||||
assert html.escape(zip_url, quote=True) in str(admin.admin_actions(snapshot))
|
||||
|
||||
|
||||
def test_admin_navigation_hides_agent_link_when_opencode_is_disabled(client, admin_user):
|
||||
from archivebox.machine.models import Machine
|
||||
|
||||
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
|
||||
client.force_login(admin_user)
|
||||
|
||||
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:8000")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert b"/admin/agent" not in response.content
|
||||
assert b">\xf0\x9f\x92\xac AI<" not in response.content
|
||||
|
||||
|
||||
def test_archiveresult_admin_zip_links():
|
||||
from archivebox.core.admin_archiveresults import ArchiveResultAdmin
|
||||
from archivebox.core.models import ArchiveResult
|
||||
|
||||
@@ -94,6 +94,28 @@ def test_discover_persona_template_profiles_finds_data_dir_personas():
|
||||
assert matching[0].user_data_dir.parent.parent == CONSTANTS.PERSONAS_DIR.resolve()
|
||||
|
||||
|
||||
def test_persona_admin_add_view_skips_unreadable_chrome_profile_entries(admin_client):
|
||||
from archivebox.config.constants import CONSTANTS
|
||||
|
||||
chrome_profile = CONSTANTS.PERSONAS_DIR / "UnreadableTemplatePersona" / "chrome_profile"
|
||||
default_profile = chrome_profile / "Default"
|
||||
default_profile.mkdir(parents=True, exist_ok=True)
|
||||
(default_profile / "Preferences").write_text('{"profile":{"name":"Default"}}')
|
||||
unreadable_dir = chrome_profile / "ActorSafetyLists"
|
||||
unreadable_dir.mkdir(parents=True, exist_ok=True)
|
||||
(unreadable_dir / "Preferences").write_text("{}")
|
||||
unreadable_dir.chmod(0)
|
||||
|
||||
try:
|
||||
response = admin_client.get(reverse("admin:personas_persona_add"), HTTP_HOST=ADMIN_TEST_HOST)
|
||||
finally:
|
||||
unreadable_dir.chmod(0o700)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert b"UnreadableTemplatePersona" in response.content
|
||||
assert b"Persona Template" in response.content
|
||||
|
||||
|
||||
def test_persona_admin_add_view_renders_import_ui(admin_client):
|
||||
source = _make_persona_template_source("RenderImportPersona")
|
||||
|
||||
|
||||
@@ -32,6 +32,25 @@ def test_snapshot_changelist_uses_stable_ordering_without_unordered_paginator_wa
|
||||
assert b"Searching matching snapshots..." in response.content
|
||||
|
||||
|
||||
def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client, snapshot):
|
||||
from archivebox.core.models import ArchiveResult
|
||||
|
||||
ArchiveResult.objects.create(
|
||||
snapshot=snapshot,
|
||||
plugin="screenshot",
|
||||
hook_name="on_Snapshot__40_screenshot.js",
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
output_size=128,
|
||||
output_files={"screenshot.png": {"size": 128, "root_relative": True}},
|
||||
)
|
||||
|
||||
response = admin_client.get(reverse("admin:core_snapshot_changelist"), HTTP_HOST=ADMIN_TEST_HOST)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert b'class="snapshot-preview' in response.content
|
||||
assert b"screenshot.png" in response.content
|
||||
|
||||
|
||||
def test_snapshot_admin_tag_editor_escapes_tag_json_script_breakout(admin_client, snapshot):
|
||||
from archivebox.core.models import Tag
|
||||
|
||||
@@ -188,10 +207,8 @@ def test_snapshot_admin_preview_uses_extension_screenshot_when_standard_screensh
|
||||
preview = admin._get_preview_data(snapshot)
|
||||
|
||||
assert preview is not None
|
||||
assert "screenshot/screenshot.png" in preview["img_url"]
|
||||
first = preview["fallback_list"].index("chrome_extension_screenshot/screenshot-1.png")
|
||||
second = preview["fallback_list"].index("chrome_extension_screenshot/screenshot.png")
|
||||
assert first < second
|
||||
assert "chrome_extension_screenshot/screenshot-1.png" in preview["img_url"]
|
||||
assert "chrome_extension_screenshot/screenshot.png" in preview["fallback_list"]
|
||||
assert "chrome_extension_screenshot/screenshot-2.png" not in preview["fallback_list"]
|
||||
|
||||
|
||||
|
||||
@@ -60,6 +60,9 @@ cat > .archivebox-deploy.override.yml <<EOF
|
||||
services:
|
||||
$DEPLOY_SERVICE:
|
||||
image: $DEPLOY_IMAGE
|
||||
user: ""
|
||||
entrypoint: null
|
||||
command: null
|
||||
EOF
|
||||
|
||||
COMPOSE=(docker compose -f "$COMPOSE_FILE" -f .archivebox-deploy.override.yml)
|
||||
|
||||
+44
-49
@@ -49,14 +49,22 @@ docker_pull_archivebox() {
|
||||
fi
|
||||
}
|
||||
|
||||
docker_run_archivebox_init() {
|
||||
docker_run_archivebox() {
|
||||
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
|
||||
docker run --platform "$ARCHIVEBOX_PLATFORM" "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" init --install
|
||||
docker run --platform "$ARCHIVEBOX_PLATFORM" "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" "$@"
|
||||
else
|
||||
docker run "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" init --install
|
||||
docker run "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
docker_run_archivebox_init() {
|
||||
docker_run_archivebox init
|
||||
}
|
||||
|
||||
docker_run_archivebox_install() {
|
||||
docker_run_archivebox install
|
||||
}
|
||||
|
||||
docker_run_archivebox_server() {
|
||||
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
|
||||
docker run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
|
||||
@@ -128,11 +136,7 @@ install_archivebox_with_uv() {
|
||||
|
||||
echo
|
||||
echo "[+] Installing ArchiveBox python tool using uv from $ARCHIVEBOX_PACKAGE..."
|
||||
if uv --no-config tool add --help > /dev/null 2>&1; then
|
||||
uv --no-config tool add --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
|
||||
else
|
||||
uv --no-config tool install --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
|
||||
fi
|
||||
uv --no-config tool install --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
|
||||
|
||||
uv_tool_bin_dir="$(uv --no-config tool dir --bin)"
|
||||
export PATH="$uv_tool_bin_dir:$PATH"
|
||||
@@ -160,7 +164,8 @@ if (command -v docker > /dev/null && docker compose version > /dev/null && docke
|
||||
fi
|
||||
curl -fsSL "$ARCHIVEBOX_COMPOSE_URL" > docker-compose.yml
|
||||
export ARCHIVEBOX_IMAGE ARCHIVEBOX_PLATFORM
|
||||
docker_compose_run_archivebox init --install
|
||||
docker_compose_run_archivebox init
|
||||
docker_compose_run_archivebox install
|
||||
echo
|
||||
echo "[+] Starting ArchiveBox server using: docker compose up -d..."
|
||||
docker compose up -d
|
||||
@@ -187,8 +192,9 @@ elif (command -v docker > /dev/null && docker_pull_archivebox); then
|
||||
fi
|
||||
cd ./data
|
||||
docker_run_archivebox_init
|
||||
docker_run_archivebox_install
|
||||
echo
|
||||
echo "[+] Starting ArchiveBox server using: docker run -d archivebox/archivebox..."
|
||||
echo "[+] Starting ArchiveBox server using: docker run -d $ARCHIVEBOX_IMAGE..."
|
||||
docker_run_archivebox_server
|
||||
wait_for_archivebox
|
||||
open_archivebox
|
||||
@@ -213,21 +219,20 @@ echo " ⚠️ If you want to use Docker, press [Ctrl-C] to cancel now. ⚠️
|
||||
echo " Get Docker: https://docs.docker.com/get-docker/"
|
||||
echo " After you've installed Docker, run this script again."
|
||||
echo
|
||||
echo "Otherwise, install will continue with apt/brew/pkg + uv in 12s... (press [Ctrl+C] to cancel)"
|
||||
echo "Otherwise, install will continue with uv in 12s... (press [Ctrl+C] to cancel)"
|
||||
echo
|
||||
sleep 12 || exit 1
|
||||
echo "Proceeding with system package manager..."
|
||||
echo "Proceeding with uv..."
|
||||
echo
|
||||
|
||||
echo "[i] ArchiveBox Setup Script 📦"
|
||||
echo
|
||||
echo " This is a helper script which installs ArchiveBox and bootstraps its Python/Node runtimes."
|
||||
echo " This is a helper script which installs ArchiveBox and bootstraps its runtime dependencies."
|
||||
echo " You may be prompted for a sudo password in order to install the following:"
|
||||
echo
|
||||
echo " - archivebox"
|
||||
echo " - python3, uv, nodejs, npm (languages used by ArchiveBox and plugin installers)"
|
||||
echo " - curl, wget (used to bootstrap package installation)"
|
||||
echo " - extractor/plugin dependencies (installed/discovered by archivebox init --install)"
|
||||
echo " - uv / curl / ca-certificates (as needed to bootstrap ArchiveBox)"
|
||||
echo " - archivebox (installed by uv with Python $ARCHIVEBOX_PYTHON)"
|
||||
echo " - extractor/plugin dependencies (installed/discovered by archivebox install)"
|
||||
echo
|
||||
echo " If you'd rather install these manually as-needed, you can find detailed documentation here:"
|
||||
echo " https://github.com/ArchiveBox/ArchiveBox/wiki/Install"
|
||||
@@ -235,43 +240,32 @@ echo
|
||||
echo "Continuing in 12s... (press [Ctrl+C] to cancel)"
|
||||
echo
|
||||
sleep 12 || exit 1
|
||||
echo "Proceeding to install dependencies..."
|
||||
echo "Proceeding to install ArchiveBox..."
|
||||
echo
|
||||
|
||||
# On Linux:
|
||||
if which apt-get > /dev/null; then
|
||||
echo "[+] Installing ArchiveBox system dependencies using apt..."
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y python3 python3-venv wget curl nodejs npm
|
||||
install_archivebox_with_uv
|
||||
# On Mac:
|
||||
elif which brew > /dev/null; then
|
||||
echo "[+] Installing ArchiveBox using Homebrew..."
|
||||
brew tap archivebox/archivebox
|
||||
brew update
|
||||
brew install archivebox
|
||||
elif which pkg > /dev/null; then
|
||||
echo "[+] Installing ArchiveBox system dependencies using pkg and uv..."
|
||||
sudo pkg install -y python3 py39-sqlite3 npm wget curl
|
||||
install_archivebox_with_uv
|
||||
else
|
||||
echo "[!] Warning: Could not find aptitude/homebrew/pkg! May not be able to install all dependencies automatically."
|
||||
echo
|
||||
echo " If you're on macOS, make sure you have homebrew installed: https://brew.sh/"
|
||||
echo " If you're on Linux, only Ubuntu/Debian/BSD systems are officially supported with this script."
|
||||
echo " If you're on Windows, this script is not officially supported (Docker is recommended instead)."
|
||||
echo
|
||||
echo "See the README.md for Manual Setup & Troubleshooting instructions if you you're unable to run ArchiveBox after this script completes."
|
||||
if ! command -v uv > /dev/null 2>&1 && ! command -v curl > /dev/null 2>&1 && ! command -v wget > /dev/null 2>&1; then
|
||||
if command -v apt-get > /dev/null 2>&1; then
|
||||
echo "[+] Installing curl and CA certificates to bootstrap uv using apt..."
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y curl ca-certificates
|
||||
elif command -v brew > /dev/null 2>&1; then
|
||||
echo "[+] Installing uv using Homebrew..."
|
||||
brew install uv
|
||||
elif command -v pkg > /dev/null 2>&1; then
|
||||
echo "[+] Installing curl and CA certificates to bootstrap uv using pkg..."
|
||||
sudo pkg install -y curl ca_root_nss
|
||||
else
|
||||
echo "[!] Warning: Could not find uv, curl, wget, apt, brew, or pkg."
|
||||
echo
|
||||
echo "See the README.md for Manual Setup & Troubleshooting instructions if you you're unable to run ArchiveBox after this script completes."
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
|
||||
if ! which archivebox > /dev/null 2>&1; then
|
||||
ensure_uv
|
||||
export PATH="$(uv --no-config tool dir --bin):$PATH"
|
||||
fi
|
||||
install_archivebox_with_uv
|
||||
|
||||
if ! which archivebox > /dev/null 2>&1; then
|
||||
if ! command -v archivebox > /dev/null 2>&1; then
|
||||
echo "[X] archivebox command was not found in PATH after installing!"
|
||||
echo " Check to see if a previous step failed."
|
||||
exit 1
|
||||
@@ -285,7 +279,8 @@ if [ -f "./index.sqlite3" ]; then
|
||||
mv -i ~/archivebox/* ~/archivebox/data/
|
||||
fi
|
||||
cd ./data
|
||||
: | archivebox init --install # pipe in empty command to make sure stdin is closed
|
||||
: | archivebox init # pipe in empty command to make sure stdin is closed
|
||||
archivebox install
|
||||
# init shows version output at the end too
|
||||
echo
|
||||
echo "[+] Starting ArchiveBox server using: nohup archivebox server &..."
|
||||
@@ -299,7 +294,7 @@ echo " archivebox server --quick-init 0.0.0.0:8000 # start server proc
|
||||
echo " archivebox manage createsuperuser # add an admin user+pass"
|
||||
echo " ps aux | grep archivebox # see server process pid"
|
||||
echo " pkill -f archivebox # stop the server"
|
||||
echo " uv tool install --python $ARCHIVEBOX_PYTHON --upgrade '$ARCHIVEBOX_PACKAGE'; archivebox init # update versions"
|
||||
echo " uv tool install --python $ARCHIVEBOX_PYTHON --upgrade '$ARCHIVEBOX_PACKAGE'; archivebox init; archivebox install # update versions"
|
||||
echo " archivebox add 'https://example.com'" # archive a new URL
|
||||
echo " archivebox list # see URLs archived"
|
||||
echo " archivebox help # see more help & examples"
|
||||
@@ -27,6 +27,9 @@ cat > "$OVERRIDE_FILE" <<EOF
|
||||
services:
|
||||
$DEPLOY_SERVICE:
|
||||
image: $DEPLOY_IMAGE
|
||||
user: ""
|
||||
entrypoint: null
|
||||
command: null
|
||||
EOF
|
||||
|
||||
COMPOSE=(docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE")
|
||||
|
||||
@@ -478,7 +478,7 @@ URL users are redirected to after logging out. The default `/` keeps users on Ar
|
||||
|
||||
### LDAP Settings
|
||||
|
||||
*Options for LDAP / Active Directory authentication via [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap). Requires `pip install archivebox[ldap]` (which also pulls in the system `libldap` / `libsasl` headers).*
|
||||
*Options for LDAP / Active Directory authentication via [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap). Requires `uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` (which also pulls in the system `libldap` / `libsasl` headers).*
|
||||
|
||||
---
|
||||
#### `LDAP_ENABLED`
|
||||
@@ -487,7 +487,7 @@ URL users are redirected to after logging out. The default `/` keeps users on Ar
|
||||
Master switch for LDAP authentication. When `True`, ArchiveBox loads the `django-auth-ldap` backend and validates that `LDAP_SERVER_URI`, `LDAP_BIND_DN`, `LDAP_BIND_PASSWORD`, and `LDAP_USER_BASE` are all set — startup fails fast otherwise.
|
||||
|
||||
```bash
|
||||
pip install archivebox[ldap]
|
||||
uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
```
|
||||
|
||||
Then set these configuration values:
|
||||
|
||||
+19
-20
@@ -24,17 +24,15 @@ Running ArchiveBox with Docker allows you to manage it in a container without ex
|
||||
|
||||
**Official Docker Hub image: [`hub.docker.com/r/archivebox/archivebox`](https://hub.docker.com/r/archivebox/archivebox)**
|
||||
```bash
|
||||
docker pull archivebox/archivebox:latest
|
||||
docker pull archivebox/archivebox:dev
|
||||
```
|
||||
|
||||
- [`Dockerfile`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile)
|
||||
- [`docker-compose.yml`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml)
|
||||
- [`archivebox-kubernetes.yml`](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml)
|
||||
|
||||
Published [Docker tags](https://hub.docker.com/r/archivebox/archivebox/tags):
|
||||
- `:latest`, `:stable` (latest stable release, the default)
|
||||
- `:x.x` and `:x.x.x` for specific versions (e.g. `:0.7` or `:0.7.2`)
|
||||
- `:dev` for unstable alpha builds (breaks often, only for developers and willing beta testers)
|
||||
- `:x.xrcN` and `:x.x.xrcN` for specific RC versions
|
||||
- `:sha-xxxxxxx` for builds of specific git commits (to test or pin specific PRs or commits)
|
||||
|
||||
<br/>
|
||||
@@ -63,7 +61,7 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
|
||||
|
||||
# download the compose file into the directory
|
||||
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
|
||||
# (shortcut for getting https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/docker-compose.yml)
|
||||
# (shortcut for getting https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml)
|
||||
|
||||
# initialize your collection and create an admin user for the Web UI (or set ADMIN_USERNAME/ADMIN_PASSWORD env vars)
|
||||
docker compose run archivebox init
|
||||
@@ -201,10 +199,11 @@ Set `BASE_URL=https://archive.example.com` in the `.env` file next to `docker-co
|
||||
Fetch and run the ArchiveBox Docker image to create your initial archive.
|
||||
|
||||
```bash
|
||||
docker pull archivebox/archivebox
|
||||
docker pull archivebox/archivebox:dev
|
||||
|
||||
mkdkir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -it -v $PWD:/data archivebox/archivebox init --setup
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev init
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev install
|
||||
```
|
||||
|
||||
*(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
|
||||
@@ -221,31 +220,31 @@ See the wiki page on [Upgrading or Merging Archives: Upgrading with plain Docker
|
||||
|
||||
### Usage
|
||||
|
||||
The Docker CLI `docker run ... archivebox/archivebox [subcommand]` works just like the non-Docker `archivebox [subcommand]` CLI.
|
||||
The Docker CLI `docker run ... archivebox/archivebox:dev [subcommand]` works just like the non-Docker `archivebox [subcommand]` CLI.
|
||||
|
||||
First, make sure you're `cd`'ed into your collection data folder (e.g. `~/archivebox/data`).
|
||||
|
||||
```bash
|
||||
docker run -it -v $PWD:/data archivebox/archivebox help
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
```
|
||||
|
||||
To add a single URL, pass it as an arg or pipe it in via stdin:
|
||||
```bash
|
||||
docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
|
||||
# OR
|
||||
echo 'https://example.com' | docker run -i -v $PWD:/data archivebox/archivebox add
|
||||
echo 'https://example.com' | docker run -i -v $PWD:/data archivebox/archivebox:dev add
|
||||
```
|
||||
|
||||
To archive multiple URLs at once, pass text containing URLs in via stdin:
|
||||
```bash
|
||||
docker run -i -v $PWD:/data archivebox/archivebox add < urls.txt
|
||||
docker run -i -v $PWD:/data archivebox/archivebox:dev add < urls.txt
|
||||
# OR
|
||||
curl 'https://example.com/some/rss/feed.xml' | docker run -i -v $PWD:/data archivebox/archivebox add
|
||||
curl 'https://example.com/some/rss/feed.xml' | docker run -i -v $PWD:/data archivebox/archivebox:dev add
|
||||
```
|
||||
|
||||
You can also use the `--depth=1` flag to tell ArchiveBox to recursively archive the URLs within a provided source.
|
||||
```bash
|
||||
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com/some/rss/feed.xml'
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com/some/rss/feed.xml'
|
||||
```
|
||||
|
||||
<br/>
|
||||
@@ -256,12 +255,12 @@ The `docker run` `-v /path/on/host:/path/inside/container` flag specifies where
|
||||
|
||||
For example to use a folder on an external USB drive (instead of the current directory `$PWD` or `~/archivebox/data`):
|
||||
```bash
|
||||
docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox ...
|
||||
docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox:dev ...
|
||||
```
|
||||
|
||||
Then to view your data, you can look in the folder on the host `/media/USB-DRIVE/archivebox/data`, or use the Web UI:
|
||||
```bash
|
||||
docker run -it -v /media/USB_DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox
|
||||
docker run -it -v /media/USB_DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
# then open https://127.0.0.1:8000
|
||||
```
|
||||
|
||||
@@ -273,7 +272,7 @@ The easiest way is to use `archivebox config --set KEY=value` or edit `./Archive
|
||||
|
||||
For example, this sets `TIMEOUT=120` as a persistent setting for the collection:
|
||||
```bash
|
||||
docker run -it -v $PWD:/data archivebox/archivebox config --set TIMEOUT=120
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev config --set TIMEOUT=120
|
||||
# OR
|
||||
echo 'TIMEOUT=120' >> ./ArchiveBox.conf
|
||||
```
|
||||
@@ -282,8 +281,8 @@ ArchiveBox in Docker also accepts config as environment variables, see more on t
|
||||
|
||||
For example, this disables the screenshot extractor for a single run (without persisting for other runs):
|
||||
```bash
|
||||
docker run -it -v $PWD:/data -e SCREENSHOT_ENABLED=False archivebox/archivebox add 'https://example.com'
|
||||
docker run -it -v $PWD:/data -e SCREENSHOT_ENABLED=False archivebox/archivebox:dev add 'https://example.com'
|
||||
# OR
|
||||
echo 'SCREENSHOT_ENABLED=False' >> ./.env
|
||||
docker run ... --env-file=./.env archivebox/archivebox ...
|
||||
docker run ... --env-file=./.env archivebox/archivebox:dev ...
|
||||
```
|
||||
+25
-25
@@ -1,6 +1,6 @@
|
||||
# Install
|
||||
|
||||
ArchiveBox is primarily distributed as a Python package via `pip`, but it also depends on some system packages that can be installed manually or automatically with Docker. It usually takes less than ~10min to get ArchiveBox set up and running.
|
||||
ArchiveBox is primarily distributed as a Python package installed with `uv`, but it also depends on some system packages that can be installed manually or automatically with Docker. It usually takes less than ~10min to get ArchiveBox set up and running.
|
||||
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/601d587d-b59f-47b9-938e-8a7fa7790176" width="20%" align="right"/>
|
||||
|
||||
@@ -59,12 +59,11 @@ It's also recommended to use a filesystem with compression and/or [deduplication
|
||||
|
||||
2. Then follow the [Quickstart](https://github.com/ArchiveBox/ArchiveBox#quickstart) guide and read the [[Docker]] wiki page for next steps. ➡️
|
||||
|
||||
> You can also run Dockerized ArchiveBox using [UNRAID/TrueNAS/Proxmox/etc.](https://github.com/ArchiveBox/ArchiveBox#-other-options) or [Kubernetes](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml).
|
||||
> You can also run Dockerized ArchiveBox using [UNRAID/TrueNAS/Proxmox/etc.](https://github.com/ArchiveBox/ArchiveBox#-other-options) or Kubernetes.
|
||||
|
||||
**More info:**
|
||||
- [`Dockerfile`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile)
|
||||
- [`docker-compose.yml`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml)
|
||||
- [`archivebox-kubernetes.yml`](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml)
|
||||
- [ArchiveBox Docker Quickstart](https://github.com/ArchiveBox/ArchiveBox#quickstart) + [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration) + [Upgrading](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives) documentation
|
||||
|
||||
<br/>
|
||||
@@ -82,9 +81,9 @@ If you're on Linux with `apt` or FreeBSD with `pkg` there is an optional auto-se
|
||||
|
||||
```bash
|
||||
curl -fsSL 'https://get.archivebox.io' | bash
|
||||
# shortcut to run https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/bin/setup.sh
|
||||
# shortcut to run https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/bin/setup.sh
|
||||
```
|
||||
The script explains what it installs beforehand, and will prompt for user confirmation before making any changes to your system. The script uses Docker if already installed, but you can decline and it will attempt to auto-install everything using `apt`/`brew`/`pkg` + `pip` instead.
|
||||
The script explains what it installs beforehand, and will prompt for user confirmation before making any changes to your system. The script uses Docker if already installed, but you can decline and it will install ArchiveBox using `uv` instead.
|
||||
|
||||
<img src="https://imgur.zervice.io/VMTzm0G.png" width="99%"/>
|
||||
|
||||
@@ -100,7 +99,7 @@ After running the setup script, continue with the [Quickstart](https://github.co
|
||||
|
||||
## Option C. Bare Metal Setup
|
||||
|
||||
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [auto-install script](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `pip` & your system package manager of choice (e.g. `apt`, `brew`, `pkg`, `nix`, etc.).
|
||||
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [auto-install script](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `uv` & your system package manager of choice (e.g. `apt`, `brew`, `pkg`, `nix`, etc.).
|
||||
|
||||
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://archivebox.github.io/abx-plugins/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://archivebox.github.io/abx-plugins/#pdf), [`SCREENSHOT_ENABLED=False`](https://archivebox.github.io/abx-plugins/#screenshot), and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom) you don't need `chromium`.
|
||||
|
||||
@@ -125,9 +124,9 @@ See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) do
|
||||
Make sure you have [Homebrew](https://brew.sh/) installed first.
|
||||
|
||||
```bash
|
||||
# Install ArchiveBox's dependencies manually (instead of using the all-in-one brew package)
|
||||
brew install python3 node git wget curl ffmpeg yt-dlp ripgrep sonic
|
||||
pip install archivebox
|
||||
# Install ArchiveBox's dependencies manually (instead of using the all-in-one brew tap)
|
||||
brew install uv node git wget curl ffmpeg yt-dlp ripgrep sonic
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
archivebox install
|
||||
|
||||
# Optional: get FFMPEG with the AAC addon
|
||||
@@ -151,13 +150,15 @@ sudo apt install archivebox
|
||||
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
archivebox init
|
||||
sudo archivebox install
|
||||
archivebox install
|
||||
archivebox add 'https://example.com'
|
||||
```
|
||||
|
||||
The apt package installs the ArchiveBox Python application. Runtime extractor
|
||||
The apt package is a thin dev-channel wrapper around the normal Python install
|
||||
flow. Runtime extractor
|
||||
dependencies such as Chromium, yt-dlp, SingleFile, and other plugin-managed
|
||||
tools are installed by `sudo archivebox install`.
|
||||
tools are installed by `archivebox install`; use `sudo archivebox install` only
|
||||
if you want it to install missing system packages via apt.
|
||||
|
||||
<img src="https://cdn0.iconfinder.com/data/icons/flat-round-system/512/freebsd-512.png" width="30px" align="right"/>
|
||||
|
||||
@@ -175,7 +176,7 @@ sudo pkg install chromium
|
||||
#### OpenBSD
|
||||
|
||||
```bash
|
||||
sudo pkg_add python3 node wget git curl yt-dlp ffmpeg ripgrep chromium
|
||||
sudo pkg_add python313 node wget git curl yt-dlp ffmpeg ripgrep chromium
|
||||
```
|
||||
|
||||
#### Arch Linux / Nix / Guix / etc. Other OSs
|
||||
@@ -187,13 +188,13 @@ See the [Quickstart](https://github.com/ArchiveBox/ArchiveBox#-package-manager-s
|
||||
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/65315723-adae-42e4-b8c6-e44b79165ae5" width="55px" align="right"/>
|
||||
|
||||
### 2. Install the Python dependencies using `pip`
|
||||
### 2. Install ArchiveBox using `uv`
|
||||
|
||||
If you are not using the apt package above, install ArchiveBox with `pip`.
|
||||
If you are not using the apt package above, install ArchiveBox with `uv`.
|
||||
|
||||
```bash
|
||||
# get the latest version of archivebox from PyPI
|
||||
pip install --upgrade --ignore-installed archivebox[ldap,sonic]
|
||||
# get the dev version of ArchiveBox
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
# if you see errors about ldap, install the C++ build tools + ldap headers and retry (only needed on some OSs + if you want ldap)
|
||||
# apt install build-essensial python3-ldap
|
||||
@@ -212,7 +213,7 @@ mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
archivebox init
|
||||
|
||||
# auto-install runtime dependencies such as Chromium, yt-dlp, SingleFile, etc.
|
||||
sudo archivebox install
|
||||
archivebox install
|
||||
|
||||
# archive a first URL
|
||||
archivebox add 'https://example.com'
|
||||
@@ -226,13 +227,12 @@ archivebox help
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
Make sure the `pip`-installed version of `archivebox` is available in your `$PATH`.
|
||||
Make sure the `uv`-installed version of `archivebox` is available in your `$PATH`.
|
||||
```bash
|
||||
pip show archivebox # show info about the pip-installed version of archivebox
|
||||
uv tool list # show info about uv-installed tools
|
||||
|
||||
echo $PATH # show the directories your system is searching for binaries
|
||||
which -a archivebox # show all installed archivebox binaries available
|
||||
which archivebox # show which archivebox binary is being called
|
||||
type -a archivebox # show all installed archivebox binaries available
|
||||
|
||||
cd ~/archivebox/data
|
||||
archivebox version # ⭐️ show lots of useful info about installed dependencies and more
|
||||
@@ -280,13 +280,13 @@ See our [[Usage]] Wiki documentation page for more examples.
|
||||
Make sure all apt/brew/pkg/etc. dependencies from above are installed & up-to-date first.
|
||||
|
||||
```bash
|
||||
# get the latest archivebox version from PyPI
|
||||
pip install --upgrade --ignore-installed archivebox
|
||||
# get the dev version of ArchiveBox
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
# run init inside any data directories to migrate the index to the latest version
|
||||
cd ~/archivebox/data
|
||||
archivebox init # update collection index & apply any migrations
|
||||
sudo archivebox install # update runtime dependencies to latest versions
|
||||
archivebox install # update runtime dependencies to latest versions
|
||||
```
|
||||
|
||||
Check our more detailed [Upgrading](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives) documentation and [Release Notes](https://github.com/ArchiveBox/ArchiveBox/releases) if you run into any problems. ➡️
|
||||
|
||||
@@ -82,9 +82,9 @@ LOGOUT_REDIRECT_URL=https://auth.yourcompany.example.com/after/logout
|
||||
|
||||
> Can be used with an SSO provider like [Authentik](https://github.com/goauthentik/authentik), [Authelia](https://github.com/authelia/authelia), [Okta / Auth0](https://www.okta.com/), [Keycloak](https://www.keycloak.org/), and others.
|
||||
|
||||
First, `pip`-install the `ldap` add-on to use this feature (not needed for Docker Archivebox).
|
||||
First, install the `ldap` add-on to use this feature (not needed for Docker Archivebox).
|
||||
```bash
|
||||
pip install archivebox[ldap]
|
||||
uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
```
|
||||
|
||||
Then set these configuration values to finish configuring LDAP:
|
||||
|
||||
@@ -144,7 +144,7 @@ Internally it functions as an index store, storing only the original IDs of the
|
||||
|
||||
*ArchiveBox has supported Sonic for years, and it is the most thoroughly tested and recommended backend for ArchiveBox users that need to scale beyond `ripgrep`.*
|
||||
|
||||
Using [sonic with ArchiveBox in Docker Compose](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml) is the easiest way to get started, though you can also use it without Docker by [installing it manually](https://github.com/valeriansaliou/sonic#installation) and then running `pip install archivebox[sonic]`.
|
||||
Using [sonic with ArchiveBox in Docker Compose](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml) is the easiest way to get started, though you can also use it without Docker by [installing it manually](https://github.com/valeriansaliou/sonic#installation) and then running `uv tool install --python 3.13 --upgrade 'archivebox[sonic] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'`.
|
||||
|
||||
```bash
|
||||
# edit docker-compose.yml to uncomment the lines that enable sonic
|
||||
|
||||
@@ -21,11 +21,10 @@ Then make sure `archivebox` is installed available in your `$PATH`.
|
||||
```bash
|
||||
apt show archivebox # show info about the apt-installed version of archivebox
|
||||
brew info archivebox # show info about the brew-installed version of archivebox
|
||||
pip show archivebox # show info about the pip-installed version of archivebox
|
||||
uv tool list # show info about uv-installed tools
|
||||
|
||||
echo $PATH # show the directories your system is searching for binaries
|
||||
which -a archivebox # show all installed archivebox binaries available
|
||||
which archivebox # show which archivebox binary is being called
|
||||
type -a archivebox # show all installed archivebox binaries available
|
||||
```
|
||||
**⭐️ Show the full archivebox version info + info about all installed dependencies:**
|
||||
```bash
|
||||
@@ -34,7 +33,7 @@ archivebox version # shows lots of useful info about installed dependencie
|
||||
(ensure the version shown is the most recent available from [Releases](https://github.com/ArchiveBox/ArchiveBox/releases))
|
||||
|
||||
### macOS
|
||||
ArchiveBox can be installed with Homebrew or `pip`/`pipx` on macOS:
|
||||
ArchiveBox can be installed with Homebrew or `uv` on macOS:
|
||||
```bash
|
||||
brew tap archivebox/archivebox
|
||||
brew install archivebox
|
||||
@@ -49,7 +48,7 @@ More info: https://github.com/ArchiveBox/homebrew-archivebox
|
||||
|
||||
### Python
|
||||
|
||||
Make sure you have at least Python 3.9 installed on your system.
|
||||
Make sure you have at least Python 3.13 installed on your system.
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
@@ -119,9 +118,9 @@ node --version # make sure you have node >=19 installed
|
||||
npm --version # make sure you have npm installed
|
||||
|
||||
cd ~/archivebox/data # go into your data directory
|
||||
sudo archivebox install # auto-installs runtime dependencies
|
||||
archivebox install # auto-installs runtime dependencies
|
||||
# equivalent to:
|
||||
# curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/archivebox/package.json' > package.json
|
||||
# curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/archivebox/package.json' > package.json
|
||||
# npm install
|
||||
|
||||
# install npm dependencies should then be present in ~/archivebox/data/node_modules/.bin
|
||||
@@ -201,7 +200,7 @@ Database and filesystem issues are uncommon but do come up from time to time (es
|
||||
*ℹ️ Generally, these commands can help you resolve most issues:*
|
||||
```bash
|
||||
archivebox init # upgrade the archivebox collection
|
||||
archivebox init --setup # upgrade the archivebox collection and all dependencies
|
||||
archivebox install # upgrade the archivebox runtime dependencies
|
||||
archivebox update --index-only # force an upgrade of some of the archivebox index/collection files
|
||||
archivebox server --debug # run the server with more verbose debug log output
|
||||
archivebox shell # access the Python API / Django management shell
|
||||
|
||||
+5
-5
@@ -4,9 +4,9 @@
|
||||
# cd /path/to/your/archivebox/data
|
||||
cd ~/archivebox/data
|
||||
|
||||
pip install --upgrade --ignore-installed archivebox
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
# or
|
||||
docker pull archivebox/archivebox:latest
|
||||
docker pull archivebox/archivebox:dev
|
||||
|
||||
# upgrade the collection to a new version
|
||||
archivebox init
|
||||
@@ -27,7 +27,7 @@ archivebox init
|
||||
---
|
||||
|
||||
*Note: It's recommended to only upgrade one major version at a time. e.g. if you're on `v0.4.14`, upgrade to `v0.5.6` next, then `v0.6.3`, and finally `v0.7.1` (as 3 separate steps).
|
||||
You can specify exact versions with pip like so: `pip install archivebox==0.6.3` or with docker `docker pull archivebox/archivebox:0.6.3`. Upgrading directly across multiple major versions may work in some cases, but is not recommended for maximum data safety.*
|
||||
You can specify exact versions with uv like so: `uv tool install --python 3.13 --upgrade archivebox==0.6.3` or with docker `docker pull archivebox/archivebox:0.6.3`. Upgrading directly across multiple major versions may work in some cases, but is not recommended for maximum data safety.*
|
||||
|
||||
|
||||
---
|
||||
@@ -99,7 +99,7 @@ cd ~/archivebox/data # or wherever your data folder is
|
||||
killall archivebox # stop the currently running archivebox version
|
||||
|
||||
# upgrade ArchiveBox using the package manager you originally used to install it
|
||||
pip install --upgrade --ignore-installed archivebox
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
# or
|
||||
sudo apt update
|
||||
sudo apt install --only-upgrade archivebox
|
||||
@@ -107,7 +107,7 @@ sudo apt install --only-upgrade archivebox
|
||||
curl -sSL 'https://get.archivebox.io' | sh
|
||||
|
||||
archivebox init # run init to upgrade the collection to the latest version
|
||||
sudo archivebox install # refresh runtime dependencies if needed
|
||||
archivebox install # refresh runtime dependencies if needed
|
||||
|
||||
archivebox update --index-only # optionally force an update of the snapshot index files (normally done lazily, see issue #962 for more info)
|
||||
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@
|
||||
All three of these ways of running ArchiveBox are equivalent and interchangeable:
|
||||
|
||||
- `archivebox [subcommand] [...args]`
|
||||
*Using the PyPI package via `pip install archivebox`*
|
||||
*Using the Python package via `uv tool install archivebox`*
|
||||
- `docker run ... archivebox/archivebox [subcommand] [...args]`
|
||||
*Using the official Docker image*
|
||||
- `docker-compose run archivebox [subcommand] [...args]`
|
||||
|
||||
@@ -496,7 +496,7 @@ You can find more organizations and initiatives on these other lists:
|
||||
- [Python PyPI](https://pypi.org/project/archivebox/)
|
||||
- [Docker Hub](https://hub.docker.com/r/archivebox/archivebox)
|
||||
- [ArchLinux AUR](https://aur.archlinux.org/packages/archivebox)
|
||||
- [Ubuntu Launchpad PPA](https://launchpad.net/~archivebox/+archive/ubuntu/archivebox)
|
||||
- [Ubuntu/Debian apt wrapper](https://github.com/ArchiveBox/debian-archivebox)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -23,7 +23,7 @@ ArchiveBox
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir my-archive; cd my-archive/
|
||||
pip install archivebox
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
archivebox init
|
||||
archivebox add https://example.com
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "archivebox",
|
||||
"version": "0.9.35rc50",
|
||||
"version": "0.9.35rc103",
|
||||
"repository": "github:ArchiveBox/ArchiveBox",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
|
||||
+4
-4
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "archivebox"
|
||||
version = "0.9.35rc50"
|
||||
version = "0.9.35rc103"
|
||||
requires-python = ">=3.13"
|
||||
description = "Self-hosted internet archiving solution."
|
||||
authors = [{name = "Nick Sweeting", email = "pyproject.toml@archivebox.io"}]
|
||||
@@ -79,9 +79,9 @@ dependencies = [
|
||||
### Extractor dependencies (optional binaries detected at runtime via shutil.which)
|
||||
### Binary/Package Management
|
||||
"abxbus==2.5.10", # EventBus API
|
||||
"abxpkg>=1.11.219", # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
|
||||
"abx-plugins>=1.11.223", # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
|
||||
"abx-dl>=1.11.223", # shared ArchiveBox downloader package with blocking install preflight
|
||||
"abxpkg>=1.11.239", # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
|
||||
"abx-plugins>=1.11.248", # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
|
||||
"abx-dl>=1.11.235", # shared ArchiveBox downloader package with blocking install preflight
|
||||
### UUID7 backport for Python <3.14
|
||||
"uuid7>=0.1.0; python_version < '3.14'", # provides the uuid_extensions module on Python 3.13
|
||||
]
|
||||
|
||||
Reference in new issue
Block a user