Compare commits

...
Author SHA1 Message Date
ArchiveBox Release Bot cc218245c3 Bump release version to 0.9.35rc485 2026-09-18 09:02:15 +00:00
ArchiveBox Release Bot b625cf4958 Bump for abx-dl 1.12.301 2026-09-18 09:01:50 +00:00
ArchiveBox Release Bot 92c0f52324 Bump release version to 0.9.35rc484 2026-09-18 08:43:52 +00:00
Nick Sweeting 880ab87da2 Improve persona import layout and support Chrome Beta profiles 2026-09-18 01:43:19 -07:00
ArchiveBox Release Bot 7e34e43cf5 Bump release version to 0.9.35rc483 2026-09-18 08:31:33 +00:00
Nick Sweeting 4a1e06fda7 Prepare XDG config directory for mounted browser profiles 2026-09-18 01:31:14 -07:00
ArchiveBox Release Bot e1647d8f93 Bump release version to 0.9.35rc482 2026-09-18 08:22:03 +00:00
ArchiveBox Release Bot 0badc70c4d Bump for abx-dl 1.12.300 2026-09-18 08:21:44 +00:00
ArchiveBox Release Bot 9fbddc5b99 Bump release version to 0.9.35rc481 2026-09-18 08:02:08 +00:00
ArchiveBox Release Bot eb7e6f2ba2 Bump for abx-dl 1.12.299 2026-09-18 08:01:47 +00:00
ArchiveBox Release Bot a7eb82ffda Bump release version to 0.9.35rc480 2026-09-18 07:48:43 +00:00
ArchiveBox Release Bot d3f2509c30 Bump for abx-dl 1.12.298 2026-09-18 07:48:16 +00:00
ArchiveBox Release Bot 3e1a31755c Bump release version to 0.9.35rc479 2026-09-18 07:30:56 +00:00
ArchiveBox Release Bot 70bce92bf3 Bump for abx-dl 1.12.297 2026-09-18 07:29:28 +00:00
ArchiveBox Release Bot 80e4a1c4f8 Bump release version to 0.9.35rc478 2026-09-18 06:40:57 +00:00
Nick Sweeting ceb3c04aff Fix explicit DISPLAY handling for Docker noVNC persona setup (#1882)
An explicit `DISPLAY=novnc:0.0` was silently removed by the Docker
entrypoint whenever `xdpyinfo` failed. In the reported `0.9.35rc381`
image, `xdpyinfo` is missing, so `docker compose run --rm` clears the
display even with a healthy noVNC server and Chromium exits with
“Missing X server or $DISPLAY”.

Preserve explicitly supplied `DISPLAY`, including an empty value to opt
out. Probe only the implicit `novnc:0.0` default, retaining automatic
headless fallback when no display was configured. Current dev already
includes `x11-utils`; this change prevents probe failure from overriding
user configuration.

Document the one-off `persona create` / `persona open` Compose flow.
`persona open` resolves Chromium automatically, avoiding the obsolete
`/data/lib/env/bin/chromium` and architecture/version-specific
Playwright paths.

Validation:
- Reproduced persona creation followed by the exact Chromium X11 failure
on the published `0.9.35rc381` image with real, healthy `theasp/novnc`.
- Mounted only the patched entrypoint into that affected image: explicit
DISPLAY survived despite the missing `xdpyinfo`, and the same Chromium
command opened a window visible through noVNC.
- Built the complete branch Docker image using the Dockerfile's default
abx-dl base. Real `docker compose run --rm archivebox persona create
personal` and `persona open personal` succeeded; viewed and controlled
Chromium through noVNC, closed it normally, and reopened in another
one-off container. A bookmark created in the UI persisted across both
runs.
- Added `IMAGE=<built-image> ./bin/validate_docker_display.sh`: six real
Compose checks cover working implicit/explicit displays, unavailable
explicit display, explicit empty opt-out, and implicit/explicit behavior
with noVNC stopped. Passed on the rebuilt image; failed on the unpatched
image at the explicit unavailable-display assertion.
- `bash -n`, `git diff --check`, and pre-commit hooks passed. No mocks
or fake binaries.

Fixes #1872.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the Docker entrypoint silently dropping an explicit `DISPLAY` when
`xdpyinfo` fails, so Chromium can connect to a healthy noVNC server in
images like `0.9.35rc381` where `xdpyinfo` is missing. The entrypoint
previously unset the display on probe failure, causing Chromium to exit
with "Missing X server or $DISPLAY"; now only the implicit `novnc:0.0`
default is probed, an explicit value (including empty to opt out) is
preserved, and headless fallback still applies when no display was
configured. Fixes #1872.

**Validation**
- Adds `bin/validate_docker_display.sh` with six real Compose checks
covering implicit/explicit displays, unavailable display, empty opt-out,
and noVNC stopped behavior.
- Documents the one-off `persona create` / `persona open` Compose flow
and that `persona open` resolves the Chromium binary automatically.

<sup>Written for commit 953f29b65e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1882?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-18 02:39:44 -04:00
Nick Sweeting 953f29b65e Preserve explicit Docker DISPLAY for noVNC persona setup 2026-09-17 23:36:08 -07:00
ArchiveBox Release Bot 404f2efc04 Bump release version to 0.9.35rc477 2026-09-18 06:32:59 +00:00
Nick Sweeting c23389c0a8 Prioritize ArchiveBox projects and TLSNotary in shared footers 2026-09-17 23:27:22 -07:00
ArchiveBox Release Bot 7837373d81 Bump release version to 0.9.35rc476 2026-09-18 06:18:47 +00:00
Nick Sweeting 3600ac54ea docs: remove unnecessary app launch command 2026-09-17 23:17:24 -07:00
ArchiveBox Release Bot ec47877277 Bump release version to 0.9.35rc475 2026-09-18 06:14:01 +00:00
Nick Sweeting 9a33749eca docs: add native app setup and client usage to README 2026-09-17 23:12:38 -07:00
Nick Sweeting c104931341 Align Apps menu inline code and add breathing room 2026-09-17 23:12:09 -07:00
ArchiveBox Release Bot 6a245d70d9 Bump release version to 0.9.35rc474 2026-09-18 06:11:08 +00:00
Nick Sweeting 4ca3b180a8 Preserve inline spacing around abx-dl in the Apps menu 2026-09-17 23:10:11 -07:00
ArchiveBox Release Bot a520e25cbb Bump release version to 0.9.35rc473 2026-09-18 06:07:19 +00:00
ArchiveBox Release Bot 271a4c1c40 Bump for abx-dl 1.12.289 2026-09-18 06:07:01 +00:00
ArchiveBox Release Bot c935a2b801 Bump release version to 0.9.35rc472 2026-09-18 06:06:47 +00:00
Nick Sweeting d1c692f044 Use the six shared ArchiveBox Apps menu destinations 2026-09-17 23:06:20 -07:00
ArchiveBox Release Bot f6f7be2ddc Bump release version to 0.9.35rc471 2026-09-18 05:58:51 +00:00
Nick Sweeting f38bb4c2e4 Let in-flight screenshot captures finish independently of newer site pushes 2026-09-17 22:57:21 -07:00
Nick Sweeting a577ea147f Merge remote-tracking branch 'origin/dev' into dev 2026-09-17 22:56:45 -07:00
Nick Sweeting 89007da097 Keep failed capture events out of the site publishing queue 2026-09-17 22:56:12 -07:00
ArchiveBox Release Bot ea243e1bcb Bump release version to 0.9.35rc470 2026-09-18 05:48:05 +00:00
Nick Sweeting f85e8d5dc0 Publish publicsite independently of screenshot capture 2026-09-17 22:47:35 -07:00
ArchiveBox Release Bot 400de11916 Bump release version to 0.9.35rc469 2026-09-18 05:28:49 +00:00
Nick Sweeting fce31ac965 Fix native snapshot action dropdown clicks and label padding 2026-09-17 22:28:07 -07:00
Nick Sweeting 014e193e6b Point project navigation to the published DigestBox Pages site 2026-09-17 22:27:46 -07:00
ArchiveBox Release Bot 505f53c733 Bump release version to 0.9.35rc468 2026-09-18 05:26:46 +00:00
Nick Sweeting fe9b59eb63 Unify ArchiveBox website navigation and screenshot galleries 2026-09-17 22:26:24 -07:00
ArchiveBox Release Bot b097af5417 Bump release version to 0.9.35rc467 2026-09-18 04:52:02 +00:00
ArchiveBox Release Bot 27ac1cc66f Bump for abx-dl 1.12.285 2026-09-18 04:51:42 +00:00
ArchiveBox Release Bot b65151dbbe Bump release version to 0.9.35rc466 2026-09-18 04:44:50 +00:00
Nick Sweeting 5c0a065242 Remove unused helpers and simplify equivalent tests (#1880)
Remove unused Python helpers and simplify equivalent test code without
changing active CLI, REST, model, runner, or UI behavior. This is the
mechanical first part of the simplification split; #1879 contains the
architectural and behavior-sensitive remainder. **Merge this PR first.**

- Delete unused hashing helpers, JSONL file loading, legacy TOML/INI
conversion helpers, and plugin-model discovery. Caller searches found no
active consumers; retained production definitions are AST-identical to
`dev`.
- Remove redundant result aliases, share identical POST payloads, and
parameterize equivalent test pairs while retaining their cases.
- Address Cubic's reviews: parse CSV and assert the exact header, URL
values, and sorted order; qualify the TLSNotary result lookup by its
full snapshot/plugin/hook identity.
- Correct the all-plugins HTTP fixture's TLSNotary expectation: every
snapshot must produce the specific HTTPS-only refusal, exit 1, and no
captured output. Other plugin statuses, output files, crawl topology,
and concurrent execution assertions remain enforced.
- Complete real parser dependency installation before public-add tests
deliberately stop the server. This prevents interrupting virtualenv
creation while preserving the crawl restart and metadata assertions.

Based on `dev` at `1fa2d4dc9`. **550 net lines removed**, excluding
`old/` and `docs/`; neither directory changes here. Runtime routing,
models, schemas, UI, dependencies, and architecture remain in #1879.

Validation:

- All-files `uv run --no-sync prek run --all-files` passes.
- The initial equivalent cleanup passed AST checks and 85 targeted
tests. Review follow-ups additionally pass both exact CSV tests, both
affected public-add runtime tests with a fresh dependency directory and
`--no-cache --no-sources`, and the full real depth-two all-plugins
crawl.
- [Full CI at
`42ed15b73`](https://github.com/ArchiveBox/ArchiveBox/actions/runs/35300660816)
**passed**, including every required lane. The subsequent one-line
TLSNotary lookup correction passed the full real all-plugins crawl
locally (131 seconds). All Cubic review threads are resolved.
- [Fresh CI at
`50a462e78`](https://github.com/ArchiveBox/ArchiveBox/actions/runs/35304078254)
is running.
2026-09-18 00:44:25 -04:00
ArchiveBox Release Bot a908323133 Bump release version to 0.9.35rc465 2026-09-18 04:33:33 +00:00
Nick Sweeting f5d4645f41 Move project links into Apps navigation and remove redundant menu items 2026-09-17 21:32:56 -07:00
Nick Sweeting 50a462e786 test: qualify TLSNotary result lookup by plugin 2026-09-17 20:40:14 -07:00
Nick Sweeting 42ed15b733 Verify TLS-only refusals and complete installs before restart tests 2026-09-17 19:45:50 -07:00
Nick Sweeting 15f6f7260e Assert exact CSV search rows and ordering 2026-09-17 19:27:11 -07:00
Nick Sweeting cd159cf7cf Remove unused helpers and mechanically simplify equivalent tests 2026-09-17 19:00:43 -07:00
45 changed files with 917 additions and 992 deletions

No files matched your search

+36 -65
View File
@@ -1,79 +1,50 @@
name: Deploy Publicsite to GitHub Pages
on:
push:
branches:
- dev
paths:
- publicsite/**
- archivebox/**
- bin/collect_ui_screenshots.*
- bin/generate_ui_screenshot_gallery.py
- bin/take_screenshot.js
- docs/Screenshots.md
- pyproject.toml
- uv.lock
- .github/workflows/deploy-publicsite.yml
branches: [dev]
pull_request:
workflow_dispatch:
workflow_run:
workflows: ['Capture publicsite screenshots']
types: [completed]
branches: [dev]
permissions:
contents: read
pages: write
id-token: write
actions: read
concurrency:
group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
# Failed captures must not displace a pending push deployment.
group: fast-pages-${{ github.event_name == 'workflow_run' && github.event.workflow_run.conclusion != 'success' && github.run_id || github.ref }}
cancel-in-progress: false
jobs:
build:
if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'pull_request' && github.ref || 'dev' }}
- uses: astral-sh/setup-uv@v7
- name: Restore latest complete screenshots
run: uv run --no-project python bin/restore_site_screenshots.py archivebox publicsite/screenshots
env:
GH_TOKEN: ${{ github.token }}
- name: Render current gallery layout without recapturing images
run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
- uses: actions/upload-pages-artifact@v3
with:
path: publicsite
deploy:
if: github.event_name != 'pull_request'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 1
- name: Setup Python and uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: false
- name: Install ArchiveBox and screenshot dependencies
run: uv sync --no-cache --frozen
- name: Build UI screenshot gallery
run: bin/collect_ui_screenshots.sh
env:
BASE_URL: http://archivebox.localhost:8000
- name: Verify gallery was generated from this exact revision
run: |
uv run --no-cache --no-sync python - <<'PY'
import json
import os
import tomllib
from pathlib import Path
provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
assert provenance["version"] == version, provenance
assert provenance["capture_count"] >= 3 * 35, provenance
assert len(provenance["files"]) == provenance["capture_count"], provenance
PY
- name: Setup Pages
uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
- name: Upload artifact
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
with:
path: ./publicsite
- name: Deploy to GitHub Pages
- uses: actions/configure-pages@v5
- uses: actions/deploy-pages@v4
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
+17
View File
@@ -246,6 +246,23 @@ jobs:
if: inputs.push_digests
run: echo ${{ steps.docker_build.outputs.digest }}
- name: Verify OpenCode install with a read-only browser profile
shell: bash
env:
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
run: |
set -Eeuo pipefail
PROFILE_DIR="$(mktemp -d)"
"$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
--volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
"$IMAGE" bash -c '
set -Eeuo pipefail
test -w "$XDG_CONFIG_HOME"
test ! -w "$XDG_CONFIG_HOME/chromium"
archivebox install opencode
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
'
- name: Validate exact built image version and commit
if: ${{ !inputs.push_digests }}
shell: bash
+70
View File
@@ -0,0 +1,70 @@
name: Capture publicsite screenshots
on:
push:
branches:
- dev
paths:
- publicsite/**
- archivebox/**
- bin/collect_ui_screenshots.*
- bin/generate_ui_screenshot_gallery.py
- bin/take_screenshot.js
- docs/Screenshots.md
- pyproject.toml
- uv.lock
- .github/workflows/screenshots.yml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.event_name == 'workflow_dispatch' && format('publicsite-captures-manual-{0}', github.run_id) || 'publicsite-captures' }}
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 1
- name: Setup Python and uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: false
- name: Install ArchiveBox and screenshot dependencies
run: uv sync --no-cache --frozen
- name: Build UI screenshot gallery
run: bin/collect_ui_screenshots.sh
env:
BASE_URL: http://archivebox.localhost:8000
- name: Verify gallery was generated from this exact revision
run: |
uv run --no-cache --no-sync python - <<'PY'
import json
import os
import tomllib
from pathlib import Path
provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
assert provenance["version"] == version, provenance
assert provenance["capture_count"] >= 3 * 35, provenance
assert len(provenance["files"]) == provenance["capture_count"], provenance
PY
- name: Save validated screenshots
uses: actions/upload-artifact@v4
with:
name: site-screenshots
path: publicsite/screenshots
if-no-files-found: error
retention-days: 90
+2 -2
View File
@@ -10,10 +10,10 @@
# --build-context abx-plugins=../abx-plugins \
# -t archivebox/abx-dl:dev
# docker buildx build . -f Dockerfile \
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.284 \
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.301 \
# -t archivebox:multistage
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.284
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.301
FROM archivebox/sonic:1.4.9 AS sonic
FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
+29 -1
View File
@@ -151,6 +151,17 @@ curl -fsSL 'https://get.archivebox.io' | bash
#### ✳️&nbsp; Easy Setup
<details>
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
<br/>
<ol>
<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
<li>Set up an admin account and generate an API Key.</li>
<li>Use the Web UI or <a href="https://archivebox.github.io/ios-archivebox/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://archivebox.github.io/ios-archivebox/">mobile app</a>, the <a href="https://archivebox.github.io/archivebox-browser-extension/">browser extension</a>, and other integrations to your server using your API Key.</li>
</ol>
<br/>
</details>
<details>
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b> (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
<br/>
@@ -561,6 +572,20 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
</code></pre>
</details>
<br/>
<details>
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
<br/>
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://archivebox.github.io/ios-archivebox/)
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://archivebox.github.io/ios-archivebox/)
- 🧩 [ArchiveBox Browser Extension](https://archivebox.github.io/archivebox-browser-extension/)
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://github.com/ArchiveBox/electron-archivebox)
<br/>
</details>
<br/>
<br/>
@@ -614,7 +639,10 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
- <img src="https://archivebox.github.io/ios-archivebox/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
<i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
<i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
+5 -1
View File
@@ -441,7 +441,11 @@ def main():
@main.command("create")
@click.argument("names", nargs=-1)
@click.option("--import", "import_from", help="Import from chrome, chromium, brave, edge, a persona directory, or a live CDP URL")
@click.option(
"--import",
"import_from",
help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
)
@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
@click.option(
"--browser-binary",
-260
View File
@@ -1,260 +0,0 @@
# Bootable hashing helpers (file content + mtime/size cache, mime detection).
# MUST NOT import archivebox.config, archivebox.core, or Django — stdlib only.
import hashlib
import mimetypes
from functools import lru_cache
from pathlib import Path
from collections.abc import Callable
from datetime import datetime
@lru_cache(maxsize=1024)
def _cached_file_hash(filepath: str, size: int, mtime: float) -> str:
"""Internal function to calculate file hash with cache key based on path, size and mtime."""
sha256_hash = hashlib.sha256()
with open(filepath, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
sha256_hash.update(chunk)
return sha256_hash.hexdigest()
@lru_cache(maxsize=10)
def hash_file(file_path: Path, pwd: Path | None = None) -> str:
"""Calculate SHA256 hash of a file with caching based on path, size and mtime."""
pwd = Path(pwd) if pwd else None
file_path = Path(file_path)
if not file_path.is_absolute():
file_path = pwd / file_path if pwd else file_path.absolute()
abs_path = file_path.resolve()
stat_info = abs_path.stat()
return _cached_file_hash(
str(abs_path),
stat_info.st_size,
stat_info.st_mtime,
)
@lru_cache(maxsize=10)
def get_dir_hashes(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict[str, str]:
"""Calculate SHA256 hashes for all files and directories recursively."""
pwd = Path(pwd) if pwd else None
dir_path = Path(dir_path)
if not dir_path.is_absolute():
dir_path = pwd / dir_path if pwd else dir_path.absolute()
if not dir_path.is_dir():
raise ValueError(f"Not a directory: {dir_path}")
if max_depth < -1:
raise ValueError(f"max_depth must be >= -1, got {max_depth}")
# Get all files recursively
all_files = get_dir_entries(
dir_path,
pwd=pwd,
recursive=True,
include_files=True,
include_dirs=False,
filter_func=filter_func,
)
hashes: dict[str, str] = {}
hashable_summary = []
# Calculate hashes for all files
for subfile in all_files:
subfile_path = dir_path / subfile
sha256_hash = hash_file(subfile_path)
hashes[subfile] = sha256_hash
hashable_summary.append(f"{sha256_hash} ./{subfile}")
# Calculate hashes for all directories
subdirs = get_dir_entries(
dir_path,
pwd=pwd,
recursive=True,
include_files=False,
include_dirs=True,
include_hidden=False,
filter_func=filter_func,
max_depth=max_depth,
)
for subdir in subdirs:
subdir_path = dir_path / subdir
subdir_hashes = get_dir_hashes(
subdir_path,
filter_func=filter_func,
max_depth=0,
)
hashes[subdir] = subdir_hashes["."]
# Filter results by max_depth
if max_depth >= 0:
hashes = {path: value for path, value in hashes.items() if len(Path(path).parts) <= max_depth + 1}
# Calculate root directory hash
hashable_summary.sort()
root_sha256 = hashlib.sha256("\n".join(hashable_summary).encode()).hexdigest()
hashes["."] = root_sha256
return hashes
@lru_cache(maxsize=128)
def get_dir_entries(
dir_path: Path,
pwd: Path | None = None,
recursive: bool = True,
include_files: bool = True,
include_dirs: bool = True,
include_hidden: bool = False,
filter_func: Callable | None = None,
max_depth: int = -1,
) -> tuple[str, ...]:
"""Get filtered list of directory entries."""
pwd = Path(pwd) if pwd else None
dir_path = Path(dir_path)
if not dir_path.is_absolute():
dir_path = pwd / dir_path if pwd else dir_path.absolute()
results = []
def process_path(path: Path, depth: int):
if not include_hidden and path.name.startswith("."):
return False
if max_depth >= 0 and depth > max_depth:
return False
if filter_func:
info = {
"abspath": str(path.absolute()),
"relpath": str(path.relative_to(dir_path)),
}
if not filter_func(info):
return False
return True
for path in dir_path.rglob("*") if recursive else dir_path.glob("*"):
current_depth = len(path.relative_to(dir_path).parts)
if path.is_file() and include_files and process_path(path, current_depth):
results.append(str(path.relative_to(dir_path)))
elif path.is_dir() and include_dirs and process_path(path, current_depth):
results.append(str(path.relative_to(dir_path)))
if not recursive:
break
return tuple(sorted(results)) # Make immutable for caching
@lru_cache(maxsize=1024)
def get_dir_sizes(dir_path: Path, pwd: Path | None = None, **kwargs) -> dict[str, int]:
"""Calculate sizes for all files and directories recursively."""
sizes: dict[str, int] = {}
hashes = get_dir_hashes(dir_path, pwd=pwd, **kwargs)
dir_path = Path(dir_path)
for path_key in hashes:
full_path = dir_path / path_key
if full_path.is_file():
sizes[path_key] = full_path.stat().st_size
else:
total = 0
for file_path in full_path.rglob("*"):
if file_path.is_file() and not file_path.name.startswith("."):
total += file_path.stat().st_size
sizes[path_key + "/"] = total
return sizes
@lru_cache(maxsize=10)
def get_dir_info(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict:
"""Get detailed information about directory contents including hashes and sizes."""
pwd = Path(pwd) if pwd else None
dir_path = Path(dir_path)
if not dir_path.is_absolute():
dir_path = pwd / dir_path if pwd else dir_path.absolute()
hashes = get_dir_hashes(dir_path, pwd=pwd, filter_func=filter_func, max_depth=max_depth)
sizes = get_dir_sizes(str(dir_path), pwd=pwd, filter_func=filter_func, max_depth=max_depth)
num_total_subpaths = sum(1 for name in hashes if name != ".")
details = {}
for filename, sha256_hash in sorted(hashes.items()):
abs_path = (dir_path / filename).resolve()
stat_info = abs_path.stat()
num_subpaths = sum(1 for p in hashes if p.startswith(filename + "/"))
is_dir = abs_path.is_dir()
if is_dir:
mime_type = "inode/directory"
basename = abs_path.name
extension = ""
num_bytes = sizes[filename + "/"]
if filename == ".":
num_subpaths = num_total_subpaths
else:
filename += "/"
num_subpaths = num_subpaths
else: # is_file
num_subpaths = None
mime_type = mimetypes.guess_type(str(abs_path))[0]
extension = abs_path.suffix
basename = abs_path.name.rsplit(extension, 1)[0]
num_bytes = sizes[filename]
details[filename] = {
"basename": basename,
"mime_type": mime_type,
"extension": extension,
"num_subpaths": num_subpaths,
"num_bytes": num_bytes,
"hash_sha256": sha256_hash,
"created_at": datetime.fromtimestamp(stat_info.st_ctime).isoformat(),
"modified_at": datetime.fromtimestamp(stat_info.st_mtime).isoformat(),
}
if filter_func and not filter_func(details[filename]):
del details[filename]
return details
if __name__ == "__main__":
import json
dir_info = get_dir_info(Path("."), max_depth=6)
with open(".hashes.json", "w") as f:
json.dump(dir_info, f, indent=4)
print("Wrote .hashes.json")
# Example output:
# {
# ".": {
# "basename": "misc",
# "mime_type": "inode/directory",
# "extension": "",
# "num_subpaths": 25,
# "num_bytes": 214677,
# "hash_sha256": "addfacf88b2ff6b564846415fb7b21dcb7e63ee4e911bc0aec255ee354958530",
# "created_at": "2024-12-04T00:08:38.537449",
# "modified_at": "2024-12-04T00:08:38.537449"
# },
# "__init__.py": {
# "basename": "__init__",
# "mime_type": "text/x-python",
# "extension": ".py",
# "num_subpaths": null,
# "num_bytes": 32,
# "hash_sha256": "b0e5e7ff17db3b60535cf664282787767c336e3e203a43e21b6326c6fe457551",
# "created_at": "2024-10-08T00:51:41.001359",
# "modified_at": "2024-10-08T00:51:41.001359"
# },
# ...
# }
-14
View File
@@ -32,7 +32,6 @@ import sys
import json
from typing import Any, TextIO
from collections.abc import Iterable, Iterator
from pathlib import Path
# Type constants for JSONL records
@@ -120,19 +119,6 @@ def read_stdin(stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
yield record
def read_file(path: Path) -> Iterator[dict[str, Any]]:
"""
Read JSONL or plain URLs from a file.
Yields parsed records as dicts.
"""
with open(path) as f:
for line in f:
record = parse_line(line)
if record:
yield record
def read_args_or_stdin(args: Iterable[str], stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
"""
Read from CLI arguments if provided, otherwise from stdin.
-83
View File
@@ -2,94 +2,11 @@ from typing import Any, cast
from collections.abc import Callable
import json
import ast
import inspect
import toml
import re
import configparser
from pathlib import Path, PosixPath
from pydantic.json_schema import GenerateJsonSchema
from pydantic_core import to_jsonable_python
JSONValue = str | bool | int | None | list["JSONValue"]
TOML_HEADER = "# Converted from INI to TOML format: https://toml.io/en/\n\n"
def load_ini_value(val: str) -> JSONValue:
"""Convert lax INI values into strict TOML-compliant (JSON) values"""
if val.lower() in ("true", "yes", "1"):
return True
if val.lower() in ("false", "no", "0"):
return False
if val.isdigit():
return int(val)
try:
return ast.literal_eval(val)
except Exception:
pass
try:
return json.loads(val)
except Exception:
pass
return val
def convert(ini_str: str) -> str:
"""Convert a string of INI config into its TOML equivalent (warning: strips comments)"""
config = configparser.ConfigParser()
setattr(config, "optionxform", str) # capitalize key names
config.read_string(ini_str)
# Initialize an empty dictionary to store the TOML representation
toml_dict = {}
# Iterate over each section in the INI configuration
for section in config.sections():
toml_dict[section] = {}
# Iterate over each key-value pair in the section
for key, value in config.items(section):
parsed_value = load_ini_value(value)
# Convert the parsed value to its TOML-compatible JSON representation
toml_dict[section.upper()][key.upper()] = json.dumps(parsed_value)
# Build the TOML string
toml_str = TOML_HEADER
for section, items in toml_dict.items():
toml_str += f"[{section}]\n"
for key, value in items.items():
toml_str += f"{key} = {value}\n"
toml_str += "\n"
return toml_str.strip()
class JSONSchemaWithLambdas(GenerateJsonSchema):
"""
Encode lambda functions in default values properly.
Usage:
>>> json.dumps(value, encoder=JSONSchemaWithLambdas())
"""
def encode_default(self, dft: Any) -> Any:
config = self._config
if isinstance(dft, Callable):
return "{{lambda " + inspect.getsource(dft).split("=lambda ")[-1].strip()[:-1] + "}}"
return to_jsonable_python(
dft,
timedelta_mode=config.ser_json_timedelta,
bytes_mode=config.ser_json_bytes,
serialize_unknown=True,
)
def better_toml_dump_str(val: Any) -> str:
try:
+2 -2
View File
@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
"Persona",
{
"fields": ("name", "created_by", "permissions"),
"classes": ("card", "persona-card-primary"),
"classes": ("card", "wide", "persona-card-primary"),
},
),
(
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
"import_extract_cookies",
"import_capture_storage",
),
"classes": ("card", "wide"),
"classes": ("card", "wide", "persona-card-import"),
},
),
(
+1 -1
View File
@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
label="Autodiscovered profiles",
widget=forms.RadioSelect,
choices=(),
help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
)
import_source = forms.CharField(
required=False,
+17 -7
View File
@@ -18,6 +18,7 @@ import subprocess
import sys
import tempfile
from dataclasses import dataclass, field
from functools import partial
from pathlib import Path
from typing import TYPE_CHECKING
from urllib.parse import urlparse
@@ -31,6 +32,7 @@ if TYPE_CHECKING:
BROWSER_LABELS = {
"chrome": "Google Chrome",
"chrome-beta": "Google Chrome Beta",
"chromium": "Chromium",
"brave": "Brave",
"edge": "Microsoft Edge",
@@ -216,24 +218,29 @@ class PersonaImportResult:
return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
def get_chrome_user_data_dir() -> Path | None:
def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
"""Get the default Chrome user data directory for the current platform."""
system = platform.system()
home = Path.home()
app_name = "Chrome Beta" if beta else "Chrome"
if system == "Darwin":
candidates = [
home / "Library" / "Application Support" / "Google" / "Chrome",
home / "Library" / "Application Support" / "Google" / app_name,
]
elif system == "Linux":
candidates = [
home / ".config" / "google-chrome",
home / ".config" / "chrome",
]
candidates = (
[home / ".config" / "google-chrome-beta"]
if beta
else [
home / ".config" / "google-chrome",
home / ".config" / "chrome",
]
)
elif system == "Windows":
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
candidates = [
local_app_data / "Google" / "Chrome" / "User Data",
local_app_data / "Google" / app_name / "User Data",
]
else:
candidates = []
@@ -320,6 +327,7 @@ def get_edge_user_data_dir() -> Path | None:
BROWSER_PROFILE_FINDERS = {
"chrome": get_chrome_user_data_dir,
"chrome-beta": partial(get_chrome_user_data_dir, beta=True),
"chromium": get_chromium_user_data_dir,
"brave": get_brave_user_data_dir,
"edge": get_edge_user_data_dir,
@@ -547,6 +555,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
return str(Path(source.browser_binary).expanduser().resolve())
names = {
"chrome": ("Google Chrome", "google-chrome"),
"chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
"chromium": ("Chromium", "chromium"),
"brave": ("Brave Browser", "brave-browser"),
"edge": ("Microsoft Edge", "microsoft-edge"),
@@ -561,6 +570,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
elif platform.system() == "Windows":
relative = {
"chrome": "Google/Chrome/Application/chrome.exe",
"chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
"edge": "Microsoft/Edge/Application/msedge.exe",
"brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
"chromium": "Chromium/Application/chrome.exe",
-4
View File
@@ -55,10 +55,6 @@ def get_plugins() -> list[str]:
return sorted(get_plugin_catalog())
def get_plugin_models():
return get_plugin_catalog().plugins
def get_plugin_name(plugin: str) -> str:
"""
Get the base plugin name without numeric prefix.
+29 -37
View File
@@ -1339,20 +1339,36 @@
display: inline-block;
}
.action-rearchive-select,
.action-permissions-select {
/* The native select receives the actual click, including in WKWebView. */
body.change-list #changelist .actions .action-rearchive-select,
body.change-list #changelist .actions .action-permissions-select {
position: absolute;
left: 0;
bottom: 0;
width: 1px;
height: 1px;
inset: 0;
width: 100%;
height: 100%;
opacity: 0;
pointer-events: none;
border: 0;
padding: 0;
cursor: pointer;
z-index: 1;
margin: 0;
}
body.change-list #changelist .action-rearchive-wrapper > .button,
body.change-list #changelist .action-permissions-wrapper > .button {
padding: 8px 16px !important;
width: max-content;
min-width: max-content;
flex: 0 0 auto;
gap: 6px;
white-space: nowrap;
}
.action-rearchive-wrapper:focus-within,
.action-permissions-wrapper:focus-within {
outline: 2px solid #2563eb;
outline-offset: 2px;
border-radius: 6px;
}
/* Object count */
.actions .action-counter {
color: #64748b;
@@ -1960,22 +1976,10 @@
.addClass('action-rearchive-select')
.attr('aria-label', 'Re-Archive')
.appendTo(wrapper)
$('<button>')
.attr('type', 'button')
$('<span>')
.attr('aria-hidden', 'true')
.addClass('button')
.html('Re-Archive <span aria-hidden="true">▾</span>')
.click(function (e) {
e.preventDefault()
e.stopPropagation()
const el = select[0]
if (typeof el.showPicker === 'function') {
try { el.showPicker() } catch (err) { el.focus(); el.click() }
} else {
el.focus()
el.click()
}
return false
})
.appendTo(wrapper)
$('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
rearchiveActions.forEach(function (action_value) {
@@ -2021,23 +2025,11 @@
.addClass('action-permissions-select')
.attr('aria-label', 'Permissions')
.appendTo(wrapper)
$('<button>')
.attr('type', 'button')
$('<span>')
.attr('aria-hidden', 'true')
.attr('name', action_type)
.addClass('button')
.text(this.text)
.click(function (e) {
e.preventDefault()
e.stopPropagation()
const el = select[0]
if (typeof el.showPicker === 'function') {
try { el.showPicker() } catch (err) { el.focus(); el.click() }
} else {
el.focus()
el.click()
}
return false
})
.appendTo(wrapper)
$('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
;[
@@ -38,8 +38,8 @@
border-bottom: 1px dashed #64748b;
}
.field-import_mode ul,
.field-import_discovered_profile ul {
.abx-import-mode,
.abx-profile-picker {
margin: 0;
padding: 0;
list-style: none;
@@ -47,20 +47,26 @@
gap: 12px;
}
.field-import_mode ul {
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
.abx-import-mode {
grid-template-columns: repeat(auto-fit, minmax(min(220px, 100%), 1fr));
}
.field-import_discovered_profile ul {
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
max-height: 460px;
overflow: auto;
padding-right: 4px;
.abx-profile-picker {
grid-template-columns: repeat(auto-fit, minmax(min(260px, 100%), 1fr));
}
.field-import_mode li,
.field-import_discovered_profile li {
.abx-import-mode > div,
.abx-profile-picker > div {
margin: 0;
min-width: 0;
}
/* Import rows grow with their contents instead of sharing a fixed flex
height. Avoid overflow-x:hidden turning overflow-y:visible into auto. */
#content-main #persona_form .persona-card-import > .form-row,
#content-main #persona_form .persona-card-import fieldset > div {
flex: none;
overflow: visible;
}
.field-import_mode label,
@@ -68,7 +74,6 @@
display: flex;
gap: 12px;
align-items: flex-start;
min-height: 100%;
padding: 14px 16px;
border-radius: 14px;
border: 1px solid #dbe4ee;
@@ -167,9 +172,14 @@
color: #991b1b;
}
body.model-persona.change-form #content-main form fieldset.persona-card-primary {
flex: 1 1 520px !important;
max-width: calc(66.66% - 10px) !important;
#content-main #persona_form .field-name .flex-container {
max-width: 100% !important;
}
#persona_form .field-name input {
width: 100%;
max-width: none;
box-sizing: border-box;
}
body.model-persona.change-form #content-main form fieldset.persona-card-timestamps {
@@ -257,6 +267,11 @@ document.addEventListener('DOMContentLoaded', function () {
or by pasting in a CDP url from
<code class="persona-import-copy" data-copy-text="chrome://inspect#remote-debugging" title="Copy chrome://inspect#remote-debugging">chrome://inspect#remote-debugging</code> (e.g. <code>http://127.0.0.1:9222</code>).
</p>
<p>
You can also import cookies into this persona from another browser using the
<a href="https://archivebox.github.io/archivebox-browser-extension/" target="_blank" rel="noopener noreferrer">ArchiveBox browser extension</a>.
Create the persona here, then select it in the extension's settings to sync cookies from that browser.
</p>
</div>
</section>
{{ block.super }}
@@ -264,9 +279,9 @@ document.addEventListener('DOMContentLoaded', function () {
{% block after_field_sets %}
{{ block.super }}
<section id="plugin-config" class="module aligned persona-plugin-config">
<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">
<h2>Plugin Config</h2>
<p>These typed controls update the same Persona config JSON shown above. Shared config keys stay synced across plugin sections.</p>
{% include "plugins/plugin_config_grid.html" with plugin_groups=adminform.form.plugin_groups %}
</section>
</fieldset>
{% endblock %}
+2 -1
View File
@@ -287,7 +287,8 @@
<code>daily</code> - Run once per day<br/>
<code>weekly</code> - Run once per week<br/>
<code>0 */6 * * *</code> - Every 6 hours (cron format)<br/>
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)<br/>
<a href="https://crontab.guru/" target="_blank" rel="noopener">Cron Format Helper</a>
</div>
</div>
+7 -10
View File
@@ -153,7 +153,7 @@ class TestArchiveBoxWithLDAP:
def test_archivebox_init_without_ldap(self, tmp_path):
"""Test that archivebox init works without LDAP enabled."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["init"],
cwd=tmp_path,
timeout=45,
@@ -161,14 +161,13 @@ class TestArchiveBoxWithLDAP:
default_cli_env=True,
disable_extractors=True,
)
_, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
# Should succeed
assert code == 0, f"archivebox init failed: {stderr}"
assert result.returncode == 0, f"archivebox init failed: {result.stderr}"
def test_archivebox_version_with_ldap_config(self, tmp_path):
"""Test that archivebox version works with LDAP config set."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["version"],
cwd=tmp_path,
timeout=10,
@@ -179,10 +178,9 @@ class TestArchiveBoxWithLDAP:
default_cli_env=True,
disable_extractors=True,
)
_, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
# Should succeed
assert code == 0, f"archivebox version failed: {stderr}"
assert result.returncode == 0, f"archivebox version failed: {result.stderr}"
class TestLDAPConfigValidationInArchiveBox:
@@ -190,7 +188,7 @@ class TestLDAPConfigValidationInArchiveBox:
def test_archivebox_init_with_incomplete_ldap_config(self, tmp_path):
"""Test that archivebox init fails with helpful error when LDAP config is incomplete."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["init"],
cwd=tmp_path,
timeout=45,
@@ -201,10 +199,9 @@ class TestLDAPConfigValidationInArchiveBox:
default_cli_env=True,
disable_extractors=True,
)
_, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
# Should fail with validation error
assert code != 0, "Should fail with incomplete LDAP config"
assert result.returncode != 0, "Should fail with incomplete LDAP config"
# Check error message
assert "LDAP_* config options must all be set" in stderr, f"Expected validation error message in: {stderr}"
assert "LDAP_* config options must all be set" in result.stderr, f"Expected validation error message in: {result.stderr}"
+2 -3
View File
@@ -1204,7 +1204,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
root_url = recursive_test_site["root_url"]
child_url = recursive_test_site["child_urls"][0]
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
[
"add",
"--depth=2",
@@ -1219,8 +1219,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
env=env,
timeout=180,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
assert result.returncode == 0, result.stderr or result.stdout
with use_archivebox_db(initialized_archive):
crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()
+7 -9
View File
@@ -142,18 +142,17 @@ class TestArchiveResultCreate:
assert crawl_result.returncode == 0, crawl_result.stderr
crawl_record = parse_jsonl_output(crawl_result.stdout)[0]
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["archiveresult", "create"],
stdin=crawl_result.stdout,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert "Passed through" in stderr
records = parse_jsonl_output(stdout)
assert result.returncode == 0
assert "Passed through" in result.stderr
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert records[0]["id"] == crawl_record["id"]
@@ -163,16 +162,15 @@ class TestArchiveResultList:
def test_list_empty(self, initialized_archive):
"""List with no archive results returns empty."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["archiveresult", "list"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert "Listed 0 archive results" in stderr
assert result.returncode == 0
assert "Listed 0 archive results" in result.stderr
def test_list_filter_by_status(self, initialized_archive):
"""Filter archive results by status."""
+28 -37
View File
@@ -33,19 +33,18 @@ class TestCrawlCreate:
"""Create crawl from URL arguments."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "create", url],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
assert "Created crawl" in stderr
assert result.returncode == 0, f"Command failed: {result.stderr}"
assert "Created crawl" in result.stderr
# Check JSONL output
records = parse_jsonl_output(stdout)
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert records[0]["type"] == "Crawl"
assert url in records[0]["urls"]
@@ -55,18 +54,17 @@ class TestCrawlCreate:
urls = [create_test_url() for _ in range(3)]
stdin = "\n".join(urls)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "create"],
stdin=stdin,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
assert result.returncode == 0, f"Command failed: {result.stderr}"
records = parse_jsonl_output(stdout)
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
crawl = records[0]
assert crawl["type"] == "Crawl"
@@ -78,32 +76,30 @@ class TestCrawlCreate:
"""Create crawl with --depth flag."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "create", "--depth=2", url],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert records[0]["max_depth"] == 2
def test_create_with_tag(self, initialized_archive):
"""Create crawl with --tag flag."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "create", "--tag=test-tag", url],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert "test-tag" in records[0].get("tags_str", "")
def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -119,17 +115,16 @@ class TestCrawlCreate:
url = create_test_url()
stdin = tag_result.stdout + json.dumps({"url": url}) + "\n"
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "create"],
stdin=stdin,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
assert any(record.get("type") == "Crawl" and url in record["urls"] for record in records)
@@ -163,32 +158,30 @@ class TestCrawlList:
def test_list_empty(self, initialized_archive):
"""List with no crawls returns empty."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "list"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert "Listed 0 crawls" in stderr
assert result.returncode == 0
assert "Listed 0 crawls" in result.stderr
def test_list_returns_created(self, initialized_archive):
"""List returns previously created crawls."""
url = create_test_url()
run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "list"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) >= 1
assert any(url in r.get("urls", "") for r in records)
@@ -197,16 +190,15 @@ class TestCrawlList:
url = create_test_url()
run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "list", "--status=queued"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
for r in records:
assert r["status"] == "queued"
@@ -221,16 +213,15 @@ class TestCrawlList:
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["crawl", "list", "--limit=2"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) == 2
+36 -45
View File
@@ -3,6 +3,8 @@ Tests for archivebox list command.
Verify list emits snapshot JSONL and applies the documented filters.
"""
import csv
import io
import json
from pathlib import Path
@@ -427,16 +429,15 @@ def test_snapshot_list_search_meta(initialized_archive):
url = create_test_url(domain="meta-search-example.com")
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list", "--search=meta", "meta-search-example.com"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
records = parse_jsonl_output(stdout)
assert result.returncode == 0, f"Command failed: {result.stderr}"
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert "meta-search-example.com" in records[0]["url"]
@@ -446,16 +447,15 @@ def test_list_search_meta_matches_metadata(initialized_archive):
url = create_test_url(domain="top-level-meta-search-example.com")
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["list", "--search=meta", "top-level-meta-search-example.com"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
records = parse_jsonl_output(stdout)
assert result.returncode == 0, f"Command failed: {result.stderr}"
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert "top-level-meta-search-example.com" in records[0]["url"]
@@ -468,11 +468,10 @@ def test_search_command_finds_snapshots(initialized_archive):
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
assert code == 0, stderr
assert "example" in stdout
assert result.returncode == 0, result.stderr
assert "example" in result.stdout
def test_search_command_returns_no_results_for_missing_term(initialized_archive):
@@ -483,22 +482,20 @@ def test_search_command_returns_no_results_for_missing_term(initialized_archive)
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["search", "nonexistentterm12345"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code in [0, 1]
assert result.returncode in [0, 1]
def test_search_command_on_empty_archive(initialized_archive):
_cmd_result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
assert code in [0, 1]
assert result.returncode in [0, 1]
def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive):
@@ -509,11 +506,10 @@ def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive)
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
assert code == 0, stderr
records = parse_jsonl_output(stdout)
assert result.returncode == 0, result.stderr
records = parse_jsonl_output(result.stdout)
assert any("example.com" in row.get("url", "") for row in records)
@@ -576,52 +572,47 @@ def test_search_command_csv_outputs_requested_column(initialized_archive):
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["search", "--csv", "url", "--with-headers"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, stderr
assert "url" in stdout
assert "example.com" in stdout
assert result.returncode == 0, result.stderr
assert list(csv.reader(io.StringIO(result.stdout))) == [["url"], ["https://example.com"]]
def test_search_command_with_headers_requires_structured_output_format(initialized_archive):
_cmd_result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
assert code != 0
assert "requires" in stderr.lower()
assert "json" in stderr.lower()
assert result.returncode != 0
assert "requires" in result.stderr.lower()
assert "json" in result.stderr.lower()
def test_search_command_sort_option_runs_successfully(initialized_archive):
for url in ["https://iana.org", "https://example.com"]:
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["search", "--csv", "url", "--sort=url"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, stderr
assert "example.com" in stdout or "iana.org" in stdout
assert result.returncode == 0, result.stderr
assert list(csv.reader(io.StringIO(result.stdout))) == [["https://example.com"], ["https://iana.org"]]
def test_search_command_help_lists_supported_filters(initialized_archive):
_cmd_result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
assert code == 0
assert "--url__icontains" in stdout
assert "--crawl-id" in stdout
assert "--status" in stdout
assert "--sort" in stdout
assert "--json" in stdout
assert "--html" in stdout
assert result.returncode == 0
assert "--url__icontains" in result.stdout
assert "--crawl-id" in result.stdout
assert "--status" in result.stdout
assert "--sort" in result.stdout
assert "--json" in result.stdout
assert "--html" in result.stdout
+32 -42
View File
@@ -53,15 +53,14 @@ def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archiv
env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
_install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "create", recursive_test_site["root_url"]],
cwd=initialized_archive,
env=env,
timeout=60,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
records = parse_jsonl_output(stdout)
assert result.returncode == 0, result.stderr or result.stdout
records = parse_jsonl_output(result.stdout)
snapshot_id = next(record["id"] for record in records if record.get("type") == "Snapshot")
with use_archivebox_db(initialized_archive):
browser_state = Snapshot.objects.get(id=snapshot_id).output_dir / "chrome" / "browser.json"
@@ -107,7 +106,7 @@ class TestRunWithCrawl:
)
assert create_result.returncode == 0, create_result.stderr
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=create_result.stdout,
cwd=initialized_archive,
@@ -116,12 +115,11 @@ class TestRunWithCrawl:
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
assert result.returncode == 0, f"Command failed: {result.stderr}"
# Should output the created Crawl
records = parse_jsonl_output(stdout)
records = parse_jsonl_output(result.stdout)
crawl_records = [r for r in records if r.get("type") == "Crawl"]
assert len(crawl_records) >= 1
assert crawl_records[0].get("id") # Should have an id now
@@ -170,7 +168,7 @@ class TestRunWithSnapshot:
)
assert create_result.returncode == 0, create_result.stderr
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=create_result.stdout,
cwd=initialized_archive,
@@ -179,11 +177,10 @@ class TestRunWithSnapshot:
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
assert result.returncode == 0, f"Command failed: {result.stderr}"
records = parse_jsonl_output(stdout)
records = parse_jsonl_output(result.stdout)
snapshot_records = [r for r in records if r.get("type") == "Snapshot"]
assert len(snapshot_records) >= 1
assert snapshot_records[0].get("id")
@@ -220,7 +217,7 @@ class TestRunWithSnapshot:
def test_run_with_plain_url(self, initialized_archive):
"""Run accepts plain URL records (no type field)."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=url + "\n",
cwd=initialized_archive,
@@ -229,10 +226,9 @@ class TestRunWithSnapshot:
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) >= 1
@@ -405,7 +401,7 @@ class TestRunRecovery:
crawl_id = crawl.id
snapshot_id = snapshot.id
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run", "--maintenance-only"],
cwd=initialized_archive,
timeout=90,
@@ -413,13 +409,12 @@ class TestRunRecovery:
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, stdout + stderr
assert "Repairing" in stderr
assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in stderr
assert "interrupted before" in stderr
assert "remaining URLs" in stderr
assert result.returncode == 0, result.stdout + result.stderr
assert "Repairing" in result.stderr
assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in result.stderr
assert "interrupted before" in result.stderr
assert "remaining URLs" in result.stderr
with use_archivebox_db(initialized_archive):
crawl = Crawl.objects.get(id=crawl_id)
@@ -444,17 +439,16 @@ class TestRunPassThrough:
assert tag_result.returncode == 0, tag_result.stderr
tag_record = parse_jsonl_output(tag_result.stdout)[0]
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=tag_result.stdout,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
tag_records = [record for record in records if record.get("type") == "Tag"]
assert len(tag_records) == 1
assert tag_records[0]["id"] == tag_record["id"]
@@ -471,7 +465,7 @@ class TestRunPassThrough:
)
assert create_result.returncode == 0, create_result.stderr
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=create_result.stdout,
cwd=initialized_archive,
@@ -480,10 +474,9 @@ class TestRunPassThrough:
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
# Should have at least the Crawl in output
assert len(records) >= 1
@@ -516,7 +509,7 @@ class TestRunMixedInput:
)
assert snapshot_result.returncode == 0, snapshot_result.stderr
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=tag_result.stdout + snapshot_result.stdout,
cwd=initialized_archive,
@@ -525,10 +518,9 @@ class TestRunMixedInput:
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
types = {record.get("type") for record in records}
assert {"Crawl", "Snapshot", "Tag"}.issubset(types)
@@ -539,16 +531,15 @@ class TestRunEmpty:
def test_run_empty_stdin(self, initialized_archive):
"""Run with empty stdin returns success."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin="",
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert result.returncode == 0
def test_run_no_runnable_records_to_process(self, initialized_archive):
"""Run with only a real non-runnable Tag reports no work."""
@@ -560,17 +551,16 @@ class TestRunEmpty:
)
assert tag_result.returncode == 0, tag_result.stderr
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["run"],
stdin=tag_result.stdout,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert "No records to process" in stderr
assert result.returncode == 0
assert "No records to process" in result.stderr
class TestRunDaemonMode:
+15 -47
View File
@@ -124,10 +124,17 @@ def test_server_auth_secret_and_cookie_settings_are_restart_stable(tmp_path):
assert first_lines[3:] == ["None", "False", "False"]
def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
@pytest.mark.parametrize(
"base_url, secure",
[
("https://archive.example.com", True),
("", False),
],
)
def test_base_url_controls_secure_cookies(tmp_path, base_url, secure):
(tmp_path / ".archivebox_id").write_text("testcoll")
env = os.environ.copy()
env["BASE_URL"] = "https://archive.example.com"
env["BASE_URL"] = base_url
env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
repo_root = Path(__file__).resolve().parents[2]
env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
@@ -155,45 +162,8 @@ def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
)
assert json.loads(result.stdout) == {
"csrf_secure": True,
"session_secure": True,
"proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
}
def test_unconfigured_base_url_enables_proxy_ssl_header_without_secure_cookies(tmp_path):
(tmp_path / ".archivebox_id").write_text("testcoll")
env = os.environ.copy()
env["BASE_URL"] = ""
env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
repo_root = Path(__file__).resolve().parents[2]
env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
result = subprocess.run(
[
sys.executable,
"-c",
(
"import django, json;"
"django.setup();"
"from django.conf import settings;"
"print(json.dumps({"
"'csrf_secure': settings.CSRF_COOKIE_SECURE,"
"'session_secure': settings.SESSION_COOKIE_SECURE,"
"'proxy_ssl_header': settings.SECURE_PROXY_SSL_HEADER,"
"}))"
),
],
capture_output=True,
text=True,
check=True,
env=env,
cwd=tmp_path,
)
assert json.loads(result.stdout) == {
"csrf_secure": False,
"session_secure": False,
"csrf_secure": secure,
"session_secure": secure,
"proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
}
@@ -759,9 +729,8 @@ def test_live_server_signal_exit_and_resume_uses_existing_supervisor_state(initi
resumed = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}-resumed.log", env=env)
resumed_log = resumed.log_path
_cmd_result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
assert result.returncode == 0, result.stderr or result.stdout
os.kill(resumed.pid, signal.SIGTERM)
resumed.wait(timeout=20)
@@ -783,14 +752,13 @@ def test_live_daemonized_server_keeps_supervisord_owned_by_archivebox_parent(ini
port = get_free_port()
bind_url = f"http://127.0.0.1:{port}"
try:
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["server", "--daemonize", f"127.0.0.1:{port}"],
cwd=initialized_archive,
env=env,
timeout=90,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
assert result.returncode == 0, result.stderr or result.stdout
_wait_for_archivebox_workers(initialized_archive, env, ("worker_daphne", "worker_runner"), timeout=30)
assert_port_open("127.0.0.1", port, timeout=30)
+31 -41
View File
@@ -33,18 +33,17 @@ class TestSnapshotCreate:
"""Create snapshot from URL arguments."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "create", url],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
assert "Created" in stderr
assert result.returncode == 0, f"Command failed: {result.stderr}"
assert "Created" in result.stderr
records = parse_jsonl_output(stdout)
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert records[0]["type"] == "Snapshot"
assert records[0]["url"] == url
@@ -83,16 +82,15 @@ class TestSnapshotCreate:
"""Create snapshot with --tag flag."""
url = create_test_url()
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "create", "--tag=test-tag", url],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert "test-tag" in records[0].get("tags", "")
def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -108,17 +106,16 @@ class TestSnapshotCreate:
url = create_test_url()
stdin = tag_result.stdout + url + "\n"
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "create"],
stdin=stdin,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
assert any(record.get("type") == "Snapshot" and record["url"] == url for record in records)
@@ -127,16 +124,15 @@ class TestSnapshotCreate:
"""Create snapshots from multiple URLs."""
urls = [create_test_url() for _ in range(3)]
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "create"] + urls,
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) == 3
created_urls = {r["url"] for r in records}
@@ -149,32 +145,30 @@ class TestSnapshotList:
def test_list_empty(self, initialized_archive):
"""List with no snapshots returns empty."""
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
_stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
assert "Listed 0 snapshots" in stderr
assert result.returncode == 0
assert "Listed 0 snapshots" in result.stderr
def test_list_returns_created(self, initialized_archive):
"""List returns previously created snapshots."""
url = create_test_url()
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) >= 1
assert any(r.get("url") == url for r in records)
@@ -183,16 +177,15 @@ class TestSnapshotList:
url = create_test_url()
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list", "--status=queued"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
for r in records:
assert r["status"] == "queued"
@@ -201,16 +194,15 @@ class TestSnapshotList:
url = create_test_url(domain="unique-domain-12345.com")
run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list", "--url__icontains=unique-domain-12345"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) == 1
assert "unique-domain-12345" in records[0]["url"]
@@ -224,16 +216,15 @@ class TestSnapshotList:
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list", "--limit=2"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0
records = parse_jsonl_output(stdout)
assert result.returncode == 0
records = parse_jsonl_output(result.stdout)
assert len(records) == 2
def test_list_with_sort_and_limit(self, initialized_archive):
@@ -246,16 +237,15 @@ class TestSnapshotList:
disable_extractors=True,
)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["snapshot", "list", "--limit=2", "--sort=-created_at"],
cwd=initialized_archive,
default_cli_env=True,
disable_extractors=True,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"Command failed: {stderr}"
records = parse_jsonl_output(stdout)
assert result.returncode == 0, f"Command failed: {result.stderr}"
records = parse_jsonl_output(result.stdout)
assert len(records) == 2
+16 -14
View File
@@ -635,7 +635,22 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
if not (status in allowed_statuses or (plugin == "archivedotorg" and status == ArchiveResult.StatusChoices.FAILED))
]
assert not unexpected_results
# This fixture serves HTTP. TLSNotary must reject every plaintext document;
# a setup, timeout, or verifier error must not count as that expected refusal.
tlsnotary_results = [row for row in archive_results if row[3] == "tlsnotary"]
assert {row[1] for row in tlsnotary_results} == expected_urls
assert len(tlsnotary_results) == len(expected_urls)
with use_archivebox_db(initialized_archive):
for snapshot_id, _url, _depth, _plugin, hook_name, status, files, size, _output in tlsnotary_results:
assert status == ArchiveResult.StatusChoices.FAILED
assert not files
assert size == 0
process = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="tlsnotary", hook_name=hook_name).process
assert process is not None
assert process.status == Process.StatusChoices.EXITED
assert process.exit_code == 1
assert process.stderr.strip() == "[tlsnotary] TLSNotary requires an HTTPS document"
assert not [result for result in unexpected_results if result["plugin"] != "tlsnotary"]
ytdlp_results = [
(url, status, output_str)
for _snapshot_id, url, _depth, plugin, _hook_name, status, _files, _size, output_str in archive_results
@@ -682,19 +697,6 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
assert set(recursive_test_site["deep_urls"]).issubset(parsed_urls)
assert processes
failed_hook_results = [
{
"url": url,
"depth": depth,
"plugin": plugin,
"hook_name": hook_name,
"status": status,
"output_str": output_str,
}
for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
if status == ArchiveResult.StatusChoices.FAILED and plugin != "archivedotorg"
]
assert not failed_hook_results
assert all(status == Process.StatusChoices.EXITED for _id, _pwd, _cmd, status, _exit_code, _started_at, _ended_at in processes)
intervals = []
@@ -963,14 +963,13 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
log_name="archivewebpage_server.log",
)
get_http_response(port, host=f"archivebox.localhost:{port}", path="/")
_cmd_result = run_archivebox_cmd(
command_result = run_archivebox_cmd(
["add", "--depth=0", "--max-urls=1", "--plugins=archivewebpage", url],
cwd=initialized_archive,
env=env,
timeout=120,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, f"archivebox add failed:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
assert command_result.returncode == 0, f"archivebox add failed:\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
capture = _get_archivewebpage_capture(initialized_archive, url)
snapshot_host = f"{get_snapshot_subdomain(capture['snapshot_id'])}.archivebox.localhost:{port}"
+4 -3
View File
@@ -163,14 +163,15 @@ def test_snapshot_service_cli_add_seals_snapshot_and_writes_indexes(tmp_path, re
port = get_free_port()
env = cli_env(port=port, server=True, PLUGINS="wget", SAVE_WGET="True")
_cmd_result = run_archivebox_cmd(
command_result = run_archivebox_cmd(
["add", "--depth=0", "--plugins=wget", recursive_test_site["root_url"]],
cwd=tmp_path,
env=env,
timeout=180,
)
stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert code == 0, f"archivebox add failed with code {code}\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
assert command_result.returncode == 0, (
f"archivebox add failed with code {command_result.returncode}\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
)
state = _snapshot_state(tmp_path, recursive_test_site["root_url"])
snapshot_dir = state["snapshot_dir"]
+10 -15
View File
@@ -387,14 +387,13 @@ def test_live_second_server_takes_over_existing_server_process(tmp_path, initial
assert "A newer archivebox process took over the orchestrator, server" in first_text
assert "Starting orchestrator, server" in second_text
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["status"],
cwd=tmp_path,
env=env,
timeout=60,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
assert result.returncode == 0, result.stderr or result.stdout
first_resumes = first_log.read_text(encoding="utf-8", errors="replace").count("Other newer archivebox process")
stop_archivebox_process(second, signal.SIGTERM)
@@ -422,15 +421,14 @@ def test_live_update_index_only_does_not_take_over_server_runtime(tmp_path, init
supervisor_pid_before = supervisor_pid_from_log(server_log)
daphne_pid_before = worker_pid_from_log(server_log, "worker_daphne")
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["update", "--index-only", "--before=0"],
cwd=tmp_path,
env=env,
timeout=90,
)
update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert update_returncode == 0, update_stderr or update_stdout
assert result.returncode == 0, result.stderr or result.stdout
assert pid_is_alive(server.pid)
assert pid_is_alive(supervisor_pid_before)
assert pid_is_alive(daphne_pid_before)
@@ -471,15 +469,14 @@ def test_live_server_keeps_http_runtime_while_update_runs_real_sqlite_indexer(tm
runner_spawn_text = "spawned: 'worker_runner' with pid"
runner_spawn_count = supervisord_log.read_text(encoding="utf-8", errors="replace").count(runner_spawn_text)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["update", "--index-only", "--batch-size=1"],
cwd=tmp_path,
env=env,
timeout=180,
)
update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert update_returncode == 0, update_stderr or update_stdout
assert result.returncode == 0, result.stderr or result.stdout
assert pid_is_alive(server.pid)
assert supervisor_pid_from_log(server_log) == supervisor_pid_before
assert pid_is_alive(daphne_pid_before)
@@ -628,14 +625,13 @@ def test_live_repeated_server_startups_take_over_cleanly(tmp_path, initialized_a
wait_for_pid_to_disappear(daphne_pids[index - 1], timeout=15)
wait_for_pid_to_disappear(runner_pids[index - 1], timeout=15)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["status"],
cwd=tmp_path,
env=env,
timeout=60,
)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, stderr or stdout
assert result.returncode == 0, result.stderr or result.stdout
assert pid_is_alive(servers[-1].pid)
assert all(pid_is_alive(server.pid) for server in servers)
@@ -673,14 +669,13 @@ def test_live_background_add_survives_server_exit_and_foreground_run_reclaims(tm
server_log = server.log_path
supervisor_pid_before = supervisor_pid_from_log(server_log)
_cmd_result = run_archivebox_cmd(
result = run_archivebox_cmd(
["update", "--index-only", "--batch-size=10"],
cwd=tmp_path,
env=env,
timeout=90,
)
update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert update_returncode == 0, update_stderr or update_stdout
assert result.returncode == 0, result.stderr or result.stdout
assert pid_is_alive(server.pid)
assert pid_is_alive(supervisor_pid_before)
assert supervisor_pid_from_log(server_log) == supervisor_pid_before
+29 -129
View File
@@ -20,6 +20,23 @@ WEB_HOST = "web.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:8000"
ADD_FORM_DEFAULTS = {
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
}
@pytest.fixture
def admin_user(db):
return User.objects.create_superuser(
@@ -140,22 +157,13 @@ def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/staff-config",
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"crawl_timeout": "0",
"timeout": "60",
"snapshot_max_size": "0",
"delete_after": "0",
"crawl_max_concurrent_snapshots": "1",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"notes": "",
"schedule": "daily",
"persona": "Default",
"permissions": "public",
"start_paused": "on",
"main_plugins": ["wget"],
"plugin_config__wget__WGET_TIMEOUT": "77",
@@ -246,6 +254,7 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com\nhttps://cdn.example.com/asset.js",
"tag": "alpha,beta",
"depth": "1",
@@ -260,11 +269,6 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
"url_filters_denylist": "cdn.example.com",
"url_filters_only_new": "1",
"notes": "Created from /add/",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -290,24 +294,12 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
def _csrf_test_add_data(url: str) -> dict[str, str]:
return {
**ADD_FORM_DEFAULTS,
"url": url,
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"crawl_timeout": "0",
"timeout": "",
"snapshot_max_size": "0",
"delete_after": "0",
"crawl_max_concurrent_snapshots": "1",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
}
@@ -337,24 +329,14 @@ def test_add_view_sanitizes_crawl_notes_before_safe_update(client, admin_user):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/notes-xss",
"tag": "",
"depth": "0",
"max_urls": "1",
"crawl_max_size": "0",
"crawl_timeout": "0",
"timeout": "",
"snapshot_max_size": "0",
"delete_after": "0",
"crawl_max_concurrent_snapshots": "1",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"notes": malicious_notes,
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -373,20 +355,8 @@ def test_add_view_unchecked_only_new_sets_crawl_override(client, admin_user):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/rearchive",
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -407,20 +377,10 @@ def test_add_view_selected_persona_wins_over_stale_config_override(client, admin
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/private",
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Private",
"permissions": "public",
"start_paused": "",
"config": '{"DEFAULT_PERSONA": "Default"}',
},
HTTP_HOST=ADMIN_HOST,
@@ -447,20 +407,9 @@ def test_add_view_applies_plugin_config_overrides(client, admin_user):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/plugin-config",
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"main_plugins": ["wget"],
"plugin_config__wget__WGET_ENABLED": "false",
"plugin_config__wget__WGET_TIMEOUT": "77",
@@ -488,9 +437,8 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/public-safe",
"tag": "",
"depth": "0",
"max_urls": "10",
"crawl_max_size": "45mb",
"crawl_timeout": "120",
@@ -503,8 +451,6 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
"url_filters_only_new": "1",
"notes": "public add",
"schedule": "daily",
"persona": "Default",
"permissions": "public",
"start_paused": "on",
"main_plugins": ["wget"],
"plugin_config__twocaptcha__TWOCAPTCHA_API_KEY": "posted-token",
@@ -541,21 +487,9 @@ def test_add_view_queues_crawl_for_background_runner(client, admin_user):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com",
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -575,21 +509,11 @@ def test_add_view_start_paused_creates_paused_crawl_without_snapshots(client, ad
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": "https://example.com/paused",
"tag": "",
"depth": "1",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "on",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -610,6 +534,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": (
"https://sweeting.me,https://google.com\n"
"Notes: [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox), https://news.ycombinator.com\n"
@@ -617,20 +542,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
'{"items":["https://example.com/three"]}\n'
"csv,https://example.com/four"
),
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -656,21 +568,9 @@ def test_add_view_trims_trailing_punctuation_from_markdown_urls(client, admin_us
response = client.post(
reverse("add"),
data={
**ADD_FORM_DEFAULTS,
"url": ("Docs: https://github.com/ArchiveBox/ArchiveBox.\nIssue: https://github.com/abc?abc#234234?."),
"tag": "",
"depth": "0",
"max_urls": "0",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"url_filters_allowlist": "",
"url_filters_denylist": "",
"url_filters_only_new": "1",
"notes": "",
"schedule": "",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
HTTP_HOST=ADMIN_HOST,
)
@@ -431,6 +431,10 @@ def test_public_add_view_import_text_formats_preserve_metadata_and_resume_withou
URL_ALLOWLIST=r"127\.0\.0\.1[:/].*",
)
create_admin_and_token(tmp_path)
# These tests deliberately stop the server while crawls are queued. Finish
# dependency installation first so shutdown cannot leave a half-built venv.
install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
assert install_result.returncode == 0, install_result.stderr or install_result.stdout
try:
start_archivebox_server(tmp_path, env=env, port=port)
@@ -536,6 +540,10 @@ def test_public_add_view_rejects_file_path_and_shell_injection_payloads(tmp_path
URL_ALLOWLIST=r"example\.com|example\.org|iana\.org|www\.iana\.org",
)
create_admin_and_token(tmp_path)
# These tests deliberately stop the server while crawls are queued. Finish
# dependency installation first so shutdown cannot leave a half-built venv.
install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
assert install_result.returncode == 0, install_result.stderr or install_result.stdout
try:
start_archivebox_server(tmp_path, env=env, port=port)
@@ -127,6 +127,10 @@ def test_persona_admin_add_view_renders_import_ui(admin_client):
assert b"auth.json" in response.content
assert b"Plugin Config" in response.content
assert b'name="plugin_config__wget__WGET_TIMEOUT"' in response.content
assert b'<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">' in response.content
assert b'href="https://archivebox.github.io/archivebox-browser-extension/"' in response.content
assert b"import cookies into this persona from another browser" in response.content
assert "wide" in response.context["adminform"].fieldsets[0][1]["classes"]
def test_import_persona_from_source_copies_profile_without_browser_export(admin_user):
+9 -27
View File
@@ -607,7 +607,14 @@ class TestSnapshotProgressStats:
assert result.embed_path_db() is None
def test_embed_path_db_prefers_valid_output_str_over_first_output_file(self, snapshot, real_hash_projection):
@pytest.mark.parametrize(
"output_str",
[
("wget/example.com/index.html"),
(""),
],
)
def test_embed_path_db_selects_html_output(self, snapshot, real_hash_projection, output_str):
from archivebox.core.models import ArchiveResult
output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
@@ -619,32 +626,7 @@ class TestSnapshotProgressStats:
_process, result = real_hash_projection
ArchiveResult.objects.filter(pk=result.pk).update(
plugin="wget",
output_str="wget/example.com/index.html",
output_files={
"example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
"example.com/index.html": {
"size": (Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").stat().st_size,
"mimetype": "text/html",
},
},
)
result.refresh_from_db()
assert result.embed_path_db() == "wget/example.com/index.html"
def test_embed_path_db_scores_output_files_instead_of_using_first_entry(self, snapshot, real_hash_projection):
from archivebox.core.models import ArchiveResult
output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
output_dir.mkdir(parents=True, exist_ok=True)
(Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(REPO_ROOT / "README.md", Path(snapshot.output_dir) / "wget" / "example.com" / "index.html")
shutil.copyfile(REPO_ROOT / "archivebox" / "templates" / "static" / "bootstrap.min.css", output_dir / "mobile.css")
_process, result = real_hash_projection
ArchiveResult.objects.filter(pk=result.pk).update(
plugin="wget",
output_str="",
output_str=output_str,
output_files={
"example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
"example.com/index.html": {
+11 -5
View File
@@ -230,11 +230,14 @@ assert_writable_dir "$DATA_DIR/logs"
assert_writable_dir "$DATA_DIR/archive"
assert_writable_dir "$PERSONAS_DIR/Default/chrome_profile"
# check if novnc x11 $DISPLAY is available
export DISPLAY="${DISPLAY:-"novnc:0.0"}"
if ! xdpyinfo > /dev/null 2>&1; then
# cant connect to x11 display, unset it so that chrome doesn't try to connect to it and hang indefinitely
unset DISPLAY
# Preserve an explicit DISPLAY (including an empty value to disable X11).
# Only probe the implicit noVNC default: a missing probe or unavailable server
# must not silently discard the user's display configuration.
if [[ -z "${DISPLAY+x}" ]]; then
export DISPLAY="novnc:0.0"
if ! xdpyinfo > /dev/null 2>&1; then
unset DISPLAY
fi
fi
# Active browser processes do not survive container restarts, but their lock
@@ -250,6 +253,9 @@ find /tmp "$TMP_DIR" -maxdepth 1 -type d -name "archivebox-chrome-profile.*" -mm
ensure_dir "/home/$ARCHIVEBOX_USER"
# A nested read-only browser-profile mount can create this parent as root.
# OpenCode's installer needs to create a sibling config directory as archivebox.
ensure_dir "${XDG_CONFIG_HOME:-/home/$ARCHIVEBOX_USER/.config}"
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR"
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR/semaphores"
ensure_small_runtime_tree "$UV_CACHE_DIR"
+74 -16
View File
@@ -4,6 +4,7 @@ import hashlib
import html
import json
import os
import re
import subprocess
import struct
import sys
@@ -123,6 +124,53 @@ def append_manifest(manifest_path: Path, screenshot_path: Path) -> None:
manifest.write(json.dumps(item) + "\n")
def render_gallery(gallery_header: str, gallery_sections: str) -> str:
return (
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
"<title>ArchiveBox UI Screenshots</title><style>"
":root{--ink:#28252a;--muted:#716c74;--accent:#9b2854;--paper:#fcfaf7;--line:#e9e2e3}"
"*{box-sizing:border-box}body{margin:0;font:16px/1.75 system-ui,sans-serif;"
"background:var(--paper);color:var(--ink)}.screenshot-content{max-width:1100px;margin:0 auto;padding:28px}"
"a{color:var(--accent)}h1,h2{line-height:1.15;letter-spacing:-.045em}"
"article{margin:40px 0 65px}article h2{font-size:29px;margin:0 0 10px}article p{font-size:13px;color:var(--muted);margin:0 0 20px}"
".profile-switch{display:flex;flex-wrap:wrap;gap:4px;background:#f0e8eb;border:1px solid var(--line);padding:5px;"
"border-radius:28px;width:max-content;max-width:100%;margin:30px 0}.profile-switch button{font:600 13px system-ui;"
"cursor:pointer;border:0;border-radius:22px;background:transparent;color:var(--muted);padding:10px 20px}"
".profile-switch button[aria-pressed=true]{background:var(--accent);color:white;box-shadow:0 3px 10px #9b285425}"
"a:focus-visible,button:focus-visible{outline:3px solid var(--accent);outline-offset:4px}"
"figure{min-width:0;margin:0 0 20px;padding:18px;border:1px solid var(--line);border-radius:20px;"
"background:linear-gradient(140deg,#fff,#f8edf2)}figure[hidden]{display:none}"
".shot-mobile{max-width:440px;margin-inline:auto}.shot-tablet{max-width:800px;margin-inline:auto}"
"figcaption{font-size:12px;color:var(--muted);text-align:center;margin:0 0 12px}"
"img{display:block;width:100%;height:auto;border-radius:8px}code{overflow-wrap:anywhere}"
"@media(max-width:550px){.screenshot-content{padding:20px}figure{padding:8px;border-radius:12px}.profile-switch button{padding:9px 16px}}"
'</style><link rel="stylesheet" href="../site-chrome.css"></head><body>'
+ (REPO_DIR / "bin/templates/screenshots-header.html").read_text(encoding="utf-8")
+ '<div class="screenshot-content">'
+ gallery_header
+ '<div class="profile-switch" role="group" aria-label="Screenshot viewport">'
+ "".join(
f'<button type="button" data-viewport="{profile}" aria-pressed="{str(profile == "desktop").lower()}">{profile.title()}</button>'
for profile in CAPTURE_PROFILES
)
+ "</div><main>"
+ gallery_sections
+ """</main><script>
const figures = document.querySelectorAll('.shot');
const buttons = document.querySelectorAll('[data-viewport]');
function selectProfile(profile) {
for (const figure of figures) figure.hidden = figure.dataset.profile !== profile;
for (const button of buttons) button.setAttribute('aria-pressed', String(button.dataset.viewport === profile));
}
for (const button of buttons) button.addEventListener('click', () => selectProfile(button.dataset.viewport));
selectProfile('desktop');
</script></div>"""
+ (REPO_DIR / "bin/templates/screenshots-footer.html").read_text(encoding="utf-8")
+ "</body></html>\n"
)
def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -> None:
provenance = build_provenance()
source_base_url = f"https://github.com/ArchiveBox/ArchiveBox/blob/{provenance['revision']}/"
@@ -205,7 +253,7 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
f'alt="{html.escape(str(capture["name"]))} — {profile}" width="{width}"></td>',
)
html_figures.append(
f'<figure class="shot shot-{profile}"><figcaption>{label}</figcaption>'
f'<figure class="shot shot-{profile}" data-profile="{profile}"><figcaption>{label}</figcaption>'
f'<a href="./{versioned_filename}">'
f'<img src="./{versioned_filename}" width="{width}" height="{height}" loading="lazy" '
f'alt="{html.escape(str(capture["name"]))} — {profile}"></a></figure>',
@@ -252,25 +300,15 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
encoding="utf-8",
)
html_path.write_text(
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
"<title>ArchiveBox UI Screenshots</title><style>"
"body{max-width:1900px;margin:0 auto;padding:20px;font:16px system-ui,sans-serif;background:#f6f7f9;color:#18202a}"
"a{color:#2563eb}article{margin:32px 0 64px}.shots{display:grid;grid-template-columns:2fr 1.35fr .8fr;gap:14px;align-items:start}"
"figure{min-width:0;margin:0}figcaption{font-weight:650;margin:0 0 8px}img{display:block;width:100%;height:auto;"
"border:1px solid #cbd5e1;border-radius:8px;background:white;box-shadow:0 8px 24px #0f172a18}code{overflow-wrap:anywhere}"
"@media(max-width:900px){.shots{grid-template-columns:1fr}body{padding:12px}}"
'</style></head><body><header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
gallery_header = (
'<header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
+ f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
f'<a href="https://github.com/ArchiveBox/ArchiveBox/commit/{html.escape(provenance["revision"])}">'
f"<code>{html.escape(provenance['revision'][:12])}</code></a> on "
f'<time datetime="{html.escape(provenance["generated_at"])}">{html.escape(provenance["generated_at"])}</time>. '
"Desktop, tablet, and mobile viewports are captured from the same build.</p></header><main>"
+ "".join(html_sections)
+ "</main></body></html>\n",
encoding="utf-8",
"Desktop, tablet, and mobile viewports are captured from the same build.</p></header>"
)
html_path.write_text(render_gallery(gallery_header, "".join(html_sections)), encoding="utf-8")
file_hashes = {
capture["filename"]: hashlib.sha256((html_path.parent / capture["filename"]).read_bytes()).hexdigest() for capture in captures
@@ -297,7 +335,27 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
screenshot_path.unlink()
def refresh_gallery(html_path: Path) -> None:
"""Apply current presentation to published capture content without changing provenance."""
document = html_path.read_text(encoding="utf-8")
header = re.search(r"<header>.*?</header>", document, re.DOTALL)
sections = re.search(r"<main>(.*?)</main>", document, re.DOTALL)
if not header or not sections:
raise SystemExit("Published screenshot gallery is missing its capture header or content")
content = sections[1]
for profile in CAPTURE_PROFILES:
# Galleries published before the switcher used classes alone.
content = content.replace(
f'<figure class="shot shot-{profile}">',
f'<figure class="shot shot-{profile}" data-profile="{profile}">',
)
html_path.write_text(render_gallery(header[0], content), encoding="utf-8")
def main() -> None:
if len(sys.argv) == 3 and sys.argv[1] == "refresh":
refresh_gallery(Path(sys.argv[2]))
return
if len(sys.argv) == 4 and sys.argv[1] == "validate":
validate_navigation(Path(sys.argv[2]), sys.argv[3])
return
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
"""Restore validated capture output without running application CI."""
import argparse
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
import os
from pathlib import Path, PurePosixPath
import subprocess
from urllib.error import HTTPError
from urllib.request import urlopen
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("kind", choices=["archivebox", "ios", "extension"])
parser.add_argument("destination", type=Path)
args = parser.parse_args()
args.destination.mkdir(parents=True, exist_ok=True)
# Only successful default-branch capture runs are eligible, never PR artifacts.
if os.environ.get("GH_TOKEN"):
repo = os.environ["GITHUB_REPOSITORY"]
branch = "dev" if args.kind == "archivebox" else "main"
runs = json.loads(
subprocess.check_output(
["gh", "api", f"repos/{repo}/actions/workflows/screenshots.yml/runs?branch={branch}&status=success&per_page=100"],
),
)
for run in runs["workflow_runs"]:
if run["event"] not in ("push", "workflow_dispatch"):
continue
artifacts = json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/actions/runs/{run['id']}/artifacts"]))
if any(a["name"] == "site-screenshots" and not a["expired"] for a in artifacts["artifacts"]):
subprocess.run(
["gh", "run", "download", str(run["id"]), "--repo", repo, "--name", "site-screenshots", "--dir", str(args.destination)],
check=True,
)
print(f"Restored validated captures from run {run['id']} ({run['head_sha']})")
raise SystemExit(0)
# Published captures remain available even after Actions artifacts expire.
base, manifest_name = {
"archivebox": ("https://archivebox.io/screenshots/", "build.json"),
"ios": ("https://archivebox.github.io/ios-archivebox/screenshots/", "manifest.json"),
"extension": ("https://archivebox.github.io/archivebox-browser-extension/screenshots/", "manifest.json"),
}[args.kind]
def fetch(name):
path = PurePosixPath(name)
if path.is_absolute() or ".." in path.parts or ":" in name or "\\" in name:
raise ValueError(f"Unsafe capture path: {name}")
with urlopen(base + name, timeout=60) as response:
data = response.read()
expected = hashes.get(name)
if expected and hashlib.sha256(data).hexdigest() != expected:
raise ValueError(f"Capture checksum mismatch: {name}")
target = args.destination / name
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(data)
try:
with urlopen(base + manifest_name, timeout=60) as response:
raw = response.read()
except HTTPError as error:
if error.code == 404 and args.kind == "ios":
print("No complete native gallery published yet; use the curated README screenshots.")
raise SystemExit(0)
raise
manifest = json.loads(raw)
hashes = {}
if args.kind == "archivebox":
hashes = manifest["files"]
names = [*hashes, "index.html"]
elif args.kind == "extension":
names = [image["file"] for capture in manifest["screenshots"] for image in capture["images"]]
else:
if manifest.get("complete") is not True:
raise ValueError("Published native gallery must be complete")
names = [capture["path"] for capture in manifest["captures"]]
with ThreadPoolExecutor(max_workers=8) as pool:
list(pool.map(fetch, names))
(args.destination / manifest_name).write_bytes(raw)
print(f"Restored {len(names)} published capture files, retaining revision {manifest['revision']}")
+26
View File
@@ -0,0 +1,26 @@
<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
<div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
<div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
</footer>
<script>
(() => {
const menu = document.querySelector('.abx-apps');
document.addEventListener('keydown', event => {
if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
});
document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
function revealAnchor() {
let target;
try { target = document.getElementById(decodeURIComponent(location.hash.slice(1))); } catch { return; }
if (!target) return;
let details = target.closest('details');
let opened = false;
while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
if (opened) requestAnimationFrame(() => target.scrollIntoView());
}
addEventListener('hashchange', revealAnchor);
revealAnchor();
})();
</script>
+9
View File
@@ -0,0 +1,9 @@
<header class="abx-header">
<a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">UI Screenshots</span></a>
<nav class="abx-nav" aria-label="Main navigation">
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
<a href="https://archivebox.io/screenshots/">Screenshots</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
<a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
<a class="abx-cta" href="https://archivebox.io/#install">Get started</a>
</nav>
</header>
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
# Real Compose regression checks. Build the image under test first, then run:
# IMAGE=archivebox/archivebox:issue-1872 ./bin/validate_docker_display.sh
set -Eeuo pipefail
export IMAGE="${IMAGE:-archivebox/archivebox:dev}"
validation_dir="$(mktemp -d "${TMPDIR:-/tmp}/archivebox-display.XXXXXX")"
compose=(docker compose --project-name "archivebox-display-$$" --file "$validation_dir/compose.yml")
cleanup() {
"${compose[@]}" down --volumes --remove-orphans
echo "Validation files: $validation_dir"
}
trap cleanup EXIT
cat > "$validation_dir/compose.yml" <<'YAML'
services:
archivebox:
image: ${IMAGE}
volumes:
- ./data:/data
- ./data/personas:/data/personas
- tmp:/tmp/archivebox
shm_size: 1gb
novnc:
image: theasp/novnc:latest
environment:
DISPLAY_WIDTH: 1280
DISPLAY_HEIGHT: 800
RUN_XTERM: "no"
healthcheck:
test: ["CMD", "xdpyinfo", "-display", ":0"]
interval: 2s
timeout: 2s
retries: 10
volumes:
tmp:
YAML
"${compose[@]}" up -d --wait novnc
# Check the real runtime dependency as well as successful X11 connectivity.
"${compose[@]}" run --rm archivebox sh -ec '
command -v xdpyinfo
test "$DISPLAY" = novnc:0.0
xdpyinfo >/dev/null
'
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
test "$DISPLAY" = novnc:0.0
xdpyinfo >/dev/null
'
# Explicit values belong to the caller, even when X11 is unavailable.
"${compose[@]}" run --rm -e DISPLAY=unavailable:0 archivebox sh -ec '
test "$DISPLAY" = unavailable:0
'
# An explicit empty value disables automatic display discovery.
"${compose[@]}" run --rm -e DISPLAY= archivebox sh -ec '
test "${DISPLAY+x}" = x
test -z "$DISPLAY"
'
"${compose[@]}" stop novnc
"${compose[@]}" run --rm archivebox sh -ec '
test "${DISPLAY+x}" != x
'
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
test "$DISPLAY" = novnc:0.0
'
echo 'PASS: Compose DISPLAY discovery, explicit override, empty opt-out, and headless fallback'
+16
View File
@@ -149,6 +149,22 @@ ssh -N -L 8080:127.0.0.1:8080 user@your-server
4. Open a page and confirm you can view it while logged in. For a new persona, log in to the sites you want to archive. Then follow [Save and check the page](#3-save-and-check-the-page), selecting the same persona.
#### One-off Compose setup
To set up a persona without starting the ArchiveBox server, enable the same `novnc` service and healthy `depends_on` block, then run:
```bash
docker compose up -d --wait novnc
docker compose run --rm archivebox persona create personal
docker compose run --rm -e DISPLAY=novnc:0.0 archivebox persona open personal
```
Connect at <http://localhost:8080/vnc.html>, log in, and close the browser when finished. The persona remains in `data/personas/personal` across one-off containers.
`persona open` resolves the installed Chromium binary automatically. Do not hard-code `/data/lib/env/bin/chromium` or a versioned Playwright cache path: the browser location varies by image version and architecture.
An explicitly set `DISPLAY` is preserved. When it is unset, ArchiveBox probes `novnc:0.0` and falls back to headless operation if unavailable. Set `DISPLAY=` to disable this automatic display selection.
## Troubleshooting Chromium Install
<a name="more-info--troubleshooting"></a>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "archivebox",
"version": "0.9.35rc464",
"version": "0.9.35rc485",
"repository": "github:ArchiveBox/ArchiveBox",
"license": "MIT",
"dependencies": {
+41 -25
View File
@@ -42,25 +42,20 @@
}
}
</script>
<link rel="stylesheet" href="site-chrome.css">
</head>
<body>
<header class="site-header">
<a class="brand" href="#top" aria-label="ArchiveBox home">
<img src="./icon.png" alt="" width="42" height="42">
<span>ArchiveBox</span>
</a>
<nav aria-label="Primary navigation"><details class="project-menu"><summary>Projects</summary><div class="project-menu__links"><a href="https://archivebox.io/" aria-current="page">ArchiveBox</a><a href="https://archivebox.github.io/ios-archivebox/">Apple apps</a><a href="https://archivebox.github.io/archivebox-browser-extension/">Browser extension</a><a href="https://archivebox.github.io/abx-dl/">abx-dl</a><a href="https://archivebox.github.io/abx-plugins/">abx-plugins</a></div></details>
<a href="#install"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 3v11"/><path d="m7 10 5 5 5-5"/><path d="M5 21h14"/></svg><span>Install</span></a>
<a href="#use-cases"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2"/><circle cx="9" cy="7" r="4"/><path d="M22 21v-2a4 4 0 0 0-3-3.87"/><path d="M16 3.13a4 4 0 0 1 0 7.75"/></svg><span>Use cases</span></a>
<a href="#configuration"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 21v-7"/><path d="M4 10V3"/><path d="M12 21v-9"/><path d="M12 8V3"/><path d="M20 21v-5"/><path d="M20 12V3"/><path d="M2 14h4"/><path d="M10 8h4"/><path d="M18 16h4"/></svg><span>Configuration</span></a>
<a href="#layout-security"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10Z"/><path d="m9 12 2 2 4-4"/></svg><span>Layout &amp; security</span></a>
<a href="#background"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 8v4l3 3"/><circle cx="12" cy="12" r="9"/></svg><span>Backstory</span></a>
<a href="#ecosystem"><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="3" y="3" width="7" height="7" rx="1.5"/><rect x="14" y="3" width="7" height="7" rx="1.5"/><rect x="14" y="14" width="7" height="7" rx="1.5"/><path d="M10 6.5h4"/><path d="M17.5 10v4"/><path d="M10 17.5h4"/></svg><span>Ecosystem</span></a>
<a href="https://github.com/ArchiveBox/ArchiveBox/wiki"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2Z"/></svg><span>Docs</span></a>
<a href="https://github.com/ArchiveBox/ArchiveBox"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M15 22v-4a4.8 4.8 0 0 0-.09-.93"/><path d="M9 22v-4a4.8 4.8 0 0 1 .09-.93"/><path d="M17.8 20.2c1.4-.5 2.2-1.7 2.2-3.7V12.7c0-.7-.2-1.4-.7-2 .1-.3.3-1.7-.1-3.2 0 0-1.1-.4-3.4.9a11.5 11.5 0 0 0-6 0C7.5 7.1 6.4 7.5 6.4 7.5c-.4 1.5-.2 2.9-.1 3.2-.5.6-.7 1.3-.7 2v3.8c0 2 1 3.2 2.4 3.7"/><path d="M9 18c-3 1.3-5 0-6-2"/></svg><span>GitHub</span></a>
<a href="https://pypi.org/project/archivebox/"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M16.5 9.4 7.5 4.2"/><path d="M21 16V8a2 2 0 0 0-1-1.73l-7-4a2 2 0 0 0-2 0l-7 4A2 2 0 0 0 3 8v8a2 2 0 0 0 1 1.73l7 4a2 2 0 0 0 2 0l7-4A2 2 0 0 0 21 16Z"/><path d="M3.3 7 12 12l8.7-5"/><path d="M12 22V12"/></svg><span>PyPI</span></a>
</nav>
</header>
<!-- BEGIN ARCHIVEBOX HEADER — generated by monorepo/site-chrome/sync.py -->
<header class="abx-header">
<a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">Web Archive</span></a>
<nav class="abx-nav" aria-label="Main navigation">
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
<a href="#use-cases">Use cases</a><a href="#configuration">Configuration</a><a href="#layout-security">Layout &amp; security</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
<a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
<a class="abx-cta" href="#install">Get started</a>
</nav>
</header>
<!-- END ARCHIVEBOX HEADER -->
<main id="top">
<section class="hero section">
@@ -463,18 +458,39 @@
</section>
</main>
<footer class="site-footer">
<div class="brand">
<img src="./icon.png" alt="" width="34" height="34">
<span>ArchiveBox</span>
</div>
<p>Open-source self-hosted web archiving. MIT licensed, nonprofit-supported, and built for durable collections.</p>
<!-- BEGIN ARCHIVEBOX FOOTER — generated by monorepo/site-chrome/sync.py -->
<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
<div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
<div class="abx-footer-local"><p>Open-source self-hosted web archiving. MIT licensed, nonprofit-supported, and built for durable collections.</p>
<div>
<a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE">License</a>
<a href="https://github.com/ArchiveBox/ArchiveBox/releases">Changelog</a>
<a href="https://zulip.archivebox.io">Community chat</a>
<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Donations">Donations</a>
</div>
</footer>
</div></div>
<div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
</footer>
<script>
(() => {
const menu = document.querySelector('.abx-apps');
document.addEventListener('keydown', event => {
if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
});
document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
function revealAnchor() {
let target;
try { target = document.getElementById(decodeURIComponent(location.hash.slice(1))); } catch { return; }
if (!target) return;
let details = target.closest('details');
let opened = false;
while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
if (opened) requestAnimationFrame(() => target.scrollIntoView());
}
addEventListener('hashchange', revealAnchor);
revealAnchor();
})();
</script>
<!-- END ARCHIVEBOX FOOTER -->
</body>
</html>
+75
View File
@@ -0,0 +1,75 @@
/* Shared ArchiveBox site chrome. Source: ArchiveBox/monorepo/site-chrome. */
.abx-header, .abx-footer { box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 14px; line-height: 1.5; letter-spacing: normal; text-align: left; }
.abx-header *, .abx-footer * { box-sizing: border-box; }
.abx-header { position: sticky; top: 0; z-index: 100; display: flex; align-items: center; justify-content: space-between; gap: 24px; width: 100%; max-width: none; margin: 0; padding: 15px max(24px, calc((100% - 1280px) / 2)); border: 0; border-bottom: 1px solid #e9e2e3; background: #fcfaf7f5; color: #28252a; backdrop-filter: blur(16px); }
.abx-header a, .abx-footer a { text-decoration: none; text-underline-offset: .24em; }
.abx-header .abx-brand, .abx-footer .abx-brand { display: inline-flex; flex: 0 0 auto; align-items: center; gap: 11px; margin: 0; padding: 0; color: inherit; font-size: 19px; font-weight: 750; line-height: 1.2; letter-spacing: -.5px; white-space: nowrap; }
.abx-brand .abx-logo { display: block; flex: 0 0 auto; width: 32px; height: 32px; margin: 0; }
.abx-header .abx-subsite { margin-left: 2px; color: #716c74; font-size: 12px; font-weight: 450; letter-spacing: 0; }
.abx-header .abx-nav { display: flex; align-items: center; justify-content: flex-end; flex-wrap: wrap; gap: 0; width: auto; max-width: none; margin: 0; padding: 0; font-size: 13px; font-weight: 600; }
.abx-nav > a, .abx-nav > details { margin: 0 0 0 15px; padding: 0 0 0 15px; border: 0; border-left: 1px solid #ded4d8; border-radius: 0; background: none; }
.abx-nav > :first-child { margin-left: 0; padding-left: 0; border-left: 0; }
.abx-header .abx-nav a { display: inline-flex; align-items: center; min-height: 36px; color: #4d424b; white-space: nowrap; }
.abx-header .abx-nav a:hover { color: #9b2854; }
.abx-header .abx-nav .abx-cta { margin-left: 18px; padding: 9px 17px; border: 1px solid #9b2854; border-radius: 24px; background: #9b2854; color: #fff; }
.abx-header .abx-nav .abx-cta:hover { background: #782040; color: #fff; }
.abx-header .abx-apps { position: relative; }
.abx-apps > summary { display: flex; align-items: center; gap: 7px; min-height: 36px; list-style: none; cursor: pointer; color: #4d424b; }
.abx-apps > summary::-webkit-details-marker { display: none; }
.abx-apps > summary::before { content: '›'; display: inline-block; color: #9b2854; font-size: 22px; line-height: 1; transition: transform .15s; }
.abx-apps[open] > summary::before { transform: rotate(90deg); }
.abx-header .abx-app-links { position: absolute; right: 0; top: calc(100% + 12px); width: 305px; max-height: min(70vh, 580px); overflow-y: auto; padding: 10px; border: 1px solid #e9e2e3; border-radius: 14px; background: #fcfaf7; box-shadow: 0 16px 48px #2d152522; }
.abx-header .abx-app-links a { display: flex; min-height: 42px; padding: 9px 12px; border-radius: 7px; white-space: normal; }
.abx-header .abx-app-links code { display: inline; vertical-align: baseline; font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 1em; font-weight: inherit; line-height: inherit; margin: 0 .3em; padding: 0; border: 0; background: transparent; color: inherit; }
.abx-header .abx-app-links a:hover { background: #f2e8ed; }
.abx-footer { position: relative; z-index: 1; display: block; width: 100%; max-width: none; margin: 72px 0 0; padding: 60px max(24px, calc((100% - 1280px) / 2)) 24px; border: 0; border-top: 3px solid #7e284a; background: radial-gradient(ellipse at 12% 0, #54243b55, transparent 48%), linear-gradient(120deg, #201923, #14151b 75%); color: #ddd2dd; color-scheme: dark; }
.abx-footer .abx-footer-intro { display: flex; align-items: flex-start; justify-content: space-between; gap: 32px; margin: 0 0 42px; }
.abx-footer .abx-brand { color: #fff4ed; font-size: 24px; }
.abx-footer .abx-brand .abx-logo { width: 40px; height: 40px; }
.abx-footer .abx-tagline { max-width: 430px; margin: 12px 0 0; color: #b8acbb; font-size: 14px; line-height: 1.7; }
.abx-footer .abx-footer-motto { margin: 2px 0 0; color: #d8b39f; font: italic 25px/1.45 Georgia, serif; text-align: right; }
.abx-footer .abx-footer-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 42px; margin: 0; }
.abx-footer .abx-footer-column { margin: 0; padding: 20px 0 0; border: 0; border-top: 1px solid #67505e66; background: none; }
.abx-footer .abx-footer-column h2 { margin: 0 0 19px; padding: 0; border: 0; color: #e4acbb; font: 650 12px/1.4 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; letter-spacing: .12em; text-transform: uppercase; }
.abx-footer .abx-footer-column ul { display: block; list-style: none; margin: 0; padding: 0; }
.abx-footer .abx-footer-column li { display: flex; align-items: center; gap: 10px; margin: 0; padding: 0; min-height: 36px; }
.abx-footer .abx-footer-column li::before { content: none; }
.abx-footer .abx-footer-column a { color: #ddd2dd; font-size: 13px; line-height: 1.55; }
.abx-footer .abx-footer-column a:first-child { padding: 6px 0; }
.abx-footer a:hover { color: #fff; text-decoration: underline; }
.abx-footer .abx-source { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 28px; width: 28px; height: 28px; border: 1px solid transparent; border-radius: 50%; color: #a58d9f; }
.abx-footer .abx-source:hover { color: #f3b5cb; border-color: #80516a; background: #ffffff08; }
.abx-source svg { width: 15px; height: 15px; fill: currentColor; }
.abx-footer .abx-footer-bottom { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 12px 24px; margin-top: 32px; padding-top: 21px; border-top: 1px solid #67505e66; color: #a99bab; font-size: 12px; }
.abx-footer .abx-footer-bottom p { margin: 0; font-size: inherit; color: inherit; }
.abx-footer .abx-footer-bottom a { color: #cebbc9; }
.abx-footer .abx-footer-local { margin: 24px 0 0; padding: 16px 0 0; border: 0; border-top: 1px solid #67505e66; border-radius: 0; background: none; color: #baacbc; font-size: 12px; }
.abx-footer-local > summary { cursor: pointer; color: #d6bfce; }
.abx-footer-local > div { padding-top: 18px; }
.abx-footer-local a { color: #e5aabd; }
.abx-footer-local .brand, .abx-footer-local .footer-intro, .abx-footer-local .footer-grid, .abx-footer-local .footer-meta { color: inherit; }
.abx-footer-local .footer-card { background: transparent; border-color: #67505e66; }
.abx-footer-local h2, .abx-footer-local h3, .abx-footer-local strong { color: #e4d8e3; }
.abx-footer-local .footer-link-copy span, .abx-footer-local .footer-card-subtitle { color: #baacbc; }
.abx-header a:focus-visible, .abx-header summary:focus-visible, .abx-footer a:focus-visible, .abx-footer summary:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
@media (max-width: 1100px) {
.abx-header { flex-wrap: wrap; gap: 8px 24px; padding: 12px 20px; }
.abx-header .abx-nav { justify-content: flex-start; }
.abx-header .abx-app-links { position: fixed; top: auto; left: 20px; right: 20px; width: auto; margin-top: 10px; }
}
@media (max-width: 700px) {
.abx-header { position: relative; padding: 14px 18px; }
.abx-header .abx-brand { flex-wrap: wrap; row-gap: 5px; }
.abx-header .abx-nav { width: 100%; gap: 4px 0; font-size: 12px; }
.abx-nav > a, .abx-nav > details { margin-left: 10px; padding-left: 10px; }
.abx-header .abx-nav .abx-cta { margin-left: 10px; padding: 7px 13px; }
.abx-header .abx-app-links { left: 12px; right: 12px; max-height: 60vh; }
.abx-footer { padding: 40px 22px 22px; margin-top: 48px; }
.abx-footer .abx-footer-intro { display: block; margin-bottom: 30px; }
.abx-footer .abx-footer-motto { margin-top: 20px; text-align: left; font-size: 22px; }
.abx-footer .abx-footer-grid { grid-template-columns: 1fr; gap: 26px; }
.abx-footer .abx-footer-column li { min-height: 40px; }
}
@media (prefers-reduced-motion: reduce) { .abx-apps > summary::before { transition: none; } }
.abx-footer-local > div > a + a { margin-left: 12px; }
+10 -5
View File
@@ -125,7 +125,8 @@ code {
gap: 8px 18px;
}
.site-header nav a {
.site-header nav a,
.site-header nav .project-menu > summary {
display: inline-flex;
align-items: center;
gap: 8px;
@@ -140,7 +141,8 @@ code {
transition: border-color 160ms ease, color 160ms ease;
}
.site-header nav a:hover {
.site-header nav a:hover,
.site-header nav .project-menu > summary:hover {
color: var(--text);
text-decoration: none;
border-color: hsl(var(--primary) / 0.42);
@@ -806,7 +808,7 @@ h3 {
}
.site-header {
position: static;
position: relative;
width: 100%;
margin: 0;
padding: 14px 16px;
@@ -816,7 +818,8 @@ h3 {
width: 100%;
}
.site-header nav a {
.site-header nav a,
.site-header nav .project-menu > summary {
font-size: 0.74rem;
letter-spacing: 0.1em;
}
@@ -884,7 +887,9 @@ h3 {
/* Shared project navigation stays usable without JavaScript. */
.project-menu { position: relative; flex: 0 0 auto; }
.project-menu > summary { cursor: pointer; padding: 10px 0; font-size: 13px; font-weight: 600; }
.project-menu > summary { cursor: pointer; list-style: none; }
.project-menu > summary::-webkit-details-marker { display: none; }
.project-menu > summary::after { content: "▾"; }
.project-menu > summary:focus-visible, .project-menu__links a:focus-visible { outline: 3px solid #9b2854; outline-offset: 3px; }
.project-menu__links { position: absolute; right: 0; top: 100%; z-index: 30; width: 220px; max-width: calc(100vw - 32px); padding: 8px; border: 1px solid #e9e2e3; border-radius: 12px; background: #fff; box-shadow: 0 12px 32px #28252a20; }
header nav .project-menu__links a { display: flex; align-items: center; min-height: 44px; padding: 8px 12px; color: #28252a; font-size: 13px; letter-spacing: normal; text-transform: none; text-decoration: none; white-space: nowrap; }
+5 -5
View File
@@ -1,6 +1,6 @@
[project]
name = "archivebox"
version = "0.9.35rc464"
version = "0.9.35rc485"
requires-python = ">=3.13"
description = "Self-hosted internet archiving solution."
authors = [{name = "Nick Sweeting", email = "pyproject.toml@archivebox.io"}]
@@ -78,9 +78,9 @@ dependencies = [
### Extractor dependencies (runtime binaries resolved through abxpkg)
### Binary/Package Management
"abxbus==2.5.56", # direct imports only; version constrained by abx-dl -> abx-plugins
"abxpkg==1.12.117", # direct imports only; version constrained by abx-dl -> abx-plugins
"abx-plugins[opencode]==1.12.246", # optional AI runtime clients are owned by the plugin
"abx-dl==1.12.284", # shared ArchiveBox downloader package
"abxpkg==1.12.125", # direct imports only; version constrained by abx-dl -> abx-plugins
"abx-plugins[opencode]==1.12.267", # optional AI runtime clients are owned by the plugin
"abx-dl==1.12.301", # shared ArchiveBox downloader package
### UUID7 backport for Python <3.14
"uuid7>=0.1.0; python_version < '3.14'", # provides the uuid_extensions module on Python 3.13
"browser-cookie3>=0.20.1",
@@ -342,7 +342,7 @@ Donate = "https://github.com/ArchiveBox/ArchiveBox/wiki/Donations"
[tool.bumpver]
current_version = "v0.9.35rc464"
current_version = "v0.9.35rc485"
version_pattern = "vMAJOR.MINOR.PATCH[PYTAGNUM]"
commit_message = "bump version {old_version} -> {new_version}"
tag_message = "{new_version}"
Generated
+15 -15
View File
@@ -13,18 +13,18 @@ supported-markers = [
]
[options]
exclude-newer = "2026-09-13T04:04:13.517458486Z"
exclude-newer = "2026-09-13T09:01:47.86829829Z"
exclude-newer-span = "P5D"
[options.exclude-newer-package]
abxbus = { timestamp = "2026-09-18T04:04:12.517470189Z", span = "PT1S" }
abxbus = { timestamp = "2026-09-18T09:01:46.868310342Z", span = "PT1S" }
abx-plugins = "2100-01-01T00:00:00Z"
abx-dl = "2100-01-01T00:00:00Z"
abxpkg = "2100-01-01T00:00:00Z"
[[package]]
name = "abx-dl"
version = "1.12.284"
version = "1.12.301"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "abx-plugins", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -38,14 +38,14 @@ dependencies = [
{ name = "rich", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/79/b1/e3e289a4d43e79df6319dfe55917b195c9f1c58e49bd0f3c0d1ce1382cc3/abx_dl-1.12.284.tar.gz", hash = "sha256:c3a07d66c1c52ad23475b4c03d65b77632057a838648422e55807772a3aaaa0e", size = 88255 }
sdist = { url = "https://files.pythonhosted.org/packages/a5/7c/b0be48270dbb8074757f200cbfe9c9a76a1f913df9cea1d854f55a7352f0/abx_dl-1.12.301.tar.gz", hash = "sha256:4e0299f61f277173e6982739998915163b1991bc11cb84de4b109864b85b444e", size = 87712 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d9/cd/3338299f434eb20cb533f66499c62eb303122e05aa14467cfa61c3104b90/abx_dl-1.12.284-py3-none-any.whl", hash = "sha256:43c8e0a902e4e0f6eec8d8d5738af5a03c563b4026f03a598186258143d731d1", size = 93261 },
{ url = "https://files.pythonhosted.org/packages/48/82/153ef21ec268c57d3efe9d58ca5b58948ef63fe7e284bbe7247444ea6b42/abx_dl-1.12.301-py3-none-any.whl", hash = "sha256:01eb13755c4b5177e3111d0b62d4d1af841e37f99c20040e5a4672002011e55e", size = 93006 },
]
[[package]]
name = "abx-plugins"
version = "1.12.246"
version = "1.12.267"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "abxbus", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -54,9 +54,9 @@ dependencies = [
{ name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "uv", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f1/b9/7cf8a8831f86f8d97b9a2a4bc73bdd3b67876ea4bb58f4c7ceb1def88f82/abx_plugins-1.12.246.tar.gz", hash = "sha256:22843c049d9412d6e5e9f68d059a24b7dbf21159900a98070e14699c6b534477", size = 309082, upload-time = "2026-09-18T03:27:03.685Z" }
sdist = { url = "https://files.pythonhosted.org/packages/36/ed/26a6777272080d5ab7b5cef8b6ba122a112eb2c3f03b8a9bbbdeb7b3b344/abx_plugins-1.12.267.tar.gz", hash = "sha256:170d1637fef0526bbec2475c96577dd35610e7e3b0c573036d901b49dc841749", size = 340635, upload-time = "2026-09-18T08:53:12.82Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/9c/cb/a218ed722799f59a2030fc92e4f2055128d966a9c205cf21a4f63fd855da/abx_plugins-1.12.246-py3-none-any.whl", hash = "sha256:09f4d89aa5b541cf2a632f10c1f710ca7b82af54c5439b63783cef376666cf52", size = 473464, upload-time = "2026-09-18T03:27:02.085Z" },
{ url = "https://files.pythonhosted.org/packages/7d/48/7edc81c7e372cb1485f797294a3bce9abe56fc3095e2480d40f9b44563c0/abx_plugins-1.12.267-py3-none-any.whl", hash = "sha256:5a151eae7eaa86e78cc0d4d9d15a7d8029b76d8ef3212a0e46ac0878199af106", size = 506879, upload-time = "2026-09-18T08:53:11.152Z" },
]
[package.optional-dependencies]
@@ -82,7 +82,7 @@ wheels = [
[[package]]
name = "abxpkg"
version = "1.12.117"
version = "1.12.125"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "platformdirs", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -90,9 +90,9 @@ dependencies = [
{ name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/00/35/bbfaccae109328c716cea4e0cdbf2d2dc5975f448203bb3f4daa605591a6/abxpkg-1.12.117.tar.gz", hash = "sha256:4d88cc293f343eee2be7ecaab1203dfbc20571a4a39b2afa2f866c9ef8032dec", size = 239387, upload-time = "2026-09-02T19:57:26.142Z" }
sdist = { url = "https://files.pythonhosted.org/packages/87/fc/cf3f9d370ee93202c2e9239d267ca0b0dece7dc3595d36eebbedf3ab2946/abxpkg-1.12.125.tar.gz", hash = "sha256:14d98f4ff305f5ac84030f4d1f16a2f6c511ff5480ebfb08ba133f367a0770e1", size = 239374, upload-time = "2026-09-18T07:29:21.549Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/73/46/372be1f8fa915eb254ad865b97fb41073acdb5573dbce3e23a6083417d87/abxpkg-1.12.117-py3-none-any.whl", hash = "sha256:76ec8afaac7b2d7aece7bdda2bbf606e6373db51eb4dcb906008052d09e59577", size = 253749, upload-time = "2026-09-02T19:57:27.437Z" },
{ url = "https://files.pythonhosted.org/packages/bf/5c/dd0d1690e4c1d6241f44a5194a525af6769acdf33dd366a6b4b56f33c2b0/abxpkg-1.12.125-py3-none-any.whl", hash = "sha256:cfef2e06ed01b8774e8bdf0b8de36ab592a0dfa273993c4824628ab17d7d0fd1", size = 253748, upload-time = "2026-09-18T07:29:23.043Z" },
]
[[package]]
@@ -127,7 +127,7 @@ wheels = [
[[package]]
name = "archivebox"
version = "0.9.35rc464"
version = "0.9.35rc485"
source = { editable = "." }
dependencies = [
{ name = "abx-dl", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -226,10 +226,10 @@ dev = [
[package.metadata]
requires-dist = [
{ name = "abx-dl", specifier = "==1.12.284" },
{ name = "abx-plugins", extras = ["opencode"], specifier = "==1.12.246" },
{ name = "abx-dl", specifier = "==1.12.301" },
{ name = "abx-plugins", extras = ["opencode"], specifier = "==1.12.267" },
{ name = "abxbus", specifier = "==2.5.56" },
{ name = "abxpkg", specifier = "==1.12.117" },
{ name = "abxpkg", specifier = "==1.12.125" },
{ name = "archivebox", extras = ["sonic", "ldap", "debug"], marker = "extra == 'all'" },
{ name = "atomicwrites", specifier = "==1.4.1" },
{ name = "base32-crockford", specifier = ">=0.3.0" },