ArchiveBox Agent Guide

ArchiveBox is the full self-hosted web archiving app. Keep this repo on the dev branch.

Shared Standards

Use uv and uv run for Python commands. Do not use system python, direct .venv/bin/python, or pip commands.
Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
Start behavior fixes with a red failing test when a test is requested or practical.
Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
Read README.md for the full setup, CLI, Docker, API, and release surface.

Concurrency Contract

A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.

Development Setup

uv sync --dev --all-extras
mkdir -p data
cd data
uv run --project .. archivebox init --install

Run collection commands from inside an initialized data directory:

cd data
uv run --project .. archivebox status
uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
uv run --project .. archivebox run

User-Facing Setup

Recommended CLI install:

uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init --install
archivebox add --plugins=parse_txt_urls 'https://example.com/'

Alternative install methods:

Docker Compose / Docker
Homebrew
Debian package
pip

Basic Usage

cd ~/archivebox/data
archivebox version
archivebox help
archivebox status
archivebox install
archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
archivebox list --json --with-headers
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
archivebox run

Verification

Run the test suite with uv run pytest archivebox/tests -x and repository checks with uv run prek run --all-files. For focused work, select the relevant tests.

Releases are published only by .github/workflows/release.yml after complete CI succeeds for the exact commit. Pushes to main prepare stable patch releases automatically; dev prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.

3.9 KiB Raw Permalink Blame History