Compare commits
180
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
810a5e4f59 | ||
|
|
372d06fe81 | ||
|
|
98b3b67f36 | ||
|
|
193b699b83 | ||
|
|
273c5d6f3c | ||
|
|
21ed8ea960 | ||
|
|
716178290a | ||
|
|
3177596156 | ||
|
|
0c1d0f561c | ||
|
|
746d487390 | ||
|
|
be35cb45b0 | ||
|
|
afd7576ad8 | ||
|
|
aba8f1342a | ||
|
|
b50f1e7692 | ||
|
|
a056d5daf7 | ||
|
|
4384d09e3f | ||
|
|
3f3166cfea | ||
|
|
3eb01ddd25 | ||
|
|
4b52eed55a | ||
|
|
105c1e793e | ||
|
|
f4b9634863 | ||
|
|
35a2e9c9c6 | ||
|
|
d76083d26c | ||
|
|
f5b2f6a030 | ||
|
|
83c83e53c3 | ||
|
|
0a4f328fa6 | ||
|
|
c4388d2435 | ||
|
|
7af15501f8 | ||
|
|
b9c9daaa65 | ||
|
|
c3f9c1a9ef | ||
|
|
e03db54bca | ||
|
|
23a5c6bba9 | ||
|
|
7a5ad4c99a | ||
|
|
1a93058585 | ||
|
|
85033aa622 | ||
|
|
922bcc05ba | ||
|
|
070b8ae9c5 | ||
|
|
f77ac105a1 | ||
|
|
f41f0afa03 | ||
|
|
31062043fc | ||
|
|
eeb06148ad | ||
|
|
80d3804e56 | ||
|
|
3a4696165d | ||
|
|
28b3bda674 | ||
|
|
8de094435e | ||
|
|
4c993242dc | ||
|
|
c3e6f62d0f | ||
|
|
f141cb841a | ||
|
|
10152bce2a | ||
|
|
d6fe46d327 | ||
|
|
a7d4245c36 | ||
|
|
8b634f9568 | ||
|
|
9cddb2b2dc | ||
|
|
7e7af50eff | ||
|
|
e019a2b07a | ||
|
|
0d2a4bf393 | ||
|
|
8914d95e83 | ||
|
|
0f965972e3 | ||
|
|
7a66e8ef6e | ||
|
|
be857415ad | ||
|
|
bfd6b3c2a7 | ||
|
|
69667e5cdb | ||
|
|
fffbdf0226 | ||
|
|
d44422fccb | ||
|
|
9c7a3f1f16 | ||
|
|
52328ad841 | ||
|
|
92008db9c4 | ||
|
|
e2d3b74dcd | ||
|
|
0b74322ea3 | ||
|
|
4d888e94d0 | ||
|
|
43c6dc2ea7 | ||
|
|
eeb057433a | ||
|
|
f31727e157 | ||
|
|
03c7074f4e | ||
|
|
845822f7ae | ||
|
|
592c79a52e | ||
|
|
d0e01fe00f | ||
|
|
e90aa46d73 | ||
|
|
deddb4d415 | ||
|
|
9a6adf146c | ||
|
|
2b568939ec | ||
|
|
8b98009cb0 | ||
|
|
01f2af413a | ||
|
|
27d243742c | ||
|
|
de595545ba | ||
|
|
22f0009a79 | ||
|
|
ab5458e22f | ||
|
|
7eddcd6e3c | ||
|
|
9b2fc58e5c | ||
|
|
e239984415 | ||
|
|
a7baeae476 | ||
|
|
b4f9633831 | ||
|
|
16b76cac0d | ||
|
|
48ced005a8 | ||
|
|
498d7715b2 | ||
|
|
736a93cb92 | ||
|
|
5c99cdf7c9 | ||
|
|
8da338a887 | ||
|
|
155097ce6f | ||
|
|
eb9bb069d6 | ||
|
|
53523a6a1e | ||
|
|
ffdc16fc22 | ||
|
|
8d14775759 | ||
|
|
cd5f96c26a | ||
|
|
877e5f3104 | ||
|
|
d91ea053a2 | ||
|
|
f326097d33 | ||
|
|
efa29074f8 | ||
|
|
99fcb02963 | ||
|
|
e4981957d4 | ||
|
|
8c83947331 | ||
|
|
13f9564fa4 | ||
|
|
4d8b162099 | ||
|
|
b8d7669c6c | ||
|
|
d08b483c3b | ||
|
|
f3f7d564e3 | ||
|
|
38b65fee63 | ||
|
|
6580ed01f3 | ||
|
|
8cc3d78b3d | ||
|
|
1dd74f3ec8 | ||
|
|
bacd3d053e | ||
|
|
287e792d29 | ||
|
|
383adc4e35 | ||
|
|
ba740b4568 | ||
|
|
e7c419ce78 | ||
|
|
363b79e13f | ||
|
|
05b1c8b829 | ||
|
|
cf03d6c8b3 | ||
|
|
6cfdbbd68b | ||
|
|
2c0d5584e2 | ||
|
|
288af543a6 | ||
|
|
afeb937db7 | ||
|
|
c8a699da0b | ||
|
|
6e49cf279f | ||
|
|
631378a80d | ||
|
|
1ef37a02dd | ||
|
|
b618e20fbf | ||
|
|
6be2bd85e5 | ||
|
|
8149cc0320 | ||
|
|
72b5439137 | ||
|
|
8f87aa9e43 | ||
|
|
e83f2c1808 | ||
|
|
a6d56657b2 | ||
|
|
67af2f3a95 | ||
|
|
b47fc056e3 | ||
|
|
007ecaea80 | ||
|
|
d2eea2e0f1 | ||
|
|
a88fa72b16 | ||
|
|
fead132b5c | ||
|
|
bfef7285f6 | ||
|
|
35efcea657 | ||
|
|
9b58aa30e5 | ||
|
|
38dce9b358 | ||
|
|
bc8daf18c0 | ||
|
|
4cd5d170ea | ||
|
|
4ec52a646b | ||
|
|
ee78c2c58b | ||
|
|
bdfa7f45fc | ||
|
|
1ec5582527 | ||
|
|
75697e4ebf | ||
|
|
7a19792be8 | ||
|
|
601b2db35b | ||
|
|
16506ed55a | ||
|
|
cc218245c3 | ||
|
|
b625cf4958 | ||
|
|
92c0f52324 | ||
|
|
880ab87da2 | ||
|
|
7e34e43cf5 | ||
|
|
4a1e06fda7 | ||
|
|
e1647d8f93 | ||
|
|
0badc70c4d | ||
|
|
9fbddc5b99 | ||
|
|
eb7e6f2ba2 | ||
|
|
a7eb82ffda | ||
|
|
d3f2509c30 | ||
|
|
3e1a31755c | ||
|
|
70bce92bf3 | ||
|
|
80e4a1c4f8 | ||
|
|
ceb3c04aff | ||
|
|
953f29b65e |
No files matched your search
@@ -30,8 +30,8 @@ body:
|
||||
description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
|
||||
render: markdown
|
||||
placeholder: |
|
||||
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
|
||||
2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
|
||||
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 5797:5797 archivebox/archivebox` in iTerm2
|
||||
2. Went to the https://127.0.0.1:5797/add/ page in Google Chrome
|
||||
3. Typed 'https://example.com/xyz' into the 'Add URL' input field
|
||||
4. Clicked the 'Add+' button
|
||||
5. Got a 500 error and saw the errors below in terminal
|
||||
@@ -138,7 +138,7 @@ body:
|
||||
archivebox:
|
||||
image: archivebox/archivebox:latest
|
||||
ports:
|
||||
- 8000:8000
|
||||
- 5797:5797
|
||||
volumes:
|
||||
- ./data:/data
|
||||
environment:
|
||||
|
||||
@@ -116,6 +116,10 @@
|
||||
}
|
||||
],
|
||||
"docker_binaries": [
|
||||
{
|
||||
"name": "{BASH_BINARY}",
|
||||
"binproviders": "env,brew,apt"
|
||||
},
|
||||
{
|
||||
"name": "{DOCKER_BINARY}",
|
||||
"binproviders": "env"
|
||||
|
||||
@@ -246,6 +246,26 @@ jobs:
|
||||
if: inputs.push_digests
|
||||
run: echo ${{ steps.docker_build.outputs.digest }}
|
||||
|
||||
- name: Verify OpenCode install with a read-only browser profile
|
||||
shell: bash
|
||||
env:
|
||||
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
PROFILE_DIR="$(mktemp -d)"
|
||||
"$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
|
||||
--volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
|
||||
"$IMAGE" bash -c '
|
||||
set -Eeuo pipefail
|
||||
test -w "$XDG_CONFIG_HOME"
|
||||
test ! -w "$XDG_CONFIG_HOME/chromium"
|
||||
# Check the shipped executable before the installer can repair
|
||||
# an image that accidentally omitted this opt-in dependency.
|
||||
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
|
||||
archivebox install opencode
|
||||
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
|
||||
'
|
||||
|
||||
- name: Validate exact built image version and commit
|
||||
if: ${{ !inputs.push_digests }}
|
||||
shell: bash
|
||||
@@ -296,8 +316,8 @@ jobs:
|
||||
MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
|
||||
fi
|
||||
COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
|
||||
LIMIT=$((750 * 1024 * 1024))
|
||||
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
|
||||
LIMIT=$((900 * 1024 * 1024))
|
||||
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=900 MiB"
|
||||
(( COMPRESSED_SIZE < LIMIT ))
|
||||
|
||||
- name: Resolve real Docker CLI test dependencies through abxpkg
|
||||
@@ -378,6 +398,12 @@ jobs:
|
||||
IMAGE="${{ matrix.local_tag }}" \
|
||||
"$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
|
||||
|
||||
- name: Validate Docker port compatibility
|
||||
shell: bash
|
||||
env:
|
||||
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
|
||||
run: '"$BASH_BINARY" bin/validate_docker_ports.sh'
|
||||
|
||||
- name: Validate Docker CLI add/list and compose behavior
|
||||
if: inputs.full_tests && matrix.platform == 'linux/amd64'
|
||||
shell: bash
|
||||
@@ -401,13 +427,13 @@ jobs:
|
||||
"$DOCKER_BINARY" compose run --rm archivebox init
|
||||
"$DOCKER_BINARY" compose up -d --wait
|
||||
"$DOCKER_BINARY" network connect archivebox_default www-nginx
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
|
||||
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
|
||||
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/static/admin/js/jquery.init.js')"
|
||||
[[ "$index_html" == *ArchiveBox* ]]
|
||||
[[ "$admin_js" == *window.django* ]]
|
||||
|
||||
"$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
|
||||
[[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
|
||||
"$DOCKER_BINARY" compose down
|
||||
|
||||
@@ -423,7 +449,7 @@ jobs:
|
||||
'${{ runner.temp }}/archivebox-image.tar' \
|
||||
'${{ steps.docker_build_pr.outputs.digest }}' \
|
||||
"$LOCAL_IMAGE_ID" \
|
||||
"$((750 * 1024 * 1024))" <<'PY'
|
||||
"$((900 * 1024 * 1024))" <<'PY'
|
||||
from hashlib import sha256
|
||||
import json
|
||||
import os
|
||||
|
||||
@@ -49,3 +49,17 @@ jobs:
|
||||
push_digests: true
|
||||
full_tests: false
|
||||
secrets: inherit
|
||||
|
||||
opencode:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
test_file:
|
||||
- archivebox/tests/test_opencode_agent.py
|
||||
- archivebox/tests/test_opencode_browser.py
|
||||
uses: ./.github/workflows/test-parallel.yml
|
||||
with:
|
||||
test_file: ${{ matrix.test_file }}
|
||||
secrets: inherit
|
||||
@@ -27,8 +27,11 @@ jobs:
|
||||
((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
|
||||
(github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
actions: read
|
||||
contents: read
|
||||
outputs:
|
||||
ready: ${{ steps.owner.outputs.ready }}
|
||||
ready: ${{ steps.verified.outputs.ready }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
@@ -45,6 +48,32 @@ jobs:
|
||||
TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
|
||||
[[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Require successful full CI for this commit
|
||||
id: verified
|
||||
if: steps.owner.outputs.ready == 'true'
|
||||
timeout-minutes: 40
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
CI_RUN_ID=''
|
||||
for _ in {1..12}; do
|
||||
CI_RUN_ID="$(gh run list \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--workflow ci.yml \
|
||||
--commit "$RELEASE_SHA" \
|
||||
--event push \
|
||||
--limit 1 \
|
||||
--json databaseId \
|
||||
--jq '.[0].databaseId // empty')"
|
||||
[[ -n "$CI_RUN_ID" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ -n "$CI_RUN_ID" ]]
|
||||
gh run watch "$CI_RUN_ID" --repo "$GITHUB_REPOSITORY" --exit-status
|
||||
echo 'ready=true' >> "$GITHUB_OUTPUT"
|
||||
|
||||
python-release:
|
||||
name: Publish tested Python artifacts
|
||||
needs: candidate
|
||||
|
||||
@@ -43,7 +43,7 @@ jobs:
|
||||
- name: Build UI screenshot gallery
|
||||
run: bin/collect_ui_screenshots.sh
|
||||
env:
|
||||
BASE_URL: http://archivebox.localhost:8000
|
||||
BASE_URL: http://archivebox.localhost:5797
|
||||
|
||||
- name: Verify gallery was generated from this exact revision
|
||||
run: |
|
||||
|
||||
@@ -2,6 +2,12 @@ name: Parallel Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
test_file:
|
||||
description: Run one discovered test file; empty runs the complete matrix
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
|
||||
env:
|
||||
PYTHONIOENCODING: utf-8
|
||||
@@ -58,10 +64,15 @@ jobs:
|
||||
|
||||
- name: Discover every test exactly once
|
||||
id: discover
|
||||
env:
|
||||
TEST_FILE: ${{ inputs.test_file }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
|
||||
test "${#output[@]}" -eq 2
|
||||
if [[ -n "$TEST_FILE" ]]; then
|
||||
output[1]="$("$JQ_BINARY" -ce '[.[] | select(.paths == [env.TEST_FILE])] | select(length == 1)' <<< "${output[1]}")"
|
||||
fi
|
||||
echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
|
||||
echo "${output[0]}"
|
||||
"$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
|
||||
|
||||
+22
-7
@@ -1,8 +1,10 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
|
||||
# Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
|
||||
# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
|
||||
# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
|
||||
# WHY: abx-dl owns Python, Node, Chromium, and extraction dependencies so the
|
||||
# standalone downloader stays independent of ArchiveBox's server features.
|
||||
# ArchiveBox-only dependencies (Sonic, OpenCode, supervisor, Django) belong in
|
||||
# the layers added by this Dockerfile, never in the shared abx-dl base image.
|
||||
# Build abx-dl first, then point this file at it:
|
||||
# docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
|
||||
# --build-context abxbus=../abxbus \
|
||||
@@ -10,10 +12,10 @@
|
||||
# --build-context abx-plugins=../abx-plugins \
|
||||
# -t archivebox/abx-dl:dev
|
||||
# docker buildx build . -f Dockerfile \
|
||||
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.289 \
|
||||
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.333 \
|
||||
# -t archivebox:multistage
|
||||
|
||||
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.289
|
||||
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.333
|
||||
|
||||
FROM archivebox/sonic:1.4.9 AS sonic
|
||||
FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
|
||||
@@ -250,6 +252,19 @@ RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777
|
||||
&& echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
|
||||
&& echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
|
||||
|
||||
# The optional AI service belongs to ArchiveBox, not the downloader base image.
|
||||
# Always bundle its executable while leaving OPENCODE_ENABLED opt-in.
|
||||
RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=archivebox-opencode-$TARGETARCH \
|
||||
chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" /var/tmp/abxpkg-cache \
|
||||
&& chmod 1777 /var/tmp/abxpkg-cache \
|
||||
&& env ABXPKG_TMP_CACHE_DIR=/var/tmp/abxpkg-cache XDG_CACHE_HOME=/var/tmp/abxpkg-cache \
|
||||
setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install opencode \
|
||||
# pnpm includes musl variants that Debian's glibc runtime cannot use.
|
||||
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
|
||||
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} +
|
||||
RUN --network=none setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
|
||||
bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install opencode'
|
||||
|
||||
WORKDIR "$DATA_DIR"
|
||||
RUN echo "[+] Initializing image collection..." \
|
||||
&& find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
|
||||
@@ -282,10 +297,10 @@ RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully
|
||||
|
||||
WORKDIR "$DATA_DIR"
|
||||
VOLUME "$DATA_DIR"
|
||||
EXPOSE 8000
|
||||
EXPOSE 5797
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
|
||||
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
|
||||
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:5797/health/' | grep -q 'OK'
|
||||
|
||||
ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
|
||||
CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]
|
||||
CMD ["archivebox", "server", "--init", "0.0.0.0:5797"]
|
||||
@@ -73,15 +73,15 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
|
||||
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
|
||||
docker compose pull
|
||||
docker compose up -d --wait # initializes new collections automatically
|
||||
# open http://admin.archivebox.localhost:8000 to finish setup
|
||||
# open http://admin.archivebox.localhost:5797 to finish setup
|
||||
# docker compose run --rm archivebox add 'https://example.com'
|
||||
# docker compose run --rm archivebox help
|
||||
<br/>
|
||||
<br/>
|
||||
# Option B: Or use it as a plain Docker container:
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
|
||||
# open http://admin.archivebox.localhost:8000 to finish setup
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
|
||||
# open http://admin.archivebox.localhost:5797 to finish setup
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
<br/>
|
||||
@@ -93,14 +93,14 @@ archivebox init
|
||||
archivebox install
|
||||
# archivebox add 'https://example.com'
|
||||
# archivebox help
|
||||
# archivebox server 0.0.0.0:8000
|
||||
# archivebox server 0.0.0.0:5797
|
||||
<br/>
|
||||
<br/>
|
||||
# Option D: Or use the uv install shortcut for Option C
|
||||
curl -fsSL 'https://get.archivebox.io' | bash
|
||||
</code></pre>
|
||||
<br/>
|
||||
<sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
|
||||
<sub>Open <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI and <a href="http://admin.archivebox.localhost:5797"><code>http://admin.archivebox.localhost:5797</code></a> for the admin UI ➡️</sub><br/>
|
||||
<sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
|
||||
</details>
|
||||
<br/>
|
||||
@@ -129,7 +129,7 @@ curl -fsSL 'https://get.archivebox.io' | bash
|
||||
- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
|
||||
- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
|
||||
- [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
|
||||
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
|
||||
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
|
||||
- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
|
||||
- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
|
||||
- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
|
||||
@@ -152,12 +152,12 @@ curl -fsSL 'https://get.archivebox.io' | bash
|
||||
#### ✳️ Easy Setup
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
|
||||
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
|
||||
<br/>
|
||||
<ol>
|
||||
<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
|
||||
<li>Set up an admin account and generate an API Key.</li>
|
||||
<li>Use the Web UI or <a href="https://archivebox.github.io/ios-archivebox/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://archivebox.github.io/ios-archivebox/">mobile app</a>, the <a href="https://archivebox.github.io/archivebox-browser-extension/">browser extension</a>, and other integrations to your server using your API Key.</li>
|
||||
<li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
|
||||
</ol>
|
||||
<br/>
|
||||
</details>
|
||||
@@ -177,7 +177,7 @@ docker compose pull
|
||||
<li>Start the server, which initializes a new collection automatically.
|
||||
<pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
|
||||
</code></pre></li>
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# run CLI commands inside the server container started above
|
||||
docker compose exec archivebox archivebox add 'https://example.com'
|
||||
@@ -198,10 +198,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
|
||||
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# docker exec archivebox archivebox [subcommand] [--help]
|
||||
@@ -256,8 +256,8 @@ archivebox init # initialize a new collection
|
||||
archivebox install # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -292,8 +292,8 @@ archivebox add 'https://example.com'
|
||||
</code></pre>
|
||||
<br/>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -325,8 +325,8 @@ archivebox init
|
||||
archivebox install
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -365,17 +365,12 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
|
||||
<br/>
|
||||
<ol>
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li>Download a binary release for your OS or build the native app from source<br/>
|
||||
<ul>
|
||||
<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
|
||||
<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
|
||||
<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
|
||||
<li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
|
||||
</ol>
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
|
||||
<br/>
|
||||
<i>✨ Alpha (contributors wanted!)</i>: for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
|
||||
<a href="https://electron.archivebox.io/">Desktop app guide</a> · <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
|
||||
<br/>
|
||||
</details>
|
||||
|
||||
@@ -544,16 +539,16 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html' # inspect snapshot da
|
||||
<summary><b>🖥 Web UI & API Usage</b></summary>
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# Start the server on bare metal (uv/apt/brew):
|
||||
archivebox server 0.0.0.0:8000 # start the server
|
||||
archivebox server 0.0.0.0:5797 # start the server
|
||||
<br/>
|
||||
# Or with Docker Compose:
|
||||
docker compose up # start the server
|
||||
<br/>
|
||||
# Or with a Docker container:
|
||||
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
|
||||
<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
|
||||
<sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI. ➡️</sup><br/>
|
||||
<sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
|
||||
<sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
|
||||
<br/><br/>
|
||||
@@ -570,18 +565,20 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
|
||||
|
||||
# restart the server to apply any config changes
|
||||
</code></pre>
|
||||
|
||||
Snapshot caching follows each snapshot's saved `PERMISSIONS`, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses `Cache-Control: private` so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.
|
||||
</details>
|
||||
|
||||
<br/>
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧 Desktop / Mobile / Extension Usage</b></summary>
|
||||
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧 Desktop / Mobile / Extension Usage</b></summary>
|
||||
<br/>
|
||||
|
||||
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://archivebox.github.io/ios-archivebox/)
|
||||
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://archivebox.github.io/ios-archivebox/)
|
||||
- 🧩 [ArchiveBox Browser Extension](https://archivebox.github.io/archivebox-browser-extension/)
|
||||
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://github.com/ArchiveBox/electron-archivebox)
|
||||
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
|
||||
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
|
||||
- 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
|
||||
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
|
||||
|
||||
<br/>
|
||||
</details>
|
||||
@@ -641,7 +638,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>
|
||||
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
|
||||
|
||||
- <img src="https://archivebox.github.io/ios-archivebox/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
|
||||
- <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://app.archivebox.io/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
|
||||
<i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
|
||||
@@ -1346,10 +1343,10 @@ archivebox init
|
||||
archivebox install # detect and install all extractor dependencies
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
archivebox server --debug --reload 0.0.0.0:8000
|
||||
archivebox server --debug --reload 0.0.0.0:5797
|
||||
|
||||
# Run the production server (with bg workers but no autoreloading)
|
||||
archivebox server 0.0.0.0:8000
|
||||
archivebox server 0.0.0.0:5797
|
||||
```
|
||||
|
||||
#### 2. Option B: Build the docker container and use that for development instead
|
||||
@@ -1361,10 +1358,10 @@ archivebox server 0.0.0.0:8000
|
||||
./bin/build_docker.sh dev
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
|
||||
|
||||
# Run the production server (with bg workers but no autoreloading)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
|
||||
|
||||
# (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
|
||||
# When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
|
||||
@@ -1386,11 +1383,11 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
|
||||
archivebox config --set DEBUG=True
|
||||
|
||||
# OR you can run a dev server with DEBUG=True in a few ways:
|
||||
archivebox server --debug --reload 0.0.0.0:8000
|
||||
archivebox server --debug --reload 0.0.0.0:5797
|
||||
# or
|
||||
archivebox server --debug 0.0.0.0:8000
|
||||
archivebox server --debug 0.0.0.0:5797
|
||||
# or
|
||||
env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application
|
||||
env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
|
||||
```
|
||||
|
||||
https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
|
||||
@@ -1425,7 +1422,7 @@ services:
|
||||
|
||||
# or with plain Docker:
|
||||
docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
|
||||
docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
|
||||
docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
|
||||
|
||||
# or with uv:
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
@@ -247,7 +247,7 @@ def get_archiveresult(request: HttpRequest, archiveresult_id: str):
|
||||
|
||||
def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
|
||||
normalized = str(plugin or "").strip().strip("/")
|
||||
if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
|
||||
if normalized in (".", "..") or not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
|
||||
raise HttpError(400, "Invalid ArchiveResult plugin name")
|
||||
return normalized
|
||||
|
||||
@@ -386,6 +386,7 @@ def _write_archiveresult_files(
|
||||
existing_output_files: dict[str, dict[str, Any]] | None = None,
|
||||
allow_empty: bool = False,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
plugin_name = _normalize_uploaded_archiveresult_plugin(plugin_name)
|
||||
files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
|
||||
output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
|
||||
mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
|
||||
|
||||
@@ -68,7 +68,7 @@ def help() -> None:
|
||||
[blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
|
||||
[grey53]# using Docker:[/grey53]
|
||||
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 5797:5797[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
"""
|
||||
if IN_DOCKER
|
||||
else ""
|
||||
@@ -109,7 +109,7 @@ def help() -> None:
|
||||
[dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
|
||||
[dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
|
||||
[dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
|
||||
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue] [grey53]# Start the Web UI / API server[/grey53]
|
||||
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:5797[/blue] [grey53]# Start the Web UI / API server[/grey53]
|
||||
"""
|
||||
print(
|
||||
Panel(
|
||||
|
||||
@@ -134,7 +134,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
|
||||
print(" > Skipping orphan snapshot import during init.")
|
||||
print()
|
||||
print(" [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
|
||||
print(" archivebox update")
|
||||
print(" archivebox update --rescan --migrate-only")
|
||||
|
||||
print("\n[green]----------------------------------------------------------------------[/green]")
|
||||
|
||||
|
||||
@@ -109,6 +109,7 @@ def create_personas(
|
||||
profile: str | None = None,
|
||||
source: str | None = None,
|
||||
browser_binary: str | None = None,
|
||||
permissions: str | None = None,
|
||||
) -> int:
|
||||
"""
|
||||
Create Personas from names.
|
||||
@@ -162,7 +163,10 @@ def create_personas(
|
||||
rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
|
||||
continue
|
||||
|
||||
persona, created = Persona.objects.get_or_create(name=name)
|
||||
persona, created = Persona.objects.get_or_create(
|
||||
name=name,
|
||||
defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
|
||||
)
|
||||
|
||||
if created:
|
||||
persona.ensure_dirs()
|
||||
@@ -441,7 +445,11 @@ def main():
|
||||
|
||||
@main.command("create")
|
||||
@click.argument("names", nargs=-1)
|
||||
@click.option("--import", "import_from", help="Import from chrome, chromium, brave, edge, a persona directory, or a live CDP URL")
|
||||
@click.option(
|
||||
"--import",
|
||||
"import_from",
|
||||
help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
|
||||
)
|
||||
@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
|
||||
@click.option(
|
||||
"--browser-binary",
|
||||
@@ -449,9 +457,30 @@ def main():
|
||||
help="Source Chromium browser executable; required for other Chromium-based browsers",
|
||||
)
|
||||
@click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
|
||||
def create_cmd(names: tuple, import_from: str | None, profile: str | None, source: str | None, browser_binary: str | None):
|
||||
@click.option(
|
||||
"--permissions",
|
||||
type=click.Choice(["public", "unlisted", "private"]),
|
||||
help="Visibility for a newly created persona (existing personas are unchanged)",
|
||||
)
|
||||
def create_cmd(
|
||||
names: tuple,
|
||||
import_from: str | None,
|
||||
profile: str | None,
|
||||
source: str | None,
|
||||
browser_binary: str | None,
|
||||
permissions: str | None,
|
||||
):
|
||||
"""Create Personas, optionally importing from a browser profile."""
|
||||
sys.exit(create_personas(names, import_from=import_from, profile=profile, source=source, browser_binary=browser_binary))
|
||||
sys.exit(
|
||||
create_personas(
|
||||
names,
|
||||
import_from=import_from,
|
||||
profile=profile,
|
||||
source=source,
|
||||
browser_binary=browser_binary,
|
||||
permissions=permissions,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@main.command("list")
|
||||
|
||||
@@ -46,13 +46,13 @@ def _split_bind_spec(spec: str) -> tuple[str, str]:
|
||||
"""Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
|
||||
|
||||
The empty strings stand in for "not provided"; the caller fills in
|
||||
defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
|
||||
defaults. Bracketed IPv6 literals like ``[::1]:5797`` are handled.
|
||||
"""
|
||||
spec = (spec or "").strip()
|
||||
if not spec:
|
||||
return "", ""
|
||||
if spec.startswith("["):
|
||||
# Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
|
||||
# Bracketed IPv6: ``[::1]`` or ``[::1]:5797``
|
||||
end = spec.find("]")
|
||||
if end == -1:
|
||||
return spec, "" # malformed; let validator reject it
|
||||
@@ -77,11 +77,11 @@ def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
|
||||
(normalized to ``127.0.0.1``). Bare hostnames are rejected because the
|
||||
bind address feeds Daphne, which has to listen on a numeric address;
|
||||
public hostnames belong in ``BASE_URL`` instead. Empty values fall back
|
||||
to ``127.0.0.1`` / ``8000``.
|
||||
to ``127.0.0.1`` / ``5797``.
|
||||
"""
|
||||
raw_host, raw_port = _split_bind_spec(spec)
|
||||
host = raw_host.strip()
|
||||
port = (raw_port or "").strip() or "8000"
|
||||
port = (raw_port or "").strip() or "5797"
|
||||
|
||||
if host == "" or host.lower() == "localhost":
|
||||
host = "127.0.0.1"
|
||||
@@ -198,7 +198,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
|
||||
"[yellow] routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
|
||||
)
|
||||
print(
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
|
||||
)
|
||||
if config.USES_SUBDOMAIN_ROUTING:
|
||||
print(
|
||||
@@ -221,7 +221,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
|
||||
"[yellow] a reverse proxy / ingress / public hostname, e.g.[/yellow]",
|
||||
)
|
||||
print(
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
|
||||
)
|
||||
print()
|
||||
|
||||
@@ -254,7 +254,7 @@ def server(
|
||||
print()
|
||||
|
||||
# First non-empty positional arg is the bind spec; otherwise inherit from
|
||||
# config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
|
||||
# config (which defaults to "127.0.0.1:5797"). _parse_and_validate_bind_spec
|
||||
# hard-errors on hostnames so the rest of the server can assume a numeric
|
||||
# bind host.
|
||||
bind_spec = next((arg for arg in runserver_args if arg), "")
|
||||
|
||||
@@ -44,6 +44,7 @@ SNAPSHOT_LIST_CHUNK_SIZE = 5000
|
||||
|
||||
def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
from archivebox.config import VERSION
|
||||
from archivebox.config.common import redact_sensitive_config
|
||||
from archivebox.core.models import SnapshotTag
|
||||
|
||||
fields = (
|
||||
@@ -53,6 +54,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
"title",
|
||||
"bookmarked_at",
|
||||
"created_at",
|
||||
"modified_at",
|
||||
"config",
|
||||
"notes",
|
||||
"timestamp",
|
||||
"depth",
|
||||
"status",
|
||||
@@ -80,6 +84,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
"tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
|
||||
"bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
|
||||
"created_at": row["created_at"].isoformat() if row["created_at"] else None,
|
||||
"modified_at": row["modified_at"].isoformat() if row["modified_at"] else None,
|
||||
"config": redact_sensitive_config(row["config"]),
|
||||
"notes": row["notes"],
|
||||
"timestamp": row["timestamp"],
|
||||
"depth": row["depth"],
|
||||
"status": row["status"],
|
||||
|
||||
@@ -7,6 +7,7 @@ import os
|
||||
import asyncio
|
||||
import shlex
|
||||
import time
|
||||
from itertools import islice
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from collections.abc import Iterable
|
||||
@@ -52,17 +53,20 @@ def _get_search_indexing_plugins() -> list[str]:
|
||||
def _build_filtered_snapshots_queryset(
|
||||
**kwargs,
|
||||
):
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
|
||||
|
||||
limit = kwargs.pop("limit", None)
|
||||
reverse = kwargs.pop("reverse", False)
|
||||
snapshots = build_snapshot_queryset(**kwargs)
|
||||
snapshots = snapshots.order_by(kwargs.get("sort") or "-created_at", "-id")
|
||||
if reverse:
|
||||
snapshots = snapshots.reverse()
|
||||
if kwargs.get("resume"):
|
||||
snapshots = snapshots.filter(timestamp__lte=kwargs["resume"])
|
||||
snapshots = snapshots.filter(**{"timestamp__gte" if reverse else "timestamp__lte": kwargs["resume"]})
|
||||
snapshots = snapshots.select_related("crawl")
|
||||
if limit is not None and limit > 0:
|
||||
snapshot_ids = list(snapshots.values_list("id", flat=True)[:limit])
|
||||
snapshots = Snapshot.objects.filter(id__in=snapshot_ids).select_related("crawl")
|
||||
snapshots = snapshots.filter(id__in=snapshot_ids)
|
||||
|
||||
return snapshots
|
||||
|
||||
@@ -73,6 +77,7 @@ def reindex_snapshots(
|
||||
search_plugins: list[str],
|
||||
batch_size: int,
|
||||
collect_ids: bool = False,
|
||||
reverse: bool = False,
|
||||
wait_for_turn=None,
|
||||
) -> dict[str, Any]:
|
||||
from archivebox.cli.archivebox_extract import run_plugins
|
||||
@@ -94,16 +99,15 @@ def reindex_snapshots(
|
||||
plugin=plugin_name,
|
||||
status__in=completed_statuses,
|
||||
)
|
||||
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False).order_by("id")
|
||||
after_id = None
|
||||
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False)
|
||||
candidates = candidates.order_by("created_at", "id") if reverse else candidates.order_by("-created_at", "-id")
|
||||
remaining = candidates.select_related(None).only("id", "timestamp", "created_at").paged_iterator(chunk_size=batch_size)
|
||||
while True:
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
page = candidates.filter(id__gt=after_id) if after_id is not None else candidates
|
||||
batch = list(page.select_related(None).only("id", "timestamp")[:batch_size])
|
||||
batch = list(islice(remaining, batch_size))
|
||||
if not batch:
|
||||
break
|
||||
after_id = batch[-1].id
|
||||
records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
|
||||
stats["processed"] += len(batch)
|
||||
stats["requested"] += len(batch)
|
||||
@@ -161,18 +165,18 @@ def update(
|
||||
continuous: bool = False,
|
||||
index_only: bool = False,
|
||||
migrate_only: bool = False,
|
||||
rescan: bool = False,
|
||||
reverse: bool = False,
|
||||
stop_daemon_stack: bool = True,
|
||||
) -> None:
|
||||
"""
|
||||
Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
|
||||
Update the database without forcing a filesystem scan by default.
|
||||
|
||||
Three-phase operation (without filters):
|
||||
- Phase 1: Drain legacy archive/ directories into the current layout
|
||||
- Phase 2: Select only stale fs_version rows through the indexed column
|
||||
- Phase 3: Run queued snapshot-level filesystem maintenance until idle
|
||||
|
||||
With filters: Only phase 2 (DB query), no filesystem operations.
|
||||
Without filters: All phases (full update).
|
||||
--migrate-only: eagerly finish pending filesystem migrations for known rows.
|
||||
--index-only: reconcile known snapshot assets/metadata and backfill search.
|
||||
--rescan: discover filesystem orphans first, then repair known snapshots.
|
||||
Explicit scans default to newest first; --reverse selects oldest first.
|
||||
Filters select existing database snapshots for targeted updates.
|
||||
"""
|
||||
|
||||
from rich import print
|
||||
@@ -248,12 +252,28 @@ def update(
|
||||
exit_code = 0
|
||||
|
||||
try:
|
||||
if rescan and (is_filtered_update or resume or continuous):
|
||||
raise click.UsageError("--rescan cannot be combined with filters, --resume, or --continuous; rerun it to resume automatically.")
|
||||
if rescan:
|
||||
migrate_only = True # Discovery must not schedule new captures.
|
||||
wait_for_turn()
|
||||
|
||||
with foreground_shutdown_signals(), foreground_parent_watchdog():
|
||||
while True:
|
||||
do_migrate = migrate_only or not index_only
|
||||
do_index = index_only or not migrate_only
|
||||
if rescan:
|
||||
from archivebox.cli.rescan import rescan_snapshots
|
||||
|
||||
rescan_snapshots(
|
||||
wait_for_turn=wait_for_turn,
|
||||
reverse=reverse,
|
||||
scan_legacy=lambda: drain_old_archive_dirs(batch_size=batch_size, reverse=reverse),
|
||||
)
|
||||
do_migrate = migrate_only or is_filtered_update and not index_only
|
||||
do_index = index_only
|
||||
if not do_migrate and not do_index:
|
||||
print(
|
||||
"[*] Collection ready. Filesystem migrations remain lazy; use --migrate-only, --index-only, or --rescan for explicit scans.",
|
||||
)
|
||||
|
||||
if do_migrate:
|
||||
if (
|
||||
@@ -285,7 +305,8 @@ def update(
|
||||
after=after,
|
||||
resume=resume,
|
||||
batch_size=batch_size,
|
||||
queue_for_archiving=True,
|
||||
queue_for_archiving=not migrate_only,
|
||||
reverse=reverse,
|
||||
wait_for_turn=wait_for_turn,
|
||||
)
|
||||
print_stats(stats)
|
||||
@@ -293,17 +314,13 @@ def update(
|
||||
else:
|
||||
stats_combined = {"phase1": {}, "phase2": {}}
|
||||
|
||||
print("[*] Phase 1: Draining old archive/ directories (0.8.x → 0.9.x migration)...")
|
||||
stats_combined["phase1"] = drain_old_archive_dirs(
|
||||
resume_from=resume,
|
||||
batch_size=batch_size,
|
||||
)
|
||||
|
||||
print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
|
||||
stats_combined["phase2"] = process_all_db_snapshots(
|
||||
batch_size=batch_size,
|
||||
resume=resume,
|
||||
wait_for_turn=wait_for_turn,
|
||||
reverse=reverse,
|
||||
migrate=True,
|
||||
)
|
||||
print_combined_stats(stats_combined)
|
||||
# The due selectors are indexed and cheap when empty, so
|
||||
@@ -311,7 +328,7 @@ def update(
|
||||
# whole-table counts merely to decide whether to call it.
|
||||
print("[*] Phase 3: Running filesystem maintenance until idle...")
|
||||
if is_filtered_update:
|
||||
for snapshot_id in sorted(touched_snapshot_ids):
|
||||
for snapshot_id in sorted(touched_snapshot_ids, reverse=not reverse):
|
||||
run_scoped_runner("--snapshot-id", snapshot_id)
|
||||
elif migrate_only:
|
||||
run_scoped_runner("--maintenance-only")
|
||||
@@ -319,6 +336,25 @@ def update(
|
||||
run_scoped_runner()
|
||||
|
||||
if do_index:
|
||||
from archivebox.cli.rescan import reconcile_known_snapshots
|
||||
|
||||
snapshots = _build_filtered_snapshots_queryset(
|
||||
filter_patterns=filter_patterns,
|
||||
filter_type=filter_type,
|
||||
status=status,
|
||||
url__icontains=url__icontains,
|
||||
url__istartswith=url__istartswith,
|
||||
tag=tag,
|
||||
crawl_id=crawl_id,
|
||||
limit=limit,
|
||||
sort=sort,
|
||||
search=search,
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
reconcile_known_snapshots(snapshots, wait_for_turn=wait_for_turn)
|
||||
search_plugins = _get_search_indexing_plugins()
|
||||
if not search_plugins:
|
||||
print("[*] No search indexing plugins are available, nothing to backfill.")
|
||||
@@ -337,12 +373,14 @@ def update(
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
stats = reindex_snapshots(
|
||||
snapshots,
|
||||
search_plugins=search_plugins,
|
||||
batch_size=batch_size,
|
||||
collect_ids=is_filtered_update,
|
||||
reverse=reverse,
|
||||
wait_for_turn=wait_for_turn,
|
||||
)
|
||||
print_index_stats(stats)
|
||||
@@ -358,6 +396,10 @@ def update(
|
||||
exit_code = 130
|
||||
exact_resume = err.__dict__.get("archivebox_resume")
|
||||
resume_cmd = ["archivebox", "update"]
|
||||
if rescan:
|
||||
resume_cmd.append("--rescan")
|
||||
if reverse:
|
||||
resume_cmd.append("--reverse")
|
||||
if migrate_only:
|
||||
resume_cmd.append("--migrate-only")
|
||||
if index_only:
|
||||
@@ -405,7 +447,7 @@ def update(
|
||||
stop_own_supervisord_process()
|
||||
|
||||
|
||||
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500) -> dict[str, int]:
|
||||
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500, reverse: bool = False) -> dict[str, int]:
|
||||
"""
|
||||
Drain old archive/ directories (0.8.x → 0.9.x migration).
|
||||
|
||||
@@ -471,18 +513,18 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
|
||||
|
||||
# Scan real legacy directories only; these still contain data to migrate.
|
||||
entries = [
|
||||
(e.stat().st_mtime, e.path)
|
||||
(float(e.name), e.path)
|
||||
for e in all_entries
|
||||
if e.is_dir(follow_symlinks=False) and Snapshot.is_legacy_archive_dir(Path(e.path)) # Skip symlinks and 0.9.x roots
|
||||
]
|
||||
entries.sort(reverse=True) # Newest first
|
||||
entries.sort(reverse=not reverse) # Newest first
|
||||
print(f"[*] Found {len(entries)} old directories to drain")
|
||||
|
||||
for mtime, entry_path in entries:
|
||||
entry_path = Path(entry_path)
|
||||
|
||||
# Resume from timestamp if specified
|
||||
if resume_from and entry_path.name > resume_from:
|
||||
if resume_from and (entry_path.name < resume_from if reverse else entry_path.name > resume_from):
|
||||
continue
|
||||
|
||||
stats["processed"] += 1
|
||||
@@ -584,7 +626,13 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
|
||||
return stats
|
||||
|
||||
|
||||
def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
|
||||
def process_all_db_snapshots(
|
||||
batch_size: int = 500,
|
||||
resume: str | None = None,
|
||||
wait_for_turn=None,
|
||||
reverse: bool = False,
|
||||
migrate: bool = False,
|
||||
) -> dict[str, int]:
|
||||
"""Queue only snapshots whose indexed filesystem version is stale."""
|
||||
from archivebox.core.models import Snapshot
|
||||
from django.db.models import Q
|
||||
@@ -601,21 +649,27 @@ def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, w
|
||||
}
|
||||
queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
|
||||
if resume:
|
||||
queryset = queryset.filter(timestamp__lte=resume)
|
||||
queryset = queryset.filter(**{"timestamp__gte" if reverse else "timestamp__lte": resume})
|
||||
initial_now = timezone.now()
|
||||
rows_to_wake = queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
|
||||
after_id = None
|
||||
rows_to_wake = queryset if migrate else queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
|
||||
ordered = rows_to_wake.order_by("created_at", "id") if reverse else rows_to_wake.order_by("-created_at", "-id")
|
||||
remaining = ordered.select_related("crawl__created_by").paged_iterator(chunk_size=batch_size)
|
||||
while True:
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
page = rows_to_wake.filter(id__gt=after_id) if after_id is not None else rows_to_wake
|
||||
batch = list(page.only("id", "fs_version", "modified_at").order_by("id")[:batch_size])
|
||||
batch = list(islice(remaining, batch_size))
|
||||
if not batch:
|
||||
break
|
||||
after_id = batch[-1].id
|
||||
now = timezone.now()
|
||||
updated = 0
|
||||
for snapshot in batch:
|
||||
if migrate:
|
||||
from archivebox.services.runner import run_snapshot_maintenance
|
||||
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
run_snapshot_maintenance(str(snapshot.id))
|
||||
continue
|
||||
updated += int(
|
||||
snapshot.safe_update(
|
||||
{"retry_at": now, "modified_at": now},
|
||||
@@ -647,6 +701,7 @@ def process_filtered_snapshots(
|
||||
resume: str | None,
|
||||
batch_size: int,
|
||||
queue_for_archiving: bool = True,
|
||||
reverse: bool = False,
|
||||
wait_for_turn=None,
|
||||
) -> dict[str, Any]:
|
||||
"""Process snapshots matching filters (DB query only)."""
|
||||
@@ -669,6 +724,7 @@ def process_filtered_snapshots(
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
|
||||
total = snapshots.count()
|
||||
@@ -687,6 +743,10 @@ def process_filtered_snapshots(
|
||||
stats["snapshot_ids"].append(str(snapshot.id))
|
||||
update_values = {}
|
||||
updated = 0
|
||||
if not queue_for_archiving and snapshot.fs_migration_needed:
|
||||
from archivebox.services.runner import run_snapshot_maintenance
|
||||
|
||||
run_snapshot_maintenance(str(snapshot.id))
|
||||
if queue_for_archiving:
|
||||
update_values.update(
|
||||
{
|
||||
@@ -784,6 +844,12 @@ def print_index_stats(stats: dict[str, Any]) -> None:
|
||||
@click.option("--continuous", is_flag=True, help="Run continuously as background worker")
|
||||
@click.option("--index-only", is_flag=True, help="Backfill available search indexes from existing archived content")
|
||||
@click.option("--migrate-only", is_flag=True, help="Only migrate filesystem and update database/index state")
|
||||
@click.option("--reverse", is_flag=True, help="Process oldest first instead of newest first for explicit scans")
|
||||
@click.option(
|
||||
"--rescan",
|
||||
is_flag=True,
|
||||
help="Scan all snapshot directories for orphans and metadata repairs (slow, opt-in, safe to restart)",
|
||||
)
|
||||
@snapshot_filter_options(default_filter_type="exact")
|
||||
@docstring(update.__doc__)
|
||||
def main(**kwargs):
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
"""Explicit discovery of portable snapshots; no checkpoint or default tree walk."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from itertools import groupby
|
||||
from pathlib import Path
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core.exceptions import ValidationError
|
||||
from django.db import IntegrityError
|
||||
from rich import print
|
||||
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.misc.util import parse_date
|
||||
|
||||
|
||||
def _directories(parent: Path):
|
||||
# Names encode dates / UUIDv7 creation time. Newest first within each
|
||||
# subtree, with bounded memory and no traversal into plugin payloads.
|
||||
with os.scandir(parent) as entries:
|
||||
paths = [Path(entry.path) for entry in entries if entry.is_dir(follow_symlinks=False)]
|
||||
yield from sorted(paths, key=lambda path: path.name, reverse=True)
|
||||
|
||||
|
||||
def _snapshot_directories(reverse=False):
|
||||
dates = []
|
||||
for user in _directories(CONSTANTS.USERS_DIR):
|
||||
root = user / CONSTANTS.SNAPSHOTS_DIR_NAME
|
||||
if root.is_symlink() or not root.is_dir():
|
||||
continue
|
||||
dates.extend(_directories(root))
|
||||
dates.sort(key=lambda path: path.name, reverse=not reverse)
|
||||
for _, date_group in groupby(dates, key=lambda path: path.name):
|
||||
directories = [directory for date in date_group for domain in _directories(date) for directory in _directories(domain)]
|
||||
yield from sorted(directories, key=lambda path: path.name, reverse=not reverse)
|
||||
|
||||
|
||||
def _fields(record, names):
|
||||
return {name: record[name] for name in names.split() if name in record}
|
||||
|
||||
|
||||
def _dates(record):
|
||||
return {name: parse_date(record[name]) for name in ("created_at", "modified_at") if record.get(name)}
|
||||
|
||||
|
||||
def _restore_dates(obj, record):
|
||||
dates = _dates(record)
|
||||
if dates:
|
||||
type(obj).objects.filter(pk=obj.pk).update(**dates)
|
||||
for name, value in dates.items():
|
||||
setattr(obj, name, value)
|
||||
|
||||
|
||||
def _reconcile_directory(directory: Path) -> str:
|
||||
snapshot_id = uuid.UUID(directory.name)
|
||||
snapshot = Snapshot.objects.select_related("crawl__created_by").filter(pk=snapshot_id).first()
|
||||
index = directory / CONSTANTS.JSONL_INDEX_FILENAME
|
||||
if index.is_symlink():
|
||||
raise ValueError("metadata must not be a symlink")
|
||||
if not index.exists():
|
||||
if snapshot is None or Path(snapshot.output_dir) != directory or snapshot.status != "sealed":
|
||||
raise ValueError("missing index.jsonl; cannot reconstruct an unknown or active snapshot")
|
||||
snapshot.write_index_jsonl(output_dir=directory)
|
||||
snapshot.reconcile_filesystem_links()
|
||||
return "Repaired"
|
||||
|
||||
records = [json.loads(line) for line in index.read_text().splitlines() if line.strip()]
|
||||
if any(not isinstance(record, dict) for record in records):
|
||||
raise ValueError("metadata records must be JSON objects")
|
||||
snapshots = [record for record in records if record.get("type") == "Snapshot"]
|
||||
if len(snapshots) != 1:
|
||||
raise ValueError("expected exactly one Snapshot record")
|
||||
record = snapshots[0]
|
||||
if uuid.UUID(record["id"]) != snapshot_id:
|
||||
raise ValueError("snapshot ID conflicts with directory name")
|
||||
if record.get("status") != "sealed":
|
||||
raise ValueError("snapshot is not sealed; finish the source capture first")
|
||||
if record.get("fs_version") not in {Snapshot._fs_current_version(), *Snapshot._FS_VERSION_MIGRATION_PATHS}:
|
||||
raise ValueError("unsupported filesystem version")
|
||||
crawl_id = uuid.UUID(record["crawl_id"])
|
||||
owner = directory.parents[3].name
|
||||
crawl_records = [item for item in records if item.get("type") == "Crawl" and uuid.UUID(item["id"]) == crawl_id]
|
||||
crawl_record = crawl_records[0] if crawl_records else {}
|
||||
if crawl_record.get("created_by", owner) != owner:
|
||||
raise ValueError("crawl owner conflicts with directory path")
|
||||
crawl = Crawl.objects.select_related("created_by").filter(pk=crawl_id).first()
|
||||
imported_crawl = crawl is None
|
||||
if crawl and crawl.created_by.username != owner:
|
||||
raise ValueError("existing crawl ID belongs to a different owner")
|
||||
if snapshot and (snapshot.url != record["url"] or snapshot.crawl_id != crawl_id or snapshot.timestamp != record["timestamp"]):
|
||||
raise ValueError("existing snapshot ID has conflicting identity; refusing to overwrite")
|
||||
if snapshot and Path(snapshot.output_dir) != directory:
|
||||
raise ValueError("existing snapshot has a different storage path")
|
||||
|
||||
# Validate identities before inserting anything. Never turn an ID collision
|
||||
# into a new ID, and never overwrite a newer destination row.
|
||||
# write_index_jsonl retains historical hook records before appending the
|
||||
# canonical DB rows. The final record for each hook is authoritative.
|
||||
results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
|
||||
pending = []
|
||||
for result in results:
|
||||
result_id = uuid.UUID(result["id"])
|
||||
if uuid.UUID(result["snapshot_id"]) != snapshot_id:
|
||||
raise ValueError("ArchiveResult belongs to another snapshot")
|
||||
existing = ArchiveResult.objects.filter(pk=result_id).first()
|
||||
identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
|
||||
if existing:
|
||||
if (existing.snapshot_id, existing.plugin, existing.hook_name) != identity:
|
||||
raise ValueError("existing ArchiveResult ID has conflicting identity")
|
||||
elif ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin=identity[1], hook_name=identity[2]).exists():
|
||||
raise ValueError("existing hook has a different ArchiveResult ID")
|
||||
else:
|
||||
pending.append(result)
|
||||
|
||||
imported = snapshot is None
|
||||
if imported:
|
||||
if Snapshot.objects.filter(timestamp=record["timestamp"]).exists():
|
||||
raise ValueError("timestamp belongs to another snapshot")
|
||||
if Snapshot.objects.filter(crawl_id=crawl_id, url=record["url"]).exists():
|
||||
raise ValueError("crawl URL belongs to another snapshot")
|
||||
user, _ = get_user_model().objects.get_or_create(
|
||||
username=owner,
|
||||
defaults={"is_active": False, "password": "!"},
|
||||
)
|
||||
if crawl is None:
|
||||
# Old exports contain only crawl_id. Keep that identity but report
|
||||
# that absent crawl metadata cannot be reconstructed exactly.
|
||||
if not crawl_record:
|
||||
print(f"[yellow] {directory}: no Crawl record; preserving ID with recovered metadata[/yellow]")
|
||||
crawl = Crawl(
|
||||
id=crawl_id,
|
||||
created_by=user,
|
||||
status="sealed",
|
||||
retry_at=None,
|
||||
**_fields(crawl_record, "urls max_depth config tags_str label"),
|
||||
)
|
||||
crawl.save(force_insert=True)
|
||||
_restore_dates(crawl, crawl_record)
|
||||
snapshot = Snapshot.create_from_directory(directory, crawl=crawl, preserve_identity=True)
|
||||
if snapshot is None:
|
||||
raise ValueError("invalid snapshot metadata")
|
||||
snapshot.status = "sealed"
|
||||
snapshot.retry_at = None
|
||||
if "config" not in record:
|
||||
snapshot.config = {"PERMISSIONS": "private"}
|
||||
if Path(snapshot.get_storage_path_for_version(Snapshot._fs_current_version())) != directory:
|
||||
raise ValueError("metadata does not match its owner/date/domain directory")
|
||||
snapshot.save(force_insert=True)
|
||||
_restore_dates(snapshot, record)
|
||||
|
||||
# Each insert is durable. An interruption after the Snapshot insert is not
|
||||
# completion: the next pass still imports missing result IDs and tags.
|
||||
for result in pending:
|
||||
snapshot._create_archive_result_if_missing(result, {}, preserve_identity=True)
|
||||
tags_before = set(snapshot.tags.values_list("name", flat=True))
|
||||
snapshot._merge_tags_from_index(record)
|
||||
tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
|
||||
link = Path(snapshot.crawl.output_dir) / "snapshots" / snapshot.extract_domain_from_url(snapshot.url) / str(snapshot.id)
|
||||
links_ok = link.is_symlink() and link.resolve() == directory.resolve()
|
||||
if not links_ok:
|
||||
snapshot.reconcile_filesystem_links()
|
||||
if snapshot.fs_migration_needed:
|
||||
snapshot.migrate_filesystem_to_current_version(source_dir=directory)
|
||||
if imported:
|
||||
_restore_dates(snapshot, record)
|
||||
if imported_crawl:
|
||||
_restore_dates(crawl, crawl_record)
|
||||
# Leave source metadata untouched, including process/binary records that
|
||||
# are historical evidence rather than runnable destination process state.
|
||||
return "Imported" if imported else "Repaired" if pending or tags_changed or not links_ok else "Unchanged"
|
||||
|
||||
|
||||
def reconcile_known_snapshots(snapshots, wait_for_turn=None):
|
||||
"""Explicit metadata / asset repair; normal updates never call this scan."""
|
||||
for snapshot in snapshots.filter(status="sealed").paged_iterator():
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
if snapshot.fs_migration_needed:
|
||||
snapshot.migrate_filesystem_to_current_version()
|
||||
directory = Path(snapshot.output_dir)
|
||||
if not directory.is_dir():
|
||||
raise ValueError(f"Missing snapshot directory: {directory}")
|
||||
_reconcile_directory(directory)
|
||||
for result in snapshot.archiveresult_set.all():
|
||||
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
|
||||
|
||||
|
||||
def rescan_snapshots(wait_for_turn=None, reverse=False, scan_legacy=None):
|
||||
stats = dict.fromkeys(("Imported", "Repaired", "Unchanged", "Errors"), 0)
|
||||
print("[*] Rescanning snapshot directories (O(N), orphans first; completed records are reused)...")
|
||||
known = {str(pk) for pk in Snapshot.objects.values_list("pk", flat=True)}
|
||||
# Two streaming passes avoid enumerating the entire archive before the
|
||||
# first import. Rows imported before interruption move to the second pass
|
||||
# next time; only one day's directory names are held in memory at a time.
|
||||
for existing in (False, True):
|
||||
if existing and scan_legacy:
|
||||
scan_legacy()
|
||||
for directory in _snapshot_directories(reverse=reverse):
|
||||
if (directory.name.replace("-", "") in known) != existing:
|
||||
continue
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
try:
|
||||
outcome = _reconcile_directory(directory)
|
||||
if directory.name.replace("-", "") in known:
|
||||
for result in ArchiveResult.objects.filter(snapshot_id=uuid.UUID(directory.name)):
|
||||
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
|
||||
stats[outcome] += 1
|
||||
if outcome != "Unchanged":
|
||||
print(f" {outcome}: {directory}")
|
||||
except (OSError, ValueError, KeyError, TypeError, ValidationError, IntegrityError) as err:
|
||||
stats["Errors"] += 1
|
||||
print(f"[yellow] Unresolved: {directory}: {err}[/yellow]")
|
||||
if sum(stats.values()) % 100 == 0:
|
||||
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
|
||||
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
|
||||
if stats["Errors"]:
|
||||
raise SystemExit(1)
|
||||
return stats
|
||||
@@ -83,12 +83,16 @@ def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
|
||||
return candidate
|
||||
|
||||
listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
|
||||
if listen_host and listen_host.lower() != "archivebox.localhost:8000":
|
||||
if listen_host and listen_host.lower() not in {"archivebox.localhost:8000", "archivebox.localhost:5797"}:
|
||||
return normalize(listen_host)
|
||||
|
||||
csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
|
||||
csrf_origins = [origin for origin in csrf_origins if origin]
|
||||
nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
|
||||
nondefault_csrf_origins = [
|
||||
origin
|
||||
for origin in csrf_origins
|
||||
if origin not in {"http://admin.archivebox.localhost:8000", "http://admin.archivebox.localhost:5797"}
|
||||
]
|
||||
if len(nondefault_csrf_origins) == 1:
|
||||
return nondefault_csrf_origins[0]
|
||||
|
||||
@@ -314,7 +318,7 @@ class ServerConfig(BaseConfigSet):
|
||||
)
|
||||
|
||||
SECRET_KEY: str = Field(default_factory=lambda: "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789_") for _ in range(50)))
|
||||
BIND_ADDR: str = Field(default="127.0.0.1:8000")
|
||||
BIND_ADDR: str = Field(default="127.0.0.1:5797")
|
||||
BASE_URL: str = Field(default="")
|
||||
ALLOWED_HOSTS: str = Field(default="*")
|
||||
CSRF_TRUSTED_ORIGINS: str = Field(default="")
|
||||
|
||||
@@ -5,9 +5,10 @@ from functools import lru_cache
|
||||
from types import SimpleNamespace
|
||||
|
||||
from django.contrib import admin, messages
|
||||
from django.core.exceptions import PermissionDenied
|
||||
from django.urls import path, reverse
|
||||
from django.shortcuts import get_object_or_404, redirect
|
||||
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
|
||||
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed, Http404
|
||||
from django.utils import timezone
|
||||
from django.utils.html import format_html, format_html_join
|
||||
from django.utils.safestring import mark_safe
|
||||
@@ -20,6 +21,7 @@ from archivebox.config.common import get_config
|
||||
from archivebox.misc.util import htmldecode, urldecode
|
||||
from archivebox.misc.paginators import AcceleratedPaginator
|
||||
from archivebox.misc.logging_util import printable_filesize
|
||||
from archivebox.misc.serve_static import serve_static_with_byterange_support
|
||||
from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
|
||||
from archivebox.search.views import admin_snapshot_search_stream_view
|
||||
from archivebox.core.routes_util import build_snapshot_url, build_web_url
|
||||
@@ -46,6 +48,12 @@ from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
|
||||
GLOBAL_CONTEXT = {}
|
||||
|
||||
SNAPSHOT_PERMISSION_META = PERMISSIONS_META
|
||||
GRID_PREVIEW_OUTPUTS = {
|
||||
("screenshot", "screenshot.png"),
|
||||
("chrome_extension_screenshot", "screenshot-1.png"),
|
||||
("chrome_extension_screenshot", "screenshot.png"),
|
||||
("favicon", "favicon.ico"),
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
@@ -569,6 +577,11 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
custom_urls = [
|
||||
path("grid/", self.admin_site.admin_view(self.grid_view), name="grid"),
|
||||
path("search-stream/", self.admin_site.admin_view(self.search_stream_view), name="core_snapshot_search_stream"),
|
||||
path(
|
||||
"<path:object_id>/preview/<str:plugin>/<path:filename>",
|
||||
self.admin_site.admin_view(self.preview_view, cacheable=True),
|
||||
name="core_snapshot_preview",
|
||||
),
|
||||
path("<path:object_id>/redo-failed/", self.admin_site.admin_view(self.redo_failed_view), name="core_snapshot_redo_failed"),
|
||||
path(
|
||||
"<path:object_id>/set-permissions/",
|
||||
@@ -578,6 +591,26 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
]
|
||||
return custom_urls + urls
|
||||
|
||||
def preview_view(self, request, object_id, plugin, filename):
|
||||
"""Serve grid thumbnails from the authenticated admin origin."""
|
||||
if (plugin, filename) not in GRID_PREVIEW_OUTPUTS:
|
||||
raise Http404("Unsupported snapshot preview")
|
||||
snapshot = get_object_or_404(Snapshot, pk=object_id)
|
||||
if not self.has_view_or_change_permission(request, snapshot):
|
||||
raise PermissionDenied
|
||||
result = snapshot.archiveresult_set.filter(plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED).first()
|
||||
file_info = (result.output_files or {}).get(filename) if result else None
|
||||
if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
|
||||
raise Http404("Snapshot preview does not exist")
|
||||
output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
|
||||
request.archivebox_cache_policy = "private"
|
||||
response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
|
||||
if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
|
||||
raise Http404("Snapshot preview is not an image")
|
||||
response.headers["Vary"] = "Cookie"
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
return response
|
||||
|
||||
def search_stream_view(self, request):
|
||||
return admin_snapshot_search_stream_view(self, request)
|
||||
|
||||
@@ -1046,7 +1079,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
return None
|
||||
|
||||
return format_html(
|
||||
'<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">',
|
||||
'<a href="{}" title="Open snapshot details">'
|
||||
'<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">'
|
||||
"</a>",
|
||||
build_web_url(f"/{obj.archive_path_from_db}/index.html", request=self.request, config=self.request.archivebox_config),
|
||||
preview["img_url"],
|
||||
preview["img_alt"],
|
||||
preview["preview_class"],
|
||||
|
||||
+15
-2
@@ -7,13 +7,26 @@ For more information on this file, see
|
||||
https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
|
||||
"""
|
||||
|
||||
from django.core.asgi import get_asgi_application
|
||||
from django.core.handlers.asgi import ASGIHandler
|
||||
|
||||
from archivebox.config.django import setup_django
|
||||
|
||||
setup_django(check_db=True)
|
||||
|
||||
django_application = get_asgi_application()
|
||||
|
||||
class ArchiveBoxASGIHandler(ASGIHandler):
|
||||
def create_request(self, scope, body_file):
|
||||
# ASGI already spools the complete body. Django's multipart parser
|
||||
# otherwise treats chunked uploads without Content-Length as empty.
|
||||
if not any(name == b"content-length" for name, _value in scope.get("headers", [])):
|
||||
body_file.seek(0, 2)
|
||||
length = body_file.tell()
|
||||
body_file.seek(0)
|
||||
scope = {**scope, "headers": [*scope.get("headers", []), (b"content-length", str(length).encode("ascii"))]}
|
||||
return super().create_request(scope, body_file)
|
||||
|
||||
|
||||
django_application = ArchiveBoxASGIHandler()
|
||||
|
||||
|
||||
async def application(scope, receive, send):
|
||||
|
||||
@@ -13,6 +13,7 @@ from django.http import HttpResponseForbidden, HttpResponseNotModified
|
||||
from django.shortcuts import redirect
|
||||
from django.utils import timezone
|
||||
from django.utils.http import http_date
|
||||
from django.utils.cache import patch_cache_control, patch_vary_headers
|
||||
|
||||
from archivebox.config import VERSION
|
||||
from archivebox.config.common import get_config, get_request_config
|
||||
@@ -123,13 +124,12 @@ def AdminCookieIsolationMiddleware(get_response):
|
||||
|
||||
|
||||
def CacheControlMiddleware(get_response):
|
||||
snapshot_path_re = re.compile(r"^/[^/]+/\\d{8}/[^/]+/[0-9a-fA-F-]{8,36}/")
|
||||
static_cache_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
|
||||
|
||||
def middleware(request):
|
||||
response = get_response(request)
|
||||
|
||||
if request.path.startswith("/static/"):
|
||||
if request.path.startswith("/static/") and response.status_code in (200, 304):
|
||||
rel_path = request.path[len("/static/") :]
|
||||
static_path = finders.find(rel_path)
|
||||
if static_path:
|
||||
@@ -154,15 +154,17 @@ def CacheControlMiddleware(get_response):
|
||||
response.headers["Last-Modified"] = http_date(mtime)
|
||||
return response
|
||||
|
||||
if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
|
||||
"Cache-Control",
|
||||
):
|
||||
config = request.__dict__.get("archivebox_config")
|
||||
if config is None:
|
||||
config = get_config(resolve_plugins=False)
|
||||
request.archivebox_config = config
|
||||
policy = "private" if config.PERMISSIONS == "private" else "public"
|
||||
response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
|
||||
# Keep artifact freshness/validators, but never let a shared cache store
|
||||
# private or unlisted snapshots. Auth redirects and errors stay unstored.
|
||||
snapshot_policy = getattr(request, "archivebox_cache_policy", None)
|
||||
if snapshot_policy and response.status_code in (200, 206, 304):
|
||||
if not response.get("Cache-Control"):
|
||||
response["Cache-Control"] = f"{snapshot_policy}, max-age=60, stale-while-revalidate=300"
|
||||
if snapshot_policy == "private":
|
||||
patch_cache_control(response, private=True)
|
||||
elif snapshot_policy or not response.get("Cache-Control"):
|
||||
response["Cache-Control"] = "private, no-store"
|
||||
patch_vary_headers(response, ("Cookie", "Authorization"))
|
||||
return response
|
||||
|
||||
return middleware
|
||||
|
||||
+37
-19
@@ -1394,7 +1394,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
Uses full hostname with sanitized special chars.
|
||||
|
||||
Examples:
|
||||
https://example.com:8080 → example.com_8080
|
||||
https://example.com:5797 → example.com_5797
|
||||
https://sub.example.com → sub.example.com
|
||||
file:///path → localhost
|
||||
data:text/html → data
|
||||
@@ -1546,7 +1546,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
return cls.objects.select_related("crawl__created_by").filter(url=url, timestamp=timestamp).first()
|
||||
|
||||
@classmethod
|
||||
def create_from_directory(cls, snapshot_dir: Path) -> Optional["Snapshot"]:
|
||||
def create_from_directory(cls, snapshot_dir: Path, *, crawl=None, preserve_identity: bool = False) -> Optional["Snapshot"]:
|
||||
"""
|
||||
Create new Snapshot from orphaned directory.
|
||||
|
||||
@@ -1618,26 +1618,25 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
return None
|
||||
|
||||
# Ensure uniqueness (reuses existing logic from create_or_update_from_dict)
|
||||
timestamp = cls._ensure_unique_timestamp(url, timestamp)
|
||||
if not preserve_identity:
|
||||
timestamp = cls._ensure_unique_timestamp(url, timestamp)
|
||||
|
||||
# Detect version
|
||||
fs_version = cls._detect_fs_version_from_index(data)
|
||||
|
||||
system_user_id = get_or_create_system_user_pk()
|
||||
catchall_crawl, _ = Crawl.objects.get_or_create(
|
||||
label="[migration] orphaned snapshots",
|
||||
defaults={
|
||||
"urls": f"# Orphaned snapshot: {url}",
|
||||
"max_depth": 0,
|
||||
"created_by_id": system_user_id,
|
||||
},
|
||||
)
|
||||
if cls.objects.filter(crawl=catchall_crawl, url=url).exists():
|
||||
catchall_crawl = crawl
|
||||
if catchall_crawl is None:
|
||||
system_user_id = get_or_create_system_user_pk()
|
||||
catchall_crawl, _ = Crawl.objects.get_or_create(
|
||||
label="[migration] orphaned snapshots",
|
||||
defaults={"urls": f"# Orphaned snapshot: {url}", "max_depth": 0, "created_by_id": system_user_id},
|
||||
)
|
||||
if not preserve_identity and cls.objects.filter(crawl=catchall_crawl, url=url).exists():
|
||||
catchall_crawl = Crawl.objects.create(
|
||||
label=f"[migration] orphaned snapshot {timestamp}",
|
||||
urls=url,
|
||||
max_depth=0,
|
||||
created_by_id=system_user_id,
|
||||
created_by_id=catchall_crawl.created_by_id,
|
||||
)
|
||||
|
||||
snapshot_kwargs = {
|
||||
@@ -1647,6 +1646,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
"fs_version": fs_version,
|
||||
"crawl": catchall_crawl,
|
||||
}
|
||||
if preserve_identity:
|
||||
snapshot_kwargs.update({key: data[key] for key in ("id", "depth", "config", "notes") if key in data})
|
||||
try:
|
||||
bookmarked_at = parse_date(data.get("bookmarked_at") or timestamp)
|
||||
except (TypeError, ValueError, OSError):
|
||||
@@ -1847,7 +1848,13 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
result_data["plugin"] = result_data.get("plugin") or result_data.get("extractor") or plugin
|
||||
self._create_archive_result_if_missing(result_data, existing, update_existing=update_existing)
|
||||
|
||||
def _create_archive_result_if_missing(self, result_data: dict, existing: dict, update_existing: bool = True):
|
||||
def _create_archive_result_if_missing(
|
||||
self,
|
||||
result_data: dict,
|
||||
existing: dict,
|
||||
update_existing: bool = True,
|
||||
preserve_identity: bool = False,
|
||||
):
|
||||
"""Create ArchiveResult if not already in DB."""
|
||||
from dateutil import parser
|
||||
from django.db import transaction
|
||||
@@ -1925,7 +1932,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
|
||||
# Machine.current() can probe the host and sanitize config. Do that before
|
||||
# atomic() so the transaction below only covers the two related row writes.
|
||||
machine = Machine.current() if cmd or pwd else None
|
||||
machine = Machine.current() if (cmd or pwd) and not preserve_identity else None
|
||||
with transaction.atomic():
|
||||
if machine is not None:
|
||||
process = Process.objects.create(
|
||||
@@ -1941,6 +1948,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
)
|
||||
|
||||
archiveresult = ArchiveResult.objects.create(
|
||||
**({"id": result_data["id"]} if preserve_identity else {}),
|
||||
snapshot=self,
|
||||
plugin=plugin,
|
||||
hook_name=hook_name,
|
||||
@@ -1954,6 +1962,10 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
end_ts=end_ts,
|
||||
process=process,
|
||||
)
|
||||
if preserve_identity:
|
||||
dates = {key: parse_date(result_data[key]) for key in ("created_at", "modified_at") if result_data.get(key)}
|
||||
if dates:
|
||||
ArchiveResult.objects.filter(pk=archiveresult.pk).update(**dates)
|
||||
existing[(plugin, hook_name)] = archiveresult
|
||||
|
||||
def write_index_json(self):
|
||||
@@ -2010,7 +2022,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
# without a corresponding DB row can be retained byte-for-byte.
|
||||
binaries_seen = set()
|
||||
processes_seen = set()
|
||||
records = [self.to_json()]
|
||||
records = [self.to_json(), self.crawl.to_json()]
|
||||
for ar in archive_results:
|
||||
process = ar.process_record
|
||||
if process and process.binary and process.binary_id not in binaries_seen:
|
||||
@@ -2837,6 +2849,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
Includes all fields needed to fully reconstruct/identify this snapshot.
|
||||
"""
|
||||
from archivebox.config import VERSION
|
||||
from archivebox.config.common import redact_sensitive_config
|
||||
|
||||
archive_size = self.archive_size
|
||||
|
||||
@@ -2850,6 +2863,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
|
||||
"tags": self.tags_str(),
|
||||
"bookmarked_at": self.bookmarked_at.isoformat() if self.bookmarked_at else None,
|
||||
"created_at": self.created_at.isoformat() if self.created_at else None,
|
||||
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
|
||||
"config": redact_sensitive_config(self.config),
|
||||
"notes": self.notes,
|
||||
"timestamp": self.timestamp,
|
||||
"depth": self.depth,
|
||||
"status": self.status,
|
||||
@@ -4009,6 +4025,8 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
|
||||
"schema_version": VERSION,
|
||||
"id": str(self.id),
|
||||
"snapshot_id": str(self.snapshot_id),
|
||||
"created_at": self.created_at.isoformat() if self.created_at else None,
|
||||
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
|
||||
"plugin": self.plugin,
|
||||
"hook_name": self.hook_name,
|
||||
"status": self.status,
|
||||
@@ -4222,7 +4240,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
|
||||
def output_file_paths(self) -> list[str]:
|
||||
return list(self.output_file_map().keys())
|
||||
|
||||
def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True) -> bool:
|
||||
def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True, full_scan: bool = False) -> bool:
|
||||
from abx_dl.output_files import OutputManifest, output_file_from_path
|
||||
|
||||
if self.plugin == "title":
|
||||
@@ -4262,7 +4280,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
|
||||
add_file(snapshot_dir / raw_output, raw_output, root_relative=True)
|
||||
|
||||
plugin_dir = snapshot_dir / self.plugin
|
||||
if not output_files and plugin_dir.is_dir():
|
||||
if (full_scan or not output_files) and plugin_dir.is_dir():
|
||||
output_files = OutputManifest.scan(plugin_dir, containment_root=snapshot_dir).as_mapping()
|
||||
|
||||
if not output_files:
|
||||
|
||||
@@ -140,6 +140,13 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
|
||||
orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
|
||||
else:
|
||||
orphaned_hook_processes = orphaned_hook_processes.none()
|
||||
else:
|
||||
# Global startup recovery must stay database-only. Process.pwd points
|
||||
# into archive payload storage, which may be a cold FUSE mount; scanning
|
||||
# every historical unlinked hook here blocks the sole scheduler before
|
||||
# it can claim newly queued work. Explicit crawl recovery below remains
|
||||
# available for the work the caller asked to resume.
|
||||
orphaned_hook_processes = orphaned_hook_processes.none()
|
||||
for process in orphaned_hook_processes.only(
|
||||
"id",
|
||||
"pwd",
|
||||
|
||||
@@ -74,7 +74,7 @@ def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwa
|
||||
had ``CSRF_TRUSTED_ORIGINS=https://archive.example.com`` set (required for
|
||||
admin login to work). On upgrade, ``BASE_URL`` is the new knob — but it
|
||||
defaults to empty, and falling through to ``BIND_ADDR`` produces an
|
||||
unreachable URL like ``http://0.0.0.0:8000``. If the user has exactly one
|
||||
unreachable URL like ``http://0.0.0.0:5797``. If the user has exactly one
|
||||
CSRF origin we treat it as the implicit BASE_URL so links/redirects keep
|
||||
pointing at the public hostname they already configured.
|
||||
|
||||
@@ -133,7 +133,7 @@ def canonical_base_host_for_request(request_host: str) -> str:
|
||||
"""Strip role subdomains and remap loopback hostnames to ``archivebox.localhost``.
|
||||
|
||||
Used by the banner suggestion and the in-browser pin endpoint: when the
|
||||
user is hitting the server on raw ``localhost:9292`` or ``127.0.0.1:9292``
|
||||
user is hitting the server on raw ``localhost:5797`` or ``127.0.0.1:5797``
|
||||
we want to suggest the wildcard-friendly ``archivebox.localhost`` family
|
||||
instead, so the eventual pinned ``BASE_URL`` plays nicely with subdomain
|
||||
routing without forcing the user to add a /etc/hosts entry.
|
||||
|
||||
@@ -87,6 +87,7 @@ DJANGO_OBJECT_ACTIONS_DEFAULT_HTTP_METHOD = "POST"
|
||||
MIDDLEWARE = [
|
||||
"archivebox.core.middleware.TimezoneMiddleware",
|
||||
"django.middleware.security.SecurityMiddleware",
|
||||
"archivebox.core.middleware.CacheControlMiddleware",
|
||||
"archivebox.core.middleware.AdminCookieIsolationMiddleware",
|
||||
"django.contrib.sessions.middleware.SessionMiddleware",
|
||||
"django.middleware.common.CommonMiddleware",
|
||||
@@ -97,7 +98,6 @@ MIDDLEWARE = [
|
||||
"archivebox.core.middleware.ServerSecurityModeMiddleware",
|
||||
"archivebox.core.middleware.HostRoutingMiddleware",
|
||||
"django.contrib.messages.middleware.MessageMiddleware",
|
||||
"archivebox.core.middleware.CacheControlMiddleware",
|
||||
# Additional middlewares from plugins (if any)
|
||||
]
|
||||
|
||||
@@ -390,7 +390,7 @@ SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
|
||||
# get_base_url() also covers deployments that only set CSRF_TRUSTED_ORIGINS (the
|
||||
# implicit-BASE_URL fallback used on 0.7.x->0.9.x upgrades), so HTTPS hardening
|
||||
# isn't lost until BASE_URL is migrated. An explicit plain-http base (e.g. local
|
||||
# http://archivebox.localhost:8000) disables proxy HTTPS handling.
|
||||
# http://archivebox.localhost:5797) disables proxy HTTPS handling.
|
||||
BASE_URL_IS_HTTPS = get_base_url(config=CONFIG).strip().lower().startswith("https://")
|
||||
if BASE_URL_IS_HTTPS or not CONFIG.BASE_URL:
|
||||
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
|
||||
|
||||
@@ -8,6 +8,7 @@ from urllib.parse import quote, urlparse
|
||||
from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
|
||||
from django import template
|
||||
from django.templatetags.static import static
|
||||
from django.urls import reverse
|
||||
from django.utils import timezone
|
||||
from django.utils.html import escape
|
||||
from django.utils.safestring import mark_safe
|
||||
@@ -460,9 +461,9 @@ def system_warnings_banner(context):
|
||||
"""Render the top-of-page warning banner for one of the conditions below,
|
||||
in priority order (highest first):
|
||||
|
||||
1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
|
||||
issue: until it's pinned, generated URLs can echo any Host the client
|
||||
sends, and admin/web/api routing has no canonical anchor.
|
||||
1. ``mode="unconfigured"``— ``BASE_URL`` is empty and the operator has
|
||||
not selected safe single-domain routing. That mode intentionally uses
|
||||
each device's connection address, including LAN and Tailscale IPs.
|
||||
2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
|
||||
an origin that differs from the configured canonical ``BASE_URL``.
|
||||
3. ``mode="unsafe"`` — ``SERVER_SECURITY_MODE`` explicitly enables
|
||||
@@ -489,7 +490,7 @@ def system_warnings_banner(context):
|
||||
|
||||
config = get_config(resolve_plugins=False)
|
||||
|
||||
if not config.BASE_URL:
|
||||
if not config.BASE_URL and config.SERVER_SECURITY_MODE != "safe-onedomain-nojsreplay":
|
||||
return get_setup_wizard_context(context.get("request"), config)
|
||||
mismatch = get_base_url_mismatch_context(context.get("request"), config)
|
||||
if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
|
||||
@@ -613,6 +614,13 @@ def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) ->
|
||||
return ""
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def admin_snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
|
||||
if not snapshot_archiveresult_url(context, snapshot, plugin, filename):
|
||||
return ""
|
||||
return reverse("admin:core_snapshot_preview", args=(snapshot.pk, plugin, filename))
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_index_row(context, link) -> str:
|
||||
snapshot_base = _snapshot_base_url_for_context(context, link)
|
||||
|
||||
@@ -23,6 +23,7 @@ from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidde
|
||||
from django.shortcuts import redirect, render
|
||||
from django.utils.decorators import method_decorator
|
||||
from django.utils.html import format_html, format_html_join
|
||||
from django.utils.cache import patch_cache_control, patch_vary_headers
|
||||
from django.utils.safestring import mark_safe
|
||||
from django.views import View
|
||||
from django.views.decorators.csrf import csrf_exempt, csrf_protect
|
||||
@@ -327,6 +328,7 @@ class SnapshotView(View):
|
||||
|
||||
@staticmethod
|
||||
def render_live_index(request, snapshot):
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
return render(
|
||||
template_name="core/snapshot.html",
|
||||
request=request,
|
||||
@@ -350,6 +352,7 @@ class SnapshotView(View):
|
||||
try:
|
||||
try:
|
||||
snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
if not can_view_snapshot(request, snapshot):
|
||||
return _private_snapshot_auth_redirect(request, snapshot, archivefile or "index.html")
|
||||
canonical_base = snapshot.url_path
|
||||
@@ -629,6 +632,7 @@ class SnapshotPathView(View):
|
||||
status=404,
|
||||
)
|
||||
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
canonical_base = snapshot.url_path
|
||||
if date:
|
||||
requested_base = f"{username}/{date}/{domain or url or ''}"
|
||||
@@ -891,7 +895,7 @@ def _serve_responses_path(request, responses_root: Path, rel_path: str, show_ind
|
||||
return None
|
||||
|
||||
|
||||
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
rel_path = path or ""
|
||||
request_config = get_request_config(
|
||||
request,
|
||||
@@ -899,6 +903,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
|
||||
)
|
||||
request.archivebox_config = request_config
|
||||
request.archivebox_snapshot_url = snapshot.url
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
snapshot._runtime_config = request_config
|
||||
|
||||
if rel_path.startswith("replay/") or rel_path == "replay":
|
||||
@@ -911,7 +916,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
|
||||
# /progress.json on through to the same view used everywhere else. The caller
|
||||
# passes snapshot_id explicitly in the query string — we don't read it from the
|
||||
# subdomain (this keeps the endpoint identical across all security modes).
|
||||
return live_progress_view(request)
|
||||
return live_progress_view(request, authorized_snapshot=snapshot)
|
||||
|
||||
is_directory_request = bool(path) and path.endswith("/")
|
||||
show_indexes = bool(request.GET.get("files")) or (request_config.USES_SUBDOMAIN_ROUTING and is_directory_request)
|
||||
@@ -954,6 +959,20 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
|
||||
raise Http404
|
||||
|
||||
|
||||
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
response = _build_snapshot_replay_response(request, snapshot, path)
|
||||
if path != "progress.json" and response.status_code in (200, 206, 304):
|
||||
if not response.get("Cache-Control"):
|
||||
policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
response.headers["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
|
||||
if snapshot.permissions != PERMISSIONS_PUBLIC:
|
||||
patch_cache_control(response, private=True)
|
||||
else:
|
||||
response.headers["Cache-Control"] = "private, no-store"
|
||||
patch_vary_headers(response, ("Cookie", "Authorization"))
|
||||
return response
|
||||
|
||||
|
||||
def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str = ""):
|
||||
request_config = get_request_config(request, resolve_plugins=False)
|
||||
request.archivebox_config = request_config
|
||||
@@ -977,6 +996,8 @@ def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str =
|
||||
|
||||
root_match = (match[0], match[1]) if match else _latest_responses_root(snapshots, domain)
|
||||
responses_root = root_match[1] if root_match else None
|
||||
if root_match:
|
||||
request.archivebox_cache_policy = "public" if root_match[0].permissions == PERMISSIONS_PUBLIC else "private"
|
||||
if request_config.USES_SUBDOMAIN_ROUTING:
|
||||
snapshot = root_match[0] if root_match else (snapshots[0] if requested_root_index and snapshots else None)
|
||||
if snapshot:
|
||||
|
||||
@@ -467,6 +467,8 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
|
||||
"tags_str": self.tags_str,
|
||||
"label": self.label,
|
||||
"created_at": self.created_at.isoformat() if self.created_at else None,
|
||||
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
|
||||
"created_by": self.created_by.username,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -88,7 +88,7 @@ def get_vm_info():
|
||||
try:
|
||||
# Getting SMBIOS data from sysfs.
|
||||
# SMBIOS 2.8 present.
|
||||
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:8000 │
|
||||
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:5797 │
|
||||
# Handle 0x0100, DMI type 1, 27 bytes
|
||||
# System Information
|
||||
# Manufacturer: DigitalOcean
|
||||
|
||||
@@ -129,7 +129,7 @@ def _resolve_archive_path(document_root: str | Path, rel_path: str) -> tuple[Pat
|
||||
|
||||
def _cache_policy(config=None, **config_kwargs) -> str:
|
||||
config = config or get_config(resolve_plugins=False, **config_kwargs)
|
||||
return "private" if config.PERMISSIONS == "private" else "public"
|
||||
return "public" if config.PERMISSIONS == "public" else "private"
|
||||
|
||||
|
||||
def _format_direntry_timestamp(stat_result: os.stat_result) -> str:
|
||||
@@ -605,9 +605,15 @@ def _apply_transformed_html_preview_style(html_text: str) -> str:
|
||||
return f"{TRANSFORMED_HTML_PREVIEW_STYLE}\n{html_text}"
|
||||
|
||||
|
||||
def _set_transformed_response_headers(response, fullpath: Path, statobj: os.stat_result, encoding: str | None, config) -> None:
|
||||
def _set_transformed_response_headers(
|
||||
response,
|
||||
fullpath: Path,
|
||||
statobj: os.stat_result,
|
||||
encoding: str | None,
|
||||
cache_policy: str,
|
||||
) -> None:
|
||||
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
|
||||
if encoding:
|
||||
response.headers["Content-Encoding"] = encoding
|
||||
@@ -858,19 +864,22 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
config = request.__dict__.get("archivebox_config")
|
||||
if config is None:
|
||||
config = get_config(resolve_plugins=False)
|
||||
cache_policy = getattr(request, "archivebox_cache_policy", None) or _cache_policy(config=config)
|
||||
fullpath, path = _resolve_archive_path(document_root, path)
|
||||
if os.access(fullpath, os.R_OK) and fullpath.is_dir():
|
||||
if request.GET.get("download") == "zip" and show_indexes:
|
||||
return _build_directory_zip_response(
|
||||
response = _build_directory_zip_response(
|
||||
fullpath,
|
||||
path,
|
||||
is_archive_replay=is_archive_replay,
|
||||
use_async_stream=_is_asgi_request(request),
|
||||
config=config,
|
||||
)
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
return response
|
||||
if show_indexes:
|
||||
response = _render_directory_index(request, path, fullpath)
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Last-Modified"] = http_date(fullpath.stat().st_mtime)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
@@ -899,7 +908,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if etag in inm_list or etag.strip('"') in [i.strip('"') for i in inm_list]:
|
||||
not_modified = HttpResponseNotModified()
|
||||
not_modified.headers["ETag"] = etag
|
||||
not_modified.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
|
||||
not_modified.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
return _apply_archive_replay_headers(
|
||||
not_modified,
|
||||
@@ -940,8 +949,10 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
request.META.get("HTTP_IF_MODIFIED_SINCE"),
|
||||
statobj.st_mtime,
|
||||
):
|
||||
not_modified = HttpResponseNotModified()
|
||||
not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
return _apply_archive_replay_headers(
|
||||
HttpResponseNotModified(),
|
||||
not_modified,
|
||||
fullpath=fullpath,
|
||||
content_type=content_type,
|
||||
is_archive_replay=is_archive_replay,
|
||||
@@ -956,7 +967,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
decoded = fullpath.read_text(encoding="utf-8", errors="replace")
|
||||
wrapped = _render_text_preview_document(decoded, fullpath.name)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -976,7 +987,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
raw_image_url = f"{raw_image_url}?{urlencode(list(preview_query.lists()), doseq=True)}"
|
||||
wrapped = _render_image_preview_document(raw_image_url, fullpath.name)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1002,9 +1013,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
last_modified=http_date(statobj.st_mtime),
|
||||
etag=etag or "",
|
||||
cache_control=(
|
||||
f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
if etag
|
||||
else f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
f"{cache_policy}, max-age=31536000, immutable" if etag else f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
),
|
||||
content_encoding=encoding or "",
|
||||
)
|
||||
@@ -1042,7 +1051,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
wrapped, _rewrite_count = _rewrite_html_image_sources_for_request(request, wrapped, document_root, rel_path)
|
||||
wrapped = _apply_transformed_html_preview_style(wrapped)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1053,7 +1062,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if rewritten_count:
|
||||
rewritten_html = _apply_transformed_html_preview_style(rewritten_html)
|
||||
response = HttpResponse(rewritten_html, content_type=content_type)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1064,7 +1073,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if escaped_count and escaped_count > tag_count * 2:
|
||||
decoded = _apply_transformed_html_preview_style(decoded)
|
||||
response = HttpResponse(decoded, content_type=content_type)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1084,13 +1093,13 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
if etag:
|
||||
response.headers["ETag"] = etag
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
|
||||
else:
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
if is_text_like:
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
|
||||
if content_type.startswith("image/"):
|
||||
response.headers["Cache-Control"] = "public, max-age=604800, immutable"
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=604800, immutable"
|
||||
|
||||
# handle byte-range requests by serving chunk of file
|
||||
if stat.S_ISREG(statobj.st_mode):
|
||||
|
||||
@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
"Persona",
|
||||
{
|
||||
"fields": ("name", "created_by", "permissions"),
|
||||
"classes": ("card", "persona-card-primary"),
|
||||
"classes": ("card", "wide", "persona-card-primary"),
|
||||
},
|
||||
),
|
||||
(
|
||||
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
"import_extract_cookies",
|
||||
"import_capture_storage",
|
||||
),
|
||||
"classes": ("card", "wide"),
|
||||
"classes": ("card", "wide", "persona-card-import"),
|
||||
},
|
||||
),
|
||||
(
|
||||
|
||||
@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
|
||||
label="Autodiscovered profiles",
|
||||
widget=forms.RadioSelect,
|
||||
choices=(),
|
||||
help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
|
||||
help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
|
||||
)
|
||||
import_source = forms.CharField(
|
||||
required=False,
|
||||
|
||||
@@ -18,6 +18,7 @@ import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import urlparse
|
||||
@@ -31,6 +32,7 @@ if TYPE_CHECKING:
|
||||
|
||||
BROWSER_LABELS = {
|
||||
"chrome": "Google Chrome",
|
||||
"chrome-beta": "Google Chrome Beta",
|
||||
"chromium": "Chromium",
|
||||
"brave": "Brave",
|
||||
"edge": "Microsoft Edge",
|
||||
@@ -216,24 +218,29 @@ class PersonaImportResult:
|
||||
return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
|
||||
|
||||
|
||||
def get_chrome_user_data_dir() -> Path | None:
|
||||
def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
|
||||
"""Get the default Chrome user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
app_name = "Chrome Beta" if beta else "Chrome"
|
||||
|
||||
if system == "Darwin":
|
||||
candidates = [
|
||||
home / "Library" / "Application Support" / "Google" / "Chrome",
|
||||
home / "Library" / "Application Support" / "Google" / app_name,
|
||||
]
|
||||
elif system == "Linux":
|
||||
candidates = [
|
||||
home / ".config" / "google-chrome",
|
||||
home / ".config" / "chrome",
|
||||
]
|
||||
candidates = (
|
||||
[home / ".config" / "google-chrome-beta"]
|
||||
if beta
|
||||
else [
|
||||
home / ".config" / "google-chrome",
|
||||
home / ".config" / "chrome",
|
||||
]
|
||||
)
|
||||
elif system == "Windows":
|
||||
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
|
||||
candidates = [
|
||||
local_app_data / "Google" / "Chrome" / "User Data",
|
||||
local_app_data / "Google" / app_name / "User Data",
|
||||
]
|
||||
else:
|
||||
candidates = []
|
||||
@@ -320,6 +327,7 @@ def get_edge_user_data_dir() -> Path | None:
|
||||
|
||||
BROWSER_PROFILE_FINDERS = {
|
||||
"chrome": get_chrome_user_data_dir,
|
||||
"chrome-beta": partial(get_chrome_user_data_dir, beta=True),
|
||||
"chromium": get_chromium_user_data_dir,
|
||||
"brave": get_brave_user_data_dir,
|
||||
"edge": get_edge_user_data_dir,
|
||||
@@ -354,6 +362,12 @@ def validate_persona_name(name: str) -> tuple[bool, str]:
|
||||
|
||||
|
||||
def discover_local_browser_profiles() -> list[PersonaImportSource]:
|
||||
# The macOS companion exports with the host Keychain before creating personas.
|
||||
# Raw mounts alone cannot decrypt macOS cookies in the Linux container.
|
||||
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
|
||||
host_templates = discover_persona_template_profiles(Path(host_profiles))
|
||||
host_names = {source.source_name for source in host_templates}
|
||||
return host_templates + [source for source in discover_persona_template_profiles() if source.source_name not in host_names]
|
||||
discovered: list[PersonaImportSource] = []
|
||||
|
||||
for browser, finder in BROWSER_PROFILE_FINDERS.items():
|
||||
@@ -430,6 +444,11 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
|
||||
|
||||
def resolve_browser_import_source(browser: str, profile_dir: str | None = None) -> PersonaImportSource:
|
||||
browser = browser.lower().strip()
|
||||
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
|
||||
label = BROWSER_LABELS.get(browser, browser)
|
||||
exported = Path(host_profiles) / f"{label} - {profile_dir or 'Default'}"
|
||||
if exported.is_dir():
|
||||
return resolve_custom_import_source(str(exported))
|
||||
if browser not in BROWSER_PROFILE_FINDERS:
|
||||
supported = ", ".join(BROWSER_PROFILE_FINDERS)
|
||||
raise ValueError(f"Unknown browser: {browser}. Supported browsers: {supported}")
|
||||
@@ -547,6 +566,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
|
||||
return str(Path(source.browser_binary).expanduser().resolve())
|
||||
names = {
|
||||
"chrome": ("Google Chrome", "google-chrome"),
|
||||
"chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
|
||||
"chromium": ("Chromium", "chromium"),
|
||||
"brave": ("Brave Browser", "brave-browser"),
|
||||
"edge": ("Microsoft Edge", "microsoft-edge"),
|
||||
@@ -561,6 +581,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
|
||||
elif platform.system() == "Windows":
|
||||
relative = {
|
||||
"chrome": "Google/Chrome/Application/chrome.exe",
|
||||
"chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
|
||||
"edge": "Microsoft/Edge/Application/msedge.exe",
|
||||
"brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
|
||||
"chromium": "Chromium/Application/chrome.exe",
|
||||
|
||||
@@ -213,7 +213,7 @@ class PluginConfigFormMixin:
|
||||
"enabled_config_key": enabled_config_key,
|
||||
"description": str(schema.get("description") or "").strip(),
|
||||
"source_url": f"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/{plugin_name}",
|
||||
"docs_url": f"https://archivebox.github.io/abx-plugins/#{plugin_name}",
|
||||
"docs_url": f"https://plugins.archivebox.io/#{plugin_name}",
|
||||
"required_plugins": [str(item) for item in schema.get("required_plugins") or []],
|
||||
"required_binary_links": _build_required_binary_links(
|
||||
schema.get("required_binaries") or [],
|
||||
|
||||
@@ -17,7 +17,7 @@ from archivebox.config.common import get_live_config_url
|
||||
from archivebox.config.views import get_environment_binary_url, is_superuser
|
||||
from archivebox.plugins.discovery import BUILTIN_PLUGINS_DIR, USER_PLUGINS_DIR, discover_plugin_configs, iter_plugin_dirs
|
||||
|
||||
ABX_PLUGINS_DOCS_BASE_URL = "https://archivebox.github.io/abx-plugins/"
|
||||
ABX_PLUGINS_DOCS_BASE_URL = "https://plugins.archivebox.io/"
|
||||
ABX_PLUGINS_GITHUB_BASE_URL = "https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/"
|
||||
LIVE_PLUGIN_BASE_URL = "/admin/environment/plugins/"
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ def _live_progress_plugin_names() -> tuple[frozenset[str], frozenset[str]]:
|
||||
return download_plugin_names, indexing_plugin_names
|
||||
|
||||
|
||||
def live_progress_view(request):
|
||||
def live_progress_view(request, *, authorized_snapshot=None):
|
||||
"""Simple JSON endpoint for live progress status - used by admin progress monitor."""
|
||||
try:
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
@@ -63,7 +63,10 @@ def live_progress_view(request):
|
||||
except (TypeError, ValueError):
|
||||
return JsonResponse({"error": "Invalid snapshot_id"}, status=400)
|
||||
scoped_snapshot = Snapshot.objects.filter(id=snapshot_id_filter).select_related("crawl").first()
|
||||
if scoped_snapshot is None or not can_view_snapshot(request, scoped_snapshot):
|
||||
authorized_snapshot_id = str(getattr(authorized_snapshot, "id", "")).replace("-", "")
|
||||
if scoped_snapshot is None or (
|
||||
authorized_snapshot_id != str(scoped_snapshot.id).replace("-", "") and not can_view_snapshot(request, scoped_snapshot)
|
||||
):
|
||||
return JsonResponse({"error": "Permission denied"}, status=403)
|
||||
elif crawl_id_filter:
|
||||
# Crawl-only scope still requires staff: there's no per-crawl ACL helper,
|
||||
@@ -221,6 +224,8 @@ def live_progress_view(request):
|
||||
|
||||
api_base = get_api_base_url(request=request, config=request_config) if scoped_snapshot is not None else ""
|
||||
|
||||
running_screencasts = {}
|
||||
|
||||
def screencast_frame_url(crawl_id: str, crawl_dir: Path) -> str:
|
||||
frame_path = crawl_dir / "chrome_screencast" / "latest.jpg"
|
||||
try:
|
||||
@@ -229,7 +234,8 @@ def live_progress_view(request):
|
||||
return ""
|
||||
if frame_stat.st_size <= 0:
|
||||
return ""
|
||||
if now.timestamp() - frame_stat.st_mtime > 15:
|
||||
started_at = running_screencasts.get(crawl_id)
|
||||
if started_at is None or frame_stat.st_mtime < started_at.timestamp():
|
||||
return ""
|
||||
rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={frame_stat.st_mtime_ns}"
|
||||
return f"{api_base}{rel}" if api_base else rel
|
||||
@@ -552,6 +558,10 @@ def live_progress_view(request):
|
||||
snapshot_id = str(matched_snapshot["id"])
|
||||
running_worker_ids.add(str(proc["id"]))
|
||||
_plugin, _label, phase, _hook_name = process_label(proc["cmd"])
|
||||
if _plugin == "chrome_screencast" and proc["started_at"] and Process(**proc, machine_id=machine_id).is_running:
|
||||
# CDP sends frames only when the page changes. Keep a still
|
||||
# frame visible for its live capture, never for a previous run.
|
||||
running_screencasts[crawl_id] = max(proc["started_at"], running_screencasts.get(crawl_id, proc["started_at"]))
|
||||
if crawl_id and proc["pid"]:
|
||||
crawl_process_pids.setdefault(crawl_id, proc["pid"])
|
||||
if phase == "snapshot" and snapshot_id and proc["pid"]:
|
||||
|
||||
@@ -1643,11 +1643,19 @@ def run_install(*, plugin_names: list[str] | None = None) -> None:
|
||||
asyncio.run(_run_install(plugin_names=plugin_names))
|
||||
|
||||
|
||||
def _first_due_id(queryset):
|
||||
return queryset.order_by("retry_at", "created_at").values_list("id", flat=True).first()
|
||||
def _first_due_id(queryset, *, newest_first: bool = False):
|
||||
ordering = ("-retry_at", "-created_at") if newest_first else ("retry_at", "created_at")
|
||||
return queryset.order_by(*ordering).values_list("id", flat=True).first()
|
||||
|
||||
|
||||
def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: int, interactive_interrupts: bool) -> bool:
|
||||
def _run_due_crawl_status(
|
||||
status: str,
|
||||
*,
|
||||
crawl_id: str | None,
|
||||
lock_seconds: int,
|
||||
interactive_interrupts: bool,
|
||||
newest_first: bool = False,
|
||||
) -> bool:
|
||||
from archivebox.crawls.models import Crawl
|
||||
|
||||
due_crawls = Crawl.objects.filter(
|
||||
@@ -1656,7 +1664,7 @@ def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: in
|
||||
)
|
||||
if crawl_id:
|
||||
due_crawls = due_crawls.filter(id=crawl_id)
|
||||
due_crawl_id = _first_due_id(due_crawls)
|
||||
due_crawl_id = _first_due_id(due_crawls, newest_first=newest_first)
|
||||
if due_crawl_id is None:
|
||||
return False
|
||||
due_crawl = Crawl.objects.filter(id=due_crawl_id).first()
|
||||
@@ -1670,8 +1678,15 @@ def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: in
|
||||
return True
|
||||
|
||||
|
||||
def _run_due_snapshot_query(queryset, *, lock_seconds: int, interactive_interrupts: bool, runtime_config) -> bool:
|
||||
due_snapshot_id = _first_due_id(queryset)
|
||||
def _run_due_snapshot_query(
|
||||
queryset,
|
||||
*,
|
||||
lock_seconds: int,
|
||||
interactive_interrupts: bool,
|
||||
runtime_config,
|
||||
newest_first: bool = False,
|
||||
) -> bool:
|
||||
due_snapshot_id = _first_due_id(queryset, newest_first=newest_first)
|
||||
return _run_due_snapshot_id(
|
||||
due_snapshot_id,
|
||||
lock_seconds=lock_seconds,
|
||||
@@ -1779,20 +1794,31 @@ def run_pending_crawls(
|
||||
)
|
||||
if crawl_id:
|
||||
active_snapshots = active_snapshots.filter(crawl_id=crawl_id)
|
||||
# Admit new submissions between captures instead of making them
|
||||
# wait behind every overdue child of an old, resumed crawl.
|
||||
if _run_due_snapshot_query(
|
||||
active_snapshots,
|
||||
active_snapshots.filter(crawl__status=Crawl.StatusChoices.QUEUED),
|
||||
lock_seconds=60,
|
||||
interactive_interrupts=interactive_interrupts,
|
||||
runtime_config=runtime_config,
|
||||
newest_first=True,
|
||||
):
|
||||
continue
|
||||
|
||||
if not maintenance_only:
|
||||
if _run_due_crawl_status(
|
||||
Crawl.StatusChoices.QUEUED,
|
||||
crawl_id=crawl_id,
|
||||
lock_seconds=crawl_claim_lock_seconds,
|
||||
interactive_interrupts=interactive_interrupts,
|
||||
newest_first=True,
|
||||
):
|
||||
continue
|
||||
|
||||
if _run_due_snapshot_query(
|
||||
active_snapshots,
|
||||
lock_seconds=60,
|
||||
interactive_interrupts=interactive_interrupts,
|
||||
runtime_config=runtime_config,
|
||||
):
|
||||
continue
|
||||
|
||||
|
||||
@@ -38,8 +38,8 @@
|
||||
border-bottom: 1px dashed #64748b;
|
||||
}
|
||||
|
||||
.field-import_mode ul,
|
||||
.field-import_discovered_profile ul {
|
||||
.abx-import-mode,
|
||||
.abx-profile-picker {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
@@ -47,20 +47,26 @@
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.field-import_mode ul {
|
||||
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
|
||||
.abx-import-mode {
|
||||
grid-template-columns: repeat(auto-fit, minmax(min(220px, 100%), 1fr));
|
||||
}
|
||||
|
||||
.field-import_discovered_profile ul {
|
||||
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
|
||||
max-height: 460px;
|
||||
overflow: auto;
|
||||
padding-right: 4px;
|
||||
.abx-profile-picker {
|
||||
grid-template-columns: repeat(auto-fit, minmax(min(260px, 100%), 1fr));
|
||||
}
|
||||
|
||||
.field-import_mode li,
|
||||
.field-import_discovered_profile li {
|
||||
.abx-import-mode > div,
|
||||
.abx-profile-picker > div {
|
||||
margin: 0;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
/* Import rows grow with their contents instead of sharing a fixed flex
|
||||
height. Avoid overflow-x:hidden turning overflow-y:visible into auto. */
|
||||
#content-main #persona_form .persona-card-import > .form-row,
|
||||
#content-main #persona_form .persona-card-import fieldset > div {
|
||||
flex: none;
|
||||
overflow: visible;
|
||||
}
|
||||
|
||||
.field-import_mode label,
|
||||
@@ -68,7 +74,6 @@
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
align-items: flex-start;
|
||||
min-height: 100%;
|
||||
padding: 14px 16px;
|
||||
border-radius: 14px;
|
||||
border: 1px solid #dbe4ee;
|
||||
@@ -167,9 +172,14 @@
|
||||
color: #991b1b;
|
||||
}
|
||||
|
||||
body.model-persona.change-form #content-main form fieldset.persona-card-primary {
|
||||
flex: 1 1 520px !important;
|
||||
max-width: calc(66.66% - 10px) !important;
|
||||
#content-main #persona_form .field-name .flex-container {
|
||||
max-width: 100% !important;
|
||||
}
|
||||
|
||||
#persona_form .field-name input {
|
||||
width: 100%;
|
||||
max-width: none;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body.model-persona.change-form #content-main form fieldset.persona-card-timestamps {
|
||||
@@ -257,6 +267,11 @@ document.addEventListener('DOMContentLoaded', function () {
|
||||
or by pasting in a CDP url from
|
||||
<code class="persona-import-copy" data-copy-text="chrome://inspect#remote-debugging" title="Copy chrome://inspect#remote-debugging">chrome://inspect#remote-debugging</code> (e.g. <code>http://127.0.0.1:9222</code>).
|
||||
</p>
|
||||
<p>
|
||||
You can also import cookies into this persona from another browser using the
|
||||
<a href="https://extension.archivebox.io/" target="_blank" rel="noopener noreferrer">ArchiveBox browser extension</a>.
|
||||
Create the persona here, then select it in the extension's settings to sync cookies from that browser.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
{{ block.super }}
|
||||
@@ -264,9 +279,9 @@ document.addEventListener('DOMContentLoaded', function () {
|
||||
|
||||
{% block after_field_sets %}
|
||||
{{ block.super }}
|
||||
<section id="plugin-config" class="module aligned persona-plugin-config">
|
||||
<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">
|
||||
<h2>Plugin Config</h2>
|
||||
<p>These typed controls update the same Persona config JSON shown above. Shared config keys stay synced across plugin sections.</p>
|
||||
{% include "plugins/plugin_config_grid.html" with plugin_groups=adminform.form.plugin_groups %}
|
||||
</section>
|
||||
</fieldset>
|
||||
{% endblock %}
|
||||
@@ -147,11 +147,21 @@
|
||||
}
|
||||
.cards .card .card-meta {
|
||||
display: grid;
|
||||
grid-template-columns: 18px minmax(0, 1fr) auto;
|
||||
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
text-align: center;
|
||||
}
|
||||
.cards .card .card-meta .action-select {
|
||||
justify-self: start;
|
||||
}
|
||||
.cards .card .card-meta .card-date {
|
||||
justify-self: center;
|
||||
max-width: 100%;
|
||||
}
|
||||
.cards .card .card-meta .card-size {
|
||||
justify-self: end;
|
||||
}
|
||||
.cards .card .card-meta .card-date,
|
||||
.cards .card .card-meta .card-size {
|
||||
min-width: 0;
|
||||
@@ -249,16 +259,16 @@
|
||||
<code class="card-url" title="{{obj.url}}">{{obj.url}}</code>
|
||||
<div class="card-media">
|
||||
<a href="{% snapshot_base_url obj %}/index.html" class="card-thumbnail">
|
||||
{% snapshot_archiveresult_url obj 'screenshot' 'screenshot.png' as screenshot_url %}
|
||||
{% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot-1.png' as extension_screenshot_1_url %}
|
||||
{% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot.png' as extension_screenshot_url %}
|
||||
{% snapshot_archiveresult_url obj 'favicon' 'favicon.ico' as favicon_url %}
|
||||
{% admin_snapshot_archiveresult_url obj 'screenshot' 'screenshot.png' as screenshot_url %}
|
||||
{% admin_snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot-1.png' as extension_screenshot_1_url %}
|
||||
{% admin_snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot.png' as extension_screenshot_url %}
|
||||
{% admin_snapshot_archiveresult_url obj 'favicon' 'favicon.ico' as favicon_url %}
|
||||
{% if screenshot_url or extension_screenshot_1_url or extension_screenshot_url or favicon_url %}
|
||||
<img
|
||||
src="{% firstof screenshot_url extension_screenshot_1_url extension_screenshot_url favicon_url %}"
|
||||
alt="{{obj.title|default:'Not yet archived...'}}"
|
||||
data-fallbacks="{{ extension_screenshot_1_url }},{{ extension_screenshot_url }},{{ favicon_url }}"
|
||||
onerror="const fallbacks=(this.dataset.fallbacks || '').split(',').filter(Boolean); if (fallbacks.length) { this.dataset.fallbacks=fallbacks.slice(1).join(','); this.src=fallbacks[0]; } else { this.onerror=null; this.closest('.card-thumbnail').classList.add('missing'); this.remove(); }"
|
||||
onerror="const current=this.src; const fallbacks=(this.dataset.fallbacks || '').split(',').filter(Boolean).map(candidate => new URL(candidate, document.baseURI).href).filter((candidate, index, all) => candidate !== current && all.indexOf(candidate) === index); if (fallbacks.length) { this.dataset.fallbacks=fallbacks.slice(1).join(','); this.src=fallbacks[0]; } else { this.onerror=null; this.closest('.card-thumbnail').classList.add('missing'); this.remove(); }"
|
||||
/>
|
||||
{% elif obj.status == 'started' %}
|
||||
<img src="{% static 'spinner.gif' %}" class="loading-preview" alt="Archiving in progress"/>
|
||||
|
||||
@@ -287,7 +287,8 @@
|
||||
<code>daily</code> - Run once per day<br/>
|
||||
<code>weekly</code> - Run once per week<br/>
|
||||
<code>0 */6 * * *</code> - Every 6 hours (cron format)<br/>
|
||||
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)
|
||||
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)<br/>
|
||||
<a href="https://crontab.guru/" target="_blank" rel="noopener">Cron Format Helper</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
<div class="abx-question-options" id="archivebox-setup-tls-options">
|
||||
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="localhost"> <span><b>⭐ Localhost</b><small>No setup needed. HTTPS is not required for local access; all replay features remain available.</small></span></label>
|
||||
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="none"> <span><b>⚠️ No TLS, private networks only</b><small>Access ArchiveBox directly on a trusted LAN, VPN, or private network without an ingress service. Public servers require HTTPS. In-browser WARC viewing will be disabled unless using <code>localhost</code> or HTTPS.</small></span></label>
|
||||
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="single"> <span><b>⚠️ Single-domain HTTPS certificate</b><small>Configure one certificate for the <code>BASE_URL</code> hostname in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your platform's ingress UI, then proxy it to ArchiveBox on port <code>8000</code>. This mode is not allowed unless also using Single-domain DNS. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
|
||||
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="single"> <span><b>⚠️ Single-domain HTTPS certificate</b><small>Configure one certificate for the <code>BASE_URL</code> hostname in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your platform's ingress UI, then proxy it to ArchiveBox on port <code>5797</code>. This mode is not allowed unless also using Single-domain DNS. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
|
||||
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="wildcard"> <span><b>⭐ Wildcard TLS</b><small>Configure one certificate covering both the <code>BASE_URL</code> hostname and <code>*.BASE_URL</code>, normally through DNS-01. Never enable on-demand TLS or request individual certificates for snapshot subdomains. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
|
||||
</div>
|
||||
<div class="abx-question-status" id="archivebox-setup-tls-status">Choose how HTTPS will reach ArchiveBox.</div>
|
||||
|
||||
@@ -1987,7 +1987,7 @@ tbody .output-link:hover {opacity: 1;}
|
||||
}
|
||||
|
||||
.model-snapshot.change-list .cards .card .card-info.card-meta {
|
||||
grid-template-columns: 18px minmax(0, 1fr) auto;
|
||||
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.model-snapshot.change-list .cards .card .card-info.card-meta > a {
|
||||
@@ -2070,7 +2070,7 @@ tbody .output-link:hover {opacity: 1;}
|
||||
}
|
||||
|
||||
.model-snapshot.change-list .cards .card .card-info.card-meta {
|
||||
grid-template-columns: 18px minmax(0, 1fr) auto;
|
||||
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -195,9 +195,11 @@
|
||||
} else {
|
||||
risks.push('⚠️ UI, API, and archive replay share one origin. CSP disables risky archived scripts, but a CSP or content-type bypass could expose the archive index, other snapshots, saved headers, admin pages, API data, and canonical-host mutations.');
|
||||
}
|
||||
risks.push(publicIndexInput.checked
|
||||
? '⚠️ Anonymous visitors can enumerate public snapshot URLs and titles; saved URLs may contain private share tokens or other secrets.'
|
||||
: '✅ Anonymous visitors cannot browse the snapshot index.');
|
||||
if (!publicIndexInput.checked) {
|
||||
risks.push('✅ Anonymous visitors cannot browse the snapshot index.');
|
||||
} else {
|
||||
risks.push('⚠️ Anonymous visitors can browse the public snapshot index; saved URLs may contain private share tokens or other secrets.');
|
||||
}
|
||||
risks.push(!controlPlaneEnabled
|
||||
? '✅ URL submission and other state-changing requests are disabled for everyone in this replay-only mode.'
|
||||
: (publicAddInput.checked ? '⚠️ Anonymous visitors can submit malicious or private/intranet URLs. A filtering or per-crawl configuration bypass could expose internal content or threaten the server.' : '✅ Only signed-in admins can submit new URLs.'));
|
||||
@@ -234,7 +236,7 @@
|
||||
var exampleBaseUrl = desiredScheme + exampleBaseHost;
|
||||
var exampleWildcardUrl = desiredScheme + exampleWildcardHost;
|
||||
var exampleAdminUrl = desiredScheme + (dnsMode === 'wildcard' ? 'admin.' + exampleBaseHost : exampleBaseHost) + '/admin/';
|
||||
var localhostAdminUrl = 'http://admin.archivebox.localhost:8000/admin/';
|
||||
var localhostAdminUrl = 'http://admin.archivebox.localhost:5797/admin/';
|
||||
|
||||
document.getElementById('archivebox-setup-hosting-status').textContent = hosting === 'localhost'
|
||||
? '❌ Visit ' + localhostAdminUrl + ' from the same machine to continue setup.'
|
||||
@@ -253,9 +255,9 @@
|
||||
document.getElementById('archivebox-setup-tls-status').textContent = tlsMode === 'localhost'
|
||||
? '❌ Visit ' + localhostAdminUrl + ' from this machine to continue setup. No certificate is needed.'
|
||||
: (tlsMode === 'wildcard'
|
||||
? '❌ In your ingress provider, proxy to ArchiveBox on port 8000 and configure one browser-trusted certificate covering ' + exampleBaseHost + ' and ' + exampleWildcardHost + ', normally through DNS-01. Never enable on-demand TLS or request individual snapshot certificates. Visit ' + exampleAdminUrl + ' to continue setup.'
|
||||
? '❌ In your ingress provider, proxy to ArchiveBox on port 5797 and configure one browser-trusted certificate covering ' + exampleBaseHost + ' and ' + exampleWildcardHost + ', normally through DNS-01. Never enable on-demand TLS or request individual snapshot certificates. Visit ' + exampleAdminUrl + ' to continue setup.'
|
||||
: (tlsMode === 'single'
|
||||
? '❌ In your ingress provider, proxy to ArchiveBox on port 8000 and configure one browser-trusted certificate for ' + exampleBaseHost + '. Visit ' + exampleAdminUrl + ' to continue setup.'
|
||||
? '❌ In your ingress provider, proxy to ArchiveBox on port 5797 and configure one browser-trusted certificate for ' + exampleBaseHost + '. Visit ' + exampleAdminUrl + ' to continue setup.'
|
||||
: (tlsMode === 'none'
|
||||
? '❌ Expose this ArchiveBox server directly over HTTP without a separate ingress or SSL termination service. Visit ' + exampleAdminUrl + ' to continue setup. In-browser WARC viewing will remain disabled unless browsing through localhost or HTTPS.'
|
||||
: '❌ Choose an ingress and TLS mode. Your choice will not be changed automatically.')));
|
||||
|
||||
@@ -1164,9 +1164,9 @@ def run_python_cwd(
|
||||
# Server/API Integration Helpers
|
||||
# =============================================================================
|
||||
|
||||
API_TEST_HOST = "api.archivebox.localhost:8000"
|
||||
ADMIN_TEST_HOST = "admin.archivebox.localhost:8000"
|
||||
WEB_TEST_HOST = "web.archivebox.localhost:8000"
|
||||
API_TEST_HOST = "api.archivebox.localhost:5797"
|
||||
ADMIN_TEST_HOST = "admin.archivebox.localhost:5797"
|
||||
WEB_TEST_HOST = "web.archivebox.localhost:5797"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -1794,8 +1794,8 @@ def real_archive_with_example(tmp_path_factory, request):
|
||||
[
|
||||
"config",
|
||||
"--set",
|
||||
"BIND_ADDR=127.0.0.1:8000",
|
||||
"BASE_URL=http://archivebox.localhost:8000",
|
||||
"BIND_ADDR=127.0.0.1:5797",
|
||||
"BASE_URL=http://archivebox.localhost:5797",
|
||||
"PUBLIC_INDEX=True",
|
||||
"PUBLIC_ADD_VIEW=True",
|
||||
],
|
||||
|
||||
@@ -67,3 +67,68 @@ def test_archiveresult_patch_upload_finalizes_queued_result(client, api_admin_us
|
||||
assert result.output_str == "output.html"
|
||||
assert snapshot.status == Snapshot.StatusChoices.SEALED
|
||||
assert snapshot.retry_at is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("chunked", [False, True])
|
||||
@pytest.mark.parametrize("method", ["POST", "PATCH"])
|
||||
def test_live_archiveresult_multipart_upload(tmp_path, chunked, method):
|
||||
import requests
|
||||
|
||||
from archivebox.tests.conftest import (
|
||||
api_auth_headers,
|
||||
cli_env,
|
||||
create_admin_and_token,
|
||||
get_free_port,
|
||||
init_archive,
|
||||
live_api_request,
|
||||
stop_archivebox_process,
|
||||
)
|
||||
from archivebox.tests.test_api_v1_cli_add import start_api_server_without_runner
|
||||
|
||||
init_archive(tmp_path)
|
||||
token = create_admin_and_token(tmp_path)
|
||||
port = get_free_port()
|
||||
server = start_api_server_without_runner(tmp_path, cli_env(port=port, server=True), port)
|
||||
try:
|
||||
snapshot = live_api_request(
|
||||
port,
|
||||
"post",
|
||||
"/api/v1/core/snapshots",
|
||||
api_token=token,
|
||||
json={"url": "https://example.com/multipart", "status": "sealed"},
|
||||
)
|
||||
assert snapshot.status_code == 200, snapshot.text
|
||||
result = live_api_request(
|
||||
port,
|
||||
"post",
|
||||
"/api/v1/core/archiveresults",
|
||||
api_token=token,
|
||||
data={"snapshot_id": snapshot.json()["id"], "plugin": "dom", "status": "started"},
|
||||
)
|
||||
assert result.status_code == 200, result.text
|
||||
payload = b"<html>Real multipart upload</html>"
|
||||
body = encode_multipart(
|
||||
BOUNDARY,
|
||||
{
|
||||
"files": SimpleUploadedFile("output.html", payload, content_type="text/html"),
|
||||
"output_paths": "output.html",
|
||||
"status": "succeeded",
|
||||
"snapshot_id": snapshot.json()["id"],
|
||||
"plugin": "dom",
|
||||
},
|
||||
)
|
||||
path = "/api/v1/core/archiveresults" if method == "POST" else f"/api/v1/core/archiveresult/{result.json()['id']}"
|
||||
response = requests.request(
|
||||
method,
|
||||
f"http://127.0.0.1:{port}{path}",
|
||||
data=iter([body]) if chunked else body,
|
||||
headers={**api_auth_headers(token, port=port), "Content-Type": MULTIPART_CONTENT},
|
||||
timeout=30,
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["status"] == "succeeded"
|
||||
files = list((tmp_path / "archive").glob("users/*/snapshots/*/*/*/dom/output.html"))
|
||||
assert len(files) == 1
|
||||
assert files[0].read_bytes() == payload
|
||||
finally:
|
||||
stop_archivebox_process(server)
|
||||
@@ -15,6 +15,79 @@ from archivebox.tests.conftest import api_client_request
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("plugin", [".", "..", "/./", "/../", " . ", " ../ "])
|
||||
def test_archiveresult_upload_rejects_dot_directory_names(client, api_admin_user, api_headers, plugin):
|
||||
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
|
||||
snapshot = Snapshot.objects.create(url="https://example.com/invalid-upload-directory", crawl=crawl)
|
||||
snapshot_dir = snapshot.output_dir
|
||||
filename = "rejected-upload.txt"
|
||||
|
||||
response = client.post(
|
||||
"/api/v1/core/archiveresults",
|
||||
{
|
||||
"snapshot_id": str(snapshot.id),
|
||||
"plugin": plugin,
|
||||
"files": SimpleUploadedFile(filename, b"must not be written", content_type="text/plain"),
|
||||
},
|
||||
**api_headers,
|
||||
)
|
||||
|
||||
assert response.status_code == 400, response.content
|
||||
assert response.json()["detail"] == "Invalid ArchiveResult plugin name"
|
||||
assert not snapshot.archiveresult_set.exists()
|
||||
assert not (snapshot_dir / filename).exists()
|
||||
assert not (snapshot_dir.parent / filename).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("plugin", [".", ".."])
|
||||
def test_archiveresult_patch_rejects_existing_dot_directory_names(client, api_admin_user, api_headers, plugin):
|
||||
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
|
||||
snapshot = Snapshot.objects.create(url="https://example.com/invalid-existing-directory", crawl=crawl)
|
||||
result = ArchiveResult.objects.create(snapshot=snapshot, plugin=plugin)
|
||||
snapshot_dir = snapshot.output_dir
|
||||
filename = "rejected-patch.txt"
|
||||
|
||||
response = client.patch(
|
||||
f"/api/v1/core/archiveresult/{result.id}",
|
||||
encode_multipart(
|
||||
BOUNDARY,
|
||||
{"files": SimpleUploadedFile(filename, b"must not be written", content_type="text/plain")},
|
||||
),
|
||||
content_type=MULTIPART_CONTENT,
|
||||
**api_headers,
|
||||
)
|
||||
|
||||
assert response.status_code == 400, response.content
|
||||
assert response.json()["detail"] == "Invalid ArchiveResult plugin name"
|
||||
result.refresh_from_db()
|
||||
assert result.output_files == {}
|
||||
assert not (snapshot_dir / filename).exists()
|
||||
assert not (snapshot_dir.parent / filename).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("plugin", ["custom.files", "custom-files_v1"])
|
||||
def test_archiveresult_upload_accepts_unregistered_directory_names(client, api_admin_user, api_headers, plugin):
|
||||
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
|
||||
snapshot = Snapshot.objects.create(url="https://example.com/custom-upload-directory", crawl=crawl)
|
||||
contents = b"uploaded original file"
|
||||
|
||||
response = client.post(
|
||||
"/api/v1/core/archiveresults",
|
||||
{
|
||||
"snapshot_id": str(snapshot.id),
|
||||
"plugin": plugin,
|
||||
"files": SimpleUploadedFile("original.txt", contents, content_type="text/plain"),
|
||||
},
|
||||
**api_headers,
|
||||
)
|
||||
|
||||
assert response.status_code == 200, response.content
|
||||
result = snapshot.archiveresult_set.get()
|
||||
assert result.plugin == plugin
|
||||
assert result.output_files["original.txt"]["size"] == len(contents)
|
||||
assert (snapshot.output_dir / plugin / "original.txt").read_bytes() == contents
|
||||
|
||||
|
||||
def test_archiveresult_upload_upserts_by_snapshot_plugin_and_hook(client, api_admin_user, api_headers):
|
||||
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
|
||||
snapshot = Snapshot.objects.create(url="https://example.com/unified-result", crawl=crawl)
|
||||
|
||||
@@ -11,7 +11,7 @@ pytestmark = pytest.mark.django_db
|
||||
|
||||
|
||||
User = get_user_model()
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -12,7 +12,7 @@ def test_basic_success_case_request(client, tmp_path, api_admin_user, api_token)
|
||||
response = client.get(
|
||||
"/api/v1/core/tags/autocomplete/",
|
||||
{"q": "api-basic", "api_key": api_token.token},
|
||||
HTTP_HOST="api.archivebox.localhost:8000",
|
||||
HTTP_HOST="api.archivebox.localhost:5797",
|
||||
)
|
||||
|
||||
assert response.status_code == 200, response.content
|
||||
@@ -33,7 +33,7 @@ from .conftest import (
|
||||
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
User = get_user_model()
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -16,6 +16,23 @@ def test_persona_help_runs_successfully(tmp_path):
|
||||
assert "list" in result.stdout
|
||||
|
||||
|
||||
def test_persona_create_permissions_preserves_existing_persona(initialized_archive):
|
||||
from archivebox.tests.conftest import run_python_cwd
|
||||
|
||||
result = run_archivebox_cmd(["persona", "create", "--permissions=private", "host-profile"], cwd=initialized_archive)
|
||||
assert result.returncode == 0, result.stderr
|
||||
existing = run_archivebox_cmd(["persona", "create", "--permissions=public", "host-profile"], cwd=initialized_archive)
|
||||
assert existing.returncode == 0, existing.stderr
|
||||
stdout, stderr, returncode = run_python_cwd(
|
||||
"import os; os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.core.settings'); "
|
||||
"import django; django.setup(); from archivebox.personas.models import Persona; "
|
||||
"p = Persona.objects.get(name='host-profile'); assert p.permissions == 'private'; print('private')",
|
||||
cwd=initialized_archive,
|
||||
)
|
||||
assert returncode == 0, stderr
|
||||
assert "private" in stdout
|
||||
|
||||
|
||||
def test_persona_import_missing_source_is_actionable_and_does_not_create(initialized_archive, tmp_path):
|
||||
result = run_archivebox_cmd(
|
||||
["persona", "create", "--import=brave", "--source", str(tmp_path / "missing"), "broken"],
|
||||
|
||||
@@ -31,6 +31,45 @@ RUN_TEST_ENV = {
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
def test_cli_run_retries_snapshot_after_filesystem_failure(initialized_archive):
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.tests.test_orm_helpers import use_archivebox_db
|
||||
|
||||
env = cli_env(live=True, PLUGINS="hashes")
|
||||
created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
|
||||
assert created.returncode == 0, created.stdout + created.stderr
|
||||
snapshot_id = next(record["id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
|
||||
with use_archivebox_db(initialized_archive):
|
||||
snapshot = Snapshot.objects.get(pk=snapshot_id)
|
||||
crawl_id = str(snapshot.crawl_id)
|
||||
directory = snapshot.output_dir
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
(directory / "input.txt").write_text("Capture survives a temporary filesystem failure")
|
||||
broken_directory = directory / "hashes"
|
||||
broken_directory.symlink_to("hashes")
|
||||
|
||||
failed = run_archivebox_cmd(["run", f"--snapshot-id={snapshot_id}"], cwd=initialized_archive, env=env, timeout=60)
|
||||
assert failed.returncode == 1, failed.stdout + failed.stderr
|
||||
assert "FileExistsError: [Errno 17]" in failed.stderr
|
||||
with use_archivebox_db(initialized_archive):
|
||||
snapshot = Snapshot.objects.get(pk=snapshot_id)
|
||||
assert snapshot.status == Snapshot.StatusChoices.STARTED
|
||||
assert snapshot.retry_at is not None
|
||||
assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
|
||||
|
||||
broken_directory.unlink()
|
||||
resumed = run_archivebox_cmd(["run", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=60)
|
||||
assert resumed.returncode == 0, resumed.stdout + resumed.stderr
|
||||
with use_archivebox_db(initialized_archive):
|
||||
snapshot = Snapshot.objects.get(pk=snapshot_id)
|
||||
assert snapshot.status == Snapshot.StatusChoices.SEALED
|
||||
result = snapshot.archiveresult_set.get(plugin="hashes")
|
||||
assert result.status == ArchiveResult.StatusChoices.SUCCEEDED
|
||||
assert result.output_size > 0
|
||||
assert (directory / "hashes").is_dir()
|
||||
|
||||
|
||||
def _install_real_chrome_for_test(data_dir, env, *, isolation):
|
||||
env["CHROME_ISOLATION"] = isolation
|
||||
env["CHROME_HEADLESS"] = "true"
|
||||
@@ -44,6 +83,52 @@ def _install_real_chrome_for_test(data_dir, env, *, isolation):
|
||||
assert install_process.returncode == 0, install_process.stderr or install_process.stdout
|
||||
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
@pytest.mark.parametrize("precreate_snapshot", [False, True])
|
||||
def test_newest_crawl_runs_before_older_queued_and_started_crawls(initialized_archive, precreate_snapshot):
|
||||
from datetime import timedelta
|
||||
|
||||
from django.utils import timezone
|
||||
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.tests.test_orm_helpers import use_archivebox_db
|
||||
|
||||
env = cli_env(live=True, PLUGINS="hashes")
|
||||
old_url, queued_url, newest_url = create_test_url(), create_test_url(), create_test_url()
|
||||
old = run_archivebox_cmd(["snapshot", "create", old_url], cwd=initialized_archive, env=env)
|
||||
assert old.returncode == 0, old.stdout + old.stderr
|
||||
with use_archivebox_db(initialized_archive):
|
||||
snapshot = Snapshot.objects.get(url=old_url)
|
||||
Crawl.objects.filter(pk=snapshot.crawl_id).update(status=Crawl.StatusChoices.STARTED)
|
||||
Snapshot.objects.filter(pk=snapshot.pk).update(retry_at=timezone.now() - timedelta(days=1))
|
||||
queued = run_archivebox_cmd(
|
||||
["snapshot" if precreate_snapshot else "crawl", "create", queued_url],
|
||||
cwd=initialized_archive,
|
||||
env=env,
|
||||
)
|
||||
assert queued.returncode == 0, queued.stdout + queued.stderr
|
||||
newest = run_archivebox_cmd(
|
||||
["snapshot" if precreate_snapshot else "crawl", "create", newest_url],
|
||||
cwd=initialized_archive,
|
||||
env=env,
|
||||
)
|
||||
assert newest.returncode == 0, newest.stdout + newest.stderr
|
||||
|
||||
result = run_archivebox_cmd(["run", "--no-stdin"], cwd=initialized_archive, env=env, timeout=60)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert all(url in result.stdout for url in (old_url, queued_url, newest_url))
|
||||
assert result.stdout.index(newest_url) < result.stdout.index(queued_url) < result.stdout.index(old_url), result.stdout
|
||||
with use_archivebox_db(initialized_archive):
|
||||
assert (
|
||||
Snapshot.objects.filter(
|
||||
url__in=[old_url, queued_url, newest_url],
|
||||
status=Snapshot.StatusChoices.SEALED,
|
||||
).count()
|
||||
== 3
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
@pytest.mark.timeout(660)
|
||||
def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archive, recursive_test_site):
|
||||
@@ -2096,7 +2181,7 @@ class TestRecoverOrchestratorStateRedFailureModes:
|
||||
stdout="\n".join(json.dumps(record) for record in records),
|
||||
)
|
||||
|
||||
recovered = recover_orchestrator_state()
|
||||
recovered = recover_orchestrator_state(crawl_id=str(crawl.id))
|
||||
|
||||
assert recovered["archiveresults_missing_for_orphaned_hook_processes"] == 1
|
||||
assert ArchiveResult.objects.filter(snapshot=snapshot, plugin="title", hook_name=hook_name).count() == 1
|
||||
@@ -2108,6 +2193,54 @@ class TestRecoverOrchestratorStateRedFailureModes:
|
||||
assert result.end_ts == newer_start + timedelta(seconds=1)
|
||||
assert result.process.started_at == newer_start
|
||||
|
||||
def test_global_recovery_skips_historical_hook_output_scan(self, tmp_path):
|
||||
from datetime import timedelta
|
||||
|
||||
from django.utils import timezone
|
||||
|
||||
from archivebox.base_models.models import get_or_create_system_user_pk
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.core.recovery_util import recover_orchestrator_state
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.machine.models import Machine, NetworkInterface, Process
|
||||
|
||||
future = timezone.now() + timedelta(days=1)
|
||||
crawl = Crawl.objects.create(
|
||||
urls="https://example.com/old\nhttps://example.com/new",
|
||||
created_by_id=get_or_create_system_user_pk(),
|
||||
status=Crawl.StatusChoices.STARTED,
|
||||
retry_at=future,
|
||||
)
|
||||
snapshot = Snapshot.objects.create(
|
||||
url="https://example.com/old",
|
||||
crawl=crawl,
|
||||
status=Snapshot.StatusChoices.STARTED,
|
||||
retry_at=future,
|
||||
)
|
||||
historical_plugin_dir = tmp_path / str(snapshot.id) / "title"
|
||||
historical_plugin_dir.mkdir(parents=True)
|
||||
(historical_plugin_dir / "title.txt").write_text("preserved historical output")
|
||||
Process.objects.create(
|
||||
machine=Machine.current(refresh=True),
|
||||
iface=NetworkInterface.current(refresh=True),
|
||||
process_type=Process.TypeChoices.HOOK,
|
||||
worker_type="archiveresult",
|
||||
pwd=str(historical_plugin_dir),
|
||||
cmd=["on_Snapshot__01_title.py"],
|
||||
status=Process.StatusChoices.EXITED,
|
||||
retry_at=None,
|
||||
exit_code=0,
|
||||
started_at=timezone.now() - timedelta(days=30),
|
||||
ended_at=timezone.now() - timedelta(days=30) + timedelta(seconds=1),
|
||||
)
|
||||
|
||||
recovered = recover_orchestrator_state()
|
||||
|
||||
assert recovered["snapshots_started_without_running_results"] == 1
|
||||
assert recovered["archiveresults_missing_for_orphaned_hook_processes"] == 0
|
||||
assert Snapshot.objects.filter(crawl=crawl, retry_at__lte=timezone.now()).count() == 1
|
||||
assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
|
||||
|
||||
def test_recovery_does_not_seal_queued_snapshot_waiting_for_future_retry_even_with_final_results(self):
|
||||
from datetime import timedelta
|
||||
|
||||
|
||||
@@ -351,12 +351,42 @@ def test_runtime_binary_wins_over_ambient_system_path(tmp_path):
|
||||
def test_daphne_worker_uses_default_application_close_timeout():
|
||||
from archivebox.workers.supervisord_util import SERVER_WORKER
|
||||
|
||||
command = SERVER_WORKER("127.0.0.1", "8000")["command"]
|
||||
command = SERVER_WORKER("127.0.0.1", "5797")["command"]
|
||||
|
||||
assert "daphne" in command
|
||||
assert "--application-close-timeout=0" not in command
|
||||
|
||||
|
||||
@pytest.mark.parametrize("in_docker,port", [(False, "5797"), (False, "8000"), (True, "9579")])
|
||||
@pytest.mark.parametrize("debug", [False, True])
|
||||
def test_port_compatibility_does_not_add_listeners_outside_docker_defaults(tmp_path, in_docker, port, debug):
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"from archivebox.config.common import get_config;"
|
||||
"from archivebox.workers.supervisord_util import build_server_worker_plan;"
|
||||
"import json,sys;"
|
||||
"config=get_config();"
|
||||
"workers,_,_=build_server_worker_plan(config=config,host='127.0.0.1',"
|
||||
"port=sys.argv[1],debug=sys.argv[2]=='True',reload=False,nothreading=False);"
|
||||
"print(json.dumps([worker for worker,lazy in workers]))",
|
||||
port,
|
||||
str(debug),
|
||||
],
|
||||
cwd=tmp_path,
|
||||
env=cli_env(IN_DOCKER=str(in_docker), SEARCH_BACKEND_SONIC_ENABLED="False", DISPLAY="", ARCHIVEBOX_VNC_PERSONA=""),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
workers = json.loads(result.stdout)
|
||||
assert [worker["name"] for worker in workers] == ["worker_runserver" if debug else "worker_daphne", "worker_runner"]
|
||||
command = shlex.split(workers[0]["command"])
|
||||
assert "--endpoint" not in command
|
||||
assert (f"127.0.0.1:{port}" if debug else f"--port={port}") in command
|
||||
|
||||
|
||||
def test_supervisord_parent_watchdog_does_not_start_another_archivebox_runtime():
|
||||
from archivebox.workers import supervisord_util
|
||||
|
||||
@@ -401,17 +431,17 @@ def test_server_worker_memory_preflight_fails_before_starting_supervisord(capsys
|
||||
def test_reload_workers_use_active_archivebox_module():
|
||||
from archivebox.workers.supervisord_util import RUNNER_WATCH_WORKER, RUNSERVER_WORKER, archivebox_cmd
|
||||
|
||||
runserver = RUNSERVER_WORKER("127.0.0.1", "8000", reload=True)
|
||||
watcher = RUNNER_WATCH_WORKER("http://127.0.0.1:8000")
|
||||
runserver = RUNSERVER_WORKER("127.0.0.1", "5797", reload=True)
|
||||
watcher = RUNNER_WATCH_WORKER("http://127.0.0.1:5797")
|
||||
|
||||
assert runserver["name"] == "worker_runserver"
|
||||
assert shlex.split(runserver["command"]) == archivebox_cmd("manage", "runserver", "127.0.0.1:8000")
|
||||
assert shlex.split(runserver["command"]) == archivebox_cmd("manage", "runserver", "127.0.0.1:5797")
|
||||
assert 'ARCHIVEBOX_RUNSERVER="1"' in runserver["environment"]
|
||||
assert 'ARCHIVEBOX_AUTORELOAD="1"' in runserver["environment"]
|
||||
assert 'ARCHIVEBOX_RUNSERVER_BIND_URL="http://127.0.0.1:8000"' in runserver["environment"]
|
||||
assert 'ARCHIVEBOX_RUNSERVER_BIND_URL="http://127.0.0.1:5797"' in runserver["environment"]
|
||||
|
||||
assert watcher["name"] == "worker_runner_watch"
|
||||
assert shlex.split(watcher["command"]) == archivebox_cmd("manage", "runner_watch", "--bind-url=http://127.0.0.1:8000")
|
||||
assert shlex.split(watcher["command"]) == archivebox_cmd("manage", "runner_watch", "--bind-url=http://127.0.0.1:5797")
|
||||
|
||||
|
||||
def test_server_daemon_starts_real_plugin_owned_sonic_worker(initialized_archive, archivebox_daemon_server):
|
||||
|
||||
@@ -5,8 +5,13 @@ Verify update drains old dirs, reconciles DB, and queues snapshots.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import shutil
|
||||
import signal
|
||||
import json
|
||||
import uuid
|
||||
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.tests.conftest import run_queued_crawls, run_archivebox_cmd, cli_env
|
||||
|
||||
from archivebox.tests.test_orm_helpers import use_archivebox_db
|
||||
@@ -14,6 +19,131 @@ from archivebox.tests.test_orm_helpers import use_archivebox_db
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
|
||||
def test_rescan_imports_current_layout_and_is_idempotent(initialized_archive, tmp_path):
|
||||
env = cli_env(disable_extractors=True)
|
||||
run_archivebox_cmd(["add", "--index-only", "https://example.com/rescan"], cwd=initialized_archive, env=env, check=True)
|
||||
run_queued_crawls(initialized_archive, env)
|
||||
run_archivebox_cmd(["update", "--migrate-only"], cwd=initialized_archive, env=env, check=True)
|
||||
with use_archivebox_db(initialized_archive):
|
||||
original = Snapshot.objects.get(url="https://example.com/rescan")
|
||||
original.title = "Imported snapshot"
|
||||
original.save()
|
||||
output = original.output_dir / "singlefile" / "singlefile.html"
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text("<html><title>Portable archive fixture</title></html>")
|
||||
result = ArchiveResult.objects.create(
|
||||
snapshot=original,
|
||||
plugin="singlefile",
|
||||
hook_name="on_Snapshot__singlefile",
|
||||
status="succeeded",
|
||||
output_str="singlefile/singlefile.html",
|
||||
)
|
||||
result.update_output_metadata_from_filesystem(full_scan=True)
|
||||
result.refresh_from_db()
|
||||
expected_result = (result.pk, result.created_at, result.modified_at, result.output_files, result.output_size)
|
||||
original.refresh_from_db()
|
||||
original.write_index_jsonl()
|
||||
index = original.output_dir / "index.jsonl"
|
||||
historical = {**result.to_json(), "id": uuid.uuid4().hex}
|
||||
index.write_text(json.dumps(historical) + "\n" + index.read_text())
|
||||
snapshot_id = original.pk
|
||||
original_fields = (original.crawl_id, original.created_at, original.modified_at, original.bookmarked_at, original.timestamp)
|
||||
original.crawl.refresh_from_db()
|
||||
crawl_dates = (original.crawl.created_at, original.crawl.modified_at)
|
||||
relative_path = original.output_dir.relative_to(initialized_archive)
|
||||
|
||||
destination = tmp_path / "destination"
|
||||
destination.mkdir()
|
||||
run_archivebox_cmd(["init", "--quick"], cwd=destination, env=env, check=True)
|
||||
shutil.copytree(initialized_archive / "archive", destination / "archive", dirs_exist_ok=True, symlinks=True)
|
||||
result = run_archivebox_cmd(["update", "--migrate-only", "--index-only"], cwd=destination, env=env, timeout=120)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
with use_archivebox_db(destination):
|
||||
assert not Snapshot.objects.filter(pk=snapshot_id).exists()
|
||||
|
||||
for expected in ("Imported: 1", "Unchanged: 1"):
|
||||
result = run_archivebox_cmd(["update", "--rescan", "--migrate-only"], cwd=destination, env=env, timeout=120)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert expected in result.stdout
|
||||
with use_archivebox_db(destination):
|
||||
imported = Snapshot.objects.get(pk=snapshot_id)
|
||||
assert imported.title == "Imported snapshot"
|
||||
assert imported.status == "sealed"
|
||||
assert imported.output_dir.relative_to(CONSTANTS.DATA_DIR) == relative_path
|
||||
assert (destination / relative_path / "index.jsonl").is_file()
|
||||
assert (
|
||||
imported.crawl_id,
|
||||
imported.created_at,
|
||||
imported.modified_at,
|
||||
imported.bookmarked_at,
|
||||
imported.timestamp,
|
||||
) == original_fields
|
||||
assert (imported.crawl.created_at, imported.crawl.modified_at) == crawl_dates
|
||||
assert Snapshot.objects.count() == 1
|
||||
imported_result = imported.archiveresult_set.get()
|
||||
assert (
|
||||
imported_result.pk,
|
||||
imported_result.created_at,
|
||||
imported_result.modified_at,
|
||||
imported_result.output_files,
|
||||
imported_result.output_size,
|
||||
) == expected_result
|
||||
assert (destination / relative_path / "singlefile" / "singlefile.html").read_text() == output.read_text()
|
||||
|
||||
copied_index = destination / relative_path / "index.jsonl"
|
||||
records = [json.loads(line) for line in copied_index.read_text().splitlines()]
|
||||
for record in records:
|
||||
if record.get("type") == "Snapshot":
|
||||
record["url"] = "https://example.com/conflicting-identity"
|
||||
conflicting_bytes = "\n".join(json.dumps(record) for record in records) + "\n"
|
||||
copied_index.write_text(conflicting_bytes)
|
||||
result = run_archivebox_cmd(["update", "--rescan"], cwd=destination, env=env)
|
||||
assert result.returncode == 1, result.stdout + result.stderr
|
||||
assert "conflicting identity" in result.stdout
|
||||
assert copied_index.read_text() == conflicting_bytes
|
||||
with use_archivebox_db(destination):
|
||||
assert Snapshot.objects.get(pk=snapshot_id).url == "https://example.com/rescan"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reverse", [False, True])
|
||||
def test_rescan_resumes_after_real_interrupt_in_requested_order(initialized_archive, tmp_path, reverse):
|
||||
env = cli_env(disable_extractors=True)
|
||||
urls = [f"https://example.com/rescan-interrupt/{number}" for number in range(30)]
|
||||
run_archivebox_cmd(["add", "--index-only", *urls], cwd=initialized_archive, env=env, check=True)
|
||||
run_queued_crawls(initialized_archive, env)
|
||||
run_archivebox_cmd(["update", "--migrate-only"], cwd=initialized_archive, env=env, check=True)
|
||||
with use_archivebox_db(initialized_archive):
|
||||
originals = list(Snapshot.objects.order_by("id" if reverse else "-id"))
|
||||
expected = [(snapshot.pk, snapshot.created_at, snapshot.crawl_id) for snapshot in originals]
|
||||
for snapshot in originals:
|
||||
snapshot.write_index_jsonl()
|
||||
destination = tmp_path / "destination"
|
||||
destination.mkdir()
|
||||
run_archivebox_cmd(["init", "--quick"], cwd=destination, env=env, check=True)
|
||||
shutil.copytree(initialized_archive / "archive", destination / "archive", dirs_exist_ok=True, symlinks=True)
|
||||
args = ["update", "--rescan", *(["--reverse"] if reverse else [])]
|
||||
process = run_archivebox_cmd(args, cwd=destination, env=env, wait=False)
|
||||
try:
|
||||
for line in process.stdout:
|
||||
if "Imported:" in line:
|
||||
process.send_signal(signal.SIGINT)
|
||||
break
|
||||
stdout, stderr = process.communicate(timeout=60)
|
||||
assert process.returncode == 130, stdout + stderr
|
||||
finally:
|
||||
if process.poll() is None:
|
||||
process.terminate()
|
||||
process.wait(timeout=60)
|
||||
with use_archivebox_db(destination):
|
||||
partial = list(Snapshot.objects.order_by("id" if reverse else "-id").values_list("pk", "created_at", "crawl_id"))
|
||||
assert 0 < len(partial) < len(expected)
|
||||
assert partial == expected[: len(partial)]
|
||||
result = run_archivebox_cmd(args, cwd=destination, env=env, timeout=120)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
with use_archivebox_db(destination):
|
||||
assert list(Snapshot.objects.order_by("id" if reverse else "-id").values_list("pk", "created_at", "crawl_id")) == expected
|
||||
|
||||
|
||||
def test_update_runs_successfully_on_empty_archive(initialized_archive):
|
||||
"""Test that update runs without error on empty archive."""
|
||||
result = run_archivebox_cmd(
|
||||
@@ -23,10 +153,9 @@ def test_update_runs_successfully_on_empty_archive(initialized_archive):
|
||||
output = result.stdout + result.stderr
|
||||
|
||||
assert result.returncode == 0, output
|
||||
assert "Phase 1: Draining old archive/ directories" in output
|
||||
assert "Phase 2: Selecting database snapshots with stale filesystem versions" in output
|
||||
assert "Updated DB rows: 0" in output
|
||||
assert "Sealed crawls: 0" in output
|
||||
assert "Filesystem migrations remain lazy" in output
|
||||
assert "Phase 1" not in output
|
||||
assert "Phase 2" not in output
|
||||
|
||||
with use_archivebox_db(initialized_archive):
|
||||
assert Snapshot.objects.count() == 0
|
||||
@@ -52,8 +181,8 @@ def test_update_reconciles_existing_snapshots(initialized_archive):
|
||||
|
||||
output = result.stdout + result.stderr
|
||||
assert result.returncode == 0, output
|
||||
assert "Phase 2: Selecting database snapshots with stale filesystem versions" in output
|
||||
assert "Updated DB rows:" in output
|
||||
assert "Filesystem migrations remain lazy" in output
|
||||
assert "Updated DB rows:" not in output
|
||||
|
||||
with use_archivebox_db(initialized_archive):
|
||||
assert Snapshot.objects.filter(url="https://example.com", status="sealed").count() == 1
|
||||
|
||||
@@ -35,9 +35,9 @@ def test_update_imports_orphaned_snapshots(tmp_path, initialized_archive):
|
||||
),
|
||||
)
|
||||
|
||||
# Run the migration phase only; default update also runs queued crawl work.
|
||||
# Filesystem-only discovery is explicit; default update stays lazy.
|
||||
update_process = run_archivebox_cmd(
|
||||
["update", "--migrate-only"],
|
||||
["update", "--rescan", "--migrate-only"],
|
||||
env=env,
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
@@ -10,8 +10,8 @@ from archivebox.tests.conftest import run_archivebox_cmd, run_queued_crawls, cli
|
||||
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
API_HOST = "api.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
API_HOST = "api.archivebox.localhost:5797"
|
||||
|
||||
|
||||
def test_delete_after_real_cli_and_orchestrator_paths_cover_all_retained_models(tmp_path, recursive_test_site):
|
||||
|
||||
@@ -620,7 +620,7 @@ class TestProcessCurrent:
|
||||
"""runner_watch should be classified as a worker, not the orchestrator itself."""
|
||||
old_argv = sys.argv
|
||||
try:
|
||||
sys.argv = ["archivebox", "manage", "runner_watch", "--bind-url=http://127.0.0.1:8000"]
|
||||
sys.argv = ["archivebox", "manage", "runner_watch", "--bind-url=http://127.0.0.1:5797"]
|
||||
result = Process._detect_process_type()
|
||||
assert result == Process.TypeChoices.WORKER
|
||||
finally:
|
||||
|
||||
@@ -85,8 +85,11 @@ def test_oldest_django_collection_migrates_end_to_end_without_data_loss(tmp_path
|
||||
result = run_archivebox_migration_cmd(tmp_path, ["init"], timeout=90)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "received a naive datetime" not in result.stderr
|
||||
result = run_archivebox_migration_cmd(tmp_path, ["update"], timeout=60)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert all((tmp_path / "archive" / timestamp).exists() for timestamp in original_trees)
|
||||
for pass_number in (1, 2):
|
||||
result = run_archivebox_migration_cmd(tmp_path, ["update"], timeout=180)
|
||||
result = run_archivebox_migration_cmd(tmp_path, ["update", "--migrate-only"], timeout=180)
|
||||
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
|
||||
|
||||
for command in (["status"], ["list", "--json"]):
|
||||
|
||||
@@ -178,7 +178,7 @@ def test_legacy_onedomain_server_upgrade_works_in_auto_mode(archive_07):
|
||||
"""[SERVER_CONFIG]
|
||||
LISTEN_HOST = archivebox.mydomain
|
||||
ALLOWED_HOSTS = archivebox.mydomain
|
||||
CSRF_TRUSTED_ORIGINS = http://admin.archivebox.localhost:8000
|
||||
CSRF_TRUSTED_ORIGINS = http://admin.archivebox.localhost:5797
|
||||
PUBLIC_INDEX = True
|
||||
PUBLIC_SNAPSHOTS = True
|
||||
PUBLIC_ADD_VIEW = False
|
||||
|
||||
@@ -720,7 +720,7 @@ def test_update_migrates_db_snapshot_when_legacy_index_missing(tmp_path):
|
||||
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
|
||||
assert result.returncode == 0, f"Init failed: {result.stderr}"
|
||||
for pass_number in (1, 2):
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
|
||||
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
|
||||
|
||||
migrated_files = list((work_dir / "archive" / "users").glob("*/snapshots/*/*/*/screenshot.png"))
|
||||
@@ -749,7 +749,7 @@ def test_update_recovers_orphan_with_corrupt_index_from_archive_org_url(tmp_path
|
||||
|
||||
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
|
||||
assert result.returncode == 0, f"Init failed: {result.stderr}"
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
|
||||
assert result.returncode == 0, f"Update failed: {result.stderr}"
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
@@ -792,7 +792,7 @@ def test_update_preserves_legacy_folder_timestamp_over_index_float_variant(tmp_p
|
||||
|
||||
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
|
||||
assert result.returncode == 0, f"Init failed: {result.stderr}"
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
|
||||
assert result.returncode == 0, f"Update failed: {result.stderr}"
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
@@ -834,7 +834,7 @@ def test_update_preserves_distinct_legacy_dirs_with_integer_and_float_timestamps
|
||||
|
||||
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
|
||||
assert result.returncode == 0, f"Init failed: {result.stderr}"
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
|
||||
assert result.returncode == 0, f"Update failed: {result.stderr}"
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
@@ -885,7 +885,7 @@ def test_update_preserves_legacy_plugin_directory_without_output_files(migration
|
||||
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
|
||||
assert result.returncode == 0, f"Init failed: {result.stderr}"
|
||||
for pass_number in (1, 2):
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
|
||||
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
|
||||
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
|
||||
|
||||
migrated_outputs = list((work_dir / "archive" / "users").glob("*/snapshots/*/*/*/media/stderr.log"))
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from concurrent.futures import TimeoutError as FutureTimeoutError
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from urllib.parse import parse_qs, quote, urlsplit
|
||||
@@ -92,9 +94,9 @@ def installed_opencode(opencode_archive_config):
|
||||
|
||||
config = get_config().model_dump(mode="json")
|
||||
settings = runtime._settings(config, opencode_archive_config.data_dir)
|
||||
settings["archivebox_base_url"] = "http://admin.archivebox.localhost:8000"
|
||||
settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:8000/admin"
|
||||
settings["archivebox_api_url"] = "http://admin.archivebox.localhost:8000/api/"
|
||||
settings["archivebox_base_url"] = "http://admin.archivebox.localhost:5797"
|
||||
settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:5797/admin"
|
||||
settings["archivebox_api_url"] = "http://admin.archivebox.localhost:5797/api/"
|
||||
binary, binary_env = runtime._resolve_binary(settings["binary"], settings["config"])
|
||||
version = binary.exec(
|
||||
cmd=("--version",),
|
||||
@@ -359,6 +361,45 @@ def test_opencode_proxy_preserves_protocol_headers(admin_client, live_opencode):
|
||||
assert deleted.status_code == 200, deleted.content
|
||||
|
||||
|
||||
def test_opencode_oauth_callback_waits_for_user_and_preserves_cancellation(live_opencode):
|
||||
from abx_plugins.plugins.opencode import runtime
|
||||
|
||||
settings = {**live_opencode.settings, "timeout": 1}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
status, _, body = runtime.proxy(settings, "POST", "provider/openai/oauth/authorize", (), headers, b'{"method":0}')
|
||||
assert status == 200
|
||||
authorization = json.loads(body)
|
||||
assert urlsplit(authorization["url"]).hostname == "auth.openai.com"
|
||||
redirect = parse_qs(urlsplit(authorization["url"]).query)["redirect_uri"][0]
|
||||
callback_origin = urlsplit(redirect)
|
||||
assert callback_origin.hostname == "localhost"
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
callback = executor.submit(runtime.proxy, settings, "POST", "provider/openai/oauth/callback", (), headers, b'{"method":0}')
|
||||
try:
|
||||
# A real pending authorization must outlive the ordinary API read
|
||||
# timeout. No tokens or substituted provider responses are used.
|
||||
with pytest.raises(FutureTimeoutError):
|
||||
callback.result(timeout=2)
|
||||
finally:
|
||||
cancelled = requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=5)
|
||||
assert cancelled.status_code == 200
|
||||
assert cancelled.text == "Login cancelled"
|
||||
status, response_headers, body = callback.result(timeout=5)
|
||||
assert status == 500
|
||||
assert response_headers["Content-Type"].startswith("application/json")
|
||||
error = json.loads(body)
|
||||
assert error["name"] == "UnknownError"
|
||||
assert error["data"]["ref"].startswith("err_")
|
||||
runtime._stop_owned_process()
|
||||
# The pnpm launcher can exit before its server child. Stopping the owned
|
||||
# process must release both listeners so another collection can authorize.
|
||||
with pytest.raises(requests.ConnectionError):
|
||||
requests.get(settings["origin"] + "/global/health", timeout=2)
|
||||
with pytest.raises(requests.ConnectionError):
|
||||
requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=2)
|
||||
|
||||
|
||||
def test_opencode_static_assets_cache_privately_and_revalidate(admin_client, live_opencode):
|
||||
headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
|
||||
page = admin_client.get("/admin/agent/opencode/", **headers)
|
||||
@@ -392,9 +433,26 @@ def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode):
|
||||
@pytest.mark.parametrize("path", ["event", "global/event"])
|
||||
def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode, path):
|
||||
from abx_plugins.plugins.opencode import runtime
|
||||
|
||||
# The real /event endpoint emits heartbeats indefinitely. A bounded
|
||||
# upstream read exposes accidental buffering without hanging the test.
|
||||
status, headers, body = runtime.proxy(
|
||||
{**live_opencode.settings, "timeout": 1},
|
||||
"GET",
|
||||
path,
|
||||
(),
|
||||
{},
|
||||
b"",
|
||||
)
|
||||
assert status == 200
|
||||
assert not isinstance(body, bytes)
|
||||
assert headers["Content-Type"] == "text/event-stream"
|
||||
asyncio.run(body.aclose())
|
||||
response = admin_client.get(
|
||||
"/admin/agent/opencode/global/event",
|
||||
f"/admin/agent/opencode/{path}",
|
||||
HTTP_HOST=ADMIN_TEST_HOST,
|
||||
HTTP_SEC_FETCH_SITE="same-origin",
|
||||
)
|
||||
@@ -440,7 +498,7 @@ def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client
|
||||
(b"sec-fetch-site", b"same-origin"),
|
||||
],
|
||||
"client": ("127.0.0.1", 12345),
|
||||
"server": ("127.0.0.1", 8000),
|
||||
"server": ("127.0.0.1", 5797),
|
||||
},
|
||||
)
|
||||
runtime._PROCESS_LOCK.acquire()
|
||||
|
||||
@@ -20,8 +20,21 @@ from .test_server_security_browser import browser_runtime as browser_runtime
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"agent_server",
|
||||
[
|
||||
("safe-onedomain-nojsreplay", False),
|
||||
("safe-subdomains-fullreplay", True),
|
||||
("danger-onedomain-fullreplay", False),
|
||||
("auto", False),
|
||||
("auto", True),
|
||||
],
|
||||
indirect=True,
|
||||
)
|
||||
def test_agent_navigation_stays_inside_mount(agent_server, browser_runtime):
|
||||
server_url, _, _ = agent_server
|
||||
server_url, data_dir, _ = agent_server
|
||||
filename = "proxy acceptance #é.txt"
|
||||
(data_dir / filename).write_text("Proxy file read: café", encoding="utf-8")
|
||||
script = r"""
|
||||
const assert = require('node:assert/strict');
|
||||
const puppeteer = require('puppeteer');
|
||||
@@ -31,6 +44,14 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
args: ['--no-sandbox', '--disable-frame-rate-limit']});
|
||||
try {
|
||||
const page = await browser.newPage();
|
||||
const escapedRequests = [];
|
||||
page.on('request', request => {
|
||||
const url = new URL(request.url());
|
||||
if (request.frame()?.parentFrame() && url.origin === config.url &&
|
||||
!url.pathname.startsWith('/admin/agent/opencode/')) {
|
||||
escapedRequests.push(request.method() + ' ' + url.pathname);
|
||||
}
|
||||
});
|
||||
await page.setViewport({width: 1440, height: 900});
|
||||
await page.goto(config.url + '/admin/login/', {waitUntil: 'domcontentloaded'});
|
||||
await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
|
||||
@@ -44,7 +65,9 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
const links = await frame.$$eval('a[href*="/session"]', nodes => nodes.map(node => node.getAttribute('href')));
|
||||
assert.ok(links.length, 'OpenCode must expose session navigation');
|
||||
for (const href of links) assert.ok(href.startsWith('/admin/agent/opencode/'), href);
|
||||
const initialSessionUrl = frame.url();
|
||||
await frame.locator('::-p-aria(New session[role="button"])').click();
|
||||
await frame.waitForFunction(previous => location.href !== previous, {}, initialSessionUrl);
|
||||
await frame.waitForSelector('[contenteditable="true"]');
|
||||
let navigation;
|
||||
for (const link of await frame.$$('a[href*="/session"]')) {
|
||||
@@ -61,6 +84,101 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
assert.equal((await frame.goto(sessionUrl, {waitUntil: 'domcontentloaded'})).status(), 200);
|
||||
await frame.waitForSelector('[contenteditable="true"]');
|
||||
assert.ok(!(await frame.$eval('body', node => node.innerText)).includes('Something went wrong'));
|
||||
// Real JSON methods, encoded file paths, and both SSE subscriptions must
|
||||
// work on the admin origin in every routing mode, without installing or
|
||||
// authenticating a paid model provider during the test.
|
||||
const requests = await frame.evaluate(async filename => {
|
||||
const base = location.origin + '/admin/agent/opencode';
|
||||
const json = async (path, method = 'GET', body) => {
|
||||
const response = await fetch(base + path, {
|
||||
method, headers: {'Content-Type': 'application/json'},
|
||||
...(body === undefined ? {} : {body: JSON.stringify(body)}),
|
||||
});
|
||||
if (!response.ok) throw new Error(method + ' ' + path + ': ' + response.status);
|
||||
if (!response.headers.get('Content-Type')?.includes('application/json')) {
|
||||
throw new Error('Non-JSON response: ' + path);
|
||||
}
|
||||
return response.json();
|
||||
};
|
||||
const providers = await json('/provider');
|
||||
const auth = await json('/provider/auth');
|
||||
const file = await json('/file/content?path=' + encodeURIComponent(filename));
|
||||
const session = await json('/session', 'POST', {title: 'Proxy request verification'});
|
||||
let renamed;
|
||||
try {
|
||||
renamed = await json('/session/' + session.id, 'PATCH', {title: 'Proxy: café & # symbols'});
|
||||
const readback = await json('/session/' + session.id);
|
||||
if (readback.title !== renamed.title) throw new Error('Session mutation was not persisted');
|
||||
} finally { await json('/session/' + session.id, 'DELETE'); }
|
||||
const streams = [];
|
||||
for (const path of ['/event', '/global/event']) {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 5000);
|
||||
try {
|
||||
const response = await fetch(base + path, {signal: controller.signal});
|
||||
if (!response.ok || !response.headers.get('Content-Type')?.includes('text/event-stream')) {
|
||||
throw new Error('Invalid event stream: ' + path);
|
||||
}
|
||||
const reader = response.body.getReader();
|
||||
const first = await reader.read();
|
||||
streams.push(new TextDecoder().decode(first.value));
|
||||
await reader.cancel();
|
||||
} finally { clearTimeout(timeout); controller.abort(); }
|
||||
}
|
||||
return {providers: providers.all.map(provider => provider.id), auth, file, renamed, streams};
|
||||
}, config.filename);
|
||||
for (const provider of ['openai', 'anthropic']) assert.ok(requests.providers.includes(provider), provider);
|
||||
assert.ok(requests.auth.openai.some(method => method.type === 'oauth'));
|
||||
assert.equal(requests.file.content, 'Proxy file read: café');
|
||||
assert.equal(requests.renamed.title, 'Proxy: café & # symbols');
|
||||
for (const event of requests.streams) assert.ok(event.includes('server.connected'), event);
|
||||
for (const provider of ['OpenAI', 'Anthropic']) {
|
||||
await frame.locator('[data-action="prompt-model"]').click();
|
||||
await frame.locator('::-p-aria(' + provider + '[role="button"])').click();
|
||||
if (provider === 'OpenAI') {
|
||||
await frame.waitForSelector('::-p-aria(ChatGPT Pro/Plus Headless[role="button"])');
|
||||
await frame.waitForSelector('::-p-aria(ChatGPT Pro/Plus Browser[role="button"])');
|
||||
await frame.locator('::-p-aria(API key Browser[role="button"])').click();
|
||||
}
|
||||
const input = await frame.waitForSelector('::-p-aria(' + provider + ' API key[role="textbox"])');
|
||||
assert.equal(await input.evaluate(node => node.value), '');
|
||||
await frame.waitForSelector('::-p-aria(Continue[role="button"])');
|
||||
await frame.locator('::-p-aria(Close[role="button"])').click();
|
||||
}
|
||||
// Submit a real shell command from a fresh draft. This exercises draft
|
||||
// promotion and the shell API without needing a paid model or API key.
|
||||
await frame.locator('::-p-aria(New session[role="button"])').click();
|
||||
await frame.waitForFunction(() => location.pathname.endsWith('/new-session'));
|
||||
await (await frame.waitForSelector('[contenteditable="true"]')).focus();
|
||||
const modifier = await page.evaluate(() => navigator.platform.includes('Mac') ? 'Meta' : 'Control');
|
||||
await page.keyboard.down(modifier);
|
||||
await page.keyboard.down('Shift');
|
||||
await page.keyboard.press('x');
|
||||
await page.keyboard.up('Shift');
|
||||
await page.keyboard.up(modifier);
|
||||
await page.keyboard.type('printf ABX_DRAFT_SHELL_OK');
|
||||
await frame.locator('::-p-aria(Send[role="button"])').click();
|
||||
await frame.waitForFunction(() => /\/session\/ses_/.test(location.pathname));
|
||||
assert.ok(new URL(frame.url()).pathname.startsWith('/admin/agent/opencode/'), frame.url());
|
||||
await frame.waitForFunction(async () => {
|
||||
const sessionID = location.pathname.match(/\/session\/(ses_[^/]+)/)[1];
|
||||
const response = await fetch('/admin/agent/opencode/session/' + sessionID + '/message');
|
||||
if (!response.ok) throw new Error('Shell message readback: ' + response.status);
|
||||
const messages = await response.json();
|
||||
return messages.some(message => message.parts.some(part =>
|
||||
part.type === 'tool' && part.state.status === 'completed' &&
|
||||
part.state.output.trim() === 'ABX_DRAFT_SHELL_OK'));
|
||||
}, {polling: 250});
|
||||
// The tabbed layout's own route parser must recognize mounted sessions.
|
||||
// Otherwise Home incorrectly appears selected and clicking it does nothing.
|
||||
const shellSessionUrl = frame.url();
|
||||
const home = await frame.waitForSelector('[aria-label="Home"]');
|
||||
assert.equal(await home.evaluate(node => node.getAttribute('aria-pressed')), 'false');
|
||||
await home.click();
|
||||
await frame.waitForFunction(() => location.pathname.replace(/\/+$/, '') === '/admin/agent/opencode');
|
||||
assert.equal(await home.evaluate(node => node.getAttribute('aria-pressed')), 'true');
|
||||
await home.click();
|
||||
await frame.waitForFunction(expected => location.href === expected, {}, shellSessionUrl);
|
||||
// Exercise the actual public PTY API and native browser WebSocket. No
|
||||
// intercepted traffic or replacement server: this runs a real shell.
|
||||
const terminal = await frame.evaluate(async () => {
|
||||
@@ -97,6 +215,7 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
} finally { await fetch(base + '/pty/' + pty.id, {method: 'DELETE'}); }
|
||||
});
|
||||
assert.ok(terminal.includes('ABX_TERMINAL_OK'), terminal);
|
||||
assert.deepEqual(escapedRequests, [], 'OpenCode requests must stay inside its proxy mount');
|
||||
assert.equal((await page.goto(config.url + '/add/', {waitUntil: 'domcontentloaded'})).status(), 200);
|
||||
console.log('AGENT_NAVIGATION_OK');
|
||||
} finally { await browser.close(); }
|
||||
@@ -104,7 +223,7 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
"""
|
||||
result = subprocess.run(
|
||||
[str(browser_runtime["node_binary"]), "-e", script],
|
||||
input=json.dumps({"chrome": str(browser_runtime["chrome_binary"]), "url": server_url}),
|
||||
input=json.dumps({"chrome": str(browser_runtime["chrome_binary"]), "url": server_url, "filename": filename}),
|
||||
env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
|
||||
text=True,
|
||||
capture_output=True,
|
||||
@@ -115,14 +234,16 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def agent_server(installed_opencode, browser_runtime):
|
||||
def agent_server(installed_opencode, browser_runtime, request):
|
||||
port = get_free_port()
|
||||
url = f"http://localhost:{port}"
|
||||
mode, subdomains = getattr(request, "param", ("safe-onedomain-nojsreplay", False))
|
||||
base_url = f"http://archivebox.localhost:{port}" if subdomains else f"http://localhost:{port}"
|
||||
url = f"http://admin.archivebox.localhost:{port}" if subdomains else base_url
|
||||
config = installed_opencode.config
|
||||
_set_archivebox_config(
|
||||
config.data_dir,
|
||||
f"BASE_URL={url}",
|
||||
"SERVER_SECURITY_MODE=safe-onedomain-nojsreplay",
|
||||
f"BASE_URL={base_url}",
|
||||
f"SERVER_SECURITY_MODE={mode}",
|
||||
)
|
||||
user = run_archivebox_cmd(
|
||||
[
|
||||
|
||||
@@ -32,8 +32,8 @@ from archivebox.tests.conftest import (
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
User = get_user_model()
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
WEB_HOST = "web.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
WEB_HOST = "web.archivebox.localhost:5797"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Exercise automatic server addressing through real HTTP login pages."""
|
||||
|
||||
import re
|
||||
|
||||
import requests
|
||||
|
||||
from .conftest import cli_env, get_free_port, run_archivebox_cmd, start_archivebox_server, stop_archivebox_process
|
||||
|
||||
|
||||
def test_safe_single_domain_login_does_not_require_base_url(tmp_path):
|
||||
port = get_free_port()
|
||||
env = cli_env(
|
||||
disable_extractors=True,
|
||||
BASE_URL="",
|
||||
SERVER_SECURITY_MODE="safe-onedomain-nojsreplay",
|
||||
ADMIN_USERNAME="automatic-url-test",
|
||||
ADMIN_PASSWORD="Automatic-url-test-password-93!",
|
||||
ALLOWED_HOSTS="*",
|
||||
)
|
||||
result = run_archivebox_cmd(["init", "--quick"], cwd=tmp_path, env=env, timeout=60)
|
||||
assert result.returncode == 0, result.stderr or result.stdout
|
||||
process = start_archivebox_server(tmp_path, port=port, env=env, daemonize=False, log_name="server.log")
|
||||
try:
|
||||
for hostname in ("localhost", "archivebox.localhost", "admin.archivebox.localhost", "192.168.1.20", "100.100.10.20"):
|
||||
session = requests.Session()
|
||||
response = session.get(
|
||||
f"http://127.0.0.1:{port}/admin/login/",
|
||||
headers={"Host": f"{hostname}:{port}"},
|
||||
timeout=15,
|
||||
allow_redirects=False,
|
||||
)
|
||||
assert response.status_code == 200, (hostname, response.status_code)
|
||||
assert 'name="password"' in response.text, hostname
|
||||
assert "base_url not set" not in response.text.lower(), hostname
|
||||
assert "archivebox-setup-wizard" not in response.text, hostname
|
||||
csrf = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', response.text)
|
||||
assert csrf, hostname
|
||||
login = session.post(
|
||||
f"http://127.0.0.1:{port}/admin/login/",
|
||||
headers={"Host": f"{hostname}:{port}"},
|
||||
data={
|
||||
"username": env["ADMIN_USERNAME"],
|
||||
"password": env["ADMIN_PASSWORD"],
|
||||
"csrfmiddlewaretoken": csrf[1],
|
||||
"next": "/admin/",
|
||||
},
|
||||
timeout=15,
|
||||
allow_redirects=False,
|
||||
)
|
||||
assert login.status_code == 302, (hostname, login.status_code)
|
||||
admin = session.get(
|
||||
f"http://127.0.0.1:{port}/admin/",
|
||||
headers={"Host": f"{hostname}:{port}"},
|
||||
timeout=15,
|
||||
allow_redirects=False,
|
||||
)
|
||||
assert admin.status_code == 200, (hostname, admin.status_code)
|
||||
assert "base_url not set" not in admin.text.lower(), hostname
|
||||
assert "archivebox-setup-wizard" not in admin.text, hostname
|
||||
finally:
|
||||
stop_archivebox_process(process)
|
||||
@@ -749,7 +749,7 @@ def test_unconfigured_public_host_superuser_can_reach_setup_wizard(tmp_path: Pat
|
||||
assert "BASE_URL" in archivebox_environment
|
||||
assert "SERVER_SECURITY_MODE" in archivebox_environment
|
||||
assert all("ARCHIVEBOX_INGRESS_BASE_URL" not in value for value in archivebox_environment)
|
||||
assert compose["services"]["archivebox"]["ports"] == ["${ARCHIVEBOX_PORT:-8000}:8000"]
|
||||
assert compose["services"]["archivebox"]["ports"] == ["${ARCHIVEBOX_PORT:-5797}:5797"]
|
||||
assert set(compose["services"]) == {"archivebox"}
|
||||
|
||||
|
||||
|
||||
@@ -16,8 +16,8 @@ from archivebox.workers.models import RETRY_AT_MAX
|
||||
pytestmark = pytest.mark.django_db
|
||||
|
||||
User = get_user_model()
|
||||
WEB_HOST = "web.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
WEB_HOST = "web.archivebox.localhost:5797"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
|
||||
|
||||
ADD_FORM_DEFAULTS = {
|
||||
@@ -99,7 +99,7 @@ def test_add_view_admin_renders_plugin_config_grid(client, admin_user):
|
||||
assert b">Source</a>" in response.content
|
||||
assert b">Docs</a>" in response.content
|
||||
assert b"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/" in response.content
|
||||
assert b"https://archivebox.github.io/abx-plugins/#" in response.content
|
||||
assert b"https://plugins.archivebox.io/#" in response.content
|
||||
assert not any(
|
||||
field["key"].endswith("_BINARY") for group in form.plugin_groups for card in group["plugins"] for field in card["config_fields"]
|
||||
)
|
||||
|
||||
@@ -99,8 +99,8 @@ def test_unconfigured_superuser_banner_uses_browser_assisted_setup_wizard():
|
||||
{
|
||||
"mode": "unconfigured",
|
||||
"can_configure": True,
|
||||
"canonical_host": "archivebox.example.test:8000",
|
||||
"suggested_base_url": "http://archivebox.example.test:8000",
|
||||
"canonical_host": "archivebox.example.test:5797",
|
||||
"suggested_base_url": "http://archivebox.example.test:5797",
|
||||
"machine_admin_url": "/admin/machine/machine/current/change/",
|
||||
"public_index": True,
|
||||
"public_add_view": False,
|
||||
@@ -111,7 +111,7 @@ def test_unconfigured_superuser_banner_uses_browser_assisted_setup_wizard():
|
||||
assert html.count('id="archivebox-setup-wizard"') == 1
|
||||
assert 'id="archivebox-system-warning-banner"' not in html
|
||||
assert "⚠ base_url not set" not in html
|
||||
assert "Setup access to your ArchiveBox Server: <code>archivebox.example.test:8000</code>" in html
|
||||
assert "Setup access to your ArchiveBox Server: <code>archivebox.example.test:5797</code>" in html
|
||||
assert 'id="archivebox-setup-base-url"' in html
|
||||
assert 'id="archivebox-setup-security-mode"' in html
|
||||
assert 'id="archivebox-setup-public-index"' in html
|
||||
@@ -245,9 +245,9 @@ def test_health_check_is_identifiable_across_ingress_origins():
|
||||
@pytest.mark.parametrize(
|
||||
("request_host", "secure", "expected_display_host"),
|
||||
(
|
||||
("admin.archivebox.localhost:8000", False, "archivebox.localhost:8000"),
|
||||
("127.0.0.1:8000", False, "archivebox.localhost:8000"),
|
||||
("0.0.0.0:8000", False, "archivebox.localhost:8000"),
|
||||
("admin.archivebox.localhost:5797", False, "archivebox.localhost:5797"),
|
||||
("127.0.0.1:5797", False, "archivebox.localhost:5797"),
|
||||
("0.0.0.0:5797", False, "archivebox.localhost:5797"),
|
||||
("archivebox.io:80", False, "archivebox.io"),
|
||||
("archivebox.io:443", True, "archivebox.io"),
|
||||
("192.0.2.10:443", True, "192.0.2.10"),
|
||||
@@ -279,7 +279,7 @@ def test_unconfigured_banner_honors_forwarded_https_from_ingress():
|
||||
("base_url", "request_host"),
|
||||
(
|
||||
("", "archivebox.example.test:18443"),
|
||||
("http://archivebox.localhost:8000", "admin.archivebox.localhost:18010"),
|
||||
("http://archivebox.localhost:5797", "admin.archivebox.localhost:18010"),
|
||||
),
|
||||
)
|
||||
def test_admin_cookie_isolation_accepts_first_run_and_external_port_mapping(base_url, request_host):
|
||||
@@ -302,7 +302,7 @@ def test_unconfigured_banner_does_not_show_setup_wizard_to_non_superusers():
|
||||
{
|
||||
"mode": "unconfigured",
|
||||
"can_configure": False,
|
||||
"suggested_base_url": "http://archivebox.example.test:8000",
|
||||
"suggested_base_url": "http://archivebox.example.test:5797",
|
||||
},
|
||||
)
|
||||
|
||||
@@ -329,13 +329,13 @@ def test_system_warning_modes_share_one_banner(context, expected_text):
|
||||
|
||||
def test_configured_base_url_mismatch_banner_shows_both_origins():
|
||||
config = get_config(include_machine=False).model_copy(update={"BASE_URL": "https://archivebox.example.test"})
|
||||
request = RequestFactory().get("/admin/", HTTP_HOST="archivebox.internal:8000")
|
||||
request = RequestFactory().get("/admin/", HTTP_HOST="archivebox.internal:5797")
|
||||
request.user = AnonymousUser()
|
||||
|
||||
context = get_base_url_mismatch_context(request, config)
|
||||
assert context == {
|
||||
"mode": "base_url_mismatch",
|
||||
"browser_url": "http://archivebox.internal:8000",
|
||||
"browser_url": "http://archivebox.internal:5797",
|
||||
"configured_base_url": "https://archivebox.example.test",
|
||||
}
|
||||
assert system_warnings_banner({"CONFIG": config, "request": request}) == context
|
||||
@@ -343,7 +343,7 @@ def test_configured_base_url_mismatch_banner_shows_both_origins():
|
||||
html = render_to_string("core/system_warnings_banner.html", context)
|
||||
assert "base_url mismatch" in html
|
||||
assert "Browser URL:" in html
|
||||
assert "http://archivebox.internal:8000" in html
|
||||
assert "http://archivebox.internal:5797" in html
|
||||
assert "Configured BASE_URL:" in html
|
||||
assert "https://archivebox.example.test" in html
|
||||
|
||||
|
||||
@@ -416,7 +416,7 @@ def test_snapshot_from_json_trims_markdown_suffixes_on_discovered_urls(crawl):
|
||||
|
||||
def test_create_snapshots_from_urls_skips_invalid_and_archivebox_internal_urls(admin_user):
|
||||
crawl = Crawl.objects.create(
|
||||
urls=("https://example.com/root\nhttp://127.0.0.1:8765/page-001.html\nnot-a-url\nhttp://admin.archivebox.localhost:8000/admin/"),
|
||||
urls=("https://example.com/root\nhttp://127.0.0.1:8765/page-001.html\nnot-a-url\nhttp://admin.archivebox.localhost:5797/admin/"),
|
||||
created_by=admin_user,
|
||||
)
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ def _admin_post_request(path):
|
||||
def _admin_get_request(path="/"):
|
||||
from archivebox.config.common import get_config
|
||||
|
||||
request = RequestFactory().get(path, HTTP_HOST="admin.archivebox.localhost:8000")
|
||||
request = RequestFactory().get(path, HTTP_HOST="admin.archivebox.localhost:5797")
|
||||
request.archivebox_config = get_config()
|
||||
return request
|
||||
|
||||
@@ -284,7 +284,7 @@ def test_admin_navigation_hides_agent_link_when_opencode_is_disabled(client, adm
|
||||
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
|
||||
client.force_login(admin_user)
|
||||
|
||||
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:8000")
|
||||
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:5797")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert b"/admin/agent" not in response.content
|
||||
|
||||
@@ -127,6 +127,10 @@ def test_persona_admin_add_view_renders_import_ui(admin_client):
|
||||
assert b"auth.json" in response.content
|
||||
assert b"Plugin Config" in response.content
|
||||
assert b'name="plugin_config__wget__WGET_TIMEOUT"' in response.content
|
||||
assert b'<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">' in response.content
|
||||
assert b'href="https://extension.archivebox.io/"' in response.content
|
||||
assert b"import cookies into this persona from another browser" in response.content
|
||||
assert "wide" in response.context["adminform"].fieldsets[0][1]["classes"]
|
||||
|
||||
|
||||
def test_import_persona_from_source_copies_profile_without_browser_export(admin_user):
|
||||
|
||||
@@ -13,7 +13,7 @@ import pytest
|
||||
from django.contrib.auth.models import AnonymousUser
|
||||
from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
|
||||
from django.core.paginator import UnorderedObjectListWarning
|
||||
from django.test import RequestFactory
|
||||
from django.test import Client, RequestFactory
|
||||
from django.urls import reverse
|
||||
|
||||
from archivebox.core.middleware import ADMIN_LOGIN_HINT_COOKIE
|
||||
@@ -152,6 +152,8 @@ def test_snapshot_changelist_uses_stable_ordering_without_unordered_paginator_wa
|
||||
def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client, snapshot, real_hash_projection):
|
||||
from archivebox.core.models import ArchiveResult
|
||||
|
||||
snapshot.title = "Preview destination test"
|
||||
snapshot.save(update_fields=["title"])
|
||||
_process, result = real_hash_projection
|
||||
ArchiveResult.objects.filter(pk=result.pk).update(
|
||||
plugin="screenshot",
|
||||
@@ -165,6 +167,103 @@ def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client,
|
||||
assert response.status_code == 200
|
||||
assert b'class="snapshot-preview' in response.content
|
||||
assert b"screenshot.png" in response.content
|
||||
html = response.content.decode()
|
||||
preview_cell = re.search(r'<td class="field-preview_icon">(.*?)</td>', html, re.S)
|
||||
title_cell = re.search(r'<td class="field-title_str">(.*?)</td>', html, re.S)
|
||||
assert preview_cell and title_cell
|
||||
preview_link = re.search(r'<a href="([^"]+)"', preview_cell.group(1))
|
||||
title_link = re.search(r'<a href="([^"]+)"', title_cell.group(1))
|
||||
assert preview_link and title_link
|
||||
assert preview_link.group(1) == title_link.group(1)
|
||||
assert preview_link.group(1).endswith(f"/{snapshot.archive_path_from_db}/index.html")
|
||||
|
||||
|
||||
def test_snapshot_grid_missing_preview_is_inside_snapshot_detail_link(admin_client, snapshot):
|
||||
snapshot.status = snapshot.StatusChoices.SEALED
|
||||
snapshot.save(update_fields=["status"])
|
||||
|
||||
response = admin_client.get(reverse("admin:grid"), HTTP_HOST=ADMIN_TEST_HOST)
|
||||
|
||||
assert response.status_code == 200
|
||||
html = response.content.decode()
|
||||
thumbnail_link = re.search(
|
||||
r'<a href="([^"]+)" class="card-thumbnail">(.*?)</a>',
|
||||
html,
|
||||
re.S,
|
||||
)
|
||||
title_link = re.search(
|
||||
r'<div class="card-title"[^>]*>\s*<a href="([^"]+)">',
|
||||
html,
|
||||
re.S,
|
||||
)
|
||||
assert thumbnail_link and title_link
|
||||
assert thumbnail_link.group(1) == title_link.group(1)
|
||||
assert thumbnail_link.group(1).endswith("/index.html")
|
||||
assert '<span class="missing-preview">No preview captured</span>' in thumbnail_link.group(2)
|
||||
|
||||
|
||||
def test_snapshot_grid_serves_extension_preview_from_authenticated_admin_origin(admin_client, snapshot, real_hash_projection):
|
||||
_process, result = real_hash_projection
|
||||
preview_dir = Path(snapshot.output_dir) / "chrome_extension_screenshot"
|
||||
preview_dir.mkdir(parents=True, exist_ok=True)
|
||||
preview_bytes = b"\x89PNG\r\n\x1a\nreal-extension-preview"
|
||||
(preview_dir / "screenshot.png").write_bytes(preview_bytes)
|
||||
result.plugin = "chrome_extension_screenshot"
|
||||
result.status = "succeeded"
|
||||
result.output_files = {"screenshot.png": {"size": len(preview_bytes), "mimetype": "image/png"}}
|
||||
result.save(update_fields=["plugin", "status", "output_files"])
|
||||
|
||||
grid_response = admin_client.get(reverse("admin:grid"), HTTP_HOST=ADMIN_TEST_HOST)
|
||||
|
||||
assert grid_response.status_code == 200
|
||||
html = grid_response.content.decode()
|
||||
preview_url = reverse(
|
||||
"admin:core_snapshot_preview",
|
||||
args=(snapshot.pk, "chrome_extension_screenshot", "screenshot.png"),
|
||||
)
|
||||
assert f'src="{preview_url}"' in html
|
||||
assert "new URL(candidate, document.baseURI).href" in html
|
||||
assert "candidate !== current" in html
|
||||
|
||||
preview_response = admin_client.get(preview_url, HTTP_HOST=ADMIN_TEST_HOST)
|
||||
|
||||
assert preview_response.status_code == 200
|
||||
assert b"".join(preview_response.streaming_content) == preview_bytes
|
||||
assert preview_response["Content-Type"] == "image/png"
|
||||
assert {part.strip() for part in preview_response["Cache-Control"].split(",")} == {
|
||||
"private",
|
||||
"max-age=604800",
|
||||
"immutable",
|
||||
}
|
||||
assert {part.strip() for part in preview_response["Vary"].split(",")} >= {"Cookie", "Authorization"}
|
||||
assert preview_response["X-Content-Type-Options"] == "nosniff"
|
||||
|
||||
not_modified = admin_client.get(
|
||||
preview_url,
|
||||
HTTP_HOST=ADMIN_TEST_HOST,
|
||||
HTTP_IF_MODIFIED_SINCE=preview_response["Last-Modified"],
|
||||
)
|
||||
assert not_modified.status_code == 304
|
||||
assert {part.strip() for part in not_modified["Cache-Control"].split(",")} == {
|
||||
"private",
|
||||
"max-age=60",
|
||||
"stale-while-revalidate=300",
|
||||
}
|
||||
assert {part.strip() for part in not_modified["Vary"].split(",")} >= {"Cookie", "Authorization"}
|
||||
|
||||
denied = Client().get(preview_url, HTTP_HOST=ADMIN_TEST_HOST)
|
||||
assert denied.status_code in (301, 302)
|
||||
assert {part.strip() for part in denied["Cache-Control"].split(",")} == {"private", "no-store"}
|
||||
|
||||
disallowed_dir = Path(snapshot.output_dir) / "singlefile"
|
||||
disallowed_dir.mkdir(parents=True, exist_ok=True)
|
||||
(disallowed_dir / "singlefile.html").write_text("<script>document.cookie</script>", encoding="utf-8")
|
||||
result.plugin = "singlefile"
|
||||
result.output_files = {"singlefile.html": {"size": 32, "mimetype": "text/html"}}
|
||||
result.save(update_fields=["plugin", "output_files"])
|
||||
disallowed_url = reverse("admin:core_snapshot_preview", args=(snapshot.pk, "singlefile", "singlefile.html"))
|
||||
|
||||
assert admin_client.get(disallowed_url, HTTP_HOST=ADMIN_TEST_HOST).status_code == 404
|
||||
|
||||
|
||||
def test_snapshot_result_health_filter_uses_live_status_rows(admin_client, snapshot):
|
||||
@@ -312,7 +411,7 @@ def test_snapshot_admin_preview_uses_extension_screenshot_when_standard_screensh
|
||||
)
|
||||
|
||||
admin = SnapshotAdmin(Snapshot, archivebox_admin)
|
||||
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:8000")
|
||||
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:5797")
|
||||
request.archivebox_config = get_config()
|
||||
admin.request = request
|
||||
|
||||
@@ -411,7 +510,7 @@ class TestSnapshotProgressStats:
|
||||
|
||||
prefetched_snapshot = Snapshot.objects.prefetch_related("archiveresult_set").get(pk=snapshot.pk)
|
||||
admin = SnapshotAdmin(Snapshot, archivebox_admin)
|
||||
request = RequestFactory().get("/", HTTP_HOST="archivebox.localhost:8000")
|
||||
request = RequestFactory().get("/", HTTP_HOST="archivebox.localhost:5797")
|
||||
request.resolver_match = resolve("/")
|
||||
request.archivebox_config = get_config()
|
||||
admin.request = request
|
||||
@@ -777,7 +876,7 @@ class TestSnapshotProgressStats:
|
||||
|
||||
assert embed_path in html
|
||||
assert "?preview=1" not in html
|
||||
assert html != "http://snap-ffa4215f6d64.archivebox.localhost:8000"
|
||||
assert html != "http://snap-ffa4215f6d64.archivebox.localhost:5797"
|
||||
|
||||
def test_plugin_full_returns_empty_for_none_result(self):
|
||||
from archivebox.core.templatetags import core_tags
|
||||
@@ -1279,13 +1378,13 @@ class TestAdminSnapshotListView:
|
||||
|
||||
admin = SnapshotAdmin(snapshot.__class__, archivebox_admin)
|
||||
|
||||
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:8000")
|
||||
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:5797")
|
||||
request.archivebox_config = get_config(overrides={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
|
||||
admin.request = request
|
||||
assert admin.get_snapshot_view_url(snapshot) == f"http://snap-{str(snapshot.pk).replace('-', '')[-12:]}.archivebox.localhost:8000"
|
||||
assert admin.get_snapshot_view_url(snapshot) == f"http://snap-{str(snapshot.pk).replace('-', '')[-12:]}.archivebox.localhost:5797"
|
||||
|
||||
request.archivebox_config = get_config(overrides={"SERVER_SECURITY_MODE": "safe-onedomain-nojsreplay"})
|
||||
assert admin.get_snapshot_view_url(snapshot) == f"http://archivebox.localhost:8000/snapshot/{snapshot.pk}"
|
||||
assert admin.get_snapshot_view_url(snapshot) == f"http://archivebox.localhost:5797/snapshot/{snapshot.pk}"
|
||||
|
||||
def test_find_snapshots_for_url_matches_fragment_suffixed_variants(self, crawl, db):
|
||||
from archivebox.core.models import Snapshot
|
||||
|
||||
@@ -132,7 +132,7 @@ def test_plugin_detail_view_renders_shipped_plugin_config_in_dedicated_sections(
|
||||
"source": "builtin",
|
||||
}
|
||||
assert "abx_plugins/plugins/wget" in summary_section["description"]
|
||||
assert "https://archivebox.github.io/abx-plugins/#wget" in summary_section["description"]
|
||||
assert "https://plugins.archivebox.io/#wget" in summary_section["description"]
|
||||
|
||||
assert hooks_section["name"] == "Hooks"
|
||||
assert hooks_section["fields"] == {}
|
||||
|
||||
@@ -3,9 +3,11 @@
|
||||
import subprocess
|
||||
import uuid
|
||||
from datetime import datetime, timezone as dt_timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from threading import Event, Thread
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from django.test import override_settings
|
||||
@@ -19,6 +21,41 @@ from archivebox.tests.test_archive_result_service import _run_shipped_snapshot_h
|
||||
pytestmark = pytest.mark.django_db(transaction=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def painted_http_server():
|
||||
"""Serve a complete static page so CDP stops emitting changed frames."""
|
||||
|
||||
request_started = Event()
|
||||
|
||||
class PaintedBlockingHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
body = b"""<!doctype html>
|
||||
<html><head><title>Still frame</title></head>
|
||||
<body style="background:#246;color:white"><h1>Visible page</h1></body></html>
|
||||
"""
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
self.wfile.flush()
|
||||
request_started.set()
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), PaintedBlockingHandler)
|
||||
thread = Thread(target=server.serve_forever, name="archivebox-test-painted-http-barrier")
|
||||
thread.start()
|
||||
yield SimpleNamespace(
|
||||
url=f"http://127.0.0.1:{server.server_port}/",
|
||||
request_started=request_started,
|
||||
)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def real_unscoped_hook_process(tmp_path):
|
||||
from archivebox.tests.conftest import run_test_hook
|
||||
@@ -329,6 +366,77 @@ class TestLiveProgressView:
|
||||
result.refresh_from_db()
|
||||
assert result.status in (ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS)
|
||||
|
||||
def test_live_progress_keeps_still_browser_frame_while_screencast_runs(
|
||||
self,
|
||||
client,
|
||||
admin_user,
|
||||
crawl,
|
||||
snapshot,
|
||||
painted_http_server,
|
||||
cached_abxpkg_lib_dir,
|
||||
):
|
||||
import time
|
||||
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.services.runner import run_due_snapshot
|
||||
|
||||
now = timezone.now()
|
||||
Crawl.objects.filter(pk=crawl.pk).update(
|
||||
status=Crawl.StatusChoices.STARTED,
|
||||
retry_at=now,
|
||||
urls=painted_http_server.url,
|
||||
config={"PLUGINS": "chrome,chrome_screencast", "CHROME_ISOLATION": "snapshot"},
|
||||
)
|
||||
Snapshot.objects.filter(pk=snapshot.pk).update(
|
||||
status=Snapshot.StatusChoices.QUEUED,
|
||||
retry_at=now,
|
||||
url=painted_http_server.url,
|
||||
config={
|
||||
"PLUGINS": "chrome,chrome_screencast",
|
||||
"CHROME_HEADLESS": True,
|
||||
"CHROME_SANDBOX": False,
|
||||
"CHROME_ISOLATION": "snapshot",
|
||||
"CHROME_TIMEOUT": 90,
|
||||
"CHROME_PAGELOAD_TIMEOUT": 90,
|
||||
"CHROME_DELAY_AFTER_LOAD": 30,
|
||||
"TIMEOUT": 90,
|
||||
},
|
||||
)
|
||||
snapshot.refresh_from_db()
|
||||
frame = Path(snapshot.crawl.output_dir) / "chrome_screencast" / "latest.jpg"
|
||||
errors = []
|
||||
|
||||
def run_snapshot():
|
||||
try:
|
||||
assert run_due_snapshot(snapshot, lock_seconds=120) is True
|
||||
except BaseException as err:
|
||||
errors.append(err)
|
||||
|
||||
runner = Thread(target=run_snapshot, name="archivebox-test-screencast-runner")
|
||||
runner.start()
|
||||
try:
|
||||
assert painted_http_server.request_started.wait(timeout=60)
|
||||
deadline = time.monotonic() + 90
|
||||
while time.monotonic() < deadline:
|
||||
assert errors == []
|
||||
if frame.is_file() and time.time() - frame.stat().st_mtime > 16:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
assert frame.is_file()
|
||||
assert frame.read_bytes().startswith(b"\xff\xd8\xff")
|
||||
assert time.time() - frame.stat().st_mtime > 16
|
||||
client.force_login(admin_user)
|
||||
response = client.get("/progress.json", HTTP_HOST=ADMIN_TEST_HOST)
|
||||
assert response.status_code == 200, response.content
|
||||
active_crawl = next(item for item in response.json()["active_crawls"] if item["id"] == str(crawl.pk))
|
||||
assert active_crawl["screencast_url"].startswith(f"/api/v1/crawls/crawl/{crawl.pk}/files/chrome_screencast/latest.jpg")
|
||||
finally:
|
||||
runner.join(timeout=120)
|
||||
assert not runner.is_alive()
|
||||
assert errors == []
|
||||
assert not frame.exists()
|
||||
|
||||
def test_live_progress_hides_finished_cancelled_crawl(self, client, admin_user, crawl, snapshot):
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.crawls.models import Crawl
|
||||
|
||||
@@ -8,7 +8,7 @@ from pathlib import Path
|
||||
from archivebox.tests.conftest import cli_env, create_test_url
|
||||
|
||||
|
||||
ADMIN_HOST = "admin.archivebox.localhost:8000"
|
||||
ADMIN_HOST = "admin.archivebox.localhost:5797"
|
||||
|
||||
|
||||
def _run_savepagenow_script(
|
||||
@@ -93,7 +93,7 @@ def _run_savepagenow_not_found_script(initialized_archive: Path, request_url: st
|
||||
client = Client()
|
||||
target_url = {request_url!r}
|
||||
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
|
||||
assert resp.status_code == 302, resp.status_code
|
||||
assert resp['Location'] == f'http://{ADMIN_HOST}/web/' + target_url
|
||||
assert Snapshot.objects.count() == 0
|
||||
@@ -226,7 +226,7 @@ def _run_savepagenow_via_web_host_redirect_script(initialized_archive: Path, req
|
||||
|
||||
target_url = {request_url!r}
|
||||
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
|
||||
assert resp.status_code == 302, resp.status_code
|
||||
assert resp['Location'] == f'http://{ADMIN_HOST}/web/' + target_url
|
||||
|
||||
@@ -274,7 +274,7 @@ def _run_savepagenow_existing_snapshot_script(initialized_archive: Path, request
|
||||
snapshot = Snapshot.objects.create(url=stored_url, crawl=crawl)
|
||||
|
||||
client = Client()
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
|
||||
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
|
||||
assert resp.status_code == 302, resp.status_code
|
||||
assert resp['Location'] == f"/{{snapshot.url_path}}"
|
||||
""",
|
||||
@@ -310,7 +310,7 @@ def test_web_add_creates_and_reuses_snapshot_public(initialized_archive):
|
||||
url,
|
||||
login=False,
|
||||
public_add_view=True,
|
||||
host="web.archivebox.localhost:8000",
|
||||
host="web.archivebox.localhost:5797",
|
||||
)
|
||||
assert result.returncode == 0, f"SavePageNow shortcut (public add) test failed.\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
|
||||
|
||||
|
||||
+404
-20
@@ -8,12 +8,51 @@ from pathlib import Path
|
||||
from threading import Thread
|
||||
|
||||
import pytest
|
||||
from types import SimpleNamespace
|
||||
|
||||
from archivebox.tests.conftest import run_archivebox_cmd
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def test_private_snapshot_image_response_is_not_publicly_cacheable(tmp_path):
|
||||
from django.test import RequestFactory
|
||||
|
||||
from archivebox.misc.serve_static import serve_static_with_byterange_support
|
||||
|
||||
(tmp_path / "screenshot.png").write_bytes(b"\x89PNG\r\n\x1a\nprivate")
|
||||
request = RequestFactory().get("/snapshot/private/screenshot.png")
|
||||
request.archivebox_config = SimpleNamespace(PERMISSIONS="public")
|
||||
request.archivebox_cache_policy = "private"
|
||||
|
||||
response = serve_static_with_byterange_support(request, "screenshot.png", document_root=tmp_path)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response["Content-Type"] == "image/png"
|
||||
assert response["Cache-Control"] == "private, max-age=604800, immutable"
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_private_snapshot_replay_never_returns_public_cache_headers(snapshot):
|
||||
from django.test import RequestFactory
|
||||
|
||||
from archivebox.core.views import _serve_snapshot_replay
|
||||
|
||||
snapshot.config = {**snapshot.config, "PERMISSIONS": "private"}
|
||||
snapshot.save(update_fields=["config"])
|
||||
snapshot.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
(snapshot.output_dir / "screenshot.png").write_bytes(b"\x89PNG\r\n\x1a\nprivate-replay")
|
||||
request = RequestFactory().get(f"/snapshot/{snapshot.id}/screenshot.png")
|
||||
|
||||
response = _serve_snapshot_replay(request, snapshot, "screenshot.png")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response["Cache-Control"].startswith("private,")
|
||||
assert "public" not in response["Cache-Control"]
|
||||
assert "Cookie" in response["Vary"]
|
||||
assert b"".join(response.streaming_content).endswith(b"private-replay")
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_archiveresult_relpath_uses_sibling_hook_that_owns_output(admin_user):
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
@@ -495,16 +534,16 @@ class TestUrlRouting:
|
||||
|
||||
bind_host, bind_port = split_host_port(bind_addr)
|
||||
assert bind_host == "127.0.0.1"
|
||||
assert bind_port == "8000"
|
||||
assert bind_port == "5797"
|
||||
host_only, port = split_host_port(base_host)
|
||||
assert host_only == "archivebox.localhost"
|
||||
assert port == "8000"
|
||||
assert web_host == "web.archivebox.localhost:8000"
|
||||
assert admin_host == "admin.archivebox.localhost:8000"
|
||||
assert api_host == "api.archivebox.localhost:8000"
|
||||
assert port == "5797"
|
||||
assert web_host == "web.archivebox.localhost:5797"
|
||||
assert admin_host == "admin.archivebox.localhost:5797"
|
||||
assert api_host == "api.archivebox.localhost:5797"
|
||||
assert snapshot_subdomain == f"snap-{snapshot_id[-12:].lower()}"
|
||||
assert snapshot_host == f"{snapshot_subdomain}.archivebox.localhost:8000"
|
||||
assert original_host == f"{domain}.archivebox.localhost:8000"
|
||||
assert snapshot_host == f"{snapshot_subdomain}.archivebox.localhost:5797"
|
||||
assert original_host == f"{domain}.archivebox.localhost:5797"
|
||||
assert get_listen_subdomain(web_host) == "web"
|
||||
assert get_listen_subdomain(admin_host) == "admin"
|
||||
assert get_listen_subdomain(api_host) == "api"
|
||||
@@ -562,7 +601,7 @@ class TestUrlRouting:
|
||||
)
|
||||
finally:
|
||||
reset_result = run_archivebox_cmd(
|
||||
["config", "--set", "BASE_URL=http://archivebox.localhost:8000"],
|
||||
["config", "--set", "BASE_URL=http://archivebox.localhost:5797"],
|
||||
cwd=self.data_dir,
|
||||
)
|
||||
assert reset_result.returncode == 0, reset_result.stderr
|
||||
@@ -622,6 +661,351 @@ class TestUrlRouting:
|
||||
""",
|
||||
)
|
||||
|
||||
def test_private_snapshot_replay_cookie_authorizes_only_its_progress(self) -> None:
|
||||
self._run(
|
||||
"""
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
ensure_admin_user()
|
||||
snapshot = get_snapshot()
|
||||
original_config = snapshot.config
|
||||
Snapshot.objects.filter(pk=snapshot.pk).update(config={**snapshot.config, "PERMISSIONS": "private"})
|
||||
other_snapshot = Snapshot.objects.create(
|
||||
url="https://other-private-progress.example/",
|
||||
crawl=snapshot.crawl,
|
||||
config={"PERMISSIONS": "private"},
|
||||
)
|
||||
snapshot_host = get_snapshot_host(str(snapshot.id))
|
||||
admin_host = get_admin_host()
|
||||
|
||||
try:
|
||||
admin_client = Client()
|
||||
assert admin_client.login(username="testadmin", password="testpassword")
|
||||
grant_response = admin_client.get(
|
||||
f"/admin/core/snapshot/replay-auth/?snapshot={snapshot.id}&next=/index.html",
|
||||
HTTP_HOST=admin_host,
|
||||
)
|
||||
assert grant_response.status_code in (301, 302)
|
||||
grant_url = urlsplit(grant_response["Location"])
|
||||
assert grant_url.netloc == snapshot_host
|
||||
|
||||
replay_client = Client()
|
||||
auth_response = replay_client.get(
|
||||
f"{grant_url.path}?{grant_url.query}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
)
|
||||
assert auth_response.status_code in (301, 302)
|
||||
|
||||
snapshot_id = str(snapshot.id).replace("-", "")
|
||||
progress_response = replay_client.get(
|
||||
f"/progress.json?snapshot_id={snapshot_id}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
)
|
||||
assert progress_response.status_code == 200, response_body(progress_response)[:500]
|
||||
assert progress_response.json()["scope"]["snapshot_id"] == snapshot_id
|
||||
|
||||
other_snapshot_id = str(other_snapshot.id).replace("-", "")
|
||||
rejected = replay_client.get(
|
||||
f"/progress.json?snapshot_id={other_snapshot_id}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
)
|
||||
assert rejected.status_code == 403
|
||||
finally:
|
||||
other_snapshot.delete()
|
||||
Snapshot.objects.filter(pk=snapshot.pk).update(config=original_config)
|
||||
|
||||
print("OK")
|
||||
""",
|
||||
mode="safe-subdomains-fullreplay",
|
||||
)
|
||||
|
||||
def test_snapshot_payload_cache_policy_tracks_effective_permissions_and_revocation(self) -> None:
|
||||
self._run(
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from archivebox.crawls.models import Crawl
|
||||
|
||||
admin = ensure_admin_user()
|
||||
snapshot = get_snapshot()
|
||||
original_config = snapshot.config
|
||||
raw_name = "cache-policy.png"
|
||||
raw_bytes = b"\\x89PNG\\r\\n\\x1a\\ncache-policy-payload"
|
||||
raw_path = Path(snapshot.output_dir) / raw_name
|
||||
raw_path.write_bytes(raw_bytes)
|
||||
metadata_path = Path(snapshot.output_dir) / "index.jsonl"
|
||||
if not metadata_path.exists():
|
||||
metadata_path.write_text('{"type":"Snapshot"}\\n', encoding="utf-8")
|
||||
hashes_dir = Path(snapshot.output_dir) / "hashes"
|
||||
hashes_dir.mkdir(parents=True, exist_ok=True)
|
||||
(hashes_dir / "hashes.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"files": [
|
||||
{"path": raw_name, "hash": hashlib.sha256(raw_bytes).hexdigest()},
|
||||
{
|
||||
"path": "index.jsonl",
|
||||
"hash": hashlib.sha256(metadata_path.read_bytes()).hexdigest(),
|
||||
},
|
||||
],
|
||||
},
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
snapshot_host = get_snapshot_host(str(snapshot.id))
|
||||
|
||||
def cache_tokens(response):
|
||||
return {token.strip().lower() for token in response.headers.get("Cache-Control", "").split(",") if token.strip()}
|
||||
|
||||
def assert_cache(response, *expected):
|
||||
assert cache_tokens(response) == set(expected), (response.status_code, response.headers)
|
||||
|
||||
def assert_auth_vary(response):
|
||||
vary = {token.strip().lower() for token in response.headers.get("Vary", "").split(",") if token.strip()}
|
||||
assert {"cookie", "authorization"} <= vary, response.headers
|
||||
|
||||
try:
|
||||
snapshot.config = {**snapshot.config, "PERMISSIONS": "public"}
|
||||
snapshot.save(update_fields=["config"])
|
||||
snapshot.refresh_from_db()
|
||||
assert snapshot.permissions == "public"
|
||||
|
||||
public_client = Client()
|
||||
public_index = public_client.get("/index.html", HTTP_HOST=snapshot_host)
|
||||
assert public_index.status_code == 200
|
||||
assert_cache(public_index, "public", "max-age=60", "stale-while-revalidate=300")
|
||||
|
||||
public_files = public_client.get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
|
||||
assert public_files.status_code == 200
|
||||
assert raw_name.encode() in response_body(public_files)
|
||||
assert_cache(public_files, "public", "max-age=60", "stale-while-revalidate=300")
|
||||
|
||||
public_metadata = public_client.get("/index.jsonl", HTTP_HOST=snapshot_host)
|
||||
assert public_metadata.status_code == 200
|
||||
assert response_body(public_metadata) == metadata_path.read_bytes()
|
||||
assert_cache(public_metadata, "public", "max-age=31536000", "immutable")
|
||||
|
||||
public_raw = public_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host)
|
||||
assert public_raw.status_code == 200
|
||||
assert response_body(public_raw) == raw_bytes
|
||||
assert public_raw.headers.get("ETag")
|
||||
assert_cache(public_raw, "public", "max-age=604800", "immutable")
|
||||
|
||||
public_preview = public_client.get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
|
||||
assert public_preview.status_code == 200
|
||||
assert public_preview["Content-Type"].startswith("text/html")
|
||||
assert_cache(public_preview, "public", "max-age=60", "stale-while-revalidate=300")
|
||||
|
||||
public_range = public_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
|
||||
assert public_range.status_code == 206
|
||||
assert response_body(public_range) == raw_bytes[:8]
|
||||
assert_cache(public_range, "public", "max-age=604800", "immutable")
|
||||
|
||||
public_not_modified = public_client.get(
|
||||
f"/{raw_name}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
HTTP_IF_NONE_MATCH=public_raw["ETag"],
|
||||
)
|
||||
assert public_not_modified.status_code == 304
|
||||
assert_cache(public_not_modified, "public", "max-age=31536000", "immutable")
|
||||
|
||||
public_progress = public_client.get(
|
||||
f"/progress.json?snapshot_id={str(snapshot.id).replace('-', '')}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
)
|
||||
assert public_progress.status_code == 200
|
||||
assert_cache(public_progress, "private", "no-store")
|
||||
public_missing = public_client.get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
|
||||
assert public_missing.status_code == 404
|
||||
assert_cache(public_missing, "private", "no-store")
|
||||
public_redirect = public_client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
|
||||
assert public_redirect.status_code in (301, 302)
|
||||
assert public_redirect["Location"].startswith(f"http://{snapshot_host}")
|
||||
assert_cache(public_redirect, "private", "no-store")
|
||||
legacy_redirect = public_client.get(
|
||||
f"/{snapshot.legacy_archive_path}/index.html",
|
||||
HTTP_HOST=get_web_host(),
|
||||
)
|
||||
assert legacy_redirect.status_code in (301, 302)
|
||||
assert snapshot.url_path in legacy_redirect["Location"]
|
||||
assert_cache(legacy_redirect, "private", "no-store")
|
||||
|
||||
static_asset = public_client.get("/static/jquery.min.js", HTTP_HOST=snapshot_host)
|
||||
assert static_asset.status_code == 200
|
||||
assert static_asset.headers.get("ETag")
|
||||
assert_cache(static_asset, "public", "max-age=31536000", "immutable")
|
||||
static_not_modified = public_client.get(
|
||||
"/static/jquery.min.js",
|
||||
HTTP_HOST=snapshot_host,
|
||||
HTTP_IF_NONE_MATCH=static_asset["ETag"],
|
||||
)
|
||||
assert static_not_modified.status_code == 304
|
||||
assert_cache(static_not_modified, "public", "max-age=31536000", "immutable")
|
||||
|
||||
# Tightening permissions must still run authorization before honoring
|
||||
# a stale validator presented in a new request.
|
||||
snapshot.config = {**snapshot.config, "PERMISSIONS": "private"}
|
||||
snapshot.save(update_fields=["config"])
|
||||
snapshot.refresh_from_db()
|
||||
assert snapshot.permissions == "private"
|
||||
revoked = Client().get(
|
||||
f"/{raw_name}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
HTTP_IF_NONE_MATCH=public_raw["ETag"],
|
||||
)
|
||||
assert revoked.status_code in (301, 302)
|
||||
assert "/admin/core/snapshot/replay-auth/" in revoked["Location"]
|
||||
assert_cache(revoked, "private", "no-store")
|
||||
assert_auth_vary(revoked)
|
||||
private_files = Client().get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
|
||||
assert private_files.status_code in (301, 302)
|
||||
assert "/admin/core/snapshot/replay-auth/" in private_files["Location"]
|
||||
assert_cache(private_files, "private", "no-store")
|
||||
assert_auth_vary(private_files)
|
||||
|
||||
admin_client = Client()
|
||||
assert admin_client.login(username=admin.username, password="testpassword")
|
||||
grant = admin_client.get(
|
||||
f"/admin/core/snapshot/replay-auth/?snapshot={snapshot.id}&next=/{raw_name}",
|
||||
HTTP_HOST=get_admin_host(),
|
||||
)
|
||||
assert grant.status_code in (301, 302)
|
||||
grant_url = urlsplit(grant["Location"])
|
||||
replay_client = Client()
|
||||
replay_auth = replay_client.get(f"{grant_url.path}?{grant_url.query}", HTTP_HOST=snapshot_host)
|
||||
assert replay_auth.status_code in (301, 302)
|
||||
assert_cache(replay_auth, "private", "no-store")
|
||||
private_raw = replay_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host)
|
||||
assert private_raw.status_code == 200
|
||||
assert response_body(private_raw) == raw_bytes
|
||||
assert_cache(private_raw, "private", "max-age=604800", "immutable")
|
||||
assert_auth_vary(private_raw)
|
||||
private_index = replay_client.get("/index.html", HTTP_HOST=snapshot_host)
|
||||
assert private_index.status_code == 200
|
||||
assert_cache(private_index, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
assert_auth_vary(private_index)
|
||||
private_zip = replay_client.get("/?files=1&download=zip", HTTP_HOST=snapshot_host)
|
||||
assert private_zip.status_code == 200
|
||||
assert private_zip["Content-Type"] == "application/zip"
|
||||
assert response_body(private_zip).startswith(b"PK")
|
||||
assert_cache(private_zip, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
private_metadata = replay_client.get("/index.jsonl", HTTP_HOST=snapshot_host)
|
||||
assert private_metadata.status_code == 200
|
||||
assert response_body(private_metadata) == metadata_path.read_bytes()
|
||||
assert_cache(private_metadata, "private", "max-age=31536000", "immutable")
|
||||
private_preview = replay_client.get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
|
||||
assert private_preview.status_code == 200
|
||||
assert private_preview["Content-Type"].startswith("text/html")
|
||||
assert_cache(private_preview, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
private_range = replay_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
|
||||
assert private_range.status_code == 206
|
||||
assert response_body(private_range) == raw_bytes[:8]
|
||||
assert_cache(private_range, "private", "max-age=604800", "immutable")
|
||||
private_not_modified = replay_client.get(
|
||||
f"/{raw_name}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
HTTP_IF_NONE_MATCH=private_raw["ETag"],
|
||||
)
|
||||
assert private_not_modified.status_code == 304
|
||||
assert_cache(private_not_modified, "private", "max-age=31536000", "immutable")
|
||||
assert_auth_vary(private_not_modified)
|
||||
private_missing = replay_client.get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
|
||||
assert private_missing.status_code == 404
|
||||
assert_cache(private_missing, "private", "no-store")
|
||||
private_redirect = admin_client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
|
||||
assert private_redirect.status_code in (301, 302)
|
||||
assert private_redirect["Location"].startswith(f"http://{snapshot_host}")
|
||||
assert_cache(private_redirect, "private", "no-store")
|
||||
|
||||
snapshot.config = {**snapshot.config, "PERMISSIONS": "unlisted"}
|
||||
snapshot.save(update_fields=["config"])
|
||||
snapshot.refresh_from_db()
|
||||
assert snapshot.permissions == "unlisted"
|
||||
unlisted_raw = Client().get(f"/{raw_name}", HTTP_HOST=snapshot_host)
|
||||
assert unlisted_raw.status_code == 200
|
||||
assert response_body(unlisted_raw) == raw_bytes
|
||||
assert_cache(unlisted_raw, "private", "max-age=604800", "immutable")
|
||||
unlisted_index = Client().get("/index.html", HTTP_HOST=snapshot_host)
|
||||
assert unlisted_index.status_code == 200
|
||||
assert_cache(unlisted_index, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
unlisted_files = Client().get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
|
||||
assert unlisted_files.status_code == 200
|
||||
assert raw_name.encode() in response_body(unlisted_files)
|
||||
assert_cache(unlisted_files, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
unlisted_zip = Client().get(
|
||||
f"/{snapshot.url_path}/index.html?files=1&download=zip",
|
||||
HTTP_HOST=get_web_host(),
|
||||
)
|
||||
assert unlisted_zip.status_code == 200
|
||||
assert unlisted_zip["Content-Type"] == "application/zip"
|
||||
assert response_body(unlisted_zip).startswith(b"PK")
|
||||
assert_cache(unlisted_zip, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
unlisted_metadata = Client().get("/index.jsonl", HTTP_HOST=snapshot_host)
|
||||
assert unlisted_metadata.status_code == 200
|
||||
assert response_body(unlisted_metadata) == metadata_path.read_bytes()
|
||||
assert_cache(unlisted_metadata, "private", "max-age=31536000", "immutable")
|
||||
unlisted_preview = Client().get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
|
||||
assert unlisted_preview.status_code == 200
|
||||
assert unlisted_preview["Content-Type"].startswith("text/html")
|
||||
assert_cache(unlisted_preview, "private", "max-age=60", "stale-while-revalidate=300")
|
||||
unlisted_range = Client().get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
|
||||
assert unlisted_range.status_code == 206
|
||||
assert response_body(unlisted_range) == raw_bytes[:8]
|
||||
assert_cache(unlisted_range, "private", "max-age=604800", "immutable")
|
||||
unlisted_not_modified = Client().get(
|
||||
f"/{raw_name}",
|
||||
HTTP_HOST=snapshot_host,
|
||||
HTTP_IF_NONE_MATCH=unlisted_raw["ETag"],
|
||||
)
|
||||
assert unlisted_not_modified.status_code == 304
|
||||
assert_cache(unlisted_not_modified, "private", "max-age=31536000", "immutable")
|
||||
unlisted_missing = Client().get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
|
||||
assert unlisted_missing.status_code == 404
|
||||
assert_cache(unlisted_missing, "private", "no-store")
|
||||
unlisted_redirect = Client().get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
|
||||
assert unlisted_redirect.status_code in (301, 302)
|
||||
assert unlisted_redirect["Location"].startswith(f"http://{snapshot_host}")
|
||||
assert_cache(unlisted_redirect, "private", "no-store")
|
||||
|
||||
# Snapshots materialize their crawl's permission when no explicit
|
||||
# snapshot override is supplied, and each effective permission drives caching.
|
||||
for inherited_permission, expected_cache in (
|
||||
("public", ("public", "max-age=604800", "immutable")),
|
||||
("unlisted", ("private", "max-age=604800", "immutable")),
|
||||
("private", ("private", "no-store")),
|
||||
):
|
||||
crawl = Crawl.objects.create(
|
||||
urls=f"https://inherited-{inherited_permission}.example/",
|
||||
created_by=admin,
|
||||
config={"PERMISSIONS": inherited_permission},
|
||||
)
|
||||
inherited = Snapshot.objects.create(
|
||||
url=f"https://inherited-{inherited_permission}.example/",
|
||||
crawl=crawl,
|
||||
config={},
|
||||
)
|
||||
inherited.refresh_from_db()
|
||||
assert inherited.permissions == inherited_permission
|
||||
inherited.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
(inherited.output_dir / raw_name).write_bytes(raw_bytes)
|
||||
inherited_host = get_snapshot_host(str(inherited.id))
|
||||
response = Client().get(f"/{raw_name}", HTTP_HOST=inherited_host)
|
||||
if inherited_permission == "private":
|
||||
assert response.status_code in (301, 302)
|
||||
else:
|
||||
assert response.status_code == 200
|
||||
assert response_body(response) == raw_bytes
|
||||
assert_cache(response, *expected_cache)
|
||||
|
||||
print("OK")
|
||||
finally:
|
||||
Snapshot.objects.filter(pk=snapshot.pk).update(config=original_config)
|
||||
""",
|
||||
mode="safe-subdomains-fullreplay",
|
||||
)
|
||||
|
||||
def test_admin_login_next_allows_archivebox_hosts_only(self) -> None:
|
||||
self._run(
|
||||
"""
|
||||
@@ -634,7 +1018,7 @@ class TestUrlRouting:
|
||||
|
||||
allowed_next_urls = [
|
||||
"/admin/core/snapshot/",
|
||||
f"http://archivebox.localhost:8000/public/",
|
||||
f"http://archivebox.localhost:5797/public/",
|
||||
f"http://{web_host}/public/",
|
||||
f"http://{admin_host}/admin/core/snapshot/",
|
||||
f"http://{api_host}/api/v1/docs",
|
||||
@@ -1205,7 +1589,7 @@ class TestUrlRouting:
|
||||
mode="safe-subdomains-fullreplay",
|
||||
)
|
||||
finally:
|
||||
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
|
||||
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
|
||||
|
||||
def test_subdomain_replay_assets_route_without_base_url(self) -> None:
|
||||
lib_dir = self.data_dir / "test-lib"
|
||||
@@ -1233,7 +1617,7 @@ class TestUrlRouting:
|
||||
env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
|
||||
)
|
||||
finally:
|
||||
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
|
||||
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
|
||||
|
||||
def test_subdomain_replay_assets_use_derived_chromewebstore_extensions_dir(self) -> None:
|
||||
lib_dir = self.data_dir / "test-lib"
|
||||
@@ -1262,7 +1646,7 @@ class TestUrlRouting:
|
||||
env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
|
||||
)
|
||||
finally:
|
||||
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
|
||||
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
|
||||
|
||||
def test_onedomain_base_url_overrides_are_preserved_for_external_links(self) -> None:
|
||||
try:
|
||||
@@ -1289,7 +1673,7 @@ class TestUrlRouting:
|
||||
mode="safe-onedomain-nojsreplay",
|
||||
)
|
||||
finally:
|
||||
self._set_config("BASE_URL=http://archivebox.localhost:8000")
|
||||
self._set_config("BASE_URL=http://archivebox.localhost:5797")
|
||||
|
||||
def test_subdomain_snapshot_urls_inherit_https_archive_base_url(self) -> None:
|
||||
try:
|
||||
@@ -1310,7 +1694,7 @@ class TestUrlRouting:
|
||||
mode="safe-subdomains-fullreplay",
|
||||
)
|
||||
finally:
|
||||
self._set_config("BASE_URL=http://archivebox.localhost:8000")
|
||||
self._set_config("BASE_URL=http://archivebox.localhost:5797")
|
||||
|
||||
def test_template_and_admin_links(self) -> None:
|
||||
self._run(
|
||||
@@ -1343,7 +1727,7 @@ class TestUrlRouting:
|
||||
|
||||
resp = client.get("/public/", HTTP_HOST=web_host)
|
||||
assert resp.status_code in (301, 302)
|
||||
assert resp["Location"] == "http://admin.archivebox.localhost:8000/admin/core/snapshot/"
|
||||
assert resp["Location"] == "http://admin.archivebox.localhost:5797/admin/core/snapshot/"
|
||||
|
||||
resp = client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=web_host)
|
||||
assert resp.status_code == 200
|
||||
@@ -1518,12 +1902,12 @@ class TestUrlRouting:
|
||||
# Exercise the same real captured files with each request-time posture,
|
||||
# including both auto branches and an HTTPS canonical reverse-proxy URL.
|
||||
for mode, base in (
|
||||
("auto", "http://archivebox.localhost:8000"),
|
||||
("auto", "http://archivebox.example:8000"),
|
||||
("auto", "http://archivebox.localhost:5797"),
|
||||
("auto", "http://archivebox.example:5797"),
|
||||
("safe-subdomains-fullreplay", "https://archivebox.example"),
|
||||
("safe-onedomain-nojsreplay", "http://archivebox.example:8000"),
|
||||
("unsafe-onedomain-noadmin", "http://archivebox.example:8000"),
|
||||
("danger-onedomain-fullreplay", "http://archivebox.example:8000"),
|
||||
("safe-onedomain-nojsreplay", "http://archivebox.example:5797"),
|
||||
("unsafe-onedomain-noadmin", "http://archivebox.example:5797"),
|
||||
("danger-onedomain-fullreplay", "http://archivebox.example:5797"),
|
||||
):
|
||||
self._run(
|
||||
"""
|
||||
|
||||
@@ -8,7 +8,7 @@ class Command(BaseCommand):
|
||||
parser.add_argument(
|
||||
"--bind-url",
|
||||
default="",
|
||||
help="Runserver bind URL to watch, e.g. http://127.0.0.1:8000",
|
||||
help="Runserver bind URL to watch, e.g. http://127.0.0.1:5797",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interval",
|
||||
|
||||
@@ -330,13 +330,18 @@ def SUPERVISORD_PARENT_WATCHDOG_WORKER(
|
||||
}
|
||||
|
||||
|
||||
SERVER_WORKER = lambda host, port: {
|
||||
SERVER_WORKER = lambda host, port, extra_port=None: {
|
||||
"name": "worker_daphne",
|
||||
"command": _shell_join(
|
||||
[
|
||||
str(resolve_env_binary("daphne")),
|
||||
f"--bind={host}",
|
||||
f"--port={port}",
|
||||
*(
|
||||
["--endpoint", "tcp:port={}:interface={}".format(extra_port, host.strip("[]").replace(":", r"\:"))]
|
||||
if extra_port is not None
|
||||
else []
|
||||
),
|
||||
"archivebox.core.asgi:application",
|
||||
],
|
||||
),
|
||||
@@ -1115,7 +1120,7 @@ def format_runtime_components(components: list[str] | tuple[str, ...]) -> str:
|
||||
def worker_runtime_component(worker_name: str) -> str | None:
|
||||
if worker_name in {"worker_runner", "worker_runner_watch"} or worker_name.startswith("worker_runner_"):
|
||||
return "orchestrator"
|
||||
if worker_name in {"worker_daphne", "worker_runserver"}:
|
||||
if worker_name in {"worker_daphne", "worker_runserver", "worker_runserver_compatibility"}:
|
||||
return "server"
|
||||
if worker_name == "worker_sonic":
|
||||
return "sonic"
|
||||
@@ -1140,6 +1145,9 @@ def active_supervisord_runtime_components(*, supervisor=None) -> list[str]:
|
||||
|
||||
def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reload: bool, nothreading: bool, supervisor=None):
|
||||
bind_url = f"http://{host}:{port}"
|
||||
# Docker does not expose its host port mappings to the container. Keep both
|
||||
# image defaults reachable so existing mappings survive an image upgrade.
|
||||
extra_port = {"5797": "8000", "8000": "5797"}.get(str(port)) if config.IN_DOCKER else None
|
||||
|
||||
if debug:
|
||||
server_worker = RUNSERVER_WORKER(host=host, port=port, reload=reload, nothreading=nothreading)
|
||||
@@ -1150,7 +1158,7 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
|
||||
if reload:
|
||||
log_files.insert(1, "logs/worker_runner_watch.log")
|
||||
else:
|
||||
server_worker = SERVER_WORKER(host=host, port=port)
|
||||
server_worker = SERVER_WORKER(host=host, port=port, extra_port=extra_port)
|
||||
bg_workers = [(RUNNER_WORKER(), False)]
|
||||
log_files = ["logs/worker_daphne.log", "logs/worker_runner.log"]
|
||||
|
||||
@@ -1178,6 +1186,12 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
|
||||
log_files.append(str(sonic_worker["stdout_logfile"]))
|
||||
|
||||
workers = [(server_worker, False), *bg_workers]
|
||||
if debug and extra_port is not None:
|
||||
compatibility_worker = RUNSERVER_WORKER(host=host, port=extra_port, reload=reload, nothreading=nothreading)
|
||||
compatibility_worker["name"] = "worker_runserver_compatibility"
|
||||
compatibility_worker["stdout_logfile"] = "logs/worker_runserver_compatibility.log"
|
||||
workers.insert(1, (compatibility_worker, False))
|
||||
log_files.append(compatibility_worker["stdout_logfile"])
|
||||
if os.environ.get("DISPLAY") and (config.IN_DOCKER or os.environ.get("ARCHIVEBOX_VNC_PERSONA")):
|
||||
vnc_persona = os.environ.get("ARCHIVEBOX_VNC_PERSONA") or config.DEFAULT_PERSONA
|
||||
workers.append(
|
||||
@@ -1478,7 +1492,7 @@ def stop_stale_sonic_processes(
|
||||
|
||||
def start_server_workers(
|
||||
host="0.0.0.0",
|
||||
port="8000",
|
||||
port="5797",
|
||||
daemonize=False,
|
||||
debug=False,
|
||||
reload=False,
|
||||
|
||||
@@ -111,7 +111,7 @@ if [[ -n "$DEPLOY_EXPECT_VERSION" && "$VERSION_OUTPUT" != *"ArchiveBox v${DEPLOY
|
||||
fi
|
||||
|
||||
echo "[+] Health check:"
|
||||
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:8000/health/ </dev/null
|
||||
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:5797/health/ </dev/null
|
||||
REMOTE
|
||||
|
||||
echo "[√] Demo deploy finished."
|
||||
@@ -230,11 +230,14 @@ assert_writable_dir "$DATA_DIR/logs"
|
||||
assert_writable_dir "$DATA_DIR/archive"
|
||||
assert_writable_dir "$PERSONAS_DIR/Default/chrome_profile"
|
||||
|
||||
# check if novnc x11 $DISPLAY is available
|
||||
export DISPLAY="${DISPLAY:-"novnc:0.0"}"
|
||||
if ! xdpyinfo > /dev/null 2>&1; then
|
||||
# cant connect to x11 display, unset it so that chrome doesn't try to connect to it and hang indefinitely
|
||||
unset DISPLAY
|
||||
# Preserve an explicit DISPLAY (including an empty value to disable X11).
|
||||
# Only probe the implicit noVNC default: a missing probe or unavailable server
|
||||
# must not silently discard the user's display configuration.
|
||||
if [[ -z "${DISPLAY+x}" ]]; then
|
||||
export DISPLAY="novnc:0.0"
|
||||
if ! xdpyinfo > /dev/null 2>&1; then
|
||||
unset DISPLAY
|
||||
fi
|
||||
fi
|
||||
|
||||
# Active browser processes do not survive container restarts, but their lock
|
||||
@@ -250,6 +253,9 @@ find /tmp "$TMP_DIR" -maxdepth 1 -type d -name "archivebox-chrome-profile.*" -mm
|
||||
|
||||
|
||||
ensure_dir "/home/$ARCHIVEBOX_USER"
|
||||
# A nested read-only browser-profile mount can create this parent as root.
|
||||
# OpenCode's installer needs to create a sibling config directory as archivebox.
|
||||
ensure_dir "${XDG_CONFIG_HOME:-/home/$ARCHIVEBOX_USER/.config}"
|
||||
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR"
|
||||
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR/semaphores"
|
||||
ensure_small_runtime_tree "$UV_CACHE_DIR"
|
||||
@@ -365,7 +371,7 @@ else
|
||||
# e.g. "docker run archivebox help"
|
||||
# "docker run archivebox add --depth=1 https://example.com"
|
||||
# "docker run archivebox manage createsupseruser"
|
||||
# "docker run archivebox server 0.0.0.0:8000"
|
||||
# "docker run archivebox server 0.0.0.0:5797"
|
||||
if [[ "$(id -u)" == "0" ]]; then
|
||||
exec setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$ARCHIVEBOX_BIN_PATH" "$@"
|
||||
else
|
||||
|
||||
@@ -128,7 +128,8 @@ def render_gallery(gallery_header: str, gallery_sections: str) -> str:
|
||||
return (
|
||||
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
|
||||
'<meta name="viewport" content="width=device-width, initial-scale=1">'
|
||||
"<title>ArchiveBox UI Screenshots</title><style>"
|
||||
+ (REPO_DIR / "bin/templates/screenshots-head.html").read_text(encoding="utf-8")
|
||||
+ "<style>"
|
||||
":root{--ink:#28252a;--muted:#716c74;--accent:#9b2854;--paper:#fcfaf7;--line:#e9e2e3}"
|
||||
"*{box-sizing:border-box}body{margin:0;font:16px/1.75 system-ui,sans-serif;"
|
||||
"background:var(--paper);color:var(--ink)}.screenshot-content{max-width:1100px;margin:0 auto;padding:28px}"
|
||||
|
||||
@@ -39,8 +39,8 @@ if os.environ.get("GH_TOKEN"):
|
||||
# Published captures remain available even after Actions artifacts expire.
|
||||
base, manifest_name = {
|
||||
"archivebox": ("https://archivebox.io/screenshots/", "build.json"),
|
||||
"ios": ("https://archivebox.github.io/ios-archivebox/screenshots/", "manifest.json"),
|
||||
"extension": ("https://archivebox.github.io/archivebox-browser-extension/screenshots/", "manifest.json"),
|
||||
"ios": ("https://app.archivebox.io/screenshots/", "manifest.json"),
|
||||
"extension": ("https://extension.archivebox.io/screenshots/", "manifest.json"),
|
||||
}[args.kind]
|
||||
|
||||
|
||||
|
||||
+14
-14
@@ -189,9 +189,9 @@ docker_run_archivebox_install() {
|
||||
|
||||
docker_run_archivebox_server() {
|
||||
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
|
||||
"$DOCKER_BINARY" run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
|
||||
"$DOCKER_BINARY" run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 5797:5797 --name=archivebox "$ARCHIVEBOX_IMAGE"
|
||||
else
|
||||
"$DOCKER_BINARY" run -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
|
||||
"$DOCKER_BINARY" run -v "$PWD":/data -d -p 5797:5797 --name=archivebox "$ARCHIVEBOX_IMAGE"
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -204,8 +204,8 @@ docker_compose_run_archivebox() {
|
||||
}
|
||||
|
||||
wait_for_archivebox() {
|
||||
url="http://127.0.0.1:8000/health/"
|
||||
host_header="admin.archivebox.localhost:8000"
|
||||
url="http://127.0.0.1:5797/health/"
|
||||
host_header="admin.archivebox.localhost:5797"
|
||||
attempts=60
|
||||
attempt=1
|
||||
|
||||
@@ -224,7 +224,7 @@ wait_for_archivebox() {
|
||||
|
||||
open_archivebox() {
|
||||
if [ -n "$OPEN_BINARY" ] && [ -t 1 ]; then
|
||||
"$OPEN_BINARY" "http://admin.archivebox.localhost:8000/admin/" || true
|
||||
"$OPEN_BINARY" "http://admin.archivebox.localhost:5797/admin/" || true
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -410,13 +410,13 @@ if [ "$DOCKER_IMAGE_READY" = "true" ] && "$DOCKER_BINARY" compose version > /dev
|
||||
wait_for_archivebox
|
||||
open_archivebox
|
||||
echo
|
||||
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo " cd $ARCHIVEBOX_HOME_DIR"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose ps"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose down"
|
||||
echo " ${FOLLOWUP_SUDO}env ARCHIVEBOX_IMAGE=$ARCHIVEBOX_IMAGE $CONTAINER_ENGINE_NAME compose pull"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose up"
|
||||
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
|
||||
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
|
||||
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose exec archivebox archivebox add 'https://example.com'"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose exec archivebox archivebox list"
|
||||
@@ -438,13 +438,13 @@ elif [ "$DOCKER_IMAGE_READY" = "true" ]; then
|
||||
wait_for_archivebox
|
||||
open_archivebox
|
||||
echo
|
||||
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo " cd $ARCHIVEBOX_DATA_DIR"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME ps --filter name=archivebox"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME rm -f archivebox"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME pull $ARCHIVEBOX_IMAGE"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -d -p 8000:8000 --name=archivebox $ARCHIVEBOX_IMAGE"
|
||||
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -d -p 5797:5797 --name=archivebox $ARCHIVEBOX_IMAGE"
|
||||
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
|
||||
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -it $ARCHIVEBOX_IMAGE add 'https://example.com'"
|
||||
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -it $ARCHIVEBOX_IMAGE list"
|
||||
@@ -501,14 +501,14 @@ select_archivebox_lib_dir
|
||||
# init shows version output at the end too
|
||||
echo
|
||||
echo "[+] Starting ArchiveBox server using: archivebox server --daemonize..."
|
||||
"$ARCHIVEBOX_BINARY" server --daemonize 0.0.0.0:8000
|
||||
"$ARCHIVEBOX_BINARY" server --daemonize 0.0.0.0:5797
|
||||
wait_for_archivebox
|
||||
open_archivebox
|
||||
echo
|
||||
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
|
||||
echo " cd $ARCHIVEBOX_DATA_DIR # see your data dir"
|
||||
echo " ${FOLLOWUP_SUDO}archivebox server 0.0.0.0:8000 # start server process"
|
||||
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
|
||||
echo " ${FOLLOWUP_SUDO}archivebox server 0.0.0.0:5797 # start server process"
|
||||
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
|
||||
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
|
||||
echo " ps aux | grep archivebox # see server process pid"
|
||||
echo " ${FOLLOWUP_SUDO}pkill -f 'archivebox server' # stop the server"
|
||||
|
||||
@@ -4,7 +4,7 @@ const fs = require('fs');
|
||||
const path = require('path');
|
||||
const puppeteer = require('puppeteer');
|
||||
|
||||
const DEFAULT_URL = 'http://web.archivebox.localhost:8000/add/';
|
||||
const DEFAULT_URL = 'http://web.archivebox.localhost:5797/add/';
|
||||
const DEFAULT_OUTPUT = 'tmp/add-page.png';
|
||||
|
||||
function usage() {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
|
||||
<div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
|
||||
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps & tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS & iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS & iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
|
||||
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps & tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS & iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS & iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abxpkg.archivebox.io/">Package &Line truncated
|
||||
|
||||
<div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security & privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
|
||||
</footer>
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
<title>ArchiveBox UI Screenshots</title>
|
||||
<link rel="canonical" href="https://archivebox.io/screenshots/">
|
||||
<meta name="description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
|
||||
<meta name="robots" content="index,follow,max-image-preview:large">
|
||||
<meta name="theme-color" content="#9b2854">
|
||||
<meta property="og:type" content="website">
|
||||
<meta property="og:site_name" content="ArchiveBox">
|
||||
<meta property="og:locale" content="en_US">
|
||||
<meta property="og:url" content="https://archivebox.io/screenshots/">
|
||||
<meta property="og:title" content="ArchiveBox UI Screenshots">
|
||||
<meta property="og:description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
|
||||
<meta property="og:image" content="https://archivebox.io/assets/social-card.png">
|
||||
<meta property="og:image:type" content="image/png">
|
||||
<meta property="og:image:width" content="1200">
|
||||
<meta property="og:image:height" content="630">
|
||||
<meta property="og:image:alt" content="ArchiveBox. ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
|
||||
<meta name="twitter:card" content="summary_large_image">
|
||||
<meta name="twitter:site" content="@ArchiveBoxApp">
|
||||
<meta name="twitter:title" content="ArchiveBox UI Screenshots">
|
||||
<meta name="twitter:description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
|
||||
<meta name="twitter:image" content="https://archivebox.io/assets/social-card.png">
|
||||
<meta name="twitter:image:alt" content="ArchiveBox. ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
|
||||
<link rel="icon" type="image/x-icon" href="https://archivebox.io/assets/favicon.ico">
|
||||
<link rel="icon" type="image/png" sizes="192x192" href="https://archivebox.io/assets/icon.png">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="https://archivebox.io/assets/apple-touch-icon.png">
|
||||
@@ -1,7 +1,14 @@
|
||||
<header class="abx-header">
|
||||
<a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">UI Screenshots</span></a>
|
||||
<nav class="abx-nav" aria-label="Main navigation">
|
||||
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS & iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
|
||||
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links">
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://archivebox.io/#install-compose">ArchiveBox Server</a><div class="abx-platforms"><a class="abx-platform abx-platform-docker" href="https://archivebox.io/#install-compose" aria-label="ArchiveBox Server for Docker" title="ArchiveBox Server for Docker"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M349.9 236.3l-66.1 0 0-59.4 66.1 0 0 59.4zm0-204.3l-66.1 0 0 60.7 66.1 0 0-60.7zm78.2 144.8l-66.1 0 0 59.4 66.1 0 0-59.4zM271.8 104.7l-66.1 0 0 60.1 66.1 0 0-60.1zm78.1 0l-66.1 0 0 60.1 66.1 0 0-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7l-434.7 0c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zM115.6 176.8l-66 0 0 59.4 66.1 0 0-59.4-.1 0zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm-78.1-72.1l-66.1 0 0 60.1 66.1 0 0-60.1z"/></svg></a><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/#archivebox-serverapp" aria-label="ArchiveBox Server for macOS" title="ArchiveBox Server for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://electron.archivebox.io/" aria-label="ArchiveBox Server for Windows" title="ArchiveBox Server for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://archivebox.io/#install-apt" aria-label="ArchiveBox Server for Linux" title="ArchiveBox Server for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-1Line truncated
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Desktop</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Desktop for macOS" title="ArchiveBox Desktop for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://windows.archivebox.io/" aria-label="ArchiveBox Desktop for Windows" title="ArchiveBox Desktop for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://linux.archivebox.io/" aria-label="ArchiveBox Desktop for Linux" title="ArchiveBox Desktop for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-10 14.1-10.5 4.7-4.4 13.5-14.2 27.9-14.2 7.1 0 15.6 2.3 25.9 8.9 6.3 4.1 11.3 4.4 22.6 9.3 8.4 3.5 13.7 9.7 10.5 18.2-2.6 7.1-11 14.4-22.7 18.1-11.1 3.6-19.8 16-38.2 14.9-3.9-.2-7-1-9.6-2.1-8-3.5-12.2-10.4-20-15-8.6-4.8-13.2-10.4-14.7-15.3-1.4-4.9 0-9 4.2-12.3zm3.3 334c-2.7 35.1-43.9 34.4-75.3 18-29.9-15.8-68.6-6.5-76.5-21.9-2.4-4.7-2.4-12.7 2.6-26.4l0-.2c2.4-7.6 .6-16-.6-23.9-1.2-7.8-1.8-15 .9-20 3.5-6.7 8.5-9.1 14.8-11.3 10.3-3.7 11.8-3.4 19.6-9.9 5.5-5.7 9.5-12.9 14.3-18 5.1-5.5 10-8.1 17.7-6.9 8.1 1.2 15.1 6.8 21.9 16l19.6 35.6c9.5 19.9 43.1 48.4 41 68.9zm-1.4-25.9c-4.1-6.6-9.6-13.6-14.4-19.6 7.1 0 14.2-2.2 16.7-8.9 2.3-6.2 0-14.9-7.4-24.9-13.5-18.2-38.3-32.5-38.3-32.5-13.5-8.4-21.1-18.7-24.6-29.9s-3-23.3-.3-35.2c5.2-22.9 18.6-45.2 27.2-59.2 2.3-1.7 .8 3.2-8.7 20.8-8.5 16.1-24.4 53.3-2.6 82.4 .6-20.7 5.5-41.8 13.8-61.5 12-27.4 37.3-74.9 39.3-112.7 1.1 .8 4.6 3.2 6.2 4.1 4.6 2.7 8.1 6.7 12.6 10.3 12.4 10 28.5 9.2 42.4 1.2 6.2-3.5 11.2-7.5 15.9-9 9.9-3.1 17.8-8.6 22.3-15 7.7 30.4 25.7 74.3 37.2 95.7 6.1 11.4 18.3 35.5 23.6 64.6 3.3-.1 7 .4 10.9 1.4 13.8-35.7-11.7-74.2-23.3-84.9-4.7-4.6-4.9-6.6-2.6-6.5 12.6 11.2 29.2 33.7 35.2 59 2.8 11.6 3.3 23.7 .4 35.7 16.4 6.8 35.9 17.9 30.7 34.8-2.2-.1-3.2 0-4Line truncated
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Mobile</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Mobile for iOS" title="ArchiveBox Mobile for iOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-android" href="https://android.archivebox.io/" aria-label="ArchiveBox Mobile for Android" title="ArchiveBox Mobile for Android"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!-- Font Awesome Free 5.0.13 by Fonticons, Inc. https://fontawesome.com License: https://creativecommons.org/licenses/by/4.0/ --><path d="M89.6 204.5v115.8c0 15.4-12.1 27.7-27.5 27.7-15.3 0-30.1-12.4-30.1-27.7V204.5c0-15.1 14.8-27.5 30.1-27.5 15.1 0 27.5 12.4 27.5 27.5zm10.8 157c0 16.4 13.2 29.6 29.6 29.6h19.9l.3 61.1c0 36.9 55.2 36.6 55.2 0v-61.1h37.2v61.1c0 36.7 55.5 36.8 55.5 0v-61.1h20.2c16.2 0 29.4-13.2 29.4-29.6V182.1H100.4v179.4zm248-189.1H99.3c0-42.8 25.6-80 63.6-99.4l-19.1-35.3c-2.8-4.9 4.3-8 6.7-3.8l19.4 35.6c34.9-15.5 75-14.7 108.3 0L297.5 34c2.5-4.3 9.5-1.1 6.7 3.8L285.1 73c37.7 19.4 63.3 56.6 63.3 99.4zm-170.7-55.5c0-5.7-4.6-10.5-10.5-10.5-5.7 0-10.2 4.8-10.2 10.5s4.6 10.5 10.2 10.5c5.9 0 10.5-4.8 10.5-10.5zm113.4 0c0-5.7-4.6-10.5-10.2-10.5-5.9 0-10.5 4.8-10.5 10.5s4.6 10.5 10.5 10.5c5.6 0 10.2-4.8 10.2-10.5zm94.8 60.1c-15.1 0-27.5 12.1-27.5 27.5v115.8c0 15.4 12.4 27.7 27.5 27.7 15.4 0 30.1-12.4 30.1-27.7V204.5c0-15.4-14.8-27.5-30.1-27.5z"/></svg></a></div></div>
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><div class="abx-platforms"><a class="abx-platform abx-platform-chrome" href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj" aria-label="ArchiveBox Browser Extension for Chrome" title="ArchiveBox Browser Extension for Chrome"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2024 Fonticons, Inc. --><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg></a><a class="abx-platform abx-platform-safari" href="https://extension.archivebox.io/" aria-label="ArchiveBox Browser Extension for Safari" title="ArchiveBox Browser Extension for Safari"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M274.7 274.7L237.3 237.3 166 346 274.7 274.7zM256 8a248 248 0 1 0 0 496 248 248 0 1 0 0-496zM411.8 182.8l14.8-6.1c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.8 1 2.6 1.7 1.3 1.6 1.7 2.6 .6 2 .6 3.1-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.3-2.6 1.7L418 197.6c-2 .8-4.2 .8-6.1 0s-3.5-2.4-4.3-4.3-.8-4.2 0-6.1 2.4-3.5 4.3-4.3zM314.4 94l6.1-14.8c.8-2 2.4-3.5 4.3-4.3s4.2-.8 6.1 0 3.5 2.4 4.3 4.3 .8 4.2 0 6.1l-6.1 14.8c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0c-1-.4-1.9-1-2.6-1.7s-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1zM256 60c2.1 0 4.2 .8 5.7 2.3S264 65.9 264 68l0 16c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7S253.9 60 256 60zM181 74.9c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.9 1 2.6 1.7 1.3 1.6 1.7 2.6L197.6 94c.4 1 .7 2 .7 3.1s-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.4-2.6 1.8-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.8-1.3-1.7-1.7-2.7l-6.1-14.8c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3zm-63.6 42.5c1.5-1.5 3.5-2.3 5.7-2.3s4.2 .8 5.7 2.3L140 128.7c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3l-11.3-11.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6l0 0zM60 256c0-2.1 .8-4.2 2.3-5.7S65.9 248 68 248l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3l-16 0c-2.1 0-4.2-.8-5.7-2.3S60 258.1 60 256zm40.2 73.2l-14.8 6.1c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1 1-1.9 1.7-2.6 1.6-1.3 2.6-1.7L94 314.4c2-.8 4.2-.8 6.1 0s3.5 2.4 4.3 4.3 .8 4.2 0 6.1-2.4 3.5-4.3 4.3zm4.3-136c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1 2.4-3.5 4.3-4.3 4.2-.8 6.1 0l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1l0 0zM197.6 418l-6.1 14.8c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1l6.1-14.8c.4-1 1-1.9 1.7-2.7s1.6-1.4 2.6-1.8 2-.6 3.1-.6 2.1 .2 3.1 .6 1.9 1 2.6 1.8 1.3 1.7 1.7 2.6 .6 2 .6 3.1-.2 2.1-.7 3.1zM264 444c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7s3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3 2.3 3.5 2.3 5.7l0 16zm67-6.9c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6L314.4 418c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3 4.2-.8 6.1 0 3.5 2.4 4.3 4.3l6.1 14.8c.8 2 .8 4.2 0 6.1s-2.4 3.5-4.3 4.3zm63.6-42.5c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3L372 383.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6 3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3l11.3 11.3c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6l0 0zM286.2 286.2L110.3 401.7 225.8 225.8 401.7 110.3 286.2 286.2zM437.1 331c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1c.4-1 1-1.9 1.7-2.6s1.6-1.3 2.6-1.7 2-.6 3.1-.6 2.1 .2 3.1 .6l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1zm6.9-67l-16 0c-2.1 0-4.2-.8-5.7-2.3s-2.3-3.5-2.3-5.7 .8-4.2 2.3-5.7 3.5-2.3 5.7-2.3l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3z"/></svg></a><a class="abx-platform abx-platform-firefox" href="https://addons.mozilla.org/firefox/addon/archivebox-exporter/" aria-label="ArchiveBox Browser Extension for Firefox" title="ArchiveBox Browser Extension for Firefox"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512Line truncated
|
||||
<div class="abx-app-row abx-plugin-row"><a class="abx-app-name" href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><div class="abx-platforms abx-icon-stack"><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Google Drive storage" title="Google Drive storage"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m38.7 419.3 22.6 39c4.7 8.2 11.4 14.7 19.4 19.4l80.6-139.6H0c0 9.1 2.3 18.2 7 26.4z" style="fill:#0066da"/><path d="M256 173.9 175.4 34.3c-7.9 4.7-14.7 11.1-19.4 19.4L7 311.7c-4.6 8-7 17.1-7 26.4h161.3z" style="fill:#00ac47"/><path d="M431.4 477.7c7.9-4.7 14.7-11.1 19.4-19.4l9.4-16.1 44.9-77.7c4.7-8.2 7-17.3 7-26.4H350.7l34.3 67.4z" style="fill:#ea4335"/><path d="m256 173.9 80.6-139.6c-7.9-4.7-17-7-26.4-7H201.8c-9.4 0-18.5 2.6-26.4 7z" style="fill:#00832d"/><path d="M350.7 338.1H161.3L80.6 477.7c7.9 4.7 17 7 26.4 7h298c9.4 0 18.5-2.6 26.4-7z" style="fill:#2684fc"/><path d="M430.5 182.7 356 53.7c-4.7-8.2-11.4-14.7-19.4-19.4L256 173.9l94.7 164.2h161c0-9.1-2.3-18.2-7-26.4z" style="fill:#ffba00"/></svg></a><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Amazon S3 storage" title="Amazon S3 storage"><svg aria-hidden="true" focusable="false" fill="#569A31" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M20.913 13.147l.12-.895c.947.576 1.258.922 1.354 1.071-.16.031-.562.046-1.474-.176zm-2.174 7.988a.547.547 0 0 0-.005.073c0 .084-.207.405-1.124.768a10.28 10.28 0 0 1-1.438.432c-1.405.325-3.128.504-4.853.504-4.612 0-7.412-1.184-7.412-1.704a.547.547 0 0 0-.005-.073L1.81 5.602c.135.078.28.154.432.227.042.02.086.038.128.057.134.062.272.122.417.18l.179.069c.154.058.314.114.478.168.043.013.084.029.13.043.207.065.423.127.646.187l.176.044c.175.044.353.087.534.127a23.414 23.414 0 0 0 .843.17l.121.023c.252.045.508.085.768.122.071.011.144.02.216.03.2.027.4.053.604.077l.24.027c.245.026.49.05.74.07l.081.009c.275.022.552.04.83.056l.233.012c.21.01.422.018.633.025a33.088 33.088 0 0 0 2.795-.026l.232-.011c.278-.016.555-.034.83-.056l.08-.008c.25-.02.497-.045.742-.072l.238-.026c.205-.024.408-.05.609-.077.07-.01.141-.019.211-.03.261-.037.519-.078.772-.122l.111-.02c.215-.04.427-.082.634-.125l.212-.047c.186-.041.368-.085.546-.13l.166-.042c.225-.06.444-.122.654-.189.04-.012.077-.026.115-.038a10.6 10.6 0 0 0 .493-.173c.058-.021.114-.044.17-.066.15-.06.293-.12.43-.185.038-.017.079-.034.116-.052.153-.073.3-.15.436-.228l-.976 7.245c-2.488-.78-5.805-2.292-7.311-3a1.09 1.09 0 0 0-1.088-1.085c-.6 0-1.088.489-1.088 1.088 0 .6.488 1.089 1.088 1.089.196 0 .378-.056.537-.148 1.72.812 5.144 2.367 7.715 3.15zm-7.42-20.047c5.677 0 9.676 1.759 9.75 2.736l-.014.113c-.01.033-.031.067-.048.101-.015.028-.026.057-.047.087-.024.033-.058.068-.09.102-.028.03-.051.06-.084.09-.038.035-.087.07-.133.105-.04.03-.074.06-.119.091-.053.036-.116.071-.177.107-.05.03-.095.06-.15.09-.068.036-.147.073-.222.11-.059.028-.114.057-.177.085-.084.038-.177.074-.268.111-.068.027-.13.054-.203.082-.097.036-.205.072-.31.107-.075.026-.148.053-.228.079-.111.035-.233.069-.35.103-.085.024-.165.05-.253.073-.124.034-.258.065-.389.098-.093.022-.181.046-.278.068-.139.032-.287.061-.433.091-.098.02-.191.041-.293.06-.155.03-.32.057-.482.084-.1.018-.198.036-.302.052-.166.026-.342.048-.515.072-.11.014-.213.03-.325.044-.181.023-.372.041-.56.06-.11.012-.218.025-.332.036-.188.016-.386.029-.58.043-.122.009-.24.02-.364.028-.207.012-.422.02-.635.028-.12.005-.234.012-.354.016a35.605 35.605 0 0 1-2.069 0c-.12-.004-.234-.011-.352-.016-.214-.008-.43-.016-.637-.028-.122-.008-.238-.02-.36-.027-.195-.015-.394-.028-.584-.044-.11-.01-.215-.024-.324-.035-.19-.02-.384-.038-.568-.06l-.315-.044c-.176-.024-.355-.046-.525-.073-.1-.015-.192-.033-.29-.05-.167-.028-.335-.055-.494-.086-.096-.018-.183-.038-.276-.056-.151-.032-.305-.062-.45-.095-.09-.02-.173-.043-.26-.064-.138-.034-.277-.067-.407-.102-.082-.022-.157-.046-.235-.069a11.75 11.75 0 0 1-.368-.108c-.075-.024-.141-.049-.213-.073-.11-.037-.223-.075-.325-.113-.067-.025-.125-.051-.188-.077-.096-.038-.195-.076-.282-.115-.06-.027-.11-.054-.166-.08-.08-.039-.162-.077-.233-.116-.052-.028-.094-.055-.142-.084-.063-.038-.13-.075-.185-.113-.043-.029-.075-.058-.113-.086-.048-.037-.098-.073-.139-.11-.032-.029-.054-.057-.08-.087-.033-.035-.069-.07-.093-.104-.02-.03-.031-.058-.046-.086-.018-.035-.039-.068-.049-.102l-.015-.113c.076-.977 4.074-2.736 9.748-2.736zm12.182 12.124c-.118-.628-.84-1.291-2.31-2.128l.963-7.16a.531.531 0 0 0 .005-.073C22.16 1.581 16.447 0 11.32 0 6.194 0 .482 1.581.482 3.851a.58.58 0 0 0 .005.072L2.819 21.25c.071 2.002 5.236 2.75 8.5 2.75 1.805 0 3.615-.188 5.098-.531.598-.138 1.133-.3 1.592-.48 1.18-.467 1.789-1.053 1.813-1.739l.945-7.018c.557.131 1.016.197 1.389.197.54 0 .902-.137 1.134-.413aLine truncated
|
||||
<div class="abx-app-row abx-tools-row"><a class="abx-app-name" href="https://abx-dl.archivebox.io/">Tools</a><div class="abx-platforms"><a class="abx-tool" href="https://abx-dl.archivebox.io/"><span>One-shot <code>abx-dl</code> CLI</span></a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/mcp">MCP</a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage">API</a><a class="abx-tool" href="https://web.archivebox.io">Demo</a></div></div>
|
||||
</div></details>
|
||||
<a href="https://archivebox.io/screenshots/">Screenshots</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
|
||||
<a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
|
||||
<a class="abx-cta" href="https://archivebox.io/#install">Get started</a>
|
||||
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Real Compose regression checks. Build the image under test first, then run:
|
||||
# IMAGE=archivebox/archivebox:issue-1872 ./bin/validate_docker_display.sh
|
||||
set -Eeuo pipefail
|
||||
|
||||
export IMAGE="${IMAGE:-archivebox/archivebox:dev}"
|
||||
validation_dir="$(mktemp -d "${TMPDIR:-/tmp}/archivebox-display.XXXXXX")"
|
||||
compose=(docker compose --project-name "archivebox-display-$$" --file "$validation_dir/compose.yml")
|
||||
cleanup() {
|
||||
"${compose[@]}" down --volumes --remove-orphans
|
||||
echo "Validation files: $validation_dir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
cat > "$validation_dir/compose.yml" <<'YAML'
|
||||
services:
|
||||
archivebox:
|
||||
image: ${IMAGE}
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- ./data/personas:/data/personas
|
||||
- tmp:/tmp/archivebox
|
||||
shm_size: 1gb
|
||||
novnc:
|
||||
image: theasp/novnc:latest
|
||||
environment:
|
||||
DISPLAY_WIDTH: 1280
|
||||
DISPLAY_HEIGHT: 800
|
||||
RUN_XTERM: "no"
|
||||
healthcheck:
|
||||
test: ["CMD", "xdpyinfo", "-display", ":0"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 10
|
||||
volumes:
|
||||
tmp:
|
||||
YAML
|
||||
|
||||
"${compose[@]}" up -d --wait novnc
|
||||
|
||||
# Check the real runtime dependency as well as successful X11 connectivity.
|
||||
"${compose[@]}" run --rm archivebox sh -ec '
|
||||
command -v xdpyinfo
|
||||
test "$DISPLAY" = novnc:0.0
|
||||
xdpyinfo >/dev/null
|
||||
'
|
||||
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
|
||||
test "$DISPLAY" = novnc:0.0
|
||||
xdpyinfo >/dev/null
|
||||
'
|
||||
|
||||
# Explicit values belong to the caller, even when X11 is unavailable.
|
||||
"${compose[@]}" run --rm -e DISPLAY=unavailable:0 archivebox sh -ec '
|
||||
test "$DISPLAY" = unavailable:0
|
||||
'
|
||||
# An explicit empty value disables automatic display discovery.
|
||||
"${compose[@]}" run --rm -e DISPLAY= archivebox sh -ec '
|
||||
test "${DISPLAY+x}" = x
|
||||
test -z "$DISPLAY"
|
||||
'
|
||||
|
||||
"${compose[@]}" stop novnc
|
||||
"${compose[@]}" run --rm archivebox sh -ec '
|
||||
test "${DISPLAY+x}" != x
|
||||
'
|
||||
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
|
||||
test "$DISPLAY" = novnc:0.0
|
||||
'
|
||||
echo 'PASS: Compose DISPLAY discovery, explicit override, empty opt-out, and headless fallback'
|
||||
Executable
+61
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
IMAGE="${IMAGE:-archivebox/archivebox:dev}"
|
||||
DOCKER_BINARY="${DOCKER_BINARY:-docker}"
|
||||
CURL_BINARY="${CURL_BINARY:-curl}"
|
||||
name="archivebox-port-check-$$"
|
||||
volume="$name-data"
|
||||
cookies=$(mktemp)
|
||||
cleanup() {
|
||||
status=$?
|
||||
if [[ "$status" != 0 ]]; then "$DOCKER_BINARY" logs "$name" || true; fi
|
||||
"$DOCKER_BINARY" rm -fv "$name" >/dev/null 2>&1 || true
|
||||
"$DOCKER_BINARY" volume rm "$volume" >/dev/null
|
||||
rm -f "$cookies"
|
||||
exit "$status"
|
||||
}
|
||||
"$DOCKER_BINARY" volume create "$volume" >/dev/null
|
||||
trap cleanup EXIT
|
||||
|
||||
"$DOCKER_BINARY" run --rm -v "$volume:/data" "${OLD_IMAGE:-$IMAGE}" init
|
||||
"$DOCKER_BINARY" run --rm -v "$volume:/data" "${OLD_IMAGE:-$IMAGE}" config --set BIND_ADDR=0.0.0.0:8000
|
||||
"$DOCKER_BINARY" run --rm -v "$volume:/data" \
|
||||
-e DJANGO_SUPERUSER_USERNAME=portcheck -e DJANGO_SUPERUSER_EMAIL=portcheck@example.com \
|
||||
-e DJANGO_SUPERUSER_PASSWORD="$name-password" "${OLD_IMAGE:-$IMAGE}" manage createsuperuser --noinput
|
||||
|
||||
for mode in default legacy debug; do
|
||||
command=()
|
||||
if [[ "$mode" == legacy ]]; then command=(server 0.0.0.0:8000); fi
|
||||
if [[ "$mode" == debug ]]; then command=(server --debug 0.0.0.0:5797); fi
|
||||
"$DOCKER_BINARY" run -d --name "$name" -v "$volume:/data" \
|
||||
-p 127.0.0.1::5797 -p 127.0.0.1::8000 "$IMAGE" "${command[@]}" >/dev/null
|
||||
for port in 5797 8000; do
|
||||
address=$("$DOCKER_BINARY" port "$name" "$port/tcp")
|
||||
ready=0
|
||||
for ((attempt=0; attempt<60; attempt++)); do
|
||||
if "$CURL_BINARY" -fsS --max-time 2 "http://$address/health/" >/dev/null 2>&1; then ready=1; break; fi
|
||||
[[ $("$DOCKER_BINARY" inspect -f '{{.State.Running}}' "$name") == true ]]
|
||||
sleep 1
|
||||
done
|
||||
[[ "$ready" == 1 ]]
|
||||
body=$("$CURL_BINARY" -fsS --max-time 10 "http://$address/health/")
|
||||
[[ "$body" == *OK* ]]
|
||||
echo "PASS: $mode server responds through Docker mapping $address -> $port"
|
||||
origin="http://admin.archivebox.localhost:${address##*:}"
|
||||
if [[ "$port" == 5797 ]]; then
|
||||
login=$("$CURL_BINARY" -fsS --max-time 10 -c "$cookies" "$origin/admin/login/?next=/admin/")
|
||||
csrf=$(printf '%s' "$login" | sed -n 's/.*name="csrfmiddlewaretoken" value="\([^"]*\)".*/\1/p')
|
||||
[[ -n "$csrf" ]]
|
||||
"$CURL_BINARY" -fsS --max-time 10 -b "$cookies" -c "$cookies" -e "$origin/admin/login/" \
|
||||
--data-urlencode username=portcheck --data-urlencode "password=$name-password" \
|
||||
--data-urlencode "csrfmiddlewaretoken=$csrf" --data-urlencode next=/admin/ \
|
||||
"$origin/admin/login/" >/dev/null
|
||||
fi
|
||||
page=$("$CURL_BINARY" -fsS --max-time 10 -b "$cookies" "$origin/admin/crawls/crawl/")
|
||||
[[ "$page" == *model-crawl* ]]
|
||||
[[ "$page" != *'name="password"'* ]]
|
||||
echo "PASS: $mode authenticated Crawls works on $port with the same browser session"
|
||||
done
|
||||
"$DOCKER_BINARY" rm -fv "$name" >/dev/null
|
||||
done
|
||||
+13
-3
@@ -9,8 +9,12 @@ DEPLOY_IMAGE="${DEPLOY_IMAGE:-archivebox/archivebox:dev}"
|
||||
DEPLOY_INTERVAL="${DEPLOY_INTERVAL:-60}"
|
||||
STATE_FILE="${STATE_FILE:-${DEPLOY_PATH}/.archivebox-dev-image.digest}"
|
||||
OVERRIDE_FILE="${OVERRIDE_FILE:-${DEPLOY_PATH}/.archivebox-deploy.override.yml}"
|
||||
DEPLOY_BASE_OVERRIDE="${DEPLOY_BASE_OVERRIDE:-}"
|
||||
DEPLOY_HEALTH_HOST="${DEPLOY_HEALTH_HOST:-admin.archivebox.io}"
|
||||
DEPLOY_HEALTH_PORT="${DEPLOY_HEALTH_PORT:-5797}"
|
||||
DEPLOY_HEALTH_PATH="${DEPLOY_HEALTH_PATH:-/health/}"
|
||||
ABXPKG_LIB_DIR="${ABXPKG_LIB_DIR:-$HOME/.config/archivebox/lib}"
|
||||
DOCKER_BINARY="$ABXPKG_LIB_DIR/env/bin/docker"
|
||||
DOCKER_BINARY="${DOCKER_BINARY:-$ABXPKG_LIB_DIR/env/bin/docker}"
|
||||
test -x "$DOCKER_BINARY"
|
||||
|
||||
cd "$DEPLOY_PATH"
|
||||
@@ -35,7 +39,12 @@ services:
|
||||
command: null
|
||||
EOF
|
||||
|
||||
COMPOSE=("$DOCKER_BINARY" compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE")
|
||||
COMPOSE=("$DOCKER_BINARY" compose -f "$COMPOSE_FILE")
|
||||
if [[ -n "$DEPLOY_BASE_OVERRIDE" ]]; then
|
||||
test -f "$DEPLOY_BASE_OVERRIDE"
|
||||
COMPOSE+=(-f "$DEPLOY_BASE_OVERRIDE")
|
||||
fi
|
||||
COMPOSE+=(-f "$OVERRIDE_FILE")
|
||||
|
||||
remote_digest() {
|
||||
local line
|
||||
@@ -71,7 +80,8 @@ deploy_digest() {
|
||||
printf '%s\n' "$line"
|
||||
line_count=$((line_count + 1))
|
||||
done <<<"$version_output"
|
||||
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:8000/health/ </dev/null
|
||||
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 \
|
||||
-H "Host: $DEPLOY_HEALTH_HOST" "http://127.0.0.1:${DEPLOY_HEALTH_PORT}${DEPLOY_HEALTH_PATH}" </dev/null
|
||||
printf '%s\n' "$digest" > "$STATE_FILE"
|
||||
}
|
||||
|
||||
|
||||
+2
-2
@@ -3,7 +3,7 @@
|
||||
# curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
|
||||
# docker compose pull
|
||||
# docker compose up -d --wait
|
||||
# Open /admin/ on the hostname or IP used to reach ArchiveBox (local: http://admin.archivebox.localhost:8000/admin/).
|
||||
# Open /admin/ on the hostname or IP used to reach ArchiveBox (local: http://admin.archivebox.localhost:5797/admin/).
|
||||
|
||||
# docker compose exec archivebox archivebox add --depth=1 'https://news.ycombinator.com'
|
||||
# docker compose exec -T archivebox archivebox add < ~/Downloads/bookmarks.txt
|
||||
@@ -20,7 +20,7 @@ services:
|
||||
# novnc:
|
||||
# condition: service_healthy
|
||||
ports:
|
||||
- "${ARCHIVEBOX_PORT:-8000}:8000"
|
||||
- "${ARCHIVEBOX_PORT:-5797}:5797"
|
||||
volumes:
|
||||
- ./data:/data
|
||||
# Override abx-dl's nested VOLUME so profiles persist across containers.
|
||||
|
||||
@@ -149,10 +149,26 @@ ssh -N -L 8080:127.0.0.1:8080 user@your-server
|
||||
|
||||
4. Open a page and confirm you can view it while logged in. For a new persona, log in to the sites you want to archive. Then follow [Save and check the page](#3-save-and-check-the-page), selecting the same persona.
|
||||
|
||||
#### One-off Compose setup
|
||||
|
||||
To set up a persona without starting the ArchiveBox server, enable the same `novnc` service and healthy `depends_on` block, then run:
|
||||
|
||||
```bash
|
||||
docker compose up -d --wait novnc
|
||||
docker compose run --rm archivebox persona create personal
|
||||
docker compose run --rm -e DISPLAY=novnc:0.0 archivebox persona open personal
|
||||
```
|
||||
|
||||
Connect at <http://localhost:8080/vnc.html>, log in, and close the browser when finished. The persona remains in `data/personas/personal` across one-off containers.
|
||||
|
||||
`persona open` resolves the installed Chromium binary automatically. Do not hard-code `/data/lib/env/bin/chromium` or a versioned Playwright cache path: the browser location varies by image version and architecture.
|
||||
|
||||
An explicitly set `DISPLAY` is preserved. When it is unset, ArchiveBox probes `novnc:0.0` and falls back to headless operation if unavailable. Set `DISPLAY=` to disable this automatic display selection.
|
||||
|
||||
## Troubleshooting Chromium Install
|
||||
|
||||
<a name="more-info--troubleshooting"></a>
|
||||
|
||||
- **Browser missing or unable to start:** see [Chromium troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#chromiumgoogle-chrome).
|
||||
- **Saved a login screen:** confirm you can view the page in the source browser, re-import the correct profile, and select that persona when saving.
|
||||
- **Need browser settings:** see the [Chrome plugin configuration](https://archivebox.github.io/abx-plugins/#chrome).
|
||||
- **Need browser settings:** see the [Chrome plugin configuration](https://plugins.archivebox.io/#chrome).
|
||||
+26
-26
@@ -68,7 +68,7 @@ archivebox add --no-only-new https://example.com # force a re-archive even
|
||||
**Possible Values:** [`60`]/`120`/...
|
||||
Maximum allowed runtime **per-extractor, per-Snapshot** in seconds. If you have a slow network connection or are seeing frequent timeout errors, you can raise this value.
|
||||
|
||||
This is a *plugin-shared* setting — each individual extractor can override it with its own `<EXTRACTOR>_TIMEOUT` (e.g. [`WGET_TIMEOUT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_TIMEOUT`](https://archivebox.github.io/abx-plugins/#chrome), [`YTDLP_TIMEOUT`](https://archivebox.github.io/abx-plugins/#ytdlp)). See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for the full list.
|
||||
This is a *plugin-shared* setting — each individual extractor can override it with its own `<EXTRACTOR>_TIMEOUT` (e.g. [`WGET_TIMEOUT`](https://plugins.archivebox.io/#wget), [`CHROME_TIMEOUT`](https://plugins.archivebox.io/#chrome), [`YTDLP_TIMEOUT`](https://plugins.archivebox.io/#ytdlp)). See the [per-plugin docs](https://plugins.archivebox.io/) for the full list.
|
||||
|
||||
> [!NOTE]
|
||||
> `TIMEOUT` only caps a single extractor invocation. To bound the *total* wall-clock runtime of an entire crawl, use [`CRAWL_TIMEOUT`](#crawl_timeout) instead.
|
||||
@@ -84,14 +84,14 @@ This is a *plugin-shared* setting — each individual extractor can override it
|
||||
**Possible Values:** [`1440,2000`]/`1024,768`/...
|
||||
Default screenshot/PDF viewport resolution in `width,height` pixels. Used as the fallback for `SCREENSHOT_RESOLUTION`, `PDF_RESOLUTION`, and `CHROME_RESOLUTION`.
|
||||
|
||||
This is a *plugin-shared* setting — individual extractors override it via `<EXTRACTOR>_RESOLUTION` (e.g. [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`PDF_RESOLUTION`](https://archivebox.github.io/abx-plugins/#pdf), [`CHROME_RESOLUTION`](https://archivebox.github.io/abx-plugins/#chrome)). See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for plugin-specific overrides.
|
||||
This is a *plugin-shared* setting — individual extractors override it via `<EXTRACTOR>_RESOLUTION` (e.g. [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`PDF_RESOLUTION`](https://plugins.archivebox.io/#pdf), [`CHROME_RESOLUTION`](https://plugins.archivebox.io/#chrome)). See the [per-plugin docs](https://plugins.archivebox.io/) for plugin-specific overrides.
|
||||
|
||||
---
|
||||
#### `CHECK_SSL_VALIDITY`
|
||||
**Possible Values:** [`True`]/`False`
|
||||
Whether to enforce HTTPS certificate validity and HSTS chain of trust when archiving sites. Set this to `False` if you want to archive pages even if they have expired or invalid certificates.
|
||||
|
||||
This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](https://archivebox.github.io/abx-plugins/#wget), [`yt-dlp`](https://archivebox.github.io/abx-plugins/#ytdlp), [`gallery-dl`](https://archivebox.github.io/abx-plugins/#gallerydl), [`chrome`](https://archivebox.github.io/abx-plugins/#chrome), etc.) honors it, and individual extractors can override with `<EXTRACTOR>_CHECK_SSL_VALIDITY`. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/).
|
||||
This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](https://plugins.archivebox.io/#wget), [`yt-dlp`](https://plugins.archivebox.io/#ytdlp), [`gallery-dl`](https://plugins.archivebox.io/#gallerydl), [`chrome`](https://plugins.archivebox.io/#chrome), etc.) honors it, and individual extractors can override with `<EXTRACTOR>_CHECK_SSL_VALIDITY`. See the [per-plugin docs](https://plugins.archivebox.io/).
|
||||
|
||||
> [!WARNING]
|
||||
> When `False`, ArchiveBox cannot guarantee that the captured content matches the real site — a man-in-the-middle could substitute responses. Only disable for trusted networks or for archiving legacy/internal sites with expired certs.
|
||||
@@ -101,7 +101,7 @@ This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](ht
|
||||
**Possible Values:** [`Mozilla/5.0 ... ArchiveBox/{VERSION} ...`]/`"Mozilla/5.0 ..."`/...
|
||||
The default `User-Agent` string sent during archiving. The built-in default identifies ArchiveBox and links back to the GitHub repo so site operators can identify and contact archivers if needed.
|
||||
|
||||
This is a *plugin-shared* setting — each extractor ([`wget`](https://archivebox.github.io/abx-plugins/#wget), [`chrome`](https://archivebox.github.io/abx-plugins/#chrome), [`yt-dlp`](https://archivebox.github.io/abx-plugins/#ytdlp), [`singlefile`](https://archivebox.github.io/abx-plugins/#singlefile), …) can override it with its own `<EXTRACTOR>_USER_AGENT`, otherwise it falls back to this value. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for per-extractor specifics.
|
||||
This is a *plugin-shared* setting — each extractor ([`wget`](https://plugins.archivebox.io/#wget), [`chrome`](https://plugins.archivebox.io/#chrome), [`yt-dlp`](https://plugins.archivebox.io/#ytdlp), [`singlefile`](https://plugins.archivebox.io/#singlefile), …) can override it with its own `<EXTRACTOR>_USER_AGENT`, otherwise it falls back to this value. See the [per-plugin docs](https://plugins.archivebox.io/) for per-extractor specifics.
|
||||
|
||||
> [!NOTE]
|
||||
> Some sites block requests that look like bots or that don't match a real browser. If you're getting 403s or empty responses, try setting this to a current Chrome/Firefox UA string.
|
||||
@@ -115,7 +115,7 @@ This is a *plugin-shared* setting — each extractor ([`wget`](https://archivebo
|
||||
|
||||
Path to a [Netscape-format `cookies.txt`](http://www.cookiecentral.com/faq/#3.5) file passed to `wget`, `curl`, `yt-dlp`, and other non-Chrome extractors for authentication. Required when archiving sites behind a login (paywalls, social media feeds, members-only forums, etc.) **if you're not using a persona**.
|
||||
|
||||
This is a *plugin-shared* setting — each extractor can override it with `<EXTRACTOR>_COOKIES_FILE` (e.g. [`WGET_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#gallerydl)). [Chrome](https://archivebox.github.io/abx-plugins/#chrome)-based extractors instead read auth state from the persona's `CHROME_USER_DATA_DIR`. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for per-extractor variants.
|
||||
This is a *plugin-shared* setting — each extractor can override it with `<EXTRACTOR>_COOKIES_FILE` (e.g. [`WGET_COOKIES_FILE`](https://plugins.archivebox.io/#wget), [`YTDLP_COOKIES_FILE`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://plugins.archivebox.io/#gallerydl)). [Chrome](https://plugins.archivebox.io/#chrome)-based extractors instead read auth state from the persona's `CHROME_USER_DATA_DIR`. See the [per-plugin docs](https://plugins.archivebox.io/) for per-extractor variants.
|
||||
|
||||
You can generate a `cookies.txt` using a [browser extension](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc), or with `wget --save-cookies` + `--user=... --password=...`.
|
||||
|
||||
@@ -130,7 +130,7 @@ archivebox add --persona=personal https://members.example.com/feed
|
||||
> **Use separate burner credentials dedicated to archiving** — don't re-use your normal daily Facebook/Instagram/Youtube/etc. account cookies as server responses often contain your name/email/PII and session tokens, which then get preserved in your snapshots forever!
|
||||
|
||||
*Related options:*
|
||||
[`DEFAULT_PERSONA`](#default_persona), [`CHROME_USER_DATA_DIR`](https://archivebox.github.io/abx-plugins/#chrome)
|
||||
[`DEFAULT_PERSONA`](#default_persona), [`CHROME_USER_DATA_DIR`](https://plugins.archivebox.io/#chrome)
|
||||
|
||||
---
|
||||
<a id="active_persona"></a>
|
||||
@@ -353,17 +353,17 @@ Django's secret key, used for cryptographic signing of sessions, CSRF tokens, pa
|
||||
|
||||
---
|
||||
#### `BIND_ADDR`
|
||||
**Possible Values:** [`127.0.0.1:8000`]/`0.0.0.0:8000`/`[::]:8000`/`0.0.0.0:80`/...
|
||||
**Possible Values:** [`127.0.0.1:5797`]/`0.0.0.0:5797`/`[::]:5797`/`0.0.0.0:80`/...
|
||||
|
||||
The `host:port` socket the ArchiveBox web server actually listens on. **This is the local bind socket, not the public URL** — for the public URL clients see, set [`BASE_URL`](#base_url).
|
||||
|
||||
- `127.0.0.1:8000` (default) — listen only on the loopback interface. Safest when you're running a reverse proxy on the same host and don't want the server reachable directly from the network.
|
||||
- `0.0.0.0:8000` — listen on **all** IPv4 interfaces. Required when running in Docker without `--network=host`, or when you want the server reachable from other machines on your LAN without a reverse proxy.
|
||||
- `[::]:8000` — listen on all IPv6 interfaces (most modern OSes will accept v4-mapped connections too).
|
||||
IPv6 literal addresses must be bracketed: `[::1]:8000`, not `::1:8000`.
|
||||
- `127.0.0.1:5797` (default) — listen only on the loopback interface. Safest when you're running a reverse proxy on the same host and don't want the server reachable directly from the network.
|
||||
- `0.0.0.0:5797` — listen on **all** IPv4 interfaces. Required when running in Docker without `--network=host`, or when you want the server reachable from other machines on your LAN without a reverse proxy.
|
||||
- `[::]:5797` — listen on all IPv6 interfaces (most modern OSes will accept v4-mapped connections too).
|
||||
IPv6 literal addresses must be bracketed: `[::1]:5797`, not `::1:5797`.
|
||||
|
||||
> [!NOTE]
|
||||
> Inside Docker, binding to `127.0.0.1` means the server is unreachable from outside the container — use `0.0.0.0:8000` and let Docker handle the port-forwarding, or publish the port with `-p 127.0.0.1:8000:8000` on the host side instead.
|
||||
> Inside Docker, binding to `127.0.0.1` means the server is unreachable from outside the container — use `0.0.0.0:5797` and let Docker handle the port-forwarding, or publish the port with `-p 127.0.0.1:5797:5797` on the host side instead.
|
||||
|
||||
*Related options:*
|
||||
[`BASE_URL`](#base_url), [`SERVER_SECURITY_MODE`](#server_security_mode)
|
||||
@@ -372,7 +372,7 @@ IPv6 literal addresses must be bracketed: `[::1]:8000`, not `::1:8000`.
|
||||
<a id="allowed_hosts"></a>
|
||||
<a id="csrf_trusted_origins"></a>
|
||||
#### `BASE_URL`
|
||||
**Possible Values:** [`""`]/`https://archive.example.com`/`http://archivebox.localhost:8000`/...
|
||||
**Possible Values:** [`""`]/`https://archive.example.com`/`http://archivebox.localhost:5797`/...
|
||||
|
||||
The canonical public URL of your ArchiveBox instance. Used to build absolute links in templates, redirects (`/admin/login/?next=...`), admin notification emails, OG/meta tags, and — in subdomain security mode — to derive the `admin.`, `web.`, `api.`, and per-snapshot `snap-<id>.` subdomains.
|
||||
|
||||
@@ -778,7 +778,7 @@ Lower values retry more aggressively (useful if you expect locks to clear quickl
|
||||
ArchiveBox can index Snapshot text/HTML output into searchable indexes that power the search bar in the Web UI and the `archivebox search <query>` CLI command. Multiple backend engines can be enabled at once; `SEARCH_BACKEND_ENGINE` selects the default used by the UI and CLI.
|
||||
|
||||
> [!NOTE]
|
||||
> Each backend has its own tuning knobs (e.g. [Sonic](https://archivebox.github.io/abx-plugins/#search_backend_sonic) host/port, [ripgrep](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) flags, [SQLite FTS](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) database path). Those backend-specific options now live with the plugin that implements them — see the [abx-plugins docs](https://archivebox.github.io/abx-plugins/) for the full per-backend schema.
|
||||
> Each backend has its own tuning knobs (e.g. [Sonic](https://plugins.archivebox.io/#search_backend_sonic) host/port, [ripgrep](https://plugins.archivebox.io/#search_backend_ripgrep) flags, [SQLite FTS](https://plugins.archivebox.io/#search_backend_sqlite) database path). Those backend-specific options now live with the plugin that implements them — see the [abx-plugins docs](https://plugins.archivebox.io/) for the full per-backend schema.
|
||||
|
||||
---
|
||||
#### `SEARCH_BACKEND_ENGINE`
|
||||
@@ -786,13 +786,13 @@ ArchiveBox can index Snapshot text/HTML output into searchable indexes that powe
|
||||
|
||||
Which search backend engine to use when running `archivebox search` and rendering the Web UI search bar.
|
||||
|
||||
- **`ripgrep`** — Pure filesystem grep across each Snapshot's archived output (HTML, text, metadata) via the [`search_backend_ripgrep`](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) plugin. No extra daemon, no extra database to maintain — just install `rg` and it works. Slow on very large collections (each query re-scans the disk) but always 100% correct: results reflect what's actually on disk *right now*, no stale index. ArchiveBox keeps it enabled as the fallback when Sonic is unavailable.
|
||||
- **`ripgrep`** — Pure filesystem grep across each Snapshot's archived output (HTML, text, metadata) via the [`search_backend_ripgrep`](https://plugins.archivebox.io/#search_backend_ripgrep) plugin. No extra daemon, no extra database to maintain — just install `rg` and it works. Slow on very large collections (each query re-scans the disk) but always 100% correct: results reflect what's actually on disk *right now*, no stale index. ArchiveBox keeps it enabled as the fallback when Sonic is unavailable.
|
||||
|
||||
- **`sonic`** *(default)* — Fast, suggest-style fuzzy search via a running [Sonic](https://github.com/valeriansaliou/sonic) daemon (configured via the [`search_backend_sonic`](https://archivebox.github.io/abx-plugins/#search_backend_sonic) plugin). ArchiveBox pushes text into Sonic at index time and queries it at search time. Sub-millisecond queries even at very large scale; ArchiveBox starts the managed service automatically when this backend is selected.
|
||||
- **`sonic`** *(default)* — Fast, suggest-style fuzzy search via a running [Sonic](https://github.com/valeriansaliou/sonic) daemon (configured via the [`search_backend_sonic`](https://plugins.archivebox.io/#search_backend_sonic) plugin). ArchiveBox pushes text into Sonic at index time and queries it at search time. Sub-millisecond queries even at very large scale; ArchiveBox starts the managed service automatically when this backend is selected.
|
||||
|
||||
- **`sqlite`** — FTS5 full-text index stored alongside ArchiveBox's main `index.sqlite3`, configured via the [`search_backend_sqlite`](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) plugin. No extra processes, no extra binary — uses the SQLite already shipped with Python. Faster than `ripgrep` on large collections, slightly slower than `sonic`, but no daemon to babysit. Good middle ground for users who want a real index without operational overhead.
|
||||
- **`sqlite`** — FTS5 full-text index stored alongside ArchiveBox's main `index.sqlite3`, configured via the [`search_backend_sqlite`](https://plugins.archivebox.io/#search_backend_sqlite) plugin. No extra processes, no extra binary — uses the SQLite already shipped with Python. Faster than `ripgrep` on large collections, slightly slower than `sonic`, but no daemon to babysit. Good middle ground for users who want a real index without operational overhead.
|
||||
|
||||
*Note: Backend-specific tuning ([Sonic](https://archivebox.github.io/abx-plugins/#search_backend_sonic) host/port/password, [ripgrep](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) flag overrides, [SQLite FTS](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) database path, indexer batch size, etc.) lives in each search-backend plugin's own config schema — see the [abx-plugins docs](https://archivebox.github.io/abx-plugins/) for the full per-backend option list.*
|
||||
*Note: Backend-specific tuning ([Sonic](https://plugins.archivebox.io/#search_backend_sonic) host/port/password, [ripgrep](https://plugins.archivebox.io/#search_backend_ripgrep) flag overrides, [SQLite FTS](https://plugins.archivebox.io/#search_backend_sqlite) database path, indexer batch size, etc.) lives in each search-backend plugin's own config schema — see the [abx-plugins docs](https://plugins.archivebox.io/) for the full per-backend option list.*
|
||||
|
||||
---
|
||||
|
||||
@@ -1087,21 +1087,21 @@ SHOW_PROGRESS=False archivebox add < urls.txt
|
||||
> **Per-plugin configuration has moved to its own documentation site.**
|
||||
> This `Configuration.md` doc covers only ArchiveBox's *core* settings. For everything that lives inside a plugin — extractor toggles, binary paths, timeouts, args, user agents, cookies, persona scoping, etc. — see:
|
||||
>
|
||||
> ## ➡️ **<https://archivebox.github.io/abx-plugins/>**
|
||||
> ## ➡️ **<https://plugins.archivebox.io/>**
|
||||
|
||||
That site is regenerated from each plugin's `config.json` schema on every release, so it stays in sync with the code. Looking for [`WGET_ARGS`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_USER_DATA_DIR`](https://archivebox.github.io/abx-plugins/#chrome), [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`YTDLP_EXTRA_ARGS`](https://archivebox.github.io/abx-plugins/#ytdlp), [`SINGLEFILE_*`](https://archivebox.github.io/abx-plugins/#singlefile), [`SONIC_HOST`](https://archivebox.github.io/abx-plugins/#search_backend_sonic), etc.? They all live there now.
|
||||
That site is regenerated from each plugin's `config.json` schema on every release, so it stays in sync with the code. Looking for [`WGET_ARGS`](https://plugins.archivebox.io/#wget), [`CHROME_USER_DATA_DIR`](https://plugins.archivebox.io/#chrome), [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`YTDLP_EXTRA_ARGS`](https://plugins.archivebox.io/#ytdlp), [`SINGLEFILE_*`](https://plugins.archivebox.io/#singlefile), [`SONIC_HOST`](https://plugins.archivebox.io/#search_backend_sonic), etc.? They all live there now.
|
||||
|
||||
### Shared core options that plugins fall back to
|
||||
|
||||
A handful of *core* options (documented above on this page) act as the **fallback default** for every plugin that has a matching per-extractor override. If you set the core option, every plugin honors it; if you also set the plugin-specific override, that wins for just that one plugin.
|
||||
|
||||
| Core option (this doc) | Plugin-level overrides (see [abx-plugins](https://archivebox.github.io/abx-plugins/)) |
|
||||
| Core option (this doc) | Plugin-level overrides (see [abx-plugins](https://plugins.archivebox.io/)) |
|
||||
|---|---|
|
||||
| [`TIMEOUT`](#timeout) | [`WGET_TIMEOUT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_TIMEOUT`](https://archivebox.github.io/abx-plugins/#chrome), [`YTDLP_TIMEOUT`](https://archivebox.github.io/abx-plugins/#ytdlp), [`SINGLEFILE_TIMEOUT`](https://archivebox.github.io/abx-plugins/#singlefile), [`TITLE_TIMEOUT`](https://archivebox.github.io/abx-plugins/#title), [`FAVICON_TIMEOUT`](https://archivebox.github.io/abx-plugins/#favicon), ... |
|
||||
| [`CHECK_SSL_VALIDITY`](#check_ssl_validity) | [`WGET_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#gallerydl), [`CHROME_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#chrome), ... |
|
||||
| [`USER_AGENT`](#user_agent) | [`WGET_USER_AGENT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_USER_AGENT`](https://archivebox.github.io/abx-plugins/#chrome), ... |
|
||||
| [`COOKIES_FILE`](#cookies_file) | [`WGET_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#gallerydl), ... |
|
||||
| [`RESOLUTION`](#resolution) | [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`PDF_RESOLUTION`](https://archivebox.github.io/abx-plugins/#pdf), [`CHROME_RESOLUTION`](https://archivebox.github.io/abx-plugins/#chrome) |
|
||||
| [`TIMEOUT`](#timeout) | [`WGET_TIMEOUT`](https://plugins.archivebox.io/#wget), [`CHROME_TIMEOUT`](https://plugins.archivebox.io/#chrome), [`YTDLP_TIMEOUT`](https://plugins.archivebox.io/#ytdlp), [`SINGLEFILE_TIMEOUT`](https://plugins.archivebox.io/#singlefile), [`TITLE_TIMEOUT`](https://plugins.archivebox.io/#title), [`FAVICON_TIMEOUT`](https://plugins.archivebox.io/#favicon), ... |
|
||||
| [`CHECK_SSL_VALIDITY`](#check_ssl_validity) | [`WGET_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#wget), [`YTDLP_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#gallerydl), [`CHROME_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#chrome), ... |
|
||||
| [`USER_AGENT`](#user_agent) | [`WGET_USER_AGENT`](https://plugins.archivebox.io/#wget), [`CHROME_USER_AGENT`](https://plugins.archivebox.io/#chrome), ... |
|
||||
| [`COOKIES_FILE`](#cookies_file) | [`WGET_COOKIES_FILE`](https://plugins.archivebox.io/#wget), [`YTDLP_COOKIES_FILE`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://plugins.archivebox.io/#gallerydl), ... |
|
||||
| [`RESOLUTION`](#resolution) | [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`PDF_RESOLUTION`](https://plugins.archivebox.io/#pdf), [`CHROME_RESOLUTION`](https://plugins.archivebox.io/#chrome) |
|
||||
|
||||
> [!TIP]
|
||||
> The resolution order for any plugin-tunable option is always:
|
||||
|
||||
+12
-10
@@ -71,7 +71,7 @@ docker compose pull
|
||||
docker compose up -d --wait
|
||||
```
|
||||
|
||||
Open <http://admin.archivebox.localhost:8000> and follow the setup wizard to create the first admin and configure web access. Existing `BASE_URL` and security settings are used as-is, so configured servers skip the web-access wizard.
|
||||
Open <http://admin.archivebox.localhost:5797> and follow the setup wizard to create the first admin and configure web access. Existing `BASE_URL` and security settings are used as-is, so configured servers skip the web-access wizard.
|
||||
|
||||
ArchiveBox installs and enables both ripgrep and [Sonic](https://github.com/valeriansaliou/sonic). Sonic is selected by default in the UI, while ripgrep remains available as the fallback. To select ripgrep explicitly:
|
||||
```bash
|
||||
@@ -148,7 +148,9 @@ Alternatively, to use the web UI, start the server with:
|
||||
docker compose up # add -d to run in the background
|
||||
```
|
||||
|
||||
Then open [`http://web.archivebox.localhost:8000`](http://web.archivebox.localhost:8000) for the public UI or [`http://admin.archivebox.localhost:8000`](http://admin.archivebox.localhost:8000) for the admin UI.
|
||||
Then open [`http://web.archivebox.localhost:5797`](http://web.archivebox.localhost:5797) for the public UI or [`http://admin.archivebox.localhost:5797`](http://admin.archivebox.localhost:5797) for the admin UI.
|
||||
|
||||
Use port **5797** for new deployments. The Docker container also listens on the old port **8000** so existing port mappings remain compatible.
|
||||
|
||||
<br/>
|
||||
|
||||
@@ -162,7 +164,7 @@ docker compose exec archivebox archivebox config --set TIMEOUT=120
|
||||
# OR edit ./data/ArchiveBox.conf and add this under its existing [ARCHIVING_CONFIG] section:
|
||||
TIMEOUT=120
|
||||
|
||||
# plugin-specific options work the same way (see https://archivebox.github.io/abx-plugins/)
|
||||
# plugin-specific options work the same way (see https://plugins.archivebox.io/)
|
||||
docker compose exec archivebox archivebox config --set YTDLP_MAX_SIZE=750m
|
||||
```
|
||||
This will apply the config to all containers or archivebox instances that access the collection.
|
||||
@@ -184,11 +186,11 @@ services:
|
||||
...
|
||||
```
|
||||
|
||||
For public HTTPS, start the default stack with `docker compose up -d`, use port `8000` only as the temporary setup/upstream endpoint, and follow the first-run wizard. It gives the DNS, upstream, and certificate settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI, then verifies the public HTTPS URLs before saving `BASE_URL` and `SERVER_SECURITY_MODE`.
|
||||
For public HTTPS, start the default stack with `docker compose up -d`, use port `5797` only as the temporary setup/upstream endpoint, and follow the first-run wizard. It gives the DNS, upstream, and certificate settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI, then verifies the public HTTPS URLs before saving `BASE_URL` and `SERVER_SECURITY_MODE`.
|
||||
|
||||
Use exactly one of these certificate layouts:
|
||||
|
||||
- **Single-domain mode:** one certificate for the `BASE_URL` hostname, proxied to ArchiveBox port `8000`.
|
||||
- **Single-domain mode:** one certificate for the `BASE_URL` hostname, proxied to ArchiveBox port `5797`.
|
||||
- **Isolated-subdomain mode:** one certificate covering both the `BASE_URL` hostname and `*.BASE_URL`, normally obtained through DNS-01.
|
||||
|
||||
Never enable on-demand TLS or request individual certificates for `snap-*` hostnames.
|
||||
@@ -211,10 +213,10 @@ Fetch and run the ArchiveBox Docker image. Starting the server creates the initi
|
||||
docker pull archivebox/archivebox:dev
|
||||
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
|
||||
```
|
||||
|
||||
Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local example: <http://admin.archivebox.localhost:8000/admin/>) to create the first admin. If `BASE_URL` is not configured yet, continue through the web setup wizard.
|
||||
Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local example: <http://admin.archivebox.localhost:5797/admin/>) to create the first admin. If `BASE_URL` is not configured yet, continue through the web setup wizard.
|
||||
|
||||
*(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
|
||||
|
||||
@@ -270,8 +272,8 @@ docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox:d
|
||||
|
||||
Then to view your data, you can look in the folder on the host `/media/USB-DRIVE/archivebox/data`, or use the Web UI:
|
||||
```bash
|
||||
docker run -it -v /media/USB-DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
# then open http://web.archivebox.localhost:8000
|
||||
docker run -it -v /media/USB-DRIVE/archivebox/data:/data -p 5797:5797 archivebox/archivebox:dev
|
||||
# then open http://web.archivebox.localhost:5797
|
||||
```
|
||||
|
||||
<br/>
|
||||
@@ -287,7 +289,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev config --set TIMEOUT=120
|
||||
TIMEOUT=120
|
||||
```
|
||||
|
||||
ArchiveBox in Docker also accepts config as environment variables, see more on the [[Configuration]] page (and the [abx-plugins config reference](https://archivebox.github.io/abx-plugins/) for per-plugin options).
|
||||
ArchiveBox in Docker also accepts config as environment variables, see more on the [[Configuration]] page (and the [abx-plugins config reference](https://plugins.archivebox.io/) for per-plugin options).
|
||||
|
||||
For example, this disables the screenshot extractor for a single run (without persisting for other runs):
|
||||
```bash
|
||||
|
||||
+5
-5
@@ -33,7 +33,7 @@ Other operating systems are not tested or supported for this release.
|
||||
|
||||
<br/>
|
||||
|
||||
You can reduce crawl-time memory pressure by disabling all Chrome-based archiving methods with [`CHROME_ENABLED=False`](https://archivebox.github.io/abx-plugins/#chrome) (or its `USE_CHROME` alias).
|
||||
You can reduce crawl-time memory pressure by disabling all Chrome-based archiving methods with [`CHROME_ENABLED=False`](https://plugins.archivebox.io/#chrome) (or its `USE_CHROME` alias).
|
||||
|
||||
It's also recommended to use a filesystem with compression and/or [deduplication](https://www.ixsystems.com/blog/ixsystems-and-klara-systems-celebrate-valentines-day-with-a-heartfelt-donation-of-fast-dedupe-to-openzfs-and-truenas/) (e.g. [ZFS](https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html) or BTRFS) for maximum efficiency.
|
||||
|
||||
@@ -94,13 +94,13 @@ After running the setup script, continue with the [Quickstart](https://github.co
|
||||
|
||||
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [`uv` install shortcut](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `uv`, `apt`, or Homebrew.
|
||||
|
||||
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://archivebox.github.io/abx-plugins/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://archivebox.github.io/abx-plugins/#pdf), [`SCREENSHOT_ENABLED=False`](https://archivebox.github.io/abx-plugins/#screenshot), and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom) you don't need `chromium`.
|
||||
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://plugins.archivebox.io/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://plugins.archivebox.io/#pdf), [`SCREENSHOT_ENABLED=False`](https://plugins.archivebox.io/#screenshot), and [`DOM_ENABLED=False`](https://plugins.archivebox.io/#dom) you don't need `chromium`.
|
||||
|
||||
<img src="https://avatars0.githubusercontent.com/u/1503512?s=200&v=4" width="100px" align="right"/>
|
||||
|
||||
**More info:**
|
||||
- For help installing these, see the [Manual Setup](#manual-setup), [[Troubleshooting]] and [[Chromium Install]] pages.
|
||||
- For per-plugin binary and enable/disable options (CHROME_BINARY, RIPGREP_BINARY, `<plugin>_ENABLED`, etc.) see the [abx-plugins config reference](https://archivebox.github.io/abx-plugins/).
|
||||
- For per-plugin binary and enable/disable options (CHROME_BINARY, RIPGREP_BINARY, `<plugin>_ENABLED`, etc.) see the [abx-plugins config reference](https://plugins.archivebox.io/).
|
||||
|
||||
|
||||
|
||||
@@ -251,8 +251,8 @@ archivebox status
|
||||
```
|
||||
```bash
|
||||
# OR start the webserver and view them in the Web UI
|
||||
archivebox server 0.0.0.0:8000
|
||||
# Visit http://admin.archivebox.localhost:8000/admin/ in a browser to finish setup
|
||||
archivebox server 0.0.0.0:5797
|
||||
# Visit http://admin.archivebox.localhost:5797/admin/ in a browser to finish setup
|
||||
```
|
||||
See our [[Usage]] Wiki documentation page for more examples.
|
||||
|
||||
|
||||
+22
-43
@@ -1,56 +1,35 @@
|
||||
# Merging Collections
|
||||
|
||||
Current ArchiveBox collections cannot be merged safely by copying their `archive/users/...` trees: the database owns the Crawl, Snapshot, user, permission, and state-machine records, and `archivebox init` intentionally does not import orphaned current-layout directories. For current collections, use a database-aware migration or export the source URLs and re-archive them into the destination collection. Copying current Snapshot directories alone is a backup operation, not a merge.
|
||||
Copy snapshot directories into the destination collection, then explicitly rescan them. Ordinary `archivebox init` and `archivebox update` intentionally avoid discovering current-layout orphan directories: that would make every startup/update scan the entire collection.
|
||||
|
||||
The workflow below is retained for **legacy collections whose real Snapshot directories are `archive/<timestamp>/`**. `archivebox update` can import those legacy directories into a fresh index.
|
||||
1. Back up both collections and finish active captures before copying. Upgrade legacy collections using the normal [[Upgrading]] instructions.
|
||||
2. Initialize the destination collection with `archivebox init`.
|
||||
3. Copy or drag the contents of each source `archive/` into the destination `archive/`, preserving the `users/<username>/snapshots/<date>/<domain>/<uuid>/` hierarchy and every snapshot's `index.jsonl`. Inspect collisions; do not overwrite different files at the same path. Do not replace the destination SQLite database.
|
||||
4. From the destination data directory, run:
|
||||
|
||||
> [!WARNING]
|
||||
> Back up every collection before merging. Confirm that the source entries are real legacy timestamp directories containing data, and inspect path conflicts instead of allowing one collection to overwrite another.
|
||||
```bash
|
||||
archivebox init
|
||||
archivebox update --rescan --migrate-only --index-only
|
||||
archivebox status
|
||||
```
|
||||
|
||||
1. Upgrade both old collections to the most recent ArchiveBox version (following instructions above)
|
||||
```bash
|
||||
cd /path/to/archivebox1/data
|
||||
archivebox init
|
||||
archivebox status
|
||||
With Docker Compose:
|
||||
|
||||
cd /path/to/archivebox2/data
|
||||
archivebox init
|
||||
archivebox status
|
||||
```bash
|
||||
docker compose run --rm archivebox init
|
||||
docker compose run --rm archivebox update --rescan --migrate-only --index-only
|
||||
docker compose run --rm archivebox status
|
||||
```
|
||||
|
||||
# ... repeat the same for each collection if merging more than two
|
||||
```
|
||||
`--rescan` deliberately performs an O(N) scan of snapshot directories. It imports sealed orphan snapshots, restores missing archive-result records and tags, and repairs missing metadata files and crawl links for known snapshots. It does not recover missing payload bytes or guess the identity of malformed metadata. Unresolved directories are reported, retained in place, and cause a nonzero exit status. Real legacy `archive/<timestamp>/` directories still use the existing filesystem migration flow.
|
||||
|
||||
2. Create a new empty archivebox collection in a new folder somewhere, this will hold the new merged collection
|
||||
```bash
|
||||
mkdir -p /path/to/archivebox_new/data
|
||||
cd /path/to/archivebox_new/data
|
||||
archivebox init
|
||||
```
|
||||
The scan preserves Snapshot, Crawl, and ArchiveResult IDs and the timestamps present in their metadata. ID/URL/timestamp conflicts are reported rather than assigned replacement IDs. Existing destination records take precedence over imported metadata. Missing owners are created as inactive accounts, without importing passwords or granting administrator privileges. Older metadata without permissions imports privately; exports without Crawl records or creation timestamps cannot reproduce those missing fields exactly. Machine-local processes, credentials, schedules, and full user accounts are not synchronized.
|
||||
|
||||
3. Copy the real legacy `archive/<timestamp>/` directories from each old collection into the new collection's `archive/` folder.
|
||||
```bash
|
||||
rsync --archive --info=progress2 /path/to/archivebox1/data/archive/ /path/to/archivebox_new/data/archive/
|
||||
rsync --archive --info=progress2 /path/to/archivebox2/data/archive/ /path/to/archivebox_new/data/archive/
|
||||
# ...repeat the same for each collection if merging more than two
|
||||
```
|
||||
Stop with Ctrl+C and rerun the same command to resume. There is no checkpoint file or new database state: each successful record is durable, and the scan compares existing IDs/results/tags/links before doing repair work. It discovers orphans before repairing known snapshots, with newest date/UUID names first in each group. Completed imports move to the known group on restart. Add `--reverse` to process oldest first instead. Known snapshot asset checks reuse the existing output manifest scanner. A restart still enumerates directories and reads metadata to find incomplete work; timestamps alone cannot prove completeness because copied files may retain old dates. Filters, `--resume`, and `--continuous` cannot be combined with `--rescan`.
|
||||
|
||||
4. Run `archivebox update` in the new collection to import the legacy directories
|
||||
```bash
|
||||
cd /path/to/archivebox_new/data
|
||||
archivebox update
|
||||
```
|
||||
`--rescan` discovers filesystem-only snapshots and implies filesystem maintenance without new captures. `--migrate-only` eagerly finishes pending migrations for known database snapshots; ordinary updates leave them lazy. `--index-only` checks known snapshots, rebuilds output metadata, and backfills missing search indexes from archived content. All three explicit modes default to newest first; `--reverse` flips the order. `archivebox update --rescan` is enough when search backfill is unnecessary.
|
||||
|
||||
5. The new collection should now contain all the entries from the old collections combined
|
||||
```bash
|
||||
cd /path/to/archivebox_new/data
|
||||
archivebox status
|
||||
|
||||
# optionally force an update of the snapshot index files (normally done lazily)
|
||||
archivebox update --index-only
|
||||
```
|
||||
For more information about why Snapshot index files are usually updated lazily, see: https://github.com/ArchiveBox/ArchiveBox/issues/962
|
||||
|
||||
After you've confirmed your Snapshots are present in the new index, the old `index.sqlite3`, `index.json`, `index.html`, etc. main index files from the old archives can be safely deleted. You can optionally merge the contents of `ArchiveBox.conf` (your ArchiveBox config options), `sources/` (copies of all URLs imported in their original format), `logs/` (ArchiveBox error logs and debug info), and other root-level items yourself if that data is important to you.
|
||||
For shared remote mounts, first make sure newly written objects have uploaded and the destination's directory cache can see them. Sharing `archive/` does not synchronize independent databases automatically. Confirm imported entries and representative archived files before removing any source or backup copies.
|
||||
|
||||
---
|
||||
|
||||
@@ -100,7 +79,7 @@ VALUES ('GENERATED_PASSWORD_HASH', NULL, 0, 'someUsername', '', '', 'someEmail@e
|
||||
Replace the values above with the desired username, email, and password hash from python output^.
|
||||
|
||||
3. Log in using the new generated user to confirm it works
|
||||
http://admin.archivebox.localhost:8000/admin/login/ user: `someUsername` pass:`somePasswordHere`
|
||||
http://admin.archivebox.localhost:5797/admin/login/ user: `someUsername` pass:`somePasswordHere`
|
||||
|
||||
More info:
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage
|
||||
|
||||
@@ -8,10 +8,10 @@ There are two ways to publish your archive: using the `archivebox server` or by
|
||||
|
||||
```bash
|
||||
# start the webserver
|
||||
archivebox server 0.0.0.0:8000
|
||||
archivebox server 0.0.0.0:5797
|
||||
```
|
||||
|
||||
Open <http://admin.archivebox.localhost:8000/admin/> in a browser to create the first admin and finish web setup. To create additional accounts from the CLI, use `archivebox manage createsuperuser`.
|
||||
Open <http://admin.archivebox.localhost:5797/admin/> in a browser to create the first admin and finish web setup. To create additional accounts from the CLI, use `archivebox manage createsuperuser`.
|
||||
|
||||
This server is enabled out-of-the-box if you're using `docker-compose` to run ArchiveBox.
|
||||
If `BASE_URL` is not configured yet, complete the first-run wizard before exposing ArchiveBox. It guides you through choosing a security mode and shows the settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI. Configure HTTPS there, then open the canonical URL you chose in the wizard. Existing `BASE_URL` and security settings are trusted, so configured servers skip this wizard.
|
||||
@@ -39,8 +39,8 @@ archivebox list --json --with-headers > index.json
|
||||
# e.g. github pages or another static hosting provider
|
||||
|
||||
# you can also serve it with the simple python HTTP server
|
||||
python3 -m http.server --bind 0.0.0.0 --directory . 8000
|
||||
open http://127.0.0.1:8000
|
||||
python3 -m http.server --bind 0.0.0.0 --directory . 5797
|
||||
open http://127.0.0.1:5797
|
||||
```
|
||||
|
||||
Here's a sample nginx configuration that works to serve your static archive folder:
|
||||
|
||||
+2
-2
@@ -63,7 +63,7 @@ Open `./archive` to view your archive data in the filesystem.
|
||||
You can also use the interactive Web UI to view/manage/add links to your archive:
|
||||
```bash
|
||||
# with plain Docker:
|
||||
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
|
||||
|
||||
# with Docker Compose:
|
||||
docker compose up -d
|
||||
@@ -72,7 +72,7 @@ docker compose up -d
|
||||
archivebox server
|
||||
|
||||
# Open /admin/ on the hostname or IP used to reach ArchiveBox.
|
||||
# Local example: http://admin.archivebox.localhost:8000/admin/
|
||||
# Local example: http://admin.archivebox.localhost:5797/admin/
|
||||
# If BASE_URL is not configured yet, continue through the web setup wizard.
|
||||
```
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ ArchiveBox v0.9 is our biggest release since the 0.7.x series. It makes everyday
|
||||
|
||||
- 🔌 **A substantially expanded automation surface.** The Django Ninja REST API now exposes more collection, crawl, Snapshot, result, tag, user, token, and operational workflows. API tokens, outbound webhooks, and browser-extension uploads make ArchiveBox easier to connect to other tools.
|
||||
|
||||
- 🤖 **Optional AI agents and Claude-powered archiving.** Disabled by default, the built-in `archivebox mcp` server lets a dedicated AI agent perform crawls, search and manage your archive, prune duplicate results, and more. New optional plugins add [Claude-powered browser interaction](https://archivebox.github.io/abx-plugins/#claudechrome), [custom content extraction](https://archivebox.github.io/abx-plugins/#claudecodeextract), and [duplicate-result cleanup](https://archivebox.github.io/abx-plugins/#claudecodecleanup).
|
||||
- 🤖 **Optional AI agents and Claude-powered archiving.** Disabled by default, the built-in `archivebox mcp` server lets a dedicated AI agent perform crawls, search and manage your archive, prune duplicate results, and more. New optional plugins add [Claude-powered browser interaction](https://plugins.archivebox.io/#claudechrome), [custom content extraction](https://plugins.archivebox.io/#claudecodeextract), and [duplicate-result cleanup](https://plugins.archivebox.io/#claudecodecleanup).
|
||||
|
||||
- 📦 **Simpler dependency and installation management.** `archivebox install` can discover and install required browser and command-line dependencies, and `uv tool install archivebox` provides a cleaner native installation path with isolated Python dependencies.
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ With the new orchestrator flow, you only need the main `archivebox` service:
|
||||
services:
|
||||
archivebox:
|
||||
image: archivebox/archivebox:dev
|
||||
command: server --init 0.0.0.0:8000
|
||||
command: server --init 0.0.0.0:5797
|
||||
volumes:
|
||||
- ./data:/data
|
||||
```
|
||||
|
||||
+72
-72
@@ -6,7 +6,7 @@ These desktop, tablet, and mobile screenshots cover ArchiveBox's major public an
|
||||
|
||||
## Login
|
||||
|
||||
View: [`/admin/login/`](http://admin.archivebox.localhost:9292/admin/login/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/login.html)
|
||||
View: [`/admin/login/`](http://admin.archivebox.localhost:5797/admin/login/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/login.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -16,7 +16,7 @@ View: [`/admin/login/`](http://admin.archivebox.localhost:9292/admin/login/) ·
|
||||
|
||||
## Public snapshot list
|
||||
|
||||
View: [`/public/`](http://web.archivebox.localhost:9292/public/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
View: [`/public/`](http://web.archivebox.localhost:5797/public/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -26,7 +26,7 @@ View: [`/public/`](http://web.archivebox.localhost:9292/public/) · [View code](
|
||||
|
||||
## Snapshot View (capture in progress)
|
||||
|
||||
View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:9292) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:5797) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -36,7 +36,7 @@ View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:9292) · [View code](h
|
||||
|
||||
## Add URLs
|
||||
|
||||
View: [`/add/`](http://admin.archivebox.localhost:9292/add/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
View: [`/add/`](http://admin.archivebox.localhost:5797/add/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -46,7 +46,7 @@ View: [`/add/`](http://admin.archivebox.localhost:9292/add/) · [View code](http
|
||||
|
||||
## Admin dashboard
|
||||
|
||||
View: [`/admin/`](http://admin.archivebox.localhost:9292/admin/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_site.py)
|
||||
View: [`/admin/`](http://admin.archivebox.localhost:5797/admin/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_site.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -56,7 +56,7 @@ View: [`/admin/`](http://admin.archivebox.localhost:9292/admin/) · [View code](
|
||||
|
||||
## AI agent
|
||||
|
||||
View: [`/admin/agent/`](http://admin.archivebox.localhost:9292/admin/agent/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/abx_plugins/plugins/opencode/views.py)
|
||||
View: [`/admin/agent/`](http://admin.archivebox.localhost:5797/admin/agent/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/abx_plugins/plugins/opencode/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -66,7 +66,7 @@ View: [`/admin/agent/`](http://admin.archivebox.localhost:9292/admin/agent/) ·
|
||||
|
||||
## Snapshots table
|
||||
|
||||
View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
|
||||
View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -76,7 +76,7 @@ View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:9292/admin/cor
|
||||
|
||||
## Snapshots grid
|
||||
|
||||
View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/grid/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/snapshots_grid.html)
|
||||
View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/grid/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/snapshots_grid.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -86,7 +86,7 @@ View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:9292/admi
|
||||
|
||||
## Snapshot admin detail
|
||||
|
||||
View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
|
||||
View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -96,7 +96,7 @@ View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://a
|
||||
|
||||
## Snapshot files
|
||||
|
||||
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292/?files=1) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/static_index.html)
|
||||
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:5797/?files=1) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/static_index.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -106,7 +106,7 @@ View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292/?files=1) · [Vie
|
||||
|
||||
## Archive results
|
||||
|
||||
View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:9292/admin/core/archiveresult/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
|
||||
View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:5797/admin/core/archiveresult/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -116,7 +116,7 @@ View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:9292/admi
|
||||
|
||||
## Archive result detail
|
||||
|
||||
View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](http://admin.archivebox.localhost:9292/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
|
||||
View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](http://admin.archivebox.localhost:5797/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -126,7 +126,7 @@ View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](htt
|
||||
|
||||
## Tags
|
||||
|
||||
View: [`/admin/core/tag/`](http://admin.archivebox.localhost:9292/admin/core/tag/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
|
||||
View: [`/admin/core/tag/`](http://admin.archivebox.localhost:5797/admin/core/tag/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -136,7 +136,7 @@ View: [`/admin/core/tag/`](http://admin.archivebox.localhost:9292/admin/core/tag
|
||||
|
||||
## Tag detail
|
||||
|
||||
View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:9292/admin/core/tag/10129/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
|
||||
View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:5797/admin/core/tag/10129/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -146,7 +146,7 @@ View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:9292/a
|
||||
|
||||
## Users
|
||||
|
||||
View: [`/admin/auth/user/`](http://admin.archivebox.localhost:9292/admin/auth/user/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
|
||||
View: [`/admin/auth/user/`](http://admin.archivebox.localhost:5797/admin/auth/user/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -156,7 +156,7 @@ View: [`/admin/auth/user/`](http://admin.archivebox.localhost:9292/admin/auth/us
|
||||
|
||||
## User detail
|
||||
|
||||
View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:9292/admin/auth/user/49/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
|
||||
View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:5797/admin/auth/user/49/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -166,7 +166,7 @@ View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:9292/adm
|
||||
|
||||
## Crawls
|
||||
|
||||
View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:9292/admin/crawls/crawl/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:5797/admin/crawls/crawl/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -176,7 +176,7 @@ View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:9292/admin/craw
|
||||
|
||||
## Crawl detail
|
||||
|
||||
View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://admin.archivebox.localhost:9292/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://admin.archivebox.localhost:5797/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -186,7 +186,7 @@ View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://ad
|
||||
|
||||
## Crawl schedules
|
||||
|
||||
View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:9292/admin/crawls/crawlschedule/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:5797/admin/crawls/crawlschedule/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -196,7 +196,7 @@ View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:9292/ad
|
||||
|
||||
## Crawl schedule detail
|
||||
|
||||
View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](http://admin.archivebox.localhost:9292/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](http://admin.archivebox.localhost:5797/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -206,7 +206,7 @@ View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](h
|
||||
|
||||
## Personas
|
||||
|
||||
View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:9292/admin/personas/persona/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
|
||||
View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:5797/admin/personas/persona/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -216,7 +216,7 @@ View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:9292/admin/
|
||||
|
||||
## Persona detail
|
||||
|
||||
View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http://admin.archivebox.localhost:9292/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
|
||||
View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http://admin.archivebox.localhost:5797/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -226,7 +226,7 @@ View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http:
|
||||
|
||||
## Machines
|
||||
|
||||
View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:9292/admin/machine/machine/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:5797/admin/machine/machine/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -236,7 +236,7 @@ View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:9292/admin/m
|
||||
|
||||
## Machine detail
|
||||
|
||||
View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http://admin.archivebox.localhost:9292/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http://admin.archivebox.localhost:5797/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -246,7 +246,7 @@ View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http:/
|
||||
|
||||
## Network interfaces
|
||||
|
||||
View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:9292/admin/machine/networkinterface/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:5797/admin/machine/networkinterface/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -256,7 +256,7 @@ View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:929
|
||||
|
||||
## Network interface detail
|
||||
|
||||
View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/`](http://admin.archivebox.localhost:9292/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/`](http://admin.archivebox.localhost:5797/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -266,7 +266,7 @@ View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/
|
||||
|
||||
## Binaries
|
||||
|
||||
View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:9292/admin/machine/binary/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:5797/admin/machine/binary/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -276,7 +276,7 @@ View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:9292/admin/ma
|
||||
|
||||
## Binary detail
|
||||
|
||||
View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://admin.archivebox.localhost:9292/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://admin.archivebox.localhost:5797/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -286,7 +286,7 @@ View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://
|
||||
|
||||
## Processes
|
||||
|
||||
View: [`/admin/machine/process/`](http://admin.archivebox.localhost:9292/admin/machine/process/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/process/`](http://admin.archivebox.localhost:5797/admin/machine/process/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -296,7 +296,7 @@ View: [`/admin/machine/process/`](http://admin.archivebox.localhost:9292/admin/m
|
||||
|
||||
## Process detail
|
||||
|
||||
View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http://admin.archivebox.localhost:9292/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http://admin.archivebox.localhost:5797/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -306,7 +306,7 @@ View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http:/
|
||||
|
||||
## API tokens
|
||||
|
||||
View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:9292/admin/api/apitoken/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:5797/admin/api/apitoken/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -316,7 +316,7 @@ View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:9292/admin/api/
|
||||
|
||||
## API token detail
|
||||
|
||||
View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://admin.archivebox.localhost:9292/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://admin.archivebox.localhost:5797/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -326,7 +326,7 @@ View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://ad
|
||||
|
||||
## Webhooks
|
||||
|
||||
View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:9292/admin/api/outboundwebhook/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:5797/admin/api/outboundwebhook/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -336,7 +336,7 @@ View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:9292/adm
|
||||
|
||||
## Webhook detail
|
||||
|
||||
View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](http://admin.archivebox.localhost:9292/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](http://admin.archivebox.localhost:5797/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -346,7 +346,7 @@ View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](ht
|
||||
|
||||
## Environment
|
||||
|
||||
View: [`/admin/environment/`](http://admin.archivebox.localhost:9292/admin/environment/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/settings.py)
|
||||
View: [`/admin/environment/`](http://admin.archivebox.localhost:5797/admin/environment/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/settings.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -356,7 +356,7 @@ View: [`/admin/environment/`](http://admin.archivebox.localhost:9292/admin/envir
|
||||
|
||||
## Configuration
|
||||
|
||||
View: [`/admin/environment/config/`](http://admin.archivebox.localhost:9292/admin/environment/config/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
View: [`/admin/environment/config/`](http://admin.archivebox.localhost:5797/admin/environment/config/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -366,7 +366,7 @@ View: [`/admin/environment/config/`](http://admin.archivebox.localhost:9292/admi
|
||||
|
||||
## Configuration detail
|
||||
|
||||
View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:9292/admin/environment/config/BASE_URL/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:5797/admin/environment/config/BASE_URL/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -376,7 +376,7 @@ View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:
|
||||
|
||||
## Dependencies
|
||||
|
||||
View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:9292/admin/environment/binaries/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:5797/admin/environment/binaries/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -386,7 +386,7 @@ View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:9292/ad
|
||||
|
||||
## Dependency detail
|
||||
|
||||
View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:9292/admin/environment/binaries/abxbus/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:5797/admin/environment/binaries/abxbus/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -396,7 +396,7 @@ View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:
|
||||
|
||||
## Plugins
|
||||
|
||||
View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:9292/admin/environment/plugins/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/plugins/views.py)
|
||||
View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:5797/admin/environment/plugins/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/plugins/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -406,7 +406,7 @@ View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:9292/adm
|
||||
|
||||
## Workers
|
||||
|
||||
View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:9292/admin/environment/workers/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:5797/admin/environment/workers/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -416,7 +416,7 @@ View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:9292/adm
|
||||
|
||||
## Worker detail
|
||||
|
||||
View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localhost:9292/admin/environment/workers/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localhost:5797/admin/environment/workers/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -426,7 +426,7 @@ View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localh
|
||||
|
||||
## Logs
|
||||
|
||||
View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:9292/admin/environment/logs/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:5797/admin/environment/logs/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -436,7 +436,7 @@ View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:9292/admin/
|
||||
|
||||
## Log detail
|
||||
|
||||
View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost:9292/admin/environment/logs/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost:5797/admin/environment/logs/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -446,7 +446,7 @@ View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost
|
||||
|
||||
## Snapshot View (singlefile)
|
||||
|
||||
View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:9292#singlefile) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:5797#singlefile) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -456,7 +456,7 @@ View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:9292#single
|
||||
|
||||
## Snapshot View (screenshot)
|
||||
|
||||
View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:9292#screenshot) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:5797#screenshot) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -466,7 +466,7 @@ View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:9292#screen
|
||||
|
||||
## Snapshot View (wget)
|
||||
|
||||
View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:9292#wget) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:5797#wget) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -476,7 +476,7 @@ View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:9292#wget) · [Vi
|
||||
|
||||
## Snapshot View (dom)
|
||||
|
||||
View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dom) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:5797#dom) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -486,7 +486,7 @@ View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dom) · [View
|
||||
|
||||
## Snapshot View (pdf)
|
||||
|
||||
View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pdf) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:5797#pdf) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -496,7 +496,7 @@ View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pdf) · [View
|
||||
|
||||
## Snapshot View (readability)
|
||||
|
||||
View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:9292#readability) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:5797#readability) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -506,7 +506,7 @@ View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:9292#reada
|
||||
|
||||
## Snapshot View (ytdlp)
|
||||
|
||||
View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ytdlp) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:5797#ytdlp) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -516,7 +516,7 @@ View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ytdlp) · [
|
||||
|
||||
## Snapshot View (responses)
|
||||
|
||||
View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:9292#responses) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:5797#responses) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -526,7 +526,7 @@ View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:9292#respons
|
||||
|
||||
## Snapshot View (chrome_mhtml)
|
||||
|
||||
View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome_mhtml) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:5797#chrome_mhtml) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -536,7 +536,7 @@ View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chro
|
||||
|
||||
## Snapshot View (defuddle)
|
||||
|
||||
View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:9292#defuddle) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:5797#defuddle) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -546,7 +546,7 @@ View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:9292#defuddle
|
||||
|
||||
## Snapshot View (mercury)
|
||||
|
||||
View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:9292#mercury) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:5797#mercury) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -556,7 +556,7 @@ View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:9292#mercury)
|
||||
|
||||
## Snapshot View (archivewebpage)
|
||||
|
||||
View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:9292#archivewebpage) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:5797#archivewebpage) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -566,7 +566,7 @@ View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ar
|
||||
|
||||
## Snapshot View (chrome)
|
||||
|
||||
View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:5797#chrome) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -576,7 +576,7 @@ View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome) ·
|
||||
|
||||
## Snapshot View (consolelog)
|
||||
|
||||
View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:9292#consolelog) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:5797#consolelog) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -586,7 +586,7 @@ View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:9292#consol
|
||||
|
||||
## Snapshot View (dns)
|
||||
|
||||
View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dns) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:5797#dns) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -596,7 +596,7 @@ View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dns) · [View
|
||||
|
||||
## Snapshot View (sslcerts)
|
||||
|
||||
View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:9292#sslcerts) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:5797#sslcerts) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -606,7 +606,7 @@ View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:9292#sslcerts
|
||||
|
||||
## Snapshot View (redirects)
|
||||
|
||||
View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:9292#redirects) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:5797#redirects) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -616,7 +616,7 @@ View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:9292#redirec
|
||||
|
||||
## Snapshot View (headers)
|
||||
|
||||
View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:9292#headers) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:5797#headers) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -626,7 +626,7 @@ View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:9292#headers)
|
||||
|
||||
## Snapshot View (seo)
|
||||
|
||||
View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:9292#seo) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:5797#seo) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -636,7 +636,7 @@ View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:9292#seo) · [View
|
||||
|
||||
## Snapshot View (accessibility)
|
||||
|
||||
View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:9292#accessibility) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:5797#accessibility) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -646,7 +646,7 @@ View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:9292#acc
|
||||
|
||||
## Snapshot View (htmltotext)
|
||||
|
||||
View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:9292#htmltotext) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:5797#htmltotext) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -656,7 +656,7 @@ View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:9292#htmlto
|
||||
|
||||
## Snapshot View (trafilatura)
|
||||
|
||||
View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:9292#trafilatura) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:5797#trafilatura) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -666,7 +666,7 @@ View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:9292#trafi
|
||||
|
||||
## Snapshot View (liteparse)
|
||||
|
||||
View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:9292#liteparse) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:5797#liteparse) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -676,7 +676,7 @@ View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:9292#litepar
|
||||
|
||||
## Snapshot View (parse_html_urls)
|
||||
|
||||
View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_html_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_html_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -686,7 +686,7 @@ View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#p
|
||||
|
||||
## Snapshot View (parse_txt_urls)
|
||||
|
||||
View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_txt_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_txt_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -696,7 +696,7 @@ View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pa
|
||||
|
||||
## Snapshot View (parse_dom_outlinks)
|
||||
|
||||
View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_dom_outlinks) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_dom_outlinks) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -706,7 +706,7 @@ View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:929
|
||||
|
||||
## Snapshot View (hashes)
|
||||
|
||||
View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:9292#hashes) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:5797#hashes) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
@@ -716,7 +716,7 @@ View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:9292#hashes) ·
|
||||
|
||||
## Snapshot View (header collapsed)
|
||||
|
||||
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:5797) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
|
||||
|
||||
<table><thead><tr>
|
||||
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
|
||||
|
||||
@@ -28,7 +28,7 @@ The default mode should not be used for archiving entire browser history or auth
|
||||
|
||||
```bash
|
||||
# (these are the defaults)
|
||||
archivebox config --set ARCHIVEDOTORG_ENABLED=True # see https://archivebox.github.io/abx-plugins/#archivedotorg
|
||||
archivebox config --set ARCHIVEDOTORG_ENABLED=True # see https://plugins.archivebox.io/#archivedotorg
|
||||
archivebox persona create public
|
||||
archivebox add --persona=public 'https://example.com'
|
||||
```
|
||||
@@ -54,10 +54,10 @@ If you're importing private links or authenticated content, you probably don't w
|
||||
|
||||
#### ⚠️ Things to watch out for: ⚠️
|
||||
|
||||
- any cookies / secret state present in a Chrome user profile or `cookies.txt` file may be reflected in server responses and saved in the Snapshot output (e.g. in [`headers`](https://archivebox.github.io/abx-plugins/#headers) extractor output) — visible in cleartext to anyone viewing the Snapshot. **Don't use your personal Chrome profile for archiving** or people viewing your archive can then authenticate as you.
|
||||
- any secret tokens embedded in URLs (e.g. secret invite links, Google Doc URLs, etc.) will be visible on `archive.org` as the URLs are not filtered when saving to it. Disable submitting to Archive.org entirely with [`ARCHIVEDOTORG_ENABLED=False`](https://archivebox.github.io/abx-plugins/#archivedotorg).
|
||||
- the domain portion in archived URLs is sent to a favicon service in order to retrieve an icon more reliably than a janky internal implementation would be able to (if leaking domains is a concern, you can change the [`FAVICON_PROVIDER`](https://archivebox.github.io/abx-plugins/#favicon) or disable favicon fetching entirely with [`FAVICON_ENABLED=False`](https://archivebox.github.io/abx-plugins/#favicon)).
|
||||
- [viewing malicious archived JS could allow an attacker to access your other archive items + the admin interface](https://github.com/ArchiveBox/ArchiveBox/issues/239) — use the default [`SERVER_SECURITY_MODE=auto`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#server_security_mode) (`*.localhost` uses isolated replay subdomains and other hosts use one-domain/no-JS raw replay), or disable risky extractors entirely with [`WGET_ENABLED=False`](https://archivebox.github.io/abx-plugins/#wget) and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom). Plugins that need their own trusted viewer can opt in with an explicit `full.html` preview template; ArchiveBox applies the no-JS policy to all other raw archived HTML without maintaining a plugin whitelist.
|
||||
- any cookies / secret state present in a Chrome user profile or `cookies.txt` file may be reflected in server responses and saved in the Snapshot output (e.g. in [`headers`](https://plugins.archivebox.io/#headers) extractor output) — visible in cleartext to anyone viewing the Snapshot. **Don't use your personal Chrome profile for archiving** or people viewing your archive can then authenticate as you.
|
||||
- any secret tokens embedded in URLs (e.g. secret invite links, Google Doc URLs, etc.) will be visible on `archive.org` as the URLs are not filtered when saving to it. Disable submitting to Archive.org entirely with [`ARCHIVEDOTORG_ENABLED=False`](https://plugins.archivebox.io/#archivedotorg).
|
||||
- the domain portion in archived URLs is sent to a favicon service in order to retrieve an icon more reliably than a janky internal implementation would be able to (if leaking domains is a concern, you can change the [`FAVICON_PROVIDER`](https://plugins.archivebox.io/#favicon) or disable favicon fetching entirely with [`FAVICON_ENABLED=False`](https://plugins.archivebox.io/#favicon)).
|
||||
- [viewing malicious archived JS could allow an attacker to access your other archive items + the admin interface](https://github.com/ArchiveBox/ArchiveBox/issues/239) — use the default [`SERVER_SECURITY_MODE=auto`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#server_security_mode) (`*.localhost` uses isolated replay subdomains and other hosts use one-domain/no-JS raw replay), or disable risky extractors entirely with [`WGET_ENABLED=False`](https://plugins.archivebox.io/#wget) and [`DOM_ENABLED=False`](https://plugins.archivebox.io/#dom). Plugins that need their own trusted viewer can opt in with an explicit `full.html` preview template; ArchiveBox applies the no-JS policy to all other raw archived HTML without maintaining a plugin whitelist.
|
||||
|
||||
<br/>
|
||||
<img src="https://imgur.zervice.io/Jszo4h2.png" width="400px"/>
|
||||
|
||||
Loaded 100 of 152 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user