Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Nick Sweeting
|
42f8a56a9d |
No files matched your search
@@ -30,8 +30,8 @@ body:
|
||||
description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
|
||||
render: markdown
|
||||
placeholder: |
|
||||
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 5797:5797 archivebox/archivebox` in iTerm2
|
||||
2. Went to the https://127.0.0.1:5797/add/ page in Google Chrome
|
||||
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
|
||||
2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
|
||||
3. Typed 'https://example.com/xyz' into the 'Add URL' input field
|
||||
4. Clicked the 'Add+' button
|
||||
5. Got a 500 error and saw the errors below in terminal
|
||||
@@ -138,7 +138,7 @@ body:
|
||||
archivebox:
|
||||
image: archivebox/archivebox:latest
|
||||
ports:
|
||||
- 5797:5797
|
||||
- 8000:8000
|
||||
volumes:
|
||||
- ./data:/data
|
||||
environment:
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
# Stable releases require live dev acceptance
|
||||
|
||||
The order is **dev CI → dev image publication → automatic Cabbage deployment →
|
||||
real capture and browser acceptance → stable publication**. Never
|
||||
substitute a published PyPI wheel, healthy container, or local build for this
|
||||
acceptance. The release workflow blocks *before* a stable upload unless Cabbage
|
||||
has successful evidence from `staging-acceptance.yml` matching all tracked
|
||||
source and dependency pins. Only ArchiveBox's own version fields may change
|
||||
between dev and stable.
|
||||
|
||||
The staging workflow observes the existing image watcher; it cannot deploy or
|
||||
restart the application. On Cabbage, install `bin/staging-acceptance-host.sh`
|
||||
as root-owned mode 0755 `/usr/local/sbin/archivebox-staging-acceptance`. Its hash
|
||||
is checked against the candidate. A dedicated SSH key must use
|
||||
`restrict,command="/usr/local/sbin/archivebox-staging-acceptance"` in authorized
|
||||
keys. It accepts only `inspect <source-sha>` and `capture <source-sha>`; it cannot
|
||||
run caller-provided commands or archive arbitrary URLs.
|
||||
|
||||
The `staging` GitHub environment is restricted to dev/main branches. It contains
|
||||
`STAGING_SSH_KEY` and `STAGING_KNOWN_HOSTS`, with independently verified host keys.
|
||||
Only trusted release events use the private ugNAS runner route. Never expose
|
||||
these credentials to PR jobs or copy an operator's existing root key.
|
||||
|
||||
Acceptance creates one tagged background crawl per source through the normal
|
||||
add/runner path, enabling every snapshot extractor, including TLSNotary. It
|
||||
checks installed pins, completed outputs and their files, unexpected/missing
|
||||
plugin results, host OOM kills, and real rendered snapshot pages. For this
|
||||
release, the explicitly accepted TLSNotary failures and exact missing-Claude-auth
|
||||
diagnostics are retained in the report's `limitations` list; they do not disable
|
||||
those plugins or excuse missing/truncated successful outputs. Remove the
|
||||
TLSNotary exception when its follow-up fix ships. Logs, result inventories and
|
||||
screenshots are retained as Actions artifacts. Other failures remain
|
||||
failures; do not publish a success deployment record to bypass them. No code
|
||||
mounts are allowed. The Cabbage job must pass before its record authorizes a
|
||||
stable upload. Queued capture progress is polled; failed tests are not retried.
|
||||
|
||||
Browser acceptance follows the default WACZ viewer into the captured document,
|
||||
matching its replay base and original URL and checking that its body is visible
|
||||
and contains substantial text. An attached viewer iframe can still be
|
||||
`about:blank`, so its presence alone cannot prove replay works. Extracted display
|
||||
titles and the captured document's `<title>` may legitimately differ; neither
|
||||
title equality nor output thumbnails substitute for checking the actual page.
|
||||
@@ -1,3 +0,0 @@
|
||||
self-hosted-runner:
|
||||
labels:
|
||||
- ugnas-linux
|
||||
@@ -116,10 +116,6 @@
|
||||
}
|
||||
],
|
||||
"docker_binaries": [
|
||||
{
|
||||
"name": "{BASH_BINARY}",
|
||||
"binproviders": "env,brew,apt"
|
||||
},
|
||||
{
|
||||
"name": "{DOCKER_BINARY}",
|
||||
"binproviders": "env"
|
||||
|
||||
@@ -1,119 +0,0 @@
|
||||
# archivebox.io website
|
||||
|
||||
This repository owns its complete website. Build and deploy it without checking
|
||||
out another repository's website, fetching shared snippets, or calling a shared
|
||||
workflow. Common files are ordinary committed copies; update them deliberately.
|
||||
|
||||
## Repository-specific files
|
||||
|
||||
- Content: `publicsite/index.html`.
|
||||
- Site/gallery generation: `bin/generate_ui_screenshot_gallery.py`.
|
||||
- Application capture: `bin/collect_ui_screenshots.sh`.
|
||||
- Deployment branch: `dev`.
|
||||
|
||||
## Common publication contract
|
||||
|
||||
1. Build pull requests without deploying them.
|
||||
2. Build website pushes immediately, independently of application capture CI.
|
||||
3. Restore validated screenshots from trusted default-branch capture jobs.
|
||||
Keep the original capture revision, timestamp, coverage and image checksums.
|
||||
4. Retain `site-screenshots` artifacts for 90 days. After expiry, restore the
|
||||
published image set. Never replace a complete gallery with an empty one when
|
||||
restoration fails. Apple device groups are independently validated artifacts;
|
||||
Android's checked-in bootstrap captures retain their local provenance.
|
||||
5. Render current content and layout around those captures. Site revision and
|
||||
capture revision are separate facts; reusing captures must not relabel them.
|
||||
6. Serialize production restore/build/deploy under one concurrency group and
|
||||
check out the latest default-branch revision when each build starts. Keep the
|
||||
built revision in provenance; later release-bot commits must not block deploys.
|
||||
7. Keep platform-specific build, installation, capture, and completeness checks
|
||||
in the application's existing capture workflow. Site-only edits do not need
|
||||
to rerun those jobs.
|
||||
|
||||
## Common presentation contract
|
||||
|
||||
The base header, Apps menu and icons, footer, brand styles and navigation behavior
|
||||
should remain identical copies across the seven product sites. Put this site's
|
||||
name, navigation, CTA and extra footer resources outside the common base files.
|
||||
Preserve real links, anchor IDs, no-JavaScript navigation, keyboard operation,
|
||||
mobile layout, reduced-motion preferences and existing product-specific content.
|
||||
Screenshots stay in English. Translation content, when added, lives in separate
|
||||
HTML files; no runtime translation service or cross-repository dependency.
|
||||
|
||||
## Editing and building
|
||||
|
||||
`base/` contains identical independent copies across the seven product sites.
|
||||
Common Python scripts share the same behavior; repository formatters may change
|
||||
line wrapping without changing their syntax trees. The base owns the shared logo, Apps menu/icons, footer directory, navigation
|
||||
behavior and styles. `site.json` and `nav.html` contain this site's name, routes,
|
||||
links and CTA. Product content and additional footer resources stay in the native
|
||||
HTML/Jinja/Liquid templates listed above. The `ARCHIVEBOX:*` comments are build
|
||||
slots; the renderer fills them with static HTML, never browser-side fetches.
|
||||
|
||||
The existing builder calls `site.py render` after generating its content. This
|
||||
copies `base/chrome.css` and `base/chrome.js` to `site-base/`, assembles the common
|
||||
HTML, validates local resources and writes the site revision to root `build.json`.
|
||||
Screenshot manifests are left untouched. No build reads another repository.
|
||||
|
||||
Common changes are manual copy/paste edits. Compare `base/` and `site.py` with
|
||||
another product repository when updating them; do not add a synchronization job,
|
||||
package dependency or reusable cross-repository workflow. `nav.html`, `site.json`
|
||||
and the marked workflow build section are intentionally repository-specific.
|
||||
|
||||
Build: `uv run --no-project python .github/pages/site.py render _site --source publicsite` after restoring screenshots and refreshing the gallery.
|
||||
|
||||
Check the built site with real Chromium (also run by Pages CI):
|
||||
|
||||
```sh
|
||||
uv run --no-config --no-project --with playwright==1.63.0 playwright install chromium
|
||||
uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py SITE_OUTPUT --evidence /tmp/site-evidence
|
||||
```
|
||||
|
||||
Use this workflow's `SITE_OUTPUT` directory. The check covers desktop/mobile
|
||||
layout, keyboard dismissal, no-JavaScript links, local resources and visible
|
||||
images, and uploads its screenshots as `site-verification` for 14 days.
|
||||
|
||||
## Branding and future translations
|
||||
|
||||
`base/chrome.css` owns the shared system font stacks, plum accent and CTA shape,
|
||||
including the existing `.button` and `.ab-btn` classes. Product styles own their
|
||||
layouts, feature sections and platform-specific button states. Shared system
|
||||
fonts need no external font downloads. `site.py` emits the same complete SEO/OG/
|
||||
Twitter tags everywhere while retaining each native template's title, description,
|
||||
canonical URL, locale and social artwork; edit those values with that page's copy.
|
||||
|
||||
For translations, keep complete separate HTML files under `es/`, `fr/` and `zh/`,
|
||||
and list their output paths in `site.json` alongside the English pages. Share CSS,
|
||||
JS, social artwork and the English screenshot gallery. Translate each page's title
|
||||
and description, set its own canonical and `lang` (`zh` for Chinese), and add
|
||||
reciprocal `hreflang` links only for translations that actually exist. The renderer
|
||||
preserves these tags and computes shared asset paths for nested pages. Localized
|
||||
header/footer copy and the small language selector can be added with those pages;
|
||||
there is no language redirect, placeholder translation or translation runtime now.
|
||||
|
||||
## Screenshot strips
|
||||
|
||||
`marquee.py` and `base/marquee.css` / `base/marquee.js` are also identical copies.
|
||||
Product homepages choose their placement with an `ARCHIVEBOX:MARQUEE` comment.
|
||||
The renderer takes real images, captions, dimensions and links from the generated
|
||||
`screenshots/index.html`; there is no second screenshot list or browser fetch.
|
||||
Desktop/tablet/mobile galleries contribute their desktop views. Apple uses
|
||||
`MARQUEE-CLIENT` and `MARQUEE-SERVER` slots to keep its two products separate.
|
||||
Catalog sites without a slot do not load the strip assets.
|
||||
|
||||
The strip scrolls once, pauses on hover, focus or manual interaction, and has an
|
||||
explicit Play/Pause button. Reduced-motion users start paused. Without JavaScript,
|
||||
normal image links and horizontal scrolling remain available. Screenshot capture
|
||||
manifests and provenance are never rewritten by this presentation step.
|
||||
|
||||
Standalone one- or two-image illustrations remain in their authored source.
|
||||
Gallery strips with three or more images use the latest restored captures.
|
||||
|
||||
## Keeping the common layout in sync
|
||||
|
||||
The comments in `base/header.html`, `base/footer-start.html`,
|
||||
`base/footer-end.html`, `base/chrome.css`, and `base/chrome.js` list the peer
|
||||
repositories. Update those committed copies together when shared branding,
|
||||
Apps links, keyboard behavior, or footer resources change. Preserve each site's
|
||||
local navigation and content. The former monorepo `site-chrome` generator is
|
||||
retired; builds must use only this repository's files.
|
||||
@@ -1,102 +0,0 @@
|
||||
"""Common screenshot artifact transport; coverage checks belong to each app."""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import PurePosixPath
|
||||
from urllib.request import urlopen
|
||||
|
||||
|
||||
def api(repo, path):
|
||||
return json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/{path}"]))
|
||||
|
||||
|
||||
def trusted(run, repo, workflow, branch, successful=True):
|
||||
return (
|
||||
(run.get("head_repository") or {}).get("full_name", "").lower() == repo.lower()
|
||||
and run["head_branch"] == branch
|
||||
and run["event"] in ("push", "workflow_dispatch")
|
||||
and run["path"] == f".github/workflows/{workflow}"
|
||||
and (not successful or run["conclusion"] == "success")
|
||||
)
|
||||
|
||||
|
||||
def runs(repo, workflow, branch, successful=True, artifact_names=("site-screenshots",)):
|
||||
# Ask for capture artifacts directly instead of checking every historical
|
||||
# workflow run (most app runs never produce a complete screenshot set).
|
||||
# Artifact creation time also includes fresh captures from old rerun jobs.
|
||||
candidates = []
|
||||
for name in artifact_names:
|
||||
rows = subprocess.check_output(
|
||||
[
|
||||
"gh",
|
||||
"api",
|
||||
f"repos/{repo}/actions/artifacts?name={name}&per_page=100",
|
||||
"--paginate",
|
||||
"--jq",
|
||||
".artifacts[] | select(.expired == false) | @json",
|
||||
],
|
||||
text=True,
|
||||
)
|
||||
candidates.extend(json.loads(row) for row in rows.splitlines())
|
||||
seen = set()
|
||||
for artifact in sorted(candidates, key=lambda item: item["created_at"], reverse=True):
|
||||
source = artifact.get("workflow_run") or {}
|
||||
run_id = source.get("id")
|
||||
if (
|
||||
artifact["name"] not in artifact_names
|
||||
or not run_id
|
||||
or run_id in seen
|
||||
or source.get("head_branch") != branch
|
||||
or source.get("head_repository_id") != source.get("repository_id")
|
||||
):
|
||||
continue
|
||||
seen.add(run_id)
|
||||
run = api(repo, f"actions/runs/{run_id}")
|
||||
if trusted(run, repo, workflow, branch, successful):
|
||||
yield run
|
||||
|
||||
|
||||
def names(repo, run):
|
||||
rows = subprocess.check_output(
|
||||
[
|
||||
"gh",
|
||||
"api",
|
||||
f"repos/{repo}/actions/runs/{run['id']}/artifacts?per_page=100",
|
||||
"--paginate",
|
||||
"--jq",
|
||||
".artifacts[] | @json",
|
||||
],
|
||||
text=True,
|
||||
)
|
||||
return {artifact["name"] for row in rows.splitlines() if not (artifact := json.loads(row))["expired"]}
|
||||
|
||||
|
||||
def download(repo, run, name, destination):
|
||||
subprocess.run(
|
||||
[
|
||||
"gh",
|
||||
"run",
|
||||
"download",
|
||||
str(run["id"]),
|
||||
"--repo",
|
||||
repo,
|
||||
"--name",
|
||||
name,
|
||||
"--dir",
|
||||
str(destination),
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
def relative_path(name):
|
||||
path = PurePosixPath(name)
|
||||
if not name or path.is_absolute() or ".." in path.parts or any(c in name for c in ":\\?#%"):
|
||||
raise ValueError(f"Unsafe screenshot path: {name}")
|
||||
return path
|
||||
|
||||
|
||||
def fetch(base, name):
|
||||
relative_path(name)
|
||||
with urlopen(base + name, timeout=60) as response:
|
||||
return response.read()
|
||||
@@ -1,156 +0,0 @@
|
||||
/*
|
||||
Shared ArchiveBox website navigation styles and behavior.
|
||||
Keep this file in sync with .github/pages/base/chrome.css in:
|
||||
ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
|
||||
electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
|
||||
Each site builds its committed copy independently; there is no central generator.
|
||||
Keep site-specific navigation, CTAs, and extra footer links outside this base.
|
||||
Also review the inline navbar/footer in abx-dl/website/index.html and
|
||||
debian-archivebox/website/index.html when shared links or branding change.
|
||||
*/
|
||||
/* Common brand tokens. Native templates own product layout and content. */
|
||||
:root {
|
||||
--abx-font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
--abx-font-mono: ui-monospace, "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace;
|
||||
--abx-accent: #9b2854;
|
||||
--abx-accent-hover: #782040;
|
||||
--abx-paper: #fcfaf7;
|
||||
--abx-ink: #28252a;
|
||||
--abx-muted: #716c74;
|
||||
--abx-line: #e9e2e3;
|
||||
--font-sans: var(--abx-font-sans);
|
||||
--font-mono: var(--abx-font-mono);
|
||||
--ab-font-sans: var(--abx-font-sans);
|
||||
--ab-font-serif: var(--abx-font-sans);
|
||||
--ab-font-label: var(--abx-font-sans);
|
||||
--ab-font-mono: var(--abx-font-mono);
|
||||
--ab-accent: var(--abx-accent);
|
||||
--ab-accent-hover: var(--abx-accent-hover);
|
||||
--ab-on-accent: #fff;
|
||||
}
|
||||
body { font-family: var(--abx-font-sans); }
|
||||
code, pre, kbd, samp { font-family: var(--abx-font-mono); }
|
||||
/* Reuse existing CTA classes; platform selection continues to control colors. */
|
||||
.button, .ab-btn {
|
||||
display: inline-flex; align-items: center; justify-content: center; gap: 10px;
|
||||
min-height: 44px; height: auto; max-width: 100%; padding: 10px 20px;
|
||||
border-radius: 24px; font-family: var(--abx-font-sans);
|
||||
font-size: 14px; font-weight: 650; line-height: 1.4;
|
||||
text-align: center; white-space: normal; text-decoration: none;
|
||||
}
|
||||
.button.primary, .ab-btn--primary { background: var(--abx-accent); border-color: var(--abx-accent); color: #fff; }
|
||||
.button.primary:hover, .ab-btn--primary:hover { background: var(--abx-accent-hover); border-color: var(--abx-accent-hover); color: #fff; }
|
||||
.button:focus-visible, .ab-btn:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
|
||||
.button svg, .button img, .ab-btn svg, .ab-btn img { flex-shrink: 0; }
|
||||
@media (prefers-reduced-motion: reduce) { .button, .ab-btn { transition: none; } .button:hover { transform: none; } }
|
||||
|
||||
/* Shared ArchiveBox site chrome. Independent copy; edit alongside the other product sites, without a build dependency. */
|
||||
/* Native anchor navigation leaves room for the heading and its surrounding content. */
|
||||
html { scroll-behavior: smooth; scroll-padding-top: clamp(96px, 25vh, 240px); }
|
||||
@media (prefers-reduced-motion: reduce) { html { scroll-behavior: auto; } }
|
||||
.abx-header, .abx-footer { box-sizing: border-box; font-family: var(--abx-font-sans); font-size: 14px; line-height: 1.5; letter-spacing: normal; text-align: left; }
|
||||
.abx-header *, .abx-footer * { box-sizing: border-box; }
|
||||
.abx-header { position: sticky; top: 0; z-index: 100; display: flex; align-items: center; justify-content: space-between; gap: 24px; width: 100%; max-width: none; margin: 0; padding: 15px max(24px, calc((100% - 1280px) / 2)); border: 0; border-bottom: 1px solid #e9e2e3; background: #fcfaf7f5; color: #28252a; backdrop-filter: blur(16px); }
|
||||
.abx-header a, .abx-footer a { text-decoration: none; text-underline-offset: .24em; }
|
||||
.abx-header .abx-brand, .abx-footer .abx-brand { display: inline-flex; flex: 0 0 auto; align-items: center; gap: 11px; margin: 0; padding: 0; color: inherit; font-size: 19px; font-weight: 750; line-height: 1.2; letter-spacing: -.5px; white-space: nowrap; }
|
||||
.abx-brand .abx-logo { display: block; flex: 0 0 auto; width: 32px; height: 32px; margin: 0; }
|
||||
.abx-header .abx-subsite { transform: translateY(3px); margin-left: 2px; color: #716c74; font-size: 12px; font-weight: 450; letter-spacing: 0; }
|
||||
.abx-header .abx-nav { display: flex; align-items: center; justify-content: flex-end; flex-wrap: wrap; gap: 0; width: auto; max-width: none; margin: 0; padding: 0; font-size: 13px; font-weight: 600; }
|
||||
.abx-nav > a, .abx-nav > details { margin: 0 0 0 15px; padding: 0 0 0 15px; border: 0; border-left: 1px solid #ded4d8; border-radius: 0; background: none; }
|
||||
.abx-nav > :first-child { margin-left: 0; padding-left: 0; border-left: 0; }
|
||||
.abx-header .abx-nav a { display: inline-flex; align-items: center; min-height: 36px; color: #4d424b; white-space: nowrap; }
|
||||
.abx-header .abx-nav a:hover { color: #9b2854; }
|
||||
.abx-header .abx-nav .abx-cta { margin-left: 18px; padding: 9px 17px; border: 1px solid #9b2854; border-radius: 24px; background: #9b2854; color: #fff; }
|
||||
.abx-header .abx-nav .abx-cta:hover { background: #782040; color: #fff; }
|
||||
.abx-header .abx-apps { position: relative; }
|
||||
.abx-apps > summary { display: flex; align-items: center; gap: 7px; min-height: 36px; list-style: none; cursor: pointer; color: #4d424b; }
|
||||
.abx-apps > summary::-webkit-details-marker { display: none; }
|
||||
.abx-apps > summary::before { content: '›'; display: inline-block; color: #9b2854; font-size: 22px; line-height: 1; transition: transform .15s; }
|
||||
.abx-apps[open] > summary::before { transform: rotate(90deg); }
|
||||
.abx-header .abx-app-links { position: absolute; right: 0; top: calc(100% + 12px); width: 440px; max-width: calc(100vw - 24px); max-height: min(70vh, 580px); overflow-y: auto; padding: 8px; border: 1px solid #e9e2e3; border-radius: 16px; background: #fcfaf7; box-shadow: 0 18px 48px #2d152522, 0 3px 8px #2d152508; }
|
||||
.abx-header .abx-app-row { display: flex; align-items: center; gap: 8px; min-height: 44px; padding: 4px 6px; border-radius: 10px; transition: background .15s; }
|
||||
.abx-header .abx-app-row:hover, .abx-header .abx-app-row:focus-within { background: #f2e8ed; }
|
||||
.abx-header .abx-app-links .abx-app-name { flex: 1; min-width: 0; min-height: 34px; padding: 5px 4px; border-radius: 6px; font-size: 13px; line-height: 1.35; white-space: normal; }
|
||||
.abx-header .abx-platforms { display: flex; flex: 0 0 auto; align-items: center; gap: 5px; }
|
||||
.abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 auto; height: 30px; min-height: 30px; padding: 5px; border: 1px solid #e8dfe3; border-radius: 9px; background: #fffdfb; color: #4d424b; box-shadow: 0 2px 3px #2d152510, inset 0 1px 0 #fff; transition: transform .15s, box-shadow .15s, border-color .15s; }
|
||||
.abx-header .abx-app-links .abx-platform { width: 30px; }
|
||||
.abx-header .abx-platform svg { display: block; width: 18px; height: 18px; margin: 0; fill: currentColor; }
|
||||
.abx-header .abx-app-links .abx-platform-docker { color: #2386ce; }
|
||||
.abx-header .abx-app-links .abx-platform-windows, .abx-header .abx-app-links .abx-platform-app-store, .abx-header .abx-app-links .abx-platform-safari { color: #1879c4; }
|
||||
.abx-header .abx-app-links .abx-platform-android { color: #29965d; }
|
||||
.abx-header .abx-app-links .abx-platform-firefox { color: #e46b24; }
|
||||
.abx-header .abx-app-links .abx-platform-brave { color: #e8552b; }
|
||||
.abx-header .abx-app-links .abx-platform-edge { color: #168b95; }
|
||||
.abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-platform:focus-visible, .abx-header .abx-app-links .abx-tool:hover, .abx-header .abx-app-links .abx-tool:focus-visible { transform: translateY(-2px); border-color: #c99bb0; background: #fff; box-shadow: 0 5px 10px #2d15251c; }
|
||||
.abx-header .abx-app-links .abx-tool { padding: 5px 8px; font-size: 11px; white-space: nowrap; }
|
||||
.abx-header .abx-tools-row { margin-top: 5px; padding-top: 9px; border-top: 1px solid #e9e2e3; border-radius: 0 0 10px 10px; }
|
||||
.abx-header .abx-tools-row .abx-app-name { flex: 0 0 auto; }
|
||||
.abx-header .abx-tools-row .abx-platforms { flex: 0 1 auto; flex-wrap: wrap; justify-content: flex-end; margin-left: auto; }
|
||||
.abx-header .abx-app-links code { display: inline; vertical-align: baseline; font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 1em; font-weight: inherit; line-height: inherit; margin: 0; padding: 0; border: 0; background: transparent; color: inherit; }
|
||||
.abx-header .abx-icon-stack { isolation: isolate; gap: 0; padding: 4px 5px; }
|
||||
.abx-header .abx-icon-stack .abx-integration { position: relative; z-index: 1; transform: translateY(var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); transition: transform .18s ease, box-shadow .18s ease, border-color .18s ease; }
|
||||
.abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -10px; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(2) { --stack-angle: 9deg; --stack-y: -2px; z-index: 2; color: #569a31; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(3) { --stack-angle: -6deg; --stack-y: 2px; z-index: 3; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(4) { --stack-angle: 7deg; --stack-y: -1px; z-index: 4; color: #f00; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(5) { --stack-angle: -10deg; --stack-y: 1px; z-index: 5; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(6) { --stack-angle: 6deg; --stack-y: -2px; z-index: 6; }
|
||||
.abx-header .abx-icon-stack .abx-integration:nth-child(7) { --stack-angle: -5deg; z-index: 7; }
|
||||
.abx-header .abx-icon-stack .abx-integration:not(:hover, :focus-visible):has(~ .abx-integration:is(:hover, :focus-visible)) { transform: translate(-6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
|
||||
.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) ~ .abx-integration:not(:hover, :focus-visible) { transform: translate(6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
|
||||
.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { z-index: 20; transform: translateY(-7px) rotate(0deg) scale(1.12); border-color: #c99bb0; box-shadow: 0 8px 14px #2d15252b; }
|
||||
@media (hover: none) { .abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -6px; } }
|
||||
@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration { transition: none; } }
|
||||
.abx-footer { position: relative; z-index: 1; display: block; width: 100%; max-width: none; margin: 72px 0 0; padding: 60px max(24px, calc((100% - 1280px) / 2)) 24px; border: 0; border-top: 3px solid #7e284a; background: radial-gradient(ellipse at 12% 0, #54243b55, transparent 48%), linear-gradient(120deg, #201923, #14151b 75%); color: #ddd2dd; color-scheme: dark; }
|
||||
.abx-footer .abx-footer-intro { display: flex; align-items: flex-start; justify-content: space-between; gap: 32px; margin: 0 0 42px; }
|
||||
.abx-footer .abx-brand { color: #fff4ed; font-size: 24px; }
|
||||
.abx-footer .abx-brand .abx-logo { width: 40px; height: 40px; }
|
||||
.abx-footer .abx-tagline { max-width: 430px; margin: 12px 0 0; color: #b8acbb; font-size: 14px; line-height: 1.7; }
|
||||
.abx-footer .abx-footer-motto { margin: 2px 0 0; color: #d8b39f; font: italic 25px/1.45 Georgia, serif; text-align: right; }
|
||||
.abx-footer .abx-footer-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 42px; margin: 0; }
|
||||
.abx-footer .abx-footer-column { margin: 0; padding: 20px 0 0; border: 0; border-top: 1px solid #67505e66; background: none; }
|
||||
.abx-footer .abx-footer-column h2 { margin: 0 0 19px; padding: 0; border: 0; color: #e4acbb; font: 650 12px/1.4 var(--abx-font-sans); letter-spacing: .12em; text-transform: uppercase; }
|
||||
.abx-footer .abx-footer-column ul { display: block; list-style: none; margin: 0; padding: 0; }
|
||||
.abx-footer .abx-footer-column li { display: flex; align-items: center; gap: 10px; margin: 0; padding: 0; min-height: 36px; }
|
||||
.abx-footer .abx-footer-column li::before { content: none; }
|
||||
.abx-footer .abx-footer-column a { color: #ddd2dd; font-size: 13px; line-height: 1.55; }
|
||||
.abx-footer .abx-footer-column a:first-child { padding: 6px 0; }
|
||||
.abx-footer a:hover { color: #fff; text-decoration: underline; }
|
||||
.abx-footer .abx-source { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 28px; width: 28px; height: 28px; border: 1px solid transparent; border-radius: 50%; color: #a58d9f; }
|
||||
.abx-footer .abx-source:hover { color: #f3b5cb; border-color: #80516a; background: #ffffff08; }
|
||||
.abx-source svg { width: 15px; height: 15px; fill: currentColor; }
|
||||
.abx-footer .abx-footer-bottom { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 12px 24px; margin-top: 32px; padding-top: 21px; border-top: 1px solid #67505e66; color: #a99bab; font-size: 12px; }
|
||||
.abx-footer .abx-footer-bottom p { margin: 0; font-size: inherit; color: inherit; }
|
||||
.abx-footer .abx-footer-bottom a { color: #cebbc9; }
|
||||
.abx-footer .abx-footer-local { margin: 24px 0 0; padding: 16px 0 0; border: 0; border-top: 1px solid #67505e66; border-radius: 0; background: none; color: #baacbc; font-size: 12px; }
|
||||
.abx-footer-local > summary { cursor: pointer; color: #d6bfce; }
|
||||
.abx-footer-local > div { padding-top: 18px; }
|
||||
.abx-footer-local a { color: #e5aabd; }
|
||||
.abx-footer-local .brand, .abx-footer-local .footer-intro, .abx-footer-local .footer-grid, .abx-footer-local .footer-meta { color: inherit; }
|
||||
.abx-footer-local .footer-card { background: transparent; border-color: #67505e66; }
|
||||
.abx-footer-local h2, .abx-footer-local h3, .abx-footer-local strong { color: #e4d8e3; }
|
||||
.abx-footer-local .footer-link-copy span, .abx-footer-local .footer-card-subtitle { color: #baacbc; }
|
||||
.abx-header a:focus-visible, .abx-header summary:focus-visible, .abx-footer a:focus-visible, .abx-footer summary:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
|
||||
@media (max-width: 1100px) {
|
||||
.abx-header { flex-wrap: wrap; gap: 8px 24px; padding: 12px 20px; }
|
||||
.abx-header .abx-nav { justify-content: flex-start; }
|
||||
.abx-header .abx-app-links { position: fixed; top: auto; left: 20px; right: auto; width: 440px; margin-top: 10px; }
|
||||
}
|
||||
@media (max-width: 700px) {
|
||||
.abx-header { position: relative; padding: 14px 18px; }
|
||||
.abx-header .abx-brand { flex-wrap: wrap; row-gap: 5px; }
|
||||
.abx-header .abx-nav { width: 100%; gap: 4px 0; font-size: 12px; }
|
||||
.abx-nav > a, .abx-nav > details { margin-left: 10px; padding-left: 10px; }
|
||||
.abx-header .abx-nav .abx-cta { margin-left: 10px; padding: 7px 13px; }
|
||||
.abx-header .abx-app-links { left: 12px; right: 12px; width: auto; max-height: 60vh; }
|
||||
.abx-footer { padding: 40px 22px 22px; margin-top: 48px; }
|
||||
.abx-footer .abx-footer-intro { display: block; margin-bottom: 30px; }
|
||||
.abx-footer .abx-footer-motto { margin-top: 20px; text-align: left; font-size: 22px; }
|
||||
.abx-footer .abx-footer-grid { grid-template-columns: 1fr; gap: 26px; }
|
||||
.abx-footer .abx-footer-column li { min-height: 40px; }
|
||||
}
|
||||
@media (prefers-reduced-motion: reduce) { .abx-apps > summary::before, .abx-header .abx-app-row, .abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { transition: none; } .abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-tool:hover { transform: none; } }
|
||||
|
||||
.abx-footer-local > div > a + a { margin-left: 12px; }
|
||||
|
||||
@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { transform: none; } }
|
||||
@@ -1,28 +0,0 @@
|
||||
/*
|
||||
Shared ArchiveBox website navigation styles and behavior.
|
||||
Keep this file in sync with .github/pages/base/chrome.js in:
|
||||
ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
|
||||
electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
|
||||
Each site builds its committed copy independently; there is no central generator.
|
||||
Keep site-specific navigation, CTAs, and extra footer links outside this base.
|
||||
Also review the inline navbar/footer in abx-dl/website/index.html and
|
||||
debian-archivebox/website/index.html when shared links or branding change.
|
||||
*/
|
||||
(() => {
|
||||
const menu = document.querySelector('.abx-apps');
|
||||
document.addEventListener('keydown', event => {
|
||||
if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
|
||||
});
|
||||
document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
|
||||
function revealAnchor() {
|
||||
let target;
|
||||
try { target = document.getElementById(decodeURIComponent(window.location.hash.slice(1))); } catch { return; }
|
||||
if (!target) return;
|
||||
let details = target.closest('details');
|
||||
let opened = false;
|
||||
while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
|
||||
if (opened) window.requestAnimationFrame(() => target.scrollIntoView());
|
||||
}
|
||||
window.addEventListener('hashchange', revealAnchor);
|
||||
revealAnchor();
|
||||
})();
|
||||
@@ -1,12 +0,0 @@
|
||||
<!--
|
||||
Shared ArchiveBox website footer.
|
||||
Keep this file in sync with .github/pages/base/footer-end.html in:
|
||||
ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
|
||||
electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
|
||||
Each site builds its committed copy independently; there is no central generator.
|
||||
Keep site-specific navigation, CTAs, and extra footer links outside this base.
|
||||
Also review the inline navbar/footer in abx-dl/website/index.html and
|
||||
debian-archivebox/website/index.html when shared links or branding change.
|
||||
-->
|
||||
<div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security & privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
|
||||
</footer>
|
||||
@@ -1,13 +0,0 @@
|
||||
<!--
|
||||
Shared ArchiveBox website footer.
|
||||
Keep this file in sync with .github/pages/base/footer-start.html in:
|
||||
ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
|
||||
electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
|
||||
Each site builds its committed copy independently; there is no central generator.
|
||||
Keep site-specific navigation, CTAs, and extra footer links outside this base.
|
||||
Also review the inline navbar/footer in abx-dl/website/index.html and
|
||||
debian-archivebox/website/index.html when shared links or branding change.
|
||||
-->
|
||||
<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
|
||||
<div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
|
||||
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps & tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS & iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS & iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated
|
||||
@@ -1,24 +0,0 @@
|
||||
<!--
|
||||
Shared ArchiveBox website navbar and Apps menu.
|
||||
Keep this file in sync with .github/pages/base/header.html in:
|
||||
ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
|
||||
electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
|
||||
Each site builds its committed copy independently; there is no central generator.
|
||||
Keep site-specific navigation, CTAs, and extra footer links outside this base.
|
||||
Also review the inline navbar/footer in abx-dl/website/index.html and
|
||||
debian-archivebox/website/index.html when shared links or branding change.
|
||||
-->
|
||||
<header class="abx-header">
|
||||
<a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">__SITE_NAME__</span></a>
|
||||
<nav class="abx-nav" aria-label="Main navigation">
|
||||
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links">
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://archivebox.io/#install-compose">ArchiveBox Server</a><div class="abx-platforms"><a class="abx-platform abx-platform-docker" href="https://archivebox.io/#install-compose" aria-label="ArchiveBox Server for Docker" title="ArchiveBox Server for Docker"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M349.9 236.3l-66.1 0 0-59.4 66.1 0 0 59.4zm0-204.3l-66.1 0 0 60.7 66.1 0 0-60.7zm78.2 144.8l-66.1 0 0 59.4 66.1 0 0-59.4zM271.8 104.7l-66.1 0 0 60.1 66.1 0 0-60.1zm78.1 0l-66.1 0 0 60.1 66.1 0 0-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7l-434.7 0c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zM115.6 176.8l-66 0 0 59.4 66.1 0 0-59.4-.1 0zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm-78.1-72.1l-66.1 0 0 60.1 66.1 0 0-60.1z"/></svg></a><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/#archivebox-serverapp" aria-label="ArchiveBox Server for macOS" title="ArchiveBox Server for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://electron.archivebox.io/" aria-label="ArchiveBox Server for Windows" title="ArchiveBox Server for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://archivebox.io/#install-apt" aria-label="ArchiveBox Server for Linux" title="ArchiveBox Server for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-1Line truncated
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Desktop</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Desktop for macOS" title="ArchiveBox Desktop for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://windows.archivebox.io/" aria-label="ArchiveBox Desktop for Windows" title="ArchiveBox Desktop for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://linux.archivebox.io/" aria-label="ArchiveBox Desktop for Linux" title="ArchiveBox Desktop for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-10 14.1-10.5 4.7-4.4 13.5-14.2 27.9-14.2 7.1 0 15.6 2.3 25.9 8.9 6.3 4.1 11.3 4.4 22.6 9.3 8.4 3.5 13.7 9.7 10.5 18.2-2.6 7.1-11 14.4-22.7 18.1-11.1 3.6-19.8 16-38.2 14.9-3.9-.2-7-1-9.6-2.1-8-3.5-12.2-10.4-20-15-8.6-4.8-13.2-10.4-14.7-15.3-1.4-4.9 0-9 4.2-12.3zm3.3 334c-2.7 35.1-43.9 34.4-75.3 18-29.9-15.8-68.6-6.5-76.5-21.9-2.4-4.7-2.4-12.7 2.6-26.4l0-.2c2.4-7.6 .6-16-.6-23.9-1.2-7.8-1.8-15 .9-20 3.5-6.7 8.5-9.1 14.8-11.3 10.3-3.7 11.8-3.4 19.6-9.9 5.5-5.7 9.5-12.9 14.3-18 5.1-5.5 10-8.1 17.7-6.9 8.1 1.2 15.1 6.8 21.9 16l19.6 35.6c9.5 19.9 43.1 48.4 41 68.9zm-1.4-25.9c-4.1-6.6-9.6-13.6-14.4-19.6 7.1 0 14.2-2.2 16.7-8.9 2.3-6.2 0-14.9-7.4-24.9-13.5-18.2-38.3-32.5-38.3-32.5-13.5-8.4-21.1-18.7-24.6-29.9s-3-23.3-.3-35.2c5.2-22.9 18.6-45.2 27.2-59.2 2.3-1.7 .8 3.2-8.7 20.8-8.5 16.1-24.4 53.3-2.6 82.4 .6-20.7 5.5-41.8 13.8-61.5 12-27.4 37.3-74.9 39.3-112.7 1.1 .8 4.6 3.2 6.2 4.1 4.6 2.7 8.1 6.7 12.6 10.3 12.4 10 28.5 9.2 42.4 1.2 6.2-3.5 11.2-7.5 15.9-9 9.9-3.1 17.8-8.6 22.3-15 7.7 30.4 25.7 74.3 37.2 95.7 6.1 11.4 18.3 35.5 23.6 64.6 3.3-.1 7 .4 10.9 1.4 13.8-35.7-11.7-74.2-23.3-84.9-4.7-4.6-4.9-6.6-2.6-6.5 12.6 11.2 29.2 33.7 35.2 59 2.8 11.6 3.3 23.7 .4 35.7 16.4 6.8 35.9 17.9 30.7 34.8-2.2-.1-3.2 0-4Line truncated
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Mobile</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Mobile for iOS" title="ArchiveBox Mobile for iOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-android" href="https://android.archivebox.io/" aria-label="ArchiveBox Mobile for Android" title="ArchiveBox Mobile for Android"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!-- Font Awesome Free 5.0.13 by Fonticons, Inc. https://fontawesome.com License: https://creativecommons.org/licenses/by/4.0/ --><path d="M89.6 204.5v115.8c0 15.4-12.1 27.7-27.5 27.7-15.3 0-30.1-12.4-30.1-27.7V204.5c0-15.1 14.8-27.5 30.1-27.5 15.1 0 27.5 12.4 27.5 27.5zm10.8 157c0 16.4 13.2 29.6 29.6 29.6h19.9l.3 61.1c0 36.9 55.2 36.6 55.2 0v-61.1h37.2v61.1c0 36.7 55.5 36.8 55.5 0v-61.1h20.2c16.2 0 29.4-13.2 29.4-29.6V182.1H100.4v179.4zm248-189.1H99.3c0-42.8 25.6-80 63.6-99.4l-19.1-35.3c-2.8-4.9 4.3-8 6.7-3.8l19.4 35.6c34.9-15.5 75-14.7 108.3 0L297.5 34c2.5-4.3 9.5-1.1 6.7 3.8L285.1 73c37.7 19.4 63.3 56.6 63.3 99.4zm-170.7-55.5c0-5.7-4.6-10.5-10.5-10.5-5.7 0-10.2 4.8-10.2 10.5s4.6 10.5 10.2 10.5c5.9 0 10.5-4.8 10.5-10.5zm113.4 0c0-5.7-4.6-10.5-10.2-10.5-5.9 0-10.5 4.8-10.5 10.5s4.6 10.5 10.5 10.5c5.6 0 10.2-4.8 10.2-10.5zm94.8 60.1c-15.1 0-27.5 12.1-27.5 27.5v115.8c0 15.4 12.4 27.7 27.5 27.7 15.4 0 30.1-12.4 30.1-27.7V204.5c0-15.4-14.8-27.5-30.1-27.5z"/></svg></a></div></div>
|
||||
<div class="abx-app-row"><a class="abx-app-name" href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><div class="abx-platforms"><a class="abx-platform abx-platform-chrome" href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj" aria-label="ArchiveBox Browser Extension for Chrome" title="ArchiveBox Browser Extension for Chrome"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2024 Fonticons, Inc. --><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg></a><a class="abx-platform abx-platform-safari" href="https://extension.archivebox.io/" aria-label="ArchiveBox Browser Extension for Safari" title="ArchiveBox Browser Extension for Safari"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M274.7 274.7L237.3 237.3 166 346 274.7 274.7zM256 8a248 248 0 1 0 0 496 248 248 0 1 0 0-496zM411.8 182.8l14.8-6.1c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.8 1 2.6 1.7 1.3 1.6 1.7 2.6 .6 2 .6 3.1-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.3-2.6 1.7L418 197.6c-2 .8-4.2 .8-6.1 0s-3.5-2.4-4.3-4.3-.8-4.2 0-6.1 2.4-3.5 4.3-4.3zM314.4 94l6.1-14.8c.8-2 2.4-3.5 4.3-4.3s4.2-.8 6.1 0 3.5 2.4 4.3 4.3 .8 4.2 0 6.1l-6.1 14.8c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0c-1-.4-1.9-1-2.6-1.7s-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1zM256 60c2.1 0 4.2 .8 5.7 2.3S264 65.9 264 68l0 16c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7S253.9 60 256 60zM181 74.9c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.9 1 2.6 1.7 1.3 1.6 1.7 2.6L197.6 94c.4 1 .7 2 .7 3.1s-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.4-2.6 1.8-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.8-1.3-1.7-1.7-2.7l-6.1-14.8c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3zm-63.6 42.5c1.5-1.5 3.5-2.3 5.7-2.3s4.2 .8 5.7 2.3L140 128.7c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3l-11.3-11.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6l0 0zM60 256c0-2.1 .8-4.2 2.3-5.7S65.9 248 68 248l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3l-16 0c-2.1 0-4.2-.8-5.7-2.3S60 258.1 60 256zm40.2 73.2l-14.8 6.1c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1 1-1.9 1.7-2.6 1.6-1.3 2.6-1.7L94 314.4c2-.8 4.2-.8 6.1 0s3.5 2.4 4.3 4.3 .8 4.2 0 6.1-2.4 3.5-4.3 4.3zm4.3-136c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1 2.4-3.5 4.3-4.3 4.2-.8 6.1 0l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1l0 0zM197.6 418l-6.1 14.8c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1l6.1-14.8c.4-1 1-1.9 1.7-2.7s1.6-1.4 2.6-1.8 2-.6 3.1-.6 2.1 .2 3.1 .6 1.9 1 2.6 1.8 1.3 1.7 1.7 2.6 .6 2 .6 3.1-.2 2.1-.7 3.1zM264 444c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7s3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3 2.3 3.5 2.3 5.7l0 16zm67-6.9c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6L314.4 418c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3 4.2-.8 6.1 0 3.5 2.4 4.3 4.3l6.1 14.8c.8 2 .8 4.2 0 6.1s-2.4 3.5-4.3 4.3zm63.6-42.5c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3L372 383.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6 3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3l11.3 11.3c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6l0 0zM286.2 286.2L110.3 401.7 225.8 225.8 401.7 110.3 286.2 286.2zM437.1 331c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1c.4-1 1-1.9 1.7-2.6s1.6-1.3 2.6-1.7 2-.6 3.1-.6 2.1 .2 3.1 .6l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1zm6.9-67l-16 0c-2.1 0-4.2-.8-5.7-2.3s-2.3-3.5-2.3-5.7 .8-4.2 2.3-5.7 3.5-2.3 5.7-2.3l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3z"/></svg></a><a class="abx-platform abx-platform-firefox" href="https://addons.mozilla.org/firefox/addon/archivebox-exporter/" aria-label="ArchiveBox Browser Extension for Firefox" title="ArchiveBox Browser Extension for Firefox"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512Line truncated
|
||||
<div class="abx-app-row abx-plugin-row"><a class="abx-app-name" href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><div class="abx-platforms abx-icon-stack"><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Google Drive storage" title="Google Drive storage"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m38.7 419.3 22.6 39c4.7 8.2 11.4 14.7 19.4 19.4l80.6-139.6H0c0 9.1 2.3 18.2 7 26.4z" style="fill:#0066da"/><path d="M256 173.9 175.4 34.3c-7.9 4.7-14.7 11.1-19.4 19.4L7 311.7c-4.6 8-7 17.1-7 26.4h161.3z" style="fill:#00ac47"/><path d="M431.4 477.7c7.9-4.7 14.7-11.1 19.4-19.4l9.4-16.1 44.9-77.7c4.7-8.2 7-17.3 7-26.4H350.7l34.3 67.4z" style="fill:#ea4335"/><path d="m256 173.9 80.6-139.6c-7.9-4.7-17-7-26.4-7H201.8c-9.4 0-18.5 2.6-26.4 7z" style="fill:#00832d"/><path d="M350.7 338.1H161.3L80.6 477.7c7.9 4.7 17 7 26.4 7h298c9.4 0 18.5-2.6 26.4-7z" style="fill:#2684fc"/><path d="M430.5 182.7 356 53.7c-4.7-8.2-11.4-14.7-19.4-19.4L256 173.9l94.7 164.2h161c0-9.1-2.3-18.2-7-26.4z" style="fill:#ffba00"/></svg></a><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Amazon S3 storage" title="Amazon S3 storage"><svg aria-hidden="true" focusable="false" fill="#569A31" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M20.913 13.147l.12-.895c.947.576 1.258.922 1.354 1.071-.16.031-.562.046-1.474-.176zm-2.174 7.988a.547.547 0 0 0-.005.073c0 .084-.207.405-1.124.768a10.28 10.28 0 0 1-1.438.432c-1.405.325-3.128.504-4.853.504-4.612 0-7.412-1.184-7.412-1.704a.547.547 0 0 0-.005-.073L1.81 5.602c.135.078.28.154.432.227.042.02.086.038.128.057.134.062.272.122.417.18l.179.069c.154.058.314.114.478.168.043.013.084.029.13.043.207.065.423.127.646.187l.176.044c.175.044.353.087.534.127a23.414 23.414 0 0 0 .843.17l.121.023c.252.045.508.085.768.122.071.011.144.02.216.03.2.027.4.053.604.077l.24.027c.245.026.49.05.74.07l.081.009c.275.022.552.04.83.056l.233.012c.21.01.422.018.633.025a33.088 33.088 0 0 0 2.795-.026l.232-.011c.278-.016.555-.034.83-.056l.08-.008c.25-.02.497-.045.742-.072l.238-.026c.205-.024.408-.05.609-.077.07-.01.141-.019.211-.03.261-.037.519-.078.772-.122l.111-.02c.215-.04.427-.082.634-.125l.212-.047c.186-.041.368-.085.546-.13l.166-.042c.225-.06.444-.122.654-.189.04-.012.077-.026.115-.038a10.6 10.6 0 0 0 .493-.173c.058-.021.114-.044.17-.066.15-.06.293-.12.43-.185.038-.017.079-.034.116-.052.153-.073.3-.15.436-.228l-.976 7.245c-2.488-.78-5.805-2.292-7.311-3a1.09 1.09 0 0 0-1.088-1.085c-.6 0-1.088.489-1.088 1.088 0 .6.488 1.089 1.088 1.089.196 0 .378-.056.537-.148 1.72.812 5.144 2.367 7.715 3.15zm-7.42-20.047c5.677 0 9.676 1.759 9.75 2.736l-.014.113c-.01.033-.031.067-.048.101-.015.028-.026.057-.047.087-.024.033-.058.068-.09.102-.028.03-.051.06-.084.09-.038.035-.087.07-.133.105-.04.03-.074.06-.119.091-.053.036-.116.071-.177.107-.05.03-.095.06-.15.09-.068.036-.147.073-.222.11-.059.028-.114.057-.177.085-.084.038-.177.074-.268.111-.068.027-.13.054-.203.082-.097.036-.205.072-.31.107-.075.026-.148.053-.228.079-.111.035-.233.069-.35.103-.085.024-.165.05-.253.073-.124.034-.258.065-.389.098-.093.022-.181.046-.278.068-.139.032-.287.061-.433.091-.098.02-.191.041-.293.06-.155.03-.32.057-.482.084-.1.018-.198.036-.302.052-.166.026-.342.048-.515.072-.11.014-.213.03-.325.044-.181.023-.372.041-.56.06-.11.012-.218.025-.332.036-.188.016-.386.029-.58.043-.122.009-.24.02-.364.028-.207.012-.422.02-.635.028-.12.005-.234.012-.354.016a35.605 35.605 0 0 1-2.069 0c-.12-.004-.234-.011-.352-.016-.214-.008-.43-.016-.637-.028-.122-.008-.238-.02-.36-.027-.195-.015-.394-.028-.584-.044-.11-.01-.215-.024-.324-.035-.19-.02-.384-.038-.568-.06l-.315-.044c-.176-.024-.355-.046-.525-.073-.1-.015-.192-.033-.29-.05-.167-.028-.335-.055-.494-.086-.096-.018-.183-.038-.276-.056-.151-.032-.305-.062-.45-.095-.09-.02-.173-.043-.26-.064-.138-.034-.277-.067-.407-.102-.082-.022-.157-.046-.235-.069a11.75 11.75 0 0 1-.368-.108c-.075-.024-.141-.049-.213-.073-.11-.037-.223-.075-.325-.113-.067-.025-.125-.051-.188-.077-.096-.038-.195-.076-.282-.115-.06-.027-.11-.054-.166-.08-.08-.039-.162-.077-.233-.116-.052-.028-.094-.055-.142-.084-.063-.038-.13-.075-.185-.113-.043-.029-.075-.058-.113-.086-.048-.037-.098-.073-.139-.11-.032-.029-.054-.057-.08-.087-.033-.035-.069-.07-.093-.104-.02-.03-.031-.058-.046-.086-.018-.035-.039-.068-.049-.102l-.015-.113c.076-.977 4.074-2.736 9.748-2.736zm12.182 12.124c-.118-.628-.84-1.291-2.31-2.128l.963-7.16a.531.531 0 0 0 .005-.073C22.16 1.581 16.447 0 11.32 0 6.194 0 .482 1.581.482 3.851a.58.58 0 0 0 .005.072L2.819 21.25c.071 2.002 5.236 2.75 8.5 2.75 1.805 0 3.615-.188 5.098-.531.598-.138 1.133-.3 1.592-.48 1.18-.467 1.789-1.053 1.813-1.739l.945-7.018c.557.131 1.016.197 1.389.197.54 0 .902-.137 1.134-.413aLine truncated
|
||||
<div class="abx-app-row abx-tools-row"><a class="abx-app-name" href="https://abx-dl.archivebox.io/">Tools</a><div class="abx-platforms"><a class="abx-tool" href="https://abx-dl.archivebox.io/"><span>One-shot <code>abx-dl</code> CLI</span></a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/mcp">MCP</a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage">API</a><a class="abx-tool" href="https://web.archivebox.io">Demo</a></div></div>
|
||||
</div></details>
|
||||
__SITE_NAV__
|
||||
</nav>
|
||||
</header>
|
||||
@@ -1,11 +0,0 @@
|
||||
/* Identical standalone gallery strip across ArchiveBox product sites. */
|
||||
/* The root query container measures the usable width, excluding scrollbars. */
|
||||
html { container-type: inline-size; }
|
||||
.abx-marquee { min-width: 0; width: 100cqw; max-width: none; margin: 24px calc(50% - 50cqw) 40px; padding: 0; overflow: hidden; }
|
||||
.abx-marquee .abx-marquee-viewport { width: 100%; overflow-x: auto; scroll-behavior: auto; overscroll-behavior-x: contain; scrollbar-width: thin; padding: 16px 24px 28px; box-sizing: border-box; }
|
||||
.abx-marquee .abx-marquee-track { display: flex; align-items: flex-start; gap: 24px; width: max-content; margin-inline: auto; }
|
||||
.abx-marquee .abx-marquee-card { display: block; flex: none; width: min-content; overflow: hidden; border-radius: 10px; background: #fff; box-shadow: 0 0 0 1px rgb(0 0 0 / 12%), 0 2px 4px rgb(0 0 0 / 10%), 0 8px 16px rgb(0 0 0 / 16%); text-decoration: none; color: #533344; }
|
||||
.abx-marquee .abx-marquee-card img { display: block; height: clamp(240px, 30vw, 400px); width: auto; max-width: none; max-height: none; object-fit: contain; box-shadow: none; border: 0; border-radius: 0; background: transparent; padding: 0; margin: 0; }
|
||||
.abx-marquee :focus-visible { outline: 2px solid #9b2854; outline-offset: 3px; }
|
||||
.abx-marquee .abx-marquee-viewport:focus-visible { outline-offset: -3px; }
|
||||
@media (max-width: 600px) { .abx-marquee .abx-marquee-track { gap: 16px; } .abx-marquee .abx-marquee-card img { height: 240px; } }
|
||||
@@ -1,44 +0,0 @@
|
||||
/* Scroll real gallery links once; keep them usable without JavaScript. */
|
||||
(() => {
|
||||
document.querySelectorAll('.abx-marquee').forEach(gallery => {
|
||||
const viewport = gallery.querySelector('.abx-marquee-viewport');
|
||||
const reducedMotion = window.matchMedia('(prefers-reduced-motion: reduce)');
|
||||
let paused = reducedMotion.matches;
|
||||
let hovering = false;
|
||||
let focused = false;
|
||||
let visible = false;
|
||||
let frame;
|
||||
let previous;
|
||||
let offset = viewport.scrollLeft;
|
||||
function tick(time) {
|
||||
const distance = viewport.scrollWidth - viewport.clientWidth;
|
||||
if (distance <= 0) return;
|
||||
if (previous !== undefined) {
|
||||
offset = Math.min(offset + Math.min(time - previous, 50) * 0.035, distance);
|
||||
viewport.scrollTo({ left: offset, behavior: 'instant' });
|
||||
}
|
||||
if (offset >= distance - 1) { paused = true; return; }
|
||||
previous = time;
|
||||
frame = window.requestAnimationFrame(tick);
|
||||
}
|
||||
function sync() {
|
||||
window.cancelAnimationFrame(frame);
|
||||
previous = undefined;
|
||||
offset = viewport.scrollLeft;
|
||||
if (!paused && !hovering && !focused && visible && !document.hidden) frame = window.requestAnimationFrame(tick);
|
||||
}
|
||||
viewport.addEventListener('pointerenter', event => { if (event.pointerType === 'mouse') { hovering = true; sync(); } });
|
||||
viewport.addEventListener('pointerleave', () => { hovering = false; sync(); });
|
||||
viewport.addEventListener('focusin', () => { focused = true; sync(); });
|
||||
viewport.addEventListener('focusout', event => { focused = viewport.contains(event.relatedTarget); sync(); });
|
||||
function pauseForInteraction() { paused = true; sync(); }
|
||||
viewport.addEventListener('pointerdown', pauseForInteraction);
|
||||
viewport.addEventListener('wheel', pauseForInteraction, { passive: true });
|
||||
viewport.addEventListener('keydown', pauseForInteraction);
|
||||
reducedMotion.addEventListener('change', () => { paused = reducedMotion.matches; sync(); });
|
||||
document.addEventListener('visibilitychange', sync);
|
||||
new window.IntersectionObserver(([entry]) => { visible = entry.isIntersecting; sync(); }).observe(gallery);
|
||||
new window.ResizeObserver(sync).observe(viewport);
|
||||
viewport.querySelectorAll('img').forEach(image => image.addEventListener('load', sync));
|
||||
});
|
||||
})();
|
||||
@@ -1,96 +0,0 @@
|
||||
"""Render homepage screenshot strips from the site's existing gallery HTML."""
|
||||
|
||||
import re
|
||||
from html import escape
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import urljoin
|
||||
|
||||
|
||||
class Gallery(HTMLParser):
|
||||
"""Only gallery figures are captures; navigation and decorative images are not."""
|
||||
|
||||
def __init__(self, base):
|
||||
super().__init__()
|
||||
self.base = base
|
||||
self.stack = []
|
||||
self.captures = []
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = dict(attrs)
|
||||
if tag == "img" and any(name == "figure" for name, _ in self.stack):
|
||||
context = [item for _, item in self.stack]
|
||||
profile = next(
|
||||
(
|
||||
item.get("data-profile") or item.get("data-viewport")
|
||||
for item in reversed(context)
|
||||
if item.get("data-profile") or item.get("data-viewport")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if profile not in (None, "desktop"):
|
||||
return
|
||||
source = attrs.get("src", "")
|
||||
if not source or not attrs.get("alt"):
|
||||
return
|
||||
anchor = next((item["id"] for name, item in reversed(self.stack) if name in ("article", "figure") and item.get("id")), "")
|
||||
server = any(item.get("data-platform") == "server" for item in context)
|
||||
self.captures.append(
|
||||
{
|
||||
"src": urljoin(self.base, source),
|
||||
"href": self.base + "#" + anchor if anchor else urljoin(self.base, source),
|
||||
"title": attrs["alt"],
|
||||
"width": attrs.get("width", ""),
|
||||
"height": attrs.get("height", ""),
|
||||
"product": "server" if server else "client",
|
||||
},
|
||||
)
|
||||
if tag not in {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"}:
|
||||
self.stack.append((tag, attrs))
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
for index in range(len(self.stack) - 1, -1, -1):
|
||||
if self.stack[index][0] == tag:
|
||||
del self.stack[index:]
|
||||
break
|
||||
|
||||
|
||||
def render(output, baseurl="", featured=()):
|
||||
homepage = output / "index.html"
|
||||
text = homepage.read_text()
|
||||
slots = re.findall(r"<!-- ARCHIVEBOX:(MARQUEE(?:-CLIENT|-SERVER)?) -->", text)
|
||||
if not slots:
|
||||
return False
|
||||
base = "/" + baseurl.strip("/") + "/" if baseurl.strip("/") else "/"
|
||||
gallery = Gallery(base + "screenshots/")
|
||||
gallery.feed((output / "screenshots/index.html").read_text())
|
||||
for slot in slots:
|
||||
product = slot.removeprefix("MARQUEE-").lower() if "-" in slot else None
|
||||
captures = [capture for capture in gallery.captures if product is None or capture["product"] == product]
|
||||
priority = {title: index for index, title in enumerate(featured)}
|
||||
captures.sort(key=lambda capture: priority.get(capture["title"], len(priority)))
|
||||
cards = []
|
||||
for capture in captures:
|
||||
dimensions = "".join(f' {key}="{escape(capture[key])}"' for key in ("width", "height") if capture[key])
|
||||
cards.append(
|
||||
f'<a class="abx-marquee-card" href="{escape(capture["href"])}"><img src="{escape(capture["src"])}" alt="{escape(capture["title"])}"{dimensions} loading="lazy" decoding="async"></a>',
|
||||
)
|
||||
viewport_id = slot.lower()
|
||||
content = (
|
||||
f'<section class="abx-marquee" aria-label="App screenshots"><div class="abx-marquee-viewport" id="{viewport_id}" tabindex="0" aria-label="Screenshots. Scroll sideways to explore."><div class="abx-marquee-track">{"".join(cards)}</div></div></section>'
|
||||
if cards
|
||||
else ""
|
||||
)
|
||||
start, end = f"<!-- ARCHIVEBOX:{slot} -->", f"<!-- /ARCHIVEBOX:{slot} -->"
|
||||
replacement = start + "\n" + content + "\n" + end
|
||||
if end in text:
|
||||
text = re.sub(
|
||||
re.escape(start) + r".*?" + re.escape(end),
|
||||
lambda _, replacement=replacement: replacement,
|
||||
text,
|
||||
count=1,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
else:
|
||||
text = text.replace(start, replacement, 1)
|
||||
homepage.write_text(text)
|
||||
return True
|
||||
@@ -1,4 +0,0 @@
|
||||
<!-- Repository-specific navigation and primary CTA. -->
|
||||
<a href="__SITE_BASE__#use-cases">Use cases</a><a href="__SITE_BASE__#configuration">Configuration</a><a href="__SITE_BASE__#layout-security">Layout & security</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
|
||||
<a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
|
||||
<a class="abx-cta" href="__SITE_BASE__#install">Get started</a>
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"name": "Web Archive",
|
||||
"url": "https://archivebox.io/",
|
||||
"pages": [
|
||||
"index.html",
|
||||
"screenshots/index.html"
|
||||
],
|
||||
"marquee_featured": [
|
||||
"Snapshots grid — desktop",
|
||||
"Snapshot View (header collapsed) — desktop",
|
||||
"AI agent — desktop",
|
||||
"Archive results — desktop",
|
||||
"Add URLs — desktop",
|
||||
"Snapshots table — desktop"
|
||||
]
|
||||
}
|
||||
@@ -1,215 +0,0 @@
|
||||
"""Assemble and verify this repository's static Pages site (standard library only)."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
from datetime import UTC, datetime
|
||||
from html import escape, unescape
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlsplit
|
||||
|
||||
from marquee import render as render_marquee
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
ROOT = HERE.parents[1]
|
||||
|
||||
|
||||
def replace_block(text, name, content):
|
||||
start, end = f"<!-- ARCHIVEBOX:{name} -->", f"<!-- /ARCHIVEBOX:{name} -->"
|
||||
pattern = re.escape(start) + r".*?" + re.escape(end)
|
||||
if re.search(pattern, text, re.DOTALL):
|
||||
return re.sub(
|
||||
pattern,
|
||||
lambda _: f"{start}\n{content}\n{end}",
|
||||
text,
|
||||
count=1,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
if text.count(start) != 1:
|
||||
raise ValueError(f"Expected one {name} placeholder")
|
||||
return text.replace(start, f"{start}\n{content}\n{end}", 1)
|
||||
|
||||
|
||||
def render_metadata(text):
|
||||
"""Keep product copy in native templates; render one consistent SEO block."""
|
||||
page = Page()
|
||||
page.feed(text)
|
||||
title = unescape(re.search(r"<title>(.*?)</title>", text, re.DOTALL)[1])
|
||||
meta = page.metadata
|
||||
description = meta.get("description", meta.get("og:description", ""))
|
||||
image = meta.get("og:image", meta.get("twitter:image", ""))
|
||||
if not description or not image or not page.canonical:
|
||||
raise ValueError("Each page needs a description, social image and canonical")
|
||||
values = {
|
||||
"description": description,
|
||||
"robots": "index,follow,max-image-preview:large",
|
||||
"theme-color": "#9b2854",
|
||||
"og:type": "website",
|
||||
"og:site_name": meta.get("og:site_name", "ArchiveBox"),
|
||||
"og:locale": meta.get("og:locale", "en_US"),
|
||||
"og:url": page.canonical,
|
||||
"og:title": title,
|
||||
"og:description": description,
|
||||
"og:image": image,
|
||||
"og:image:type": meta.get("og:image:type", "image/png"),
|
||||
"og:image:width": meta.get("og:image:width", "1200"),
|
||||
"og:image:height": meta.get("og:image:height", "630"),
|
||||
"og:image:alt": meta.get("og:image:alt", title),
|
||||
"twitter:card": "summary_large_image",
|
||||
"twitter:site": "@ArchiveBoxApp",
|
||||
"twitter:title": title,
|
||||
"twitter:description": description,
|
||||
"twitter:image": image,
|
||||
"twitter:image:alt": meta.get("og:image:alt", title),
|
||||
}
|
||||
|
||||
def strip_managed_tag(match):
|
||||
tag = Page()
|
||||
tag.feed(match[0])
|
||||
if tag.canonicals or any(key in values for key in tag.metadata):
|
||||
return ""
|
||||
# System fonts are shared across sites, without a network font dependency.
|
||||
if "fonts.googleapis.com" in match[0] or "fonts.gstatic.com" in match[0]:
|
||||
return ""
|
||||
return match[0]
|
||||
|
||||
text = re.sub(r"<(?:meta|link)\b[^>]*>", strip_managed_tag, text)
|
||||
text = re.sub(r"<title>.*?</title>", "", text, flags=re.DOTALL)
|
||||
tags = [
|
||||
f"<title>{escape(title)}</title>",
|
||||
f'<link rel="canonical" href="{escape(page.canonical, quote=True)}">',
|
||||
]
|
||||
for key, value in values.items():
|
||||
attr = "property" if key.startswith("og:") else "name"
|
||||
tags.append(f'<meta {attr}="{key}" content="{escape(value, quote=True)}">')
|
||||
if "<!-- ARCHIVEBOX:METADATA -->" not in text:
|
||||
text = text.replace("</head>", "<!-- ARCHIVEBOX:METADATA -->\n</head>", 1)
|
||||
return replace_block(text, "METADATA", "\n".join(tags))
|
||||
|
||||
|
||||
def render(output, source=None, baseurl=""):
|
||||
config = json.loads((HERE / "site.json").read_text())
|
||||
output = output.resolve()
|
||||
if source:
|
||||
source = source.resolve()
|
||||
if output == source or output in source.parents or source in output.parents:
|
||||
raise ValueError("Build output must be separate from the source directory")
|
||||
if output == ROOT or output in ROOT.parents:
|
||||
raise ValueError("Build output must not replace a repository")
|
||||
if output.exists():
|
||||
shutil.rmtree(output)
|
||||
shutil.copytree(source, output)
|
||||
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
|
||||
assets = output / "site-base"
|
||||
assets.mkdir(parents=True, exist_ok=True)
|
||||
has_marquee = render_marquee(output, baseurl, config.get("marquee_featured", []))
|
||||
common_assets = ["chrome.css", "chrome.js"] + (["marquee.css", "marquee.js"] if has_marquee else [])
|
||||
for name in common_assets:
|
||||
shutil.copyfile(HERE / "base" / name, assets / name)
|
||||
for name in config["pages"]:
|
||||
page = output / name
|
||||
text = render_metadata(page.read_text())
|
||||
base = "../" * (len(Path(name).parts) - 1) or "./"
|
||||
header = (HERE / "base/header.html").read_text().replace("__SITE_NAME__", escape(config["name"]))
|
||||
header = header.replace("__SITE_NAV__", (HERE / "nav.html").read_text().strip())
|
||||
text = replace_block(text, "HEADER", header)
|
||||
for kind in ["FOOTER-START", "FOOTER-END"]:
|
||||
text = replace_block(text, kind, (HERE / "base" / f"{kind.lower()}.html").read_text().strip())
|
||||
text = text.replace("__SITE_BASE__", base).replace("__SITE_REVISION__", revision)
|
||||
resources = f'<link rel="stylesheet" href="{base}site-base/chrome.css?v={revision}">\n<script src="{base}site-base/chrome.js?v={revision}" defer></script>'
|
||||
if has_marquee and name == "index.html":
|
||||
resources += f'\n<link rel="stylesheet" href="{base}site-base/marquee.css?v={revision}">\n<script src="{base}site-base/marquee.js?v={revision}" defer></script>'
|
||||
if "<!-- ARCHIVEBOX:ASSETS -->" not in text:
|
||||
text = text.replace("</head>", "<!-- ARCHIVEBOX:ASSETS -->\n</head>", 1)
|
||||
text = replace_block(text, "ASSETS", resources)
|
||||
page.write_text(text)
|
||||
# Site build provenance is independent of all original screenshot manifests.
|
||||
provenance_path = output / "build.json"
|
||||
provenance = json.loads(provenance_path.read_text()) if provenance_path.exists() else {}
|
||||
provenance.update(
|
||||
revision=revision,
|
||||
generatedAt=datetime.now(UTC).isoformat(),
|
||||
site=config["url"],
|
||||
)
|
||||
provenance_path.write_text(json.dumps(provenance, indent=2) + "\n")
|
||||
(output / ".nojekyll").touch()
|
||||
verify(output, baseurl)
|
||||
print(f"Assembled {len(config['pages'])} pages at {output} from {revision}")
|
||||
|
||||
|
||||
class Page(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.ids = set()
|
||||
self.metadata = {}
|
||||
self.canonical = ""
|
||||
self.resources = []
|
||||
self.headers = self.footers = self.canonicals = 0
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = dict(attrs)
|
||||
if tag == "meta":
|
||||
for key in ["name", "property"]:
|
||||
if attrs.get(key):
|
||||
self.metadata[attrs[key]] = attrs.get("content", "")
|
||||
if tag == "link" and attrs.get("rel") == "canonical":
|
||||
self.canonical = attrs["href"]
|
||||
if attrs.get("id"):
|
||||
self.ids.add(attrs["id"])
|
||||
classes = attrs.get("class", "").split()
|
||||
self.headers += "abx-header" in classes
|
||||
self.footers += "abx-footer" in classes
|
||||
self.canonicals += tag == "link" and attrs.get("rel") == "canonical"
|
||||
if tag in ["img", "script"] and attrs.get("src"):
|
||||
self.resources.append(attrs["src"])
|
||||
if tag == "link" and attrs.get("rel") in [
|
||||
"stylesheet",
|
||||
"icon",
|
||||
"apple-touch-icon",
|
||||
]:
|
||||
self.resources.append(attrs["href"])
|
||||
|
||||
|
||||
def verify(output, baseurl=""):
|
||||
config = json.loads((HERE / "site.json").read_text())
|
||||
output = output.resolve()
|
||||
prefix = "/" + baseurl.strip("/") if baseurl.strip("/") else ""
|
||||
for name in config["pages"]:
|
||||
path = output / name
|
||||
text = path.read_text()
|
||||
page = Page()
|
||||
page.feed(text)
|
||||
if (page.headers, page.footers, page.canonicals) != (1, 1, 1):
|
||||
raise ValueError(f"{name}: expected exactly one shared header, footer and canonical")
|
||||
if "__SITE_" in text:
|
||||
raise ValueError(f"{name}: unresolved site placeholder")
|
||||
for resource in page.resources:
|
||||
url = urlsplit(resource)
|
||||
if url.scheme or url.netloc or not url.path:
|
||||
continue
|
||||
resource_path = unquote(url.path)
|
||||
if prefix and resource_path.startswith(prefix + "/"):
|
||||
resource_path = resource_path[len(prefix) :]
|
||||
local = (output / resource_path.lstrip("/") if resource_path.startswith("/") else path.parent / resource_path).resolve()
|
||||
if not local.is_relative_to(output) or not local.is_file():
|
||||
raise ValueError(f"{name}: missing local resource {resource}")
|
||||
for asset in ["chrome.css", "chrome.js"]:
|
||||
if (output / "site-base" / asset).read_bytes() != (HERE / "base" / asset).read_bytes():
|
||||
raise ValueError(f"{name}: stale common asset {asset}")
|
||||
print(f"Verified shared layout and local resources for {len(config['pages'])} pages")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("command", choices=["render", "verify"])
|
||||
parser.add_argument("output", type=Path)
|
||||
parser.add_argument("--source", type=Path)
|
||||
parser.add_argument("--baseurl", default="")
|
||||
args = parser.parse_args()
|
||||
if args.command == "render":
|
||||
render(args.output, args.source, args.baseurl)
|
||||
else:
|
||||
verify(args.output, args.baseurl)
|
||||
@@ -1,255 +0,0 @@
|
||||
"""Check the built site in real Chromium at desktop and phone widths."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from functools import partial
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from urllib.parse import unquote, urljoin, urlsplit
|
||||
|
||||
from playwright.sync_api import expect, sync_playwright
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
class Handler(SimpleHTTPRequestHandler):
|
||||
def handle(self):
|
||||
# Browser navigation can cancel an in-flight lazy image response.
|
||||
try:
|
||||
super().handle()
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
pass
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
class ScreenshotReferences(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.references = []
|
||||
self.ids = set()
|
||||
|
||||
def handle_starttag(self, tag, attributes):
|
||||
attrs = dict(attributes)
|
||||
if attrs.get("id"):
|
||||
self.ids.add(attrs["id"])
|
||||
for attribute in ("src", "href"):
|
||||
if attrs.get(attribute):
|
||||
self.references.append(attrs[attribute])
|
||||
|
||||
|
||||
def verify_screenshot_references(output, canonical):
|
||||
"""Check gallery embeds on every static page, including optional translations."""
|
||||
pages = {}
|
||||
for path in output.rglob("*.html"):
|
||||
page = ScreenshotReferences()
|
||||
page.feed(path.read_text())
|
||||
pages[path.resolve()] = page
|
||||
for path, page in pages.items():
|
||||
page_url = urljoin(canonical, path.relative_to(output.resolve()).as_posix())
|
||||
for reference in page.references:
|
||||
url = urlsplit(urljoin(page_url, reference))
|
||||
if url.netloc != urlsplit(canonical).netloc or "/screenshots/" not in url.path:
|
||||
continue
|
||||
target = output / unquote(url.path).lstrip("/")
|
||||
if target.is_dir():
|
||||
target /= "index.html"
|
||||
target = target.resolve()
|
||||
assert target.is_relative_to(output.resolve()) and target.is_file(), f"{path}: missing screenshot target {reference}"
|
||||
assert not re.match(r"^\d+-", target.name), f"{path}: screenshot filename depends on capture order: {reference}"
|
||||
if url.fragment and target in pages:
|
||||
assert unquote(url.fragment) in pages[target].ids, f"{path}: missing screenshot anchor {reference}"
|
||||
|
||||
|
||||
def verify(output, evidence):
|
||||
config = json.loads((HERE / "site.json").read_text())
|
||||
verify_screenshot_references(output, config["url"])
|
||||
evidence.mkdir(parents=True, exist_ok=True)
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), partial(Handler, directory=str(output.resolve())))
|
||||
thread = Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
origin = f"http://127.0.0.1:{server.server_port}"
|
||||
try:
|
||||
with sync_playwright() as playwright:
|
||||
browser = playwright.chromium.launch(ignore_default_args=["--hide-scrollbars"])
|
||||
for name in config["pages"]:
|
||||
route = name.removesuffix("index.html")
|
||||
page = browser.new_page(reduced_motion="reduce")
|
||||
errors = []
|
||||
missing = []
|
||||
page.on("pageerror", lambda error, errors=errors: errors.append(str(error)))
|
||||
page.on(
|
||||
"response",
|
||||
lambda response, missing=missing: (
|
||||
missing.append(response.url) if response.url.startswith(origin) and response.status >= 400 else None
|
||||
),
|
||||
)
|
||||
for width in [1716, 2560, 390]:
|
||||
page.set_viewport_size({"width": width, "height": 900})
|
||||
page.goto(f"{origin}/{route}", wait_until="domcontentloaded")
|
||||
header = page.locator(".abx-header")
|
||||
expect(header).to_have_count(1)
|
||||
expect(header).to_be_visible()
|
||||
expect(header).to_be_in_viewport()
|
||||
brand = header.locator(".abx-brand")
|
||||
expect(brand).to_be_visible()
|
||||
expect(brand).to_contain_text("ArchiveBox")
|
||||
expect(brand.locator(".abx-logo")).to_be_visible()
|
||||
expect(brand.locator(".abx-logo")).to_be_in_viewport()
|
||||
expect(header).to_have_css("display", "flex")
|
||||
expect(page.locator(".abx-footer")).to_have_count(1)
|
||||
expect(page.locator(".abx-footer-column")).to_have_count(3)
|
||||
expect(header.locator(".abx-cta")).to_be_visible()
|
||||
menu = header.locator(".abx-apps")
|
||||
menu.locator("summary").focus()
|
||||
page.keyboard.press("Enter")
|
||||
expect(menu).to_have_attribute("open", "")
|
||||
expect(menu.locator("a").first).to_be_visible()
|
||||
page.keyboard.press("Escape")
|
||||
expect(menu).not_to_have_attribute("open", "")
|
||||
menu.locator("summary").click()
|
||||
box = header.bounding_box()
|
||||
assert box
|
||||
page.mouse.click(box["x"] + 2, box["y"] + 5)
|
||||
expect(menu).not_to_have_attribute("open", "")
|
||||
assert page.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth"), (
|
||||
f"{name}: overflow at {width}"
|
||||
)
|
||||
if not route and "screenshots/index.html" in config["pages"]:
|
||||
strips = page.locator(".abx-marquee")
|
||||
assert strips.count() > 0, "Homepage must use its screenshot gallery"
|
||||
for strip in strips.all():
|
||||
strip.scroll_into_view_if_needed()
|
||||
bounds = strip.bounding_box()
|
||||
usable_width = page.evaluate("document.documentElement.clientWidth")
|
||||
assert bounds and abs(bounds["x"]) < 1
|
||||
assert abs(bounds["width"] - usable_width) < 1
|
||||
assert not strip.inner_text().strip(), "Screenshot strips should have no visible captions or links"
|
||||
cards = strip.locator(".abx-marquee-card")
|
||||
assert cards.count() > 0, "Screenshot strip is empty"
|
||||
expect(strip.locator("button")).to_have_count(0)
|
||||
expect(strip.locator("a:not(.abx-marquee-card)")).to_have_count(0)
|
||||
viewport = strip.locator(".abx-marquee-viewport")
|
||||
overflowing = viewport.evaluate(
|
||||
"node => node.scrollWidth > node.clientWidth",
|
||||
)
|
||||
if overflowing:
|
||||
page.mouse.move(0, 0)
|
||||
page.emulate_media(reduced_motion="no-preference")
|
||||
initial = viewport.evaluate("node => node.scrollLeft")
|
||||
page.wait_for_function(
|
||||
"([node, initial]) => node.scrollLeft > initial",
|
||||
arg=[viewport.element_handle(), initial],
|
||||
)
|
||||
viewport.hover()
|
||||
stopped = viewport.evaluate("node => node.scrollLeft")
|
||||
viewport.evaluate(
|
||||
"node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
|
||||
)
|
||||
assert viewport.evaluate("node => node.scrollLeft") == stopped
|
||||
page.emulate_media(reduced_motion="reduce")
|
||||
page.mouse.move(0, 0)
|
||||
stopped = viewport.evaluate("node => node.scrollLeft")
|
||||
viewport.evaluate(
|
||||
"node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
|
||||
)
|
||||
assert viewport.evaluate("node => node.scrollLeft") == stopped
|
||||
viewport.focus()
|
||||
page.keyboard.press("ArrowRight")
|
||||
assert viewport.evaluate("node => node.scrollWidth > node.clientWidth")
|
||||
else:
|
||||
centered = viewport.evaluate(
|
||||
"""node => {
|
||||
const view = node.getBoundingClientRect();
|
||||
const track = node.querySelector('.abx-marquee-track').getBoundingClientRect();
|
||||
return Math.abs((track.left - view.left) - (view.right - track.right)) < 2;
|
||||
}""",
|
||||
)
|
||||
assert centered, "Short screenshot strips must be centered"
|
||||
for card in cards.all():
|
||||
href = card.get_attribute("href")
|
||||
assert href and "/screenshots/" in href
|
||||
response = page.request.get(urljoin(page.url, href))
|
||||
assert response.ok, f"Broken screenshot link: {href}"
|
||||
anchor = urlsplit(href).fragment
|
||||
if anchor:
|
||||
assert f'id="{anchor}"' in response.text(), f"Missing screenshot anchor: {href}"
|
||||
# Decode every local content image using the real browser loader,
|
||||
# including images in offscreen carousel tracks and lazy galleries.
|
||||
images = page.locator("main img").evaluate_all("nodes => [...new Set(nodes.map(image => image.src))]")
|
||||
local_images = [url for url in images if url.startswith(origin + "/")]
|
||||
page.evaluate(
|
||||
"""async urls => {
|
||||
for (const url of urls) {
|
||||
const image = new Image();
|
||||
image.src = url;
|
||||
await image.decode();
|
||||
if (!image.naturalWidth) throw new Error(`Broken image: ${url}`);
|
||||
}
|
||||
}""",
|
||||
local_images,
|
||||
)
|
||||
page.evaluate("scrollTo({top: 0, behavior: 'instant'})")
|
||||
page.evaluate(
|
||||
"() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
|
||||
)
|
||||
page.screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}{width}.png"))
|
||||
page.locator(".abx-footer").screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}footer-{width}.png"))
|
||||
for link in header.locator(".abx-nav > a").all():
|
||||
url = urlsplit(link.get_attribute("href") or "")
|
||||
if url.scheme or url.netloc:
|
||||
continue
|
||||
response = page.request.get(urljoin(page.url, url.geturl()))
|
||||
assert response.ok, f"Broken navigation: {url.geturl()}"
|
||||
if url.fragment:
|
||||
body = response.text()
|
||||
assert f'id="{url.fragment}"' in body or f"id='{url.fragment}'" in body, (
|
||||
f"Missing navigation anchor: {url.geturl()}"
|
||||
)
|
||||
if route == "" and page.locator("#resources").count():
|
||||
page.goto(f"{origin}/#resources")
|
||||
expect(page.locator("#resources")).to_be_visible()
|
||||
assert not errors, errors
|
||||
assert not missing, missing
|
||||
page.close()
|
||||
context = browser.new_context(java_script_enabled=False, viewport={"width": 390, "height": 900})
|
||||
plain = context.new_page()
|
||||
plain.goto(f"{origin}/{route}", wait_until="domcontentloaded")
|
||||
expect(plain.locator(".abx-header")).to_be_visible()
|
||||
expect(plain.locator(".abx-header .abx-brand")).to_be_in_viewport()
|
||||
expect(plain.locator(".abx-header .abx-logo")).to_be_visible()
|
||||
plain.locator(".abx-apps summary").click()
|
||||
expect(plain.locator(".abx-app-links a").first).to_be_visible()
|
||||
expect(plain.locator(".abx-footer-column a").first).to_be_visible()
|
||||
if not route and "screenshots/index.html" in config["pages"]:
|
||||
assert plain.locator(".abx-marquee-card").count() > 0
|
||||
for strip in plain.locator(".abx-marquee").all():
|
||||
bounds = strip.bounding_box()
|
||||
usable_width = plain.evaluate("document.documentElement.clientWidth")
|
||||
assert bounds and abs(bounds["x"]) < 1
|
||||
assert abs(bounds["width"] - usable_width) < 1
|
||||
assert plain.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth")
|
||||
expect(plain.locator(".abx-marquee button")).to_have_count(0)
|
||||
expect(plain.locator(".abx-marquee-card").first).to_be_visible()
|
||||
context.close()
|
||||
print(
|
||||
f"PASS {name}: desktop/mobile, keyboard, no-JS, images and resources",
|
||||
flush=True,
|
||||
)
|
||||
browser.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("output", type=Path)
|
||||
parser.add_argument("--evidence", type=Path, default=Path("test-results/site"))
|
||||
args = parser.parse_args()
|
||||
verify(args.output, args.evidence)
|
||||
@@ -1,78 +1,37 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Discover every test once, batching measured short files."""
|
||||
"""Discover every ArchiveBox test file and expose it as one CI job."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main() -> None:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
root = Path.cwd().resolve()
|
||||
archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))
|
||||
|
||||
if not archivebox_tests:
|
||||
raise SystemExit("No ArchiveBox tests discovered")
|
||||
|
||||
# These are observed whole-job seconds, including setup. Unknown/long files,
|
||||
# optional dependencies and hosted-only requirements keep independent jobs.
|
||||
durations = json.loads((root / ".github/test-durations.json").read_text())["seconds"]
|
||||
hosted = {path for path in archivebox_tests if "# ci-runner: hosted" in path.read_text().splitlines()[:5]}
|
||||
requested = os.environ.get("TEST_FILE", "")
|
||||
batches: list[list[str]] = []
|
||||
batch: list[str] = []
|
||||
batch_seconds = 0
|
||||
matrix: list[dict[str, object]] = []
|
||||
for path in archivebox_tests:
|
||||
test_path = path.relative_to(root).as_posix()
|
||||
duration = durations.get(test_path, 60)
|
||||
if duration >= 60 or path in hosted or path.stem == "test_auth_ldap" or test_path == requested:
|
||||
batches.append([test_path])
|
||||
continue
|
||||
if batch and (len(batch) == 8 or batch_seconds + duration > 180):
|
||||
batches.append(batch)
|
||||
batch = []
|
||||
batch_seconds = 0
|
||||
batch.append(test_path)
|
||||
batch_seconds += duration
|
||||
if batch:
|
||||
batches.append(batch)
|
||||
|
||||
matrix: list[dict[str, object]] = []
|
||||
for paths in batches:
|
||||
matrix.append(
|
||||
{
|
||||
"name": f"main/{Path(paths[0]).stem}" if len(paths) == 1 else f"batch/{Path(paths[0]).stem}",
|
||||
"paths": paths,
|
||||
"paths_arg": " ".join(paths),
|
||||
"extra": "ldap" if paths == ["archivebox/tests/test_auth_ldap.py"] else "",
|
||||
"count": len(paths),
|
||||
"ugnas": False,
|
||||
"name": f"main/{path.stem}",
|
||||
"paths": [test_path],
|
||||
"paths_arg": test_path,
|
||||
"extra": "ldap" if test_path.endswith("/test_auth_ldap.py") else "",
|
||||
"count": 1,
|
||||
},
|
||||
)
|
||||
|
||||
# Ordinary Linux tests can use either pool. Only genuine hosted-only
|
||||
# requirements need a file header; filenames never enter runner policy.
|
||||
# Keep a bounded NAS share so hosted capacity continues working in parallel.
|
||||
# This is queued work, not a worker count: several waves keep the capped
|
||||
# runners occupied while hosted jobs run in parallel.
|
||||
nas_job_limit = int(os.environ.get("UGNAS_CI_MAX_JOBS", "3"))
|
||||
# Keep long jobs early in the hosted queue. Giving the longest jobs to the
|
||||
# CPU-capped NAS instead created a tail after hosted work had finished; use
|
||||
# its extra slots for a bounded share of shorter work without filename rules.
|
||||
matrix.sort(key=lambda item: sum(durations.get(path, 60) for path in item["paths"]), reverse=True)
|
||||
eligible = [item for item in reversed(matrix) if not any(root / path in hosted for path in item["paths"])]
|
||||
for item in eligible[:nas_job_limit]:
|
||||
item["ugnas"] = True
|
||||
|
||||
discovered_paths = [path for entry in matrix for path in entry["paths"]]
|
||||
if len(discovered_paths) != len(set(discovered_paths)):
|
||||
raise SystemExit("Tests were not discovered exactly once")
|
||||
if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
|
||||
raise SystemExit("Discovered test shard coverage does not match test files")
|
||||
|
||||
print(
|
||||
f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs; "
|
||||
f"{sum(item['ugnas'] for item in matrix)} selected for ugNAS, subject to the availability check",
|
||||
)
|
||||
print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
|
||||
print(json.dumps(matrix, separators=(",", ":")))
|
||||
|
||||
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
{
|
||||
"source_runs": [
|
||||
"https://github.com/ArchiveBox/ArchiveBox/actions/runs/36302511923",
|
||||
"https://github.com/ArchiveBox/ArchiveBox/actions/runs/36353196552",
|
||||
"https://github.com/ArchiveBox/ArchiveBox/actions/runs/36454992009"
|
||||
],
|
||||
"measurement": "Whole job seconds, including setup and excluding queue time. Hosted unbatched files use the latest complete successful run; other files retain their individual baseline measurement. Unknown files stay independent.",
|
||||
"seconds": {
|
||||
"archivebox/tests/test_api_v1_auth_browser_session.py": 29,
|
||||
"archivebox/tests/test_api_v1_auth_check_api_token.py": 27,
|
||||
"archivebox/tests/test_api_v1_auth_get_api_token.py": 24,
|
||||
"archivebox/tests/test_api_v1_cli_add.py": 96,
|
||||
"archivebox/tests/test_api_v1_cli_remove.py": 50,
|
||||
"archivebox/tests/test_api_v1_cli_schedule.py": 41,
|
||||
"archivebox/tests/test_api_v1_cli_search.py": 26,
|
||||
"archivebox/tests/test_api_v1_cli_update.py": 74,
|
||||
"archivebox/tests/test_api_v1_cli_workflow_add_search_update_remove.py": 64,
|
||||
"archivebox/tests/test_api_v1_core_any_id.py": 21,
|
||||
"archivebox/tests/test_api_v1_core_archiveresult_archiveresult_id.py": 55,
|
||||
"archivebox/tests/test_api_v1_core_archiveresults.py": 32,
|
||||
"archivebox/tests/test_api_v1_core_snapshot_snapshot_id.py": 199,
|
||||
"archivebox/tests/test_api_v1_core_snapshots_rss.py": 27,
|
||||
"archivebox/tests/test_api_v1_core_snapshots.py": 28,
|
||||
"archivebox/tests/test_api_v1_core_tag_tag_id_rename.py": 23,
|
||||
"archivebox/tests/test_api_v1_core_tag_tag_id_snapshots_jsonl.py": 27,
|
||||
"archivebox/tests/test_api_v1_core_tag_tag_id_urls_txt.py": 28,
|
||||
"archivebox/tests/test_api_v1_core_tag_tag_id.py": 27,
|
||||
"archivebox/tests/test_api_v1_core_tags_add_to_snapshot.py": 27,
|
||||
"archivebox/tests/test_api_v1_core_tags_autocomplete.py": 26,
|
||||
"archivebox/tests/test_api_v1_core_tags_create.py": 21,
|
||||
"archivebox/tests/test_api_v1_core_tags_remove_from_snapshot.py": 25,
|
||||
"archivebox/tests/test_api_v1_core_tags_search.py": 26,
|
||||
"archivebox/tests/test_api_v1_core_tags.py": 25,
|
||||
"archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_filename.py": 22,
|
||||
"archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_filename.py": 25,
|
||||
"archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_subfolder_filename.py": 24,
|
||||
"archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py": 86,
|
||||
"archivebox/tests/test_api_v1_crawls_crawls.py": 21,
|
||||
"archivebox/tests/test_api_v1_machine_binaries.py": 23,
|
||||
"archivebox/tests/test_api_v1_machine_binary_binary_id.py": 23,
|
||||
"archivebox/tests/test_api_v1_machine_binary_by_name_name.py": 26,
|
||||
"archivebox/tests/test_api_v1_machine_machine_current.py": 23,
|
||||
"archivebox/tests/test_api_v1_machine_machine_machine_id.py": 23,
|
||||
"archivebox/tests/test_api_v1_machine_machines.py": 27,
|
||||
"archivebox/tests/test_api_v1_personas_personas.py": 23,
|
||||
"archivebox/tests/test_api_v1_personas_sync.py": 29,
|
||||
"archivebox/tests/test_api_v1_workflow_core_token_auth_side_effects.py": 36,
|
||||
"archivebox/tests/test_api_v1_workflow_frozen_crawl_config_sources.py": 33,
|
||||
"archivebox/tests/test_archive_result_service.py": 137,
|
||||
"archivebox/tests/test_auth_ldap.py": 44,
|
||||
"archivebox/tests/test_binary_service.py": 83,
|
||||
"archivebox/tests/test_cli_add_1.py": 168,
|
||||
"archivebox/tests/test_cli_add_2.py": 203,
|
||||
"archivebox/tests/test_cli_add_3.py": 215,
|
||||
"archivebox/tests/test_cli_add_4.py": 201,
|
||||
"archivebox/tests/test_cli_archiveresult.py": 199,
|
||||
"archivebox/tests/test_cli_binary.py": 20,
|
||||
"archivebox/tests/test_cli_config.py": 148,
|
||||
"archivebox/tests/test_cli_crawl.py": 177,
|
||||
"archivebox/tests/test_cli_extract_input.py": 155,
|
||||
"archivebox/tests/test_cli_extract.py": 106,
|
||||
"archivebox/tests/test_cli_help.py": 30,
|
||||
"archivebox/tests/test_cli_init.py": 135,
|
||||
"archivebox/tests/test_cli_install.py": 58,
|
||||
"archivebox/tests/test_cli_list.py": 260,
|
||||
"archivebox/tests/test_cli_machine.py": 50,
|
||||
"archivebox/tests/test_cli_manage.py": 48,
|
||||
"archivebox/tests/test_cli_mcp.py": 51,
|
||||
"archivebox/tests/test_cli_oneshot.py": 23,
|
||||
"archivebox/tests/test_cli_persona.py": 82,
|
||||
"archivebox/tests/test_cli_piping.py": 97,
|
||||
"archivebox/tests/test_cli_pluginmap.py": 21,
|
||||
"archivebox/tests/test_cli_process.py": 20,
|
||||
"archivebox/tests/test_cli_remove.py": 162,
|
||||
"archivebox/tests/test_cli_run_1.py": 195,
|
||||
"archivebox/tests/test_cli_run_2.py": 255,
|
||||
"archivebox/tests/test_cli_run_3.py": 167,
|
||||
"archivebox/tests/test_cli_run_4.py": 178,
|
||||
"archivebox/tests/test_cli_schedule.py": 87,
|
||||
"archivebox/tests/test_cli_search.py": 35,
|
||||
"archivebox/tests/test_cli_server.py": 259,
|
||||
"archivebox/tests/test_cli_shell.py": 32,
|
||||
"archivebox/tests/test_cli_snapshot.py": 181,
|
||||
"archivebox/tests/test_cli_status.py": 102,
|
||||
"archivebox/tests/test_cli_tag.py": 21,
|
||||
"archivebox/tests/test_cli_update_reindex_snapshots.py": 163,
|
||||
"archivebox/tests/test_cli_update.py": 142,
|
||||
"archivebox/tests/test_cli_version.py": 143,
|
||||
"archivebox/tests/test_config_DELETE_AFTER.py": 169,
|
||||
"archivebox/tests/test_config_MAX_limits.py": 34,
|
||||
"archivebox/tests/test_config_ONLY_NEW.py": 24,
|
||||
"archivebox/tests/test_config_SAVE_TITLE.py": 99,
|
||||
"archivebox/tests/test_config_URL_filters.py": 26,
|
||||
"archivebox/tests/test_core_config.py": 20,
|
||||
"archivebox/tests/test_core_forms.py": 28,
|
||||
"archivebox/tests/test_crawl_pause.py": 20,
|
||||
"archivebox/tests/test_crawl_runner.py": 45,
|
||||
"archivebox/tests/test_crawl_service.py": 34,
|
||||
"archivebox/tests/test_frozen_crawl_config.py": 47,
|
||||
"archivebox/tests/test_hook_result_lifecycle.py": 78,
|
||||
"archivebox/tests/test_hooks.py": 90,
|
||||
"archivebox/tests/test_machine_models.py": 52,
|
||||
"archivebox/tests/test_machine_service.py": 89,
|
||||
"archivebox/tests/test_migrations_04_to_09.py": 36,
|
||||
"archivebox/tests/test_migrations_07_to_09.py": 183,
|
||||
"archivebox/tests/test_migrations_08_to_09.py": 244,
|
||||
"archivebox/tests/test_migrations_fresh.py": 99,
|
||||
"archivebox/tests/test_misc_checks.py": 22,
|
||||
"archivebox/tests/test_misc_monkey_patches.py": 20,
|
||||
"archivebox/tests/test_opencode_agent_1.py": 158,
|
||||
"archivebox/tests/test_opencode_agent_2.py": 145,
|
||||
"archivebox/tests/test_opencode_agent_3.py": 154,
|
||||
"archivebox/tests/test_opencode_browser_1.py": 262,
|
||||
"archivebox/tests/test_opencode_browser_2.py": 146,
|
||||
"archivebox/tests/test_opencode_browser_3.py": 148,
|
||||
"archivebox/tests/test_orm_helpers.py": 22,
|
||||
"archivebox/tests/test_permissions.py": 21,
|
||||
"archivebox/tests/test_persona_runtime.py": 56,
|
||||
"archivebox/tests/test_postgres_backend.py": 85,
|
||||
"archivebox/tests/test_process_runtime_paths.py": 28,
|
||||
"archivebox/tests/test_process_service.py": 49,
|
||||
"archivebox/tests/test_proof_previews.py": 54,
|
||||
"archivebox/tests/test_recursive_crawl_1.py": 339,
|
||||
"archivebox/tests/test_recursive_crawl_2.py": 171,
|
||||
"archivebox/tests/test_recursive_crawl_3.py": 176,
|
||||
"archivebox/tests/test_release_notes.py": 14,
|
||||
"archivebox/tests/test_release_source_fingerprint.py": 22,
|
||||
"archivebox/tests/test_release_workflow.py": 21,
|
||||
"archivebox/tests/test_resource_admission.py": 21,
|
||||
"archivebox/tests/test_runner_watch.py": 177,
|
||||
"archivebox/tests/test_search.py": 153,
|
||||
"archivebox/tests/test_serve_static.py": 31,
|
||||
"archivebox/tests/test_server_automatic_url.py": 36,
|
||||
"archivebox/tests/test_server_security_browser.py": 278,
|
||||
"archivebox/tests/test_settings_signal_webhooks.py": 21,
|
||||
"archivebox/tests/test_setup_script.py": 21,
|
||||
"archivebox/tests/test_shutdown_util.py": 22,
|
||||
"archivebox/tests/test_snapshot_filesystem_migration.py": 29,
|
||||
"archivebox/tests/test_snapshot_output_stacks.py": 62,
|
||||
"archivebox/tests/test_snapshot_ref_lookup.py": 23,
|
||||
"archivebox/tests/test_snapshot_service.py": 41,
|
||||
"archivebox/tests/test_snapshot_url_length.py": 26,
|
||||
"archivebox/tests/test_tag_service.py": 24,
|
||||
"archivebox/tests/test_takeover_util_1.py": 143,
|
||||
"archivebox/tests/test_takeover_util_2.py": 201,
|
||||
"archivebox/tests/test_takeover_util_3.py": 138,
|
||||
"archivebox/tests/test_takeover_util_4.py": 156,
|
||||
"archivebox/tests/test_test_harness.py": 22,
|
||||
"archivebox/tests/test_ui_add_view_runtime.py": 221,
|
||||
"archivebox/tests/test_ui_add_view.py": 46,
|
||||
"archivebox/tests/test_ui_admin_apitoken.py": 23,
|
||||
"archivebox/tests/test_ui_admin_archiveresult.py": 30,
|
||||
"archivebox/tests/test_ui_admin_config_widget.py": 25,
|
||||
"archivebox/tests/test_ui_admin_crawl.py": 44,
|
||||
"archivebox/tests/test_ui_admin_links.py": 35,
|
||||
"archivebox/tests/test_ui_admin_machine.py": 29,
|
||||
"archivebox/tests/test_ui_admin_persona.py": 28,
|
||||
"archivebox/tests/test_ui_admin_snapshot.py": 273,
|
||||
"archivebox/tests/test_ui_admin_tag.py": 26,
|
||||
"archivebox/tests/test_ui_admin_user.py": 28,
|
||||
"archivebox/tests/test_ui_config_views.py": 43,
|
||||
"archivebox/tests/test_ui_live_progress.py": 297,
|
||||
"archivebox/tests/test_ui_public_snapshot.py": 179,
|
||||
"archivebox/tests/test_ui_savepagenow.py": 48,
|
||||
"archivebox/tests/test_urls.py": 220,
|
||||
"archivebox/tests/test_util.py": 20,
|
||||
"archivebox/tests/test_version_metadata.py": 24
|
||||
}
|
||||
}
|
||||
@@ -18,68 +18,47 @@ concurrency:
|
||||
cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
permissions:
|
||||
contents: write
|
||||
uses: ./.github/workflows/prepare-release.yml
|
||||
secrets: inherit
|
||||
|
||||
lint:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Linters
|
||||
uses: ./.github/workflows/lint.yml
|
||||
secrets: inherit
|
||||
|
||||
install-cli-platform:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Install and CLI platform compatibility
|
||||
uses: ./.github/workflows/test.yml
|
||||
secrets: inherit
|
||||
|
||||
tests:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Discovered test matrix
|
||||
uses: ./.github/workflows/test-parallel.yml
|
||||
secrets: inherit
|
||||
|
||||
documentation:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Documentation and root tests
|
||||
uses: ./.github/workflows/docs.yml
|
||||
secrets: inherit
|
||||
|
||||
codeql:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: CodeQL
|
||||
uses: ./.github/workflows/codeql.yml
|
||||
secrets: inherit
|
||||
|
||||
python-artifacts:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Tested Python artifacts
|
||||
uses: ./.github/workflows/pip.yml
|
||||
secrets: inherit
|
||||
|
||||
docker-digests:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
name: Tested Docker images
|
||||
uses: ./.github/workflows/docker.yml
|
||||
with:
|
||||
push_digests: ${{ github.event_name == 'push' && (github.ref_name == 'dev' || github.ref_name == 'main') }}
|
||||
push_digests: false
|
||||
secrets: inherit
|
||||
|
||||
required:
|
||||
name: All required CI lanes
|
||||
if: ${{ always() && needs.prepare.outputs.run_jobs == 'true' }}
|
||||
if: ${{ always() }}
|
||||
needs:
|
||||
- prepare
|
||||
- lint
|
||||
- install-cli-platform
|
||||
- tests
|
||||
|
||||
@@ -1,87 +1,79 @@
|
||||
name: Publish website
|
||||
name: Deploy Publicsite to GitHub Pages
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [dev]
|
||||
pull_request:
|
||||
branches:
|
||||
- dev
|
||||
paths:
|
||||
- publicsite/**
|
||||
- archivebox/**
|
||||
- bin/collect_ui_screenshots.*
|
||||
- bin/generate_ui_screenshot_gallery.py
|
||||
- bin/take_screenshot.js
|
||||
- docs/Screenshots.md
|
||||
- pyproject.toml
|
||||
- uv.lock
|
||||
- .github/workflows/deploy-publicsite.yml
|
||||
workflow_dispatch:
|
||||
workflow_run:
|
||||
workflows: ['Capture publicsite screenshots']
|
||||
types: [completed]
|
||||
branches: [dev]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
env:
|
||||
# Repository-specific output; all publication logic below is a local copy.
|
||||
SITE_OUTPUT: _site
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
|
||||
# One production queue covers restore, build and deploy. PRs have separate queues.
|
||||
concurrency:
|
||||
group: pages-${{ github.event_name == 'pull_request' && github.ref || 'production' }}
|
||||
cancel-in-progress: false
|
||||
group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
|
||||
cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
outputs:
|
||||
revision: ${{ steps.source.outputs.revision }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ github.event_name == 'pull_request' && github.ref || env.DEFAULT_BRANCH }}
|
||||
persist-credentials: false
|
||||
- name: Record website source revision
|
||||
id: source
|
||||
run: echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# BEGIN REPOSITORY-SPECIFIC BUILD
|
||||
- uses: astral-sh/setup-uv@v7
|
||||
- name: Restore latest complete screenshots
|
||||
run: uv run --no-project python bin/restore_site_screenshots.py publicsite/screenshots
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
- name: Render current gallery layout without recapturing images
|
||||
run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
|
||||
- name: Assemble website with the local base components
|
||||
run: uv run --no-project python .github/pages/site.py render _site --source publicsite
|
||||
# END REPOSITORY-SPECIFIC BUILD
|
||||
|
||||
- name: Verify website in Chromium
|
||||
run: |
|
||||
uv run --no-config --no-project --with playwright==1.63.0 playwright install --with-deps chromium
|
||||
uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py "$SITE_OUTPUT" --evidence "$RUNNER_TEMP/site-evidence"
|
||||
- name: Save website verification evidence
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: site-verification
|
||||
path: ${{ runner.temp }}/site-evidence
|
||||
if-no-files-found: warn
|
||||
retention-days: 14
|
||||
|
||||
- name: Upload website
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: ${{ env.SITE_OUTPUT }}
|
||||
|
||||
deploy:
|
||||
if: github.event_name != 'pull_request'
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/configure-pages@v5
|
||||
- uses: actions/deploy-pages@v4
|
||||
- name: Checkout
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Setup Python and uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
|
||||
- name: Install ArchiveBox and screenshot dependencies
|
||||
run: uv sync --no-cache --frozen
|
||||
|
||||
- name: Build UI screenshot gallery
|
||||
run: bin/collect_ui_screenshots.sh
|
||||
env:
|
||||
BASE_URL: http://archivebox.localhost:8000
|
||||
|
||||
- name: Verify gallery was generated from this exact revision
|
||||
run: |
|
||||
uv run --no-cache --no-sync python - <<'PY'
|
||||
import json
|
||||
import os
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
|
||||
version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
|
||||
assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
|
||||
assert provenance["version"] == version, provenance
|
||||
assert provenance["capture_count"] >= 3 * 35, provenance
|
||||
assert len(provenance["files"]) == provenance["capture_count"], provenance
|
||||
PY
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
|
||||
with:
|
||||
path: ./publicsite
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
|
||||
@@ -13,22 +13,6 @@ on:
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
source_ref:
|
||||
description: Exact 40-character commit SHA to build
|
||||
required: true
|
||||
type: string
|
||||
push_digests:
|
||||
description: Push validated per-platform images by digest for the release workflow
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
full_tests:
|
||||
description: Run storage, CLI, and size validation in addition to release smoke checks
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
DOCKERHUB_IMAGE: archivebox/archivebox
|
||||
@@ -58,18 +42,6 @@ jobs:
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
ref: ${{ inputs.source_ref || github.sha }}
|
||||
|
||||
- name: Verify exact source commit
|
||||
if: github.event_name == 'workflow_dispatch'
|
||||
shell: bash
|
||||
env:
|
||||
SOURCE_REF: ${{ inputs.source_ref || github.sha }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
[[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
|
||||
[[ "$(git rev-parse HEAD)" == "$SOURCE_REF" ]]
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
|
||||
@@ -211,7 +183,7 @@ jobs:
|
||||
{
|
||||
echo 'labels<<EOF'
|
||||
echo "org.opencontainers.image.version=${VERSION}"
|
||||
echo "org.opencontainers.image.revision=${{ inputs.source_ref || github.sha }}"
|
||||
echo "org.opencontainers.image.revision=${GITHUB_SHA}"
|
||||
echo 'org.opencontainers.image.source=https://github.com/ArchiveBox/ArchiveBox'
|
||||
echo "io.archivebox.abx-dl.image=${{ steps.abx_dl_image.outputs.image }}"
|
||||
echo 'EOF'
|
||||
@@ -231,7 +203,7 @@ jobs:
|
||||
labels: ${{ steps.docker_meta.outputs.labels }}
|
||||
build-args: |
|
||||
ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
|
||||
ARCHIVEBOX_COMMIT_HASH=${{ inputs.source_ref || github.sha }}
|
||||
ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
|
||||
pull: true
|
||||
platforms: ${{ matrix.platform }}
|
||||
# Keep the released image as the one cache authority. A second GHA
|
||||
@@ -260,7 +232,7 @@ jobs:
|
||||
labels: ${{ steps.docker_meta.outputs.labels }}
|
||||
build-args: |
|
||||
ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
|
||||
ARCHIVEBOX_COMMIT_HASH=${{ inputs.source_ref || github.sha }}
|
||||
ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
|
||||
pull: true
|
||||
platforms: ${{ matrix.platform }}
|
||||
# Test the same layer lineage users receive. Reusing :dev avoids a
|
||||
@@ -274,34 +246,13 @@ jobs:
|
||||
if: inputs.push_digests
|
||||
run: echo ${{ steps.docker_build.outputs.digest }}
|
||||
|
||||
- name: Verify OpenCode install with a read-only browser profile
|
||||
shell: bash
|
||||
env:
|
||||
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
PROFILE_DIR="$(mktemp -d)"
|
||||
"$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
|
||||
--volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
|
||||
"$IMAGE" bash -c '
|
||||
set -Eeuo pipefail
|
||||
test -w "$XDG_CONFIG_HOME"
|
||||
test ! -w "$XDG_CONFIG_HOME/chromium"
|
||||
# Check the shipped executable before the installer can repair
|
||||
# an image that accidentally omitted this opt-in dependency.
|
||||
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
|
||||
archivebox install opencode
|
||||
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
|
||||
'
|
||||
|
||||
- name: Validate exact built image version and commit
|
||||
if: inputs.full_tests
|
||||
if: ${{ !inputs.push_digests }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
IMAGE="${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}"
|
||||
SHORT_SHA="${{ (inputs.source_ref || github.sha) }}"
|
||||
SHORT_SHA="${SHORT_SHA::7}"
|
||||
IMAGE="${{ matrix.local_tag }}"
|
||||
SHORT_SHA="${GITHUB_SHA::7}"
|
||||
|
||||
DATA_DIR="$(mktemp -d)"
|
||||
"$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
|
||||
@@ -345,8 +296,8 @@ jobs:
|
||||
MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
|
||||
fi
|
||||
COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
|
||||
LIMIT=$((1025 * 1024 * 1024))
|
||||
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=1025 MiB"
|
||||
LIMIT=$((750 * 1024 * 1024))
|
||||
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
|
||||
(( COMPRESSED_SIZE < LIMIT ))
|
||||
|
||||
- name: Resolve real Docker CLI test dependencies through abxpkg
|
||||
@@ -406,7 +357,7 @@ jobs:
|
||||
"$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
|
||||
"$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
|
||||
"$SUDO_BINARY" "$BASH_BINARY" -c \
|
||||
"printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
|
||||
"printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
|
||||
"$SUDO_BINARY" "$BASH_BINARY" -c \
|
||||
"printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
|
||||
|
||||
@@ -424,20 +375,14 @@ jobs:
|
||||
|
||||
NFS_TEST_DIR="$NFS_MOUNT" \
|
||||
SMB_TEST_DIR="$SMB_MOUNT" \
|
||||
IMAGE="${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}" \
|
||||
IMAGE="${{ matrix.local_tag }}" \
|
||||
"$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
|
||||
|
||||
- name: Validate Docker port compatibility
|
||||
shell: bash
|
||||
env:
|
||||
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
|
||||
run: '"$BASH_BINARY" bin/validate_docker_ports.sh'
|
||||
|
||||
- name: Validate Docker CLI add/list and compose behavior
|
||||
if: inputs.full_tests && matrix.platform == 'linux/amd64'
|
||||
shell: bash
|
||||
env:
|
||||
ARCHIVEBOX_IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
|
||||
ARCHIVEBOX_IMAGE: ${{ matrix.local_tag }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
"$MKDIR_BINARY" "${{ github.workspace }}/data"
|
||||
@@ -456,13 +401,13 @@ jobs:
|
||||
"$DOCKER_BINARY" compose run --rm archivebox init
|
||||
"$DOCKER_BINARY" compose up -d --wait
|
||||
"$DOCKER_BINARY" network connect archivebox_default www-nginx
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
|
||||
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/static/admin/js/jquery.init.js')"
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
|
||||
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
|
||||
[[ "$index_html" == *ArchiveBox* ]]
|
||||
[[ "$admin_js" == *window.django* ]]
|
||||
|
||||
"$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
|
||||
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
|
||||
[[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
|
||||
"$DOCKER_BINARY" compose down
|
||||
|
||||
@@ -478,7 +423,7 @@ jobs:
|
||||
'${{ runner.temp }}/archivebox-image.tar' \
|
||||
'${{ steps.docker_build_pr.outputs.digest }}' \
|
||||
"$LOCAL_IMAGE_ID" \
|
||||
"$((1025 * 1024 * 1024))" <<'PY'
|
||||
"$((750 * 1024 * 1024))" <<'PY'
|
||||
from hashlib import sha256
|
||||
import json
|
||||
import os
|
||||
@@ -552,17 +497,12 @@ jobs:
|
||||
|
||||
- name: Export digest
|
||||
if: inputs.push_digests
|
||||
env:
|
||||
ABX_DL_IMAGE_REF: ${{ steps.abx_dl_image.outputs.image }}
|
||||
ARTIFACT_NAME: ${{ matrix.artifact_name }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mkdir -p /tmp/digests
|
||||
digest="${{ steps.docker_build.outputs.digest }}"
|
||||
touch "/tmp/digests/${digest#sha256:}"
|
||||
printf '%s\n' '${{ inputs.source_ref || github.sha }}' > /tmp/digests/source-ref
|
||||
printf '%s\n' "$ABX_DL_IMAGE_REF" > "/tmp/digests/abx-dl-image-$ARTIFACT_NAME"
|
||||
|
||||
- name: Upload digest
|
||||
if: inputs.push_digests
|
||||
|
||||
@@ -67,7 +67,7 @@ jobs:
|
||||
timeout-minutes: 15
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
max-parallel: 4
|
||||
matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
@@ -146,9 +146,7 @@ jobs:
|
||||
fi
|
||||
- name: Build the exact checkout's documentation image
|
||||
if: ${{ matrix.environment == 'docker' }}
|
||||
run: |
|
||||
ABX_DL_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(dep.removeprefix("abx-dl==") for dep in tomllib.load(open("pyproject.toml", "rb"))["project"]["dependencies"] if dep.startswith("abx-dl==")))')"
|
||||
"$DOCKER_BINARY" build --build-arg "ABX_DL_IMAGE=archivebox/abx-dl:${ABX_DL_VERSION}" --tag archivebox/archivebox:dev .
|
||||
run: '"$DOCKER_BINARY" build --tag archivebox/archivebox:dev .'
|
||||
- name: Run the Ubuntu, Docker, or macOS documentation lane
|
||||
if: ${{ !contains(fromJSON('["root", "freebsd", "openbsd"]'), matrix.environment) }}
|
||||
run: uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
name: Duplicate Issue Detection
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened]
|
||||
|
||||
jobs:
|
||||
check-duplicates:
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
|
||||
with:
|
||||
version: "0.11.3"
|
||||
enable-cache: false
|
||||
|
||||
- name: Resolve OpenCode and workflow tools through abxpkg
|
||||
env:
|
||||
ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
uv sync --no-cache --locked --no-sources --no-install-project
|
||||
OPENCODE_CONFIG="$(uv run --no-cache --no-sync --no-sources python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "opencode" / "config.json")')"
|
||||
mkdir -p "$ABXPKG_LIB_DIR/env/bin"
|
||||
export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
|
||||
echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
|
||||
echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
|
||||
opencode_env="$(
|
||||
uv run --no-cache --no-sync --no-sources abxpkg env \
|
||||
--install \
|
||||
--json \
|
||||
--lib="$ABXPKG_LIB_DIR" \
|
||||
--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
|
||||
--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:github_binaries" \
|
||||
--deps-from="$OPENCODE_CONFIG:required_binaries"
|
||||
)"
|
||||
jq_binary="$ABXPKG_LIB_DIR/env/bin/jq"
|
||||
"$jq_binary" -r 'to_entries[] | "\(.key)=\(.value)"' \
|
||||
<<< "$opencode_env" >> "$GITHUB_ENV"
|
||||
|
||||
opencode_binary="$ABXPKG_LIB_DIR/env/bin/opencode"
|
||||
echo "OPENCODE_BINARY=$opencode_binary" >> "$GITHUB_ENV"
|
||||
test -L "$opencode_binary"
|
||||
test -x "$opencode_binary"
|
||||
|
||||
- name: Check for duplicate issues
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OPENCODE_PERMISSION: |
|
||||
{
|
||||
"bash": {
|
||||
"gh issue*": "allow",
|
||||
"gh pr*": "allow",
|
||||
"*": "deny"
|
||||
},
|
||||
"webfetch": "allow"
|
||||
}
|
||||
run: |
|
||||
"$OPENCODE_BINARY" run -m anthropic/claude-haiku-4-5 "A new issue has been created:
|
||||
|
||||
Issue number:
|
||||
${{ github.event.issue.number }}
|
||||
|
||||
Lookup this issue and search through existing issues and PRs (excluding #${{ github.event.issue.number }}) in this repository to find any potential duplicates of this new issue.
|
||||
Consider:
|
||||
1. Similar titles or descriptions
|
||||
2. Same error messages or symptoms
|
||||
3. Related functionality or components
|
||||
4. Similar feature requests
|
||||
|
||||
If you find any potential duplicates, please comment on the new issue with:
|
||||
- A brief explanation of why it might be a duplicate
|
||||
- Links to the potentially duplicate issues or PRs
|
||||
- A suggestion to check those issues first
|
||||
|
||||
Use this format for the comment:
|
||||
'This issue might be a duplicate of existing issues. Please check:
|
||||
- #[issue_number]: [brief description of similarity]
|
||||
|
||||
Feel free to ignore if none of these address your specific case.'
|
||||
|
||||
If no clear duplicates are found, do not comment."
|
||||
@@ -1,55 +0,0 @@
|
||||
name: Prepare release version
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
outputs:
|
||||
run_jobs:
|
||||
description: Whether this revision owns its release version and should run expensive jobs
|
||||
value: ${{ jobs.prepare.outputs.run_jobs }}
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
name: Prepare the version before starting expensive jobs
|
||||
permissions:
|
||||
contents: write
|
||||
runs-on: ubuntu-24.04
|
||||
outputs:
|
||||
run_jobs: ${{ github.event_name != 'push' || steps.channel.outputs.stable == 'true' || steps.version.outputs.action == 'ready' || steps.version.outputs.action == 'reserved' }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
submodules: true
|
||||
token: ${{ github.event_name == 'push' && secrets.RELEASE_GH_TOKEN || github.token }}
|
||||
|
||||
- name: Keep synchronized stable branches on the released version
|
||||
id: channel
|
||||
if: github.event_name == 'push'
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
|
||||
git fetch --no-tags origin main
|
||||
MAIN_SHA="$(git rev-parse origin/main)"
|
||||
# Stable reconciliation may preserve dev history with a merge commit.
|
||||
# Its tree still matches main; bumping it would undo synchronization
|
||||
# before the stable publisher can advance the Docker :dev tag.
|
||||
if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] &&
|
||||
git merge-base --is-ancestor "$MAIN_SHA" "$GITHUB_SHA" &&
|
||||
git diff --quiet "$MAIN_SHA" "$GITHUB_SHA"; then
|
||||
echo 'stable=true' >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo 'stable=false' >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- id: version
|
||||
if: github.event_name == 'push' && steps.channel.outputs.stable != 'true'
|
||||
uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
|
||||
with:
|
||||
package: archivebox
|
||||
branch: ${{ github.ref_name }}
|
||||
tag-prefix: v
|
||||
version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}
|
||||
@@ -0,0 +1,51 @@
|
||||
name: Release candidate
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [dev]
|
||||
|
||||
permissions:
|
||||
actions: read
|
||||
contents: write
|
||||
packages: write
|
||||
|
||||
concurrency:
|
||||
group: release-candidate-dev
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
runs-on: ubuntu-24.04
|
||||
outputs:
|
||||
action: ${{ steps.version.outputs.action }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
submodules: true
|
||||
token: ${{ secrets.RELEASE_GH_TOKEN }}
|
||||
|
||||
- id: version
|
||||
uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
|
||||
with:
|
||||
package: archivebox
|
||||
branch: dev
|
||||
tag-prefix: v
|
||||
version-scheme: rc
|
||||
|
||||
python-artifacts:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
|
||||
uses: ./.github/workflows/pip.yml
|
||||
with:
|
||||
full_tests: false
|
||||
secrets: inherit
|
||||
|
||||
docker-digests:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
|
||||
uses: ./.github/workflows/docker.yml
|
||||
with:
|
||||
push_digests: true
|
||||
full_tests: false
|
||||
secrets: inherit
|
||||
+19
-272
@@ -2,56 +2,33 @@ name: Release
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: [CI]
|
||||
workflows: [CI, Release candidate]
|
||||
types: [completed]
|
||||
branches: [dev, main]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_branch:
|
||||
description: Branch being promoted
|
||||
required: true
|
||||
type: choice
|
||||
options: [main, dev]
|
||||
release_sha:
|
||||
description: Exact tested commit to release
|
||||
required: true
|
||||
type: string
|
||||
ci_run_id:
|
||||
description: Successful CI run containing the tested Python artifacts
|
||||
required: true
|
||||
type: string
|
||||
docker_digest_run_id:
|
||||
description: Manual Docker digest build run containing the tested image digests
|
||||
required: true
|
||||
type: string
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
group: release-${{ github.repository }}-${{ github.event_name == 'workflow_dispatch' && inputs.release_branch || github.event.workflow_run.head_branch }}
|
||||
group: release-${{ github.repository }}-${{ github.event.workflow_run.head_branch }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
DOCKERHUB_IMAGE: archivebox/archivebox
|
||||
GHCR_IMAGE: ghcr.io/archivebox/archivebox
|
||||
RELEASE_BRANCH: ${{ github.event_name == 'workflow_dispatch' && inputs.release_branch || github.event.workflow_run.head_branch }}
|
||||
RELEASE_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.release_sha || github.event.workflow_run.head_sha }}
|
||||
RELEASE_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
RELEASE_SHA: ${{ github.event.workflow_run.head_sha }}
|
||||
CI_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
|
||||
jobs:
|
||||
candidate:
|
||||
if: >-
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
(github.event.workflow_run.head_branch == 'dev' || github.event.workflow_run.head_branch == 'main') ||
|
||||
(github.event_name == 'workflow_dispatch' && inputs.release_branch == 'main')
|
||||
((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
|
||||
(github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
actions: read
|
||||
contents: read
|
||||
outputs:
|
||||
ready: ${{ steps.verified.outputs.ready }}
|
||||
artifact_run_id: ${{ steps.verified.outputs.artifact_run_id }}
|
||||
digest_run_id: ${{ steps.verified.outputs.digest_run_id }}
|
||||
ready: ${{ steps.owner.outputs.ready }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
@@ -60,84 +37,23 @@ jobs:
|
||||
- id: owner
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
BRANCH_SHA="$(git ls-remote origin "refs/heads/$RELEASE_BRANCH" | cut -f1)"
|
||||
if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch && "$BRANCH_SHA" != "$RELEASE_SHA" ]]; then
|
||||
echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
|
||||
if [[ "$RELEASE_BRANCH" == main ]]; then
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
|
||||
echo "Refusing to publish non-stable version $VERSION as a main release" >&2
|
||||
exit 1
|
||||
}
|
||||
git fetch --no-tags origin main
|
||||
git merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
|
||||
echo "Refusing to publish $RELEASE_SHA: it is not on main" >&2
|
||||
exit 1
|
||||
}
|
||||
TAG_TARGET="$(git ls-remote origin "refs/tags/v$VERSION" | cut -f1)"
|
||||
if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch && -n "$TAG_TARGET" && "$TAG_TARGET" != "$RELEASE_SHA" ]]; then
|
||||
echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo 'ready=true' >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
# A stable release fast-forwards dev to main. Only main publishes it;
|
||||
# the resulting dev workflow is not a new release candidate.
|
||||
if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]]; then
|
||||
echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
|
||||
TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
|
||||
[[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Require successful full CI for this commit
|
||||
id: verified
|
||||
if: steps.owner.outputs.ready == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
MANUAL_CI_RUN_ID: ${{ inputs.ci_run_id }}
|
||||
MANUAL_DIGEST_RUN_ID: ${{ inputs.docker_digest_run_id }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
if [[ "$GITHUB_EVENT_NAME" == workflow_dispatch ]]; then
|
||||
CI_RUN_ID="$MANUAL_CI_RUN_ID"
|
||||
DIGEST_RUN_ID="$MANUAL_DIGEST_RUN_ID"
|
||||
CI_STATUS="$(gh run view "$CI_RUN_ID" --repo "$GITHUB_REPOSITORY" --json status,conclusion,headSha,event,name \
|
||||
--jq '[.status, .conclusion, .headSha, .event, .name] | @tsv')"
|
||||
[[ "$CI_STATUS" == "$(printf 'completed\tsuccess\t%s\tpush\tCI' "$RELEASE_SHA")" ]]
|
||||
else
|
||||
CI_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml \
|
||||
--commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
|
||||
--json databaseId,status,conclusion \
|
||||
--jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
|
||||
[[ -n "$CI_RUN_ID" ]] || exit 0
|
||||
DIGEST_RUN_ID="$CI_RUN_ID"
|
||||
fi
|
||||
ARTIFACT_RUN_ID="$CI_RUN_ID"
|
||||
{
|
||||
echo 'ready=true'
|
||||
echo "artifact_run_id=$ARTIFACT_RUN_ID"
|
||||
echo "digest_run_id=$DIGEST_RUN_ID"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
python-release:
|
||||
name: Publish tested Python artifacts
|
||||
needs: candidate
|
||||
if: needs.candidate.outputs.ready == 'true'
|
||||
runs-on: ubuntu-24.04
|
||||
env:
|
||||
CI_RUN_ID: ${{ needs.candidate.outputs.artifact_run_id }}
|
||||
DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
|
||||
environment: pypi
|
||||
permissions:
|
||||
actions: read
|
||||
contents: write
|
||||
deployments: read
|
||||
id-token: write
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.version }}
|
||||
@@ -161,26 +77,6 @@ jobs:
|
||||
version: "0.11.3"
|
||||
enable-cache: false
|
||||
|
||||
# This precedes every stable upload, including workflow_dispatch. A green
|
||||
# build is not evidence that our automatically updated test servers ran it.
|
||||
- name: Download tested Docker base-image metadata
|
||||
if: env.RELEASE_BRANCH == 'main'
|
||||
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
|
||||
with:
|
||||
path: ${{ runner.temp }}/tested-docker-artifacts
|
||||
pattern: digest-*
|
||||
merge-multiple: true
|
||||
github-token: ${{ github.token }}
|
||||
repository: ${{ github.repository }}
|
||||
run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
|
||||
|
||||
- name: Require matching acceptance on Cabbage
|
||||
if: env.RELEASE_BRANCH == 'main'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
STAGING_DOCKER_ARTIFACT_DIR: ${{ runner.temp }}/tested-docker-artifacts
|
||||
run: uv run --no-project python bin/verify-staging-release.py
|
||||
|
||||
- name: Resolve release binaries through abxpkg
|
||||
env:
|
||||
ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
|
||||
@@ -229,10 +125,6 @@ jobs:
|
||||
VERSION='${{ steps.version.outputs.version }}'
|
||||
case "$RELEASE_BRANCH" in
|
||||
main)
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
|
||||
echo "Refusing to publish non-stable version $VERSION from main" >&2
|
||||
exit 1
|
||||
}
|
||||
;;
|
||||
dev)
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] || {
|
||||
@@ -270,92 +162,20 @@ jobs:
|
||||
docker-release:
|
||||
name: Publish tested multiarch image
|
||||
if: >-
|
||||
(github.event_name == 'workflow_dispatch') ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push')
|
||||
needs: [candidate, python-release]
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push'
|
||||
needs: python-release
|
||||
runs-on: ubuntu-24.04
|
||||
env:
|
||||
DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
|
||||
permissions:
|
||||
actions: read
|
||||
contents: write
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout the tested commit
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
fetch-depth: 1
|
||||
ref: ${{ env.RELEASE_SHA }}
|
||||
token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
|
||||
|
||||
- name: Reconcile dev to the tested stable release
|
||||
if: env.RELEASE_BRANCH == 'main' && needs.python-release.outputs.release_ready == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
git fetch --no-tags origin main dev
|
||||
MAIN_SHA="$(git rev-parse origin/main)"
|
||||
DEV_SHA="$(git rev-parse origin/dev)"
|
||||
if [[ "$MAIN_SHA" != "$RELEASE_SHA" || "$DEV_SHA" == "$RELEASE_SHA" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
push_dev() {
|
||||
if ! git push origin "$1:refs/heads/dev"; then
|
||||
CURRENT_DEV="$(git ls-remote origin refs/heads/dev | cut -f1)"
|
||||
if [[ "$CURRENT_DEV" == "$DEV_SHA" ]]; then
|
||||
echo "Failed to push dev although it still points to $DEV_SHA" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Leaving dev at $CURRENT_DEV: it advanced during release reconciliation"
|
||||
fi
|
||||
}
|
||||
if git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
|
||||
push_dev "$RELEASE_SHA"
|
||||
exit 0
|
||||
else
|
||||
STATUS=$?
|
||||
[[ "$STATUS" -eq 1 ]] || exit "$STATUS"
|
||||
fi
|
||||
BASE_SHA="$(git merge-base "$MAIN_SHA" "$DEV_SHA")"
|
||||
if git diff --quiet "$BASE_SHA" "$DEV_SHA" -- . ':!pyproject.toml' ':!uv.lock' ':!etc/package.json'; then
|
||||
:
|
||||
else
|
||||
STATUS=$?
|
||||
[[ "$STATUS" -eq 1 ]] || exit "$STATUS"
|
||||
echo "Leaving dev at $DEV_SHA: it contains changes beyond release metadata"
|
||||
exit 0
|
||||
fi
|
||||
normalized_metadata() {
|
||||
case "$2" in
|
||||
pyproject.toml)
|
||||
git show "$1:$2" | awk '
|
||||
/^\[/ {section = $0}
|
||||
section == "[project]" && /^version = "[^"]+"$/ {print "version = \"__VERSION__\""; next}
|
||||
section == "[tool.bumpver]" && /^current_version = "[^"]+"$/ {print "current_version = \"__VERSION__\""; next}
|
||||
{print}
|
||||
' ;;
|
||||
uv.lock)
|
||||
git show "$1:$2" | awk 'previous == "name = \"archivebox\"" && /^version = / {print "version = \"__VERSION__\""; previous = $0; next} {print; previous = $0}' ;;
|
||||
etc/package.json)
|
||||
git show "$1:$2" | sed -E 's/^( "version": )"[^"]+"(,?)$/\1"__VERSION__"\2/' ;;
|
||||
esac
|
||||
}
|
||||
for path in pyproject.toml uv.lock etc/package.json; do
|
||||
BASE_METADATA="$(normalized_metadata "$BASE_SHA" "$path")"
|
||||
DEV_METADATA="$(normalized_metadata "$DEV_SHA" "$path")"
|
||||
if [[ "$BASE_METADATA" != "$DEV_METADATA" ]]; then
|
||||
echo "Leaving dev at $DEV_SHA: $path contains changes beyond release version metadata"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
# A concurrent release bot may bump only dev's own version metadata.
|
||||
# Keep that commit as a parent while preserving the exact tested stable tree.
|
||||
git switch --detach "$MAIN_SHA"
|
||||
git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
|
||||
merge -s ours --no-ff -m "Merge tested stable release into dev" "$DEV_SHA"
|
||||
MERGE_SHA="$(git rev-parse HEAD)"
|
||||
push_dev "$MERGE_SHA"
|
||||
|
||||
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
|
||||
with:
|
||||
@@ -378,14 +198,13 @@ jobs:
|
||||
--lib="$ABXPKG_LIB_DIR" \
|
||||
--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
|
||||
--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries" >/dev/null
|
||||
for name in uv gh git jq docker; do
|
||||
for name in uv git jq docker; do
|
||||
test -L "$ABXPKG_LIB_DIR/env/bin/$name"
|
||||
test -x "$ABXPKG_LIB_DIR/env/bin/$name"
|
||||
done
|
||||
{
|
||||
echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
|
||||
echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
|
||||
echo "GH_BINARY=$ABXPKG_LIB_DIR/env/bin/gh"
|
||||
echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
|
||||
echo "DOCKER_BINARY=$ABXPKG_LIB_DIR/env/bin/docker"
|
||||
} >> "$GITHUB_ENV"
|
||||
@@ -413,31 +232,16 @@ jobs:
|
||||
merge-multiple: true
|
||||
github-token: ${{ github.token }}
|
||||
repository: ${{ github.repository }}
|
||||
run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
|
||||
|
||||
- name: Verify exact successful digest build run
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
RUN_STATUS="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json status --jq '.status')"
|
||||
RUN_CONCLUSION="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json conclusion --jq '.conclusion')"
|
||||
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]]
|
||||
if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch ]]; then
|
||||
RUN_SHA="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json headSha --jq '.headSha')"
|
||||
[[ "$RUN_SHA" == "$RELEASE_SHA" ]]
|
||||
fi
|
||||
run-id: ${{ env.CI_RUN_ID }}
|
||||
|
||||
- name: Verify digest set and collect tags
|
||||
id: docker_meta
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
|
||||
[[ "${#DIGESTS[@]}" -eq 2 ]]
|
||||
for digest in "${DIGESTS[@]}"; do [[ "$digest" =~ ^[0-9a-f]{64}$ ]]; done
|
||||
[[ "$(< /tmp/digests/source-ref)" == "$RELEASE_SHA" ]]
|
||||
|
||||
VERSION='${{ needs.python-release.outputs.version }}'
|
||||
SHORT_SHA="${RELEASE_SHA::12}"
|
||||
@@ -447,46 +251,12 @@ jobs:
|
||||
TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}")"
|
||||
TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
|
||||
fi
|
||||
if [[ "$RELEASE_BRANCH" == main ]]; then
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
|
||||
echo "Refusing to tag :latest with prerelease $VERSION" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$TAG_TARGET" == "$RELEASE_SHA" ]] || {
|
||||
echo "Refusing to tag :latest: v$VERSION does not point to $RELEASE_SHA" >&2
|
||||
exit 1
|
||||
}
|
||||
$GIT_BINARY fetch --no-tags origin main
|
||||
$GIT_BINARY merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
|
||||
echo "Refusing to tag :latest: $RELEASE_SHA is not on main" >&2
|
||||
exit 1
|
||||
}
|
||||
MAIN_VERSION="$($GIT_BINARY show origin/main:pyproject.toml | sed -nE 's/^version = "([^"]+)".*/\1/p' | head -n 1)"
|
||||
[[ "$MAIN_VERSION" == "$VERSION" ]] || {
|
||||
echo "Refusing to tag :latest: main is now $MAIN_VERSION, not $VERSION" >&2
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
SYNC_DEV=false
|
||||
if [[ "$RELEASE_BRANCH" == main ]]; then
|
||||
$GIT_BINARY fetch --no-tags origin main dev
|
||||
MAIN_TARGET="$($GIT_BINARY rev-parse origin/main)"
|
||||
DEV_TARGET="$($GIT_BINARY rev-parse origin/dev)"
|
||||
# Recovery can add CI/docs fixes after the immutable release tag.
|
||||
# Both branches and every shipped application file must agree.
|
||||
if $GIT_BINARY merge-base --is-ancestor "$MAIN_TARGET" "$DEV_TARGET" &&
|
||||
$GIT_BINARY diff --quiet "$MAIN_TARGET" "$DEV_TARGET" &&
|
||||
$GIT_BINARY diff --quiet "$RELEASE_SHA" "$MAIN_TARGET" -- . ':!.github' ':!README.md'; then
|
||||
SYNC_DEV=true
|
||||
fi
|
||||
fi
|
||||
{
|
||||
echo 'dockerhub_tags<<EOF'
|
||||
if [[ "$RELEASE_BRANCH" == dev ]]; then
|
||||
echo "${DOCKERHUB_IMAGE}:dev"
|
||||
elif [[ "$RELEASE_BRANCH" == main ]]; then
|
||||
echo "${DOCKERHUB_IMAGE}:latest"
|
||||
[[ "$SYNC_DEV" == true ]] && echo "${DOCKERHUB_IMAGE}:dev"
|
||||
fi
|
||||
echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
|
||||
[[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${DOCKERHUB_IMAGE}:${VERSION}"
|
||||
@@ -496,7 +266,6 @@ jobs:
|
||||
echo "${GHCR_IMAGE}:dev"
|
||||
elif [[ "$RELEASE_BRANCH" == main ]]; then
|
||||
echo "${GHCR_IMAGE}:latest"
|
||||
[[ "$SYNC_DEV" == true ]] && echo "${GHCR_IMAGE}:dev"
|
||||
fi
|
||||
echo "${GHCR_IMAGE}:sha-${SHORT_SHA}"
|
||||
[[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${GHCR_IMAGE}:${VERSION}"
|
||||
@@ -507,7 +276,7 @@ jobs:
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
|
||||
TAG_ARGS=()
|
||||
while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.dockerhub_tags }}'
|
||||
REFS=()
|
||||
@@ -518,7 +287,7 @@ jobs:
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
|
||||
mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
|
||||
TAG_ARGS=()
|
||||
while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.ghcr_tags }}'
|
||||
REFS=()
|
||||
@@ -537,26 +306,6 @@ jobs:
|
||||
test "$INSTALLED_VERSION" = "$VERSION"
|
||||
done
|
||||
|
||||
- name: Record published image source
|
||||
shell: bash
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
# workflow_run can report the default branch's head even when this
|
||||
# run published a different branch/SHA. Preserve the verified target.
|
||||
[[ "$RELEASE_BRANCH" == dev || "$RELEASE_BRANCH" == main ]]
|
||||
[[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
|
||||
mkdir -p "$RUNNER_TEMP/published-image-source"
|
||||
# shellcheck disable=SC2016
|
||||
"$JQ_BINARY" -n --arg branch "$RELEASE_BRANCH" --arg sha "$RELEASE_SHA" \
|
||||
'{branch: $branch, sha: $sha}' > "$RUNNER_TEMP/published-image-source/source.json"
|
||||
|
||||
- name: Upload published image source
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
||||
with:
|
||||
name: published-image-source
|
||||
path: ${{ runner.temp }}/published-image-source/source.json
|
||||
if-no-files-found: error
|
||||
|
||||
- name: Update Docker Hub README
|
||||
uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
|
||||
with:
|
||||
@@ -568,7 +317,6 @@ jobs:
|
||||
name: Cascade verified release
|
||||
if: needs.python-release.outputs.release_ready == 'true'
|
||||
needs:
|
||||
- candidate
|
||||
- python-release
|
||||
- docker-release
|
||||
permissions:
|
||||
@@ -579,6 +327,5 @@ jobs:
|
||||
upstream_package: archivebox
|
||||
upstream_version: ${{ needs.python-release.outputs.version }}
|
||||
upstream_artifact_name: python-distributions
|
||||
upstream_run_id: ${{ needs.candidate.outputs.artifact_run_id }}
|
||||
secrets:
|
||||
release_token: ${{ secrets.RELEASE_GH_TOKEN }}
|
||||
@@ -1,80 +0,0 @@
|
||||
name: Capture publicsite screenshots
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- dev
|
||||
paths:
|
||||
- archivebox/**
|
||||
- bin/collect_ui_screenshots.*
|
||||
- bin/generate_ui_screenshot_gallery.py
|
||||
- bin/take_screenshot.js
|
||||
- docs/Screenshots.md
|
||||
- pyproject.toml
|
||||
- uv.lock
|
||||
- .github/workflows/screenshots.yml
|
||||
- .github/workflows/prepare-release.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.event_name == 'workflow_dispatch' && format('publicsite-captures-manual-{0}', github.run_id) || 'publicsite-captures' }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
permissions:
|
||||
contents: write
|
||||
uses: ./.github/workflows/prepare-release.yml
|
||||
secrets: inherit
|
||||
|
||||
deploy:
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.run_jobs == 'true'
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Setup Python and uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
|
||||
- name: Install ArchiveBox and screenshot dependencies
|
||||
run: uv sync --no-cache --frozen
|
||||
|
||||
- name: Build UI screenshot gallery
|
||||
run: bin/collect_ui_screenshots.sh
|
||||
env:
|
||||
BASE_URL: http://archivebox.localhost:5797
|
||||
|
||||
- name: Verify gallery was generated from this exact revision
|
||||
run: |
|
||||
uv run --no-cache --no-sync python - <<'PY'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
|
||||
version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
|
||||
assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
|
||||
assert provenance["version"] == version, provenance
|
||||
assert provenance["capture_count"] >= 3 * 35, provenance
|
||||
assert len(provenance["files"]) == provenance["capture_count"], provenance
|
||||
assert all(not re.match(r"^\d+-", name) for name in provenance["files"]), provenance
|
||||
PY
|
||||
|
||||
- name: Save validated screenshots
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: site-screenshots
|
||||
path: publicsite/screenshots
|
||||
if-no-files-found: error
|
||||
retention-days: 90
|
||||
@@ -1,109 +0,0 @@
|
||||
name: Staging acceptance
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: [Release]
|
||||
types: [completed]
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
group: staging-acceptance
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
published:
|
||||
if: github.event.workflow_run.conclusion == 'success'
|
||||
runs-on: ubuntu-24.04
|
||||
permissions:
|
||||
actions: read
|
||||
outputs:
|
||||
ready: ${{ steps.source.outputs.ready }}
|
||||
sha: ${{ steps.source.outputs.sha }}
|
||||
steps:
|
||||
- id: image
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
RELEASE_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
published=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RELEASE_RUN_ID/jobs?per_page=100" --jq 'any(.jobs[]; .name == "Publish tested multiarch image" and .conclusion == "success")')
|
||||
echo "published=$published" >> "$GITHUB_OUTPUT"
|
||||
- name: Download published image source
|
||||
if: steps.image.outputs.published == 'true'
|
||||
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
|
||||
with:
|
||||
name: published-image-source
|
||||
path: ${{ runner.temp }}/published-image-source
|
||||
github-token: ${{ github.token }}
|
||||
repository: ${{ github.repository }}
|
||||
run-id: ${{ github.event.workflow_run.id }}
|
||||
- name: Select published dev image
|
||||
id: source
|
||||
if: steps.image.outputs.published == 'true'
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
# The triggering run may report the default branch as its head;
|
||||
# this artifact names the image source verified by the publisher.
|
||||
source="$RUNNER_TEMP/published-image-source/source.json"
|
||||
jq -e 'type == "object" and (keys | sort) == ["branch", "sha"]' "$source" >/dev/null
|
||||
branch=$(jq -er '.branch | select(type == "string")' "$source")
|
||||
sha=$(jq -er '.sha | select(type == "string")' "$source")
|
||||
[[ "$branch" == dev || "$branch" == main ]]
|
||||
[[ "$sha" =~ ^[0-9a-f]{40}$ ]]
|
||||
echo "sha=$sha" >> "$GITHUB_OUTPUT"
|
||||
if [[ "$branch" == dev ]]; then
|
||||
echo 'ready=true' >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo 'ready=false' >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
acceptance:
|
||||
# The privileged key is never exposed to PRs or arbitrary dispatch refs.
|
||||
needs: published
|
||||
if: needs.published.outputs.ready == 'true'
|
||||
runs-on: [self-hosted, Linux, X64, ugnas-linux]
|
||||
environment: staging
|
||||
timeout-minutes: 110
|
||||
permissions:
|
||||
contents: read
|
||||
deployments: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
target: [cabbage]
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
|
||||
with:
|
||||
ref: ${{ needs.published.outputs.sha }}
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e
|
||||
with:
|
||||
version: '0.11.3'
|
||||
- name: Prepare browser and restricted staging identity
|
||||
env:
|
||||
STAGING_SSH_KEY: ${{ secrets.STAGING_SSH_KEY }}
|
||||
STAGING_KNOWN_HOSTS: ${{ secrets.STAGING_KNOWN_HOSTS }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
printf '%s\n' "$STAGING_SSH_KEY" > "$RUNNER_TEMP/staging-key"
|
||||
printf '%s\n' "$STAGING_KNOWN_HOSTS" > "$RUNNER_TEMP/staging-known-hosts"
|
||||
uv run --no-project --with playwright==1.63.0 playwright install chromium
|
||||
- name: Verify actual deployment, full captures and rendered snapshots
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
STAGING_TARGET: ${{ matrix.target }}
|
||||
STAGING_SHA: ${{ needs.published.outputs.sha }}
|
||||
STAGING_KEY_FILE: ${{ runner.temp }}/staging-key
|
||||
STAGING_KNOWN_HOSTS_FILE: ${{ runner.temp }}/staging-known-hosts
|
||||
run: uv run --no-project --with playwright==1.63.0 python bin/check-staging-deployment.py
|
||||
- name: Remove staging identity
|
||||
if: always()
|
||||
run: rm -f "$RUNNER_TEMP/staging-key" "$RUNNER_TEMP/staging-known-hosts"
|
||||
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
|
||||
if: always()
|
||||
with:
|
||||
name: staging-${{ matrix.target }}-${{ needs.published.outputs.sha }}
|
||||
path: staging-evidence/
|
||||
if-no-files-found: error
|
||||
@@ -1,20 +1,7 @@
|
||||
name: Parallel Tests
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
test_file:
|
||||
description: Run one discovered test file
|
||||
required: true
|
||||
default: archivebox/tests/test_cli_list.py
|
||||
type: string
|
||||
workflow_call:
|
||||
inputs:
|
||||
test_file:
|
||||
description: Run one discovered test file; empty runs the complete matrix
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
|
||||
env:
|
||||
PYTHONIOENCODING: utf-8
|
||||
@@ -26,27 +13,11 @@ env:
|
||||
jobs:
|
||||
discover-tests:
|
||||
name: Discover every test
|
||||
# Selection must run on GitHub so an offline NAS cannot block the fallback.
|
||||
runs-on: ubuntu-24.04
|
||||
outputs:
|
||||
tests: ${{ steps.discover.outputs.tests }}
|
||||
runner: ${{ steps.runner.outputs.use-runner || '["ubuntu-24.04"]' }}
|
||||
|
||||
steps:
|
||||
- name: Select available CI capacity
|
||||
id: runner
|
||||
if: vars.UGNAS_CI_ENABLED == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main')
|
||||
uses: mikehardy/runner-fallback-action@dc7732f9e532c451b2527b288ba8d58fd4b0a4ca # v1.1.0
|
||||
with:
|
||||
github-token: ${{ secrets.RUNNER_STATUS_TOKEN }}
|
||||
organization: ArchiveBox
|
||||
primary-runner: self-hosted,ugnas-linux
|
||||
fallback-runner: ubuntu-24.04
|
||||
# Jobs queued for this pool must not be mistaken for idle runner slots:
|
||||
# requiring MAX_JOBS idle runners prevents assigning a second wave.
|
||||
primaries-required: ${{ vars.UGNAS_CI_MIN_IDLE || 1 }}
|
||||
fallback-on-error: true
|
||||
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -87,16 +58,10 @@ jobs:
|
||||
|
||||
- name: Discover every test exactly once
|
||||
id: discover
|
||||
env:
|
||||
TEST_FILE: ${{ inputs.test_file }}
|
||||
UGNAS_CI_MAX_JOBS: ${{ vars.UGNAS_CI_MAX_JOBS || 3 }}
|
||||
run: |
|
||||
set -Eeuo pipefail
|
||||
mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
|
||||
test "${#output[@]}" -eq 2
|
||||
if [[ -n "$TEST_FILE" ]]; then
|
||||
output[1]="$("$JQ_BINARY" -ce '[.[] | select(.paths == [env.TEST_FILE])] | select(length == 1)' <<< "${output[1]}")"
|
||||
fi
|
||||
echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
|
||||
echo "${output[0]}"
|
||||
"$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
|
||||
@@ -104,13 +69,13 @@ jobs:
|
||||
tests:
|
||||
name: ${{ matrix.test.name }}
|
||||
needs: discover-tests
|
||||
# Opt in trusted branches only; PRs always use GitHub-hosted runners.
|
||||
# Ordinary Linux tests are eligible; # ci-runner: hosted declares exceptions.
|
||||
# The hosted discovery job falls back when workers are offline/busy or the
|
||||
# availability API fails. UGNAS_CI_ENABLED=false disables the pool entirely.
|
||||
runs-on: ${{ fromJSON(matrix.test.ugnas && needs.discover-tests.outputs.runner || '["ubuntu-24.04"]') }}
|
||||
runs-on: ubuntu-24.04
|
||||
env:
|
||||
ABXPKG_LIB_DIR: /tmp/archivebox-lib
|
||||
CHROME_HEADLESS: "true"
|
||||
DATA_DIR: /tmp/archivebox-test-data
|
||||
PERSONAS_DIR: /tmp/abx-personas
|
||||
CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -118,15 +83,6 @@ jobs:
|
||||
test: ${{ fromJson(needs.discover-tests.outputs.tests) }}
|
||||
|
||||
steps:
|
||||
- name: Set job-local runtime paths
|
||||
run: |
|
||||
{
|
||||
echo "ABXPKG_LIB_DIR=$RUNNER_TEMP/archivebox-lib"
|
||||
echo "DATA_DIR=$RUNNER_TEMP/archivebox-test-data"
|
||||
echo "PERSONAS_DIR=$RUNNER_TEMP/abx-personas"
|
||||
echo "CHROME_USER_DATA_DIR=$RUNNER_TEMP/abx-personas/Default/chrome_profile"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -176,9 +132,6 @@ jobs:
|
||||
extra_args=(--extra "$TEST_EXTRA")
|
||||
fi
|
||||
uv sync --no-cache --locked --dev "${extra_args[@]}"
|
||||
- name: Install Playwright Chromium for browser tests
|
||||
if: contains(matrix.test.paths_arg, 'test_proof_previews') || contains(matrix.test.paths_arg, 'test_snapshot_output_stacks') || contains(matrix.test.paths_arg, 'test_ui_admin_snapshot') || contains(matrix.test.paths_arg, 'test_cli_extract')
|
||||
run: uv run --no-cache --no-sync --no-sources playwright install chromium
|
||||
|
||||
- name: Run ${{ matrix.test.name }}
|
||||
env:
|
||||
@@ -189,35 +142,13 @@ jobs:
|
||||
for test_path in "${test_paths[@]}"; do
|
||||
test -f "$test_path"
|
||||
done
|
||||
test_status=0
|
||||
for test_path in "${test_paths[@]}"; do
|
||||
test_name="$(basename "$test_path" .py)"
|
||||
test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
|
||||
export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
|
||||
export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
|
||||
export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
|
||||
if (( ${#test_paths[@]} > 1 )); then
|
||||
export ABXPKG_LIB_DIR="$RUNNER_TEMP/archivebox-lib/$test_slug"
|
||||
fi
|
||||
mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
|
||||
uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" --durations=20 \
|
||||
--basetemp="$RUNNER_TEMP/pytest/$test_slug" \
|
||||
--junitxml="$RUNNER_TEMP/test-results/$test_slug.xml" || test_status=1
|
||||
uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
|
||||
--basetemp="$RUNNER_TEMP/pytest/$test_slug"
|
||||
done
|
||||
exit "$test_status"
|
||||
- name: Save test results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
||||
with:
|
||||
name: test-results-${{ strategy.job-index }}
|
||||
path: ${{ runner.temp }}/test-results/*.xml
|
||||
if-no-files-found: error
|
||||
retention-days: 14
|
||||
- name: Save failed runtime diagnostics
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
||||
with:
|
||||
name: runtime-logs-${{ strategy.job-index }}
|
||||
path: ${{ runner.temp }}/pytest/**/*.log
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
@@ -68,5 +68,3 @@ ArchiveBox.conf
|
||||
|
||||
# Local Claude Code task locks (not project state)
|
||||
.claude/scheduled_tasks.lock
|
||||
|
||||
/_site/
|
||||
@@ -56,8 +56,6 @@ repos:
|
||||
- id: mixed-line-ending
|
||||
- id: fix-byte-order-marker
|
||||
- id: end-of-file-fixer
|
||||
# Preserve the exact bytes of captured cryptographic proof fixtures.
|
||||
exclude: ^archivebox/tests/fixtures/legacy_proofs/.*\.json$
|
||||
- id: detect-private-key
|
||||
- id: debug-statements
|
||||
- id: forbid-submodules
|
||||
|
||||
@@ -77,7 +77,11 @@ archivebox run
|
||||
|
||||
## Verification
|
||||
|
||||
Run the test suite with `uv run pytest archivebox/tests -x` and repository checks
|
||||
with `uv run prek run --all-files`. For focused work, select the relevant tests.
|
||||
Use targeted tests for focused work:
|
||||
|
||||
Releases are published only by `.github/workflows/release.yml` after complete CI succeeds for the exact commit. Pushes to `main` prepare stable patch releases automatically; `dev` prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
|
||||
```bash
|
||||
uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
|
||||
uv run prek run --all-files
|
||||
```
|
||||
|
||||
Releases are published only by `.github/workflows/release.yml` after the complete `dev` CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
|
||||
+62
-86
@@ -1,38 +1,8 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
|
||||
# Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
|
||||
# WHY: abx-dl owns Python, Node, Chromium, and extraction dependencies so the
|
||||
# standalone downloader stays independent of ArchiveBox's server features.
|
||||
# ArchiveBox-only dependencies (Sonic, OpenCode, supervisor, Django) belong in
|
||||
# the layers added by this Dockerfile, never in the shared abx-dl base image.
|
||||
# REQUIRED INSTALLATION CONTRACT — do not work around failures:
|
||||
# 1. IMAGE CONSTRUCTION: preinstall ALL plugins supported by runtime="archivebox"
|
||||
# and ALL their required_binaries, regardless of default enabled state.
|
||||
# Inherit the complete downloader dependencies from the tested abx-dl image;
|
||||
# install every remaining ArchiveBox-only dependency in this image's layers,
|
||||
# including OpenCode and Sonic. Never put server dependencies in abx-dl.
|
||||
# Use the normal plugin dependency pipeline: install command -> config.json
|
||||
# required_binaries -> abxpkg. Never add manual apt/pip/npm installs to hide
|
||||
# failures in that pipeline. OS/interpreter bootstrap is a separate concern,
|
||||
# not a second hand-maintained list of plugin dependencies.
|
||||
# 2. FINISHED IMAGE: archivebox install and the full plugin installation check
|
||||
# must discover everything already present and be NO-OPS offline as UID 911.
|
||||
# No downloads, package installs, sudo, or root repair may be needed to enable
|
||||
# ANY plugin or start a new crawl. Never run verification as root or manually
|
||||
# preinstall packages before it. If the inherited image lacks a downloader
|
||||
# dependency, fix/release abx-dl and consume that tested image; do not patch
|
||||
# over the missing base dependency here or repair deployed containers by SSH.
|
||||
# 3. NON-CONTAINER CI: exercise real runtime auto-installation through the same
|
||||
# CLI/required_binaries/abxpkg path users invoke. GitHub's runner has
|
||||
# passwordless sudo; abxpkg must escalate automatically when needed. Never
|
||||
# manually preinstall dependencies or wrap the tested command in sudo to make
|
||||
# a failure disappear. This is distinct from the finished-image no-op test.
|
||||
# 4. SIZE: the complete compressed image budget is 1025 MiB. Preserve ALL
|
||||
# user-facing functionality. Only mechanical/internal
|
||||
# optimizations are allowed. Never remove dependencies, disable features or
|
||||
# plugins, weaken assertions, or skip checks to meet a size limit. If the
|
||||
# complete image exceeds the configured limit, LET CI FAIL and explicitly
|
||||
# ask the maintainer what to do. Do not raise the limit unilaterally.
|
||||
# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
|
||||
# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
|
||||
# Build abx-dl first, then point this file at it:
|
||||
# docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
|
||||
# --build-context abxbus=../abxbus \
|
||||
@@ -40,10 +10,10 @@
|
||||
# --build-context abx-plugins=../abx-plugins \
|
||||
# -t archivebox/abx-dl:dev
|
||||
# docker buildx build . -f Dockerfile \
|
||||
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:dev \
|
||||
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.261 \
|
||||
# -t archivebox:multistage
|
||||
|
||||
ARG ABX_DL_IMAGE=archivebox/abx-dl:main
|
||||
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.261
|
||||
|
||||
FROM archivebox/sonic:1.4.9 AS sonic
|
||||
FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
|
||||
@@ -77,7 +47,6 @@ ENV ARCHIVEBOX_USER=archivebox \
|
||||
IN_DOCKER=True
|
||||
|
||||
ENV CODE_DIR=/app \
|
||||
ABX_RUNTIME=archivebox \
|
||||
DATA_DIR=/data \
|
||||
CONFIG_DIR=/opt/archivebox \
|
||||
ABXPKG_LIB_DIR=/opt/archivebox/lib \
|
||||
@@ -125,58 +94,87 @@ FROM archivebox-runtime-base AS archivebox-builder
|
||||
WORKDIR "$CODE_DIR"
|
||||
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
|
||||
--mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
|
||||
--mount=type=bind,source=uv.lock,target=/app/uv.lock \
|
||||
<<'EOF'
|
||||
echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
|
||||
export UV_NO_CACHE=true
|
||||
echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
|
||||
echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
|
||||
echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
|
||||
rm -f /etc/apt/apt.conf.d/docker-clean
|
||||
apt-get update -qq
|
||||
apt-get install -qq -y --no-install-recommends x11-utils
|
||||
apt-get install -qq -y --no-install-recommends \
|
||||
build-essential gcc libldap2-dev libsasl2-dev libssl-dev
|
||||
# Extend abx-dl's existing venv instead of clearing it. Clearing and later
|
||||
# copying a complete replacement over the base duplicates every venv byte in
|
||||
# the final overlay history even when most packages are unchanged.
|
||||
# Do not rewrite inherited native libraries merely to strip them again.
|
||||
/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries
|
||||
/usr/bin/uv pip install --no-cache setuptools pip wheel
|
||||
|
||||
mkdir -p /tmp/archivebox-uv-project
|
||||
/venv/bin/python - <<'PY'
|
||||
from pathlib import Path
|
||||
import json
|
||||
import re
|
||||
import urllib.request
|
||||
|
||||
source = Path("/app/pyproject.toml")
|
||||
target = Path("/tmp/archivebox-uv-project/pyproject.toml")
|
||||
text = source.read_text()
|
||||
text = text.replace(
|
||||
'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
|
||||
'environments = ["sys_platform == \'linux\'"]',
|
||||
)
|
||||
|
||||
# Docker builds need the just-published internal abx wheels immediately, but
|
||||
# PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
|
||||
# Docker-only dependency view from the version JSON so the published package
|
||||
# metadata stays normal while image builds remain resumable after a release.
|
||||
for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
|
||||
match = re.search(
|
||||
rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
|
||||
text,
|
||||
)
|
||||
if not match:
|
||||
continue
|
||||
version = match.group("version")
|
||||
with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
|
||||
data = json.load(response)
|
||||
wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
|
||||
text = re.sub(
|
||||
rf'"{re.escape(package)}(?:==|>=)[^"]+"',
|
||||
f'"{package} @ {wheel_url}"',
|
||||
text,
|
||||
count=1,
|
||||
)
|
||||
|
||||
target.write_text(text)
|
||||
PY
|
||||
|
||||
# The old Docker-only dependency rewrite predated this checkout's lockfile:
|
||||
# it fetched version JSON to avoid stale PyPI Simple listings. That extra API
|
||||
# can itself lag or return 503s. The verified lock already records exact public
|
||||
# artifact URLs and hashes, including extras and platform markers. Consume it
|
||||
# directly, retaining the freshness check instead of resolving a second graph.
|
||||
/usr/bin/uv sync \
|
||||
--project /app \
|
||||
--locked \
|
||||
--project /tmp/archivebox-uv-project \
|
||||
--no-cache \
|
||||
--no-dev \
|
||||
--inexact \
|
||||
--no-install-project \
|
||||
--no-install-workspace \
|
||||
--no-install-package abxbus \
|
||||
--no-install-package abxpkg \
|
||||
--no-install-package abx-plugins \
|
||||
--no-install-package abx-dl \
|
||||
--no-sources
|
||||
builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
|
||||
ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
|
||||
/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
|
||||
while IFS= read -r -d '' native_library; do
|
||||
if grep -Fxq "$native_library" /tmp/archivebox-inherited-libraries; then
|
||||
continue
|
||||
fi
|
||||
magic=''
|
||||
if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
|
||||
"$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
|
||||
fi
|
||||
done < /tmp/archivebox-native-libraries
|
||||
rm -f /tmp/archivebox-native-libraries /tmp/archivebox-inherited-libraries
|
||||
rm -f /tmp/archivebox-native-libraries
|
||||
rm -f /venv/bin/uv /venv/bin/uvx
|
||||
abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils
|
||||
abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
|
||||
abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
|
||||
rm -rf "$builder_abxpkg_lib_dir"
|
||||
rm -rf /venv/lib/python3.*/site-packages/pip* \
|
||||
/venv/lib/python3.*/site-packages/wheel* \
|
||||
/venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
|
||||
rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
|
||||
EOF
|
||||
|
||||
COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
|
||||
@@ -204,7 +202,7 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
|
||||
&& test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
|
||||
&& /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
|
||||
|
||||
# The builder installs and purges stripping tools in one layer, so its final
|
||||
# The builder installs and purges compilers in one layer, so its final
|
||||
# filesystem is already runtime-clean. Preserve that ancestry: starting again
|
||||
# from archivebox-runtime-base and COPYing /venv would bake a second full venv
|
||||
# over the inherited one instead of recording only ArchiveBox's package delta.
|
||||
@@ -227,8 +225,7 @@ COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
|
||||
# The builder invokes abxpkg as root for temporary ELF stripping, which can
|
||||
# rewrite derived state ownership. Restore UID 911 before resolving binaries
|
||||
# as the runtime user; the old copy-based final stage hid this dependency.
|
||||
RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777,id=archivebox-runtime-config-$TARGETARCH \
|
||||
echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
|
||||
RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
|
||||
&& printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
|
||||
&& ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
|
||||
&& ln -sf /venv/bin/daphne /usr/local/bin/daphne \
|
||||
@@ -242,33 +239,12 @@ RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777
|
||||
&& install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
|
||||
&& install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
|
||||
&& chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
|
||||
&& find "$ABXPKG_LIB_DIR" \( ! -user "$DEFAULT_ARCHIVEBOX_UID" -o ! -group "$DEFAULT_ARCHIVEBOX_GID" \) -exec chown -h "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" {} + \
|
||||
&& UV_NO_CACHE=true setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
|
||||
&& chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
|
||||
&& setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
|
||||
&& openssl rand -hex 16 > /etc/machine-id \
|
||||
&& echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
|
||||
&& echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
|
||||
|
||||
# Install the complete ArchiveBox catalog, including disabled server plugins.
|
||||
# Explicit plugin names make installation independent of enabled defaults.
|
||||
RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=archivebox-opencode-$TARGETARCH \
|
||||
chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" /var/tmp/abxpkg-cache \
|
||||
&& chmod 1777 /var/tmp/abxpkg-cache \
|
||||
&& export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
|
||||
&& env ABXPKG_TMP_CACHE_DIR=/var/tmp/abxpkg-cache XDG_CACHE_HOME=/var/tmp/abxpkg-cache \
|
||||
setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install $ABX_DOCKER_PLUGINS \
|
||||
# pnpm includes musl variants that Debian's glibc runtime cannot use.
|
||||
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
|
||||
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} + \
|
||||
# pnpm's explicit store lives under ABXPKG_LIB_DIR, regardless of XDG_CACHE_HOME.
|
||||
# Retaining its download copies made image size depend on whether the build
|
||||
# preserved hardlinks to installed files. Remove the store in this layer;
|
||||
# installed files retain their contents, and the offline check below proves
|
||||
# the complete catalog still works without the disposable package payloads.
|
||||
&& rm -rf "$ABXPKG_LIB_DIR/cache/pnpm"
|
||||
RUN --network=none export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
|
||||
&& setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
|
||||
bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install $ABX_DOCKER_PLUGINS'
|
||||
|
||||
WORKDIR "$DATA_DIR"
|
||||
RUN echo "[+] Initializing image collection..." \
|
||||
&& find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
|
||||
@@ -301,10 +277,10 @@ RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully
|
||||
|
||||
WORKDIR "$DATA_DIR"
|
||||
VOLUME "$DATA_DIR"
|
||||
EXPOSE 5797
|
||||
EXPOSE 8000
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
|
||||
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:5797/health/' | grep -q 'OK'
|
||||
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
|
||||
|
||||
ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
|
||||
CMD ["archivebox", "server", "--init", "0.0.0.0:5797"]
|
||||
CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]
|
||||
@@ -73,15 +73,15 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
|
||||
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
|
||||
docker compose pull
|
||||
docker compose up -d --wait # initializes new collections automatically
|
||||
# open http://admin.archivebox.localhost:5797 to finish setup
|
||||
# open http://admin.archivebox.localhost:8000 to finish setup
|
||||
# docker compose run --rm archivebox add 'https://example.com'
|
||||
# docker compose run --rm archivebox help
|
||||
<br/>
|
||||
<br/>
|
||||
# Option B: Or use it as a plain Docker container:
|
||||
mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
|
||||
# open http://admin.archivebox.localhost:5797 to finish setup
|
||||
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
|
||||
# open http://admin.archivebox.localhost:8000 to finish setup
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
|
||||
# docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
<br/>
|
||||
@@ -93,23 +93,24 @@ archivebox init
|
||||
archivebox install
|
||||
# archivebox add 'https://example.com'
|
||||
# archivebox help
|
||||
# archivebox server 0.0.0.0:5797
|
||||
# archivebox server 0.0.0.0:8000
|
||||
<br/>
|
||||
<br/>
|
||||
# Option D: Or use the uv install shortcut for Option C
|
||||
curl -fsSL 'https://get.archivebox.io' | bash
|
||||
</code></pre>
|
||||
<br/>
|
||||
<sub>Open <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI and <a href="http://admin.archivebox.localhost:5797"><code>http://admin.archivebox.localhost:5797</code></a> for the admin UI ➡️</sub><br/>
|
||||
<sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
|
||||
<sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
|
||||
</details>
|
||||
<br/>
|
||||
|
||||
|
||||
<div align="center" style="text-align: center">
|
||||
<br/><br/>
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
|
||||
<br/><br/>
|
||||
<small><a href="https://demo.archivebox.io">Demo</a> | <a href="https://archivebox.io/screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
|
||||
<small><a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
|
||||
<br/>
|
||||
<sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
|
||||
<br/><br/>
|
||||
@@ -128,10 +129,10 @@ curl -fsSL 'https://get.archivebox.io' | bash
|
||||
- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
|
||||
- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
|
||||
- [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
|
||||
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
|
||||
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
|
||||
- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
|
||||
- Support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
|
||||
- Support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [infiniscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
|
||||
- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
|
||||
- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
|
||||
|
||||
<br/>
|
||||
|
||||
@@ -150,17 +151,6 @@ curl -fsSL 'https://get.archivebox.io' | bash
|
||||
|
||||
#### ✳️ Easy Setup
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
|
||||
<br/>
|
||||
<ol>
|
||||
<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
|
||||
<li>Set up an admin account and generate an API Key.</li>
|
||||
<li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
|
||||
</ol>
|
||||
<br/>
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b> (macOS/Linux) <b>👈 recommended</b> <i>(click to expand)</i></summary>
|
||||
<br/>
|
||||
@@ -176,7 +166,7 @@ docker compose pull
|
||||
<li>Start the server, which initializes a new collection automatically.
|
||||
<pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
|
||||
</code></pre></li>
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# run CLI commands inside the server container started above
|
||||
docker compose exec archivebox archivebox add 'https://example.com'
|
||||
@@ -197,10 +187,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
|
||||
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
|
||||
docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
|
||||
docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# docker exec archivebox archivebox [subcommand] [--help]
|
||||
@@ -255,8 +245,8 @@ archivebox init # initialize a new collection
|
||||
archivebox install # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -291,8 +281,8 @@ archivebox add 'https://example.com'
|
||||
</code></pre>
|
||||
<br/>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -324,8 +314,8 @@ archivebox init
|
||||
archivebox install
|
||||
</code></pre>
|
||||
</li>
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
|
||||
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
|
||||
# completely optional, CLI can always be used without running a server
|
||||
# archivebox [subcommand] [--help]
|
||||
archivebox help
|
||||
@@ -364,12 +354,17 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
|
||||
<br/>
|
||||
<ol>
|
||||
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
|
||||
<li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
|
||||
<li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
|
||||
<li>Download a binary release for your OS or build the native app from source<br/>
|
||||
<ul>
|
||||
<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
|
||||
<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
|
||||
<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ol>
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
|
||||
<br/>
|
||||
<a href="https://electron.archivebox.io/">Desktop app guide</a> · <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
|
||||
<i>✨ Alpha (contributors wanted!)</i>: for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
|
||||
<br/>
|
||||
</details>
|
||||
|
||||
@@ -380,7 +375,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
|
||||
> *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*
|
||||
|
||||
<ul>
|
||||
<li><a href="https://apps.truenas.com/catalog/archivebox_community/">TrueNAS</a></li>
|
||||
<li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
|
||||
<li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
|
||||
<li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
|
||||
<li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
|
||||
@@ -431,7 +426,7 @@ For more discussion on third-party hosting options see here: <a href="https://gi
|
||||
#### ➡️ Next Steps
|
||||
|
||||
- Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
|
||||
- (Optional) [Import browser cookies and settings into a persona](docs/Chromium-Install.md#import-an-existing-browser-profile).
|
||||
- (Optional) Create a persona and import browser cookies to archive logged-in sites: `archivebox persona create --import=chrome personal`
|
||||
- Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
|
||||
- Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
|
||||
- Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
|
||||
@@ -459,6 +454,11 @@ docker compose run --rm archivebox help
|
||||
# equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
|
||||
docker run -it -v $PWD:/data archivebox/archivebox:dev help
|
||||
|
||||
# optional: import your browser cookies into a persona for logged-in archiving
|
||||
archivebox persona create --import=chrome personal
|
||||
# supported: chrome/chromium/brave/edge (Chromium-based only)
|
||||
# use --profile to target a specific profile (e.g. Default, Profile 1)
|
||||
# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
|
||||
```
|
||||
|
||||
#### ArchiveBox Subcommands
|
||||
@@ -538,16 +538,16 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html' # inspect snapshot da
|
||||
<summary><b>🖥 Web UI & API Usage</b></summary>
|
||||
<pre lang="bash"><code style="white-space: pre-line">
|
||||
# Start the server on bare metal (uv/apt/brew):
|
||||
archivebox server 0.0.0.0:5797 # start the server
|
||||
archivebox server 0.0.0.0:8000 # start the server
|
||||
<br/>
|
||||
# Or with Docker Compose:
|
||||
docker compose up # start the server
|
||||
<br/>
|
||||
# Or with a Docker container:
|
||||
docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
|
||||
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
|
||||
</code></pre>
|
||||
|
||||
<sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI. ➡️</sup><br/>
|
||||
<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
|
||||
<sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
|
||||
<sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
|
||||
<br/><br/>
|
||||
@@ -564,22 +564,6 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
|
||||
|
||||
# restart the server to apply any config changes
|
||||
</code></pre>
|
||||
|
||||
Snapshot caching follows each snapshot's saved `PERMISSIONS`, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses `Cache-Control: private` so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.
|
||||
</details>
|
||||
|
||||
<br/>
|
||||
|
||||
<details>
|
||||
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧 Desktop / Mobile / Extension Usage</b></summary>
|
||||
<br/>
|
||||
|
||||
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
|
||||
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
|
||||
- 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
|
||||
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
|
||||
|
||||
<br/>
|
||||
</details>
|
||||
|
||||
<br/>
|
||||
@@ -635,10 +619,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
|
||||
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>
|
||||
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
|
||||
|
||||
- <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://app.archivebox.io/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
|
||||
<i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
|
||||
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
|
||||
<i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
|
||||
@@ -951,7 +932,7 @@ archivebox manage createsuperuser
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
|
||||
</ul>
|
||||
|
||||
@@ -1004,11 +985,14 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
|
||||
|
||||
|
||||
<ul>
|
||||
<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">importing a chrome profile into a persona</a></li>
|
||||
<li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
|
||||
<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
|
||||
<li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
|
||||
<code>reddit.com/some/url</code> -> <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
|
||||
</ul>
|
||||
|
||||
In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/51">Issue #51</a>.
|
||||
|
||||
</details>
|
||||
<br/>
|
||||
|
||||
@@ -1057,8 +1041,8 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
|
||||
</li>
|
||||
<li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
|
||||
</li>
|
||||
<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
|
||||
<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
|
||||
<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
|
||||
<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
@@ -1339,10 +1323,10 @@ archivebox init
|
||||
archivebox install # detect and install all extractor dependencies
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
archivebox server --debug --reload 0.0.0.0:5797
|
||||
archivebox server --debug --reload 0.0.0.0:8000
|
||||
|
||||
# Run the production server (with bg workers but no autoreloading)
|
||||
archivebox server 0.0.0.0:5797
|
||||
archivebox server 0.0.0.0:8000
|
||||
```
|
||||
|
||||
#### 2. Option B: Build the docker container and use that for development instead
|
||||
@@ -1354,10 +1338,10 @@ archivebox server 0.0.0.0:5797
|
||||
./bin/build_docker.sh dev
|
||||
|
||||
# Run the development server w/ autoreloading (but no bg workers)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
|
||||
|
||||
# Run the production server (with bg workers but no autoreloading)
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
|
||||
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
|
||||
|
||||
# (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
|
||||
# When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
|
||||
@@ -1379,11 +1363,11 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
|
||||
archivebox config --set DEBUG=True
|
||||
|
||||
# OR you can run a dev server with DEBUG=True in a few ways:
|
||||
archivebox server --debug --reload 0.0.0.0:5797
|
||||
archivebox server --debug --reload 0.0.0.0:8000
|
||||
# or
|
||||
archivebox server --debug 0.0.0.0:5797
|
||||
archivebox server --debug 0.0.0.0:8000
|
||||
# or
|
||||
env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
|
||||
env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application
|
||||
```
|
||||
|
||||
https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
|
||||
@@ -1418,7 +1402,7 @@ services:
|
||||
|
||||
# or with plain Docker:
|
||||
docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
|
||||
docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
|
||||
docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
|
||||
|
||||
# or with uv:
|
||||
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
|
||||
|
||||
@@ -7,11 +7,6 @@ class APIConfig(AppConfig):
|
||||
name = "archivebox.api"
|
||||
label = "api"
|
||||
|
||||
def ready(self):
|
||||
from archivebox.api.webhooks import register_delete_webhook_field_loader
|
||||
|
||||
register_delete_webhook_field_loader()
|
||||
|
||||
|
||||
def register_admin(admin_site):
|
||||
from archivebox.api.admin import register_admin
|
||||
|
||||
@@ -4,7 +4,7 @@ from django.http import HttpRequest
|
||||
|
||||
from ninja import Router, Schema
|
||||
|
||||
from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token, HeaderTokenAuth, BearerTokenAuth
|
||||
from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
|
||||
|
||||
|
||||
router = Router(tags=["Authentication"], auth=None)
|
||||
@@ -63,44 +63,3 @@ def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
|
||||
return {"success": True, "user_id": str(user.pk)}
|
||||
|
||||
return {"success": False, "user_id": None}
|
||||
|
||||
|
||||
@router.post(
|
||||
"/browser_session",
|
||||
auth=[HeaderTokenAuth(), BearerTokenAuth()],
|
||||
summary="Exchange an administrator API key for a normal browser session",
|
||||
)
|
||||
def browser_session(request: HttpRequest):
|
||||
from django.conf import settings
|
||||
from django.contrib.auth import login
|
||||
from django.http import JsonResponse
|
||||
from django.utils import timezone
|
||||
from ninja.errors import HttpError
|
||||
from archivebox.core.routes_util import get_admin_base_url
|
||||
|
||||
user = request.auth
|
||||
if not user.is_active or not user.is_superuser:
|
||||
raise HttpError(403, "An active administrator is required")
|
||||
login(request, user, backend="django.contrib.auth.backends.ModelBackend")
|
||||
# A browser session must not outlive the key used to establish it.
|
||||
token = request._api_token
|
||||
lifetime = settings.SESSION_COOKIE_AGE
|
||||
if token.expires:
|
||||
lifetime = min(lifetime, max(0, int((token.expires - timezone.now()).total_seconds())))
|
||||
request.session.set_expiry(lifetime)
|
||||
request.session.save()
|
||||
response = JsonResponse(
|
||||
{
|
||||
"admin_url": get_admin_base_url(request=request).rstrip("/") + "/admin/",
|
||||
"cookie": {
|
||||
"name": settings.SESSION_COOKIE_NAME,
|
||||
"value": request.session.session_key,
|
||||
"expires": request.session.get_expiry_date().timestamp(),
|
||||
"secure": settings.SESSION_COOKIE_SECURE,
|
||||
},
|
||||
},
|
||||
)
|
||||
# The native client installs a host-only HttpOnly cookie on the admin origin;
|
||||
# API and admin hosts can differ, so a Set-Cookie on the API host is insufficient.
|
||||
response["Cache-Control"] = "no-store"
|
||||
return response
|
||||
@@ -32,7 +32,7 @@ from archivebox.core.models import Snapshot, ArchiveResult, Tag
|
||||
from archivebox.core.permissions import public_snapshots_queryset
|
||||
from archivebox.api.auth import authenticated_user_from_request
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.core.routes_util import build_snapshot_detail_url, build_web_url
|
||||
from archivebox.core.routes_util import build_web_url
|
||||
from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url_length
|
||||
from archivebox.core.tag_util import (
|
||||
add_snapshot_counts,
|
||||
@@ -247,7 +247,7 @@ def get_archiveresult(request: HttpRequest, archiveresult_id: str):
|
||||
|
||||
def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
|
||||
normalized = str(plugin or "").strip().strip("/")
|
||||
if normalized in (".", "..") or not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
|
||||
if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
|
||||
raise HttpError(400, "Invalid ArchiveResult plugin name")
|
||||
return normalized
|
||||
|
||||
@@ -386,7 +386,6 @@ def _write_archiveresult_files(
|
||||
existing_output_files: dict[str, dict[str, Any]] | None = None,
|
||||
allow_empty: bool = False,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
plugin_name = _normalize_uploaded_archiveresult_plugin(plugin_name)
|
||||
files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
|
||||
output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
|
||||
mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
|
||||
@@ -800,7 +799,7 @@ def _snapshots_rss_response(
|
||||
)
|
||||
|
||||
for snapshot in snapshots:
|
||||
archived_url = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
|
||||
archived_url = build_web_url(f"/{snapshot.archive_path_from_db}", request=request)
|
||||
tags = [tag.name for tag in snapshot.tags.all()]
|
||||
crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
|
||||
description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
|
||||
|
||||
@@ -182,14 +182,6 @@ def crawl_file(request: HttpRequest, crawl_id: str, path: str):
|
||||
if not file_path.is_file() or crawl_root not in file_path.parents:
|
||||
raise HttpError(404, "Crawl file not found")
|
||||
|
||||
if path == "chrome_screencast/latest.jpg" and request.GET.get("after"):
|
||||
try:
|
||||
started_at = float(request.GET["after"])
|
||||
except ValueError:
|
||||
raise HttpError(400, "Invalid screencast start time")
|
||||
if file_path.stat().st_mtime < started_at:
|
||||
raise HttpError(404, "No frame for this capture yet")
|
||||
|
||||
response = FileResponse(file_path.open("rb"))
|
||||
response["Cache-Control"] = "no-store, no-cache, max-age=0, must-revalidate"
|
||||
response["Pragma"] = "no-cache"
|
||||
|
||||
@@ -2,7 +2,7 @@ __package__ = "archivebox.api"
|
||||
|
||||
import json
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
from django.db.models import Q
|
||||
@@ -23,7 +23,6 @@ class PersonaBrowserSettingsSchema(Schema):
|
||||
user_agent: str = ""
|
||||
viewport_size: str = ""
|
||||
viewport_device_scale_factor: float | None = None
|
||||
color_scheme: Literal["", "light", "dark"] = ""
|
||||
language: str = ""
|
||||
timezone: str = ""
|
||||
geolocation: dict[str, Any] | None = None
|
||||
@@ -95,8 +94,6 @@ def browser_settings_to_config(extension_persona_id: str, settings: PersonaBrows
|
||||
)
|
||||
if settings.viewport_device_scale_factor is not None:
|
||||
config["BROWSER_DEVICE_SCALE_FACTOR"] = settings.viewport_device_scale_factor
|
||||
if "color_scheme" in settings.model_fields_set:
|
||||
config["BROWSER_COLOR_SCHEME"] = settings.color_scheme
|
||||
if settings.language:
|
||||
config["BROWSER_LANGUAGE"] = settings.language
|
||||
if settings.timezone:
|
||||
@@ -153,12 +150,12 @@ def sync_persona(request: HttpRequest, payload: PersonaSyncSchema):
|
||||
persona.ensure_dirs()
|
||||
|
||||
cookies_written = False
|
||||
if "cookies_txt" in payload.model_fields_set:
|
||||
if payload.cookies_txt.strip():
|
||||
(persona.path / "cookies.txt").write_text(payload.cookies_txt)
|
||||
cookies_written = True
|
||||
|
||||
auth_written = False
|
||||
if "auth_json" in payload.model_fields_set:
|
||||
if payload.auth_json:
|
||||
(persona.path / "auth.json").write_text(json.dumps(payload.auth_json, indent=2, sort_keys=True) + "\n")
|
||||
auth_written = True
|
||||
|
||||
|
||||
@@ -5,40 +5,12 @@ from collections.abc import Callable
|
||||
import logging
|
||||
|
||||
from django.db import transaction
|
||||
from django.db.models.signals import pre_delete
|
||||
from signal_webhooks.handlers import sync_task_handler
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def load_deferred_fields_for_delete_webhook(sender, instance, using, **kwargs) -> None:
|
||||
"""Materialize deferred fields while DELETE webhook rows still exist.
|
||||
|
||||
Admin lists defer large fields, but signal_webhooks serializes all concrete
|
||||
fields in post_delete. Loading one then would query an already-deleted row
|
||||
and drop the event. Hydrate only deferred fields on webhook-enabled models
|
||||
here, preserving the normal list-query optimization.
|
||||
"""
|
||||
from signal_webhooks.handlers import find_hook_handler
|
||||
from signal_webhooks.utils import reference_for_model
|
||||
|
||||
ref = reference_for_model(type(instance))
|
||||
deferred_fields = instance.get_deferred_fields()
|
||||
if not deferred_fields or find_hook_handler(ref, "DELETE") is None:
|
||||
return
|
||||
|
||||
instance.refresh_from_db(using=using, fields=deferred_fields)
|
||||
|
||||
|
||||
def register_delete_webhook_field_loader() -> None:
|
||||
pre_delete.connect(
|
||||
load_deferred_fields_for_delete_webhook,
|
||||
dispatch_uid="archivebox.load_deferred_fields_for_delete_webhook",
|
||||
weak=False,
|
||||
)
|
||||
|
||||
|
||||
def warning_error_handler(hook: Any, error: Exception | None) -> None:
|
||||
if error is not None:
|
||||
logger.warning("Outbound webhook %r failed: %s", hook.name, error)
|
||||
|
||||
@@ -155,10 +155,13 @@ def cli(ctx, help=False):
|
||||
wants_help = any(arg in ("-h", "--help", "--version") for arg in sys.argv[1:])
|
||||
if not wants_help and (subcommand in ArchiveBoxGroup.archive_commands or subcommand in ArchiveBoxGroup.model_commands):
|
||||
try:
|
||||
# RUNSERVER_WORKER sets the web-worker markers on its own child.
|
||||
# Setting them on the server parent leaks them through supervisord
|
||||
# into runners waiting for takeover, which then register themselves
|
||||
# as web workers and trigger spurious runner_watch reloads.
|
||||
if subcommand == "server":
|
||||
run_in_debug = "--reload" in sys.argv or os.environ.get("DEBUG") in ("1", "true", "True", "TRUE", "yes")
|
||||
if run_in_debug:
|
||||
os.environ["ARCHIVEBOX_RUNSERVER"] = "1"
|
||||
if "--reload" in sys.argv:
|
||||
os.environ["ARCHIVEBOX_AUTORELOAD"] = "1"
|
||||
|
||||
from archivebox.config.django import setup_django
|
||||
from archivebox.misc.checks import check_data_folder, check_migrations
|
||||
|
||||
@@ -174,12 +177,8 @@ def cli(ctx, help=False):
|
||||
archivebox_init(quick=True)
|
||||
os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
|
||||
check_data_folder()
|
||||
if subcommand != "update" and not check_migrations(auto_apply=True):
|
||||
from archivebox.machine.models import Machine
|
||||
|
||||
# Host registration and config reconciliation belong to CLI
|
||||
# startup; get_config() is also used by read-only HTTP views.
|
||||
Machine.current()
|
||||
if subcommand != "update":
|
||||
check_migrations(auto_apply=True)
|
||||
except Exception as e:
|
||||
STDERR.print(f"[red][X] Error setting up Django or checking data folder: {e}[/red]")
|
||||
if subcommand not in ("manage", "shell"): # not all management commands need django to be setup beforehand
|
||||
|
||||
@@ -6,7 +6,6 @@ __command__ = "archivebox add"
|
||||
|
||||
import sys
|
||||
import os
|
||||
import signal
|
||||
from pathlib import Path
|
||||
|
||||
from typing import Any, TYPE_CHECKING
|
||||
@@ -228,10 +227,6 @@ def add(
|
||||
|
||||
assert command is not None
|
||||
exit_code = 0
|
||||
interactive_interrupts = sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())
|
||||
|
||||
def interrupt_hook() -> None:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
def crawl_is_complete() -> bool:
|
||||
crawl.refresh_from_db(fields=["status"])
|
||||
@@ -239,13 +234,7 @@ def add(
|
||||
|
||||
try:
|
||||
try:
|
||||
with (
|
||||
foreground_shutdown_signals(
|
||||
first_signal_message=None,
|
||||
interrupt_handlers={signal.SIGINT: interrupt_hook},
|
||||
),
|
||||
foreground_parent_watchdog(),
|
||||
):
|
||||
with foreground_shutdown_signals(first_signal_message=None), foreground_parent_watchdog():
|
||||
while True:
|
||||
standby = standby_until_foreground_runner_needed(
|
||||
command,
|
||||
@@ -257,7 +246,7 @@ def add(
|
||||
exit_code = run_runner_worker(
|
||||
["--crawl-id", str(crawl.id)],
|
||||
name=f"worker_runner_add_{os.getpid()}",
|
||||
interactive_interrupts=interactive_interrupts,
|
||||
interactive_interrupts=True,
|
||||
config=get_config(crawl=crawl),
|
||||
)
|
||||
crawl.refresh_from_db(fields=["status", "retry_at"])
|
||||
@@ -322,13 +311,9 @@ def add(
|
||||
|
||||
admin_url = build_admin_url(f"/admin/crawls/crawl/{crawl.id}/change/", config=config)
|
||||
|
||||
if snapshots_count:
|
||||
print("\n[bold]snapshot output saved to:[/bold]")
|
||||
for snapshot in crawl.snapshot_set.order_by("created_at").iterator():
|
||||
click.echo(f" {snapshot.output_dir}")
|
||||
print("\n[bold]crawl logs and setup:[/bold]")
|
||||
click.echo(f" {rel_output_str}")
|
||||
click.echo(f" {admin_url}")
|
||||
print("\n[bold]crawl output saved to:[/bold]")
|
||||
print(f" {rel_output_str}")
|
||||
print(f" {admin_url}")
|
||||
print(f"\n[bold]total urls snapshotted:[/bold] {snapshots_count}")
|
||||
print(f"[bold]total size:[/bold] {total_size}")
|
||||
print(f"[bold]total time:[/bold] {duration_str}")
|
||||
@@ -403,6 +388,10 @@ def main(**kwargs):
|
||||
raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
|
||||
if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
|
||||
raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
|
||||
if not kwargs.get("index_only"):
|
||||
from archivebox.workers.supervisord_util import require_crawl_memory
|
||||
|
||||
require_crawl_memory()
|
||||
|
||||
# Translate --only-new/--no-only-new into a crawl config override.
|
||||
# add() takes config overrides as a dict; no per-flag kwargs.
|
||||
|
||||
@@ -68,7 +68,7 @@ def help() -> None:
|
||||
[blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
|
||||
[grey53]# using Docker:[/grey53]
|
||||
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 5797:5797[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
|
||||
"""
|
||||
if IN_DOCKER
|
||||
else ""
|
||||
@@ -109,7 +109,7 @@ def help() -> None:
|
||||
[dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
|
||||
[dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
|
||||
[dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
|
||||
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:5797[/blue] [grey53]# Start the Web UI / API server[/grey53]
|
||||
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue] [grey53]# Start the Web UI / API server[/grey53]
|
||||
"""
|
||||
print(
|
||||
Panel(
|
||||
|
||||
@@ -111,11 +111,6 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
|
||||
else:
|
||||
os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init
|
||||
|
||||
from archivebox.machine.models import Machine
|
||||
|
||||
# Registration requires the final schema and reconciles legacy file config.
|
||||
Machine.current()
|
||||
|
||||
from archivebox.misc.db import database_display_location, is_postgres
|
||||
|
||||
assert database_exists()
|
||||
@@ -139,7 +134,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
|
||||
print(" > Skipping orphan snapshot import during init.")
|
||||
print()
|
||||
print(" [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
|
||||
print(" archivebox update --rescan --migrate-only")
|
||||
print(" archivebox update")
|
||||
|
||||
print("\n[green]----------------------------------------------------------------------[/green]")
|
||||
|
||||
|
||||
@@ -27,10 +27,11 @@ Examples:
|
||||
__package__ = "archivebox.cli"
|
||||
__command__ = "archivebox persona"
|
||||
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
import platform
|
||||
from pathlib import Path
|
||||
from dataclasses import replace
|
||||
from collections.abc import Iterable
|
||||
|
||||
import rich_click as click
|
||||
@@ -40,6 +41,116 @@ from archivebox.cli.cli_util import apply_filters
|
||||
from archivebox.personas import importers as persona_importers
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Browser Profile Locations
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def get_chrome_user_data_dir() -> Path | None:
|
||||
"""Get the default Chrome user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
|
||||
if system == "Darwin": # macOS
|
||||
candidates = [
|
||||
home / "Library" / "Application Support" / "Google" / "Chrome",
|
||||
home / "Library" / "Application Support" / "Chromium",
|
||||
]
|
||||
elif system == "Linux":
|
||||
candidates = [
|
||||
home / ".config" / "google-chrome",
|
||||
home / ".config" / "chromium",
|
||||
home / ".config" / "chrome",
|
||||
home / "snap" / "chromium" / "common" / "chromium",
|
||||
]
|
||||
elif system == "Windows":
|
||||
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
|
||||
candidates = [
|
||||
local_app_data / "Google" / "Chrome" / "User Data",
|
||||
local_app_data / "Chromium" / "User Data",
|
||||
]
|
||||
else:
|
||||
candidates = []
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.exists() and (candidate / "Default").exists():
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_brave_user_data_dir() -> Path | None:
|
||||
"""Get the default Brave user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
|
||||
if system == "Darwin":
|
||||
candidates = [
|
||||
home / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser",
|
||||
]
|
||||
elif system == "Linux":
|
||||
candidates = [
|
||||
home / ".config" / "BraveSoftware" / "Brave-Browser",
|
||||
]
|
||||
elif system == "Windows":
|
||||
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
|
||||
candidates = [
|
||||
local_app_data / "BraveSoftware" / "Brave-Browser" / "User Data",
|
||||
]
|
||||
else:
|
||||
candidates = []
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.exists() and (candidate / "Default").exists():
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_edge_user_data_dir() -> Path | None:
|
||||
"""Get the default Edge user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
|
||||
if system == "Darwin":
|
||||
candidates = [
|
||||
home / "Library" / "Application Support" / "Microsoft Edge",
|
||||
]
|
||||
elif system == "Linux":
|
||||
candidates = [
|
||||
home / ".config" / "microsoft-edge",
|
||||
home / ".config" / "microsoft-edge-beta",
|
||||
home / ".config" / "microsoft-edge-dev",
|
||||
]
|
||||
elif system == "Windows":
|
||||
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
|
||||
candidates = [
|
||||
local_app_data / "Microsoft" / "Edge" / "User Data",
|
||||
]
|
||||
else:
|
||||
candidates = []
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.exists() and (candidate / "Default").exists():
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
BROWSER_PROFILE_FINDERS = {
|
||||
"chrome": get_chrome_user_data_dir,
|
||||
"chromium": get_chrome_user_data_dir, # Same locations
|
||||
"brave": get_brave_user_data_dir,
|
||||
"edge": get_edge_user_data_dir,
|
||||
}
|
||||
|
||||
CHROMIUM_BROWSERS = {"chrome", "chromium", "brave", "edge"}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Cookie Extraction via CDP
|
||||
# =============================================================================
|
||||
|
||||
# =============================================================================
|
||||
# Validation Helpers
|
||||
# =============================================================================
|
||||
@@ -107,9 +218,6 @@ def create_personas(
|
||||
names: Iterable[str],
|
||||
import_from: str | None = None,
|
||||
profile: str | None = None,
|
||||
source: str | None = None,
|
||||
browser_binary: str | None = None,
|
||||
permissions: str | None = None,
|
||||
) -> int:
|
||||
"""
|
||||
Create Personas from names.
|
||||
@@ -131,26 +239,25 @@ def create_personas(
|
||||
rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
import_source = None
|
||||
if source and not import_from:
|
||||
rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
|
||||
return 1
|
||||
# Validate import source if specified
|
||||
source_profile_dir = None
|
||||
if import_from:
|
||||
try:
|
||||
if source:
|
||||
import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
|
||||
if import_source.browser != "persona":
|
||||
import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
|
||||
elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
|
||||
import_source = persona_importers.resolve_custom_import_source(import_from)
|
||||
else:
|
||||
import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
|
||||
if browser_binary:
|
||||
import_source = replace(import_source, browser_binary=browser_binary)
|
||||
except ValueError as err:
|
||||
rprint(f"[red]{err}[/red]", file=sys.stderr)
|
||||
import_from = import_from.lower()
|
||||
if import_from not in BROWSER_PROFILE_FINDERS:
|
||||
rprint(f"[red]Unknown browser: {import_from}[/red]", file=sys.stderr)
|
||||
rprint(f"[dim]Supported browsers: {', '.join(BROWSER_PROFILE_FINDERS.keys())}[/dim]", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
source_profile_dir = BROWSER_PROFILE_FINDERS[import_from]()
|
||||
if not source_profile_dir:
|
||||
rprint(f"[red]Could not find {import_from} profile directory[/red]", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
rprint(f"[dim]Found {import_from} profile: {source_profile_dir}[/dim]", file=sys.stderr)
|
||||
|
||||
if profile is None and (source_profile_dir / "Default").exists():
|
||||
profile = "Default"
|
||||
|
||||
created_count = 0
|
||||
for name in name_list:
|
||||
name = name.strip()
|
||||
@@ -163,24 +270,21 @@ def create_personas(
|
||||
rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
|
||||
continue
|
||||
|
||||
persona, created = Persona.objects.get_or_create(
|
||||
name=name,
|
||||
defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
|
||||
)
|
||||
persona, created = Persona.objects.get_or_create(name=name)
|
||||
|
||||
if created:
|
||||
persona.ensure_dirs()
|
||||
created_count += 1
|
||||
|
||||
rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
|
||||
else:
|
||||
rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
|
||||
|
||||
cookies_file = Path(persona.path) / "cookies.txt"
|
||||
|
||||
# Import browser profile if requested
|
||||
if import_source is not None:
|
||||
if import_from in CHROMIUM_BROWSERS and source_profile_dir is not None:
|
||||
try:
|
||||
rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
|
||||
import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
|
||||
import_result = persona_importers.import_persona_from_source(
|
||||
persona,
|
||||
import_source,
|
||||
@@ -188,25 +292,20 @@ def create_personas(
|
||||
import_cookies=True,
|
||||
capture_storage=False,
|
||||
)
|
||||
except (Exception, KeyboardInterrupt) as e:
|
||||
if created:
|
||||
shutil.rmtree(persona.path, ignore_errors=True)
|
||||
persona.delete()
|
||||
except Exception as e:
|
||||
rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if import_result.profile_copied:
|
||||
rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
|
||||
if import_result.cookies_imported:
|
||||
rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
|
||||
rprint(f"[green]Extracted cookies to {cookies_file}[/green]", file=sys.stderr)
|
||||
elif not import_result.profile_copied:
|
||||
rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
|
||||
|
||||
for warning in import_result.warnings:
|
||||
rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
|
||||
|
||||
if created:
|
||||
rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
|
||||
if not is_tty:
|
||||
write_record(
|
||||
{
|
||||
@@ -445,42 +544,11 @@ def main():
|
||||
|
||||
@main.command("create")
|
||||
@click.argument("names", nargs=-1)
|
||||
@click.option(
|
||||
"--import",
|
||||
"import_from",
|
||||
help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
|
||||
)
|
||||
@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
|
||||
@click.option(
|
||||
"--browser-binary",
|
||||
type=click.Path(exists=True, dir_okay=False),
|
||||
help="Source Chromium browser executable; required for other Chromium-based browsers",
|
||||
)
|
||||
@click.option("--import", "import_from", help="Import profile from browser (chrome, chromium, brave, edge)")
|
||||
@click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
|
||||
@click.option(
|
||||
"--permissions",
|
||||
type=click.Choice(["public", "unlisted", "private"]),
|
||||
help="Visibility for a newly created persona (existing personas are unchanged)",
|
||||
)
|
||||
def create_cmd(
|
||||
names: tuple,
|
||||
import_from: str | None,
|
||||
profile: str | None,
|
||||
source: str | None,
|
||||
browser_binary: str | None,
|
||||
permissions: str | None,
|
||||
):
|
||||
def create_cmd(names: tuple, import_from: str | None, profile: str | None):
|
||||
"""Create Personas, optionally importing from a browser profile."""
|
||||
sys.exit(
|
||||
create_personas(
|
||||
names,
|
||||
import_from=import_from,
|
||||
profile=profile,
|
||||
source=source,
|
||||
browser_binary=browser_binary,
|
||||
permissions=permissions,
|
||||
),
|
||||
)
|
||||
sys.exit(create_personas(names, import_from=import_from, profile=profile))
|
||||
|
||||
|
||||
@main.command("list")
|
||||
@@ -492,84 +560,6 @@ def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
|
||||
sys.exit(list_personas(name=name, name__icontains=name__icontains, limit=limit))
|
||||
|
||||
|
||||
@main.command("open")
|
||||
@click.argument("name", default="Default")
|
||||
def open_cmd(name: str):
|
||||
"""Open a persona's browser, including its imported logins."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
import psutil
|
||||
|
||||
from abx_plugins import get_plugins_dir
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.personas.models import Persona
|
||||
|
||||
valid, reason = validate_persona_name(name)
|
||||
if not valid:
|
||||
raise click.ClickException(reason)
|
||||
persona = Persona.get_or_create_named(name)
|
||||
persona.ensure_dirs()
|
||||
config = get_config(persona=persona)
|
||||
if os.environ.get("DISPLAY"):
|
||||
from archivebox.config import CONSTANTS
|
||||
|
||||
for process in psutil.process_iter():
|
||||
try:
|
||||
args = process.cmdline()
|
||||
profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
|
||||
if (
|
||||
not profile
|
||||
or any(arg.startswith(("--type=", "--headless")) for arg in args)
|
||||
or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
|
||||
or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
|
||||
):
|
||||
continue
|
||||
rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
|
||||
process.wait()
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
continue
|
||||
plugins_dir = Path(get_plugins_dir()).resolve()
|
||||
env = os.environ.copy()
|
||||
dependencies = subprocess.run(
|
||||
[
|
||||
str(Path(sys.executable).with_name("abxpkg")),
|
||||
"env",
|
||||
"--install",
|
||||
"--json",
|
||||
f"--lib={config.ABXPKG_LIB_DIR}",
|
||||
f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
if dependencies.returncode:
|
||||
raise click.ClickException(dependencies.stderr.strip())
|
||||
env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
|
||||
env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
|
||||
env["ACTIVE_PERSONA"] = persona.name
|
||||
env["PERSONAS_DIR"] = str(persona.path.parent)
|
||||
env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
|
||||
payload = config.model_dump(mode="json")
|
||||
payload.update(
|
||||
ACTIVE_PERSONA=persona.name,
|
||||
CHROME_HEADLESS=False,
|
||||
CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
|
||||
AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
|
||||
COOKIES_FILE=str(persona.COOKIES_FILE or ""),
|
||||
)
|
||||
script = Path(__file__).parent.parent / "personas" / "open_browser.js"
|
||||
result = subprocess.run(
|
||||
[env.get("NODE_BINARY") or "node", str(script)],
|
||||
input=json.dumps(payload),
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
@main.command("update")
|
||||
@click.option("--name", "-n", help="Set new name")
|
||||
def update_cmd(name: str | None):
|
||||
|
||||
@@ -54,9 +54,12 @@ RUNNER_DAEMON_ENV = "ARCHIVEBOX_RUNNER_DAEMON"
|
||||
|
||||
|
||||
def _exit_daemon_runner_on_signal(sig: signal.Signals) -> None:
|
||||
# A supervised daemon must exit non-zero so supervisord restarts it, while
|
||||
# still unwinding the active crawl's hooks and persisted process state.
|
||||
raise SystemExit(128 + sig.value)
|
||||
# A supervised `archivebox run --daemon` is intentionally a disposable
|
||||
# child. If it receives SIGINT/SIGTERM directly, exit with the conventional
|
||||
# signal status so supervisord treats it as an unexpected worker death and
|
||||
# restarts only the runner. The parent `archivebox server` owns supervisord
|
||||
# shutdown and must not be pulled down by a killed daemon worker.
|
||||
os._exit(128 + int(sig))
|
||||
|
||||
|
||||
def process_stdin_records() -> int:
|
||||
@@ -282,26 +285,19 @@ def run_runner(
|
||||
# future ownership lease cannot hide it from the unified scheduler.
|
||||
if crawl is not None:
|
||||
crawl.update_and_requeue(retry_at=now, refresh=False)
|
||||
# Interactivity follows terminal ownership, not the spelling of the command.
|
||||
# A supervised add worker has pipes and receives terminal intent/answers from
|
||||
# its parent; direct run owns the terminal itself. Daemons and redirected
|
||||
# commands must never wait for a person who cannot answer their prompt.
|
||||
# Trust the explicit terminal-owner flag, not Process.root: when add borrows
|
||||
# a server-owned supervisord, its worker's OS ancestry still leads to server.
|
||||
# Gating on root=ADD silently made the first Ctrl+C abort after takeover.
|
||||
interactive_interrupts = not daemon and (
|
||||
(sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())) or os.environ.get("ARCHIVEBOX_INTERACTIVE_INTERRUPTS") == "1"
|
||||
)
|
||||
# Only a foreground `archivebox add` gets the interactive "abort current
|
||||
# hook, continue/retry, second Ctrl+C exits" flow. Server/update/run owned
|
||||
# orchestrators should shut down immediately and cleanly on the first signal.
|
||||
interactive_interrupts = current.root.process_type == Process.TypeChoices.ADD
|
||||
if daemon:
|
||||
os.environ[RUNNER_DAEMON_ENV] = "1"
|
||||
|
||||
shutdown_state = None
|
||||
try:
|
||||
with (
|
||||
foreground_shutdown_signals(
|
||||
on_signal=_exit_daemon_runner_on_signal if daemon else None,
|
||||
raise_on_first_signal=not daemon,
|
||||
) as shutdown_state,
|
||||
),
|
||||
foreground_parent_watchdog(enabled=not daemon),
|
||||
):
|
||||
run_pending_crawls(
|
||||
@@ -310,16 +306,10 @@ def run_runner(
|
||||
maintenance_only=maintenance_only,
|
||||
interactive_interrupts=interactive_interrupts,
|
||||
)
|
||||
if daemon and shutdown_state.signal_name:
|
||||
return 128 + signal.Signals[shutdown_state.signal_name].value
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
if daemon and shutdown_state and shutdown_state.signal_name:
|
||||
return 128 + signal.Signals[shutdown_state.signal_name].value
|
||||
return 130
|
||||
return 0
|
||||
except asyncio.CancelledError as e:
|
||||
if daemon and shutdown_state and shutdown_state.signal_name:
|
||||
return 128 + signal.Signals[shutdown_state.signal_name].value
|
||||
if daemon:
|
||||
rprint(f"[red]Runner cancelled unexpectedly: {type(e).__name__}: {e}[/red]", file=sys.stderr)
|
||||
return 1
|
||||
@@ -372,11 +362,11 @@ def main(
|
||||
):
|
||||
sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only))
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
sys.exit(0)
|
||||
|
||||
with foreground_shutdown_signals(), foreground_parent_watchdog(enabled=not daemon):
|
||||
if snapshot_id:
|
||||
sys.exit(run_snapshot_worker(snapshot_id, daemon=daemon))
|
||||
sys.exit(run_snapshot_worker(snapshot_id))
|
||||
|
||||
if binary_id:
|
||||
try:
|
||||
@@ -396,7 +386,7 @@ def main(
|
||||
if crawl_id:
|
||||
sys.exit(
|
||||
run_runner(
|
||||
daemon=daemon,
|
||||
daemon=False,
|
||||
crawl_id=crawl_id,
|
||||
maintenance_only=maintenance_only,
|
||||
),
|
||||
@@ -411,7 +401,7 @@ def main(
|
||||
sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only))
|
||||
|
||||
|
||||
def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
|
||||
def run_snapshot_worker(snapshot_id: str) -> int:
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.core.takeover_util import enter_single_runner_gate
|
||||
from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
|
||||
@@ -426,7 +416,6 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
|
||||
return 0
|
||||
|
||||
snapshot = None
|
||||
interactive_interrupts = not daemon and sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())
|
||||
try:
|
||||
with foreground_shutdown_signals(), foreground_parent_watchdog():
|
||||
for _ in range(10):
|
||||
@@ -435,7 +424,7 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
|
||||
snapshot.update_and_requeue(retry_at=timezone.now())
|
||||
elif snapshot.retry_at > timezone.now():
|
||||
break
|
||||
if not run_due_snapshot(snapshot, lock_seconds=60, interactive_interrupts=interactive_interrupts):
|
||||
if not run_due_snapshot(snapshot, lock_seconds=60):
|
||||
break
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
@@ -448,7 +437,7 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
|
||||
snapshot.update_and_requeue(retry_at=timezone.now())
|
||||
except Exception:
|
||||
pass
|
||||
return 130
|
||||
return 0
|
||||
except Exception as e:
|
||||
rprint(f"[red]Runner error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
|
||||
import traceback
|
||||
|
||||
@@ -46,13 +46,13 @@ def _split_bind_spec(spec: str) -> tuple[str, str]:
|
||||
"""Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
|
||||
|
||||
The empty strings stand in for "not provided"; the caller fills in
|
||||
defaults. Bracketed IPv6 literals like ``[::1]:5797`` are handled.
|
||||
defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
|
||||
"""
|
||||
spec = (spec or "").strip()
|
||||
if not spec:
|
||||
return "", ""
|
||||
if spec.startswith("["):
|
||||
# Bracketed IPv6: ``[::1]`` or ``[::1]:5797``
|
||||
# Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
|
||||
end = spec.find("]")
|
||||
if end == -1:
|
||||
return spec, "" # malformed; let validator reject it
|
||||
@@ -77,11 +77,11 @@ def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
|
||||
(normalized to ``127.0.0.1``). Bare hostnames are rejected because the
|
||||
bind address feeds Daphne, which has to listen on a numeric address;
|
||||
public hostnames belong in ``BASE_URL`` instead. Empty values fall back
|
||||
to ``127.0.0.1`` / ``5797``.
|
||||
to ``127.0.0.1`` / ``8000``.
|
||||
"""
|
||||
raw_host, raw_port = _split_bind_spec(spec)
|
||||
host = raw_host.strip()
|
||||
port = (raw_port or "").strip() or "5797"
|
||||
port = (raw_port or "").strip() or "8000"
|
||||
|
||||
if host == "" or host.lower() == "localhost":
|
||||
host = "127.0.0.1"
|
||||
@@ -198,7 +198,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
|
||||
"[yellow] routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
|
||||
)
|
||||
print(
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
|
||||
)
|
||||
if config.USES_SUBDOMAIN_ROUTING:
|
||||
print(
|
||||
@@ -221,7 +221,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
|
||||
"[yellow] a reverse proxy / ingress / public hostname, e.g.[/yellow]",
|
||||
)
|
||||
print(
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
|
||||
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
|
||||
)
|
||||
print()
|
||||
|
||||
@@ -254,7 +254,7 @@ def server(
|
||||
print()
|
||||
|
||||
# First non-empty positional arg is the bind spec; otherwise inherit from
|
||||
# config (which defaults to "127.0.0.1:5797"). _parse_and_validate_bind_spec
|
||||
# config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
|
||||
# hard-errors on hostnames so the rest of the server can assume a numeric
|
||||
# bind host.
|
||||
bind_spec = next((arg for arg in runserver_args if arg), "")
|
||||
@@ -363,10 +363,12 @@ def server(
|
||||
command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
|
||||
|
||||
def still_owns_runtime_stack() -> bool:
|
||||
# This is polled at the log-tail cadence. Keep the autocommit
|
||||
# connection open so SQLite can reuse its page cache between polls;
|
||||
# reconnecting here repeatedly rereads the process table while idle.
|
||||
return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
|
||||
from django.db import connections
|
||||
|
||||
try:
|
||||
return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
|
||||
finally:
|
||||
connections.close_all()
|
||||
|
||||
shutdown_state = None
|
||||
try:
|
||||
@@ -414,13 +416,8 @@ def server(
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
from django.db import connections
|
||||
|
||||
try:
|
||||
if not shutdown_state or not shutdown_state.signal_name:
|
||||
command.mark_exited()
|
||||
finally:
|
||||
connections.close_all()
|
||||
if not shutdown_state or not shutdown_state.signal_name:
|
||||
command.mark_exited()
|
||||
print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
|
||||
|
||||
|
||||
|
||||
@@ -44,7 +44,6 @@ SNAPSHOT_LIST_CHUNK_SIZE = 5000
|
||||
|
||||
def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
from archivebox.config import VERSION
|
||||
from archivebox.config.common import redact_sensitive_config
|
||||
from archivebox.core.models import SnapshotTag
|
||||
|
||||
fields = (
|
||||
@@ -54,9 +53,6 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
"title",
|
||||
"bookmarked_at",
|
||||
"created_at",
|
||||
"modified_at",
|
||||
"config",
|
||||
"notes",
|
||||
"timestamp",
|
||||
"depth",
|
||||
"status",
|
||||
@@ -84,9 +80,6 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
|
||||
"tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
|
||||
"bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
|
||||
"created_at": row["created_at"].isoformat() if row["created_at"] else None,
|
||||
"modified_at": row["modified_at"].isoformat() if row["modified_at"] else None,
|
||||
"config": redact_sensitive_config(row["config"]),
|
||||
"notes": row["notes"],
|
||||
"timestamp": row["timestamp"],
|
||||
"depth": row["depth"],
|
||||
"status": row["status"],
|
||||
|
||||
@@ -7,7 +7,6 @@ import os
|
||||
import asyncio
|
||||
import shlex
|
||||
import time
|
||||
from itertools import islice
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from collections.abc import Iterable
|
||||
@@ -53,20 +52,17 @@ def _get_search_indexing_plugins() -> list[str]:
|
||||
def _build_filtered_snapshots_queryset(
|
||||
**kwargs,
|
||||
):
|
||||
from archivebox.core.models import Snapshot
|
||||
from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
|
||||
|
||||
limit = kwargs.pop("limit", None)
|
||||
reverse = kwargs.pop("reverse", False)
|
||||
snapshots = build_snapshot_queryset(**kwargs)
|
||||
snapshots = snapshots.order_by(kwargs.get("sort") or "-created_at", "-id")
|
||||
if reverse:
|
||||
snapshots = snapshots.reverse()
|
||||
if kwargs.get("resume"):
|
||||
snapshots = snapshots.filter(**{"timestamp__gte" if reverse else "timestamp__lte": kwargs["resume"]})
|
||||
snapshots = snapshots.filter(timestamp__lte=kwargs["resume"])
|
||||
snapshots = snapshots.select_related("crawl")
|
||||
if limit is not None and limit > 0:
|
||||
snapshot_ids = list(snapshots.values_list("id", flat=True)[:limit])
|
||||
snapshots = snapshots.filter(id__in=snapshot_ids)
|
||||
snapshots = Snapshot.objects.filter(id__in=snapshot_ids).select_related("crawl")
|
||||
|
||||
return snapshots
|
||||
|
||||
@@ -77,7 +73,6 @@ def reindex_snapshots(
|
||||
search_plugins: list[str],
|
||||
batch_size: int,
|
||||
collect_ids: bool = False,
|
||||
reverse: bool = False,
|
||||
wait_for_turn=None,
|
||||
) -> dict[str, Any]:
|
||||
from archivebox.cli.archivebox_extract import run_plugins
|
||||
@@ -99,15 +94,16 @@ def reindex_snapshots(
|
||||
plugin=plugin_name,
|
||||
status__in=completed_statuses,
|
||||
)
|
||||
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False)
|
||||
candidates = candidates.order_by("created_at", "id") if reverse else candidates.order_by("-created_at", "-id")
|
||||
remaining = candidates.select_related(None).only("id", "timestamp", "created_at").paged_iterator(chunk_size=batch_size)
|
||||
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False).order_by("id")
|
||||
after_id = None
|
||||
while True:
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
batch = list(islice(remaining, batch_size))
|
||||
page = candidates.filter(id__gt=after_id) if after_id is not None else candidates
|
||||
batch = list(page.select_related(None).only("id", "timestamp")[:batch_size])
|
||||
if not batch:
|
||||
break
|
||||
after_id = batch[-1].id
|
||||
records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
|
||||
stats["processed"] += len(batch)
|
||||
stats["requested"] += len(batch)
|
||||
@@ -165,18 +161,18 @@ def update(
|
||||
continuous: bool = False,
|
||||
index_only: bool = False,
|
||||
migrate_only: bool = False,
|
||||
rescan: bool = False,
|
||||
reverse: bool = False,
|
||||
stop_daemon_stack: bool = True,
|
||||
) -> None:
|
||||
"""
|
||||
Update the database without forcing a filesystem scan by default.
|
||||
Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
|
||||
|
||||
--migrate-only: eagerly finish pending filesystem migrations for known rows.
|
||||
--index-only: reconcile known snapshot assets/metadata and backfill search.
|
||||
--rescan: discover filesystem orphans first, then repair known snapshots.
|
||||
Explicit scans default to newest first; --reverse selects oldest first.
|
||||
Filters select existing database snapshots for targeted updates.
|
||||
Three-phase operation (without filters):
|
||||
- Phase 1: Drain legacy archive/ directories into the current layout
|
||||
- Phase 2: Select only stale fs_version rows through the indexed column
|
||||
- Phase 3: Run queued snapshot-level filesystem maintenance until idle
|
||||
|
||||
With filters: Only phase 2 (DB query), no filesystem operations.
|
||||
Without filters: All phases (full update).
|
||||
"""
|
||||
|
||||
from rich import print
|
||||
@@ -252,28 +248,12 @@ def update(
|
||||
exit_code = 0
|
||||
|
||||
try:
|
||||
if rescan and (is_filtered_update or resume or continuous):
|
||||
raise click.UsageError("--rescan cannot be combined with filters, --resume, or --continuous; rerun it to resume automatically.")
|
||||
if rescan:
|
||||
migrate_only = True # Discovery must not schedule new captures.
|
||||
wait_for_turn()
|
||||
|
||||
with foreground_shutdown_signals(), foreground_parent_watchdog():
|
||||
while True:
|
||||
if rescan:
|
||||
from archivebox.cli.rescan import rescan_snapshots
|
||||
|
||||
rescan_snapshots(
|
||||
wait_for_turn=wait_for_turn,
|
||||
reverse=reverse,
|
||||
scan_legacy=lambda: drain_old_archive_dirs(batch_size=batch_size, reverse=reverse),
|
||||
)
|
||||
do_migrate = migrate_only or is_filtered_update and not index_only
|
||||
do_index = index_only
|
||||
if not do_migrate and not do_index:
|
||||
print(
|
||||
"[*] Collection ready. Filesystem migrations remain lazy; use --migrate-only, --index-only, or --rescan for explicit scans.",
|
||||
)
|
||||
do_migrate = migrate_only or not index_only
|
||||
do_index = index_only or not migrate_only
|
||||
|
||||
if do_migrate:
|
||||
if (
|
||||
@@ -305,8 +285,7 @@ def update(
|
||||
after=after,
|
||||
resume=resume,
|
||||
batch_size=batch_size,
|
||||
queue_for_archiving=not migrate_only,
|
||||
reverse=reverse,
|
||||
queue_for_archiving=True,
|
||||
wait_for_turn=wait_for_turn,
|
||||
)
|
||||
print_stats(stats)
|
||||
@@ -314,13 +293,17 @@ def update(
|
||||
else:
|
||||
stats_combined = {"phase1": {}, "phase2": {}}
|
||||
|
||||
print("[*] Phase 1: Draining old archive/ directories (0.8.x → 0.9.x migration)...")
|
||||
stats_combined["phase1"] = drain_old_archive_dirs(
|
||||
resume_from=resume,
|
||||
batch_size=batch_size,
|
||||
)
|
||||
|
||||
print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
|
||||
stats_combined["phase2"] = process_all_db_snapshots(
|
||||
batch_size=batch_size,
|
||||
resume=resume,
|
||||
wait_for_turn=wait_for_turn,
|
||||
reverse=reverse,
|
||||
migrate=True,
|
||||
)
|
||||
print_combined_stats(stats_combined)
|
||||
# The due selectors are indexed and cheap when empty, so
|
||||
@@ -328,7 +311,7 @@ def update(
|
||||
# whole-table counts merely to decide whether to call it.
|
||||
print("[*] Phase 3: Running filesystem maintenance until idle...")
|
||||
if is_filtered_update:
|
||||
for snapshot_id in sorted(touched_snapshot_ids, reverse=not reverse):
|
||||
for snapshot_id in sorted(touched_snapshot_ids):
|
||||
run_scoped_runner("--snapshot-id", snapshot_id)
|
||||
elif migrate_only:
|
||||
run_scoped_runner("--maintenance-only")
|
||||
@@ -336,25 +319,6 @@ def update(
|
||||
run_scoped_runner()
|
||||
|
||||
if do_index:
|
||||
from archivebox.cli.rescan import reconcile_known_snapshots
|
||||
|
||||
snapshots = _build_filtered_snapshots_queryset(
|
||||
filter_patterns=filter_patterns,
|
||||
filter_type=filter_type,
|
||||
status=status,
|
||||
url__icontains=url__icontains,
|
||||
url__istartswith=url__istartswith,
|
||||
tag=tag,
|
||||
crawl_id=crawl_id,
|
||||
limit=limit,
|
||||
sort=sort,
|
||||
search=search,
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
reconcile_known_snapshots(snapshots, wait_for_turn=wait_for_turn)
|
||||
search_plugins = _get_search_indexing_plugins()
|
||||
if not search_plugins:
|
||||
print("[*] No search indexing plugins are available, nothing to backfill.")
|
||||
@@ -373,14 +337,12 @@ def update(
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
stats = reindex_snapshots(
|
||||
snapshots,
|
||||
search_plugins=search_plugins,
|
||||
batch_size=batch_size,
|
||||
collect_ids=is_filtered_update,
|
||||
reverse=reverse,
|
||||
wait_for_turn=wait_for_turn,
|
||||
)
|
||||
print_index_stats(stats)
|
||||
@@ -396,10 +358,6 @@ def update(
|
||||
exit_code = 130
|
||||
exact_resume = err.__dict__.get("archivebox_resume")
|
||||
resume_cmd = ["archivebox", "update"]
|
||||
if rescan:
|
||||
resume_cmd.append("--rescan")
|
||||
if reverse:
|
||||
resume_cmd.append("--reverse")
|
||||
if migrate_only:
|
||||
resume_cmd.append("--migrate-only")
|
||||
if index_only:
|
||||
@@ -447,7 +405,7 @@ def update(
|
||||
stop_own_supervisord_process()
|
||||
|
||||
|
||||
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500, reverse: bool = False) -> dict[str, int]:
|
||||
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500) -> dict[str, int]:
|
||||
"""
|
||||
Drain old archive/ directories (0.8.x → 0.9.x migration).
|
||||
|
||||
@@ -513,18 +471,18 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
|
||||
|
||||
# Scan real legacy directories only; these still contain data to migrate.
|
||||
entries = [
|
||||
(float(e.name), e.path)
|
||||
(e.stat().st_mtime, e.path)
|
||||
for e in all_entries
|
||||
if e.is_dir(follow_symlinks=False) and Snapshot.is_legacy_archive_dir(Path(e.path)) # Skip symlinks and 0.9.x roots
|
||||
]
|
||||
entries.sort(reverse=not reverse) # Newest first
|
||||
entries.sort(reverse=True) # Newest first
|
||||
print(f"[*] Found {len(entries)} old directories to drain")
|
||||
|
||||
for mtime, entry_path in entries:
|
||||
entry_path = Path(entry_path)
|
||||
|
||||
# Resume from timestamp if specified
|
||||
if resume_from and (entry_path.name < resume_from if reverse else entry_path.name > resume_from):
|
||||
if resume_from and entry_path.name > resume_from:
|
||||
continue
|
||||
|
||||
stats["processed"] += 1
|
||||
@@ -626,13 +584,7 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
|
||||
return stats
|
||||
|
||||
|
||||
def process_all_db_snapshots(
|
||||
batch_size: int = 500,
|
||||
resume: str | None = None,
|
||||
wait_for_turn=None,
|
||||
reverse: bool = False,
|
||||
migrate: bool = False,
|
||||
) -> dict[str, int]:
|
||||
def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
|
||||
"""Queue only snapshots whose indexed filesystem version is stale."""
|
||||
from archivebox.core.models import Snapshot
|
||||
from django.db.models import Q
|
||||
@@ -649,27 +601,21 @@ def process_all_db_snapshots(
|
||||
}
|
||||
queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
|
||||
if resume:
|
||||
queryset = queryset.filter(**{"timestamp__gte" if reverse else "timestamp__lte": resume})
|
||||
queryset = queryset.filter(timestamp__lte=resume)
|
||||
initial_now = timezone.now()
|
||||
rows_to_wake = queryset if migrate else queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
|
||||
ordered = rows_to_wake.order_by("created_at", "id") if reverse else rows_to_wake.order_by("-created_at", "-id")
|
||||
remaining = ordered.select_related("crawl__created_by").paged_iterator(chunk_size=batch_size)
|
||||
rows_to_wake = queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
|
||||
after_id = None
|
||||
while True:
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
batch = list(islice(remaining, batch_size))
|
||||
page = rows_to_wake.filter(id__gt=after_id) if after_id is not None else rows_to_wake
|
||||
batch = list(page.only("id", "fs_version", "modified_at").order_by("id")[:batch_size])
|
||||
if not batch:
|
||||
break
|
||||
after_id = batch[-1].id
|
||||
now = timezone.now()
|
||||
updated = 0
|
||||
for snapshot in batch:
|
||||
if migrate:
|
||||
from archivebox.services.runner import run_snapshot_maintenance
|
||||
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
run_snapshot_maintenance(str(snapshot.id))
|
||||
continue
|
||||
updated += int(
|
||||
snapshot.safe_update(
|
||||
{"retry_at": now, "modified_at": now},
|
||||
@@ -701,7 +647,6 @@ def process_filtered_snapshots(
|
||||
resume: str | None,
|
||||
batch_size: int,
|
||||
queue_for_archiving: bool = True,
|
||||
reverse: bool = False,
|
||||
wait_for_turn=None,
|
||||
) -> dict[str, Any]:
|
||||
"""Process snapshots matching filters (DB query only)."""
|
||||
@@ -724,7 +669,6 @@ def process_filtered_snapshots(
|
||||
before=before,
|
||||
after=after,
|
||||
resume=resume,
|
||||
reverse=reverse,
|
||||
)
|
||||
|
||||
total = snapshots.count()
|
||||
@@ -743,10 +687,6 @@ def process_filtered_snapshots(
|
||||
stats["snapshot_ids"].append(str(snapshot.id))
|
||||
update_values = {}
|
||||
updated = 0
|
||||
if not queue_for_archiving and snapshot.fs_migration_needed:
|
||||
from archivebox.services.runner import run_snapshot_maintenance
|
||||
|
||||
run_snapshot_maintenance(str(snapshot.id))
|
||||
if queue_for_archiving:
|
||||
update_values.update(
|
||||
{
|
||||
@@ -844,12 +784,6 @@ def print_index_stats(stats: dict[str, Any]) -> None:
|
||||
@click.option("--continuous", is_flag=True, help="Run continuously as background worker")
|
||||
@click.option("--index-only", is_flag=True, help="Backfill available search indexes from existing archived content")
|
||||
@click.option("--migrate-only", is_flag=True, help="Only migrate filesystem and update database/index state")
|
||||
@click.option("--reverse", is_flag=True, help="Process oldest first instead of newest first for explicit scans")
|
||||
@click.option(
|
||||
"--rescan",
|
||||
is_flag=True,
|
||||
help="Scan all snapshot directories for orphans and metadata repairs (slow, opt-in, safe to restart)",
|
||||
)
|
||||
@snapshot_filter_options(default_filter_type="exact")
|
||||
@docstring(update.__doc__)
|
||||
def main(**kwargs):
|
||||
|
||||
@@ -80,6 +80,20 @@ def _render_binary_abspath(abspath: str):
|
||||
return Text(abspath, style="green")
|
||||
|
||||
|
||||
def _binary_record_matches_runtime(installed, lib_dir: Path) -> bool:
|
||||
if not installed or not installed.is_valid or not installed.version:
|
||||
return False
|
||||
try:
|
||||
abspath = Path(installed.abspath).expanduser().resolve(strict=False)
|
||||
if not abspath.exists():
|
||||
return False
|
||||
if installed.binprovider not in {"", "env", "apt", "brew"}:
|
||||
abspath.relative_to(lib_dir)
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _binary_row_dedupe_key(
|
||||
*,
|
||||
display_name: str,
|
||||
@@ -113,6 +127,7 @@ def version(
|
||||
|
||||
from rich.panel import Panel
|
||||
from rich.console import Console
|
||||
from rich.live import Live
|
||||
from abx_dl.tables import binary_dependency_status, binary_dependency_table
|
||||
|
||||
from archivebox.config import CONSTANTS
|
||||
@@ -248,6 +263,7 @@ def version(
|
||||
prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
|
||||
failures = []
|
||||
seen_failures: set[str] = set()
|
||||
seen_rows: set[tuple[str, str, str, str]] = set()
|
||||
|
||||
from archivebox.plugins.discovery import get_enabled_plugins, get_plugin_catalog
|
||||
from abx_dl.config import get_required_binary_requests
|
||||
@@ -256,29 +272,37 @@ def version(
|
||||
from abxpkg.binary_service import BinaryEvent, BinaryService
|
||||
|
||||
plugins = get_plugin_catalog()
|
||||
enabled_plugin_names = set(get_enabled_plugins(config=config))
|
||||
runtime_config = normalize_runtime_config(dict(config.items()), json_safe=False)
|
||||
enabled_plugins = plugins.select(get_enabled_plugins(config=config))
|
||||
enabled_plugin_names = set(enabled_plugins)
|
||||
runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
|
||||
derived_config: dict[str, object] = {}
|
||||
db_binaries = {}
|
||||
db_available = False
|
||||
if in_data_dir:
|
||||
try:
|
||||
from archivebox.config.django import setup_django
|
||||
|
||||
setup_django()
|
||||
|
||||
from archivebox.machine.models import Machine
|
||||
from archivebox.machine.models import Machine, Binary
|
||||
|
||||
machine = Machine.current()
|
||||
derived_config = normalize_runtime_config(machine.config, json_safe=False)
|
||||
for binary in Binary.objects.filter(machine=machine).order_by("name", "-modified_at"):
|
||||
if _binary_record_matches_runtime(binary, config.ABXPKG_LIB_DIR):
|
||||
db_binaries.setdefault(binary.name, binary)
|
||||
db_available = True
|
||||
|
||||
except Exception as e:
|
||||
prnt()
|
||||
prnt("", f"[yellow]Warning: Could not query collection machine config; resolving through abxpkg: {e}[/yellow]")
|
||||
prnt("", f"[yellow]Warning: Could not query collection binary records; resolving through abxpkg: {e}[/yellow]")
|
||||
|
||||
declared_binary_specs: dict[str, dict[str, object]] = {}
|
||||
for plugin_name, plugin in plugins.items():
|
||||
if not plugin_may_have_requested_binary(plugin):
|
||||
continue
|
||||
plugin_requested = plugin_name.lower() in {name.lower() for name in requested_names}
|
||||
plugin_requested = bool(requested_names)
|
||||
plugin_enabled = plugin_name in enabled_plugin_names
|
||||
binary_records = get_required_binary_requests(
|
||||
plugin,
|
||||
plugin.config.required_binaries,
|
||||
@@ -296,9 +320,10 @@ def version(
|
||||
display_name = logical_name
|
||||
if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
|
||||
continue
|
||||
# This is a diagnostic: recheck the provider instead of trusting an
|
||||
# installation record or a cached version, including disabled plugins.
|
||||
binary_record["no_cache"] = True
|
||||
if not plugin_enabled and not requested_names:
|
||||
continue
|
||||
if _binary_record_matches_runtime(db_binaries.get(logical_name), config.ABXPKG_LIB_DIR):
|
||||
continue
|
||||
signature = json.dumps(binary_record, sort_keys=True, default=str)
|
||||
declared_binary_specs.setdefault(signature, binary_record)
|
||||
|
||||
@@ -316,47 +341,126 @@ def version(
|
||||
|
||||
loaded_binaries = asyncio.run(resolve_declared_binaries())
|
||||
|
||||
rows_by_binary: dict[str, dict] = {}
|
||||
rows: list[dict[str, object]] = []
|
||||
any_rows = False
|
||||
any_available = False
|
||||
compact_paths = console.is_terminal
|
||||
for plugin_name, plugin in plugins.items():
|
||||
if not plugin_may_have_requested_binary(plugin):
|
||||
continue
|
||||
plugin_requested = plugin_name.lower() in {name.lower() for name in requested_names}
|
||||
plugin_enabled = plugin_name in enabled_plugin_names
|
||||
disabled_by = ", ".join(
|
||||
f"{dependency.enabled_key}=False"
|
||||
for dependency in plugins.select([plugin_name]).values()
|
||||
if not config.get(dependency.enabled_key, True)
|
||||
)
|
||||
binary_records = get_required_binary_requests(
|
||||
plugin,
|
||||
plugin.config.required_binaries,
|
||||
overrides=runtime_config,
|
||||
derived_overrides=derived_config,
|
||||
run_output_dir=CONSTANTS.DATA_DIR,
|
||||
)
|
||||
for binary_record in binary_records:
|
||||
binary_record["no_cache"] = True
|
||||
actual_name = str(binary_record["name"])
|
||||
logical_name = (
|
||||
Path(actual_name).expanduser().name
|
||||
if ("/" in actual_name or "\\" in actual_name or actual_name.startswith("~"))
|
||||
else actual_name
|
||||
)
|
||||
display_name = logical_name
|
||||
if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
|
||||
live_enabled = console.is_terminal
|
||||
live_cm = Live(binary_dependency_table(rows), console=console, refresh_per_second=8) if live_enabled else None
|
||||
|
||||
def emit_row(row: dict[str, object]) -> None:
|
||||
rows.append(row)
|
||||
if live_cm is not None:
|
||||
live_cm.update(binary_dependency_table(rows), refresh=True)
|
||||
|
||||
if live_cm is not None:
|
||||
live_cm.start()
|
||||
try:
|
||||
for plugin_name, plugin in plugins.items():
|
||||
if not plugin_may_have_requested_binary(plugin):
|
||||
continue
|
||||
plugin_requested = bool(requested_names)
|
||||
plugin_enabled = plugin_name in enabled_plugin_names
|
||||
binary_records = get_required_binary_requests(
|
||||
plugin,
|
||||
plugin.config.required_binaries,
|
||||
overrides=runtime_config,
|
||||
derived_overrides=derived_config,
|
||||
run_output_dir=CONSTANTS.DATA_DIR,
|
||||
)
|
||||
for binary_record in binary_records:
|
||||
actual_name = str(binary_record["name"])
|
||||
logical_name = (
|
||||
Path(actual_name).expanduser().name
|
||||
if ("/" in actual_name or "\\" in actual_name or actual_name.startswith("~"))
|
||||
else actual_name
|
||||
)
|
||||
display_name = logical_name
|
||||
if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
|
||||
continue
|
||||
|
||||
loaded = loaded_binaries[json.dumps(binary_record, sort_keys=True, default=str)]
|
||||
abspath = loaded.abspath if loaded is not None else ""
|
||||
version_str = str(loaded.version or "unknown") if loaded is not None else "unknown"
|
||||
provider = str(loaded.binprovider or "env") if loaded is not None else "env"
|
||||
valid = loaded is not None and Path(abspath).is_file()
|
||||
installed = db_binaries.get(logical_name) if db_available else None
|
||||
if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
|
||||
abspath = installed.abspath
|
||||
version_str = (installed.version or "unknown")[:15]
|
||||
provider = (installed.binprovider or "env")[:8]
|
||||
valid = True
|
||||
elif not plugin_enabled and not requested_names:
|
||||
# `archivebox version` is expected to verify the active runtime, not
|
||||
# cold-load every optional plugin provider. Migration and status
|
||||
# checks often run with PLUGINS narrowed to a tiny set; resolving
|
||||
# disabled plugin binaries there can spend most of the command on
|
||||
# providers the current collection will never execute.
|
||||
continue
|
||||
else:
|
||||
loaded = loaded_binaries[json.dumps(binary_record, sort_keys=True, default=str)]
|
||||
abspath = loaded.abspath if loaded is not None else ""
|
||||
version_str = str(loaded.version or "unknown")[:15] if loaded is not None else "unknown"
|
||||
provider = str(loaded.binprovider or "env")[:8] if loaded is not None else "env"
|
||||
valid = loaded is not None
|
||||
|
||||
any_rows = True
|
||||
if valid:
|
||||
display_path = (
|
||||
_format_binary_abspath(
|
||||
abspath,
|
||||
pwd=Path.cwd(),
|
||||
lib_dir=config.ABXPKG_LIB_DIR,
|
||||
personas_dir=CONSTANTS.PERSONAS_DIR,
|
||||
home=Path.home(),
|
||||
)
|
||||
if compact_paths
|
||||
else abspath
|
||||
)
|
||||
rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
|
||||
any_available = True
|
||||
else:
|
||||
rendered_path = "[grey53]not installed[/grey53]"
|
||||
if plugin_enabled and display_name not in seen_failures:
|
||||
failures.append(display_name)
|
||||
seen_failures.add(display_name)
|
||||
|
||||
row_key = _binary_row_dedupe_key(
|
||||
display_name=display_name,
|
||||
valid=valid,
|
||||
version=version_str if valid else "-",
|
||||
provider=provider if valid else "-",
|
||||
abspath=abspath,
|
||||
)
|
||||
if row_key in seen_rows:
|
||||
continue
|
||||
seen_rows.add(row_key)
|
||||
|
||||
emit_row(
|
||||
{
|
||||
"plugin": plugin_name,
|
||||
"status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
|
||||
"binary": display_name,
|
||||
"version": version_str if valid else "-",
|
||||
"provider": provider if valid else "-",
|
||||
"path": rendered_path,
|
||||
"style": "" if plugin_enabled else "dim",
|
||||
},
|
||||
)
|
||||
|
||||
if db_available:
|
||||
for binary_name, installed in db_binaries.items():
|
||||
if requested_names and binary_name not in requested_names:
|
||||
continue
|
||||
abspath = installed.abspath
|
||||
version_str = (installed.version or "unknown")[:15]
|
||||
provider = (installed.binprovider or "env")[:8]
|
||||
row_key = _binary_row_dedupe_key(
|
||||
display_name=binary_name,
|
||||
valid=True,
|
||||
version=version_str,
|
||||
provider=provider,
|
||||
abspath=abspath,
|
||||
)
|
||||
if row_key in seen_rows:
|
||||
continue
|
||||
seen_rows.add(row_key)
|
||||
|
||||
any_rows = True
|
||||
if valid:
|
||||
display_path = (
|
||||
_format_binary_abspath(
|
||||
abspath,
|
||||
@@ -368,67 +472,24 @@ def version(
|
||||
if compact_paths
|
||||
else abspath
|
||||
)
|
||||
rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
|
||||
emit_row(
|
||||
{
|
||||
"plugin": "(database)",
|
||||
"status": binary_dependency_status(enabled=True, valid=True),
|
||||
"binary": binary_name,
|
||||
"version": version_str,
|
||||
"provider": provider,
|
||||
"path": _render_binary_abspath(display_path) if compact_paths else display_path,
|
||||
"style": "dim",
|
||||
},
|
||||
)
|
||||
any_rows = True
|
||||
any_available = True
|
||||
else:
|
||||
rendered_path = "[grey53]not installed[/grey53]"
|
||||
if plugin_enabled and display_name not in seen_failures:
|
||||
failures.append(display_name)
|
||||
seen_failures.add(display_name)
|
||||
|
||||
if not plugin_enabled:
|
||||
rendered_path = f"[grey53]disabled by {disabled_by or 'PLUGINS configuration'}[/grey53]"
|
||||
|
||||
row = {
|
||||
"plugin": plugin_name,
|
||||
"status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
|
||||
"binary": display_name,
|
||||
"version": version_str if valid else "-",
|
||||
"provider": provider if valid else "-",
|
||||
"path": rendered_path,
|
||||
"style": "" if plugin_enabled else "dim",
|
||||
"enabled": plugin_enabled,
|
||||
"valid": valid,
|
||||
"plugins": [plugin_name],
|
||||
"disabled_by": [disabled_by or "PLUGINS configuration"] if not plugin_enabled else [],
|
||||
}
|
||||
existing = rows_by_binary.get(display_name)
|
||||
if existing is None:
|
||||
rows_by_binary[display_name] = row
|
||||
else:
|
||||
consumers = list(dict.fromkeys([*existing["plugins"], plugin_name]))
|
||||
disabled_reasons = list(dict.fromkeys([*existing["disabled_by"], *row["disabled_by"]]))
|
||||
# Disabled consumers cannot mask an active requirement. If any
|
||||
# active request fails (e.g. a version floor), keep that failure.
|
||||
if plugin_enabled and (not existing["enabled"] or not valid):
|
||||
existing.update(row)
|
||||
existing["plugins"] = consumers
|
||||
existing["disabled_by"] = disabled_reasons
|
||||
existing["plugin"] = ", ".join(consumers)
|
||||
if not existing["enabled"]:
|
||||
existing["path"] = f"[grey53]disabled by {', '.join(disabled_reasons)}[/grey53]"
|
||||
|
||||
rows = list(rows_by_binary.values())
|
||||
|
||||
if console.is_terminal and console.width < 120:
|
||||
from rich.text import Text
|
||||
|
||||
for row in rows:
|
||||
heading = Text.from_markup(str(row["status"]))
|
||||
heading.append(f" {row['binary']} {row['version']} ({row['provider']}) · {row['plugin']}")
|
||||
prnt(heading, style=str(row["style"]))
|
||||
path = row["path"]
|
||||
prnt(Text.assemble(" ", path if isinstance(path, Text) else Text.from_markup(str(path))), style=str(row["style"]))
|
||||
else:
|
||||
table = binary_dependency_table(rows)
|
||||
table.expand = False
|
||||
table.title = None
|
||||
table.box = None
|
||||
for column in table.columns:
|
||||
column.width = None
|
||||
column.max_width = None
|
||||
column.ratio = None
|
||||
prnt(table)
|
||||
finally:
|
||||
if live_cm is not None:
|
||||
live_cm.stop()
|
||||
else:
|
||||
prnt(binary_dependency_table(rows))
|
||||
|
||||
if not any_rows:
|
||||
prnt("", "[grey53]No required binaries declared for discovered plugins.[/grey53]")
|
||||
|
||||
@@ -1,222 +0,0 @@
|
||||
"""Explicit discovery of portable snapshots; no checkpoint or default tree walk."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from itertools import groupby
|
||||
from pathlib import Path
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core.exceptions import ValidationError
|
||||
from django.db import IntegrityError
|
||||
from rich import print
|
||||
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.misc.util import parse_date
|
||||
|
||||
|
||||
def _directories(parent: Path):
|
||||
# Names encode dates / UUIDv7 creation time. Newest first within each
|
||||
# subtree, with bounded memory and no traversal into plugin payloads.
|
||||
with os.scandir(parent) as entries:
|
||||
paths = [Path(entry.path) for entry in entries if entry.is_dir(follow_symlinks=False)]
|
||||
yield from sorted(paths, key=lambda path: path.name, reverse=True)
|
||||
|
||||
|
||||
def _snapshot_directories(reverse=False):
|
||||
dates = []
|
||||
for user in _directories(CONSTANTS.USERS_DIR):
|
||||
root = user / CONSTANTS.SNAPSHOTS_DIR_NAME
|
||||
if root.is_symlink() or not root.is_dir():
|
||||
continue
|
||||
dates.extend(_directories(root))
|
||||
dates.sort(key=lambda path: path.name, reverse=not reverse)
|
||||
for _, date_group in groupby(dates, key=lambda path: path.name):
|
||||
directories = [directory for date in date_group for domain in _directories(date) for directory in _directories(domain)]
|
||||
yield from sorted(directories, key=lambda path: path.name, reverse=not reverse)
|
||||
|
||||
|
||||
def _fields(record, names):
|
||||
return {name: record[name] for name in names.split() if name in record}
|
||||
|
||||
|
||||
def _dates(record):
|
||||
return {name: parse_date(record[name]) for name in ("created_at", "modified_at") if record.get(name)}
|
||||
|
||||
|
||||
def _restore_dates(obj, record):
|
||||
dates = _dates(record)
|
||||
if dates:
|
||||
type(obj).objects.filter(pk=obj.pk).update(**dates)
|
||||
for name, value in dates.items():
|
||||
setattr(obj, name, value)
|
||||
|
||||
|
||||
def _reconcile_directory(directory: Path) -> str:
|
||||
snapshot_id = uuid.UUID(directory.name)
|
||||
snapshot = Snapshot.objects.select_related("crawl__created_by").filter(pk=snapshot_id).first()
|
||||
index = directory / CONSTANTS.JSONL_INDEX_FILENAME
|
||||
if index.is_symlink():
|
||||
raise ValueError("metadata must not be a symlink")
|
||||
if not index.exists():
|
||||
if snapshot is None or Path(snapshot.output_dir) != directory or snapshot.status != "sealed":
|
||||
raise ValueError("missing index.jsonl; cannot reconstruct an unknown or active snapshot")
|
||||
snapshot.write_index_jsonl(output_dir=directory)
|
||||
snapshot.reconcile_filesystem_links()
|
||||
return "Repaired"
|
||||
|
||||
records = [json.loads(line) for line in index.read_text().splitlines() if line.strip()]
|
||||
if any(not isinstance(record, dict) for record in records):
|
||||
raise ValueError("metadata records must be JSON objects")
|
||||
snapshots = [record for record in records if record.get("type") == "Snapshot"]
|
||||
if len(snapshots) != 1:
|
||||
raise ValueError("expected exactly one Snapshot record")
|
||||
record = snapshots[0]
|
||||
if uuid.UUID(record["id"]) != snapshot_id:
|
||||
raise ValueError("snapshot ID conflicts with directory name")
|
||||
if record.get("status") != "sealed":
|
||||
raise ValueError("snapshot is not sealed; finish the source capture first")
|
||||
if record.get("fs_version") not in {Snapshot._fs_current_version(), *Snapshot._FS_VERSION_MIGRATION_PATHS}:
|
||||
raise ValueError("unsupported filesystem version")
|
||||
crawl_id = uuid.UUID(record["crawl_id"])
|
||||
owner = directory.parents[3].name
|
||||
crawl_records = [item for item in records if item.get("type") == "Crawl" and uuid.UUID(item["id"]) == crawl_id]
|
||||
crawl_record = crawl_records[0] if crawl_records else {}
|
||||
if crawl_record.get("created_by", owner) != owner:
|
||||
raise ValueError("crawl owner conflicts with directory path")
|
||||
crawl = Crawl.objects.select_related("created_by").filter(pk=crawl_id).first()
|
||||
imported_crawl = crawl is None
|
||||
if crawl and crawl.created_by.username != owner:
|
||||
raise ValueError("existing crawl ID belongs to a different owner")
|
||||
if snapshot and (snapshot.url != record["url"] or snapshot.crawl_id != crawl_id or snapshot.timestamp != record["timestamp"]):
|
||||
raise ValueError("existing snapshot ID has conflicting identity; refusing to overwrite")
|
||||
if snapshot and Path(snapshot.output_dir) != directory:
|
||||
raise ValueError("existing snapshot has a different storage path")
|
||||
|
||||
# Validate identities before inserting anything. Never turn an ID collision
|
||||
# into a new ID, and never overwrite a newer destination row.
|
||||
# write_index_jsonl retains historical hook records before appending the
|
||||
# canonical DB rows. The final record for each hook is authoritative.
|
||||
results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
|
||||
pending = []
|
||||
for result in results:
|
||||
result_id = uuid.UUID(result["id"])
|
||||
if uuid.UUID(result["snapshot_id"]) != snapshot_id:
|
||||
raise ValueError("ArchiveResult belongs to another snapshot")
|
||||
existing = ArchiveResult.objects.filter(pk=result_id).first()
|
||||
identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
|
||||
if existing:
|
||||
if (existing.snapshot_id, existing.plugin, existing.hook_name) != identity:
|
||||
raise ValueError("existing ArchiveResult ID has conflicting identity")
|
||||
elif ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin=identity[1], hook_name=identity[2]).exists():
|
||||
raise ValueError("existing hook has a different ArchiveResult ID")
|
||||
else:
|
||||
pending.append(result)
|
||||
|
||||
imported = snapshot is None
|
||||
if imported:
|
||||
if Snapshot.objects.filter(timestamp=record["timestamp"]).exists():
|
||||
raise ValueError("timestamp belongs to another snapshot")
|
||||
if Snapshot.objects.filter(crawl_id=crawl_id, url=record["url"]).exists():
|
||||
raise ValueError("crawl URL belongs to another snapshot")
|
||||
user, _ = get_user_model().objects.get_or_create(
|
||||
username=owner,
|
||||
defaults={"is_active": False, "password": "!"},
|
||||
)
|
||||
if crawl is None:
|
||||
# Old exports contain only crawl_id. Keep that identity but report
|
||||
# that absent crawl metadata cannot be reconstructed exactly.
|
||||
if not crawl_record:
|
||||
print(f"[yellow] {directory}: no Crawl record; preserving ID with recovered metadata[/yellow]")
|
||||
crawl = Crawl(
|
||||
id=crawl_id,
|
||||
created_by=user,
|
||||
status="sealed",
|
||||
retry_at=None,
|
||||
**_fields(crawl_record, "urls max_depth config tags_str label"),
|
||||
)
|
||||
crawl.save(force_insert=True)
|
||||
_restore_dates(crawl, crawl_record)
|
||||
snapshot = Snapshot.create_from_directory(directory, crawl=crawl, preserve_identity=True)
|
||||
if snapshot is None:
|
||||
raise ValueError("invalid snapshot metadata")
|
||||
snapshot.status = "sealed"
|
||||
snapshot.retry_at = None
|
||||
if "config" not in record:
|
||||
snapshot.config = {"PERMISSIONS": "private"}
|
||||
if Path(snapshot.get_storage_path_for_version(Snapshot._fs_current_version())) != directory:
|
||||
raise ValueError("metadata does not match its owner/date/domain directory")
|
||||
snapshot.save(force_insert=True)
|
||||
_restore_dates(snapshot, record)
|
||||
|
||||
# Each insert is durable. An interruption after the Snapshot insert is not
|
||||
# completion: the next pass still imports missing result IDs and tags.
|
||||
for result in pending:
|
||||
snapshot._create_archive_result_if_missing(result, {}, preserve_identity=True)
|
||||
tags_before = set(snapshot.tags.values_list("name", flat=True))
|
||||
snapshot._merge_tags_from_index(record)
|
||||
tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
|
||||
link = Path(snapshot.crawl.output_dir) / "snapshots" / snapshot.extract_domain_from_url(snapshot.url) / str(snapshot.id)
|
||||
links_ok = link.is_symlink() and link.resolve() == directory.resolve()
|
||||
if not links_ok:
|
||||
snapshot.reconcile_filesystem_links()
|
||||
if snapshot.fs_migration_needed:
|
||||
snapshot.migrate_filesystem_to_current_version(source_dir=directory)
|
||||
if imported:
|
||||
_restore_dates(snapshot, record)
|
||||
if imported_crawl:
|
||||
_restore_dates(crawl, crawl_record)
|
||||
# Leave source metadata untouched, including process/binary records that
|
||||
# are historical evidence rather than runnable destination process state.
|
||||
return "Imported" if imported else "Repaired" if pending or tags_changed or not links_ok else "Unchanged"
|
||||
|
||||
|
||||
def reconcile_known_snapshots(snapshots, wait_for_turn=None):
|
||||
"""Explicit metadata / asset repair; normal updates never call this scan."""
|
||||
for snapshot in snapshots.filter(status="sealed").paged_iterator():
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
if snapshot.fs_migration_needed:
|
||||
snapshot.migrate_filesystem_to_current_version()
|
||||
directory = Path(snapshot.output_dir)
|
||||
if not directory.is_dir():
|
||||
raise ValueError(f"Missing snapshot directory: {directory}")
|
||||
_reconcile_directory(directory)
|
||||
for result in snapshot.archiveresult_set.all():
|
||||
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
|
||||
|
||||
|
||||
def rescan_snapshots(wait_for_turn=None, reverse=False, scan_legacy=None):
|
||||
stats = dict.fromkeys(("Imported", "Repaired", "Unchanged", "Errors"), 0)
|
||||
print("[*] Rescanning snapshot directories (O(N), orphans first; completed records are reused)...")
|
||||
known = {str(pk) for pk in Snapshot.objects.values_list("pk", flat=True)}
|
||||
# Two streaming passes avoid enumerating the entire archive before the
|
||||
# first import. Rows imported before interruption move to the second pass
|
||||
# next time; only one day's directory names are held in memory at a time.
|
||||
for existing in (False, True):
|
||||
if existing and scan_legacy:
|
||||
scan_legacy()
|
||||
for directory in _snapshot_directories(reverse=reverse):
|
||||
if (directory.name.replace("-", "") in known) != existing:
|
||||
continue
|
||||
if wait_for_turn:
|
||||
wait_for_turn()
|
||||
try:
|
||||
outcome = _reconcile_directory(directory)
|
||||
if directory.name.replace("-", "") in known:
|
||||
for result in ArchiveResult.objects.filter(snapshot_id=uuid.UUID(directory.name)):
|
||||
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
|
||||
stats[outcome] += 1
|
||||
if outcome != "Unchanged":
|
||||
print(f" {outcome}: {directory}")
|
||||
except (OSError, ValueError, KeyError, TypeError, ValidationError, IntegrityError) as err:
|
||||
stats["Errors"] += 1
|
||||
print(f"[yellow] Unresolved: {directory}: {err}[/yellow]")
|
||||
if sum(stats.values()) % 100 == 0:
|
||||
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
|
||||
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
|
||||
if stats["Errors"]:
|
||||
raise SystemExit(1)
|
||||
return stats
|
||||
@@ -17,7 +17,7 @@ CONFIG_FILE_HEADER = (
|
||||
"# You can add options here manually in INI format, or automatically by running:\n"
|
||||
"# archivebox config --set KEY=VALUE\n"
|
||||
"#\n"
|
||||
"# This file is kept in sync 1:1 with the current Machine.config in the DB —\n"
|
||||
"# This file is kept in sync 1:1 with Machine.config in the index DB —\n"
|
||||
"# editing either side propagates to the other. ``archivebox init`` reads\n"
|
||||
"# this file on startup; the admin Machine.config editor writes both.\n"
|
||||
"#\n"
|
||||
|
||||
@@ -18,7 +18,7 @@ from urllib.parse import quote, urlparse
|
||||
|
||||
from abx_plugins.plugins.base.utils import build_config_model
|
||||
from django.db import DatabaseError
|
||||
from pydantic import BaseModel, Field, PrivateAttr, TypeAdapter, create_model, field_validator, model_validator
|
||||
from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
from rich.console import Console
|
||||
|
||||
@@ -83,16 +83,12 @@ def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
|
||||
return candidate
|
||||
|
||||
listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
|
||||
if listen_host and listen_host.lower() not in {"archivebox.localhost:8000", "archivebox.localhost:5797"}:
|
||||
if listen_host and listen_host.lower() != "archivebox.localhost:8000":
|
||||
return normalize(listen_host)
|
||||
|
||||
csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
|
||||
csrf_origins = [origin for origin in csrf_origins if origin]
|
||||
nondefault_csrf_origins = [
|
||||
origin
|
||||
for origin in csrf_origins
|
||||
if origin not in {"http://admin.archivebox.localhost:8000", "http://admin.archivebox.localhost:5797"}
|
||||
]
|
||||
nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
|
||||
if len(nondefault_csrf_origins) == 1:
|
||||
return nondefault_csrf_origins[0]
|
||||
|
||||
@@ -318,7 +314,7 @@ class ServerConfig(BaseConfigSet):
|
||||
)
|
||||
|
||||
SECRET_KEY: str = Field(default_factory=lambda: "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789_") for _ in range(50)))
|
||||
BIND_ADDR: str = Field(default="127.0.0.1:5797")
|
||||
BIND_ADDR: str = Field(default="127.0.0.1:8000")
|
||||
BASE_URL: str = Field(default="")
|
||||
ALLOWED_HOSTS: str = Field(default="*")
|
||||
CSRF_TRUSTED_ORIGINS: str = Field(default="")
|
||||
@@ -465,16 +461,9 @@ class ArchivingConfig(BaseConfigSet):
|
||||
USER_AGENT: str = Field(
|
||||
default=f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)",
|
||||
)
|
||||
COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
|
||||
AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
|
||||
COOKIES_FILE: Path | None = Field(default=None)
|
||||
|
||||
URL_DENYLIST: str = Field(
|
||||
default=(
|
||||
r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$"
|
||||
r"|^https?://news\.ycombinator\.com/(?:vote|hide)/?(?:[?#]|$)"
|
||||
),
|
||||
alias="URL_BLACKLIST",
|
||||
)
|
||||
URL_DENYLIST: str = Field(default=r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$", alias="URL_BLACKLIST")
|
||||
URL_ALLOWLIST: str | None = Field(default=None, alias="URL_WHITELIST")
|
||||
|
||||
DEFAULT_PERSONA: str = Field(default="Default", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
|
||||
@@ -1113,7 +1102,7 @@ def get_config(
|
||||
if apps.ready:
|
||||
from archivebox.machine.models import Machine
|
||||
|
||||
machine = Machine.current_readonly()
|
||||
machine = Machine.current()
|
||||
except (DatabaseError, ImportError, RuntimeError, TypeError, ValueError):
|
||||
machine = None
|
||||
|
||||
@@ -1264,7 +1253,7 @@ def get_config(
|
||||
config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
|
||||
for key in explicit_plugin_enabled_keys:
|
||||
if key in config_data:
|
||||
setattr(config, key, TypeAdapter(bool).validate_python(config_data[key]))
|
||||
setattr(config, key, config_data[key])
|
||||
if config.PLUGINS:
|
||||
config._derive_plugin_enabled_config(respect_current_enabled=True)
|
||||
if redact_sensitive:
|
||||
|
||||
@@ -63,14 +63,7 @@ def setup_django(check_db=False) -> None:
|
||||
os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
|
||||
if CONSTANTS.DATA_DIR.exists():
|
||||
for child in CONSTANTS.DATA_DIR.iterdir():
|
||||
try:
|
||||
os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
|
||||
except FileNotFoundError:
|
||||
# A concurrent SQLite transaction can remove its journal
|
||||
# between directory enumeration and chown. There is no
|
||||
# ownership left to fix; other permission errors must
|
||||
# still fail startup rather than being hidden.
|
||||
continue
|
||||
os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
|
||||
|
||||
# Suppress the "database access during app initialization" warning
|
||||
# This warning can be triggered during django.setup() but is safe to ignore
|
||||
|
||||
+13
-21
@@ -85,31 +85,23 @@ def get_collection_id(DATA_DIR=DATA_DIR) -> str:
|
||||
|
||||
|
||||
@cache
|
||||
def get_machine_id(length: int = 8) -> str:
|
||||
"""Get a machine ID, using the network identity when no OS ID is available."""
|
||||
import machineid
|
||||
def get_machine_id() -> str:
|
||||
"""Get a short, stable, unique ID for the current machine (e.g. abc45678)"""
|
||||
|
||||
MACHINE_ID = "unknown"
|
||||
try:
|
||||
return machineid.hashed_id("archivebox")[:length]
|
||||
except machineid.MachineIdNotFound:
|
||||
import uuid
|
||||
import machineid
|
||||
|
||||
import psutil
|
||||
MACHINE_ID = machineid.hashed_id("archivebox")[:8]
|
||||
except Exception:
|
||||
try:
|
||||
import uuid
|
||||
import hashlib
|
||||
|
||||
node = uuid.getnode()
|
||||
identity = str(node)
|
||||
if node & (1 << 40):
|
||||
# uuid.getnode() silently generates a per-process random value when
|
||||
# tools such as ip/ifconfig are absent. Read interfaces directly so
|
||||
# separate CLI processes agree even in a minimal container.
|
||||
addresses = sorted(
|
||||
address.address
|
||||
for interface in psutil.net_if_addrs().values()
|
||||
for address in interface
|
||||
if address.family == psutil.AF_LINK and address.address not in ("", "00:00:00:00:00:00")
|
||||
)
|
||||
identity = str(int(addresses[0].replace(":", "").replace("-", ""), 16)) if addresses else f"hostname:{socket.gethostname()}"
|
||||
return hashlib.sha256(identity.encode()).hexdigest()[:length]
|
||||
MACHINE_ID = hashlib.sha256(str(uuid.getnode()).encode()).hexdigest()[:8]
|
||||
except Exception:
|
||||
pass
|
||||
return MACHINE_ID
|
||||
|
||||
|
||||
@cache
|
||||
|
||||
@@ -2,9 +2,11 @@ __package__ = "archivebox.core"
|
||||
|
||||
import html
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
from functools import reduce
|
||||
from operator import and_
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
from django.contrib import admin
|
||||
@@ -22,13 +24,7 @@ from django.utils.text import smart_split
|
||||
|
||||
from archivebox.base_models.admin import BaseModelAdmin
|
||||
from archivebox.core.models import ArchiveResult, Snapshot
|
||||
from archivebox.core.routes_util import (
|
||||
build_snapshot_detail_url,
|
||||
build_snapshot_files_url,
|
||||
build_snapshot_url,
|
||||
build_snapshot_zip_url,
|
||||
get_snapshot_output_anchor,
|
||||
)
|
||||
from archivebox.core.routes_util import build_snapshot_url
|
||||
from archivebox.core.widgets import InlineTagEditorWidget
|
||||
from archivebox.machine.env_util import env_to_shell_exports
|
||||
from archivebox.misc.logging_util import printable_filesize
|
||||
@@ -190,7 +186,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None, can_del
|
||||
if embed_path and result.status == "succeeded":
|
||||
output_link = build_snapshot_url(snapshot_id, embed_path, config=config)
|
||||
else:
|
||||
output_link = build_snapshot_detail_url(result.snapshot.archive_path_from_db, config=config)
|
||||
output_link = build_snapshot_url(snapshot_id, "", config=config)
|
||||
output_link_attr = html.escape(output_link, quote=True)
|
||||
|
||||
# Get version - try cmd_version field
|
||||
@@ -559,8 +555,8 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
try:
|
||||
queryset = self.get_queryset(request).filter(pk__in=selected)
|
||||
if not queryset.exists():
|
||||
snapshot = Snapshot.objects.filter(pk=request.GET.get("snapshot")).first()
|
||||
return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request) if snapshot else request.path)
|
||||
snapshot = Snapshot.objects.only("id").filter(pk=request.GET.get("snapshot")).first()
|
||||
return redirect(build_snapshot_url(str(snapshot.id), "index.html", request=request) if snapshot else request.path)
|
||||
except (ValidationError, ValueError):
|
||||
return redirect(request.path)
|
||||
return delete_selected(self, request, queryset)
|
||||
@@ -572,7 +568,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
request.GET.clear()
|
||||
request.META["QUERY_STRING"] = ""
|
||||
try:
|
||||
handoff_snapshot = Snapshot.objects.filter(pk=handoff_snapshot).first()
|
||||
handoff_snapshot = Snapshot.objects.only("id").filter(pk=handoff_snapshot).first()
|
||||
except (ValidationError, ValueError):
|
||||
handoff_snapshot = None
|
||||
saved_list_per_page = self.list_per_page
|
||||
@@ -586,7 +582,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
pk__in=request.POST.getlist(ACTION_CHECKBOX_NAME),
|
||||
).exists()
|
||||
):
|
||||
return redirect(build_snapshot_detail_url(handoff_snapshot.archive_path_from_db, request=request))
|
||||
return redirect(build_snapshot_url(str(handoff_snapshot.id), "index.html", request=request))
|
||||
return response
|
||||
finally:
|
||||
self.list_per_page = saved_list_per_page
|
||||
@@ -660,40 +656,22 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
|
||||
def get_snapshot_view_url(self, result: ArchiveResult) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_detail_url(
|
||||
result.snapshot.archive_path_from_db,
|
||||
request=request,
|
||||
config=request.archivebox_config,
|
||||
)
|
||||
return build_snapshot_url(str(result.snapshot_id), request=request, config=request.archivebox_config)
|
||||
|
||||
def get_output_view_url(self, result: ArchiveResult) -> str:
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
output_path = get_snapshot_output_anchor(result.plugin, result.embed_path() or "")
|
||||
return build_snapshot_detail_url(
|
||||
result.snapshot.archive_path_from_db,
|
||||
output_path=output_path,
|
||||
request=request,
|
||||
config=config,
|
||||
)
|
||||
output_path = result.embed_path()
|
||||
if not output_path:
|
||||
output_path = result.plugin or ""
|
||||
return build_snapshot_url(str(result.snapshot_id), output_path, request=request, config=config)
|
||||
|
||||
def get_output_files_url(self, result: ArchiveResult) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_files_url(
|
||||
str(result.snapshot_id),
|
||||
result.output_folder_path(),
|
||||
request=request,
|
||||
config=request.archivebox_config,
|
||||
)
|
||||
return f"{build_snapshot_url(str(result.snapshot_id), result.plugin, request=request, config=request.archivebox_config)}/?files=1"
|
||||
|
||||
def get_output_zip_url(self, result: ArchiveResult) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_zip_url(
|
||||
str(result.snapshot_id),
|
||||
result.output_folder_path(),
|
||||
request=request,
|
||||
config=request.archivebox_config,
|
||||
)
|
||||
return f"{self.get_output_files_url(result)}&download=zip"
|
||||
|
||||
@admin.display(description="Details", ordering="id")
|
||||
def details_link(self, result):
|
||||
@@ -751,11 +729,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
request = self.request
|
||||
return format_html(
|
||||
'<a href="{}"><b><code>[{}]</code></b> {} {}</a><br/>',
|
||||
build_snapshot_detail_url(
|
||||
result.snapshot.archive_path_from_db,
|
||||
request=request,
|
||||
config=request.archivebox_config,
|
||||
),
|
||||
build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
|
||||
snapshot_id[:8],
|
||||
result.snapshot.bookmarked_at.strftime("%Y-%m-%d %H:%M"),
|
||||
result.snapshot.url[:128],
|
||||
@@ -853,14 +827,11 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
config = request.archivebox_config
|
||||
# Determine output link path - use embed_path() which checks output_files
|
||||
embed_path = result.embed_path()
|
||||
output_url = (
|
||||
build_snapshot_url(str(result.snapshot_id), embed_path, request=request, config=config)
|
||||
if result.status == "succeeded" and embed_path
|
||||
else build_snapshot_detail_url(result.snapshot.archive_path_from_db, request=request, config=config)
|
||||
)
|
||||
output_path = embed_path if (result.status == "succeeded" and embed_path) else "index.html"
|
||||
snapshot_id = str(result.snapshot_id)
|
||||
return format_html(
|
||||
'<a href="{}" class="output-link">↗️</a><pre>{}</pre>',
|
||||
output_url,
|
||||
build_snapshot_url(snapshot_id, output_path, request=request, config=config),
|
||||
result.output_str_for_display(),
|
||||
)
|
||||
|
||||
@@ -887,16 +858,46 @@ class ArchiveResultAdmin(BaseModelAdmin):
|
||||
)
|
||||
|
||||
def output_summary(self, result):
|
||||
output_html = format_html('<pre style="display: inline-block">{}</pre>', result.output_str_for_display())
|
||||
file_map = result.output_file_map()
|
||||
paths = sorted({path for key in file_map if (path := result.output_file_path(key, output_file_map=file_map))})
|
||||
if paths:
|
||||
output_html += format_html(
|
||||
'<br/><a href="{}">See result files ...</a><pre><code>{}</code></pre>',
|
||||
self.get_output_files_url(result),
|
||||
"\n".join(paths),
|
||||
)
|
||||
return output_html
|
||||
snapshot_dir = Path(result.snapshot.output_dir)
|
||||
output_html = format_html(
|
||||
'<pre style="display: inline-block">{}</pre><br/>',
|
||||
result.output_str_for_display(),
|
||||
)
|
||||
snapshot_id = str(result.snapshot_id)
|
||||
request = self.request
|
||||
output_html += format_html(
|
||||
'<a href="{}#all">See result files ...</a><br/><pre><code>',
|
||||
build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
|
||||
)
|
||||
embed_path = result.embed_path() or ""
|
||||
path_from_embed = snapshot_dir / (embed_path or "")
|
||||
output_html += format_html(
|
||||
'<i style="padding: 1px">{}</i><b style="padding-right: 20px">/</b><i>{}</i><br/><hr/>',
|
||||
str(snapshot_dir),
|
||||
str(embed_path),
|
||||
)
|
||||
if os.access(path_from_embed, os.R_OK):
|
||||
root_dir = str(path_from_embed)
|
||||
else:
|
||||
root_dir = str(snapshot_dir)
|
||||
|
||||
for root, dirs, files in os.walk(root_dir):
|
||||
depth = root.replace(root_dir, "").count(os.sep) + 1
|
||||
if depth > 2:
|
||||
continue
|
||||
indent = " " * 4 * (depth)
|
||||
output_html += format_html('<b style="padding: 1px">{}{}/</b><br/>', indent, os.path.basename(root))
|
||||
indentation_str = " " * 4 * (depth + 1)
|
||||
for filename in sorted(files):
|
||||
is_hidden = filename.startswith(".")
|
||||
output_html += format_html(
|
||||
'<span style="opacity: {}.2">{}{}</span><br/>',
|
||||
int(not is_hidden),
|
||||
indentation_str,
|
||||
filename.strip(),
|
||||
)
|
||||
|
||||
return output_html + mark_safe("</code></pre>")
|
||||
|
||||
|
||||
def register_admin(admin_site):
|
||||
|
||||
+176
-248
@@ -1,56 +1,38 @@
|
||||
__package__ = "archivebox.core"
|
||||
|
||||
import json
|
||||
from math import ceil
|
||||
from typing import Any
|
||||
from functools import lru_cache
|
||||
from types import SimpleNamespace
|
||||
|
||||
from django.contrib import admin, messages
|
||||
from django.core.exceptions import PermissionDenied
|
||||
from django.urls import path, reverse
|
||||
from django.shortcuts import get_object_or_404, redirect
|
||||
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed, Http404
|
||||
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
|
||||
from django.utils import timezone
|
||||
from django.utils.formats import date_format
|
||||
from django.utils.html import format_html, format_html_join
|
||||
from django.utils.safestring import mark_safe
|
||||
from django.db.models import Q, Count, Exists, F, OuterRef, Prefetch
|
||||
from django import forms
|
||||
from django.template import Context, Template, RequestContext
|
||||
from django.template import Template, RequestContext
|
||||
from django.contrib.admin.helpers import ActionForm
|
||||
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.misc.util import htmldecode, urldecode
|
||||
from archivebox.misc.paginators import AcceleratedPaginator
|
||||
from archivebox.misc.logging_util import printable_filesize
|
||||
from archivebox.misc.serve_static import serve_static_with_byterange_support
|
||||
from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
|
||||
from archivebox.search.views import admin_snapshot_search_stream_view
|
||||
from archivebox.core.routes_util import (
|
||||
build_snapshot_detail_url,
|
||||
build_snapshot_files_url,
|
||||
build_snapshot_role_output_url,
|
||||
build_snapshot_url,
|
||||
build_snapshot_zip_url,
|
||||
get_snapshot_output_anchor,
|
||||
)
|
||||
from archivebox.core.routes_util import build_snapshot_url, build_web_url
|
||||
from archivebox.core.tag_util import get_or_create_tag
|
||||
from archivebox.core.templatetags.core_tags import snapshot_thumbnail, snapshot_url_text
|
||||
from archivebox.plugins.hooks import discover_hooks
|
||||
from archivebox.plugins.discovery import get_plugin_icon, get_snapshot_role_names
|
||||
from archivebox.plugins.output_groups import (
|
||||
OUTPUT_GROUPS,
|
||||
display_plugin_name,
|
||||
output_group_for_plugin,
|
||||
order_output_plugins,
|
||||
plugin_icon_is_hidden,
|
||||
plugin_output_sizes,
|
||||
)
|
||||
from archivebox.plugins.discovery import get_plugin_icon, get_plugin_name, get_plugins
|
||||
|
||||
from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
|
||||
|
||||
from archivebox.core.models import Tag, Snapshot, ArchiveResult
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.core.admin_archiveresults import render_archiveresults_list
|
||||
from archivebox.core.preview_util import EXTENSION_SCREENSHOT_PLUGIN
|
||||
from archivebox.progressmonitor.views import progress_endpoint
|
||||
from archivebox.core.permissions import (
|
||||
PERMISSIONS_CHOICES,
|
||||
@@ -66,12 +48,9 @@ GLOBAL_CONTEXT = {}
|
||||
SNAPSHOT_PERMISSION_META = PERMISSIONS_META
|
||||
|
||||
|
||||
def _format_size_column(num_bytes: int) -> str:
|
||||
size = num_bytes
|
||||
for unit in ("B", "KB", "MB", "GB", "TB"):
|
||||
if size < 1024 or unit == "TB":
|
||||
return f"{size:.1f} {unit}" if unit == "GB" else f"{ceil(size)} {unit}"
|
||||
size /= 1024
|
||||
@lru_cache(maxsize=1)
|
||||
def _plugin_sort_order() -> dict[str, int]:
|
||||
return {get_plugin_name(plugin): idx for idx, plugin in enumerate(get_plugins())}
|
||||
|
||||
|
||||
class SnapshotActionForm(ActionForm):
|
||||
@@ -283,28 +262,15 @@ class SnapshotChangeList(SearchResultsChangeList):
|
||||
rows = (
|
||||
ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
|
||||
.order_by("snapshot_id", "plugin")
|
||||
.only("snapshot_id", "plugin", "status", "output_str", "output_size", "output_files")
|
||||
.values_list("snapshot_id", "plugin", "status", "output_size", "output_files")
|
||||
)
|
||||
for result in rows.iterator(chunk_size=1000):
|
||||
if result.plugin in seen_plugins[result.snapshot_id]:
|
||||
for snapshot_id, plugin, status, output_size, output_files in rows.iterator(chunk_size=1000):
|
||||
if plugin in seen_plugins[snapshot_id]:
|
||||
continue
|
||||
seen_plugins[result.snapshot_id].add(result.plugin)
|
||||
output_results_by_snapshot[result.snapshot_id].append(result)
|
||||
|
||||
card_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
|
||||
for result in (
|
||||
ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids)
|
||||
.exclude(plugin="")
|
||||
.only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_size",
|
||||
"output_files",
|
||||
seen_plugins[snapshot_id].add(plugin)
|
||||
output_results_by_snapshot[snapshot_id].append(
|
||||
SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
|
||||
)
|
||||
):
|
||||
card_results_by_snapshot[result.snapshot_id].append(result)
|
||||
|
||||
for obj in self.result_list:
|
||||
counts = status_counts_by_snapshot.get(obj.pk, {})
|
||||
@@ -328,7 +294,6 @@ class SnapshotChangeList(SearchResultsChangeList):
|
||||
"is_sealed": obj.status not in (obj.StatusChoices.QUEUED, obj.StatusChoices.STARTED, obj.StatusChoices.PAUSED),
|
||||
}
|
||||
obj.__dict__["_admin_output_results"] = output_results_by_snapshot[obj.pk]
|
||||
obj.__dict__["_snapshot_card_results"] = card_results_by_snapshot[obj.pk]
|
||||
|
||||
def get_results(self, request):
|
||||
super().get_results(request)
|
||||
@@ -341,7 +306,6 @@ class SnapshotChangeList(SearchResultsChangeList):
|
||||
class SnapshotAdminForm(forms.ModelForm):
|
||||
"""Custom form for Snapshot admin with tag editor widget."""
|
||||
|
||||
url = forms.CharField(widget=forms.TextInput)
|
||||
tags_editor = forms.CharField(
|
||||
label="Tags",
|
||||
required=False,
|
||||
@@ -404,7 +368,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
|
||||
list_display = (
|
||||
"permissions_badge",
|
||||
"created_at_display",
|
||||
"created_at",
|
||||
"preview_icon",
|
||||
"title_str",
|
||||
"tags_inline",
|
||||
@@ -412,7 +376,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
"files",
|
||||
"size_with_stats",
|
||||
)
|
||||
list_display_links = ("created_at_display",)
|
||||
list_display_links = ("created_at",)
|
||||
sort_fields = ("title_str", "created_at", "status", "crawl")
|
||||
readonly_fields = (
|
||||
"admin_actions",
|
||||
@@ -526,15 +490,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
inlines = [] # Removed TagInline, using TagEditorWidget instead
|
||||
list_per_page = 50
|
||||
|
||||
@admin.display(description="Date", ordering="created_at")
|
||||
def created_at_display(self, obj):
|
||||
created_at = timezone.localtime(obj.created_at)
|
||||
return format_html(
|
||||
'<span class="snapshot-created-at"><span class="snapshot-created-at__date">{}</span><span class="snapshot-created-at__time">{}</span></span>',
|
||||
date_format(created_at, "Y-m-d"),
|
||||
date_format(created_at, "g:i A"),
|
||||
)
|
||||
|
||||
action_form = SnapshotActionForm
|
||||
paginator = AcceleratedPaginator
|
||||
|
||||
@@ -600,26 +555,20 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
|
||||
def get_snapshot_view_url(self, obj: Snapshot) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=request.archivebox_config)
|
||||
return build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)
|
||||
|
||||
def get_snapshot_files_url(self, obj: Snapshot) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_files_url(str(obj.id), request=request, config=request.archivebox_config)
|
||||
return f"{build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)}/?files=1"
|
||||
|
||||
def get_snapshot_zip_url(self, obj: Snapshot) -> str:
|
||||
request = self.request
|
||||
return build_snapshot_zip_url(str(obj.id), request=request, config=request.archivebox_config)
|
||||
return f"{self.get_snapshot_files_url(obj)}&download=zip"
|
||||
|
||||
def get_urls(self):
|
||||
urls = super().get_urls()
|
||||
custom_urls = [
|
||||
path("grid/", self.admin_site.admin_view(self.grid_view), name="grid"),
|
||||
path("search-stream/", self.admin_site.admin_view(self.search_stream_view), name="core_snapshot_search_stream"),
|
||||
path(
|
||||
"<path:object_id>/preview/<str:plugin>/<path:filename>",
|
||||
self.admin_site.admin_view(self.preview_view, cacheable=True),
|
||||
name="core_snapshot_preview",
|
||||
),
|
||||
path("<path:object_id>/redo-failed/", self.admin_site.admin_view(self.redo_failed_view), name="core_snapshot_redo_failed"),
|
||||
path(
|
||||
"<path:object_id>/set-permissions/",
|
||||
@@ -629,26 +578,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
]
|
||||
return custom_urls + urls
|
||||
|
||||
def preview_view(self, request, object_id, plugin, filename):
|
||||
"""Serve a saved image output from the authenticated admin origin."""
|
||||
snapshot = get_object_or_404(Snapshot, pk=object_id)
|
||||
if not self.has_view_or_change_permission(request, snapshot):
|
||||
raise PermissionDenied
|
||||
result = snapshot.archiveresult_set.filter(plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED).first()
|
||||
try:
|
||||
output_path = result.output_file_path(filename) if result else None
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
output_path = None
|
||||
if not output_path:
|
||||
raise Http404("Snapshot image does not exist")
|
||||
request.archivebox_cache_policy = "private"
|
||||
response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
|
||||
if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
|
||||
raise Http404("Snapshot preview is not an image")
|
||||
response.headers["Vary"] = "Cookie"
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
return response
|
||||
|
||||
def search_stream_view(self, request):
|
||||
return admin_snapshot_search_stream_view(self, request)
|
||||
|
||||
@@ -780,7 +709,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
|
||||
return qs
|
||||
|
||||
@admin.display(description="🔒", ordering="permissions")
|
||||
@admin.display(description="👁", ordering="permissions")
|
||||
def permissions_badge(self, obj):
|
||||
permissions = obj.__dict__.get("snapshot_permissions")
|
||||
if permissions is None:
|
||||
@@ -926,13 +855,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
def status_info(self, obj):
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
favicon_url = build_snapshot_role_output_url(
|
||||
obj,
|
||||
"list_icon",
|
||||
request=request,
|
||||
config=config,
|
||||
fallback_to_default=True,
|
||||
)
|
||||
favicon_url = build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config)
|
||||
return format_html(
|
||||
"""
|
||||
Archived: {} ({} files {})
|
||||
@@ -970,7 +893,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
show_title = bool(title_raw) and title_normalized != "pending..." and title_normalized != url_normalized
|
||||
css_class = "fetched" if show_title else "pending"
|
||||
|
||||
detail_url = build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config)
|
||||
detail_url = build_web_url(f"/{obj.archive_path_from_db}/index.html", request=request, config=config)
|
||||
title_html = ""
|
||||
if show_title:
|
||||
title_html = format_html(
|
||||
@@ -980,26 +903,14 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
urldecode(htmldecode(title_raw))[:128],
|
||||
)
|
||||
|
||||
favicon = self._get_favicon_data(obj) or {}
|
||||
favicon_html = (
|
||||
format_html(
|
||||
'<img class="snapshot-url-favicon" src="{}" alt="" loading="lazy" decoding="async" onerror="this.hidden=true">',
|
||||
favicon["url"],
|
||||
)
|
||||
if favicon.get("url")
|
||||
else ""
|
||||
)
|
||||
return format_html(
|
||||
'<a class="snapshot-title-detail-hitbox" href="{}" aria-label="Open snapshot details"></a>'
|
||||
"{}"
|
||||
'<div class="snapshot-title-url">'
|
||||
'<a class="snapshot-original-url" href="{}">{}<code style="user-select: all;">{}</code></a>'
|
||||
'<div style="font-size: 11px; color: #64748b; margin-top: 2px;">'
|
||||
'<a href="{}"><code style="user-select: all;">{}</code></a>'
|
||||
"</div>",
|
||||
detail_url,
|
||||
title_html,
|
||||
url_raw or obj.url,
|
||||
favicon_html,
|
||||
snapshot_url_text(url_raw or obj.url),
|
||||
(url_raw or obj.url)[:128],
|
||||
)
|
||||
|
||||
@admin.display(description="Tags", ordering="tag_count")
|
||||
@@ -1026,47 +937,90 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
)
|
||||
return mark_safe(f'<span class="tags-inline-editor">{tags_html}</span>')
|
||||
|
||||
def _get_favicon_data(self, obj):
|
||||
list_icon_plugins = set(get_snapshot_role_names("list_icon"))
|
||||
def _get_preview_data(self, obj):
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
results = self._get_prefetched_results(obj)
|
||||
if results is None:
|
||||
results = obj.archiveresult_set.filter(
|
||||
plugin__in=list_icon_plugins,
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
if results is not None:
|
||||
results = [r for r in results if r.plugin in ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")]
|
||||
else:
|
||||
results = list(
|
||||
obj.archiveresult_set.filter(plugin__in=("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")).only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_files",
|
||||
"output_str",
|
||||
),
|
||||
)
|
||||
urls = []
|
||||
for result in results:
|
||||
if result.plugin not in list_icon_plugins or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
|
||||
continue
|
||||
try:
|
||||
output_path = result.embed_path()
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
continue
|
||||
if output_path:
|
||||
urls.append(
|
||||
build_snapshot_url(str(obj.id), output_path, request=self.request, config=self.request.archivebox_config),
|
||||
)
|
||||
return {"url": urls[0], "fallbacks": ",".join(urls[1:])} if urls else None
|
||||
|
||||
def _render_thumbnail(self, obj, width="100px", height="100px"):
|
||||
return snapshot_thumbnail(
|
||||
Context({"request": self.request, "CONFIG": self.request.archivebox_config}),
|
||||
obj,
|
||||
admin=True,
|
||||
width=width,
|
||||
height=height,
|
||||
def result_output_path(result, filename: str) -> str | None:
|
||||
output_files = result.output_files or {}
|
||||
file_info = output_files.get(filename)
|
||||
if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
|
||||
return None
|
||||
if file_info.get("root_relative"):
|
||||
return filename
|
||||
return f"{result.plugin}/{filename}"
|
||||
|
||||
def result_urls(plugin: str, filenames: tuple[str, ...]) -> list[str]:
|
||||
urls: list[str] = []
|
||||
for result in results:
|
||||
if result.plugin != plugin or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
|
||||
continue
|
||||
for filename in filenames:
|
||||
output_path = result_output_path(result, filename)
|
||||
if output_path:
|
||||
urls.append(build_snapshot_url(str(obj.id), output_path, request=request, config=config))
|
||||
return urls
|
||||
|
||||
screenshot_urls = result_urls("screenshot", ("screenshot.png",))
|
||||
extension_screenshot_urls = result_urls(EXTENSION_SCREENSHOT_PLUGIN, ("screenshot-1.png", "screenshot.png"))
|
||||
favicon_urls = result_urls("favicon", ("favicon.ico",))
|
||||
|
||||
if not screenshot_urls and not extension_screenshot_urls and not favicon_urls:
|
||||
return None
|
||||
|
||||
all_screenshot_urls = [*screenshot_urls, *extension_screenshot_urls]
|
||||
if all_screenshot_urls:
|
||||
img_url = all_screenshot_urls[0]
|
||||
fallbacks = [*all_screenshot_urls[1:], *favicon_urls]
|
||||
img_alt = "Screenshot"
|
||||
preview_class = "screenshot"
|
||||
else:
|
||||
img_url = favicon_urls[0]
|
||||
fallbacks = favicon_urls[1:]
|
||||
img_alt = "Favicon"
|
||||
preview_class = "favicon"
|
||||
|
||||
fallback_list = ",".join(fallbacks)
|
||||
onerror_js = (
|
||||
"this.dataset.fallbacks && this.dataset.fallbacks.length ? "
|
||||
"(this.src=this.dataset.fallbacks.split(',').shift(), "
|
||||
"this.dataset.fallbacks=this.dataset.fallbacks.split(',').slice(1).join(',')) : "
|
||||
"this.remove()"
|
||||
)
|
||||
|
||||
return {
|
||||
"img_url": img_url,
|
||||
"img_alt": img_alt,
|
||||
"preview_class": preview_class,
|
||||
"onerror_js": onerror_js,
|
||||
"fallback_list": fallback_list,
|
||||
"favicon_url": favicon_urls[0] if favicon_urls else "",
|
||||
"favicon_fallback_list": ",".join(favicon_urls[1:]),
|
||||
}
|
||||
|
||||
@admin.display(description="", empty_value="")
|
||||
def url_favicon(self, obj):
|
||||
favicon = self._get_favicon_data(obj)
|
||||
if not favicon:
|
||||
preview = self._get_preview_data(obj)
|
||||
if not preview:
|
||||
return ""
|
||||
|
||||
favicon_url = favicon.get("url") or ""
|
||||
favicon_url = preview.get("favicon_url") or ""
|
||||
if not favicon_url:
|
||||
return ""
|
||||
fallback_list = favicon.get("fallbacks") or ""
|
||||
fallback_list = preview.get("favicon_fallback_list") or ""
|
||||
onerror_js = (
|
||||
"this.dataset.fallbacks && this.dataset.fallbacks.length ? "
|
||||
"(this.src=this.dataset.fallbacks.split(',').shift(), "
|
||||
@@ -1087,24 +1041,42 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
|
||||
@admin.display(description="Preview", empty_value="")
|
||||
def preview_icon(self, obj):
|
||||
preview = self._get_preview_data(obj)
|
||||
if not preview:
|
||||
return None
|
||||
|
||||
return format_html(
|
||||
'<a href="{}" title="Open snapshot details">{}</a>',
|
||||
build_snapshot_detail_url(obj.archive_path_from_db, request=self.request, config=self.request.archivebox_config),
|
||||
self._render_thumbnail(obj, height="38px"),
|
||||
'<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">',
|
||||
preview["img_url"],
|
||||
preview["img_alt"],
|
||||
preview["preview_class"],
|
||||
preview["onerror_js"],
|
||||
preview["fallback_list"],
|
||||
)
|
||||
|
||||
@admin.display(description=" ", empty_value="")
|
||||
def snapshot_summary(self, obj):
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
preview = self._get_preview_data(obj)
|
||||
stats = self._get_progress_stats(obj)
|
||||
archive_size = stats["output_size"] or 0
|
||||
size_txt = printable_filesize(archive_size) if archive_size else "pending"
|
||||
screenshot_html = format_html(
|
||||
'<a href="{}" title="Open snapshot live view" style="display:block;flex:0 0 220px;">{}</a>',
|
||||
build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config),
|
||||
self._render_thumbnail(obj, width="220px", height="138px"),
|
||||
)
|
||||
screenshot_html = ""
|
||||
|
||||
if preview:
|
||||
screenshot_html = format_html(
|
||||
'<a href="{href}" title="Open snapshot live view" style="display:block; flex:0 0 220px; width:220px;">'
|
||||
'<img src="{src}" alt="{alt}" decoding="async" loading="lazy" onerror="{onerror}" data-fallbacks="{fallbacks}" '
|
||||
'style="display:block; width:100%; max-width:220px; aspect-ratio: 16 / 10; object-fit: cover; object-position: top; '
|
||||
'border-radius: 10px; border: 1px solid #e2e8f0; background: #f8fafc;">'
|
||||
"</a>",
|
||||
href=build_web_url(f"/{obj.archive_path}", request=request, config=config),
|
||||
src=preview["img_url"],
|
||||
alt=preview["img_alt"],
|
||||
onerror=preview["onerror_js"],
|
||||
fallbacks=preview["fallback_list"],
|
||||
)
|
||||
|
||||
return format_html(
|
||||
'<div style="display:flex; gap:16px; align-items:flex-start;">'
|
||||
@@ -1119,19 +1091,18 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
"</div>",
|
||||
screenshot_html,
|
||||
size_txt,
|
||||
self.get_snapshot_files_url(obj),
|
||||
obj.archive_path_from_db,
|
||||
build_web_url(f"/{obj.archive_path}", request=request, config=config),
|
||||
obj.archive_path,
|
||||
)
|
||||
|
||||
@admin.display(
|
||||
description="Files Saved",
|
||||
ordering="ar_succeeded_count",
|
||||
empty_value="",
|
||||
)
|
||||
def files(self, obj):
|
||||
results = self._get_prefetched_results(obj)
|
||||
if results is None:
|
||||
results = obj.archiveresult_set.only("plugin", "status", "output_size", "output_files")
|
||||
results = obj.archiveresult_set.only("plugin", "status", "output_size")
|
||||
|
||||
plugins_with_output: dict[str, ArchiveResult] = {}
|
||||
for result in results:
|
||||
@@ -1142,111 +1113,56 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
plugins_with_output.setdefault(result.plugin, result)
|
||||
|
||||
if not plugins_with_output:
|
||||
return ""
|
||||
return mark_safe('<span style="opacity: 0.35;">...</span>')
|
||||
|
||||
sorted_results = [plugins_with_output[plugin] for plugin in order_output_plugins(plugins_with_output, plugin_output_sizes(results))]
|
||||
icons_by_group: dict[str, list[dict[str, Any]]] = {group_id: [] for group_id, _, _ in OUTPUT_GROUPS}
|
||||
sorted_results = sorted(
|
||||
plugins_with_output.values(),
|
||||
key=lambda result: (_plugin_sort_order().get(result.plugin, 9999), result.plugin),
|
||||
)
|
||||
visible_results = sorted_results[:14]
|
||||
output = []
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
for result in sorted_results:
|
||||
if plugin_icon_is_hidden(result.plugin):
|
||||
continue
|
||||
for result in visible_results:
|
||||
icon = mark_safe(get_plugin_icon(result.plugin))
|
||||
if not icon.strip():
|
||||
continue
|
||||
embed_path = result.embed_path()
|
||||
if not embed_path:
|
||||
continue
|
||||
group_id = output_group_for_plugin(result.plugin)
|
||||
anchor_path = get_snapshot_output_anchor(result.plugin, embed_path)
|
||||
icons_by_group[group_id].append(
|
||||
{
|
||||
"plugin": result.plugin,
|
||||
"icon": icon,
|
||||
"href": build_snapshot_detail_url(
|
||||
obj.archive_path_from_db,
|
||||
output_path=anchor_path,
|
||||
request=request,
|
||||
config=config,
|
||||
),
|
||||
},
|
||||
output.append(
|
||||
format_html(
|
||||
'<a href="{}" class="exists-True" title="{}">{}</a>',
|
||||
build_web_url(f"/{obj.archive_path_from_db}/{result.plugin}/", request=request, config=config),
|
||||
result.plugin,
|
||||
icon,
|
||||
),
|
||||
)
|
||||
if len(sorted_results) > len(visible_results):
|
||||
output.append(
|
||||
format_html(
|
||||
'<span title="{} more outputs">+{}</span>',
|
||||
len(sorted_results) - len(visible_results),
|
||||
len(sorted_results) - len(visible_results),
|
||||
),
|
||||
)
|
||||
|
||||
output = []
|
||||
for group_id, label, _ in OUTPUT_GROUPS:
|
||||
icons = icons_by_group[group_id]
|
||||
if icons:
|
||||
top = icons[0]
|
||||
front_index = 0
|
||||
pile_column = len(output)
|
||||
top_html = format_html(
|
||||
'<a href="{}" class="exists-True files-icon-pile-top files-icon-pile-cover files-icon-plugin--{}" aria-label="{}" data-tooltip="{}" style="--files-icon-front-index:{};--files-icon-front-offset:{}px">{}</a>',
|
||||
top["href"],
|
||||
top["plugin"],
|
||||
label,
|
||||
label,
|
||||
front_index,
|
||||
front_index * 2,
|
||||
top["icon"],
|
||||
)
|
||||
members = []
|
||||
for icon_index, item in enumerate(icons):
|
||||
row = icon_index // 5
|
||||
row_count = min(5, len(icons) - row * 5)
|
||||
member_top = 8 + row * 18
|
||||
row_offset = (5 - row_count) * 11 + 3
|
||||
member_left = 3 + row_offset + (icon_index % 5) * 22
|
||||
members.append(
|
||||
format_html(
|
||||
'<a href="{}" class="exists-True files-icon-plugin--{}" aria-label="{}" data-tooltip="{}" style="--files-icon-index:{};--files-icon-row:{};--files-icon-column:{};--files-icon-row-offset:{}px;--files-icon-member-left:{}px;--files-icon-member-top:{}px">{}</a>',
|
||||
item["href"],
|
||||
item["plugin"],
|
||||
display_plugin_name(item["plugin"]),
|
||||
item["plugin"],
|
||||
icon_index,
|
||||
row,
|
||||
icon_index % 5,
|
||||
row_offset,
|
||||
member_left,
|
||||
member_top,
|
||||
item["icon"],
|
||||
),
|
||||
)
|
||||
popup = format_html(
|
||||
'<span class="files-icon-pile-popup" popover="manual">{}<span class="files-icon-pile-label">{}</span></span>',
|
||||
mark_safe("".join(str(icon) for icon in members)),
|
||||
label,
|
||||
)
|
||||
pile_contents = format_html("{}{}", top_html, popup)
|
||||
output.append(
|
||||
format_html(
|
||||
'<span class="files-icon-pile files-icon-pile--{}" role="group" aria-label="{} saved outputs" style="--files-icon-popup-left:{}px;--files-icon-rows:{};--files-icon-card-width:{}px">{}</span>',
|
||||
group_id,
|
||||
label,
|
||||
pile_column * -22 - front_index * 2,
|
||||
(len(icons) + 4) // 5,
|
||||
116,
|
||||
pile_contents,
|
||||
),
|
||||
)
|
||||
|
||||
return format_html(
|
||||
'<span class="files-icons files-icons--compact" style="font-size: 1em;">{}</span>',
|
||||
'<span class="files-icons files-icons--compact" style="font-size: 1em; opacity: 0.8;">{}</span>',
|
||||
mark_safe("".join(output)),
|
||||
)
|
||||
|
||||
@admin.display(empty_value="")
|
||||
@admin.display()
|
||||
def size(self, obj):
|
||||
request = self.request
|
||||
config = request.archivebox_config
|
||||
archive_size = self._get_progress_stats(obj)["output_size"] or 0
|
||||
if archive_size:
|
||||
size_txt = _format_size_column(archive_size)
|
||||
size_txt = printable_filesize(archive_size)
|
||||
if archive_size > 52428800:
|
||||
size_txt = mark_safe(f"<b>{size_txt}</b>")
|
||||
else:
|
||||
return ""
|
||||
size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
|
||||
return format_html(
|
||||
'<a href="{}" title="View all files">{}</a>',
|
||||
self.get_snapshot_files_url(obj),
|
||||
build_web_url(f"/{obj.archive_path}", request=request, config=config),
|
||||
size_txt,
|
||||
)
|
||||
|
||||
@@ -1316,7 +1232,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
@admin.display(
|
||||
description="Size",
|
||||
ordering="output_size",
|
||||
empty_value="",
|
||||
)
|
||||
def size_with_stats(self, obj):
|
||||
"""Show archive size with output size from archive results."""
|
||||
@@ -1325,11 +1240,24 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
size_bytes = output_size or 0
|
||||
|
||||
if size_bytes:
|
||||
size_txt = _format_size_column(size_bytes)
|
||||
size_txt = printable_filesize(size_bytes)
|
||||
if size_bytes > 52428800: # 50MB
|
||||
size_txt = mark_safe(f"<b>{size_txt}</b>")
|
||||
else:
|
||||
return ""
|
||||
size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
|
||||
|
||||
# Show hook statistics
|
||||
if stats["total"] > 0:
|
||||
return format_html(
|
||||
'<a href="{}" title="View all files" style="white-space: nowrap;">'
|
||||
"{}</a>"
|
||||
'<div style="font-size: 10px; color: #94a3b8; margin-top: 2px;">'
|
||||
"{}/{} hooks</div>",
|
||||
self.get_snapshot_files_url(obj),
|
||||
size_txt,
|
||||
stats["succeeded"],
|
||||
stats["total"],
|
||||
)
|
||||
|
||||
return format_html(
|
||||
'<a href="{}" title="View all files">{}</a>',
|
||||
@@ -1457,9 +1385,9 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
|
||||
)
|
||||
def url_str(self, obj):
|
||||
return format_html(
|
||||
'<a class="snapshot-original-url" href="{}"><code style="user-select: all;">{}</code></a>',
|
||||
'<a href="{}"><code style="user-select: all;">{}</code></a>',
|
||||
obj.url,
|
||||
snapshot_url_text(obj.url),
|
||||
obj.url[:128],
|
||||
)
|
||||
|
||||
@admin.display(description="Health", ordering="health")
|
||||
|
||||
@@ -7,13 +7,12 @@ from django.contrib import admin, messages
|
||||
from django.contrib.admin.options import IS_POPUP_VAR
|
||||
from django.http import HttpRequest, HttpResponseRedirect
|
||||
from django.template.response import TemplateResponse
|
||||
from django.db.models import Prefetch
|
||||
from django.urls import reverse
|
||||
from django.utils.html import format_html
|
||||
from django.utils.safestring import mark_safe
|
||||
|
||||
from archivebox.base_models.admin import BaseModelAdmin
|
||||
from archivebox.core.models import ArchiveResult, SnapshotTag, Tag
|
||||
from archivebox.core.models import SnapshotTag, Tag
|
||||
from archivebox.core.tag_util import (
|
||||
TAG_HAS_SNAPSHOTS_CHOICES,
|
||||
TAG_SORT_CHOICES,
|
||||
@@ -25,8 +24,7 @@ from archivebox.core.tag_util import (
|
||||
normalize_has_snapshots_filter,
|
||||
normalize_tag_sort,
|
||||
)
|
||||
from archivebox.core.routes_util import build_snapshot_role_output_url
|
||||
from archivebox.plugins.discovery import get_snapshot_role_names
|
||||
from archivebox.core.routes_util import build_snapshot_url
|
||||
|
||||
|
||||
class TagInline(admin.TabularInline):
|
||||
@@ -193,26 +191,7 @@ class TagAdmin(BaseModelAdmin):
|
||||
|
||||
@admin.display(description="Snapshots")
|
||||
def snapshots(self, tag: Tag):
|
||||
snapshots = (
|
||||
tag.snapshot_set.select_related("crawl__created_by")
|
||||
.prefetch_related(
|
||||
Prefetch(
|
||||
"archiveresult_set",
|
||||
queryset=ArchiveResult.objects.filter(
|
||||
plugin__in=get_snapshot_role_names("list_icon"),
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
).only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_files",
|
||||
),
|
||||
to_attr="_list_icon_results",
|
||||
),
|
||||
)
|
||||
.order_by("-downloaded_at", "-created_at", "-pk")[:10]
|
||||
)
|
||||
snapshots = tag.snapshot_set.select_related("crawl__created_by").order_by("-downloaded_at", "-created_at", "-pk")[:10]
|
||||
total_count = tag.snapshot_set.count()
|
||||
if not snapshots:
|
||||
return mark_safe(
|
||||
@@ -235,12 +214,7 @@ class TagAdmin(BaseModelAdmin):
|
||||
</a>
|
||||
""",
|
||||
reverse("admin:core_snapshot_change", args=[snapshot.pk]),
|
||||
build_snapshot_role_output_url(
|
||||
snapshot,
|
||||
"list_icon",
|
||||
archive_results=snapshot._list_icon_results,
|
||||
fallback_to_default=True,
|
||||
),
|
||||
build_snapshot_url(str(snapshot.pk), "favicon.ico"),
|
||||
title[:120],
|
||||
snapshot.url[:120],
|
||||
),
|
||||
|
||||
+3
-23
@@ -7,31 +7,11 @@ For more information on this file, see
|
||||
https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
|
||||
"""
|
||||
|
||||
from django.core.handlers.asgi import ASGIHandler
|
||||
from django.core.asgi import get_asgi_application
|
||||
|
||||
from archivebox.config.django import setup_django
|
||||
|
||||
setup_django(check_db=True)
|
||||
|
||||
|
||||
class ArchiveBoxASGIHandler(ASGIHandler):
|
||||
def create_request(self, scope, body_file):
|
||||
# ASGI already spools the complete body. Django's multipart parser
|
||||
# otherwise treats chunked uploads without Content-Length as empty.
|
||||
if not any(name == b"content-length" for name, _value in scope.get("headers", [])):
|
||||
body_file.seek(0, 2)
|
||||
length = body_file.tell()
|
||||
body_file.seek(0)
|
||||
scope = {**scope, "headers": [*scope.get("headers", []), (b"content-length", str(length).encode("ascii"))]}
|
||||
return super().create_request(scope, body_file)
|
||||
|
||||
|
||||
django_application = ArchiveBoxASGIHandler()
|
||||
|
||||
|
||||
async def application(scope, receive, send):
|
||||
if scope["type"] == "websocket":
|
||||
from archivebox.opencode.views import websocket_view
|
||||
|
||||
return await websocket_view(scope, receive, send)
|
||||
return await django_application(scope, receive, send)
|
||||
# Standard Django ASGI application (no websockets/channels needed)
|
||||
application = get_asgi_application()
|
||||
@@ -13,7 +13,6 @@ from django.http import HttpResponseForbidden, HttpResponseNotModified
|
||||
from django.shortcuts import redirect
|
||||
from django.utils import timezone
|
||||
from django.utils.http import http_date
|
||||
from django.utils.cache import patch_cache_control, patch_vary_headers
|
||||
|
||||
from archivebox.config import VERSION
|
||||
from archivebox.config.common import get_config, get_request_config
|
||||
@@ -33,7 +32,6 @@ from archivebox.core.routes_util import (
|
||||
split_host_port,
|
||||
)
|
||||
from archivebox.core.views import OriginalDomainHostView, SnapshotHostView
|
||||
from archivebox.misc.serve_static import FAVICON_CACHE_CONTROL
|
||||
|
||||
ADMIN_LOGIN_HINT_COOKIE = "archivebox_admin_logged_in"
|
||||
|
||||
@@ -91,7 +89,13 @@ def AdminCookieIsolationMiddleware(get_response):
|
||||
response = get_response(request)
|
||||
|
||||
if request.path == "/admin" or request.path.startswith("/admin/"):
|
||||
if not getattr(response, "xframe_options_exempt", False):
|
||||
from archivebox.opencode.views import _PROXY_PREFIX
|
||||
|
||||
is_opencode_proxy = request.path == _PROXY_PREFIX or request.path.startswith(f"{_PROXY_PREFIX}/")
|
||||
if is_opencode_proxy:
|
||||
response.headers["X-Frame-Options"] = "SAMEORIGIN"
|
||||
response.headers["Content-Security-Policy"] = "frame-ancestors 'self'"
|
||||
else:
|
||||
response.headers["X-Frame-Options"] = "DENY"
|
||||
response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
|
||||
|
||||
@@ -125,12 +129,13 @@ def AdminCookieIsolationMiddleware(get_response):
|
||||
|
||||
|
||||
def CacheControlMiddleware(get_response):
|
||||
snapshot_path_re = re.compile(r"^/[^/]+/\\d{8}/[^/]+/[0-9a-fA-F-]{8,36}/")
|
||||
static_cache_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
|
||||
|
||||
def middleware(request):
|
||||
response = get_response(request)
|
||||
|
||||
if request.path.startswith("/static/") and response.status_code in (200, 304):
|
||||
if request.path.startswith("/static/"):
|
||||
rel_path = request.path[len("/static/") :]
|
||||
static_path = finders.find(rel_path)
|
||||
if static_path:
|
||||
@@ -155,30 +160,15 @@ def CacheControlMiddleware(get_response):
|
||||
response.headers["Last-Modified"] = http_date(mtime)
|
||||
return response
|
||||
|
||||
if response.status_code in (200, 206, 304) and response.get("Cache-Control") == FAVICON_CACHE_CONTROL:
|
||||
# The same raw favicon bytes are shared across authenticated viewers.
|
||||
# Session middleware may have added Cookie to Vary during authorization.
|
||||
vary = [
|
||||
value.strip()
|
||||
for value in response.get("Vary", "").split(",")
|
||||
if value.strip().lower() not in {"", "cookie", "authorization"}
|
||||
]
|
||||
if vary:
|
||||
response["Vary"] = ", ".join(vary)
|
||||
else:
|
||||
response.headers.pop("Vary", None)
|
||||
return response
|
||||
|
||||
# Other private/unlisted artifacts remain private; auth errors stay unstored.
|
||||
snapshot_policy = getattr(request, "archivebox_cache_policy", None)
|
||||
if snapshot_policy and response.status_code in (200, 206, 304):
|
||||
if not response.get("Cache-Control"):
|
||||
response["Cache-Control"] = f"{snapshot_policy}, max-age=60, stale-while-revalidate=300"
|
||||
if snapshot_policy == "private":
|
||||
patch_cache_control(response, private=True)
|
||||
elif snapshot_policy or not response.get("Cache-Control"):
|
||||
response["Cache-Control"] = "private, no-store"
|
||||
patch_vary_headers(response, ("Cookie", "Authorization"))
|
||||
if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
|
||||
"Cache-Control",
|
||||
):
|
||||
config = request.__dict__.get("archivebox_config")
|
||||
if config is None:
|
||||
config = get_config(resolve_plugins=False)
|
||||
request.archivebox_config = config
|
||||
policy = "private" if config.PERMISSIONS == "private" else "public"
|
||||
response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
|
||||
return response
|
||||
|
||||
return middleware
|
||||
|
||||
@@ -100,12 +100,11 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
|
||||
if schema_editor.connection.vendor != "sqlite":
|
||||
return
|
||||
|
||||
from django.utils import timezone
|
||||
from django.db import connection
|
||||
import uuid as uuid_lib
|
||||
from datetime import datetime
|
||||
|
||||
connection = schema_editor.connection
|
||||
cursor = connection.cursor()
|
||||
now = connection.ops.adapt_datetimefield_value(timezone.now())
|
||||
|
||||
# Check if there are any snapshots without a crawl
|
||||
cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
|
||||
@@ -126,13 +125,14 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
|
||||
INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
|
||||
VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
|
||||
""",
|
||||
[now],
|
||||
[datetime.now().isoformat()],
|
||||
)
|
||||
|
||||
# Create a default crawl for migrated snapshots.
|
||||
# Depending on migration graph order, later crawls migrations may already
|
||||
# have removed output_dir by the time this data migration runs.
|
||||
crawl_id = uuid_lib.uuid4().hex
|
||||
now = datetime.now().isoformat()
|
||||
cursor.execute("PRAGMA table_info(crawls_crawl)")
|
||||
crawl_columns = {row[1] for row in cursor.fetchall()}
|
||||
default_columns: list[str] = []
|
||||
@@ -155,9 +155,9 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
|
||||
status, retry_at, created_by_id, schedule_id, config, persona_id
|
||||
) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
|
||||
'Auto-created crawl for migrated snapshots', {default_values_sql}
|
||||
'sealed', NULL, 1, NULL, '{{}}', NULL)
|
||||
'sealed', %s, 1, NULL, '{{}}', NULL)
|
||||
""",
|
||||
[crawl_id, now, now, crawl_urls],
|
||||
[crawl_id, now, now, crawl_urls, now],
|
||||
)
|
||||
|
||||
# Assign all snapshots without a crawl to the default crawl
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Repair SQLite queue timestamps written as raw ISO strings by old migrations."""
|
||||
|
||||
from django.db import migrations
|
||||
from django.utils.dateparse import parse_datetime
|
||||
|
||||
|
||||
def repair_legacy_retry_timestamps(apps, schema_editor):
|
||||
connection = schema_editor.connection
|
||||
if connection.vendor != "sqlite":
|
||||
return
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 0024 created this Crawl already sealed, but left a due retry marker.
|
||||
# Its orphaned snapshots have their own scheduler rows for maintenance.
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE crawls_crawl SET retry_at = NULL
|
||||
WHERE status = 'sealed'
|
||||
AND label = 'Migrated from v0.7.2/v0.8.6'
|
||||
AND notes = 'Auto-created crawl for migrated snapshots'
|
||||
AND substr(retry_at, 11, 1) = 'T'
|
||||
""",
|
||||
)
|
||||
|
||||
# SQLite compares stored date strings lexically. A raw `T` timestamp
|
||||
# from a legacy migration sorts differently from Django's space-separated
|
||||
# parameter, so a due row can be selected but never claimed. Preserve
|
||||
# the datetime Django already reads from each row while rewriting its
|
||||
# storage representation through the active database adapter.
|
||||
for table in ("crawls_crawl", "core_snapshot", "machine_binary", "machine_process"):
|
||||
last_id = ""
|
||||
while True:
|
||||
cursor.execute(
|
||||
f"SELECT id, CAST(retry_at AS TEXT) FROM {table} WHERE id > %s AND substr(retry_at, 11, 1) = 'T' ORDER BY id LIMIT 500",
|
||||
[last_id],
|
||||
)
|
||||
rows = cursor.fetchall()
|
||||
if not rows:
|
||||
break
|
||||
for row_id, raw_retry_at in rows:
|
||||
try:
|
||||
parsed = parse_datetime(raw_retry_at)
|
||||
except ValueError:
|
||||
continue
|
||||
if parsed is None:
|
||||
continue
|
||||
normalized = connection.ops.adapt_datetimefield_value(parsed)
|
||||
cursor.execute(
|
||||
f"UPDATE {table} SET retry_at = %s WHERE id = %s AND retry_at = %s",
|
||||
[normalized, row_id, raw_retry_at],
|
||||
)
|
||||
last_id = rows[-1][0]
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("core", "0054_snapshot_fs_version_index"),
|
||||
]
|
||||
|
||||
operations = [migrations.RunPython(repair_legacy_retry_timestamps, migrations.RunPython.noop)]
|
||||
@@ -1,25 +0,0 @@
|
||||
"""Repair process links after both core's legacy copy and machine migrations."""
|
||||
|
||||
from importlib import import_module
|
||||
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
repair_process_binary_iface_links = import_module(
|
||||
"archivebox.machine.migrations.0020_repair_process_binary_iface_links",
|
||||
).repair_process_binary_iface_links
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
atomic = False
|
||||
|
||||
dependencies = [
|
||||
("core", "0055_repair_legacy_retry_timestamps"),
|
||||
("machine", "0022_networkinterface_identity_without_mac"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
# The original repair can run before core.0027 creates legacy Process
|
||||
# rows. Repair again after the copy, including already upgraded DBs.
|
||||
migrations.RunPython(repair_process_binary_iface_links, migrations.RunPython.noop),
|
||||
]
|
||||
+278
-534
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
__package__ = "archivebox.core"
|
||||
|
||||
|
||||
EXTENSION_SCREENSHOT_PLUGIN = "chrome_extension_screenshot"
|
||||
@@ -74,9 +74,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
|
||||
**snapshot_filter,
|
||||
).update(retry_at=now, modified_at=now)
|
||||
|
||||
# Find rare uploads through hook_name's index before checking their parents;
|
||||
# a correlated upload probe otherwise visits every sealed snapshot.
|
||||
extension_upload_results = ArchiveResult.objects.filter(
|
||||
snapshot_id=OuterRef("pk"),
|
||||
hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
|
||||
)
|
||||
server_results = ArchiveResult.objects.filter(snapshot_id=OuterRef("pk")).exclude(
|
||||
@@ -84,15 +83,15 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
|
||||
)
|
||||
extension_only_snapshots = (
|
||||
Snapshot.objects.filter(
|
||||
id__in=extension_upload_results.values("snapshot_id"),
|
||||
status=Snapshot.StatusChoices.SEALED,
|
||||
crawl__status__in=[Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED],
|
||||
**snapshot_filter,
|
||||
)
|
||||
.annotate(
|
||||
has_extension_upload=Exists(extension_upload_results),
|
||||
has_server_result=Exists(server_results),
|
||||
)
|
||||
.filter(has_server_result=False)
|
||||
.filter(has_extension_upload=True, has_server_result=False)
|
||||
)
|
||||
# Older browser-extension uploads could win a race with runner startup:
|
||||
# their successful external rows made the fresh Snapshot look finished
|
||||
@@ -141,13 +140,6 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
|
||||
orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
|
||||
else:
|
||||
orphaned_hook_processes = orphaned_hook_processes.none()
|
||||
else:
|
||||
# Global startup recovery must stay database-only. Process.pwd points
|
||||
# into archive payload storage, which may be a cold FUSE mount; scanning
|
||||
# every historical unlinked hook here blocks the sole scheduler before
|
||||
# it can claim newly queued work. Explicit crawl recovery below remains
|
||||
# available for the work the caller asked to resume.
|
||||
orphaned_hook_processes = orphaned_hook_processes.none()
|
||||
for process in orphaned_hook_processes.only(
|
||||
"id",
|
||||
"pwd",
|
||||
|
||||
+23
-114
@@ -2,7 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qsl, quote, urlencode, urlparse, urlsplit, urlunsplit
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from django.utils.http import url_has_allowed_host_and_scheme
|
||||
|
||||
@@ -11,8 +11,6 @@ from archivebox.config.common import get_config, get_request_config
|
||||
_SNAPSHOT_ID_RE = re.compile(r"^[0-9a-fA-F-]{8,36}$")
|
||||
_SNAPSHOT_SUBDOMAIN_RE = re.compile(r"^snap-(?P<suffix>[0-9a-fA-F]{12})$")
|
||||
_ROLE_SUBDOMAIN_LABELS = ("admin", "web", "api")
|
||||
_URL_PATH_SAFE = "/@-._~!$&'()*+,;="
|
||||
_URL_FRAGMENT_SAFE = "/@-._~!$&'()*+,;="
|
||||
|
||||
|
||||
def split_host_port(host: str) -> tuple[str, str | None]:
|
||||
@@ -62,6 +60,13 @@ def _csrf_trusted_origins(config) -> list[str]:
|
||||
return seen
|
||||
|
||||
|
||||
def _allowed_hosts(config) -> set[str]:
|
||||
raw = (config.ALLOWED_HOSTS or "").strip()
|
||||
if not raw:
|
||||
return set()
|
||||
return {entry.strip().lower() for entry in raw.split(",") if entry.strip() and entry.strip() != "*"}
|
||||
|
||||
|
||||
def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
"""Pick a single CSRF_TRUSTED_ORIGINS entry to act as the implicit BASE_URL.
|
||||
|
||||
@@ -69,7 +74,7 @@ def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwa
|
||||
had ``CSRF_TRUSTED_ORIGINS=https://archive.example.com`` set (required for
|
||||
admin login to work). On upgrade, ``BASE_URL`` is the new knob — but it
|
||||
defaults to empty, and falling through to ``BIND_ADDR`` produces an
|
||||
unreachable URL like ``http://0.0.0.0:5797``. If the user has exactly one
|
||||
unreachable URL like ``http://0.0.0.0:8000``. If the user has exactly one
|
||||
CSRF origin we treat it as the implicit BASE_URL so links/redirects keep
|
||||
pointing at the public hostname they already configured.
|
||||
|
||||
@@ -88,6 +93,11 @@ def get_listen_host(config: dict[str, Any] | None = None, **config_kwargs: Any)
|
||||
return (config.BIND_ADDR or "").strip()
|
||||
|
||||
|
||||
def get_listen_parts(config: dict[str, Any] | None = None, **config_kwargs: Any) -> tuple[str, str | None]:
|
||||
config = config or get_config(**config_kwargs)
|
||||
return split_host_port(get_listen_host(config=config))
|
||||
|
||||
|
||||
def _with_port(host: str, port: str | None) -> str:
|
||||
return f"{host}:{port}" if port else host
|
||||
|
||||
@@ -123,7 +133,7 @@ def canonical_base_host_for_request(request_host: str) -> str:
|
||||
"""Strip role subdomains and remap loopback hostnames to ``archivebox.localhost``.
|
||||
|
||||
Used by the banner suggestion and the in-browser pin endpoint: when the
|
||||
user is hitting the server on raw ``localhost:5797`` or ``127.0.0.1:5797``
|
||||
user is hitting the server on raw ``localhost:9292`` or ``127.0.0.1:9292``
|
||||
we want to suggest the wildcard-friendly ``archivebox.localhost`` family
|
||||
instead, so the eventual pinned ``BASE_URL`` plays nicely with subdomain
|
||||
routing without forcing the user to add a /etc/hosts entry.
|
||||
@@ -136,7 +146,7 @@ def canonical_base_host_for_request(request_host: str) -> str:
|
||||
|
||||
def _root_host_from_listen(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
config = config or get_config(**config_kwargs)
|
||||
listen_host, listen_port = split_host_port(get_listen_host(config=config))
|
||||
listen_host, listen_port = get_listen_parts(config=config)
|
||||
root_host = "archivebox.localhost" if _is_local_bind_host(listen_host) else listen_host
|
||||
return _with_port(root_host, listen_port) if root_host else ""
|
||||
|
||||
@@ -370,6 +380,11 @@ def get_snapshot_base_url(snapshot_id: str, request=None, config: dict[str, Any]
|
||||
)
|
||||
|
||||
|
||||
def get_original_base_url(domain: str, request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
|
||||
return _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
|
||||
|
||||
|
||||
def build_admin_url(path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
return _build_url(get_admin_base_url(request, config=config, **config_kwargs), path)
|
||||
|
||||
@@ -379,109 +394,11 @@ def build_web_url(path: str = "", request=None, config: dict[str, Any] | None =
|
||||
|
||||
|
||||
def build_snapshot_url(snapshot_id: str, path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
# `path` is a filesystem-relative output path, not an already-encoded URL.
|
||||
# In particular, wget can save literal `%3A` in filenames; emitting that
|
||||
# unchanged makes the browser request `:`, which cannot match the file.
|
||||
output_path = quote(str(path).lstrip("/"), safe=_URL_PATH_SAFE)
|
||||
return _build_url(get_snapshot_base_url(snapshot_id, request=request, config=config, **config_kwargs), output_path)
|
||||
|
||||
|
||||
def build_snapshot_detail_path(archive_path: str, output_path: str = "", prefix: str = "/") -> str:
|
||||
"""Build the live Django snapshot-detail path and optional output anchor."""
|
||||
normalized_archive_path = quote(str(archive_path or "").strip("/"), safe=_URL_PATH_SAFE)
|
||||
detail_path = f"{prefix}{normalized_archive_path}"
|
||||
normalized_output_path = str(output_path or "").lstrip("#/")
|
||||
if not normalized_output_path:
|
||||
return detail_path
|
||||
fragment = quote(normalized_output_path, safe=_URL_FRAGMENT_SAFE)
|
||||
return f"{detail_path}#{fragment}"
|
||||
|
||||
|
||||
def build_snapshot_index_path(archive_path: str, output_path: str = "", prefix: str = "/") -> str:
|
||||
"""Build a path to the literal exported ``index.html`` snapshot file."""
|
||||
normalized_archive_path = quote(str(archive_path or "").strip("/"), safe=_URL_PATH_SAFE)
|
||||
index_path = f"{prefix}{normalized_archive_path}/index.html"
|
||||
normalized_output_path = str(output_path or "").lstrip("#/")
|
||||
if not normalized_output_path:
|
||||
return index_path
|
||||
fragment = quote(normalized_output_path, safe=_URL_FRAGMENT_SAFE)
|
||||
return f"{index_path}#{fragment}"
|
||||
|
||||
|
||||
def get_snapshot_output_anchor(plugin: str, output_path: str) -> str:
|
||||
"""Map root-relative legacy outputs to the plugin card that displays them."""
|
||||
normalized_output_path = str(output_path or "").lstrip("/")
|
||||
return normalized_output_path if normalized_output_path.startswith(f"{plugin}/") else plugin
|
||||
|
||||
|
||||
def build_snapshot_detail_url(
|
||||
archive_path: str,
|
||||
output_path: str = "",
|
||||
request=None,
|
||||
config: dict[str, Any] | None = None,
|
||||
**config_kwargs: Any,
|
||||
) -> str:
|
||||
"""Build a request-aware URL for trusted snapshot UI on the web origin."""
|
||||
return build_web_url(
|
||||
build_snapshot_detail_path(archive_path, output_path=output_path),
|
||||
request=request,
|
||||
config=config,
|
||||
**config_kwargs,
|
||||
)
|
||||
|
||||
|
||||
def build_snapshot_role_output_url(
|
||||
snapshot,
|
||||
role: str,
|
||||
request=None,
|
||||
config: dict[str, Any] | None = None,
|
||||
archive_results=None,
|
||||
fallback_to_default: bool = False,
|
||||
**config_kwargs: Any,
|
||||
) -> str:
|
||||
"""Build the first available output URL for a plugin-owned snapshot role."""
|
||||
from archivebox.plugins.discovery import get_plugin_default_output_path, get_snapshot_role_names
|
||||
|
||||
for plugin in get_snapshot_role_names(role):
|
||||
output_path = snapshot.plugin_output_path(plugin, archive_results=archive_results)
|
||||
if not output_path and fallback_to_default:
|
||||
output_path = get_plugin_default_output_path(plugin)
|
||||
if output_path:
|
||||
return build_snapshot_url(str(snapshot.id), output_path, request=request, config=config, **config_kwargs)
|
||||
return ""
|
||||
|
||||
|
||||
def build_snapshot_files_url(
|
||||
snapshot_id: str,
|
||||
path: str = "",
|
||||
request=None,
|
||||
config: dict[str, Any] | None = None,
|
||||
**config_kwargs: Any,
|
||||
) -> str:
|
||||
"""Build a request-aware directory-browser URL on the isolated replay origin."""
|
||||
directory_url = build_snapshot_url(snapshot_id, path, request=request, config=config, **config_kwargs)
|
||||
parts = urlsplit(directory_url)
|
||||
normalized_directory_path = f"{parts.path.rstrip('/')}/"
|
||||
directory_url = urlunsplit((parts.scheme, parts.netloc, normalized_directory_path, parts.query, parts.fragment))
|
||||
return _replace_url_query_params(directory_url, files="1")
|
||||
|
||||
|
||||
def build_snapshot_zip_url(
|
||||
snapshot_id: str,
|
||||
path: str = "",
|
||||
request=None,
|
||||
config: dict[str, Any] | None = None,
|
||||
**config_kwargs: Any,
|
||||
) -> str:
|
||||
"""Build a request-aware ZIP download URL for a snapshot directory."""
|
||||
files_url = build_snapshot_files_url(snapshot_id, path, request=request, config=config, **config_kwargs)
|
||||
return _replace_url_query_params(files_url, download="zip")
|
||||
return _build_url(get_snapshot_base_url(snapshot_id, request=request, config=config, **config_kwargs), path)
|
||||
|
||||
|
||||
def build_original_url(domain: str, path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
|
||||
config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
|
||||
base_url = _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
|
||||
return _build_url(base_url, path)
|
||||
return _build_url(get_original_base_url(domain, request=request, config=config, **config_kwargs), path)
|
||||
|
||||
|
||||
def _build_url(base_url: str, path: str) -> str:
|
||||
@@ -492,11 +409,3 @@ def _build_url(base_url: str, path: str) -> str:
|
||||
if not path:
|
||||
return base_url
|
||||
return f"{base_url}{path if path.startswith('/') else f'/{path}'}"
|
||||
|
||||
|
||||
def _replace_url_query_params(url: str, **params: str) -> str:
|
||||
parts = urlsplit(url)
|
||||
replacement_keys = set(params)
|
||||
query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key not in replacement_keys]
|
||||
query.extend((key, value) for key, value in params.items())
|
||||
return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))
|
||||
@@ -76,6 +76,7 @@ INSTALLED_APPS = [
|
||||
"archivebox.crawls", # handles Crawl and CrawlSchedule models and management (depends on core)
|
||||
"archivebox.progressmonitor", # live progress endpoint and admin monitor template
|
||||
"archivebox.api", # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.
|
||||
"archivebox.opencode",
|
||||
# 3rd-party apps from PyPI that need to be loaded last
|
||||
"admin_data_views", # handles rendering some convenient automatic read-only views of data in Django admin
|
||||
"django_extensions", # provides Django Debug Toolbar (and other non-debug helpers)
|
||||
@@ -87,7 +88,6 @@ DJANGO_OBJECT_ACTIONS_DEFAULT_HTTP_METHOD = "POST"
|
||||
MIDDLEWARE = [
|
||||
"archivebox.core.middleware.TimezoneMiddleware",
|
||||
"django.middleware.security.SecurityMiddleware",
|
||||
"archivebox.core.middleware.CacheControlMiddleware",
|
||||
"archivebox.core.middleware.AdminCookieIsolationMiddleware",
|
||||
"django.contrib.sessions.middleware.SessionMiddleware",
|
||||
"django.middleware.common.CommonMiddleware",
|
||||
@@ -98,6 +98,7 @@ MIDDLEWARE = [
|
||||
"archivebox.core.middleware.ServerSecurityModeMiddleware",
|
||||
"archivebox.core.middleware.HostRoutingMiddleware",
|
||||
"django.contrib.messages.middleware.MessageMiddleware",
|
||||
"archivebox.core.middleware.CacheControlMiddleware",
|
||||
# Additional middlewares from plugins (if any)
|
||||
]
|
||||
|
||||
@@ -390,7 +391,7 @@ SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
|
||||
# get_base_url() also covers deployments that only set CSRF_TRUSTED_ORIGINS (the
|
||||
# implicit-BASE_URL fallback used on 0.7.x->0.9.x upgrades), so HTTPS hardening
|
||||
# isn't lost until BASE_URL is migrated. An explicit plain-http base (e.g. local
|
||||
# http://archivebox.localhost:5797) disables proxy HTTPS handling.
|
||||
# http://archivebox.localhost:8000) disables proxy HTTPS handling.
|
||||
BASE_URL_IS_HTTPS = get_base_url(config=CONFIG).strip().lower().startswith("https://")
|
||||
if BASE_URL_IS_HTTPS or not CONFIG.BASE_URL:
|
||||
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
|
||||
|
||||
@@ -1,28 +1,3 @@
|
||||
"""Shared foreground signal and child-process cleanup contracts.
|
||||
|
||||
Shutdown intent and process exit are separate: takeover must preserve the
|
||||
replacement owner's work, while a user abort must not later resume the crawl.
|
||||
Callers own that policy; this module preserves the received signal and stops
|
||||
only the process objects they explicitly supply.
|
||||
|
||||
Cooperative boundaries are essential, not just quieter exception handling.
|
||||
A real Linux CI regression interrupted Popen.poll() with SIGINT and stranded
|
||||
its wait lock. SIGKILL stopped the child (it became a zombie), but Popen.wait()
|
||||
still timed out. Raising a different exception or waiting longer cannot repair
|
||||
that interrupted critical section. Polling loops must record the signal first
|
||||
and honor it between operations; async owners may cancel their own task while
|
||||
their loop is running. Once exiting, the last-resort process cleanup must stay
|
||||
synchronous: the original loop may be closed, and creating a replacement loop
|
||||
cannot recover tasks or resources owned by it.
|
||||
|
||||
Keep immediate signal notices and repeated-signal force exit independent of
|
||||
the event bus: it may already be draining or closed during shutdown. Resumable
|
||||
pause/retry/skip belongs to the existing crawl controller and must not set
|
||||
sticky command-shutdown state. See foreground_shutdown_signals below for both
|
||||
cooperative and legacy blocking callers; do not change all callers' policy at
|
||||
once without supplying their interruption boundaries.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
@@ -48,7 +23,7 @@ _active_shutdown_state: ShutdownSignalState | None = None
|
||||
|
||||
|
||||
def raise_if_shutdown_requested() -> None:
|
||||
"""Honor sticky shutdown intent at a caller-owned interruption boundary."""
|
||||
"""Let long foreground loops honor a signal even if Python ignored it once."""
|
||||
|
||||
if _active_shutdown_state and _active_shutdown_state.signal_name:
|
||||
raise KeyboardInterrupt
|
||||
@@ -73,37 +48,14 @@ def wait_popen_and_kill_children(
|
||||
timeout: float,
|
||||
kill_timeout: float = 2.0,
|
||||
) -> None:
|
||||
"""Reap our own Popen child and stop its explicitly captured descendants.
|
||||
|
||||
A dead child still needs its parent to reap it. Keep wait failures visible;
|
||||
bypassing Popen's lock would hide the interrupted-owner bug described above.
|
||||
Descendants must still be cleaned up when that wait fails.
|
||||
"""
|
||||
"""Wait for a Popen parent and then hard-kill any surviving descendants."""
|
||||
|
||||
try:
|
||||
try:
|
||||
proc.wait(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=kill_timeout)
|
||||
except subprocess.TimeoutExpired as err:
|
||||
try:
|
||||
state = psutil.Process(proc.pid).as_dict(attrs=["pid", "ppid", "status", "num_threads", "cpu_times"])
|
||||
except psutil.NoSuchProcess:
|
||||
state = {"pid": proc.pid, "status": "exited"}
|
||||
err.add_note(f"Process state after SIGKILL: {state}")
|
||||
# A locked Popen wait can time out even after the OS process exits.
|
||||
# Inspect CPython's lock only for diagnostics; never release it here.
|
||||
waitpid_lock = getattr(proc, "_waitpid_lock", None)
|
||||
err.add_note(
|
||||
f"Popen state: returncode={proc.returncode}, "
|
||||
f"waitpid_lock_locked={waitpid_lock.locked() if waitpid_lock is not None else None}, "
|
||||
f"threads={[thread.name for thread in threading.enumerate()]}",
|
||||
)
|
||||
raise
|
||||
finally:
|
||||
# A parent that cannot be reaped must not leave its descendants running.
|
||||
kill_remaining_processes(children, timeout=kill_timeout)
|
||||
proc.wait(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=kill_timeout)
|
||||
kill_remaining_processes(children, timeout=kill_timeout)
|
||||
|
||||
|
||||
def wait_psutil_and_kill_children(
|
||||
@@ -149,17 +101,13 @@ def foreground_shutdown_signals(
|
||||
*,
|
||||
first_signal_message: str | None = "\n[🛑] Got {signal_name}, stopping gracefully...\n",
|
||||
on_signal: Callable[[signal.Signals], None] | None = None,
|
||||
interrupt_handlers: dict[signal.Signals, Callable[[], None]] | None = None,
|
||||
raise_on_first_signal: bool = True,
|
||||
) -> Iterator[ShutdownSignalState]:
|
||||
"""Keep shutdown intent visible until the owning foreground command unwinds.
|
||||
"""Install foreground signal handlers that print an immediate exit notice.
|
||||
|
||||
Signal state was added because inner loops can swallow KeyboardInterrupt
|
||||
while their caller still needs to stop its children. Cooperative callers use
|
||||
raise_on_first_signal=False and check raise_if_shutdown_requested() between
|
||||
operations, or cancel their owned async task through on_signal. Raising from
|
||||
the handler can otherwise interrupt library locks or partially sent I/O.
|
||||
Blocking callers without a cooperative boundary retain immediate exceptions.
|
||||
Some log-tail loops intentionally swallow KeyboardInterrupt so that callers
|
||||
can centralize cleanup in finally blocks. The handler writes the signal name
|
||||
immediately, then raises KeyboardInterrupt to break out of the blocking read.
|
||||
"""
|
||||
|
||||
global _active_shutdown_state
|
||||
@@ -170,20 +118,10 @@ def foreground_shutdown_signals(
|
||||
|
||||
state = ShutdownSignalState()
|
||||
previous_active_state = _active_shutdown_state
|
||||
previous_handlers = {sig: signal.getsignal(sig) for sig in (*handled_signals, *(interrupt_handlers or {}))}
|
||||
previous_handlers = {sig: signal.getsignal(sig) for sig in handled_signals}
|
||||
|
||||
def raise_keyboard_interrupt(signum, _frame):
|
||||
sig = signal.Signals(signum)
|
||||
if interrupt_handlers and sig in interrupt_handlers:
|
||||
# Shield resumable terminal intent from sticky shutdown state.
|
||||
# Ctrl+C is delegated to abx-dl's pause/abort controller; SIGUSR1/2
|
||||
# carry prompt answers. Recording any of these as shutdown here
|
||||
# makes a later skip/retry fail raise_if_shutdown_requested(), or
|
||||
# makes a second Ctrl+C hard-exit before hooks can flush output.
|
||||
# SIGTERM/SIGHUP deliberately bypass this shield: ownership takeover
|
||||
# and parent loss must cancel the old worker without asking for input.
|
||||
interrupt_handlers[sig]()
|
||||
return
|
||||
already_requested = state.signal_name is not None
|
||||
if not already_requested:
|
||||
state.signal_name = sig.name
|
||||
@@ -191,9 +129,11 @@ def foreground_shutdown_signals(
|
||||
os.write(sys.stdout.fileno(), first_signal_message.format(signal_name=state.signal_name).encode())
|
||||
if on_signal is not None:
|
||||
on_signal(sig)
|
||||
# Actual shutdown requests remain sticky and a second signal forces
|
||||
# exit. Resumable interactive actions were handled above without
|
||||
# marking this context as shutting down.
|
||||
# Foreground `archivebox add` uses the first signal to abort the active
|
||||
# hook through the bus-facing runner code, then reserves the second
|
||||
# signal for hard foreground-command shutdown. Server/update/run and
|
||||
# other non-interactive commands raise immediately so their finally
|
||||
# blocks can stop owned children without prompting.
|
||||
if already_requested:
|
||||
os.write(sys.stdout.fileno(), f"\n[🛑] Got {sig.name} again, exiting immediately.\n".encode())
|
||||
os._exit(130)
|
||||
@@ -202,7 +142,7 @@ def foreground_shutdown_signals(
|
||||
|
||||
try:
|
||||
_active_shutdown_state = state
|
||||
for sig in previous_handlers:
|
||||
for sig in handled_signals:
|
||||
signal.signal(sig, raise_keyboard_interrupt)
|
||||
yield state
|
||||
finally:
|
||||
|
||||
@@ -6,16 +6,15 @@ from typing import Any
|
||||
from urllib.parse import unquote
|
||||
|
||||
from django.contrib.auth.models import User
|
||||
from django.db.models import Count, Exists, F, OuterRef, Prefetch, QuerySet
|
||||
from django.db.models import Count, Exists, F, OuterRef, QuerySet
|
||||
from django.db.models.functions import Lower
|
||||
from django.http import HttpRequest
|
||||
from django.urls import reverse
|
||||
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag, Tag
|
||||
from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_role_output_url
|
||||
from archivebox.misc.util import sanitize_html_text
|
||||
from archivebox.plugins.discovery import get_snapshot_role_names
|
||||
from archivebox.core.routes_util import build_snapshot_url, build_web_url
|
||||
from archivebox.core.models import Snapshot, SnapshotTag, Tag
|
||||
|
||||
|
||||
TAG_SNAPSHOT_PREVIEW_LIMIT = 10
|
||||
@@ -203,16 +202,9 @@ def _build_snapshot_preview(snapshot: Snapshot, request: HttpRequest | None = No
|
||||
"id": str(snapshot.pk),
|
||||
"title": _display_snapshot_title(snapshot),
|
||||
"url": snapshot.url,
|
||||
"favicon_url": build_snapshot_role_output_url(
|
||||
snapshot,
|
||||
"list_icon",
|
||||
request=request,
|
||||
config=config,
|
||||
archive_results=snapshot._list_icon_results,
|
||||
fallback_to_default=True,
|
||||
),
|
||||
"favicon_url": build_snapshot_url(str(snapshot.pk), "favicon.ico", request=request, config=config),
|
||||
"admin_url": reverse("admin:core_snapshot_change", args=[snapshot.pk]),
|
||||
"archive_url": build_snapshot_detail_url(snapshot.archive_path_from_db, request=request, config=config),
|
||||
"archive_url": build_web_url(f"/{snapshot.archive_path_from_db}/index.html", request=request, config=config),
|
||||
"downloaded_at": snapshot.downloaded_at.isoformat() if snapshot.downloaded_at else None,
|
||||
}
|
||||
|
||||
@@ -229,22 +221,6 @@ def _build_snapshot_preview_map(
|
||||
snapshot_tags = (
|
||||
SnapshotTag.objects.filter(tag_id__in=tag_ids)
|
||||
.select_related("snapshot__crawl__created_by")
|
||||
.prefetch_related(
|
||||
Prefetch(
|
||||
"snapshot__archiveresult_set",
|
||||
queryset=ArchiveResult.objects.filter(
|
||||
plugin__in=get_snapshot_role_names("list_icon"),
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
).only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_files",
|
||||
),
|
||||
to_attr="_list_icon_results",
|
||||
),
|
||||
)
|
||||
.order_by(
|
||||
"tag_id",
|
||||
F("snapshot__downloaded_at").desc(nulls_last=True),
|
||||
|
||||
@@ -76,8 +76,6 @@ def command_owns_runtime_stack(command, *, data_dir: str | Path) -> bool:
|
||||
|
||||
def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
"""Return the newest live local parent allowed to borrow runner/sonic."""
|
||||
from django.db.models import Q
|
||||
|
||||
from archivebox.machine.models import Machine, Process
|
||||
|
||||
machine = Machine.current()
|
||||
@@ -85,13 +83,7 @@ def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
|
||||
machine=machine,
|
||||
status=Process.StatusChoices.RUNNING,
|
||||
pwd=str(data_dir),
|
||||
).filter(
|
||||
Q(process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ADD, Process.TypeChoices.UPDATE))
|
||||
# A direct `run` owns its own hooks, just as add/update own theirs via
|
||||
# a supervised one-shot worker. Respawning the server daemon must not
|
||||
# steal the gate back from that newer foreground command. Supervised
|
||||
# runner children cannot elect themselves above their parent command.
|
||||
| (Q(process_type=Process.TypeChoices.ORCHESTRATOR) & ~Q(parent__process_type=Process.TypeChoices.SUPERVISORD)),
|
||||
process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ADD, Process.TypeChoices.UPDATE),
|
||||
)
|
||||
if exclude_id is not None:
|
||||
qs = qs.exclude(id=exclude_id)
|
||||
@@ -134,25 +126,12 @@ def ensure_daemon_stack(*, reason: str = ""):
|
||||
|
||||
supervisor = get_existing_supervisord_process() or get_or_create_supervisord_process(daemonize=False)
|
||||
worker = get_worker(supervisor, sonic_worker["name"])
|
||||
if not (isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING")):
|
||||
if reason:
|
||||
rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
|
||||
worker = start_worker(supervisor, sonic_worker)
|
||||
if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
|
||||
return worker
|
||||
|
||||
# Supervisord marks this worker RUNNING with startsecs=0, before Sonic has
|
||||
# opened its search socket. A query launched at that point fails even though
|
||||
# the daemon is about to become ready.
|
||||
deadline = time.monotonic() + 30.0
|
||||
while time.monotonic() < deadline:
|
||||
if is_port_listening(sonic_event.host, sonic_event.port):
|
||||
if time.monotonic() < deadline:
|
||||
return worker
|
||||
break
|
||||
worker = get_worker(supervisor, sonic_worker["name"])
|
||||
if not isinstance(worker, dict) or worker.get("statename") not in ("STARTING", "RUNNING"):
|
||||
raise RuntimeError(f"Sonic search backend worker failed to start: {worker}")
|
||||
time.sleep(min(0.1, max(0.0, deadline - time.monotonic())))
|
||||
raise RuntimeError(f"Sonic search backend is not listening at {sonic_event.url}")
|
||||
if reason:
|
||||
rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
|
||||
return start_worker(supervisor, sonic_worker)
|
||||
|
||||
|
||||
def live_runner_processes(*, data_dir: str | Path):
|
||||
|
||||
@@ -1,90 +1,32 @@
|
||||
import hashlib
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
from html import unescape
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote, urlparse
|
||||
from urllib.parse import quote
|
||||
|
||||
from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
|
||||
from django import template
|
||||
from django.templatetags.static import static
|
||||
from django.utils import timezone
|
||||
from django.utils.html import escape, format_html
|
||||
from django.utils.html import escape
|
||||
from django.utils.safestring import mark_safe
|
||||
from django.utils.text import Truncator
|
||||
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.core.routes_util import (
|
||||
build_snapshot_detail_url,
|
||||
build_snapshot_files_url,
|
||||
build_snapshot_role_output_url,
|
||||
build_snapshot_url,
|
||||
build_snapshot_zip_url,
|
||||
get_admin_base_url,
|
||||
get_snapshot_base_url,
|
||||
get_web_base_url,
|
||||
)
|
||||
from archivebox.core.setup_wizard import get_base_url_mismatch_context, get_setup_wizard_context
|
||||
from archivebox.plugins.discovery import (
|
||||
get_plugin_catalog,
|
||||
get_plugin_icon,
|
||||
get_plugin_name,
|
||||
get_plugin_template,
|
||||
plugin_card_is_interactive,
|
||||
)
|
||||
|
||||
register = template.Library()
|
||||
register.filter("plugin_card_is_interactive", plugin_card_is_interactive)
|
||||
|
||||
# Preview cards are clipped windows onto their content, never scroll areas.
|
||||
# Apply inside the document too: parent overflow cannot constrain an iframe.
|
||||
CARD_OVERFLOW_STYLE = (
|
||||
"<style>html,body{height:100%!important;min-height:0!important}"
|
||||
"html,body,body *{overflow:clip!important;scrollbar-width:none!important}</style>"
|
||||
)
|
||||
|
||||
|
||||
@register.filter
|
||||
def snapshot_url_text(url):
|
||||
"""Emphasize the hostname while preserving the exact selectable URL."""
|
||||
url = str(url or "")
|
||||
match = re.match(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:)?//([^/?#]*)", url)
|
||||
if not match:
|
||||
return escape(url)
|
||||
netloc = match[1]
|
||||
authority = netloc.rsplit("@", 1)[-1]
|
||||
start = match.start(1) + len(netloc) - len(authority)
|
||||
host_end = authority.find("]") + 1 if authority.startswith("[") else authority.find(":")
|
||||
hostname = authority[:host_end] if host_end > 0 else authority
|
||||
if hostname.lower().startswith("www."):
|
||||
start += 4
|
||||
hostname = hostname[4:]
|
||||
remainder = url[start + len(hostname) :]
|
||||
suffix = re.search(r"[?&#]", remainder)
|
||||
suffix_start = suffix.start() if suffix else len(remainder)
|
||||
return format_html(
|
||||
'<span class="snapshot-url-prefix">{}</span><span class="snapshot-url-domain">{}</span>'
|
||||
'<span class="snapshot-url-path">{}</span><span class="snapshot-url-suffix">{}</span>',
|
||||
url[:start],
|
||||
hostname,
|
||||
remainder[:suffix_start],
|
||||
remainder[suffix_start:],
|
||||
)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def plugin_ui(context, plugin: str, slot: str):
|
||||
"""Optional plugin UI must never prevent the host page from rendering."""
|
||||
try:
|
||||
source = get_plugin_template(plugin, slot, fallback=False)
|
||||
if not source:
|
||||
return ""
|
||||
return template.engines["django"].from_string(source).render(context.flatten())
|
||||
except Exception:
|
||||
logging.getLogger(__name__).exception("Unable to render optional plugin UI: %s/%s", plugin, slot)
|
||||
return ""
|
||||
|
||||
|
||||
_TEXT_PREVIEW_EXTS = (".json", ".jsonl", ".txt", ".csv", ".tsv", ".xml", ".yml", ".yaml", ".md", ".log")
|
||||
_IMAGE_PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".ico", ".avif")
|
||||
@@ -132,21 +74,6 @@ _BROWSER_VIDEO_FILE_EXTS = {".mp4", ".webm", ".m4v", ".ogv"}
|
||||
_BROWSER_AUDIO_FILE_EXTS = {".mp3", ".m4a", ".aac", ".ogg", ".oga", ".opus", ".wav", ".flac"}
|
||||
|
||||
|
||||
def _is_web_subdomain_alias(browser_url: str, configured_base_url: str) -> bool:
|
||||
"""Hide the informational mismatch banner on the conventional public web alias.
|
||||
|
||||
This is deliberately presentation-only: request routing and the restriction
|
||||
on state-changing requests continue to use the configured BASE_URL.
|
||||
"""
|
||||
browser = urlparse(browser_url)
|
||||
configured = urlparse(configured_base_url)
|
||||
return (
|
||||
browser.scheme.lower() == configured.scheme.lower()
|
||||
and browser.hostname == f"web.{configured.hostname}"
|
||||
and browser.port == configured.port
|
||||
)
|
||||
|
||||
|
||||
def _normalize_output_files(output_files: Any) -> dict[str, dict[str, Any]]:
|
||||
from abx_dl.output_files import OutputManifest
|
||||
|
||||
@@ -197,10 +124,10 @@ def _count_media_files(result) -> int:
|
||||
return count
|
||||
|
||||
|
||||
def _list_media_files(result, *, include_filesystem_fallback=True) -> list[dict]:
|
||||
def _list_media_files(result) -> list[dict]:
|
||||
media_files: list[dict] = []
|
||||
try:
|
||||
plugin_dir = Path(result.snapshot_dir) / result.plugin if include_filesystem_fallback else None
|
||||
plugin_dir = Path(result.snapshot_dir) / result.plugin
|
||||
except (AttributeError, TypeError, ValueError):
|
||||
return media_files
|
||||
|
||||
@@ -212,7 +139,7 @@ def _list_media_files(result, *, include_filesystem_fallback=True) -> list[dict]
|
||||
if rel_path.suffix.lower() in _MEDIA_FILE_EXTS:
|
||||
candidates.append((rel_path, _coerce_output_file_size(metadata.get("size"))))
|
||||
|
||||
if not candidates and plugin_dir is not None and plugin_dir.exists():
|
||||
if not candidates and plugin_dir.exists():
|
||||
scanned = 0
|
||||
max_scan = 2000
|
||||
for file_path in plugin_dir.rglob("*"):
|
||||
@@ -233,13 +160,7 @@ def _list_media_files(result, *, include_filesystem_fallback=True) -> list[dict]
|
||||
candidates.append((rel_path, size))
|
||||
|
||||
for rel_path, size in candidates:
|
||||
if callable(getattr(result, "output_file_path", None)):
|
||||
href = result.output_file_path(str(rel_path), output_file_map=output_files)
|
||||
else:
|
||||
metadata = output_files.get(str(rel_path)) or {}
|
||||
href = str(rel_path) if metadata.get("root_relative") else str(Path(result.plugin) / rel_path)
|
||||
if not href:
|
||||
continue
|
||||
href = str(Path(result.plugin) / rel_path)
|
||||
suffix = rel_path.suffix.lower()
|
||||
media_type = "video" if suffix in _VIDEO_FILE_EXTS else "audio"
|
||||
media_files.append(
|
||||
@@ -318,14 +239,12 @@ def _snapshot_base_url_for_context(context, snapshot) -> str:
|
||||
|
||||
def _snapshot_url_for_context(context, snapshot, path: str = "") -> str:
|
||||
if not context.get("STATIC_EXPORT"):
|
||||
path_part, separator, query = str(path or "").partition("?")
|
||||
url = build_snapshot_url(
|
||||
return build_snapshot_url(
|
||||
str(_snapshot_id(snapshot)),
|
||||
path_part,
|
||||
path,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
)
|
||||
return f"{url}?{query}" if separator else url
|
||||
|
||||
base_url = _static_snapshot_base_url(context, snapshot)
|
||||
raw_path = str(path or "")
|
||||
@@ -337,6 +256,14 @@ def _snapshot_url_for_context(context, snapshot, path: str = "") -> str:
|
||||
return f"{base_url.rstrip('/')}/{quoted_path}{suffix}"
|
||||
|
||||
|
||||
def _build_snapshot_files_url(snapshot_id: str, request=None, config=None, base_url: str | None = None) -> str:
|
||||
return (
|
||||
f"{base_url.rstrip('/')}/index.jsonl"
|
||||
if base_url
|
||||
else build_snapshot_url(str(snapshot_id), "/?files=1", request=request, config=config)
|
||||
)
|
||||
|
||||
|
||||
def _build_snapshot_preview_url(
|
||||
snapshot_id: str,
|
||||
path: str = "",
|
||||
@@ -348,25 +275,20 @@ def _build_snapshot_preview_url(
|
||||
if path == "about:blank":
|
||||
return path
|
||||
if _is_root_snapshot_output_path(path):
|
||||
return (
|
||||
f"{base_url.rstrip('/')}/index.jsonl"
|
||||
if base_url
|
||||
else build_snapshot_files_url(str(snapshot_id), request=request, config=config)
|
||||
)
|
||||
return _build_snapshot_files_url(snapshot_id, request=request, config=config, base_url=base_url)
|
||||
if base_url:
|
||||
path_part, separator, query = str(path).lstrip("/").partition("?")
|
||||
url = f"{base_url.rstrip('/')}/{quote(path_part, safe=_STATIC_URL_SAFE)}"
|
||||
if separator:
|
||||
url = f"{url}?{query}"
|
||||
else:
|
||||
path_part, separator, query = str(path).partition("?")
|
||||
url = build_snapshot_url(str(snapshot_id), path_part, request=request, config=config)
|
||||
if separator:
|
||||
url = f"{url}?{query}"
|
||||
url = build_snapshot_url(str(snapshot_id), path, request=request, config=config)
|
||||
path_parts = Path(path).parts
|
||||
plugin = get_plugin_name(plugin) if plugin else (path_parts[0] if len(path_parts) > 1 else "")
|
||||
has_plugin_preview = bool(plugin and get_plugin_template(plugin, "full", fallback=False))
|
||||
if not (_is_text_preview_path(path) or _is_image_preview_path(path) or has_plugin_preview):
|
||||
if not (
|
||||
_is_text_preview_path(path) or _is_image_preview_path(path) or has_plugin_preview or is_archivewebpage_replay_target(path or "")
|
||||
):
|
||||
return url
|
||||
separator = "&" if "?" in url else "?"
|
||||
return f"{url}{separator}preview=1"
|
||||
@@ -377,7 +299,7 @@ def _render_text_preview(plugin: str, icon_html: str, snippet: str) -> str:
|
||||
plugin_label = escape(plugin or "")
|
||||
escaped = escape(snippet)
|
||||
return (
|
||||
f'<div class="thumbnail-text" data-plugin="{plugin_attr}">'
|
||||
f'<div class="thumbnail-text" data-plugin="{plugin_attr}" data-compact="1">'
|
||||
f'<div class="thumbnail-text-header">'
|
||||
f'<span class="thumbnail-compact-icon">{icon_html}</span>'
|
||||
f'<span class="thumbnail-text-title">{plugin_label}</span>'
|
||||
@@ -392,7 +314,7 @@ def _render_fallback_card(plugin: str, icon_html: str, fallback_label: str) -> s
|
||||
plugin_label = escape(plugin or "")
|
||||
fallback_attr = escape(fallback_label)
|
||||
return (
|
||||
f'<div class="thumbnail-compact" data-plugin="{plugin_attr}">'
|
||||
f'<div class="thumbnail-compact" data-plugin="{plugin_attr}" data-compact="1">'
|
||||
f'<span class="thumbnail-compact-icon">{icon_html}</span>'
|
||||
f'<span class="thumbnail-compact-label">{plugin_label}</span>'
|
||||
f'<span class="thumbnail-compact-meta">{fallback_attr}</span>'
|
||||
@@ -432,29 +354,12 @@ def index(value, position):
|
||||
|
||||
|
||||
@register.filter
|
||||
def file_size(num_bytes: float, decimal_places: int = 1) -> str:
|
||||
def file_size(num_bytes: float) -> str:
|
||||
for count in ["Bytes", "KB", "MB", "GB"]:
|
||||
if num_bytes > -1024.0 and num_bytes < 1024.0:
|
||||
if num_bytes >= 0:
|
||||
precision = 10 ** max(0, int(decimal_places))
|
||||
num_bytes = math.ceil(num_bytes * precision) / precision
|
||||
return f"{num_bytes:.{decimal_places}f} {count}"
|
||||
return f"{num_bytes:3.1f} {count}"
|
||||
num_bytes /= 1024.0
|
||||
if num_bytes >= 0:
|
||||
precision = 10 ** max(0, int(decimal_places))
|
||||
num_bytes = math.ceil(num_bytes * precision) / precision
|
||||
return f"{num_bytes:.{decimal_places}f} TB"
|
||||
|
||||
|
||||
@register.filter
|
||||
def grid_file_size(num_bytes: float | None) -> str:
|
||||
if not num_bytes or num_bytes <= 0:
|
||||
return ""
|
||||
if num_bytes < 1024**2:
|
||||
return f"{max(1, num_bytes / 1024):.0f}kb"
|
||||
if num_bytes < 1024**3:
|
||||
return f"{num_bytes / 1024**2:.0f}mb"
|
||||
return f"{num_bytes / 1024**3:.1f}gb"
|
||||
return "{:3.1f} {}".format(num_bytes, "TB")
|
||||
|
||||
|
||||
@register.filter
|
||||
@@ -525,9 +430,9 @@ def system_warnings_banner(context):
|
||||
"""Render the top-of-page warning banner for one of the conditions below,
|
||||
in priority order (highest first):
|
||||
|
||||
1. ``mode="unconfigured"``— ``BASE_URL`` is empty and the operator has
|
||||
not selected safe single-domain routing. That mode intentionally uses
|
||||
each device's connection address, including LAN and Tailscale IPs.
|
||||
1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
|
||||
issue: until it's pinned, generated URLs can echo any Host the client
|
||||
sends, and admin/web/api routing has no canonical anchor.
|
||||
2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
|
||||
an origin that differs from the configured canonical ``BASE_URL``.
|
||||
3. ``mode="unsafe"`` — ``SERVER_SECURITY_MODE`` explicitly enables
|
||||
@@ -554,10 +459,10 @@ def system_warnings_banner(context):
|
||||
|
||||
config = get_config(resolve_plugins=False)
|
||||
|
||||
if not config.BASE_URL and config.SERVER_SECURITY_MODE != "safe-onedomain-nojsreplay":
|
||||
if not config.BASE_URL:
|
||||
return get_setup_wizard_context(context.get("request"), config)
|
||||
mismatch = get_base_url_mismatch_context(context.get("request"), config)
|
||||
if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
|
||||
if mismatch:
|
||||
return mismatch
|
||||
if config.IS_LOWER_SECURITY_MODE:
|
||||
return {"mode": "unsafe"}
|
||||
@@ -621,140 +526,66 @@ def snapshot_base_url(context, snapshot) -> str:
|
||||
return _snapshot_base_url_for_context(context, snapshot)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_detail_url(context, snapshot) -> str:
|
||||
"""Keep the trusted snapshot detail page on web.*, not the replay host."""
|
||||
if context.get("STATIC_EXPORT"):
|
||||
return _snapshot_url_for_context(context, snapshot, "index.html")
|
||||
return build_snapshot_detail_url(
|
||||
snapshot.archive_path_from_db,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_url(context, snapshot, path: str = "") -> str:
|
||||
return _snapshot_url_for_context(context, snapshot, path)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_files_url(context, snapshot, path: str = "") -> str:
|
||||
if context.get("STATIC_EXPORT"):
|
||||
return _snapshot_url_for_context(context, snapshot, "index.jsonl")
|
||||
return build_snapshot_files_url(
|
||||
str(_snapshot_id(snapshot)),
|
||||
path,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_zip_url(context, snapshot, path: str = "") -> str:
|
||||
if context.get("STATIC_EXPORT"):
|
||||
return ""
|
||||
return build_snapshot_zip_url(
|
||||
str(_snapshot_id(snapshot)),
|
||||
path,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_role_output_url(context, snapshot, role: str, fallback_to_default: bool = False) -> str:
|
||||
url_cache = snapshot.__dict__.setdefault("_snapshot_role_output_url_cache", {})
|
||||
def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
|
||||
snapshot_id = str(_snapshot_id(snapshot))
|
||||
url_cache = snapshot.__dict__.setdefault("_snapshot_archiveresult_url_cache", {})
|
||||
cache_key = (
|
||||
role,
|
||||
plugin,
|
||||
filename,
|
||||
bool(context.get("STATIC_EXPORT")),
|
||||
str(context.get("STATIC_EXPORT_DIR") or ""),
|
||||
fallback_to_default,
|
||||
)
|
||||
if cache_key in url_cache:
|
||||
return url_cache[cache_key]
|
||||
|
||||
from archivebox.plugins.discovery import get_snapshot_role_names
|
||||
results = None
|
||||
if "_admin_archiveresults" in snapshot.__dict__:
|
||||
results = snapshot.__dict__["_admin_archiveresults"]
|
||||
elif "archiveresult_set" in snapshot.__dict__.get("_prefetched_objects_cache", {}):
|
||||
results = snapshot.archiveresult_set.all()
|
||||
elif "_snapshot_archiveresult_url_results" in snapshot.__dict__:
|
||||
results = snapshot.__dict__["_snapshot_archiveresult_url_results"]
|
||||
|
||||
if context.get("STATIC_EXPORT"):
|
||||
output_path = None
|
||||
for plugin in get_snapshot_role_names(role):
|
||||
output_path = snapshot.plugin_output_path(plugin)
|
||||
if output_path:
|
||||
break
|
||||
url_cache[cache_key] = _snapshot_url_for_context(context, snapshot, output_path) if output_path else ""
|
||||
else:
|
||||
url_cache[cache_key] = build_snapshot_role_output_url(
|
||||
snapshot,
|
||||
role,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
fallback_to_default=fallback_to_default,
|
||||
if results is None:
|
||||
from archivebox.core.models import ArchiveResult
|
||||
|
||||
results = list(
|
||||
ArchiveResult.objects.filter(
|
||||
snapshot_id=snapshot_id,
|
||||
plugin__in=("screenshot", "chrome_extension_screenshot", "favicon"),
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
).only(
|
||||
"plugin",
|
||||
"status",
|
||||
"output_files",
|
||||
),
|
||||
)
|
||||
return url_cache[cache_key]
|
||||
snapshot.__dict__["_snapshot_archiveresult_url_results"] = results
|
||||
|
||||
for result in results:
|
||||
if result.plugin != plugin or result.status != "succeeded":
|
||||
continue
|
||||
output_files = result.output_files or {}
|
||||
file_info = output_files.get(filename)
|
||||
if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
|
||||
continue
|
||||
output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
|
||||
url_cache[cache_key] = _snapshot_url_for_context(context, snapshot, output_path)
|
||||
return url_cache[cache_key]
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_result_file_url(context, snapshot, result, path: str) -> str:
|
||||
"""Build a saved-file URL from one ArchiveResult declaration."""
|
||||
try:
|
||||
output_path = result.output_file_path(path)
|
||||
except (AttributeError, TypeError, ValueError, OverflowError):
|
||||
return ""
|
||||
if not output_path:
|
||||
return ""
|
||||
return _snapshot_url_for_context(context, snapshot, output_path)
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_thumbnail(context, snapshot, admin=False, width="100px", height="100px"):
|
||||
from django.template.loader import render_to_string
|
||||
|
||||
from archivebox.plugins.output_groups import snapshot_thumbnail_result
|
||||
|
||||
results = snapshot.__dict__.get("_snapshot_card_results")
|
||||
if results is None:
|
||||
results = snapshot.__dict__.get("_admin_output_results")
|
||||
if results is None:
|
||||
results = snapshot.__dict__.get("_admin_archiveresults")
|
||||
if results is None:
|
||||
results = snapshot.__dict__.get("_prefetched_objects_cache", {}).get("archiveresult_set")
|
||||
if results is None:
|
||||
results = snapshot.archiveresult_set.exclude(plugin="").only(
|
||||
"id",
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_files",
|
||||
"output_size",
|
||||
)
|
||||
|
||||
try:
|
||||
host = urlparse(snapshot.url).hostname or "Archive"
|
||||
except ValueError:
|
||||
host = "Archive"
|
||||
host = host.removeprefix("www.")
|
||||
hue = int(hashlib.sha256(host.encode()).hexdigest()[:6], 16) % 360
|
||||
return mark_safe(
|
||||
render_to_string(
|
||||
"core/snapshot_thumbnail.html",
|
||||
{
|
||||
**context.flatten(),
|
||||
"card_result": snapshot_thumbnail_result(results),
|
||||
"host": host,
|
||||
"initial": host[:1].upper(),
|
||||
"hue": hue,
|
||||
"width": width,
|
||||
"height": height,
|
||||
},
|
||||
),
|
||||
)
|
||||
url_cache[cache_key] = ""
|
||||
return ""
|
||||
|
||||
|
||||
@register.simple_tag(takes_context=True)
|
||||
def snapshot_index_row(context, link) -> str:
|
||||
detail_url = snapshot_detail_url(context, link)
|
||||
snapshot_base = _snapshot_base_url_for_context(context, link)
|
||||
|
||||
status = getattr(link, "status", None) or "unknown"
|
||||
bookmarked_at = getattr(link, "bookmarked_at", None)
|
||||
@@ -773,11 +604,10 @@ def snapshot_index_row(context, link) -> str:
|
||||
|
||||
url = getattr(link, "url", "") or ""
|
||||
title = unescape(getattr(link, "title", "") or "")
|
||||
is_pending = status in {"queued", "started", "backoff"}
|
||||
title_text = title or url
|
||||
tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
|
||||
tag_html = "".join(
|
||||
f'<span class="snapshot-tag" title="{escape(tag)}">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag
|
||||
)
|
||||
tag_html = "".join(f'<span class="snapshot-tag">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag)
|
||||
if tag_html:
|
||||
tag_cell = f'<span class="snapshot-tags">{tag_html}</span>'
|
||||
else:
|
||||
@@ -792,14 +622,50 @@ def snapshot_index_row(context, link) -> str:
|
||||
archive_size = int(getattr(link, "archive_size", 0) or 0)
|
||||
size_cell = file_size(archive_size) if archive_size else '<span class="empty-value">...</span>'
|
||||
output_plural = "" if num_outputs == 1 else "s"
|
||||
files_url = snapshot_files_url(context, link)
|
||||
files_url = _snapshot_url_for_context(context, link, "index.jsonl") if context.get("STATIC_EXPORT") else f"{snapshot_base}/?files=1"
|
||||
|
||||
preview_html = snapshot_thumbnail(context, link)
|
||||
if is_pending:
|
||||
preview_html = '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress"></span>'
|
||||
if not context.get("STATIC_EXPORT"):
|
||||
preview_html = (
|
||||
'<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress">'
|
||||
f'<img src="{escape(static("spinner.gif"))}" alt="" decoding="async" loading="lazy">'
|
||||
"</span>"
|
||||
)
|
||||
elif "_public_preview_paths" in link.__dict__:
|
||||
preview_paths = list(getattr(link, "_public_preview_paths", []) or [])
|
||||
if preview_paths:
|
||||
preview_urls = [_snapshot_url_for_context(context, link, path) for path in preview_paths]
|
||||
preview_html = (
|
||||
f'<img src="{escape(preview_urls[0])}" '
|
||||
f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
|
||||
'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
|
||||
)
|
||||
else:
|
||||
preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
|
||||
else:
|
||||
preview_urls = [
|
||||
url
|
||||
for url in (
|
||||
snapshot_archiveresult_url(context, link, "screenshot", "screenshot.png"),
|
||||
snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot-1.png"),
|
||||
snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot.png"),
|
||||
)
|
||||
if url
|
||||
]
|
||||
if preview_urls:
|
||||
preview_html = (
|
||||
f'<img src="{escape(preview_urls[0])}" '
|
||||
f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
|
||||
'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
|
||||
)
|
||||
else:
|
||||
preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
|
||||
|
||||
if "_public_list_icon_paths" in link.__dict__:
|
||||
list_icon_paths = list(getattr(link, "_public_list_icon_paths", []) or [])
|
||||
if list_icon_paths:
|
||||
favicon_urls = [_snapshot_url_for_context(context, link, path) for path in list_icon_paths]
|
||||
if "_public_favicon_paths" in link.__dict__:
|
||||
favicon_paths = list(getattr(link, "_public_favicon_paths", []) or [])
|
||||
if favicon_paths:
|
||||
favicon_urls = [_snapshot_url_for_context(context, link, path) for path in favicon_paths]
|
||||
favicon_html = (
|
||||
f'<img src="{escape(favicon_urls[0])}" '
|
||||
f'data-fallbacks="{escape(",".join(favicon_urls[1:]))}" '
|
||||
@@ -809,7 +675,7 @@ def snapshot_index_row(context, link) -> str:
|
||||
else:
|
||||
favicon_html = '<span class="link-favicon link-favicon-empty" aria-hidden="true"></span>'
|
||||
else:
|
||||
favicon_url = snapshot_role_output_url(context, link, "list_icon")
|
||||
favicon_url = snapshot_archiveresult_url(context, link, "favicon", "favicon.ico")
|
||||
favicon_html = (
|
||||
(
|
||||
f'<img src="{escape(favicon_url)}" '
|
||||
@@ -824,24 +690,27 @@ def snapshot_index_row(context, link) -> str:
|
||||
html = f"""
|
||||
<tr class="snapshot-row status-{escape(status)}">
|
||||
<td class="snapshot-time" title="{escape(title_time)}" data-sort="{escape(sort_value)}">
|
||||
<a href="{escape(detail_url)}">
|
||||
<a href="{escape(snapshot_base)}/index.html">
|
||||
<span>{escape(date_text)}</span>
|
||||
<small>{escape(time_text)}</small>
|
||||
</a>
|
||||
</td>
|
||||
<td class="snapshot-preview-cell">
|
||||
<a href="{escape(detail_url)}" title="Open archived snapshot">
|
||||
<a href="{escape(snapshot_base)}/index.html" title="Open archived snapshot">
|
||||
{preview_html}
|
||||
</a>
|
||||
</td>
|
||||
<td class="snapshot-title-cell" title="{escape(title or url)}">
|
||||
<div class="snapshot-title-line">
|
||||
<a href="{escape(detail_url)}" class="snapshot-title">
|
||||
{escape(Truncator(title_text).chars(110) if title_text != url else url)}
|
||||
<a href="{escape(snapshot_base)}/index.html" class="snapshot-favicon-link" title="Open archived snapshot">
|
||||
{favicon_html}
|
||||
</a>
|
||||
<a href="{escape(snapshot_base)}/index.html" class="snapshot-title">
|
||||
{escape(Truncator(title_text).chars(110))}
|
||||
</a>
|
||||
</div>
|
||||
<a href="{escape(url)}" class="snapshot-url" title="{escape(url)}" target="_blank" rel="noopener noreferrer">
|
||||
{favicon_html}<span class="snapshot-url-text">{snapshot_url_text(url)}</span>
|
||||
{escape(url)}
|
||||
</a>
|
||||
<span class="snapshot-mobile-saved">Saved {escape(date_text)} at {escape(time_text)}</span>
|
||||
</td>
|
||||
@@ -913,28 +782,6 @@ def plugin_card(context, result) -> str:
|
||||
return ""
|
||||
|
||||
plugin = get_plugin_name(result.plugin)
|
||||
if not context.get("STATIC_EXPORT"):
|
||||
# The cover and expanded stack both embed this one card document. Run
|
||||
# plugin code on the snapshot origin so it can read saved files without
|
||||
# exposing raw archives or the admin session to the collection page.
|
||||
card_url = build_snapshot_url(
|
||||
str(_snapshot_id(result.snapshot)),
|
||||
f"_card/{result.id}",
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
)
|
||||
return mark_safe(
|
||||
f'<iframe src="{escape(card_url)}" title="{escape(plugin)} card" '
|
||||
'loading="lazy" fetchpriority="low" sandbox="allow-scripts allow-same-origin" '
|
||||
'style="width:100% !important;height:100% !important;transform:none !important;border:0"></iframe>',
|
||||
)
|
||||
return render_plugin_card_document(context, result)
|
||||
|
||||
|
||||
def render_plugin_card_document(context, result) -> str:
|
||||
"""Render one plugin card template for the trusted snapshot-origin frame."""
|
||||
plugin = get_plugin_name(result.plugin)
|
||||
has_card_template = bool(get_plugin_template(plugin, "card", fallback=False))
|
||||
template_str = get_plugin_template(plugin, "card")
|
||||
|
||||
# Use embed_path() for the display path
|
||||
@@ -942,8 +789,9 @@ def render_plugin_card_document(context, result) -> str:
|
||||
output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path or "")
|
||||
|
||||
icon_html = get_plugin_icon(plugin)
|
||||
media_files = _list_media_files(result, include_filesystem_fallback=bool(context.get("STATIC_EXPORT")))
|
||||
media_file_count = len(media_files)
|
||||
plugin_lower = (plugin or "").lower()
|
||||
media_file_count = _count_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else 0
|
||||
media_files = _list_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else []
|
||||
if context.get("STATIC_EXPORT") and media_files:
|
||||
media_files = [
|
||||
item
|
||||
@@ -962,22 +810,12 @@ def render_plugin_card_document(context, result) -> str:
|
||||
output_lower = (raw_output_path or "").lower()
|
||||
force_text_preview = output_lower.endswith(_TEXT_PREVIEW_EXTS)
|
||||
|
||||
# Static exports cannot execute live preview routes; keep their text snippets.
|
||||
if context.get("STATIC_EXPORT") and force_text_preview:
|
||||
template_str = None
|
||||
|
||||
# Create a mini template and render it with context
|
||||
try:
|
||||
if (
|
||||
template_str
|
||||
and raw_output_path
|
||||
and str(raw_output_path).strip() not in (".", "/", "./")
|
||||
and (has_card_template or not force_text_preview)
|
||||
):
|
||||
if template_str and raw_output_path and str(raw_output_path).strip() not in (".", "/", "./") and not force_text_preview:
|
||||
tpl = template.Template(template_str)
|
||||
ctx = template.Context(
|
||||
{
|
||||
**context.flatten(),
|
||||
"result": result,
|
||||
"snapshot": result.snapshot,
|
||||
"output_path": output_url,
|
||||
@@ -991,39 +829,11 @@ def render_plugin_card_document(context, result) -> str:
|
||||
rendered = tpl.render(ctx)
|
||||
# Only return non-empty content (strip whitespace to check)
|
||||
if rendered.strip():
|
||||
if rendered.lstrip().lower().startswith("<!doctype html"):
|
||||
rendered = rendered.replace("</head>", CARD_OVERFLOW_STYLE + "</head>", 1)
|
||||
if context.get("STATIC_EXPORT") and rendered.lstrip().lower().startswith("<!doctype html"):
|
||||
# Static exports have no _card route. Keep the same card
|
||||
# document in both stack positions, with relative saved-file
|
||||
# URLs resolved against the exported index page.
|
||||
return mark_safe(
|
||||
f'<iframe srcdoc="{escape(rendered)}" title="{escape(plugin)} card" '
|
||||
'loading="lazy" sandbox="allow-scripts allow-same-origin" '
|
||||
'style="width:100% !important;height:100% !important;max-width:100% !important;'
|
||||
'transform:none !important;border:0"></iframe>',
|
||||
)
|
||||
return mark_safe(rendered)
|
||||
except (template.TemplateSyntaxError, AttributeError, TypeError, ValueError):
|
||||
rendered = ""
|
||||
|
||||
if force_text_preview and not context.get("STATIC_EXPORT"):
|
||||
# Read archived text only when the browser requests its preview, never
|
||||
# while rendering the outer snapshot HTML (payloads may live on B2).
|
||||
preview_url = _build_snapshot_preview_url(
|
||||
str(_snapshot_id(result.snapshot)),
|
||||
raw_output_path,
|
||||
request=context.get("request"),
|
||||
config=context.get("CONFIG"),
|
||||
plugin=plugin,
|
||||
)
|
||||
return mark_safe(
|
||||
f'<iframe src="{escape(preview_url)}" title="{escape(plugin)} preview" '
|
||||
'loading="lazy" fetchpriority="low" sandbox="allow-scripts allow-same-origin" '
|
||||
'style="border:0;pointer-events:none"></iframe>',
|
||||
)
|
||||
|
||||
if force_text_preview and context.get("STATIC_EXPORT"):
|
||||
if force_text_preview:
|
||||
preview = _render_text_file_preview(result.snapshot_dir, raw_output_path, plugin, icon_html)
|
||||
if preview:
|
||||
return mark_safe(preview)
|
||||
@@ -1076,11 +886,7 @@ def plugin_full(context, result) -> str:
|
||||
output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path)
|
||||
|
||||
try:
|
||||
catalog = get_plugin_catalog()
|
||||
tpl = template.Engine(
|
||||
dirs=[catalog[plugin].path] if plugin in catalog else [],
|
||||
libraries=template.Engine.get_default().libraries,
|
||||
).from_string(template_str)
|
||||
tpl = template.Template(template_str)
|
||||
ctx = template.Context(
|
||||
{
|
||||
"result": result,
|
||||
@@ -1088,7 +894,6 @@ def plugin_full(context, result) -> str:
|
||||
"output_path": output_url,
|
||||
"output_path_raw": raw_output_path,
|
||||
"plugin": plugin,
|
||||
"plugin_icon": get_plugin_icon(plugin),
|
||||
},
|
||||
)
|
||||
rendered = tpl.render(ctx)
|
||||
|
||||
+145
-330
@@ -6,8 +6,8 @@ import posixpath
|
||||
from pathlib import Path
|
||||
from typing import ClassVar, cast
|
||||
from urllib.parse import quote, urlparse
|
||||
from uuid import UUID
|
||||
|
||||
from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
|
||||
from admin_data_views.typing import ItemContext, SectionData, TableContext
|
||||
from admin_data_views.utils import ItemLink, render_with_item_view, render_with_table_view
|
||||
from django import template
|
||||
@@ -17,12 +17,12 @@ from django.contrib.auth import HASH_SESSION_KEY, SESSION_KEY, get_user_model
|
||||
from django.contrib.auth.mixins import UserPassesTestMixin
|
||||
from django.contrib.sessions.models import Session
|
||||
from django.core import signing
|
||||
from django.core.paginator import InvalidPage
|
||||
from django.db.models import Case, IntegerField, Q, Value, When
|
||||
from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidden, QueryDict
|
||||
from django.shortcuts import redirect, render
|
||||
from django.utils.decorators import method_decorator
|
||||
from django.utils.html import format_html, format_html_join
|
||||
from django.utils.cache import patch_cache_control, patch_vary_headers
|
||||
from django.utils.safestring import mark_safe
|
||||
from django.views import View
|
||||
from django.views.decorators.csrf import csrf_exempt, csrf_protect
|
||||
@@ -47,6 +47,7 @@ from archivebox.config.configset import BaseConfigSet
|
||||
from archivebox.core.forms import AddLinkForm
|
||||
from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
|
||||
from archivebox.core.permissions import (
|
||||
PERMISSIONS_PRIVATE,
|
||||
PERMISSIONS_PUBLIC,
|
||||
PERMISSIONS_UNLISTED,
|
||||
can_view_snapshot,
|
||||
@@ -57,8 +58,6 @@ from archivebox.core.permissions import (
|
||||
)
|
||||
from archivebox.core.routes_util import (
|
||||
build_admin_url,
|
||||
build_snapshot_detail_url,
|
||||
build_snapshot_files_url,
|
||||
build_snapshot_url,
|
||||
build_web_url,
|
||||
get_admin_host,
|
||||
@@ -69,7 +68,7 @@ from archivebox.core.routes_util import (
|
||||
)
|
||||
from archivebox.crawls.models import Crawl
|
||||
from archivebox.misc.paginators import AcceleratedPaginator
|
||||
from archivebox.misc.serve_static import FAVICON_CACHE_CONTROL, serve_static_with_byterange_support
|
||||
from archivebox.misc.serve_static import serve_static_with_byterange_support
|
||||
from archivebox.misc.util import (
|
||||
base_url,
|
||||
filter_queryset_by_uuid_substring,
|
||||
@@ -78,19 +77,10 @@ from archivebox.misc.util import (
|
||||
validate_url,
|
||||
without_fragment,
|
||||
)
|
||||
from archivebox.plugins.discovery import (
|
||||
get_plugin_catalog,
|
||||
get_plugin_icon,
|
||||
get_plugin_name,
|
||||
get_plugin_template,
|
||||
get_snapshot_role_names,
|
||||
plugin_has_custom_full_response,
|
||||
serve_plugin_replay_asset,
|
||||
)
|
||||
from archivebox.plugins.discovery import get_plugin_name, get_plugin_template
|
||||
from archivebox.plugins.forms import get_plugin_config_binary_urls
|
||||
from archivebox.plugins.views import get_config_definition_link
|
||||
from archivebox.progressmonitor.views import live_progress_view
|
||||
from archivebox.progressmonitor.collection import collection_summary
|
||||
from archivebox.search.config import (
|
||||
get_search_mode,
|
||||
get_search_mode_backend,
|
||||
@@ -120,10 +110,7 @@ def _find_snapshot_by_ref(snapshot_ref: str) -> Snapshot | None:
|
||||
snapshots = Snapshot.objects.select_related("crawl", "crawl__created_by")
|
||||
|
||||
if len(lookup) == 12 and "-" not in lookup:
|
||||
# Resolve suffixes using the covering primary-key index before fetching
|
||||
# wide snapshot rows and joining their crawl/user data.
|
||||
matches = list(Snapshot.objects.filter(id__endswith=lookup).order_by().values_list("id", flat=True))
|
||||
return snapshots.filter(pk__in=matches).order_by("-created_at", "-downloaded_at").first()
|
||||
return snapshots.filter(id__endswith=lookup).order_by("-created_at", "-downloaded_at").first()
|
||||
|
||||
try:
|
||||
return snapshots.get(pk=lookup)
|
||||
@@ -340,20 +327,6 @@ class SnapshotView(View):
|
||||
|
||||
@staticmethod
|
||||
def render_live_index(request, snapshot):
|
||||
if request.GET.get("discover_outputs") == "1":
|
||||
# Reuse the existing output discovery and card/badge classification.
|
||||
request.archivebox_cache_policy = "private"
|
||||
initial_context = snapshot.get_html_details_context(request=request)
|
||||
known_paths = {output["path"] for output in initial_context["archiveresults"]}
|
||||
context = snapshot.get_html_details_context(request=request, discover_files=True)
|
||||
context["archiveresults"] = [output for output in context["archiveresults"] if output["path"] not in known_paths]
|
||||
for output in context["archiveresults"]:
|
||||
output["output_group"] = "other"
|
||||
output["folder_path"] = str(Path(output["path"]).parent)
|
||||
response = render(request, "core/snapshot_output_cards.html", context)
|
||||
patch_cache_control(response, private=True, no_store=True)
|
||||
return response
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
return render(
|
||||
template_name="core/snapshot.html",
|
||||
request=request,
|
||||
@@ -370,19 +343,18 @@ class SnapshotView(View):
|
||||
|
||||
# slug is a timestamp
|
||||
if slug.replace(".", "").isdigit():
|
||||
# missing trailing slash -> redirect to index
|
||||
if "/" not in path:
|
||||
return redirect(f"{path}/index.html")
|
||||
|
||||
try:
|
||||
try:
|
||||
snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
if not can_view_snapshot(request, snapshot):
|
||||
return _private_snapshot_auth_redirect(request, snapshot, archivefile or "index.html")
|
||||
canonical_base = snapshot.url_path
|
||||
if canonical_base != snapshot.legacy_archive_path:
|
||||
target_path = (
|
||||
build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
|
||||
if not archivefile or archivefile == "index.html"
|
||||
else build_snapshot_url(str(snapshot.id), archivefile, request=request)
|
||||
)
|
||||
target_path = f"/{canonical_base}/{archivefile or 'index.html'}"
|
||||
query = request.META.get("QUERY_STRING")
|
||||
if query:
|
||||
target_path = f"{target_path}?{query}"
|
||||
@@ -397,9 +369,8 @@ class SnapshotView(View):
|
||||
show_indexes=True,
|
||||
is_archive_replay=True,
|
||||
)
|
||||
elif archivefile in ("", "index.html"):
|
||||
# Both the canonical extensionless URL and the legacy
|
||||
# index.html URL render the live Django snapshot page.
|
||||
elif archivefile == "index.html":
|
||||
# if they requested snapshot index, serve live rendered template instead of static html
|
||||
response = self.render_live_index(request, snapshot)
|
||||
else:
|
||||
target = build_snapshot_url(str(snapshot.id), archivefile, request=request)
|
||||
@@ -433,15 +404,15 @@ class SnapshotView(View):
|
||||
except Snapshot.MultipleObjectsReturned:
|
||||
snapshot_hrefs = mark_safe("<br/>").join(
|
||||
format_html(
|
||||
'{} <a href="{}"><b><code>{}</code></b></a> {} <b>{}</b>',
|
||||
'{} <a href="/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
|
||||
snap.bookmarked_at.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
build_snapshot_detail_url(snap.archive_path_from_db, request=request),
|
||||
snap.archive_path,
|
||||
snap.timestamp,
|
||||
snap.url,
|
||||
snap.title_stripped[:64] or "",
|
||||
)
|
||||
for snap in direct_snapshots_queryset(request, Snapshot.objects.filter(timestamp__startswith=slug))
|
||||
.select_related("crawl__created_by")
|
||||
.only("url", "timestamp", "title", "bookmarked_at")
|
||||
.order_by("-bookmarked_at")
|
||||
)
|
||||
return HttpResponse(
|
||||
@@ -465,27 +436,27 @@ class SnapshotView(View):
|
||||
<title>Snapshot Not Found</title>
|
||||
</head><body>
|
||||
<center><br/><br/><br/>
|
||||
Snapshot <a href="{}" target="_top"><b><code>[{}]</code></b></a>: <a href="{}" target="_blank" rel="noreferrer">{}</a><br/>
|
||||
was queued on {}, but no files have been saved yet in:<br/><b><a href="{}" target="_top"><code>{}</code></a><code>/{}</code></b><br/><br/>
|
||||
Snapshot <a href="/{}/index.html" target="_top"><b><code>[{}]</code></b></a>: <a href="{}" target="_blank" rel="noreferrer">{}</a><br/>
|
||||
was queued on {}, but no files have been saved yet in:<br/><b><a href="/{}/" target="_top"><code>{}</code></a><code>/{}</code></b><br/><br/>
|
||||
It's possible {} during the last capture on {},<br/>or that the archiving process has not completed yet.<br/>
|
||||
<pre><code># run this cmd to finish/retry archiving this Snapshot</code><br/>
|
||||
<code style="user-select: all; color: #333">archivebox update -t timestamp {}</code></pre><br/><br/>
|
||||
<div class="text-align: left; width: 100%; max-width: 400px">
|
||||
<i><b>Next steps:</i></b><br/>
|
||||
- list all the <a href="{}" target="_top">Snapshot files <code>.*</code></a><br/>
|
||||
- view the <a href="{}" target="_top">Snapshot <code>./index.html</code></a><br/>
|
||||
- list all the <a href="/{}/" target="_top">Snapshot files <code>.*</code></a><br/>
|
||||
- view the <a href="/{}/index.html" target="_top">Snapshot <code>./index.html</code></a><br/>
|
||||
- go to the <a href="/admin/core/snapshot/{}/change/" target="_top">Snapshot admin</a> to edit<br/>
|
||||
- go to the <a href="/admin/core/snapshot/?id__exact={}" target="_top">Snapshot actions</a> to re-archive<br/>
|
||||
- or return to <a href="/" target="_top">the main index...</a></div>
|
||||
</center>
|
||||
</body></html>
|
||||
""",
|
||||
build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
|
||||
snapshot.archive_path,
|
||||
snapshot.timestamp,
|
||||
snapshot.url,
|
||||
snapshot.url,
|
||||
str(snapshot.bookmarked_at).split(".")[0],
|
||||
build_snapshot_files_url(str(snapshot.id), request=request),
|
||||
snapshot.archive_path,
|
||||
snapshot.timestamp,
|
||||
archivefile if str(archivefile) != "None" else "",
|
||||
f"the {archivefile} resource could not be fetched"
|
||||
@@ -493,8 +464,8 @@ class SnapshotView(View):
|
||||
else "the original site was not available",
|
||||
str(snapshot.bookmarked_at).split(".")[0],
|
||||
snapshot.timestamp,
|
||||
build_snapshot_files_url(str(snapshot.id), request=request),
|
||||
build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
|
||||
snapshot.archive_path,
|
||||
snapshot.archive_path,
|
||||
snapshot.pk,
|
||||
snapshot.id,
|
||||
),
|
||||
@@ -513,7 +484,7 @@ class SnapshotView(View):
|
||||
return id_qs
|
||||
return SnapshotView.find_snapshots_for_url(slug)
|
||||
|
||||
snapshots = direct_snapshots_queryset(request, _resolve_snapshots_for_slug(path)).select_related("crawl__created_by")
|
||||
snapshots = direct_snapshots_queryset(request, _resolve_snapshots_for_slug(path))
|
||||
try:
|
||||
if "://" in path:
|
||||
snapshot = snapshots.order_by("-bookmarked_at").first()
|
||||
@@ -541,15 +512,15 @@ class SnapshotView(View):
|
||||
except Snapshot.MultipleObjectsReturned:
|
||||
snapshot_hrefs = mark_safe("<br/>").join(
|
||||
format_html(
|
||||
'{} <code style="font-size: 0.8em">{}</code> <a href="{}"><b><code>{}</code></b></a> {} <b>{}</b>',
|
||||
'{} <code style="font-size: 0.8em">{}</code> <a href="/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
|
||||
snap.bookmarked_at.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
str(snap.id)[:8],
|
||||
build_snapshot_detail_url(snap.archive_path_from_db, request=request),
|
||||
snap.archive_path,
|
||||
snap.timestamp,
|
||||
snap.url,
|
||||
snap.title_stripped[:64] or "",
|
||||
)
|
||||
for snap in snapshots.select_related("crawl__created_by").order_by("-bookmarked_at")
|
||||
for snap in snapshots.only("url", "timestamp", "title", "bookmarked_at").order_by("-bookmarked_at")
|
||||
)
|
||||
return HttpResponse(
|
||||
format_html(
|
||||
@@ -562,7 +533,7 @@ class SnapshotView(View):
|
||||
status=404,
|
||||
)
|
||||
|
||||
target_path = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
|
||||
target_path = build_snapshot_url(str(snapshot.id), "index.html", request=request)
|
||||
query = request.META.get("QUERY_STRING")
|
||||
if query:
|
||||
target_path = f"{target_path}?{query}"
|
||||
@@ -658,7 +629,6 @@ class SnapshotPathView(View):
|
||||
status=404,
|
||||
)
|
||||
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
canonical_base = snapshot.url_path
|
||||
if date:
|
||||
requested_base = f"{username}/{date}/{domain or url or ''}"
|
||||
@@ -667,9 +637,7 @@ class SnapshotPathView(View):
|
||||
if snapshot_id:
|
||||
requested_base = f"{requested_base}/{snapshot_id}"
|
||||
if canonical_base != requested_base:
|
||||
target = f"/{canonical_base}"
|
||||
if path:
|
||||
target = f"{target}/{path}"
|
||||
target = f"/{canonical_base}/{path or 'index.html'}"
|
||||
query = request.META.get("QUERY_STRING")
|
||||
if query:
|
||||
target = f"{target}?{query}"
|
||||
@@ -715,14 +683,11 @@ def _safe_archive_relpath(path: str) -> str | None:
|
||||
return cleaned
|
||||
|
||||
|
||||
def _resolve_archiveresult_relpath(
|
||||
snapshot: Snapshot,
|
||||
rel_path: str,
|
||||
) -> tuple[str, ArchiveResult | None, tuple[str, ...]]:
|
||||
"""Resolve a declared output path and its exact historical fallbacks."""
|
||||
def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[str, ArchiveResult | None]:
|
||||
"""Resolve plugin-relative output paths through ArchiveResult.output_files."""
|
||||
parts = Path(rel_path).parts
|
||||
if len(parts) < 2:
|
||||
return rel_path, None, ()
|
||||
return rel_path, None
|
||||
|
||||
plugin = parts[0]
|
||||
plugin_relpath = posixpath.join(*parts[1:])
|
||||
@@ -734,38 +699,19 @@ def _resolve_archiveresult_relpath(
|
||||
).only("plugin", "output_files"),
|
||||
)
|
||||
if not results:
|
||||
from archivebox.plugins.discovery import get_plugin_default_output_path
|
||||
return rel_path, None
|
||||
|
||||
fallbacks = (plugin_relpath,) if get_plugin_default_output_path(plugin) == rel_path else ()
|
||||
return rel_path, None, fallbacks
|
||||
|
||||
declared_paths: list[tuple[str, ArchiveResult]] = []
|
||||
historical_fallbacks: list[str] = []
|
||||
for result in results:
|
||||
output_files = result.output_file_map()
|
||||
output_files = result.output_files or {}
|
||||
for candidate in (plugin_relpath, rel_path):
|
||||
output_path = result.output_file_path(candidate, output_file_map=output_files)
|
||||
if output_path and output_path not in {path for path, _result in declared_paths}:
|
||||
declared_paths.append((output_path, result))
|
||||
file_info = output_files.get(candidate)
|
||||
if not isinstance(file_info, dict):
|
||||
continue
|
||||
if file_info.get("root_relative"):
|
||||
return candidate, result
|
||||
return rel_path, result
|
||||
|
||||
# Partially migrated 0.7 rows can declare ``singlefile.html`` while
|
||||
# the old file still lives at SNAP_DIR/singlefile.html, without the
|
||||
# later root_relative metadata bit. Keep that one exact declared key as
|
||||
# a fallback after the normal path returns 404. Do not preflight either
|
||||
# path: the static response performs the only happy-path filesystem IO.
|
||||
if plugin_relpath in output_files and plugin_relpath not in historical_fallbacks:
|
||||
historical_fallbacks.append(plugin_relpath)
|
||||
|
||||
if declared_paths:
|
||||
primary_path, primary_result = declared_paths[0]
|
||||
fallbacks = tuple(
|
||||
dict.fromkeys(
|
||||
path for path in (*[path for path, _result in declared_paths[1:]], *historical_fallbacks) if path != primary_path
|
||||
),
|
||||
)
|
||||
return primary_path, primary_result, fallbacks
|
||||
|
||||
return rel_path, results[0], ()
|
||||
return rel_path, results[0]
|
||||
|
||||
|
||||
def _plugin_full_preview_response(
|
||||
@@ -775,7 +721,7 @@ def _plugin_full_preview_response(
|
||||
result: ArchiveResult | None,
|
||||
) -> HttpResponse | None:
|
||||
"""Render an explicit plugin full template as a trusted preview wrapper."""
|
||||
if not request.GET.get("preview") or request.GET.get("raw") or request.GET.get("files"):
|
||||
if not request.GET.get("preview"):
|
||||
return None
|
||||
|
||||
path_parts = Path(rel_path).parts
|
||||
@@ -783,25 +729,23 @@ def _plugin_full_preview_response(
|
||||
if not plugin:
|
||||
return None
|
||||
|
||||
# ReplayWeb.page needs plugin-owned WACZ inspection and service-worker
|
||||
# context, so it remains the one narrow preview exception below.
|
||||
if plugin == "archivewebpage" and archivewebpage_replay.is_replay_target(rel_path):
|
||||
return None
|
||||
|
||||
template_str = get_plugin_template(plugin, "full", fallback=False)
|
||||
if not template_str:
|
||||
return None
|
||||
if plugin_has_custom_full_response(plugin):
|
||||
# The plugin owns path selection and rendering for this full template.
|
||||
# Defer to the file-serving response hook so unrelated outputs from the
|
||||
# same plugin continue through their normal MIME-specific renderer.
|
||||
return None
|
||||
|
||||
raw_query = request.GET.copy()
|
||||
raw_query.pop("preview", None)
|
||||
raw_query.pop("titlebar", None)
|
||||
output_url = request.path
|
||||
if raw_query:
|
||||
output_url = f"{output_url}?{raw_query.urlencode()}"
|
||||
|
||||
catalog = get_plugin_catalog()
|
||||
rendered = (
|
||||
template.Engine(debug=False, dirs=[catalog[plugin].path] if plugin in catalog else [])
|
||||
template.Engine(debug=False)
|
||||
.from_string(template_str)
|
||||
.render(
|
||||
template.Context(
|
||||
@@ -810,40 +754,18 @@ def _plugin_full_preview_response(
|
||||
"snapshot": snapshot,
|
||||
"output_path": output_url,
|
||||
"output_path_raw": rel_path,
|
||||
"snapshot_details_url": build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
|
||||
"plugin": plugin,
|
||||
"plugin_icon": get_plugin_icon(plugin),
|
||||
"preview_base": f"{request.path.rsplit('/', 1)[0]}/",
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
if request.GET.get("titlebar") == "0":
|
||||
from archivebox.core.templatetags.core_tags import CARD_OVERFLOW_STYLE
|
||||
|
||||
# The card already has a result header and actions. Only its embedded
|
||||
# trusted full viewer drops the duplicate top bar; opening the same
|
||||
# output directly keeps the complete viewer and controls.
|
||||
rendered = rendered.replace("</head>", CARD_OVERFLOW_STYLE + "<style>body > header { display: none !important; }</style></head>", 1)
|
||||
response = HttpResponse(rendered, content_type="text/html; charset=utf-8")
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{Path(rel_path).stem}.html"'
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
response.headers["X-ArchiveBox-Security-Mode"] = request.archivebox_config.SERVER_SECURITY_MODE
|
||||
response.headers["Referrer-Policy"] = "no-referrer"
|
||||
# Full plugin viewers are trusted code, but must fetch saved data from the
|
||||
# isolated snapshot origin. The outer collection UI stays on web.*.
|
||||
response.headers["Content-Security-Policy"] = _trusted_snapshot_frame_csp(request)
|
||||
return response
|
||||
|
||||
|
||||
def _trusted_snapshot_frame_csp(request: HttpRequest) -> str:
|
||||
# A trusted plugin frame on snap-* may be embedded by web.* or admin.*.
|
||||
# Archived HTML/JS/CSS is still served only inside that snapshot origin;
|
||||
# neither this CSP nor the frame grants it admin cookies or cross-snapshot
|
||||
# reads. In one-domain no-JS mode the hosts coincide, but raw replay scripts
|
||||
# remain blocked by the separate replay response policy.
|
||||
viewer_origins = " ".join(dict.fromkeys((build_web_url(request=request), build_admin_url(request=request))))
|
||||
return (
|
||||
response.headers["Content-Security-Policy"] = (
|
||||
"default-src 'self' data: blob:; "
|
||||
"script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; "
|
||||
"style-src 'unsafe-inline' data: blob: 'self'; "
|
||||
@@ -856,49 +778,8 @@ def _trusted_snapshot_frame_csp(request: HttpRequest) -> str:
|
||||
"object-src 'none'; "
|
||||
"base-uri 'none'; "
|
||||
"form-action 'none'; "
|
||||
f"frame-ancestors 'self' {viewer_origins};"
|
||||
"frame-ancestors 'self';"
|
||||
)
|
||||
|
||||
|
||||
def _plugin_card_document_response(request: HttpRequest, snapshot: Snapshot, result_id: str) -> HttpResponse:
|
||||
"""Serve the existing card template from the snapshot origin, not web.*.
|
||||
|
||||
srcdoc and inline card scripts inherit the containing page's origin. This
|
||||
frame route gives them same-origin access to this snapshot's saved files
|
||||
without opening CORS on raw archives or sharing the admin session cookie.
|
||||
The parent web page receives only the one-bit login hint; delete buttons
|
||||
navigate to admin.* for a real authenticated confirmation.
|
||||
"""
|
||||
try:
|
||||
result_pk = UUID(result_id)
|
||||
except (TypeError, ValueError):
|
||||
raise Http404 from None
|
||||
result = snapshot.archiveresult_set.filter(pk=result_pk).first()
|
||||
if result is None:
|
||||
raise Http404
|
||||
|
||||
# There is exactly one plugin card template. The stack cover and expanded
|
||||
# tray point at this same document; only their containing card width differs.
|
||||
from archivebox.core.templatetags.core_tags import CARD_OVERFLOW_STYLE, render_plugin_card_document
|
||||
|
||||
fragment = render_plugin_card_document(
|
||||
template.Context({"request": request, "CONFIG": get_request_config(request, resolve_plugins=False)}),
|
||||
result,
|
||||
)
|
||||
if str(fragment).lstrip().lower().startswith("<!doctype html"):
|
||||
card_html = str(fragment)
|
||||
else:
|
||||
card_html = (
|
||||
'<!doctype html><html><head><meta charset="utf-8"><style>'
|
||||
"html,body{margin:0;width:100%;height:100%;overflow:hidden}"
|
||||
"body>iframe{display:block;width:100%;height:100%;border:0}"
|
||||
f"</style>{CARD_OVERFLOW_STYLE}</head><body>"
|
||||
f"{fragment}</body></html>"
|
||||
)
|
||||
response = HttpResponse(card_html, content_type="text/html; charset=utf-8")
|
||||
response.headers["Content-Security-Policy"] = _trusted_snapshot_frame_csp(request)
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
response.headers["Referrer-Policy"] = "no-referrer"
|
||||
return response
|
||||
|
||||
|
||||
@@ -1005,7 +886,7 @@ def _serve_responses_path(request, responses_root: Path, rel_path: str, show_ind
|
||||
return None
|
||||
|
||||
|
||||
def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
rel_path = path or ""
|
||||
request_config = get_request_config(
|
||||
request,
|
||||
@@ -1013,11 +894,10 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
|
||||
)
|
||||
request.archivebox_config = request_config
|
||||
request.archivebox_snapshot_url = snapshot.url
|
||||
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
snapshot._runtime_config = request_config
|
||||
|
||||
if rel_path.startswith("replay/") or rel_path == "replay":
|
||||
response = serve_plugin_replay_asset(rel_path, request_config, HttpResponse)
|
||||
response = archivewebpage_replay.serve_replay_asset_response(rel_path, request_config, HttpResponse)
|
||||
if response is not None:
|
||||
return response
|
||||
|
||||
@@ -1026,26 +906,11 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
|
||||
# /progress.json on through to the same view used everywhere else. The caller
|
||||
# passes snapshot_id explicitly in the query string — we don't read it from the
|
||||
# subdomain (this keeps the endpoint identical across all security modes).
|
||||
return live_progress_view(request, authorized_snapshot=snapshot)
|
||||
|
||||
if rel_path.startswith("_card/"):
|
||||
# SnapshotHostView/SnapshotReplayView have already authorized this
|
||||
# snapshot. Look up the result through its relation so a card ID from
|
||||
# another isolated snapshot can never render here.
|
||||
return _plugin_card_document_response(request, snapshot, rel_path.removeprefix("_card/"))
|
||||
return live_progress_view(request)
|
||||
|
||||
is_directory_request = bool(path) and path.endswith("/")
|
||||
show_indexes = bool(request.GET.get("files")) or (request_config.USES_SUBDOMAIN_ROUTING and is_directory_request)
|
||||
if not show_indexes and (not rel_path or rel_path == "index.html"):
|
||||
if request_config.CONTROL_PLANE_ENABLED and snapshot.permissions in (PERMISSIONS_PUBLIC, PERMISSIONS_UNLISTED):
|
||||
# Snapshot detail is trusted application UI and belongs on web.*.
|
||||
# snap-* is for isolated saved outputs and plugin frames. Keep the
|
||||
# private replay-cookie path below: its grant is host-only and must
|
||||
# never be mistaken for an admin session on web.*.
|
||||
target = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request, config=request_config)
|
||||
if request.META.get("QUERY_STRING"):
|
||||
target = f"{target}?{request.META['QUERY_STRING']}"
|
||||
return redirect(target)
|
||||
return SnapshotView.render_live_index(request, snapshot)
|
||||
|
||||
if not rel_path or rel_path.endswith("/"):
|
||||
@@ -1057,23 +922,22 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
|
||||
if rel_path is None:
|
||||
raise Http404
|
||||
|
||||
rel_path, archive_result, fallback_relpaths = _resolve_archiveresult_relpath(snapshot, rel_path)
|
||||
rel_path, archive_result = _resolve_archiveresult_relpath(snapshot, rel_path)
|
||||
|
||||
plugin_preview = _plugin_full_preview_response(request, snapshot, rel_path, archive_result)
|
||||
if plugin_preview is not None:
|
||||
return plugin_preview
|
||||
|
||||
for candidate_path in (rel_path, *fallback_relpaths):
|
||||
try:
|
||||
return serve_static_with_byterange_support(
|
||||
request,
|
||||
candidate_path,
|
||||
document_root=snapshot.output_dir,
|
||||
show_indexes=show_indexes,
|
||||
is_archive_replay=True,
|
||||
)
|
||||
except Http404:
|
||||
continue
|
||||
try:
|
||||
return serve_static_with_byterange_support(
|
||||
request,
|
||||
rel_path,
|
||||
document_root=snapshot.output_dir,
|
||||
show_indexes=show_indexes,
|
||||
is_archive_replay=True,
|
||||
)
|
||||
except Http404:
|
||||
pass
|
||||
|
||||
host = urlparse(snapshot.url).hostname or snapshot.domain
|
||||
responses_root = Path(snapshot.output_dir) / "responses" / host
|
||||
@@ -1085,22 +949,6 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
|
||||
raise Http404
|
||||
|
||||
|
||||
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
|
||||
response = _build_snapshot_replay_response(request, snapshot, path)
|
||||
if response.status_code in (200, 206, 304) and response.get("Cache-Control") == FAVICON_CACHE_CONTROL:
|
||||
return response
|
||||
if path != "progress.json" and response.status_code in (200, 206, 304):
|
||||
if not response.get("Cache-Control"):
|
||||
policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
|
||||
response.headers["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
|
||||
if snapshot.permissions != PERMISSIONS_PUBLIC:
|
||||
patch_cache_control(response, private=True)
|
||||
else:
|
||||
response.headers["Cache-Control"] = "private, no-store"
|
||||
patch_vary_headers(response, ("Cookie", "Authorization"))
|
||||
return response
|
||||
|
||||
|
||||
def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str = ""):
|
||||
request_config = get_request_config(request, resolve_plugins=False)
|
||||
request.archivebox_config = request_config
|
||||
@@ -1124,8 +972,6 @@ def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str =
|
||||
|
||||
root_match = (match[0], match[1]) if match else _latest_responses_root(snapshots, domain)
|
||||
responses_root = root_match[1] if root_match else None
|
||||
if root_match:
|
||||
request.archivebox_cache_policy = "public" if root_match[0].permissions == PERMISSIONS_PUBLIC else "private"
|
||||
if request_config.USES_SUBDOMAIN_ROUTING:
|
||||
snapshot = root_match[0] if root_match else (snapshots[0] if requested_root_index and snapshots else None)
|
||||
if snapshot:
|
||||
@@ -1212,18 +1058,6 @@ class OriginalDomainReplayView(View):
|
||||
|
||||
|
||||
class PublicIndexView(ListView):
|
||||
"""Keep browsing responsive while the same small server is archiving.
|
||||
|
||||
Performance takes priority over perfect initial totals: on a 1 vCPU / 1 GB
|
||||
host with hundreds of thousands of snapshots, even an indexed count can
|
||||
consume the entire roughly 0.5-0.7 second page-load budget. Benchmark with
|
||||
the runner active, since it shares the CPU, memory, and SQLite database.
|
||||
|
||||
The unfiltered list therefore renders rows first and accepts missing/stale
|
||||
display totals. A later ordinary reload can show the background-cached
|
||||
count; do not restore a synchronous COUNT or add polling just to fill it in.
|
||||
"""
|
||||
|
||||
template_name = "public_index.html"
|
||||
model = Snapshot
|
||||
ordering: ClassVar[list[str]] = ["-bookmarked_at", "-created_at"]
|
||||
@@ -1249,28 +1083,14 @@ class PublicIndexView(ListView):
|
||||
"status",
|
||||
"output_size",
|
||||
"permissions",
|
||||
# Modern replay URLs need the filesystem version and owner's name.
|
||||
# Deferring these caused four extra SELECTs per displayed snapshot.
|
||||
"fs_version",
|
||||
"crawl__created_by__username",
|
||||
)
|
||||
|
||||
def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot]:
|
||||
# One extra row determines Next without a full collection count. Cached
|
||||
# totals can be stale while the runner adds/removes snapshots, so they
|
||||
# must never decide whether the requested page exists.
|
||||
target_count = page_number * page_size + 1
|
||||
def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot] | None:
|
||||
target_count = page_number * page_size
|
||||
public_snapshots: list[Snapshot] = []
|
||||
scanned = 0
|
||||
chunk_size = max(self.public_page_scan_chunk_size, page_size + 1)
|
||||
# Read in snapshot_public_order_idx order and filter visibility in small
|
||||
# batches. Combining the permissions predicate with ORDER BY can make
|
||||
# SQLite select the permissions index and sort the whole public archive
|
||||
# before LIMIT. This favors the common early pages without a new index;
|
||||
# deep pages or mostly private collections can still require more reads.
|
||||
ordered_snapshots = (
|
||||
Snapshot.objects.select_related("crawl__created_by").order_by(*self.ordering).only(*self._base_public_snapshot_fields())
|
||||
)
|
||||
chunk_size = max(self.public_page_scan_chunk_size, page_size)
|
||||
ordered_snapshots = Snapshot.objects.order_by(*self.ordering).only(*self._base_public_snapshot_fields())
|
||||
|
||||
while len(public_snapshots) < target_count:
|
||||
chunk = list(ordered_snapshots[scanned : scanned + chunk_size])
|
||||
@@ -1286,39 +1106,20 @@ class PublicIndexView(ListView):
|
||||
if self.request.GET.get("q", default="").strip():
|
||||
return super().paginate_queryset(queryset, page_size)
|
||||
|
||||
public_count = self.get_exact_public_snapshot_count()
|
||||
paginator = self.get_paginator(range(public_count), page_size)
|
||||
page_kwarg = self.kwargs.get(self.page_kwarg)
|
||||
page_query = self.request.GET.get(self.page_kwarg)
|
||||
page_number = page_kwarg or page_query or 1
|
||||
|
||||
try:
|
||||
page_number = int(page_number)
|
||||
if page_number < 1:
|
||||
raise ValueError("Page must be positive")
|
||||
except (ValueError, TypeError) as err:
|
||||
page = paginator.page(page_number)
|
||||
except InvalidPage as err:
|
||||
raise Http404(f"Invalid page ({page_number}): {err}") from err
|
||||
|
||||
snapshots = self._ordered_public_page_from_order_index(page_number=page_number, page_size=page_size)
|
||||
if not snapshots and page_number > 1:
|
||||
raise Http404(f"Invalid page ({page_number}): No snapshots")
|
||||
# Exhausting the rows gives an exact total for this read at no extra
|
||||
# cost. Otherwise the lookahead is only a lower bound: an absent count
|
||||
# is preferable to an on-request scan or a cached total smaller than
|
||||
# the rows we just observed. Cached overestimates are acceptable here.
|
||||
observed_count = (page_number - 1) * page_size + len(snapshots)
|
||||
self.public_snapshot_count = observed_count
|
||||
if len(snapshots) > page_size:
|
||||
summary = collection_summary(self.request.user, public=True)
|
||||
cached_count = summary["snapshots"] if summary is not None else None
|
||||
self.public_snapshot_count = cached_count if cached_count is not None and cached_count >= observed_count else None
|
||||
|
||||
# Give Django an in-memory lower bound, never the collection QuerySet
|
||||
# whose Paginator.count would issue COUNT(*). Navigation uses this read's
|
||||
# lookahead even when a stale total would hide a newly added page.
|
||||
# The template must use the separate display totals, not num_pages here.
|
||||
paginator = self.get_paginator(range(observed_count), page_size)
|
||||
page = paginator.page(page_number)
|
||||
page.object_list = snapshots[:page_size]
|
||||
return paginator, page, page.object_list, page.has_other_pages()
|
||||
object_list = self._ordered_public_page_from_order_index(page_number=page.number, page_size=page_size)
|
||||
page.object_list = object_list
|
||||
return paginator, page, object_list, page.has_other_pages()
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
runtime_config = self.__dict__.get("runtime_config")
|
||||
@@ -1347,20 +1148,11 @@ class PublicIndexView(ListView):
|
||||
and search_mode_backend
|
||||
and context["paginator"].count == 0,
|
||||
)
|
||||
context["public_snapshot_count"] = context["paginator"].count if query else self.public_snapshot_count
|
||||
count = context["public_snapshot_count"]
|
||||
context["public_page_count"] = (
|
||||
max(1, (count + context["paginator"].per_page - 1) // context["paginator"].per_page) if count is not None else None
|
||||
)
|
||||
snapshots = list(context.get("object_list") or ())
|
||||
from archivebox.plugins.output_groups import plugin_output_sizes
|
||||
|
||||
all_results_by_snapshot = {str(snapshot.id): [] for snapshot in snapshots}
|
||||
snapshots_by_id = {str(snapshot.id): snapshot for snapshot in snapshots}
|
||||
icons_by_snapshot: dict[str, dict[str, ArchiveResult]] = {str(snapshot.id): {} for snapshot in snapshots}
|
||||
icons_by_snapshot: dict[str, set[str]] = {str(snapshot.id): set() for snapshot in snapshots}
|
||||
tag_names_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
|
||||
list_icon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
|
||||
list_icon_plugins = set(get_snapshot_role_names("list_icon"))
|
||||
preview_paths_by_snapshot: dict[str, list[tuple[int, int, str]]] = {str(snapshot.id): [] for snapshot in snapshots}
|
||||
favicon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
|
||||
progress_by_snapshot: dict[str, dict[str, int]] = {
|
||||
str(snapshot.id): {
|
||||
"total": 0,
|
||||
@@ -1381,66 +1173,89 @@ class PublicIndexView(ListView):
|
||||
):
|
||||
tag_names_by_snapshot[str(snapshot_id)].append(tag_name)
|
||||
|
||||
results = (
|
||||
ArchiveResult.objects.filter(snapshot_id__in=icons_by_snapshot.keys())
|
||||
.exclude(plugin="")
|
||||
.only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_files",
|
||||
"output_size",
|
||||
preview_plugin_order = {
|
||||
"screenshot": 0,
|
||||
"chrome_extension_screenshot": 1,
|
||||
}
|
||||
preview_candidates = {
|
||||
"screenshot": ("screenshot.png",),
|
||||
"chrome_extension_screenshot": ("screenshot-1.png", "screenshot.png"),
|
||||
}
|
||||
|
||||
def result_output_path(result: ArchiveResult, filename: str) -> str | None:
|
||||
output_files = result.output_files or {}
|
||||
file_info = output_files.get(filename)
|
||||
if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
|
||||
return None
|
||||
if file_info.get("root_relative"):
|
||||
return filename
|
||||
return f"{result.plugin}/{filename}"
|
||||
|
||||
for snapshot_id, plugin, status in (
|
||||
ArchiveResult.objects.filter(
|
||||
snapshot_id__in=icons_by_snapshot.keys(),
|
||||
)
|
||||
)
|
||||
# This is already bounded by the displayed snapshot page. Finish
|
||||
# the SELECT before parsing output manifests: a streaming cursor
|
||||
# otherwise keeps SQLite's read lock throughout rendering.
|
||||
for result in list(results):
|
||||
snapshot_key = str(result.snapshot_id)
|
||||
# Thumbnail URLs traverse result.snapshot too; share the page's
|
||||
# already-loaded parent rather than fetching it once per result.
|
||||
result.snapshot = snapshots_by_id[snapshot_key]
|
||||
# These read-only page objects are shared by icons, sizes, and
|
||||
# thumbnails. Normalize each manifest once for this render.
|
||||
result._render_output_file_map = result.output_file_map()
|
||||
all_results_by_snapshot[snapshot_key].append(result)
|
||||
.exclude(plugin="")
|
||||
.values_list("snapshot_id", "plugin", "status")
|
||||
.iterator(chunk_size=1000)
|
||||
):
|
||||
snapshot_key = str(snapshot_id)
|
||||
progress = progress_by_snapshot[snapshot_key]
|
||||
progress["total"] += 1
|
||||
if result.status == ArchiveResult.StatusChoices.SUCCEEDED:
|
||||
icons_by_snapshot[snapshot_key].setdefault(result.plugin, result)
|
||||
if status == ArchiveResult.StatusChoices.SUCCEEDED:
|
||||
icons_by_snapshot[snapshot_key].add(plugin)
|
||||
progress["succeeded"] += 1
|
||||
elif result.status == ArchiveResult.StatusChoices.FAILED:
|
||||
elif status == ArchiveResult.StatusChoices.FAILED:
|
||||
progress["failed"] += 1
|
||||
elif result.status == ArchiveResult.StatusChoices.STARTED:
|
||||
elif status == ArchiveResult.StatusChoices.STARTED:
|
||||
progress["running"] += 1
|
||||
elif result.status == ArchiveResult.StatusChoices.SKIPPED:
|
||||
elif status == ArchiveResult.StatusChoices.SKIPPED:
|
||||
progress["skipped"] += 1
|
||||
elif result.status == ArchiveResult.StatusChoices.NORESULTS:
|
||||
elif status == ArchiveResult.StatusChoices.NORESULTS:
|
||||
progress["noresults"] += 1
|
||||
if result.plugin in list_icon_plugins and result.status == ArchiveResult.StatusChoices.SUCCEEDED:
|
||||
if output_path := result.embed_path():
|
||||
list_icon_paths_by_snapshot[snapshot_key].append(output_path)
|
||||
|
||||
for result in (
|
||||
ArchiveResult.objects.filter(
|
||||
snapshot_id__in=icons_by_snapshot.keys(),
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
plugin__in=(*preview_candidates, "favicon"),
|
||||
)
|
||||
.only("snapshot_id", "plugin", "output_files")
|
||||
.iterator(chunk_size=1000)
|
||||
):
|
||||
snapshot_key = str(result.snapshot_id)
|
||||
if result.plugin in preview_candidates:
|
||||
plugin_rank = preview_plugin_order[result.plugin]
|
||||
for filename_rank, filename in enumerate(preview_candidates[result.plugin]):
|
||||
output_path = result_output_path(result, filename)
|
||||
if output_path:
|
||||
preview_paths_by_snapshot[snapshot_key].append((plugin_rank, filename_rank, output_path))
|
||||
elif result.plugin == "favicon":
|
||||
output_path = result_output_path(result, "favicon.ico")
|
||||
if output_path:
|
||||
favicon_paths_by_snapshot[snapshot_key].append(output_path)
|
||||
|
||||
for snapshot in snapshots:
|
||||
snapshot._icons_compact = True
|
||||
snapshot._icons_output_sizes = plugin_output_sizes(all_results_by_snapshot[str(snapshot.id)])
|
||||
snapshot._icons_archive_results = icons_by_snapshot.get(str(snapshot.id), set())
|
||||
snapshot._icons_progress_stats = progress_by_snapshot.get(str(snapshot.id), {})
|
||||
snapshot.num_outputs_cached = snapshot._icons_progress_stats.get("succeeded", 0)
|
||||
snapshot._tags_str_cached = ",".join(tag_names_by_snapshot.get(str(snapshot.id), []))
|
||||
snapshot._snapshot_card_results = all_results_by_snapshot[str(snapshot.id)]
|
||||
snapshot._public_list_icon_paths = list_icon_paths_by_snapshot.get(str(snapshot.id), [])
|
||||
snapshot._public_preview_paths = [
|
||||
output_path for _plugin_rank, _filename_rank, output_path in sorted(preview_paths_by_snapshot.get(str(snapshot.id), []))
|
||||
]
|
||||
snapshot._public_favicon_paths = favicon_paths_by_snapshot.get(str(snapshot.id), [])
|
||||
snapshot._is_archived_cached = bool(snapshot.downloaded_at or snapshot.status == Snapshot.StatusChoices.SEALED)
|
||||
context["object_list"] = snapshots
|
||||
return context
|
||||
|
||||
def get_exact_public_snapshot_count(self) -> int:
|
||||
hidden_count = Snapshot.objects.filter(permissions=PERMISSIONS_PRIVATE).count()
|
||||
hidden_count += Snapshot.objects.filter(permissions=PERMISSIONS_UNLISTED).count()
|
||||
return Snapshot.objects.count() - hidden_count
|
||||
|
||||
def get_queryset(self, **kwargs):
|
||||
qs = (
|
||||
public_snapshots_queryset(super().get_queryset(**kwargs))
|
||||
.select_related("crawl__created_by")
|
||||
.only(*self._base_public_snapshot_fields())
|
||||
)
|
||||
qs = public_snapshots_queryset(super().get_queryset(**kwargs)).only(*self._base_public_snapshot_fields())
|
||||
query = self.request.GET.get("q", default="").strip()
|
||||
|
||||
if not query:
|
||||
@@ -1719,7 +1534,7 @@ class WebAddView(AddView):
|
||||
if requested_url:
|
||||
snapshot = self._latest_snapshot_for_url(requested_url)
|
||||
if snapshot:
|
||||
return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))
|
||||
return redirect(f"/{snapshot.url_path}")
|
||||
|
||||
request_host = (request.get_host() or "").lower()
|
||||
request_config = get_request_config(request)
|
||||
@@ -1759,7 +1574,7 @@ class WebAddView(AddView):
|
||||
|
||||
snapshot = self._latest_snapshot_for_url(requested_url)
|
||||
if snapshot:
|
||||
return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))
|
||||
return redirect(f"/{snapshot.url_path}")
|
||||
|
||||
add_url = self._normalize_add_url(requested_url)
|
||||
assert self.form_class is not None
|
||||
@@ -1791,7 +1606,7 @@ class WebAddView(AddView):
|
||||
crawl = self._create_crawl_from_form(form)
|
||||
snapshot = Snapshot.from_json({"url": add_url, "tags": form.cleaned_data.get("tag", "")}, overrides={"crawl": crawl})
|
||||
assert snapshot is not None
|
||||
return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))
|
||||
return redirect(f"/{snapshot.url_path}")
|
||||
|
||||
|
||||
class HealthCheckView(View):
|
||||
|
||||
@@ -717,7 +717,7 @@ class InlineTagEditorWidget(TagEditorWidget):
|
||||
)
|
||||
pills_html += f'''
|
||||
<span class="tag-pill" data-tag="{self._escape(td["name"])}" data-tag-id="{td["id"]}" style="{self._tag_style(td["name"])}">
|
||||
<a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link" title="{self._escape(td["name"])}">{self._escape(td["name"])}</a>
|
||||
<a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link">{self._escape(td["name"])}</a>
|
||||
{remove_button}
|
||||
</span>
|
||||
'''
|
||||
|
||||
+12
-55
@@ -8,7 +8,7 @@ from urllib.parse import urlencode, urlparse
|
||||
from django import forms
|
||||
from django.contrib import admin, messages
|
||||
from django.core.paginator import Paginator
|
||||
from django.db.models import Count, F, Prefetch, Q
|
||||
from django.db.models import Count, F, Q
|
||||
from django.http import HttpRequest, HttpResponseBadRequest, HttpResponseNotAllowed, JsonResponse
|
||||
from django.shortcuts import get_object_or_404, redirect
|
||||
from django.template.loader import render_to_string
|
||||
@@ -30,11 +30,9 @@ from archivebox.core.permissions import (
|
||||
PERMISSIONS_VALUES,
|
||||
normalize_permissions,
|
||||
)
|
||||
from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_role_output_url
|
||||
from archivebox.core.widgets import TagEditorWidget, URLFiltersWidget
|
||||
from archivebox.crawls.models import Crawl, CrawlSchedule
|
||||
from archivebox.misc.paginators import AcceleratedPaginator
|
||||
from archivebox.plugins.discovery import get_snapshot_role_names
|
||||
from archivebox.progressmonitor.views import progress_endpoint
|
||||
from archivebox.workers.models import RETRY_AT_MAX
|
||||
|
||||
@@ -74,32 +72,13 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,
|
||||
|
||||
# Keep ArchiveResult counters as scalar subqueries so the paginated
|
||||
# Snapshot queryset does not become a join+GROUP BY over every result row.
|
||||
snapshots_qs = (
|
||||
filtered_qs.order_by("-created_at")
|
||||
.annotate(
|
||||
total_results=ArchiveResult.snapshot_count_expr(),
|
||||
succeeded_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SUCCEEDED),
|
||||
failed_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.FAILED),
|
||||
started_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.STARTED),
|
||||
skipped_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SKIPPED),
|
||||
snapshot_permissions=F("permissions"),
|
||||
)
|
||||
.prefetch_related(
|
||||
Prefetch(
|
||||
"archiveresult_set",
|
||||
queryset=ArchiveResult.objects.filter(
|
||||
plugin__in=get_snapshot_role_names("list_icon"),
|
||||
status=ArchiveResult.StatusChoices.SUCCEEDED,
|
||||
).only(
|
||||
"snapshot_id",
|
||||
"plugin",
|
||||
"status",
|
||||
"output_str",
|
||||
"output_files",
|
||||
),
|
||||
to_attr="_list_icon_results",
|
||||
),
|
||||
)
|
||||
snapshots_qs = filtered_qs.order_by("-created_at").annotate(
|
||||
total_results=ArchiveResult.snapshot_count_expr(),
|
||||
succeeded_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SUCCEEDED),
|
||||
failed_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.FAILED),
|
||||
started_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.STARTED),
|
||||
skipped_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SKIPPED),
|
||||
snapshot_permissions=F("permissions"),
|
||||
)
|
||||
|
||||
page_number = request.GET.get(page_param, 1) if request is not None else 1
|
||||
@@ -240,14 +219,6 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,
|
||||
|
||||
# Format date
|
||||
date_str = snapshot.created_at.strftime("%Y-%m-%d %H:%M") if snapshot.created_at else "-"
|
||||
detail_url = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
|
||||
favicon_url = build_snapshot_role_output_url(
|
||||
snapshot,
|
||||
"list_icon",
|
||||
request=request,
|
||||
archive_results=snapshot._list_icon_results,
|
||||
fallback_to_default=True,
|
||||
)
|
||||
|
||||
rows.append(f'''
|
||||
<tr style="border-bottom: 1px solid #eee;">
|
||||
@@ -260,8 +231,8 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,
|
||||
<span title="{permissions}" style="display:inline-flex; align-items:center; justify-content:center; width:22px; height:22px; border-radius:999px; font-size:12px; color:{permission_fg}; background:{permission_bg};">{permission_icon}</span>
|
||||
</td>
|
||||
<td style="padding: 6px 8px; white-space: nowrap;">
|
||||
<a href="{escape(detail_url)}" style="text-decoration: none;">
|
||||
<img src="{escape(favicon_url)}"
|
||||
<a href="/{snapshot.archive_path}/" style="text-decoration: none;">
|
||||
<img src="/{snapshot.archive_path}/favicon.ico"
|
||||
style="width: 16px; height: 16px; vertical-align: middle; margin-right: 4px;"
|
||||
onerror="this.style.display='none'"/>
|
||||
</a>
|
||||
@@ -1259,7 +1230,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
|
||||
(
|
||||
"Configuration",
|
||||
{
|
||||
"fields": ("is_enabled", "schedule", "template"),
|
||||
"fields": ("schedule", "template"),
|
||||
"classes": ("card",),
|
||||
},
|
||||
),
|
||||
@@ -1305,21 +1276,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
|
||||
|
||||
def change_view(self, request, object_id, form_url="", extra_context=None):
|
||||
self.request = request
|
||||
from django.db import connections, router
|
||||
|
||||
connection = connections[router.db_for_write(self.model)]
|
||||
if request.method != "POST" or connection.vendor != "sqlite":
|
||||
return super().change_view(request, object_id, form_url, extra_context)
|
||||
# Reserve the writer before Django's atomic admin form reads. A
|
||||
# deferred read transaction cannot safely upgrade while the runner
|
||||
# writes, even when SQLite's existing busy timeout has time left.
|
||||
connection.ensure_connection()
|
||||
previous_mode = connection.transaction_mode
|
||||
connection.transaction_mode = "IMMEDIATE"
|
||||
try:
|
||||
return super().change_view(request, object_id, form_url, extra_context)
|
||||
finally:
|
||||
connection.transaction_mode = previous_mode
|
||||
return super().change_view(request, object_id, form_url, extra_context)
|
||||
|
||||
def add_view(self, request, form_url="", extra_context=None):
|
||||
return redirect("/add/#schedule")
|
||||
|
||||
+17
-22
@@ -467,8 +467,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
|
||||
"tags_str": self.tags_str,
|
||||
"label": self.label,
|
||||
"created_at": self.created_at.isoformat() if self.created_at else None,
|
||||
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
|
||||
"created_by": self.created_by.username,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
@@ -846,31 +844,24 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
|
||||
output_dir: Path | None = None,
|
||||
num_snapshots: int | None = None,
|
||||
) -> str:
|
||||
from abx_dl.limits import parse_filesize_to_bytes
|
||||
from abx_dl.limits import CrawlLimitState
|
||||
|
||||
if output_dir is None:
|
||||
output_dir = self.output_dir
|
||||
if config is None:
|
||||
from archivebox.config.common import get_config
|
||||
|
||||
config = get_config(crawl=self, include_machine=False)
|
||||
config = get_config(crawl=self, include_machine=False).for_crawl_runtime(
|
||||
crawl=self,
|
||||
persona=self.resolve_persona(),
|
||||
crawl_output_dir=output_dir,
|
||||
)
|
||||
|
||||
# The DB is the shared authority for workers and the UI. A second
|
||||
# ledger beside the artifacts can disagree after cancellation/retry and
|
||||
# would require filesystem access just to render a crawl's status.
|
||||
# Recompute against the current limits so increasing a cap permits work.
|
||||
timeout = int(self._config_value(config, "CRAWL_TIMEOUT", 0) or 0)
|
||||
if timeout:
|
||||
# Snapshot creation already supplies a durable crawl clock. Using
|
||||
# that existing fact avoids another timestamp to keep in sync and
|
||||
# survives cancellation, which deliberately removes unfinished
|
||||
# ArchiveResult rows while preserving the Snapshot for retry.
|
||||
first_snapshot_time = self.snapshot_set.aggregate(first=models.Min("created_at"))["first"]
|
||||
if first_snapshot_time and (timezone.now() - first_snapshot_time).total_seconds() >= timeout:
|
||||
return "crawl_timeout"
|
||||
max_size = parse_filesize_to_bytes(self._config_value(config, "CRAWL_MAX_SIZE", 0) or 0)
|
||||
if max_size:
|
||||
total_size = self.snapshot_set.aggregate(total=models.Sum("output_size"))["total"] or 0
|
||||
if total_size >= max_size:
|
||||
return "crawl_max_size"
|
||||
limits_path = output_dir / ".abx-dl" / "limits.json"
|
||||
if limits_path.exists():
|
||||
stop_reason = CrawlLimitState.from_config(config).get_stop_reason()
|
||||
if stop_reason:
|
||||
return stop_reason
|
||||
|
||||
max_urls = int(self._config_value(config, "CRAWL_MAX_URLS", 0) or 0)
|
||||
if num_snapshots is None:
|
||||
@@ -1257,6 +1248,10 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
|
||||
}
|
||||
if tag_names:
|
||||
tag_names_by_url[snapshot.url] = tag_names
|
||||
# Snapshot.save() handles model-level validation. The crawl symlink
|
||||
# can still wait until after commit so SQLite does not hold a write
|
||||
# lock while touching the filesystem.
|
||||
transaction.on_commit(lambda snapshot=snapshot: snapshot.ensure_crawl_symlink())
|
||||
|
||||
tag_names = {tag for tags in tag_names_by_url.values() for tag in tags}
|
||||
if tag_names:
|
||||
|
||||
@@ -57,10 +57,7 @@ def get_vm_info():
|
||||
|
||||
hw_manufacturer = "Docker" if hw_in_docker else "Unknown"
|
||||
hw_product = "Container" if hw_in_docker else "Unknown"
|
||||
try:
|
||||
hw_uuid = machineid.id()
|
||||
except machineid.MachineIdNotFound:
|
||||
hw_uuid = "Unknown"
|
||||
hw_uuid = machineid.id()
|
||||
|
||||
if platform.system().lower() == "darwin":
|
||||
# Get macOS machine info
|
||||
@@ -91,7 +88,7 @@ def get_vm_info():
|
||||
try:
|
||||
# Getting SMBIOS data from sysfs.
|
||||
# SMBIOS 2.8 present.
|
||||
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:5797 │
|
||||
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:8000 │
|
||||
# Handle 0x0100, DMI type 1, 27 bytes
|
||||
# System Information
|
||||
# Manufacturer: DigitalOcean
|
||||
@@ -304,9 +301,7 @@ def get_host_stats() -> dict[str, Any]:
|
||||
|
||||
|
||||
def get_host_guid() -> str:
|
||||
from archivebox.config.paths import get_machine_id
|
||||
|
||||
return get_machine_id(length=64)
|
||||
return machineid.hashed_id("archivebox")
|
||||
|
||||
|
||||
# Example usage
|
||||
|
||||
@@ -221,23 +221,29 @@ class Machine(ModelWithHealthStats):
|
||||
app_label = "machine"
|
||||
|
||||
@classmethod
|
||||
def current_readonly(cls) -> Machine | None:
|
||||
"""Read the current machine without registration, refresh, or config writes."""
|
||||
if _CURRENT_MACHINE is not None:
|
||||
return _CURRENT_MACHINE
|
||||
return cls.objects.filter(guid=get_host_guid()).first()
|
||||
|
||||
@classmethod
|
||||
def current(cls) -> Machine:
|
||||
def current(cls, refresh: bool = False) -> Machine:
|
||||
global _CURRENT_MACHINE
|
||||
if _CURRENT_MACHINE is not None:
|
||||
return _CURRENT_MACHINE
|
||||
if refresh:
|
||||
_CURRENT_MACHINE = None
|
||||
if _CURRENT_MACHINE:
|
||||
if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
|
||||
# One-time-per-process reconciliation between ArchiveBox.conf
|
||||
# and Machine.config. Fast-path: bool check + early-return when
|
||||
# the sync has already run, so the cached-machine return path
|
||||
# stays sub-microsecond.
|
||||
try:
|
||||
from archivebox.config.collection import sync_machine_and_file
|
||||
|
||||
sync_machine_and_file(_CURRENT_MACHINE)
|
||||
except Exception:
|
||||
pass
|
||||
return _CURRENT_MACHINE
|
||||
else:
|
||||
_CURRENT_MACHINE = None
|
||||
|
||||
host_guid = get_host_guid()
|
||||
# Host detection, sanitation, and config reconciliation run only on
|
||||
# initial registration, never on cached reads.
|
||||
try:
|
||||
machine = cls.objects.get(guid=host_guid)
|
||||
_CURRENT_MACHINE = cls.objects.get(guid=host_guid)
|
||||
except cls.DoesNotExist:
|
||||
config = {}
|
||||
try:
|
||||
@@ -247,7 +253,7 @@ class Machine(ModelWithHealthStats):
|
||||
config = _coerce_from_str_dict(file_config)
|
||||
except Exception:
|
||||
config = {}
|
||||
machine = cls.objects.create(
|
||||
_CURRENT_MACHINE = cls.objects.create(
|
||||
guid=host_guid,
|
||||
hostname=socket.gethostname(),
|
||||
config=config,
|
||||
@@ -256,15 +262,15 @@ class Machine(ModelWithHealthStats):
|
||||
stats=get_host_stats(),
|
||||
)
|
||||
else:
|
||||
if timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
|
||||
if timezone.now() >= _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
|
||||
for key, value in {
|
||||
"hostname": socket.gethostname(),
|
||||
**get_os_info(),
|
||||
**get_vm_info(),
|
||||
"stats": get_host_stats(),
|
||||
}.items():
|
||||
setattr(machine, key, value)
|
||||
machine.save(
|
||||
setattr(_CURRENT_MACHINE, key, value)
|
||||
_CURRENT_MACHINE.save(
|
||||
update_fields=[
|
||||
"hostname",
|
||||
"hw_in_docker",
|
||||
@@ -281,16 +287,16 @@ class Machine(ModelWithHealthStats):
|
||||
"modified_at",
|
||||
],
|
||||
)
|
||||
machine = cls._sanitize_config(machine)
|
||||
# Reconcile once, on the first Machine.current() call in this process.
|
||||
machine = cls._sanitize_config(_CURRENT_MACHINE)
|
||||
# Same one-time sync as the cached-return path. Triggers here on the
|
||||
# very first ``Machine.current()`` call in a process before any
|
||||
# cached return can occur.
|
||||
try:
|
||||
from archivebox.config.collection import sync_machine_and_file
|
||||
|
||||
sync_machine_and_file(machine)
|
||||
sync_machine_and_file(_CURRENT_MACHINE)
|
||||
except Exception:
|
||||
pass
|
||||
# Publish only after initialization is complete.
|
||||
_CURRENT_MACHINE = machine
|
||||
return machine
|
||||
|
||||
@classmethod
|
||||
@@ -360,15 +366,18 @@ class Machine(ModelWithHealthStats):
|
||||
if update_fields is not None:
|
||||
kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))
|
||||
|
||||
# Apply explicit edits to the cached instance without making the next
|
||||
# read repeat host detection, sanitation, or config reconciliation.
|
||||
# Drop the ``Machine.current()`` module-level cache on every save so
|
||||
# config edits (admin form, from_json, etc.) become live in the same
|
||||
# process without waiting out the 7-day ``MACHINE_RECHECK_INTERVAL``.
|
||||
# The cache reads ``_CURRENT_MACHINE.modified_at`` and that value
|
||||
# never moves forward on the cached object even when the row is
|
||||
# updated in the DB, so without this we'd keep serving stale
|
||||
# ``machine.config`` (incl. ``BASE_URL``) until the worker restarts.
|
||||
update_fields = kwargs.get("update_fields")
|
||||
super().save(*args, **kwargs)
|
||||
global _CURRENT_MACHINE
|
||||
if _CURRENT_MACHINE is not None and _CURRENT_MACHINE.pk == self.pk:
|
||||
for field in self._meta.concrete_fields:
|
||||
if field.attname in self.__dict__ and (update_fields is None or field.name in update_fields):
|
||||
setattr(_CURRENT_MACHINE, field.attname, self.__dict__[field.attname])
|
||||
_CURRENT_MACHINE = None
|
||||
|
||||
# Mirror Machine.config into ArchiveBox.conf so the two stores stay
|
||||
# 1:1. Skipped when ``update_fields`` is set and doesn't touch
|
||||
@@ -423,7 +432,7 @@ class NetworkInterface(ModelWithHealthStats):
|
||||
@classmethod
|
||||
def current(cls, refresh: bool = False) -> NetworkInterface:
|
||||
global _CURRENT_INTERFACE
|
||||
machine = Machine.current()
|
||||
machine = Machine.current(refresh=refresh)
|
||||
if _CURRENT_INTERFACE and _CURRENT_INTERFACE.machine_id == machine.id:
|
||||
if not refresh:
|
||||
# Callers that pass refresh=False are asking for attribution to
|
||||
@@ -1758,8 +1767,7 @@ class Process(ModelWithDeleteAfter, models.Model):
|
||||
return None
|
||||
|
||||
# Can't validate processes on other machines
|
||||
machine = Machine.current_readonly()
|
||||
if machine is None or self.machine_id != machine.id:
|
||||
if self.machine_id != Machine.current().id:
|
||||
return None
|
||||
|
||||
try:
|
||||
|
||||
@@ -255,7 +255,7 @@ def check_data_dir_permissions(config=None, **config_kwargs):
|
||||
" [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions]https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions[/link]",
|
||||
)
|
||||
STDERR.print(
|
||||
" [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid[/link]",
|
||||
" [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid[/link]",
|
||||
)
|
||||
STDERR.print(
|
||||
" [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts]https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts[/link]",
|
||||
|
||||
+3
-13
@@ -445,10 +445,7 @@ def migration_lock(stdout: TextIO | None = None):
|
||||
|
||||
|
||||
# Migration names that previously existed in ArchiveBox's source tree but have
|
||||
# since been deleted (squashed away, renamed, moved between apps, etc.), plus
|
||||
# exact older migrations generated in the installed package by v0.6.2's
|
||||
# startup makemigrations. Those runtime-generated names were never in Git, so
|
||||
# the deleted-file history alone cannot cover them. DBs
|
||||
# since been deleted (squashed away, renamed, moved between apps, etc.). DBs
|
||||
# upgraded incrementally through 0.8.x → 0.9.x dev rcs accumulate rows for
|
||||
# these in ``django_migrations``; the newer-DB guard added in 65dc2521 would
|
||||
# otherwise refuse to start with "applied migrations missing from this build"
|
||||
@@ -457,18 +454,13 @@ def migration_lock(stdout: TextIO | None = None):
|
||||
# splits the migration graph when only a *subset* of the replaces list is
|
||||
# applied (which is exactly what happens for users at different intermediate
|
||||
# dev branch states). This set is the authoritative compat list — extend it
|
||||
# when squashing more migrations away. The source-tracked names were generated
|
||||
# from ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
|
||||
# when squashing more migrations away. Generated from
|
||||
# ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
|
||||
HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
|
||||
{
|
||||
# core: 0023→0075 sequence plus every transient dev rename
|
||||
("core", "0002_auto_20190417_0739"),
|
||||
("core", "0006_auto_20200915_2006"),
|
||||
# Reported v0.6.2 runtime-generated 0021 migrations; the checked-in
|
||||
# replacement only changes ArchiveResult.extractor choices.
|
||||
("core", "0021_auto_20220510_0644"),
|
||||
("core", "0021_auto_20220724_1254"),
|
||||
("core", "0021_auto_20221128_1116"),
|
||||
("core", "0023_alter_archiveresult_options_archiveresult_abid_and_more"),
|
||||
("core", "0023_new_schema"),
|
||||
("core", "0024_auto_20240513_1143"),
|
||||
@@ -526,8 +518,6 @@ HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
|
||||
("core", "0053_remove_snapshottag_snapshot_old"),
|
||||
("core", "0054_alter_snapshot_timestamp"),
|
||||
("core", "0055_alter_tag_slug"),
|
||||
# Dev-only progress indexes removed in favor of existing-index queries.
|
||||
("core", "0056_progress_covering_indexes"),
|
||||
("core", "0056_remove_tag_uuid"),
|
||||
("core", "0057_rename_id_tag_old_id"),
|
||||
("core", "0058_alter_tag_old_id"),
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
# Bootable hashing helpers (file content + mtime/size cache, mime detection).
|
||||
# MUST NOT import archivebox.config, archivebox.core, or Django — stdlib only.
|
||||
|
||||
import hashlib
|
||||
import mimetypes
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from collections.abc import Callable
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
@lru_cache(maxsize=1024)
|
||||
def _cached_file_hash(filepath: str, size: int, mtime: float) -> str:
|
||||
"""Internal function to calculate file hash with cache key based on path, size and mtime."""
|
||||
sha256_hash = hashlib.sha256()
|
||||
|
||||
with open(filepath, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
sha256_hash.update(chunk)
|
||||
|
||||
return sha256_hash.hexdigest()
|
||||
|
||||
|
||||
@lru_cache(maxsize=10)
|
||||
def hash_file(file_path: Path, pwd: Path | None = None) -> str:
|
||||
"""Calculate SHA256 hash of a file with caching based on path, size and mtime."""
|
||||
pwd = Path(pwd) if pwd else None
|
||||
file_path = Path(file_path)
|
||||
if not file_path.is_absolute():
|
||||
file_path = pwd / file_path if pwd else file_path.absolute()
|
||||
|
||||
abs_path = file_path.resolve()
|
||||
stat_info = abs_path.stat()
|
||||
|
||||
return _cached_file_hash(
|
||||
str(abs_path),
|
||||
stat_info.st_size,
|
||||
stat_info.st_mtime,
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=10)
|
||||
def get_dir_hashes(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict[str, str]:
|
||||
"""Calculate SHA256 hashes for all files and directories recursively."""
|
||||
pwd = Path(pwd) if pwd else None
|
||||
dir_path = Path(dir_path)
|
||||
if not dir_path.is_absolute():
|
||||
dir_path = pwd / dir_path if pwd else dir_path.absolute()
|
||||
|
||||
if not dir_path.is_dir():
|
||||
raise ValueError(f"Not a directory: {dir_path}")
|
||||
if max_depth < -1:
|
||||
raise ValueError(f"max_depth must be >= -1, got {max_depth}")
|
||||
|
||||
# Get all files recursively
|
||||
all_files = get_dir_entries(
|
||||
dir_path,
|
||||
pwd=pwd,
|
||||
recursive=True,
|
||||
include_files=True,
|
||||
include_dirs=False,
|
||||
filter_func=filter_func,
|
||||
)
|
||||
|
||||
hashes: dict[str, str] = {}
|
||||
hashable_summary = []
|
||||
|
||||
# Calculate hashes for all files
|
||||
for subfile in all_files:
|
||||
subfile_path = dir_path / subfile
|
||||
sha256_hash = hash_file(subfile_path)
|
||||
hashes[subfile] = sha256_hash
|
||||
hashable_summary.append(f"{sha256_hash} ./{subfile}")
|
||||
|
||||
# Calculate hashes for all directories
|
||||
subdirs = get_dir_entries(
|
||||
dir_path,
|
||||
pwd=pwd,
|
||||
recursive=True,
|
||||
include_files=False,
|
||||
include_dirs=True,
|
||||
include_hidden=False,
|
||||
filter_func=filter_func,
|
||||
max_depth=max_depth,
|
||||
)
|
||||
|
||||
for subdir in subdirs:
|
||||
subdir_path = dir_path / subdir
|
||||
subdir_hashes = get_dir_hashes(
|
||||
subdir_path,
|
||||
filter_func=filter_func,
|
||||
max_depth=0,
|
||||
)
|
||||
hashes[subdir] = subdir_hashes["."]
|
||||
|
||||
# Filter results by max_depth
|
||||
if max_depth >= 0:
|
||||
hashes = {path: value for path, value in hashes.items() if len(Path(path).parts) <= max_depth + 1}
|
||||
|
||||
# Calculate root directory hash
|
||||
hashable_summary.sort()
|
||||
root_sha256 = hashlib.sha256("\n".join(hashable_summary).encode()).hexdigest()
|
||||
hashes["."] = root_sha256
|
||||
|
||||
return hashes
|
||||
|
||||
|
||||
@lru_cache(maxsize=128)
|
||||
def get_dir_entries(
|
||||
dir_path: Path,
|
||||
pwd: Path | None = None,
|
||||
recursive: bool = True,
|
||||
include_files: bool = True,
|
||||
include_dirs: bool = True,
|
||||
include_hidden: bool = False,
|
||||
filter_func: Callable | None = None,
|
||||
max_depth: int = -1,
|
||||
) -> tuple[str, ...]:
|
||||
"""Get filtered list of directory entries."""
|
||||
pwd = Path(pwd) if pwd else None
|
||||
dir_path = Path(dir_path)
|
||||
if not dir_path.is_absolute():
|
||||
dir_path = pwd / dir_path if pwd else dir_path.absolute()
|
||||
|
||||
results = []
|
||||
|
||||
def process_path(path: Path, depth: int):
|
||||
if not include_hidden and path.name.startswith("."):
|
||||
return False
|
||||
if max_depth >= 0 and depth > max_depth:
|
||||
return False
|
||||
if filter_func:
|
||||
info = {
|
||||
"abspath": str(path.absolute()),
|
||||
"relpath": str(path.relative_to(dir_path)),
|
||||
}
|
||||
if not filter_func(info):
|
||||
return False
|
||||
return True
|
||||
|
||||
for path in dir_path.rglob("*") if recursive else dir_path.glob("*"):
|
||||
current_depth = len(path.relative_to(dir_path).parts)
|
||||
|
||||
if path.is_file() and include_files and process_path(path, current_depth):
|
||||
results.append(str(path.relative_to(dir_path)))
|
||||
elif path.is_dir() and include_dirs and process_path(path, current_depth):
|
||||
results.append(str(path.relative_to(dir_path)))
|
||||
|
||||
if not recursive:
|
||||
break
|
||||
|
||||
return tuple(sorted(results)) # Make immutable for caching
|
||||
|
||||
|
||||
@lru_cache(maxsize=1024)
|
||||
def get_dir_sizes(dir_path: Path, pwd: Path | None = None, **kwargs) -> dict[str, int]:
|
||||
"""Calculate sizes for all files and directories recursively."""
|
||||
sizes: dict[str, int] = {}
|
||||
hashes = get_dir_hashes(dir_path, pwd=pwd, **kwargs)
|
||||
dir_path = Path(dir_path)
|
||||
|
||||
for path_key in hashes:
|
||||
full_path = dir_path / path_key
|
||||
if full_path.is_file():
|
||||
sizes[path_key] = full_path.stat().st_size
|
||||
else:
|
||||
total = 0
|
||||
for file_path in full_path.rglob("*"):
|
||||
if file_path.is_file() and not file_path.name.startswith("."):
|
||||
total += file_path.stat().st_size
|
||||
sizes[path_key + "/"] = total
|
||||
|
||||
return sizes
|
||||
|
||||
|
||||
@lru_cache(maxsize=10)
|
||||
def get_dir_info(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict:
|
||||
"""Get detailed information about directory contents including hashes and sizes."""
|
||||
pwd = Path(pwd) if pwd else None
|
||||
dir_path = Path(dir_path)
|
||||
if not dir_path.is_absolute():
|
||||
dir_path = pwd / dir_path if pwd else dir_path.absolute()
|
||||
|
||||
hashes = get_dir_hashes(dir_path, pwd=pwd, filter_func=filter_func, max_depth=max_depth)
|
||||
sizes = get_dir_sizes(str(dir_path), pwd=pwd, filter_func=filter_func, max_depth=max_depth)
|
||||
|
||||
num_total_subpaths = sum(1 for name in hashes if name != ".")
|
||||
details = {}
|
||||
|
||||
for filename, sha256_hash in sorted(hashes.items()):
|
||||
abs_path = (dir_path / filename).resolve()
|
||||
stat_info = abs_path.stat()
|
||||
num_subpaths = sum(1 for p in hashes if p.startswith(filename + "/"))
|
||||
is_dir = abs_path.is_dir()
|
||||
if is_dir:
|
||||
mime_type = "inode/directory"
|
||||
basename = abs_path.name
|
||||
extension = ""
|
||||
num_bytes = sizes[filename + "/"]
|
||||
if filename == ".":
|
||||
num_subpaths = num_total_subpaths
|
||||
else:
|
||||
filename += "/"
|
||||
num_subpaths = num_subpaths
|
||||
else: # is_file
|
||||
num_subpaths = None
|
||||
mime_type = mimetypes.guess_type(str(abs_path))[0]
|
||||
extension = abs_path.suffix
|
||||
basename = abs_path.name.rsplit(extension, 1)[0]
|
||||
num_bytes = sizes[filename]
|
||||
|
||||
details[filename] = {
|
||||
"basename": basename,
|
||||
"mime_type": mime_type,
|
||||
"extension": extension,
|
||||
"num_subpaths": num_subpaths,
|
||||
"num_bytes": num_bytes,
|
||||
"hash_sha256": sha256_hash,
|
||||
"created_at": datetime.fromtimestamp(stat_info.st_ctime).isoformat(),
|
||||
"modified_at": datetime.fromtimestamp(stat_info.st_mtime).isoformat(),
|
||||
}
|
||||
|
||||
if filter_func and not filter_func(details[filename]):
|
||||
del details[filename]
|
||||
|
||||
return details
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import json
|
||||
|
||||
dir_info = get_dir_info(Path("."), max_depth=6)
|
||||
with open(".hashes.json", "w") as f:
|
||||
json.dump(dir_info, f, indent=4)
|
||||
print("Wrote .hashes.json")
|
||||
|
||||
# Example output:
|
||||
# {
|
||||
# ".": {
|
||||
# "basename": "misc",
|
||||
# "mime_type": "inode/directory",
|
||||
# "extension": "",
|
||||
# "num_subpaths": 25,
|
||||
# "num_bytes": 214677,
|
||||
# "hash_sha256": "addfacf88b2ff6b564846415fb7b21dcb7e63ee4e911bc0aec255ee354958530",
|
||||
# "created_at": "2024-12-04T00:08:38.537449",
|
||||
# "modified_at": "2024-12-04T00:08:38.537449"
|
||||
# },
|
||||
# "__init__.py": {
|
||||
# "basename": "__init__",
|
||||
# "mime_type": "text/x-python",
|
||||
# "extension": ".py",
|
||||
# "num_subpaths": null,
|
||||
# "num_bytes": 32,
|
||||
# "hash_sha256": "b0e5e7ff17db3b60535cf664282787767c336e3e203a43e21b6326c6fe457551",
|
||||
# "created_at": "2024-10-08T00:51:41.001359",
|
||||
# "modified_at": "2024-10-08T00:51:41.001359"
|
||||
# },
|
||||
# ...
|
||||
# }
|
||||
@@ -32,6 +32,7 @@ import sys
|
||||
import json
|
||||
from typing import Any, TextIO
|
||||
from collections.abc import Iterable, Iterator
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Type constants for JSONL records
|
||||
@@ -119,6 +120,19 @@ def read_stdin(stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
|
||||
yield record
|
||||
|
||||
|
||||
def read_file(path: Path) -> Iterator[dict[str, Any]]:
|
||||
"""
|
||||
Read JSONL or plain URLs from a file.
|
||||
|
||||
Yields parsed records as dicts.
|
||||
"""
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
record = parse_line(line)
|
||||
if record:
|
||||
yield record
|
||||
|
||||
|
||||
def read_args_or_stdin(args: Iterable[str], stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
|
||||
"""
|
||||
Read from CLI arguments if provided, otherwise from stdin.
|
||||
|
||||
+48
-125
@@ -14,10 +14,10 @@ import time
|
||||
import zipfile
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote, urlencode, urljoin
|
||||
|
||||
from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
|
||||
from django.contrib.staticfiles import finders
|
||||
from django.core.handlers.asgi import ASGIRequest
|
||||
from django.http import Http404, HttpResponse, HttpResponseNotModified, StreamingHttpResponse
|
||||
@@ -29,10 +29,8 @@ from django.views import static
|
||||
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.misc.logging_util import printable_filesize
|
||||
from archivebox.plugins.discovery import render_plugin_full_response
|
||||
|
||||
_HASHES_CACHE: dict[Path, tuple[float, dict[str, str]]] = {}
|
||||
FAVICON_CACHE_CONTROL = "public, max-age=31536000, s-maxage=31536000, immutable"
|
||||
IMG_SRC_ATTR_RE = re.compile(r'(<img\b[^>]*?\s(?:src|data-src)=["\'])([^"\']+)(["\'])', re.IGNORECASE)
|
||||
TRANSFORMED_HTML_PREVIEW_STYLE = """<style id="archivebox-static-html-preview-style">
|
||||
html {
|
||||
@@ -131,7 +129,7 @@ def _resolve_archive_path(document_root: str | Path, rel_path: str) -> tuple[Pat
|
||||
|
||||
def _cache_policy(config=None, **config_kwargs) -> str:
|
||||
config = config or get_config(resolve_plugins=False, **config_kwargs)
|
||||
return "public" if config.PERMISSIONS == "public" else "private"
|
||||
return "private" if config.PERMISSIONS == "private" else "public"
|
||||
|
||||
|
||||
def _format_direntry_timestamp(stat_result: os.stat_result) -> str:
|
||||
@@ -529,26 +527,6 @@ def _index_responses_paths_for_html_images(
|
||||
|
||||
best_matches: dict[str, str] = {}
|
||||
image_matches: set[str] = set()
|
||||
# Optional responses plugin file contract: index.jsonl records contain method,
|
||||
# url, status, and a path relative to responses/. No plugin code is imported.
|
||||
# Legacy captures without an index still use the filename lookup below.
|
||||
# The index retains complete URLs even when response filenames are shortened.
|
||||
index = responses_root / "index.jsonl"
|
||||
if index.is_file():
|
||||
with index.open(errors="replace") as records:
|
||||
for line in records:
|
||||
try:
|
||||
record = json.loads(line)
|
||||
encoded = _encoded_responses_image_url(record.get("url", ""), None)
|
||||
if encoded not in encoded_urls or record.get("method") != "GET" or record.get("status") != 200:
|
||||
continue
|
||||
candidate = (responses_root / record.get("path", "")).resolve()
|
||||
if not candidate.is_relative_to(responses_root.resolve()) or not candidate.is_file():
|
||||
continue
|
||||
best_matches[encoded] = posixpath.relpath(candidate, start=snapshot_root / posixpath.dirname(html_rel_path))
|
||||
image_matches.add(encoded)
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
continue
|
||||
image_suffixes = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".ico"}
|
||||
for candidate in responses_root.rglob("*"):
|
||||
if not candidate.is_file():
|
||||
@@ -583,30 +561,10 @@ def _rewrite_html_image_sources_to_responses(
|
||||
if image_url not in encoded_urls_by_src:
|
||||
encoded_urls_by_src[image_url] = _encoded_responses_image_url(image_url, page_url)
|
||||
|
||||
# Optional dom plugin artifact: dom/output.html. Only image src and
|
||||
# data-canonical-src attributes are read; no scripts or plugin code execute.
|
||||
# Absence of this artifact simply disables proxy-to-original aliases.
|
||||
aliases: dict[str, str] = {}
|
||||
|
||||
class ImageAliases(HTMLParser):
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = dict(attrs)
|
||||
original = attrs.get("data-canonical-src")
|
||||
proxy = attrs.get("src")
|
||||
if tag == "img" and original and proxy:
|
||||
original_key = _encoded_responses_image_url(original, page_url)
|
||||
proxy_key = _encoded_responses_image_url(proxy, page_url)
|
||||
if original_key and proxy_key:
|
||||
aliases[original_key] = proxy_key
|
||||
|
||||
dom = snapshot_root / "dom" / "output.html"
|
||||
if dom.is_file() and dom.stat().st_size <= 10 * 1024 * 1024:
|
||||
ImageAliases().feed(dom.read_text(errors="replace"))
|
||||
wanted = {encoded for encoded in encoded_urls_by_src.values() if encoded}
|
||||
response_paths = _index_responses_paths_for_html_images(
|
||||
snapshot_root,
|
||||
html_rel_path,
|
||||
wanted | {aliases[key] for key in wanted if key in aliases},
|
||||
{encoded_url for encoded_url in encoded_urls_by_src.values() if encoded_url},
|
||||
)
|
||||
rewrites = 0
|
||||
|
||||
@@ -614,7 +572,7 @@ def _rewrite_html_image_sources_to_responses(
|
||||
nonlocal rewrites
|
||||
image_url = html.unescape(match.group(2))
|
||||
encoded_url = encoded_urls_by_src.get(image_url)
|
||||
local_path = response_paths.get(encoded_url or "") or response_paths.get(aliases.get(encoded_url or "", ""))
|
||||
local_path = response_paths.get(encoded_url or "")
|
||||
if not local_path:
|
||||
return match.group(0)
|
||||
rewrites += 1
|
||||
@@ -631,17 +589,12 @@ def _rewrite_html_image_sources_for_request(
|
||||
) -> tuple[str, int]:
|
||||
if not document_root:
|
||||
return html_text, 0
|
||||
try:
|
||||
return _rewrite_html_image_sources_to_responses(
|
||||
html_text,
|
||||
document_root,
|
||||
html_rel_path,
|
||||
request.__dict__.get("archivebox_snapshot_url"),
|
||||
)
|
||||
except (OSError, ValueError, TypeError, RecursionError):
|
||||
# Saved plugin artifacts are optional and may be incomplete, malformed,
|
||||
# or removed concurrently. Image enrichment must not break file replay.
|
||||
return html_text, 0
|
||||
return _rewrite_html_image_sources_to_responses(
|
||||
html_text,
|
||||
document_root,
|
||||
html_rel_path,
|
||||
request.__dict__.get("archivebox_snapshot_url"),
|
||||
)
|
||||
|
||||
|
||||
def _apply_transformed_html_preview_style(html_text: str) -> str:
|
||||
@@ -652,22 +605,16 @@ def _apply_transformed_html_preview_style(html_text: str) -> str:
|
||||
return f"{TRANSFORMED_HTML_PREVIEW_STYLE}\n{html_text}"
|
||||
|
||||
|
||||
def _set_transformed_response_headers(
|
||||
response,
|
||||
fullpath: Path,
|
||||
statobj: os.stat_result,
|
||||
encoding: str | None,
|
||||
cache_policy: str,
|
||||
) -> None:
|
||||
def _set_transformed_response_headers(response, fullpath: Path, statobj: os.stat_result, encoding: str | None, config) -> None:
|
||||
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
|
||||
if encoding:
|
||||
response.headers["Content-Encoding"] = encoding
|
||||
|
||||
|
||||
def _render_markdown_fallback(text: str) -> str:
|
||||
if _markdown is not None:
|
||||
if _markdown is not None and not HTML_TAG_RE.search(text):
|
||||
try:
|
||||
return _markdown(
|
||||
text,
|
||||
@@ -911,22 +858,19 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
config = request.__dict__.get("archivebox_config")
|
||||
if config is None:
|
||||
config = get_config(resolve_plugins=False)
|
||||
cache_policy = getattr(request, "archivebox_cache_policy", None) or _cache_policy(config=config)
|
||||
fullpath, path = _resolve_archive_path(document_root, path)
|
||||
if os.access(fullpath, os.R_OK) and fullpath.is_dir():
|
||||
if request.GET.get("download") == "zip" and show_indexes:
|
||||
response = _build_directory_zip_response(
|
||||
return _build_directory_zip_response(
|
||||
fullpath,
|
||||
path,
|
||||
is_archive_replay=is_archive_replay,
|
||||
use_async_stream=_is_asgi_request(request),
|
||||
config=config,
|
||||
)
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
return response
|
||||
if show_indexes:
|
||||
response = _render_directory_index(request, path, fullpath)
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Last-Modified"] = http_date(fullpath.stat().st_mtime)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
@@ -940,15 +884,6 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
raise Http404(_("“%(path)s” does not exist") % {"path": fullpath})
|
||||
|
||||
statobj = fullpath.stat()
|
||||
# Captured site icons are intentionally shareable even for private snapshots.
|
||||
# Only raw icon responses qualify, not generated previews or directory pages.
|
||||
favicon_cache_control = (
|
||||
FAVICON_CACHE_CONTROL
|
||||
if fullpath.stem.lower() == "favicon"
|
||||
and fullpath.suffix.lower() in {".ico", ".svg", ".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif"}
|
||||
and not request.GET.get("preview")
|
||||
else None
|
||||
)
|
||||
document_root = Path(document_root) if document_root else None
|
||||
rel_path = path
|
||||
etag = None
|
||||
@@ -964,7 +899,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if etag in inm_list or etag.strip('"') in [i.strip('"') for i in inm_list]:
|
||||
not_modified = HttpResponseNotModified()
|
||||
not_modified.headers["ETag"] = etag
|
||||
not_modified.headers["Cache-Control"] = favicon_cache_control or f"{cache_policy}, max-age=31536000, immutable"
|
||||
not_modified.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
not_modified.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
return _apply_archive_replay_headers(
|
||||
not_modified,
|
||||
@@ -975,16 +910,6 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
)
|
||||
|
||||
content_type, encoding = mimetypes.guess_type(str(fullpath))
|
||||
# Captured favicons retain the conventional .ico filename even when a site
|
||||
# returns SVG. Browsers cannot decode SVG served as image/x-icon.
|
||||
if fullpath.suffix.lower() == ".ico":
|
||||
with fullpath.open("rb") as icon_file:
|
||||
icon_head = icon_file.read(4096)
|
||||
if re.search(rb"<svg(?:\s|>)", icon_head):
|
||||
content_type = "image/svg+xml"
|
||||
preserve_plain_text = fullpath.suffix.lower() in {".log", ".sh", ".jsonl"}
|
||||
if preserve_plain_text:
|
||||
content_type = "text/plain"
|
||||
content_type = content_type or "application/octet-stream"
|
||||
# Add charset for text-like types (best guess), but don't override the type.
|
||||
is_text_like = content_type.startswith("text/") or content_type in {
|
||||
@@ -999,21 +924,21 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
preview_as_text_html = (
|
||||
bool(request.GET.get("preview"))
|
||||
and is_text_like
|
||||
and (bool(request.GET.get("raw")) or not content_type.startswith("text/html"))
|
||||
and not content_type.startswith("text/html")
|
||||
and not content_type.startswith("image/svg+xml")
|
||||
)
|
||||
preview_as_image_html = (
|
||||
bool(request.GET.get("preview")) and content_type.startswith("image/") and not content_type.startswith("image/svg+xml")
|
||||
)
|
||||
preview_as_archivewebpage_html = bool(request.GET.get("preview")) and archivewebpage_replay.is_replay_target(fullpath.name)
|
||||
|
||||
# Respect the If-Modified-Since header for non-markdown responses.
|
||||
if not content_type.startswith(("text/plain", "text/html")) and not static.was_modified_since(
|
||||
request.META.get("HTTP_IF_MODIFIED_SINCE"),
|
||||
statobj.st_mtime,
|
||||
):
|
||||
not_modified = HttpResponseNotModified()
|
||||
not_modified.headers["Cache-Control"] = favicon_cache_control or f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
return _apply_archive_replay_headers(
|
||||
not_modified,
|
||||
HttpResponseNotModified(),
|
||||
fullpath=fullpath,
|
||||
content_type=content_type,
|
||||
is_archive_replay=is_archive_replay,
|
||||
@@ -1028,7 +953,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
decoded = fullpath.read_text(encoding="utf-8", errors="replace")
|
||||
wrapped = _render_text_preview_document(decoded, fullpath.name)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1037,7 +962,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
config=config,
|
||||
)
|
||||
except (OSError, UnicodeDecodeError, ValueError):
|
||||
pass
|
||||
preview_as_text_html = False
|
||||
|
||||
if preview_as_image_html:
|
||||
try:
|
||||
@@ -1048,7 +973,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
raw_image_url = f"{raw_image_url}?{urlencode(list(preview_query.lists()), doseq=True)}"
|
||||
wrapped = _render_image_preview_document(raw_image_url, fullpath.name)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1057,16 +982,16 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
config=config,
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
preview_as_image_html = False
|
||||
|
||||
if request.GET.get("preview"):
|
||||
if preview_as_archivewebpage_html:
|
||||
try:
|
||||
raw_query = request.GET.copy()
|
||||
raw_query.pop("preview", None)
|
||||
raw_output_path = request.path
|
||||
if raw_query:
|
||||
raw_output_path = f"{raw_output_path}?{raw_query.urlencode()}"
|
||||
plugin_replay = render_plugin_full_response(
|
||||
body, preview_content_type, headers = archivewebpage_replay.render_preview_response(
|
||||
fullpath.name,
|
||||
raw_output_path,
|
||||
wacz_path=fullpath,
|
||||
@@ -1074,28 +999,28 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
last_modified=http_date(statobj.st_mtime),
|
||||
etag=etag or "",
|
||||
cache_control=(
|
||||
f"{cache_policy}, max-age=31536000, immutable" if etag else f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
if etag
|
||||
else f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
),
|
||||
content_encoding=encoding or "",
|
||||
)
|
||||
if plugin_replay is not None:
|
||||
body, replay_content_type, headers = plugin_replay
|
||||
response = HttpResponse(body, content_type=replay_content_type)
|
||||
for key, value in headers.items():
|
||||
response.headers[key] = value
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
content_type=replay_content_type,
|
||||
is_archive_replay=is_archive_replay,
|
||||
config=config,
|
||||
)
|
||||
response = HttpResponse(body, content_type=preview_content_type)
|
||||
for key, value in headers.items():
|
||||
response.headers[key] = value
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
content_type=preview_content_type,
|
||||
is_archive_replay=is_archive_replay,
|
||||
config=config,
|
||||
)
|
||||
except (OSError, RuntimeError, ValueError):
|
||||
pass
|
||||
preview_as_archivewebpage_html = False
|
||||
|
||||
# Heuristic fix: some archived HTML outputs are stored with HTML-escaped markup
|
||||
# or markdown sources. If so, render sensibly.
|
||||
if not request.GET.get("raw") and not preserve_plain_text and content_type.startswith(("text/plain", "text/html", "text/markdown")):
|
||||
if content_type.startswith(("text/plain", "text/html")):
|
||||
try:
|
||||
max_unescape_size = 10 * 1024 * 1024 # 10MB cap to avoid heavy memory use
|
||||
if statobj.st_size <= max_unescape_size:
|
||||
@@ -1109,12 +1034,12 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if content_type.startswith("text/html") and document_root:
|
||||
rewritten_html, rewritten_count = _rewrite_html_image_sources_for_request(request, decoded, document_root, rel_path)
|
||||
markdown_candidate = _extract_markdown_candidate(decoded)
|
||||
if content_type.startswith("text/markdown") or _looks_like_markdown(markdown_candidate):
|
||||
if _looks_like_markdown(markdown_candidate):
|
||||
wrapped = _render_markdown_document(markdown_candidate)
|
||||
wrapped, _rewrite_count = _rewrite_html_image_sources_for_request(request, wrapped, document_root, rel_path)
|
||||
wrapped = _apply_transformed_html_preview_style(wrapped)
|
||||
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1125,7 +1050,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if rewritten_count:
|
||||
rewritten_html = _apply_transformed_html_preview_style(rewritten_html)
|
||||
response = HttpResponse(rewritten_html, content_type=content_type)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1136,7 +1061,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
if escaped_count and escaped_count > tag_count * 2:
|
||||
decoded = _apply_transformed_html_preview_style(decoded)
|
||||
response = HttpResponse(decoded, content_type=content_type)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
|
||||
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
|
||||
return _apply_archive_replay_headers(
|
||||
response,
|
||||
fullpath=fullpath,
|
||||
@@ -1156,15 +1081,13 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
|
||||
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
|
||||
if etag:
|
||||
response.headers["ETag"] = etag
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
|
||||
else:
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
|
||||
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
|
||||
if is_text_like:
|
||||
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
|
||||
if content_type.startswith("image/"):
|
||||
response.headers["Cache-Control"] = f"{cache_policy}, max-age=604800, immutable"
|
||||
if favicon_cache_control:
|
||||
response.headers["Cache-Control"] = favicon_cache_control
|
||||
response.headers["Cache-Control"] = "public, max-age=604800, immutable"
|
||||
|
||||
# handle byte-range requests by serving chunk of file
|
||||
if stat.S_ISREG(statobj.st_mode):
|
||||
|
||||
@@ -2,11 +2,94 @@ from typing import Any, cast
|
||||
from collections.abc import Callable
|
||||
|
||||
import json
|
||||
import ast
|
||||
import inspect
|
||||
import toml
|
||||
import re
|
||||
import configparser
|
||||
|
||||
from pathlib import Path, PosixPath
|
||||
|
||||
from pydantic.json_schema import GenerateJsonSchema
|
||||
from pydantic_core import to_jsonable_python
|
||||
|
||||
JSONValue = str | bool | int | None | list["JSONValue"]
|
||||
|
||||
TOML_HEADER = "# Converted from INI to TOML format: https://toml.io/en/\n\n"
|
||||
|
||||
|
||||
def load_ini_value(val: str) -> JSONValue:
|
||||
"""Convert lax INI values into strict TOML-compliant (JSON) values"""
|
||||
if val.lower() in ("true", "yes", "1"):
|
||||
return True
|
||||
if val.lower() in ("false", "no", "0"):
|
||||
return False
|
||||
if val.isdigit():
|
||||
return int(val)
|
||||
|
||||
try:
|
||||
return ast.literal_eval(val)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
return json.loads(val)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return val
|
||||
|
||||
|
||||
def convert(ini_str: str) -> str:
|
||||
"""Convert a string of INI config into its TOML equivalent (warning: strips comments)"""
|
||||
|
||||
config = configparser.ConfigParser()
|
||||
setattr(config, "optionxform", str) # capitalize key names
|
||||
config.read_string(ini_str)
|
||||
|
||||
# Initialize an empty dictionary to store the TOML representation
|
||||
toml_dict = {}
|
||||
|
||||
# Iterate over each section in the INI configuration
|
||||
for section in config.sections():
|
||||
toml_dict[section] = {}
|
||||
|
||||
# Iterate over each key-value pair in the section
|
||||
for key, value in config.items(section):
|
||||
parsed_value = load_ini_value(value)
|
||||
|
||||
# Convert the parsed value to its TOML-compatible JSON representation
|
||||
toml_dict[section.upper()][key.upper()] = json.dumps(parsed_value)
|
||||
|
||||
# Build the TOML string
|
||||
toml_str = TOML_HEADER
|
||||
for section, items in toml_dict.items():
|
||||
toml_str += f"[{section}]\n"
|
||||
for key, value in items.items():
|
||||
toml_str += f"{key} = {value}\n"
|
||||
toml_str += "\n"
|
||||
|
||||
return toml_str.strip()
|
||||
|
||||
|
||||
class JSONSchemaWithLambdas(GenerateJsonSchema):
|
||||
"""
|
||||
Encode lambda functions in default values properly.
|
||||
Usage:
|
||||
>>> json.dumps(value, encoder=JSONSchemaWithLambdas())
|
||||
"""
|
||||
|
||||
def encode_default(self, dft: Any) -> Any:
|
||||
config = self._config
|
||||
if isinstance(dft, Callable):
|
||||
return "{{lambda " + inspect.getsource(dft).split("=lambda ")[-1].strip()[:-1] + "}}"
|
||||
return to_jsonable_python(
|
||||
dft,
|
||||
timedelta_mode=config.ser_json_timedelta,
|
||||
bytes_mode=config.ser_json_bytes,
|
||||
serialize_unknown=True,
|
||||
)
|
||||
|
||||
|
||||
def better_toml_dump_str(val: Any) -> str:
|
||||
try:
|
||||
|
||||
+103
-15
@@ -3,7 +3,7 @@ __package__ = "archivebox.misc"
|
||||
# Bootable utility functions (URL parsing, date parsing, JSON encoding, decorators).
|
||||
# MUST NOT import archivebox.config, archivebox.core, or Django — this module is
|
||||
# loaded by hooks.py and other early code paths. Only depends on stdlib and the
|
||||
# bootable .logging module and standalone abx_plugins URL helpers.
|
||||
# bootable .logging module.
|
||||
|
||||
import re
|
||||
import json as pyjson
|
||||
@@ -19,9 +19,6 @@ from urllib.parse import urlparse, quote, unquote
|
||||
from html import escape, unescape
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from abx_plugins.plugins.base.url_cleaning import URL_REGEX, fix_url_from_markdown
|
||||
from abx_plugins.plugins.base.url_cleaning import sanitize_extracted_url as sanitize_plugin_url
|
||||
|
||||
from .logging import COLOR_DICT
|
||||
|
||||
|
||||
@@ -93,6 +90,18 @@ def ts_to_date_str(ts: Any) -> str | None:
|
||||
COLOR_REGEX = re.compile(r"\[(?P<arg_1>\d+)(;(?P<arg_2>\d+)(;(?P<arg_3>\d+))?)?m")
|
||||
|
||||
|
||||
# https://mathiasbynens.be/demo/url-regex
|
||||
URL_REGEX = re.compile(
|
||||
r"(?=("
|
||||
r"http[s]?://" # start matching from allowed schemes
|
||||
r"(?:[a-zA-Z]|[0-9]" # followed by allowed alphanum characters
|
||||
r"|[-_$@.&+!*\(\),]" # or allowed symbols (keep hyphen first to match literal hyphen)
|
||||
r"|[^\u0000-\u007F])+" # or allowed unicode bytes
|
||||
r'[^\]\[<>"\'\s]+' # stop parsing at these symbols
|
||||
r"))",
|
||||
re.IGNORECASE | re.UNICODE,
|
||||
)
|
||||
|
||||
# Maximum supported URL length. Very long URLs are rare but must be supported correctly
|
||||
# (e.g. data: URLs, deeply nested query strings). The Snapshot.url column is stored as a
|
||||
# variable-length TextField (so short URLs don't reserve space and very long URLs still
|
||||
@@ -100,6 +109,29 @@ COLOR_REGEX = re.compile(r"\[(?P<arg_1>\d+)(;(?P<arg_2>\d+)(;(?P<arg_3>\d+))?)?m
|
||||
# all keep working.
|
||||
MAX_URL_LENGTH = 65535
|
||||
|
||||
QUOTE_DELIMITERS = (
|
||||
'"',
|
||||
"'",
|
||||
"`",
|
||||
"“",
|
||||
"”",
|
||||
"‘",
|
||||
"’",
|
||||
)
|
||||
QUOTE_ENTITY_DELIMITERS = (
|
||||
""",
|
||||
""",
|
||||
""",
|
||||
"'",
|
||||
"'",
|
||||
"'",
|
||||
)
|
||||
URL_ENTITY_REPLACEMENTS = (
|
||||
("&", "&"),
|
||||
("&", "&"),
|
||||
("&", "&"),
|
||||
)
|
||||
|
||||
FILESIZE_UNITS: dict[str, int] = {
|
||||
"": 1,
|
||||
"b": 1,
|
||||
@@ -121,9 +153,36 @@ FILESIZE_UNITS: dict[str, int] = {
|
||||
|
||||
|
||||
def sanitize_extracted_url(url: str) -> str:
|
||||
"""Use the standalone parser's quote rules, then trim dangling prose punctuation."""
|
||||
cleaned = sanitize_plugin_url(url)
|
||||
return cleaned.rstrip('.,;:!?\\"')
|
||||
"""Trim quote garbage and dangling prose punctuation from an extracted URL candidate."""
|
||||
cleaned = (url or "").strip()
|
||||
if not cleaned:
|
||||
return cleaned
|
||||
|
||||
lower_cleaned = cleaned.lower()
|
||||
cut_index = len(cleaned)
|
||||
|
||||
for delimiter in QUOTE_DELIMITERS:
|
||||
found_index = cleaned.find(delimiter)
|
||||
if found_index != -1:
|
||||
cut_index = min(cut_index, found_index)
|
||||
|
||||
for delimiter in QUOTE_ENTITY_DELIMITERS:
|
||||
found_index = lower_cleaned.find(delimiter)
|
||||
if found_index != -1:
|
||||
cut_index = min(cut_index, found_index)
|
||||
|
||||
cleaned = cleaned[:cut_index].strip()
|
||||
lower_cleaned = cleaned.lower()
|
||||
for entity, replacement in URL_ENTITY_REPLACEMENTS:
|
||||
while entity in lower_cleaned:
|
||||
entity_index = lower_cleaned.find(entity)
|
||||
cleaned = cleaned[:entity_index] + replacement + cleaned[entity_index + len(entity) :]
|
||||
lower_cleaned = cleaned.lower()
|
||||
|
||||
cleaned = cleaned.rstrip(".,;:!?\\'\"")
|
||||
cleaned = cleaned.rstrip('"')
|
||||
|
||||
return cleaned
|
||||
|
||||
|
||||
def validate_url_length(url: str) -> str:
|
||||
@@ -156,6 +215,39 @@ def parens_are_matched(string: str, open_char="(", close_char=")"):
|
||||
return count == 0
|
||||
|
||||
|
||||
def fix_url_from_markdown(url_str: str) -> str:
|
||||
"""
|
||||
cleanup a regex-parsed url that may contain dangling trailing parens from markdown link syntax
|
||||
helpful to fix URLs parsed from markdown e.g.
|
||||
input: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def).somemoretext
|
||||
result: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def
|
||||
|
||||
IMPORTANT ASSUMPTION: valid urls wont have unbalanced or incorrectly nested parentheses
|
||||
e.g. this will fail the user actually wants to ingest a url like 'https://example.com/some_wei)(rd_url'
|
||||
in that case it will return https://example.com/some_wei (truncated up to the first unbalanced paren)
|
||||
This assumption is true 99.9999% of the time, and for the rare edge case the user can use url_list parser.
|
||||
"""
|
||||
trimmed_url = url_str
|
||||
if len(trimmed_url) > 2048:
|
||||
return trimmed_url
|
||||
|
||||
# cut off one trailing character at a time
|
||||
# until parens are balanced e.g. /a(b)c).x(y)z -> /a(b)c
|
||||
trim_attempts = 0
|
||||
while trimmed_url and not parens_are_matched(trimmed_url) and trim_attempts < 256:
|
||||
trimmed_url = trimmed_url[:-1]
|
||||
trim_attempts += 1
|
||||
|
||||
if not trimmed_url or not parens_are_matched(trimmed_url):
|
||||
return url_str
|
||||
|
||||
# make sure trimmed url is still valid
|
||||
if any(match == trimmed_url for match in re.findall(URL_REGEX, trimmed_url)):
|
||||
return trimmed_url
|
||||
|
||||
return url_str
|
||||
|
||||
|
||||
def split_comma_separated_urls(url: str):
|
||||
offset = 0
|
||||
while True:
|
||||
@@ -182,8 +274,7 @@ def find_all_urls(urls_str: str):
|
||||
if match.start() in skipped_starts:
|
||||
continue
|
||||
|
||||
cleaned_match = fix_url_from_markdown(match.group(1), preceding_text=urls_str[max(0, match.start(1) - 6) : match.start(1)])
|
||||
cleaned_match = sanitize_extracted_url(cleaned_match)
|
||||
cleaned_match = sanitize_extracted_url(fix_url_from_markdown(match.group(1)))
|
||||
for offset, url in split_comma_separated_urls(cleaned_match):
|
||||
if offset:
|
||||
skipped_starts.add(match.start() + offset)
|
||||
@@ -297,12 +388,9 @@ def parse_date(date: Any) -> datetime | None:
|
||||
if date.tzinfo is None:
|
||||
return date.replace(tzinfo=timezone.utc)
|
||||
|
||||
# Legacy SQLite ArchiveResult rows can retain explicit timezone offsets
|
||||
# through migration. Django loads those as aware datetimes, whereas JSON
|
||||
# imports reach the string branch below. Normalize both representations
|
||||
# consistently so opening a migrated snapshot does not raise (#1889).
|
||||
# Convert the instant; replacing tzinfo here would shift the capture time.
|
||||
return date.astimezone(timezone.utc)
|
||||
offset = date.utcoffset()
|
||||
assert offset == datetime.now(timezone.utc).utcoffset(), "Refusing to load a non-UTC date!"
|
||||
return date
|
||||
|
||||
if isinstance(date, (float, int)):
|
||||
date = str(date)
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
from django.apps import AppConfig
|
||||
|
||||
|
||||
class OpencodeConfig(AppConfig):
|
||||
name = "archivebox.opencode"
|
||||
label = "opencode_plugin"
|
||||
@@ -0,0 +1,283 @@
|
||||
{% extends "admin/base.html" %}
|
||||
{% load i18n %}
|
||||
|
||||
{% block title %}Agent{% endblock %}
|
||||
|
||||
{% block breadcrumbs %}
|
||||
<div class="breadcrumbs">
|
||||
<a href="{% url 'admin:index' %}">{% trans 'Home' %}</a>
|
||||
› Agent
|
||||
</div>
|
||||
{% endblock %}
|
||||
|
||||
{% block extrastyle %}
|
||||
{{ block.super }}
|
||||
<style>
|
||||
body.opencode-agent #content {
|
||||
padding: 0;
|
||||
}
|
||||
body.opencode-agent #content-main {
|
||||
height: calc(100vh - 150px);
|
||||
min-height: 560px;
|
||||
}
|
||||
.opencode-agent-error {
|
||||
margin: 24px;
|
||||
color: #0f172a;
|
||||
font: 14px -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}
|
||||
.opencode-agent-error code {
|
||||
color: #0369a1;
|
||||
}
|
||||
.opencode-agent-frame {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
border: 0;
|
||||
display: block;
|
||||
}
|
||||
.opencode-agent-shell {
|
||||
position: relative;
|
||||
height: 100%;
|
||||
}
|
||||
.opencode-agent-welcome {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
z-index: 20;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
padding: 24px;
|
||||
background: rgba(0, 0, 0, 0.48);
|
||||
color: #111827;
|
||||
}
|
||||
.opencode-agent-welcome[hidden] {
|
||||
display: none;
|
||||
}
|
||||
.opencode-agent-welcome-panel {
|
||||
width: min(720px, 100%);
|
||||
max-height: min(720px, calc(100vh - 190px));
|
||||
overflow: auto;
|
||||
border-radius: 8px;
|
||||
border: 1px solid #d1d5db;
|
||||
background: Canvas;
|
||||
color: CanvasText;
|
||||
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.32);
|
||||
padding: 24px;
|
||||
font: 14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}
|
||||
.opencode-agent-welcome-panel h1 {
|
||||
margin: 0 0 10px;
|
||||
font-size: 22px;
|
||||
line-height: 1.25;
|
||||
}
|
||||
.opencode-agent-welcome-panel p {
|
||||
margin: 0 0 14px;
|
||||
}
|
||||
.opencode-agent-welcome-panel ul {
|
||||
margin: 0 0 16px 20px;
|
||||
padding: 0;
|
||||
}
|
||||
.opencode-agent-welcome-panel li {
|
||||
margin: 0 0 8px;
|
||||
}
|
||||
.opencode-agent-welcome-warning {
|
||||
margin: 16px 0;
|
||||
padding: 12px 14px;
|
||||
border: 1px solid #f59e0b;
|
||||
border-radius: 6px;
|
||||
background: color-mix(in srgb, #f59e0b 12%, Canvas);
|
||||
}
|
||||
.opencode-agent-welcome-actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 8px;
|
||||
}
|
||||
.opencode-agent-welcome button {
|
||||
cursor: pointer;
|
||||
border: 1px solid #2563eb;
|
||||
border-radius: 6px;
|
||||
background: #2563eb;
|
||||
color: #fff;
|
||||
padding: 8px 14px;
|
||||
font-weight: 600;
|
||||
}
|
||||
@media (max-width: 700px) {
|
||||
.opencode-agent-welcome {
|
||||
padding: 12px;
|
||||
}
|
||||
.opencode-agent-welcome-panel {
|
||||
max-height: calc(100vh - 170px);
|
||||
padding: 18px;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
{% endblock %}
|
||||
|
||||
{% block bodyclass %}{{ block.super }} opencode-agent{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div id="content-main">
|
||||
{% if error %}
|
||||
<div class="opencode-agent-error">
|
||||
{{ error }}
|
||||
{% if command %}<br><br><code>{{ command }}</code>{% endif %}
|
||||
</div>
|
||||
{% else %}
|
||||
<script>
|
||||
(() => {
|
||||
const workdir = "{{ workdir|escapejs }}";
|
||||
const recentSessionId = "{{ recent_session_id|escapejs }}";
|
||||
const merge = (left, right) => {
|
||||
if (!left || typeof left !== "object" || Array.isArray(left)) return right;
|
||||
const out = {...left};
|
||||
for (const [key, value] of Object.entries(right)) {
|
||||
out[key] = value && typeof value === "object" && !Array.isArray(value) ? merge(out[key], value) : value;
|
||||
}
|
||||
return out;
|
||||
};
|
||||
const seed = (key, value) => {
|
||||
try {
|
||||
localStorage.setItem(key, JSON.stringify(merge(JSON.parse(localStorage.getItem(key) || "{}"), value)));
|
||||
} catch {
|
||||
localStorage.setItem(key, JSON.stringify(value));
|
||||
}
|
||||
};
|
||||
const setJSON = (key, value) => localStorage.setItem(key, JSON.stringify(value));
|
||||
const page = {
|
||||
activeProject: workdir,
|
||||
activeWorkspace: workdir,
|
||||
gettingStartedDismissed: true,
|
||||
workspaceExpanded: {[workdir]: true},
|
||||
};
|
||||
if (recentSessionId) {
|
||||
page.lastProjectSession = {[workdir]: {directory: workdir, id: recentSessionId, at: Date.now()}};
|
||||
}
|
||||
seed("opencode.global.dat:layout.page", page);
|
||||
seed("opencode.global.dat:layout", {
|
||||
sidebar: {opened: true, width: 280},
|
||||
fileTree: {opened: true, width: 280, tab: "all"},
|
||||
});
|
||||
const serverKey = `${location.origin}/admin/agent/opencode`;
|
||||
setJSON("opencode.global.dat:server", {
|
||||
list: [],
|
||||
projects: {
|
||||
[serverKey]: [{worktree: workdir, expanded: true}],
|
||||
},
|
||||
lastProject: {
|
||||
[serverKey]: workdir,
|
||||
},
|
||||
});
|
||||
localStorage.setItem("opencode.settings.dat:defaultServerUrl", serverKey);
|
||||
})();
|
||||
</script>
|
||||
<div class="opencode-agent-shell">
|
||||
<iframe src="{{ proxy_url }}" title="OpenCode Agent" class="opencode-agent-frame"></iframe>
|
||||
<div class="opencode-agent-welcome" id="opencode-agent-welcome" hidden>
|
||||
<div class="opencode-agent-welcome-panel" role="dialog" aria-modal="true" aria-labelledby="opencode-agent-welcome-title">
|
||||
<h1 id="opencode-agent-welcome-title">ArchiveBox AI Agent</h1>
|
||||
<p>This agent can work directly with your ArchiveBox collection. It can inspect the crawl database, create and monitor crawls, run maintenance operations, answer research questions, query archived data, and help with complex collection workflows.</p>
|
||||
<p>Example prompts:</p>
|
||||
<ul>
|
||||
<li>Archive this list of URLs with depth 0, then report which ones failed and why.</li>
|
||||
<li>Find snapshots from the last month that failed PDF or screenshot extraction and retry them.</li>
|
||||
<li>Create a constrained crawl for this site, avoid login/privacy/sitemap URLs, and watch the logs as it runs.</li>
|
||||
<li>Summarize what my collection contains about this topic and link to the best saved pages.</li>
|
||||
</ul>
|
||||
<div class="opencode-agent-welcome-warning">
|
||||
The agent has unrestricted access to this collection and can make destructive edits. Review requests carefully. Permission settings can be changed from the OpenCode gear icon in the lower left.
|
||||
</div>
|
||||
<div class="opencode-agent-welcome-actions">
|
||||
<button type="button" id="opencode-agent-welcome-dismiss">Start using Agent</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
(() => {
|
||||
const key = "archivebox.opencode.agentWelcomeDismissed.v1";
|
||||
const welcome = document.getElementById("opencode-agent-welcome");
|
||||
const dismiss = document.getElementById("opencode-agent-welcome-dismiss");
|
||||
if (!welcome || !dismiss) return;
|
||||
if (localStorage.getItem(key) !== "1") {
|
||||
welcome.hidden = false;
|
||||
}
|
||||
dismiss.addEventListener("click", () => {
|
||||
localStorage.setItem(key, "1");
|
||||
welcome.hidden = true;
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
<script>
|
||||
(() => {
|
||||
const frame = document.querySelector(".opencode-agent-frame");
|
||||
if (!frame) return;
|
||||
|
||||
const healthUrl = "{{ proxy_prefix|escapejs }}/global/health";
|
||||
const monitorUrl = "{{ proxy_prefix|escapejs }}/_archivebox/health";
|
||||
const fetchOptions = {
|
||||
cache: "no-store",
|
||||
credentials: "same-origin",
|
||||
redirect: "manual",
|
||||
};
|
||||
let checking = false;
|
||||
let waking = false;
|
||||
let wasUnavailable = false;
|
||||
let healthyChecks = 0;
|
||||
let knownVersion = "{{ opencode_version|escapejs }}";
|
||||
const wake = () => {
|
||||
if (waking) return;
|
||||
waking = true;
|
||||
fetch(healthUrl, fetchOptions)
|
||||
.catch(() => {})
|
||||
.finally(() => { waking = false; });
|
||||
};
|
||||
const check = async () => {
|
||||
if (checking || (document.hidden && wasUnavailable)) return;
|
||||
checking = true;
|
||||
try {
|
||||
const response = await fetch(monitorUrl, fetchOptions);
|
||||
if (!response.ok || response.redirected) {
|
||||
wasUnavailable = true;
|
||||
healthyChecks = 0;
|
||||
if (response.status === 503) {
|
||||
wake();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const health = await response.json();
|
||||
if (!health || health.healthy !== true) {
|
||||
wasUnavailable = true;
|
||||
healthyChecks = 0;
|
||||
return;
|
||||
}
|
||||
const currentVersion = typeof health.version === "string" ? health.version : "";
|
||||
if (knownVersion && currentVersion && currentVersion !== knownVersion) {
|
||||
wasUnavailable = true;
|
||||
}
|
||||
if (currentVersion) {
|
||||
knownVersion = currentVersion;
|
||||
}
|
||||
if (wasUnavailable && ++healthyChecks >= 2) {
|
||||
wasUnavailable = false;
|
||||
healthyChecks = 0;
|
||||
frame.contentWindow.location.reload();
|
||||
}
|
||||
} catch {
|
||||
wasUnavailable = true;
|
||||
healthyChecks = 0;
|
||||
} finally {
|
||||
checking = false;
|
||||
}
|
||||
};
|
||||
|
||||
window.setInterval(check, 3000);
|
||||
window.addEventListener("online", check);
|
||||
document.addEventListener("visibilitychange", () => {
|
||||
if (!document.hidden) check();
|
||||
});
|
||||
check();
|
||||
})();
|
||||
</script>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -1,10 +1,11 @@
|
||||
from django.urls import path, re_path
|
||||
|
||||
from archivebox.opencode.views import agent_view, opencode_proxy_view
|
||||
from archivebox.opencode.views import agent_health_view, agent_view, opencode_proxy_view
|
||||
|
||||
|
||||
urlpatterns = [
|
||||
path("", agent_view, name="opencode-agent"),
|
||||
path("opencode/_archivebox/health", agent_health_view, name="opencode-health"),
|
||||
re_path(
|
||||
r"^opencode(?:/(?P<path>.*))?$",
|
||||
opencode_proxy_view,
|
||||
|
||||
+769
-116
@@ -1,140 +1,793 @@
|
||||
"""Django adapter for the optional plugin; never import its runtime at startup."""
|
||||
from __future__ import annotations
|
||||
|
||||
import atexit
|
||||
import base64
|
||||
import logging
|
||||
from io import BytesIO
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
|
||||
import httpx
|
||||
import requests
|
||||
from abx_plugins.plugins import opencode as opencode_plugin
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.config.common import get_config
|
||||
from archivebox.core.routes_util import build_admin_url, get_api_base_url, get_base_url
|
||||
from asgiref.sync import sync_to_async
|
||||
|
||||
from django.contrib.auth.middleware import AuthenticationMiddleware
|
||||
from django.contrib.auth.views import redirect_to_login
|
||||
from django.contrib.sessions.middleware import SessionMiddleware
|
||||
from django.core.exceptions import PermissionDenied
|
||||
from django.core.handlers.asgi import ASGIRequest
|
||||
from django.db import close_old_connections
|
||||
from django.http import Http404, HttpResponse, HttpResponseForbidden, StreamingHttpResponse
|
||||
from django.template import engines
|
||||
from django.urls import Resolver404, resolve
|
||||
from django.http import (
|
||||
Http404,
|
||||
HttpRequest,
|
||||
HttpResponse,
|
||||
HttpResponseForbidden,
|
||||
JsonResponse,
|
||||
StreamingHttpResponse,
|
||||
)
|
||||
from django.shortcuts import redirect, render
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
|
||||
from archivebox.config import CONSTANTS
|
||||
from archivebox.config.common import get_request_config
|
||||
from archivebox.core.middleware import ReverseProxyAuthMiddleware
|
||||
from archivebox.core.routes_util import build_admin_url, get_admin_host, get_api_base_url, get_base_url, host_matches
|
||||
from archivebox.plugins.discovery import get_plugin_template
|
||||
|
||||
_PROCESS: subprocess.Popen | None = None
|
||||
_PROCESS_READY: subprocess.Popen | None = None
|
||||
_PROCESS_LOCK = threading.Lock()
|
||||
_SESSION_LOCK = threading.Lock()
|
||||
_LOGGER = logging.getLogger(__name__)
|
||||
_PROXY_PREFIX = "/admin/agent/opencode"
|
||||
_PROXY_PREFIX_REGEX = _PROXY_PREFIX.replace("/", r"\/")
|
||||
_PROXY_PREFIX_NO_SLASH_REGEX = _PROXY_PREFIX.lstrip("/").replace("/", r"\/")
|
||||
_CONFIG_PATH = Path(opencode_plugin.__file__).with_name("config.json")
|
||||
|
||||
_TEXT_CONTENT_TYPES = (
|
||||
"text/",
|
||||
"application/javascript",
|
||||
"application/x-javascript",
|
||||
)
|
||||
_HOP_BY_HOP_HEADERS = {
|
||||
"connection",
|
||||
"keep-alive",
|
||||
"proxy-authenticate",
|
||||
"proxy-authorization",
|
||||
"te",
|
||||
"trailers",
|
||||
"transfer-encoding",
|
||||
"upgrade",
|
||||
}
|
||||
_ARCHIVEBOX_SKILL = """---
|
||||
name: archivebox
|
||||
description: Use ArchiveBox's CLI and local REST API from an ArchiveBox collection.
|
||||
---
|
||||
|
||||
You are running inside an ArchiveBox collection directory.
|
||||
|
||||
- ArchiveBox collection directory: {archivebox_data_dir}
|
||||
- ArchiveBox BASE_URL: {archivebox_base_url}
|
||||
- ArchiveBox Admin URL: {archivebox_admin_url}
|
||||
- ArchiveBox REST API URL: {archivebox_api_url}
|
||||
- Prefer the `archivebox` CLI for authenticated changes, e.g. `archivebox add`, `archivebox schedule`, `archivebox update`, and `archivebox shell`.
|
||||
- Run ArchiveBox CLI commands from the ArchiveBox collection directory above.
|
||||
- Get command help with `archivebox list --help`, `archivebox add --help`, `archivebox schedule --help`, etc. Do not use `archivebox help <command>`.
|
||||
- Use `--depth=0` by default. Only use recursive crawling when the user explicitly asks for it; use `--depth=1` when you need pages one hop out.
|
||||
- Before any recursive crawl, constrain scope with ArchiveBox config such as `CRAWL_MAX_URLS`, `CRAWL_MAX_SIZE`, `SNAPSHOT_MAX_*`, `URL_ALLOWLIST`, `URL_DENYLIST`, and related limits.
|
||||
- Respect the configured `archivebox config --get ONLY_NEW` behavior unless the user explicitly says otherwise. Remind users that expected crawl URLs can be skipped when the collection already contains snapshots with the same URL.
|
||||
- Always audit newly discovered crawl URLs before letting a crawl run broadly. Treat junk URLs such as privacy policies, legal pages, tag archives, sitemap files, feeds, login/logout URLs, and other low-value boilerplate as unwanted unless the user explicitly asked to archive them.
|
||||
- Always watch crawl output and logs as the crawl progresses, and correct errors early instead of waiting until the crawl finishes.
|
||||
- If a crawl contains bad URLs, pause it, edit the crawl's `urls` field to remove them, delete any unneeded snapshots already created under that crawl, then resume the crawl.
|
||||
- Use `archivebox shell -c '...'` or `archivebox shell <<'PY' ... PY` for Django ORM work. Shell Plus prints an import banner first; keep stderr visible while debugging.
|
||||
- Use full ArchiveBox module paths in shell code: `from archivebox.crawls.models import Crawl, CrawlSchedule` and `from archivebox.core.models import Snapshot, ArchiveResult`.
|
||||
- If a model/field/relation is unclear, inspect `_meta.fields` before guessing, e.g. `archivebox shell -c "from archivebox.crawls.models import Crawl; print([f.name for f in Crawl._meta.fields])"`.
|
||||
- Use `archivebox config --get BASE_URL` only to verify the configured base URL; prefer the seeded URLs above for API/admin requests.
|
||||
- Use `$ARCHIVEBOX_API_URL` for REST API inspection when helpful. Do not assume admin session cookies authenticate API subdomain requests; prefer CLI/shell for authenticated mutations unless the admin provides or asks you to create an API token.
|
||||
- Discover REST endpoints from `${{ARCHIVEBOX_API_URL}}v1/openapi.json`; crawl endpoints live under `/api/v1/crawls/`, snapshots under `/api/v1/core/`.
|
||||
- Do not bypass ArchiveBox auth, expose API keys, or modify config unless the admin explicitly asks.
|
||||
- After creating crawls or snapshots, report the crawl/snapshot IDs and the exact command or API request used.
|
||||
"""
|
||||
|
||||
|
||||
def _runtime_settings(request, config):
|
||||
from abx_plugins.plugins.opencode import runtime
|
||||
|
||||
settings = runtime._settings(config, CONSTANTS.DATA_DIR)
|
||||
route_config = request.__dict__.get("archivebox_config")
|
||||
settings.update(
|
||||
archivebox_base_url=get_base_url(request=request, config=route_config).rstrip("/"),
|
||||
archivebox_admin_url=build_admin_url("/admin/", request=request, config=route_config).rstrip("/"),
|
||||
archivebox_api_url=f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/",
|
||||
)
|
||||
return runtime, settings
|
||||
|
||||
|
||||
def _dispatch(request, path=None):
|
||||
def _signal_owned_process(process: subprocess.Popen, sig: signal.Signals) -> None:
|
||||
try:
|
||||
# Middleware already merged server defaults and live Machine overrides.
|
||||
# Re-resolving every extractor's config here stalls the UI request burst.
|
||||
config = get_request_config(request).model_dump(mode="json")
|
||||
if not config.get("OPENCODE_ENABLED", False):
|
||||
raise Http404
|
||||
if not request.user.is_authenticated:
|
||||
return redirect_to_login(request.get_full_path(), login_url="/admin/login/")
|
||||
if not request.user.is_active or not request.user.is_superuser:
|
||||
return HttpResponseForbidden("Agent access requires a superuser account.")
|
||||
os.killpg(process.pid, sig)
|
||||
except OSError:
|
||||
try:
|
||||
process.send_signal(sig)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
|
||||
runtime, settings = _runtime_settings(request, config)
|
||||
if path is None:
|
||||
from archivebox.core.admin_site import archivebox_admin
|
||||
|
||||
context = {**archivebox_admin.each_context(request), **runtime.agent_context(settings)}
|
||||
source = get_plugin_template("opencode", "agent", fallback=False)
|
||||
if source is None:
|
||||
raise RuntimeError("Agent template unavailable")
|
||||
return HttpResponse(engines["django"].from_string(source).render(context, request))
|
||||
def _stop_owned_process(process: subprocess.Popen | None = None) -> None:
|
||||
global _PROCESS, _PROCESS_READY
|
||||
owned_process = process or _PROCESS
|
||||
if owned_process is None:
|
||||
return
|
||||
if owned_process.poll() is None:
|
||||
_signal_owned_process(owned_process, signal.SIGCONT)
|
||||
_signal_owned_process(owned_process, signal.SIGTERM)
|
||||
try:
|
||||
owned_process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
_signal_owned_process(owned_process, signal.SIGKILL)
|
||||
owned_process.wait()
|
||||
if _PROCESS is owned_process:
|
||||
_PROCESS = None
|
||||
if _PROCESS_READY is owned_process:
|
||||
_PROCESS_READY = None
|
||||
|
||||
if not runtime._origin_allowed(request.method, request.get_host(), request.headers):
|
||||
return HttpResponseForbidden("Cross-origin agent requests are blocked.")
|
||||
status, headers, body = runtime.proxy(
|
||||
settings,
|
||||
request.method,
|
||||
path,
|
||||
tuple((key, value) for key, values in request.GET.lists() for value in values),
|
||||
request.headers,
|
||||
request.body,
|
||||
|
||||
atexit.register(_stop_owned_process)
|
||||
|
||||
|
||||
def _machine_config() -> dict[str, Any]:
|
||||
resolved = get_config()
|
||||
return dict(resolved.model_dump(mode="json"))
|
||||
|
||||
|
||||
def _archivebox_data_dir_default() -> Path:
|
||||
return Path(CONSTANTS.DATA_DIR)
|
||||
|
||||
|
||||
def _archivebox_route_urls(request: HttpRequest, route_config) -> tuple[str, str, str]:
|
||||
base_url = get_base_url(
|
||||
request=request,
|
||||
config=route_config,
|
||||
).rstrip("/")
|
||||
admin_url = build_admin_url(
|
||||
"/admin/",
|
||||
request=request,
|
||||
config=route_config,
|
||||
).rstrip("/")
|
||||
api_url = f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/"
|
||||
return base_url, admin_url, api_url
|
||||
|
||||
|
||||
def _config_value(config: dict, key: str, default):
|
||||
value = config.get(key, default)
|
||||
if value in (None, ""):
|
||||
return default
|
||||
return value
|
||||
|
||||
|
||||
def _opencode_enabled(config: dict) -> bool:
|
||||
value = config.get("OPENCODE_ENABLED", False)
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
return str(value).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _require_enabled(config: dict) -> None:
|
||||
if not _opencode_enabled(config):
|
||||
raise Http404
|
||||
|
||||
|
||||
def _require_superuser(request: HttpRequest):
|
||||
user = getattr(request, "user", None)
|
||||
if user is None:
|
||||
return redirect(f"/admin/login/?next={request.get_full_path()}")
|
||||
if (
|
||||
bool(getattr(user, "is_authenticated", False))
|
||||
and bool(getattr(user, "is_active", False))
|
||||
and bool(getattr(user, "is_superuser", False))
|
||||
):
|
||||
return None
|
||||
if bool(getattr(user, "is_authenticated", False)):
|
||||
return HttpResponseForbidden(
|
||||
b"ArchiveBox agent access requires a superuser account.",
|
||||
)
|
||||
response_type = HttpResponse if isinstance(body, bytes) else StreamingHttpResponse
|
||||
response = response_type(body, status=status, headers=headers)
|
||||
response.xframe_options_exempt = True
|
||||
response.headers["X-Frame-Options"] = "SAMEORIGIN"
|
||||
response.headers["Content-Security-Policy"] = "frame-ancestors 'self'"
|
||||
return response
|
||||
except Http404:
|
||||
raise
|
||||
except Exception:
|
||||
# Optional-service boundary, including imports and template rendering.
|
||||
_LOGGER.exception("Optional AI service failed")
|
||||
return HttpResponse("AI service unavailable. See server logs.", status=503, content_type="text/plain")
|
||||
return redirect(f"/admin/login/?next={request.get_full_path()}")
|
||||
|
||||
|
||||
def agent_view(request):
|
||||
return _dispatch(request)
|
||||
def _origin_allowed(request: HttpRequest, path: str | None = None) -> bool:
|
||||
if request.method in {"GET", "HEAD", "OPTIONS", "TRACE"}:
|
||||
return True
|
||||
|
||||
expected_host = request.get_host()
|
||||
pty_connect = bool(
|
||||
path and path.startswith("pty/") and path.endswith("/connect-token"),
|
||||
)
|
||||
if pty_connect:
|
||||
return True
|
||||
|
||||
origin = _request_header(request, "Origin")
|
||||
if origin:
|
||||
return _same_host(origin, expected_host)
|
||||
|
||||
referer = _request_header(request, "Referer")
|
||||
if referer:
|
||||
return _same_host(referer, expected_host)
|
||||
|
||||
fetch_site = _request_header(request, "Sec-Fetch-Site")
|
||||
if fetch_site:
|
||||
return fetch_site in {"same-origin", "same-site", "none"}
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _same_host(value: str, expected_host: str) -> bool:
|
||||
parsed = urlsplit(value)
|
||||
return parsed.scheme in {"http", "https"} and parsed.netloc == expected_host
|
||||
|
||||
|
||||
def _settings(config: dict) -> dict:
|
||||
host = str(_config_value(config, "OPENCODE_HOST", "127.0.0.1"))
|
||||
port = int(_config_value(config, "OPENCODE_PORT", 4096))
|
||||
default_data_dir = _archivebox_data_dir_default()
|
||||
opencode_dir = Path(
|
||||
str(_config_value(config, "OPENCODE_STATE_DIR", default_data_dir / "opencode")),
|
||||
).expanduser()
|
||||
workdir = Path(
|
||||
str(_config_value(config, "OPENCODE_WORKDIR", opencode_dir / "workdir")),
|
||||
).expanduser()
|
||||
binary = str(_config_value(config, "OPENCODE_BINARY", "opencode"))
|
||||
timeout = int(_config_value(config, "OPENCODE_TIMEOUT", 120))
|
||||
return {
|
||||
"host": host,
|
||||
"port": port,
|
||||
"origin": f"http://{host}:{port}",
|
||||
"archivebox_data_dir": default_data_dir,
|
||||
"workdir": workdir,
|
||||
"opencode_dir": opencode_dir,
|
||||
"config_home": opencode_dir / "config",
|
||||
"data_home": opencode_dir / "data",
|
||||
"state_home": opencode_dir / "state",
|
||||
"cache_home": opencode_dir / "cache",
|
||||
"home": opencode_dir / "home",
|
||||
"binary": binary,
|
||||
"config": config,
|
||||
"timeout": timeout,
|
||||
}
|
||||
|
||||
|
||||
def _resolve_binary(binary: str, config: dict) -> tuple[Any, Any, dict[str, str]]:
|
||||
try:
|
||||
from abxpkg import BinProvider
|
||||
from abx_plugins.plugins.base.utils import load_required_binary_from_config
|
||||
|
||||
binary_environ = os.environ.copy()
|
||||
lib_dir = config.get("ABXPKG_LIB_DIR")
|
||||
if lib_dir:
|
||||
binary_environ["ABXPKG_LIB_DIR"] = str(lib_dir)
|
||||
loaded_dependencies = [
|
||||
load_required_binary_from_config(
|
||||
required_binary,
|
||||
_CONFIG_PATH,
|
||||
global_config=config,
|
||||
environ=binary_environ,
|
||||
install=False,
|
||||
)
|
||||
for required_binary in (
|
||||
str(config.get("NODE_BINARY") or "node"),
|
||||
str(config.get("GIT_BINARY") or "git"),
|
||||
binary,
|
||||
)
|
||||
]
|
||||
except Exception as err:
|
||||
raise RuntimeError(
|
||||
f"OpenCode dependency is not installed from required_binaries: {err}",
|
||||
) from err
|
||||
|
||||
if any(not loaded.loaded_abspath for loaded in loaded_dependencies):
|
||||
raise RuntimeError(
|
||||
"OpenCode dependency is not installed from required_binaries.",
|
||||
)
|
||||
|
||||
providers = [loaded.loaded_binprovider for loaded in loaded_dependencies if loaded.loaded_binprovider is not None]
|
||||
binary_env = BinProvider.build_exec_env(
|
||||
providers=providers,
|
||||
base_env=binary_environ,
|
||||
)
|
||||
return (
|
||||
loaded_dependencies[-1],
|
||||
loaded_dependencies[1],
|
||||
binary_env,
|
||||
)
|
||||
|
||||
|
||||
def _project_route(workdir: Path, session_id: str = "") -> str:
|
||||
encoded = base64.b64encode(str(workdir.resolve()).encode()).decode()
|
||||
encoded = encoded.replace("+", "-").replace("/", "_").rstrip("=")
|
||||
route = f"{_PROXY_PREFIX}/{encoded}/session"
|
||||
return f"{route}/{session_id}" if session_id else route
|
||||
|
||||
|
||||
def _ensure_project_files(settings: dict) -> None:
|
||||
workdir = settings["workdir"].resolve()
|
||||
workdir.mkdir(parents=True, exist_ok=True)
|
||||
git_marker = workdir / ".git" / "not-a-git"
|
||||
if git_marker.exists():
|
||||
# Current OpenCode hangs on the legacy fake marker, so remove only
|
||||
# that invalid shape before initializing the real worktree.
|
||||
shutil.rmtree(git_marker.parent)
|
||||
|
||||
editable_skill_path = settings["opencode_dir"] / "SKILL.md"
|
||||
editable_skill_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not editable_skill_path.exists():
|
||||
editable_skill_path.write_text(
|
||||
_ARCHIVEBOX_SKILL.format(
|
||||
archivebox_data_dir=settings["archivebox_data_dir"],
|
||||
archivebox_base_url=settings.get("archivebox_base_url", ""),
|
||||
archivebox_admin_url=settings.get("archivebox_admin_url", ""),
|
||||
archivebox_api_url=settings.get("archivebox_api_url", ""),
|
||||
),
|
||||
)
|
||||
|
||||
opencode_skill_path = settings["config_home"] / "opencode" / "skills" / "archivebox" / "SKILL.md"
|
||||
opencode_skill_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if opencode_skill_path.resolve() != editable_skill_path.resolve():
|
||||
if opencode_skill_path.exists() or opencode_skill_path.is_symlink():
|
||||
opencode_skill_path.unlink()
|
||||
opencode_skill_path.symlink_to(editable_skill_path)
|
||||
|
||||
|
||||
def _ensure_default_session(settings: dict) -> str:
|
||||
workdir = settings["workdir"].resolve()
|
||||
params = {"directory": str(workdir)}
|
||||
timeout = settings["timeout"]
|
||||
project = requests.post(
|
||||
f"{settings['origin']}/project/git/init",
|
||||
params=params,
|
||||
timeout=timeout,
|
||||
)
|
||||
project.raise_for_status()
|
||||
project_data = project.json()
|
||||
if Path(str(project_data.get("worktree") or "/")).resolve() != workdir:
|
||||
raise RuntimeError(
|
||||
f"OpenCode initialized the wrong project worktree: {project_data.get('worktree')!r}",
|
||||
)
|
||||
|
||||
sessions = requests.get(
|
||||
f"{settings['origin']}/session",
|
||||
params={**params, "roots": "true", "limit": 55},
|
||||
timeout=timeout,
|
||||
)
|
||||
sessions.raise_for_status()
|
||||
session_data = sessions.json()
|
||||
if not isinstance(session_data, list):
|
||||
raise RuntimeError("OpenCode returned an invalid project session list.")
|
||||
for session_data_item in session_data:
|
||||
if not isinstance(session_data_item, dict):
|
||||
continue
|
||||
session_id = str(session_data_item.get("id") or "")
|
||||
session_directory = session_data_item.get("directory")
|
||||
if session_id and session_directory and Path(str(session_directory)).resolve() == workdir:
|
||||
return session_id
|
||||
|
||||
session = requests.post(
|
||||
f"{settings['origin']}/session",
|
||||
params=params,
|
||||
json={},
|
||||
timeout=timeout,
|
||||
)
|
||||
session.raise_for_status()
|
||||
session_data = session.json()
|
||||
session_id = str(session_data.get("id") or "")
|
||||
session_directory = session_data.get("directory")
|
||||
if not session_id or not session_directory or Path(str(session_directory)).resolve() != workdir:
|
||||
raise RuntimeError(
|
||||
"OpenCode did not create a session for the requested worktree.",
|
||||
)
|
||||
return session_id
|
||||
|
||||
|
||||
def _owned_process_running() -> bool:
|
||||
process = _PROCESS
|
||||
return process is not None and process.poll() is None
|
||||
|
||||
|
||||
def _owned_process_ready() -> bool:
|
||||
process = _PROCESS_READY
|
||||
return process is not None and process is _PROCESS and process.poll() is None
|
||||
|
||||
|
||||
def _health(settings: dict) -> bool:
|
||||
try:
|
||||
response = requests.get(
|
||||
f"{settings['origin']}/global/health",
|
||||
timeout=2,
|
||||
)
|
||||
return response.status_code == 200
|
||||
except requests.RequestException:
|
||||
return False
|
||||
|
||||
|
||||
def _opencode_version(settings: dict) -> str:
|
||||
try:
|
||||
response = requests.get(
|
||||
f"{settings['origin']}/global/health",
|
||||
timeout=2,
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
return str(data.get("version") or "") if isinstance(data, dict) else ""
|
||||
except (requests.RequestException, ValueError):
|
||||
return ""
|
||||
|
||||
|
||||
def _ensure_opencode(settings: dict) -> tuple[bool, str]:
|
||||
global _PROCESS, _PROCESS_READY
|
||||
started_process: subprocess.Popen | None = None
|
||||
workdir = settings["workdir"].resolve()
|
||||
|
||||
with _PROCESS_LOCK:
|
||||
if _owned_process_ready():
|
||||
return True, ""
|
||||
if _health(settings):
|
||||
if _owned_process_running():
|
||||
_PROCESS_READY = _PROCESS
|
||||
return True, ""
|
||||
if _owned_process_running():
|
||||
_stop_owned_process(_PROCESS)
|
||||
|
||||
try:
|
||||
binary, git_binary, binary_env = _resolve_binary(
|
||||
settings["binary"],
|
||||
settings["config"],
|
||||
)
|
||||
except RuntimeError as err:
|
||||
return False, str(err)
|
||||
|
||||
env = {
|
||||
**os.environ,
|
||||
**binary_env,
|
||||
"ARCHIVEBOX_BASE_URL": str(settings.get("archivebox_base_url", "")),
|
||||
"ARCHIVEBOX_ADMIN_URL": str(settings.get("archivebox_admin_url", "")),
|
||||
"ARCHIVEBOX_API_URL": str(settings.get("archivebox_api_url", "")),
|
||||
"BROWSER": "false",
|
||||
"GIT_CEILING_DIRECTORIES": str(workdir),
|
||||
"HOME": str(settings["home"]),
|
||||
"OPENCODE_DISABLE_PROJECT_CONFIG": "true",
|
||||
"XDG_CONFIG_HOME": str(settings["config_home"]),
|
||||
"XDG_DATA_HOME": str(settings["data_home"]),
|
||||
"XDG_STATE_HOME": str(settings["state_home"]),
|
||||
"XDG_CACHE_HOME": str(settings["cache_home"]),
|
||||
}
|
||||
|
||||
settings["workdir"].mkdir(parents=True, exist_ok=True)
|
||||
settings["config_home"].mkdir(parents=True, exist_ok=True)
|
||||
settings["data_home"].mkdir(parents=True, exist_ok=True)
|
||||
settings["state_home"].mkdir(parents=True, exist_ok=True)
|
||||
settings["cache_home"].mkdir(parents=True, exist_ok=True)
|
||||
settings["home"].mkdir(parents=True, exist_ok=True)
|
||||
_ensure_project_files(settings)
|
||||
|
||||
if not (workdir / ".git").exists():
|
||||
try:
|
||||
git_init = git_binary.exec(
|
||||
cmd=("init", "--quiet"),
|
||||
cwd=workdir,
|
||||
env=env,
|
||||
timeout=settings["timeout"],
|
||||
)
|
||||
except (AssertionError, OSError, subprocess.SubprocessError) as err:
|
||||
return False, f"OpenCode project initialization failed: {err}"
|
||||
if git_init.returncode != 0:
|
||||
output = (git_init.stderr or git_init.stdout or "").strip()
|
||||
return (
|
||||
False,
|
||||
f"OpenCode project initialization failed: {output or f'git exited with {git_init.returncode}'}",
|
||||
)
|
||||
|
||||
if _health(settings):
|
||||
return True, ""
|
||||
|
||||
binary_abspath = binary.loaded_abspath
|
||||
if binary.loaded_binprovider is not None:
|
||||
binary_abspath = binary.loaded_binprovider._exec_bin_abspath(
|
||||
Path(binary.loaded_abspath),
|
||||
)
|
||||
cmd = [
|
||||
str(binary_abspath),
|
||||
"serve",
|
||||
"--hostname",
|
||||
settings["host"],
|
||||
"--port",
|
||||
str(settings["port"]),
|
||||
]
|
||||
try:
|
||||
_PROCESS = subprocess.Popen(
|
||||
cmd,
|
||||
cwd=workdir,
|
||||
env=env,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
start_new_session=True,
|
||||
)
|
||||
_PROCESS_READY = None
|
||||
started_process = _PROCESS
|
||||
except FileNotFoundError:
|
||||
return False, f"OpenCode binary not found: {settings['binary']}"
|
||||
|
||||
deadline = time.monotonic() + settings["timeout"]
|
||||
while time.monotonic() < deadline:
|
||||
if _health(settings):
|
||||
_PROCESS_READY = started_process
|
||||
return True, ""
|
||||
if started_process and started_process.poll() is not None:
|
||||
if _PROCESS is started_process:
|
||||
_PROCESS = None
|
||||
if _PROCESS_READY is started_process:
|
||||
_PROCESS_READY = None
|
||||
return False, "OpenCode exited before the web server became ready."
|
||||
time.sleep(0.25)
|
||||
|
||||
_stop_owned_process(started_process)
|
||||
return False, "Timed out waiting for OpenCode to start."
|
||||
|
||||
|
||||
def agent_view(request: HttpRequest):
|
||||
config = _machine_config()
|
||||
_require_enabled(config)
|
||||
auth_response = _require_superuser(request)
|
||||
if auth_response:
|
||||
return auth_response
|
||||
|
||||
settings = _settings(config)
|
||||
route_config = request.__dict__.get("archivebox_config")
|
||||
base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
|
||||
settings["archivebox_base_url"] = base_url
|
||||
settings["archivebox_admin_url"] = admin_url
|
||||
settings["archivebox_api_url"] = api_url
|
||||
ok, error = _ensure_opencode(settings)
|
||||
recent_session_id = ""
|
||||
opencode_version = ""
|
||||
if ok:
|
||||
try:
|
||||
with _SESSION_LOCK:
|
||||
recent_session_id = _ensure_default_session(settings)
|
||||
opencode_version = _opencode_version(settings)
|
||||
if not opencode_version:
|
||||
ok = False
|
||||
error = "OpenCode health check did not report its version."
|
||||
except (requests.RequestException, RuntimeError, ValueError) as err:
|
||||
ok = False
|
||||
error = f"OpenCode project initialization failed: {err}"
|
||||
from archivebox.core.admin_site import archivebox_admin
|
||||
|
||||
context = {
|
||||
**archivebox_admin.each_context(request),
|
||||
"title": "Agent",
|
||||
"error": "" if ok else error,
|
||||
"command": f"{settings['binary']} serve --hostname {settings['host']} --port {settings['port']}" if error else "",
|
||||
# OpenCode 1.17+ keeps durable sessions on the explicit
|
||||
# /<dirBase64>/session/<sessionId> route. We still seed localStorage
|
||||
# below because the sidebar state uses it, but the iframe itself must
|
||||
# open the durable session URL so a fresh browser does not land on the
|
||||
# transient new-session route and appear to have lost prior sessions.
|
||||
"proxy_url": _project_route(settings["workdir"], recent_session_id),
|
||||
"proxy_prefix": _PROXY_PREFIX,
|
||||
"opencode_version": opencode_version,
|
||||
"workdir": str(settings["workdir"].resolve()),
|
||||
"recent_session_id": recent_session_id,
|
||||
}
|
||||
return render(
|
||||
request,
|
||||
"opencode/agent.html",
|
||||
context,
|
||||
status=200 if ok else 502,
|
||||
)
|
||||
|
||||
|
||||
def agent_health_view(request: HttpRequest):
|
||||
config = _machine_config()
|
||||
_require_enabled(config)
|
||||
auth_response = _require_superuser(request)
|
||||
if auth_response:
|
||||
return auth_response
|
||||
|
||||
version = _opencode_version(_settings(config))
|
||||
healthy = bool(version)
|
||||
response = JsonResponse(
|
||||
{"healthy": healthy, "version": version},
|
||||
status=200 if healthy else 503,
|
||||
)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return response
|
||||
|
||||
|
||||
def _proxy_url(settings: dict, path: str | None) -> str:
|
||||
rel = "/" if not path else f"/{path}"
|
||||
return urljoin(settings["origin"], rel)
|
||||
|
||||
|
||||
def _request_header(request: HttpRequest, name: str) -> str | None:
|
||||
meta = getattr(request, "META", {})
|
||||
if name == "Content-Type":
|
||||
value = meta.get("CONTENT_TYPE")
|
||||
elif name == "Content-Length":
|
||||
value = meta.get("CONTENT_LENGTH")
|
||||
else:
|
||||
value = meta.get(f"HTTP_{name.upper().replace('-', '_')}")
|
||||
return str(value) if value else None
|
||||
|
||||
|
||||
def _request_headers(request: HttpRequest, settings: dict) -> dict[str, str]:
|
||||
forwarded = {}
|
||||
for key in ("Accept", "Accept-Language", "Content-Type", "Range", "User-Agent"):
|
||||
value = _request_header(request, key)
|
||||
if value:
|
||||
forwarded[key] = value
|
||||
return forwarded
|
||||
|
||||
|
||||
def _request_params(request: HttpRequest) -> tuple[tuple[str, str], ...]:
|
||||
if hasattr(request.GET, "lists"):
|
||||
return tuple((key, str(value)) for key, values in request.GET.lists() for value in values)
|
||||
return tuple((key, str(value)) for key, value in dict(request.GET).items())
|
||||
|
||||
|
||||
async def _event_chunks(
|
||||
settings: dict,
|
||||
path: str | None,
|
||||
method: str,
|
||||
params: tuple[tuple[str, str], ...],
|
||||
headers: dict[str, str],
|
||||
):
|
||||
if not _owned_process_ready():
|
||||
ok, error = await sync_to_async(_ensure_opencode, thread_sensitive=False)(settings)
|
||||
if not ok:
|
||||
_LOGGER.warning("OpenCode event stream unavailable: %s", error)
|
||||
yield b'event: error\ndata: {"error":"OpenCode upstream unavailable"}\n\n'
|
||||
return
|
||||
|
||||
timeout = httpx.Timeout(settings["timeout"], read=None)
|
||||
url = _proxy_url(settings, path)
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
|
||||
async with client.stream(
|
||||
method,
|
||||
url,
|
||||
params=params,
|
||||
headers=headers,
|
||||
) as upstream:
|
||||
async for chunk in upstream.aiter_raw(chunk_size=512):
|
||||
yield chunk
|
||||
except httpx.RequestError as err:
|
||||
_LOGGER.warning("OpenCode event stream ended: %s", err)
|
||||
|
||||
|
||||
def _rewrite_text(body: bytes, settings: dict) -> bytes:
|
||||
text = body.decode("utf-8", errors="replace")
|
||||
text = text.replace(settings["origin"], _PROXY_PREFIX)
|
||||
text = text.replace("location.origin", f'location.origin+"{_PROXY_PREFIX}"')
|
||||
text = text.replace(
|
||||
"k(k5,{get component(){return t.router??Az},",
|
||||
f'k(k5,{{base:"{_PROXY_PREFIX}",get component(){{return t.router??Az}},',
|
||||
)
|
||||
text = text.replace('"/assets/', f'"{_PROXY_PREFIX}/assets/')
|
||||
text = text.replace("'/assets/", f"'{_PROXY_PREFIX}/assets/")
|
||||
proxy_path = rf'(\1.replace(/^{_PROXY_PREFIX_REGEX}(?=\/|$)/,"")||"/")'
|
||||
text = re.sub(r"\b(window\.location\.pathname)\b", proxy_path, text)
|
||||
text = re.sub(r"(?<![.\w])(location\.pathname)\b", proxy_path, text)
|
||||
text = text.replace(
|
||||
'window.history.replaceState(nz(o),"",r):window.history.pushState(o,"",r)',
|
||||
(
|
||||
f'window.history.replaceState(nz(o),"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r):'
|
||||
f'window.history.pushState(o,"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r)'
|
||||
),
|
||||
)
|
||||
text = text.replace(
|
||||
'const BL="modulepreload",UL=function(t){return"/"+t}',
|
||||
f'const BL="modulepreload",UL=function(t){{return"{_PROXY_PREFIX}/"+t}}',
|
||||
)
|
||||
text = re.sub(
|
||||
rf"""(?P<prefix>\b(?:href|src|action)=["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
|
||||
rf"\g<prefix>{_PROXY_PREFIX}/",
|
||||
text,
|
||||
)
|
||||
text = re.sub(
|
||||
rf"""(?P<prefix>\b(?:fetch|EventSource)\(["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
|
||||
rf"\g<prefix>{_PROXY_PREFIX}/",
|
||||
text,
|
||||
)
|
||||
text = re.sub(
|
||||
rf"""(?P<prefix>\burl\(["']?)/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
|
||||
rf"\g<prefix>{_PROXY_PREFIX}/",
|
||||
text,
|
||||
)
|
||||
return text.encode("utf-8")
|
||||
|
||||
|
||||
def _response_headers(upstream: requests.Response, settings: dict) -> dict[str, str]:
|
||||
headers = {}
|
||||
for key, value in upstream.headers.items():
|
||||
lower = key.lower()
|
||||
if lower in _HOP_BY_HOP_HEADERS or lower in {
|
||||
"content-length",
|
||||
"content-encoding",
|
||||
"x-frame-options",
|
||||
}:
|
||||
continue
|
||||
if lower == "location":
|
||||
if value.startswith(settings["origin"]):
|
||||
value = value.replace(settings["origin"], _PROXY_PREFIX, 1)
|
||||
elif value.startswith("/"):
|
||||
value = f"{_PROXY_PREFIX}{value}"
|
||||
headers[key] = value
|
||||
return headers
|
||||
|
||||
|
||||
def _proxy_error_response(error: Exception | str) -> HttpResponse:
|
||||
_LOGGER.warning("OpenCode upstream request failed: %s", error)
|
||||
return HttpResponse(
|
||||
b"OpenCode upstream request failed.",
|
||||
status=502,
|
||||
content_type="text/plain; charset=utf-8",
|
||||
)
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def opencode_proxy_view(request, path=None):
|
||||
return _dispatch(request, path=path or "")
|
||||
|
||||
|
||||
def _websocket_context(scope):
|
||||
close_old_connections()
|
||||
try:
|
||||
match = resolve(scope["path"])
|
||||
if match.url_name != "opencode-proxy":
|
||||
raise PermissionDenied
|
||||
request = ASGIRequest(
|
||||
{**scope, "type": "http", "method": "GET", "scheme": "https" if scope.get("scheme") == "wss" else "http"},
|
||||
BytesIO(),
|
||||
def opencode_proxy_view(request: HttpRequest, path: str | None = None):
|
||||
config = _machine_config()
|
||||
_require_enabled(config)
|
||||
auth_response = _require_superuser(request)
|
||||
if auth_response:
|
||||
return auth_response
|
||||
if not _origin_allowed(request, path):
|
||||
return HttpResponseForbidden(
|
||||
b"Cross-origin OpenCode agent requests are blocked.",
|
||||
)
|
||||
route_config = get_request_config(request, resolve_plugins=False)
|
||||
request.archivebox_config = route_config
|
||||
if not route_config.CONTROL_PLANE_ENABLED:
|
||||
raise PermissionDenied
|
||||
if (
|
||||
route_config.USES_SUBDOMAIN_ROUTING
|
||||
and route_config.BASE_URL
|
||||
and not host_matches(request.get_host(), get_admin_host(config=route_config, request=request))
|
||||
):
|
||||
raise PermissionDenied
|
||||
SessionMiddleware(_dispatch).process_request(request)
|
||||
AuthenticationMiddleware(_dispatch).process_request(request)
|
||||
ReverseProxyAuthMiddleware(_dispatch).process_request(request)
|
||||
config = route_config.model_dump(mode="json")
|
||||
if not config.get("OPENCODE_ENABLED") or not request.user.is_active or not request.user.is_superuser:
|
||||
raise PermissionDenied
|
||||
runtime, settings = _runtime_settings(request, config)
|
||||
if not request.headers.get("Origin") or not runtime._origin_allowed("POST", request.get_host(), request.headers):
|
||||
raise PermissionDenied
|
||||
return runtime, settings, match.kwargs.get("path", "")
|
||||
finally:
|
||||
close_old_connections()
|
||||
|
||||
settings = _settings(config)
|
||||
route_config = request.__dict__.get("archivebox_config")
|
||||
base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
|
||||
settings["archivebox_base_url"] = base_url
|
||||
settings["archivebox_admin_url"] = admin_url
|
||||
settings["archivebox_api_url"] = api_url
|
||||
|
||||
if request.method == "GET" and (path or "").endswith("/event"):
|
||||
response = StreamingHttpResponse(
|
||||
_event_chunks(
|
||||
settings,
|
||||
path,
|
||||
request.method or "GET",
|
||||
_request_params(request),
|
||||
_request_headers(request, settings),
|
||||
),
|
||||
content_type="text/event-stream",
|
||||
)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
response.headers["X-Accel-Buffering"] = "no"
|
||||
return response
|
||||
|
||||
if path == "global/health" or not _owned_process_ready():
|
||||
ok, error = _ensure_opencode(settings)
|
||||
if not ok:
|
||||
return _proxy_error_response(error)
|
||||
|
||||
async def websocket_view(scope, receive, send):
|
||||
if (await receive())["type"] != "websocket.connect":
|
||||
return
|
||||
try:
|
||||
runtime, settings, path = await sync_to_async(_websocket_context)(scope)
|
||||
await runtime.websocket_proxy(settings, path, scope.get("query_string", b""), scope.get("subprotocols", []), receive, send)
|
||||
except (PermissionDenied, Resolver404, Http404):
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
except Exception:
|
||||
_LOGGER.exception("Optional AI WebSocket failed")
|
||||
await send({"type": "websocket.close", "code": 1011})
|
||||
method = request.method or "GET"
|
||||
upstream = requests.request(
|
||||
method,
|
||||
_proxy_url(settings, path),
|
||||
params=_request_params(request),
|
||||
data=request.body if method not in {"GET", "HEAD"} else None,
|
||||
headers=_request_headers(request, settings),
|
||||
stream=True,
|
||||
timeout=settings["timeout"],
|
||||
allow_redirects=False,
|
||||
)
|
||||
except requests.RequestException as err:
|
||||
return _proxy_error_response(err)
|
||||
|
||||
try:
|
||||
body = upstream.content
|
||||
except requests.RequestException as err:
|
||||
upstream.close()
|
||||
return _proxy_error_response(err)
|
||||
|
||||
content_type = upstream.headers.get("Content-Type", "")
|
||||
headers = _response_headers(upstream, settings)
|
||||
if any(content_type.startswith(prefix) for prefix in _TEXT_CONTENT_TYPES):
|
||||
body = _rewrite_text(body, settings)
|
||||
response = HttpResponse(
|
||||
body,
|
||||
status=upstream.status_code,
|
||||
content_type=content_type or "application/octet-stream",
|
||||
)
|
||||
for key, value in headers.items():
|
||||
response.headers[key] = value
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return response
|
||||
@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
"Persona",
|
||||
{
|
||||
"fields": ("name", "created_by", "permissions"),
|
||||
"classes": ("card", "wide", "persona-card-primary"),
|
||||
"classes": ("card", "persona-card-primary"),
|
||||
},
|
||||
),
|
||||
(
|
||||
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
"import_extract_cookies",
|
||||
"import_capture_storage",
|
||||
),
|
||||
"classes": ("card", "wide", "persona-card-import"),
|
||||
"classes": ("card", "wide"),
|
||||
},
|
||||
),
|
||||
(
|
||||
@@ -63,6 +63,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
"classes": ("card", "persona-card-timestamps"),
|
||||
},
|
||||
),
|
||||
add_fieldsets[1],
|
||||
add_fieldsets[2],
|
||||
(
|
||||
"Artifacts",
|
||||
@@ -135,7 +136,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
|
||||
return super().get_form(request, obj=obj, change=change, **kwargs)
|
||||
|
||||
def render_change_form(self, request, context, add=False, change=False, form_url="", obj=None):
|
||||
context["detected_profile_count"] = len(discover_local_browser_profiles()) if add else 0
|
||||
context["detected_profile_count"] = len(discover_local_browser_profiles())
|
||||
return super().render_change_form(request, context, add=add, change=change, form_url=form_url, obj=obj)
|
||||
|
||||
def save_model(self, request, obj, form, change):
|
||||
|
||||
@@ -9,11 +9,11 @@
|
||||
* COOKIES_OUTPUT_FILE Optional output path for Netscape cookies.txt
|
||||
* AUTH_STORAGE_OUTPUT_FILE Optional output path for auth.json
|
||||
* CHROME_BINARY Absolute browser path resolved by abxpkg
|
||||
* CHROME_SANDBOX Set to false when the Linux sandbox is unavailable
|
||||
* NODE_MODULES_DIR node_modules path resolved by abxpkg
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
|
||||
const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR || process.env.ABX_PLUGINS_DIR;
|
||||
@@ -26,7 +26,7 @@ const baseUtils = require(path.join(pluginsDir, 'base', 'utils.js'));
|
||||
baseUtils.ensureNodeModuleResolution(module);
|
||||
|
||||
const chromeUtils = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
|
||||
const puppeteer = require('puppeteer');
|
||||
const puppeteer = require('puppeteer-core');
|
||||
|
||||
function cookieToNetscape(cookie) {
|
||||
let domain = cookie.domain;
|
||||
@@ -115,28 +115,38 @@ async function openBrowser() {
|
||||
throw new Error(`User data directory does not exist: ${userDataDir}`);
|
||||
}
|
||||
|
||||
const outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'abx-browser-state-'));
|
||||
const binary = process.env.CHROME_BINARY;
|
||||
if (!binary || !path.isAbsolute(binary) || !fs.existsSync(binary)) {
|
||||
throw new Error('CHROME_BINARY must be an absolute executable path resolved by abxpkg');
|
||||
}
|
||||
|
||||
// The ordinary archiving launcher uses a mock keychain. An import must
|
||||
// instead let the original signed browser decrypt its own profile cookies.
|
||||
const sandboxDisabled = ['false', '0', 'no', 'off'].includes(
|
||||
String(process.env.CHROME_SANDBOX || '').trim().toLowerCase(),
|
||||
);
|
||||
const browser = await puppeteer.launch({
|
||||
executablePath: binary,
|
||||
const launched = await chromeUtils.launchChromium({
|
||||
binary,
|
||||
outputDir,
|
||||
userDataDir,
|
||||
headless: true,
|
||||
ignoreDefaultArgs: ['--use-mock-keychain', '--password-store=basic'],
|
||||
args: ['--no-first-run', '--disable-sync', '--disable-background-networking',
|
||||
'--profile-directory=Default', ...(process.platform === 'linux' && (process.getuid?.() === 0 || sandboxDisabled) ? ['--no-sandbox'] : [])],
|
||||
killZombies: false,
|
||||
});
|
||||
|
||||
if (!launched.success) {
|
||||
throw new Error(launched.error || 'Chrome launch failed');
|
||||
}
|
||||
|
||||
const browser = await chromeUtils.connectToBrowserEndpoint(puppeteer, launched.cdpUrl, { defaultViewport: null });
|
||||
|
||||
return {
|
||||
browser,
|
||||
async cleanup() {
|
||||
await browser.close();
|
||||
try {
|
||||
await browser.disconnect();
|
||||
} catch (error) {}
|
||||
try {
|
||||
await chromeUtils.killChrome(launched.pid, outputDir);
|
||||
} catch (error) {}
|
||||
try {
|
||||
fs.rmSync(outputDir, { recursive: true, force: true });
|
||||
} catch (error) {}
|
||||
},
|
||||
sourceDescription: userDataDir,
|
||||
};
|
||||
|
||||
@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
|
||||
label="Autodiscovered profiles",
|
||||
widget=forms.RadioSelect,
|
||||
choices=(),
|
||||
help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
|
||||
help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
|
||||
)
|
||||
import_source = forms.CharField(
|
||||
required=False,
|
||||
|
||||
@@ -11,14 +11,10 @@ import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import sqlite3
|
||||
import time
|
||||
from http.cookiejar import MozillaCookieJar
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import urlparse
|
||||
@@ -32,7 +28,6 @@ if TYPE_CHECKING:
|
||||
|
||||
BROWSER_LABELS = {
|
||||
"chrome": "Google Chrome",
|
||||
"chrome-beta": "Google Chrome Beta",
|
||||
"chromium": "Chromium",
|
||||
"brave": "Brave",
|
||||
"edge": "Microsoft Edge",
|
||||
@@ -47,27 +42,25 @@ BROWSER_PROFILE_DIR_NAMES = (
|
||||
)
|
||||
|
||||
VOLATILE_PROFILE_COPY_PATTERNS = (
|
||||
# Chromium/macOS atomic-write staging files (e.g.
|
||||
# .com.brave.Browser.TransportSecurity.Ze7UBU) disappear after rename.
|
||||
".*.??????",
|
||||
"Cache",
|
||||
"Code Cache",
|
||||
"GPUCache",
|
||||
"ShaderCache",
|
||||
"Service Worker",
|
||||
"GCM Store",
|
||||
"chrome-extension_*",
|
||||
"DNR Extension Rules",
|
||||
"Extension Rules",
|
||||
"Extension Scripts",
|
||||
"Extension State",
|
||||
"Local Extension Settings",
|
||||
"*.log",
|
||||
"Crashpad",
|
||||
"BrowserMetrics",
|
||||
"BrowserMetrics-spare.pma",
|
||||
"RunningChromeVersion",
|
||||
"DevToolsActivePort",
|
||||
"SingletonLock",
|
||||
"SingletonSocket",
|
||||
"SingletonCookie",
|
||||
"Sessions",
|
||||
"Sessions_Encrypted",
|
||||
"Current Session",
|
||||
"Current Tabs",
|
||||
"Last Session",
|
||||
"Last Tabs",
|
||||
)
|
||||
|
||||
PERSONA_PROFILE_DIR_CANDIDATES = (
|
||||
@@ -208,7 +201,6 @@ class PersonaImportResult:
|
||||
source: PersonaImportSource
|
||||
profile_copied: bool = False
|
||||
cookies_imported: bool = False
|
||||
cookie_count: int = 0
|
||||
storage_captured: bool = False
|
||||
user_agent_imported: bool = False
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
@@ -218,29 +210,28 @@ class PersonaImportResult:
|
||||
return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
|
||||
|
||||
|
||||
def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
|
||||
def get_chrome_user_data_dir() -> Path | None:
|
||||
"""Get the default Chrome user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
app_name = "Chrome Beta" if beta else "Chrome"
|
||||
|
||||
if system == "Darwin":
|
||||
candidates = [
|
||||
home / "Library" / "Application Support" / "Google" / app_name,
|
||||
home / "Library" / "Application Support" / "Google" / "Chrome",
|
||||
home / "Library" / "Application Support" / "Chromium",
|
||||
]
|
||||
elif system == "Linux":
|
||||
candidates = (
|
||||
[home / ".config" / "google-chrome-beta"]
|
||||
if beta
|
||||
else [
|
||||
home / ".config" / "google-chrome",
|
||||
home / ".config" / "chrome",
|
||||
]
|
||||
)
|
||||
candidates = [
|
||||
home / ".config" / "google-chrome",
|
||||
home / ".config" / "chromium",
|
||||
home / ".config" / "chrome",
|
||||
home / "snap" / "chromium" / "common" / "chromium",
|
||||
]
|
||||
elif system == "Windows":
|
||||
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
|
||||
candidates = [
|
||||
local_app_data / "Google" / app_name / "User Data",
|
||||
local_app_data / "Google" / "Chrome" / "User Data",
|
||||
local_app_data / "Chromium" / "User Data",
|
||||
]
|
||||
else:
|
||||
candidates = []
|
||||
@@ -252,21 +243,6 @@ def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
|
||||
return None
|
||||
|
||||
|
||||
def get_chromium_user_data_dir() -> Path | None:
|
||||
"""Select Chromium's own profile even when Google Chrome is installed."""
|
||||
home = Path.home()
|
||||
if platform.system() == "Darwin":
|
||||
candidates = [home / "Library/Application Support/Chromium"]
|
||||
elif platform.system() == "Windows":
|
||||
candidates = [Path(os.environ.get("LOCALAPPDATA", home / "AppData/Local")) / "Chromium/User Data"]
|
||||
else:
|
||||
candidates = [
|
||||
Path(os.environ.get("XDG_CONFIG_HOME", home / ".config")) / "chromium",
|
||||
home / "snap/chromium/common/chromium",
|
||||
]
|
||||
return next((path for path in candidates if path.is_dir() and _list_profile_names(path)), None)
|
||||
|
||||
|
||||
def get_brave_user_data_dir() -> Path | None:
|
||||
"""Get the default Brave user data directory for the current platform."""
|
||||
system = platform.system()
|
||||
@@ -327,8 +303,7 @@ def get_edge_user_data_dir() -> Path | None:
|
||||
|
||||
BROWSER_PROFILE_FINDERS = {
|
||||
"chrome": get_chrome_user_data_dir,
|
||||
"chrome-beta": partial(get_chrome_user_data_dir, beta=True),
|
||||
"chromium": get_chromium_user_data_dir,
|
||||
"chromium": get_chrome_user_data_dir,
|
||||
"brave": get_brave_user_data_dir,
|
||||
"edge": get_edge_user_data_dir,
|
||||
}
|
||||
@@ -362,12 +337,6 @@ def validate_persona_name(name: str) -> tuple[bool, str]:
|
||||
|
||||
|
||||
def discover_local_browser_profiles() -> list[PersonaImportSource]:
|
||||
# The macOS companion exports with the host Keychain before creating personas.
|
||||
# Raw mounts alone cannot decrypt macOS cookies in the Linux container.
|
||||
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
|
||||
host_templates = discover_persona_template_profiles(Path(host_profiles))
|
||||
host_names = {source.source_name for source in host_templates}
|
||||
return host_templates + [source for source in discover_persona_template_profiles() if source.source_name not in host_names]
|
||||
discovered: list[PersonaImportSource] = []
|
||||
|
||||
for browser, finder in BROWSER_PROFILE_FINDERS.items():
|
||||
@@ -444,21 +413,13 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
|
||||
|
||||
def resolve_browser_import_source(browser: str, profile_dir: str | None = None) -> PersonaImportSource:
|
||||
browser = browser.lower().strip()
|
||||
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
|
||||
label = BROWSER_LABELS.get(browser, browser)
|
||||
exported = Path(host_profiles) / f"{label} - {profile_dir or 'Default'}"
|
||||
if exported.is_dir():
|
||||
return resolve_custom_import_source(str(exported))
|
||||
if browser not in BROWSER_PROFILE_FINDERS:
|
||||
supported = ", ".join(BROWSER_PROFILE_FINDERS)
|
||||
raise ValueError(f"Unknown browser: {browser}. Supported browsers: {supported}")
|
||||
|
||||
user_data_dir = BROWSER_PROFILE_FINDERS[browser]()
|
||||
if not user_data_dir:
|
||||
raise ValueError(
|
||||
f"Could not find {browser} profile directory. Use --source /path/to/browser-data. "
|
||||
"For Docker on macOS/Windows, import on the host into the shared data directory first.",
|
||||
)
|
||||
raise ValueError(f"Could not find {browser} profile directory")
|
||||
|
||||
chosen_profile = profile_dir or pick_default_profile_dir(user_data_dir)
|
||||
if not chosen_profile:
|
||||
@@ -483,8 +444,6 @@ def resolve_browser_profile_source(
|
||||
resolved_root = resolved_root.resolve()
|
||||
if not resolved_root.exists():
|
||||
raise ValueError(f"Profile root does not exist: {resolved_root}")
|
||||
if Path(profile_dir).name != profile_dir or profile_dir in {".", ".."}:
|
||||
raise ValueError("Profile must be a directory name within the source browser root.")
|
||||
if not profile_dir.strip():
|
||||
raise ValueError("Profile directory name cannot be empty.")
|
||||
|
||||
@@ -517,17 +476,6 @@ def resolve_custom_import_source(raw_value: str, profile_dir: str | None = None)
|
||||
raise ValueError(f"Custom browser path does not exist: {source_path}")
|
||||
|
||||
explicit_profile = profile_dir.strip() if profile_dir else ""
|
||||
for profile_root in PERSONA_PROFILE_DIR_CANDIDATES:
|
||||
root = source_path / profile_root
|
||||
if root.is_dir():
|
||||
selected = explicit_profile or pick_default_profile_dir(root)
|
||||
if selected:
|
||||
return resolve_browser_profile_source(
|
||||
browser="persona",
|
||||
source_name=source_path.name,
|
||||
user_data_dir=root.resolve(),
|
||||
profile_dir=selected,
|
||||
)
|
||||
if _looks_like_profile_dir(source_path):
|
||||
if explicit_profile and explicit_profile != source_path.name:
|
||||
raise ValueError("Profile name does not match the provided profile directory path.")
|
||||
@@ -560,48 +508,6 @@ def pick_default_profile_dir(user_data_dir: Path) -> str | None:
|
||||
return profiles[0]
|
||||
|
||||
|
||||
def resolve_source_browser_binary(source: PersonaImportSource) -> str:
|
||||
"""Use the originating browser so OS-protected cookies use the correct keychain."""
|
||||
if source.browser_binary:
|
||||
return str(Path(source.browser_binary).expanduser().resolve())
|
||||
names = {
|
||||
"chrome": ("Google Chrome", "google-chrome"),
|
||||
"chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
|
||||
"chromium": ("Chromium", "chromium"),
|
||||
"brave": ("Brave Browser", "brave-browser"),
|
||||
"edge": ("Microsoft Edge", "microsoft-edge"),
|
||||
}
|
||||
app, executable = names.get(source.browser, ("", ""))
|
||||
candidates = []
|
||||
if platform.system() == "Darwin" and app:
|
||||
candidates = [
|
||||
Path("/Applications") / f"{app}.app/Contents/MacOS/{app}",
|
||||
Path.home() / f"Applications/{app}.app/Contents/MacOS/{app}",
|
||||
]
|
||||
elif platform.system() == "Windows":
|
||||
relative = {
|
||||
"chrome": "Google/Chrome/Application/chrome.exe",
|
||||
"chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
|
||||
"edge": "Microsoft/Edge/Application/msedge.exe",
|
||||
"brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
|
||||
"chromium": "Chromium/Application/chrome.exe",
|
||||
}.get(source.browser)
|
||||
if relative:
|
||||
candidates = [
|
||||
Path(os.environ[root]) / relative for root in ("LOCALAPPDATA", "PROGRAMFILES", "PROGRAMFILES(X86)") if os.environ.get(root)
|
||||
]
|
||||
for candidate in candidates:
|
||||
if candidate.is_file() and os.access(candidate, os.X_OK):
|
||||
return str(candidate)
|
||||
if executable and (found := shutil.which(executable)):
|
||||
return found
|
||||
raise ValueError(
|
||||
f"Source {source.browser} browser executable not found. Use --browser-binary with the originating Chromium browser. "
|
||||
"For Docker, run persona create --import on the browser's host into the shared data directory first: "
|
||||
"the container cannot decrypt cookies using the host's macOS Keychain or Windows credentials.",
|
||||
)
|
||||
|
||||
|
||||
def import_persona_from_source(
|
||||
persona: Persona,
|
||||
source: PersonaImportSource,
|
||||
@@ -610,182 +516,67 @@ def import_persona_from_source(
|
||||
import_cookies: bool = True,
|
||||
capture_storage: bool = False,
|
||||
) -> PersonaImportResult:
|
||||
# Stage all work before replacing an existing, working identity.
|
||||
persona.ensure_dirs()
|
||||
result = PersonaImportResult(source=source)
|
||||
expected_cookies = 0
|
||||
browser_binary = None
|
||||
native_cookie_browsers = {"chrome", "chromium", "brave", "edge", "vivaldi", "opera", "opera_gx"}
|
||||
# browser-cookie3 needs a desktop D-Bus session on Linux. In a headless
|
||||
# environment, let the originating browser decode its own profile instead.
|
||||
has_native_cookie_decoder = platform.system() != "Linux" or bool(os.environ.get("DBUS_SESSION_BUS_ADDRESS"))
|
||||
native_cookie_import = source.kind == "browser-profile" and source.browser in native_cookie_browsers and has_native_cookie_decoder
|
||||
portable_import = source.kind == "browser-profile" and source.browser == "persona"
|
||||
if source.kind == "browser-profile" and not portable_import and (import_cookies or capture_storage):
|
||||
if not native_cookie_import:
|
||||
browser_binary = resolve_source_browser_binary(source)
|
||||
for cookie_db in (source.profile_path / "Network" / "Cookies", source.profile_path / "Cookies"):
|
||||
if cookie_db.is_file():
|
||||
with sqlite3.connect(f"{cookie_db.as_uri()}?mode=ro", uri=True) as conn:
|
||||
expected_cookies = conn.execute(
|
||||
"SELECT COUNT(*) FROM cookies WHERE expires_utc = 0 OR expires_utc > ?",
|
||||
(int((time.time() + 11644473600) * 1_000_000),),
|
||||
).fetchone()[0]
|
||||
break
|
||||
with tempfile.TemporaryDirectory(prefix=".import-", dir=persona.path.parent) as tmp:
|
||||
stage = Path(tmp)
|
||||
staged_profile = stage / "chrome_profile"
|
||||
if source.kind == "browser-profile":
|
||||
assert source.user_data_dir and source.profile_path
|
||||
if copy_profile or import_cookies or capture_storage:
|
||||
staged_profile.mkdir()
|
||||
# Only the selected profile belongs to this persona. Normalize its
|
||||
# name so downstream Chromium launches always select it.
|
||||
copy_browser_user_data_dir(source.profile_path, staged_profile / "Default")
|
||||
local_state = source.user_data_dir / "Local State"
|
||||
if local_state.exists():
|
||||
state = json.loads(local_state.read_text())
|
||||
profile_state = state.setdefault("profile", {})
|
||||
profile_state["last_used"] = "Default"
|
||||
profile_state["last_active_profiles"] = ["Default"]
|
||||
info = profile_state.get("info_cache", {}).get(source.profile_dir)
|
||||
profile_state["info_cache"] = {"Default": info} if info else {}
|
||||
(staged_profile / "Local State").write_text(json.dumps(state))
|
||||
persona.cleanup_chrome_profile(staged_profile)
|
||||
result.profile_copied = copy_profile
|
||||
elif copy_profile:
|
||||
result.warnings.append(
|
||||
"CDP imports capture cookies and open-tab storage, not browser settings. Use --source for a full profile import.",
|
||||
)
|
||||
|
||||
if import_cookies or capture_storage:
|
||||
if portable_import:
|
||||
export_dir = source.user_data_dir.parent
|
||||
if not all((export_dir / name).is_file() for name in ("auth.json", "cookies.txt")):
|
||||
raise ValueError("This persona has no portable cookie export. Re-import it on the original browser host first.")
|
||||
auth_payload = json.loads((export_dir / "auth.json").read_text())
|
||||
if not isinstance(auth_payload, dict) or not isinstance(auth_payload.get("cookies"), list):
|
||||
raise ValueError("Invalid persona auth.json: expected a cookies list.")
|
||||
for name in ("auth.json", "cookies.txt"):
|
||||
shutil.copy2(export_dir / name, stage / name)
|
||||
success, message = True, ""
|
||||
elif native_cookie_import:
|
||||
auth_payload = export_profile_cookies(source, staged_profile, stage)
|
||||
success, message = True, ""
|
||||
persona_chrome_dir = Path(persona.CHROME_USER_DATA_DIR)
|
||||
cookies_file = persona.path / "cookies.txt"
|
||||
auth_file = persona.path / "auth.json"
|
||||
|
||||
launch_user_data_dir: Path | None = None
|
||||
|
||||
if source.kind == "browser-profile":
|
||||
if copy_profile and source.user_data_dir:
|
||||
resolved_source_root = source.user_data_dir.resolve()
|
||||
resolved_persona_root = persona_chrome_dir.resolve()
|
||||
if resolved_source_root == resolved_persona_root:
|
||||
result.warnings.append(
|
||||
"Skipped profile copy because the selected source is already this persona's chrome_profile directory.",
|
||||
)
|
||||
else:
|
||||
# Custom browsers and headless Linux use the originating browser.
|
||||
# Desktop imports use their OS decoder without launching or probing.
|
||||
launch_profile = stage / "export_profile"
|
||||
if source.kind == "browser-profile":
|
||||
copy_browser_user_data_dir(staged_profile, launch_profile)
|
||||
success, auth_payload, message = export_browser_state(
|
||||
user_data_dir=launch_profile if source.kind == "browser-profile" else None,
|
||||
cdp_url=source.cdp_url,
|
||||
profile_dir="Default" if source.kind == "browser-profile" else None,
|
||||
chrome_binary=browser_binary,
|
||||
cookies_output_file=stage / "cookies.txt" if import_cookies else None,
|
||||
auth_output_file=stage / "auth.json",
|
||||
)
|
||||
if not success:
|
||||
raise ValueError(message or "Browser state export failed; persona was not replaced.")
|
||||
if expected_cookies and not (auth_payload or {}).get("cookies"):
|
||||
raise ValueError(
|
||||
"The source has cookies but the browser exported none. Run the import on the source host "
|
||||
"with the original browser and unlock its OS keychain; the existing persona was not replaced.",
|
||||
)
|
||||
if import_cookies:
|
||||
result.cookies_imported = True
|
||||
result.cookie_count = len((auth_payload or {}).get("cookies", []))
|
||||
if capture_storage:
|
||||
result.storage_captured = True
|
||||
# Keep full CDP cookie attributes (sameSite, httpOnly, partition keys)
|
||||
# as well as cookies.txt for non-browser extractors.
|
||||
result.user_agent_imported = _apply_imported_user_agent(persona, auth_payload)
|
||||
copy_browser_user_data_dir(resolved_source_root, resolved_persona_root)
|
||||
persona.cleanup_chrome_profile(resolved_persona_root)
|
||||
result.profile_copied = True
|
||||
launch_user_data_dir = resolved_persona_root
|
||||
else:
|
||||
launch_user_data_dir = source.user_data_dir
|
||||
elif copy_profile:
|
||||
result.warnings.append(
|
||||
"Profile copying is only available for local Chromium profile paths. CDP imports can only pull cookies and open-tab storage.",
|
||||
)
|
||||
|
||||
if not import_cookies and not capture_storage:
|
||||
return result
|
||||
|
||||
if source.kind == "cdp":
|
||||
export_success, auth_payload, export_message = export_browser_state(
|
||||
cdp_url=source.cdp_url,
|
||||
cookies_output_file=cookies_file if import_cookies else None,
|
||||
auth_output_file=auth_file if capture_storage else None,
|
||||
)
|
||||
else:
|
||||
export_success, auth_payload, export_message = export_browser_state(
|
||||
user_data_dir=launch_user_data_dir,
|
||||
profile_dir=source.profile_dir,
|
||||
chrome_binary=source.browser_binary,
|
||||
cookies_output_file=cookies_file if import_cookies else None,
|
||||
auth_output_file=auth_file if capture_storage else None,
|
||||
)
|
||||
|
||||
if not export_success:
|
||||
result.warnings.append(export_message or "Browser import failed.")
|
||||
return result
|
||||
|
||||
if import_cookies and cookies_file.exists():
|
||||
result.cookies_imported = True
|
||||
if capture_storage and auth_file.exists():
|
||||
result.storage_captured = True
|
||||
if _apply_imported_user_agent(persona, auth_payload):
|
||||
result.user_agent_imported = True
|
||||
|
||||
if result.profile_copied:
|
||||
target = Path(persona.CHROME_USER_DATA_DIR)
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
staged_profile.rename(target)
|
||||
for filename in ("cookies.txt", "auth.json"):
|
||||
exported = stage / filename
|
||||
if exported.exists():
|
||||
exported.chmod(0o600)
|
||||
exported.replace(persona.path / filename)
|
||||
return result
|
||||
|
||||
|
||||
def export_profile_cookies(source: PersonaImportSource, staged_profile: Path, output: Path) -> dict:
|
||||
"""Decode with the host keychain, never by starting the user's browser."""
|
||||
import browser_cookie3
|
||||
|
||||
payload = {"TYPE": "auth", "cookies": [], "localStorage": {}, "sessionStorage": {}}
|
||||
jar = MozillaCookieJar(str(output / "cookies.txt"))
|
||||
assert source.profile_path
|
||||
for relative in (Path("Network/Cookies"), Path("Cookies")):
|
||||
original = source.profile_path / relative
|
||||
if not original.is_file():
|
||||
continue
|
||||
database = staged_profile / "Default" / relative
|
||||
for suffix in ("", "-wal", "-shm"):
|
||||
database.with_name(database.name + suffix).unlink(missing_ok=True)
|
||||
# SQLite backup includes committed WAL data even while the browser is open.
|
||||
with sqlite3.connect(f"{original.as_uri()}?mode=ro", uri=True) as reader:
|
||||
with sqlite3.connect(database) as writer:
|
||||
reader.backup(writer)
|
||||
try:
|
||||
cookies = getattr(browser_cookie3, source.browser)(
|
||||
cookie_file=str(database),
|
||||
key_file=str(staged_profile / "Local State"),
|
||||
)
|
||||
except Exception as err:
|
||||
raise ValueError(
|
||||
"Could not decrypt browser cookies. Run the import as your desktop user on the browser host "
|
||||
"with its OS keychain unlocked. Mounting macOS/Windows browser files into Linux does not provide "
|
||||
"the decryption keys. The existing persona was not replaced.",
|
||||
) from err
|
||||
with sqlite3.connect(database) as connection:
|
||||
connection.row_factory = sqlite3.Row
|
||||
columns = {row[1] for row in connection.execute("PRAGMA table_info(cookies)")}
|
||||
metadata_columns = [
|
||||
name
|
||||
for name in ("host_key", "path", "name", "samesite", "top_frame_site_key", "has_cross_site_ancestor")
|
||||
if name in columns
|
||||
]
|
||||
rows = {
|
||||
(row["host_key"], row["path"], row["name"]): dict(row)
|
||||
for row in connection.execute("SELECT " + ", ".join(metadata_columns) + " FROM cookies")
|
||||
}
|
||||
for cookie in cookies:
|
||||
if cookie.is_expired():
|
||||
continue
|
||||
jar.set_cookie(cookie)
|
||||
item = {
|
||||
"name": cookie.name,
|
||||
"value": cookie.value,
|
||||
"domain": cookie.domain,
|
||||
"path": cookie.path,
|
||||
"secure": bool(cookie.secure),
|
||||
"httpOnly": cookie.has_nonstandard_attr("HTTPOnly"),
|
||||
}
|
||||
if cookie.expires:
|
||||
item["expires"] = cookie.expires
|
||||
row = rows.get((cookie.domain, cookie.path, cookie.name), {})
|
||||
same_site = {0: "None", 1: "Lax", 2: "Strict"}.get(row.get("samesite"))
|
||||
if same_site:
|
||||
item["sameSite"] = same_site
|
||||
if row.get("top_frame_site_key"):
|
||||
item["partitionKey"] = {
|
||||
"topLevelSite": row["top_frame_site_key"],
|
||||
"hasCrossSiteAncestor": bool(row.get("has_cross_site_ancestor", False)),
|
||||
}
|
||||
payload["cookies"].append(item)
|
||||
break
|
||||
jar.save(ignore_discard=True, ignore_expires=False)
|
||||
(output / "auth.json").write_text(json.dumps(payload) + "\n")
|
||||
return payload
|
||||
|
||||
|
||||
def copy_browser_user_data_dir(source_dir: Path, destination_dir: Path) -> None:
|
||||
destination_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.rmtree(destination_dir, ignore_errors=True)
|
||||
@@ -820,27 +611,21 @@ def export_browser_state(
|
||||
chrome_config = chrome_plugin_dir / "chrome" / "config.json"
|
||||
|
||||
env = os.environ.copy()
|
||||
dependency_config = json.loads(chrome_config.read_text())
|
||||
dependency_config["required_binaries"] = [dep for dep in dependency_config["required_binaries"] if dep["name"] != "{CHROME_BINARY}"]
|
||||
dependency_file = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False)
|
||||
with dependency_file:
|
||||
json.dump(dependency_config, dependency_file)
|
||||
try:
|
||||
dependency_env = subprocess.run(
|
||||
[
|
||||
str(Path(sys.executable).with_name("abxpkg")),
|
||||
"env",
|
||||
"--install",
|
||||
"--json",
|
||||
f"--lib={get_config().ABXPKG_LIB_DIR}",
|
||||
f"--deps-from={dependency_file.name}:required_binaries",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
finally:
|
||||
Path(dependency_file.name).unlink()
|
||||
if chrome_binary:
|
||||
env["CHROME_BINARY"] = str(chrome_binary)
|
||||
dependency_env = subprocess.run(
|
||||
[
|
||||
str(Path(sys.executable).with_name("abxpkg")),
|
||||
"env",
|
||||
"--install",
|
||||
"--json",
|
||||
f"--lib={get_config().ABXPKG_LIB_DIR}",
|
||||
f"--deps-from={chrome_config}:required_binaries",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
if dependency_env.returncode != 0:
|
||||
return False, None, dependency_env.stderr.strip() or "abxpkg could not resolve browser export dependencies."
|
||||
try:
|
||||
@@ -854,9 +639,6 @@ def export_browser_state(
|
||||
if not node_projection.is_symlink() or not os.access(node_projection, os.X_OK):
|
||||
return False, None, f"abxpkg did not resolve Node.js into {node_projection}."
|
||||
env.update({str(key): str(value) for key, value in resolved_env.items()})
|
||||
if chrome_binary:
|
||||
env["CHROME_BINARY"] = chrome_binary
|
||||
env["NODE_MODULES_DIR"] = str(abxpkg_lib_dir / "pnpm" / "packages" / "chrome" / "node_modules")
|
||||
env["NODE_BINARY"] = str(node_projection)
|
||||
env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(chrome_plugin_dir)
|
||||
|
||||
@@ -1083,7 +865,7 @@ def _apply_imported_user_agent(persona: Persona, auth_payload: dict | None) -> b
|
||||
if not auth_payload:
|
||||
return False
|
||||
|
||||
user_agent = str(auth_payload.get("user_agent") or "").strip().replace("HeadlessChrome/", "Chrome/")
|
||||
user_agent = str(auth_payload.get("user_agent") or "").strip()
|
||||
if not user_agent:
|
||||
return False
|
||||
|
||||
|
||||
@@ -45,23 +45,17 @@ VOLATILE_PROFILE_DIR_NAMES = {
|
||||
"Code Cache",
|
||||
"GPUCache",
|
||||
"ShaderCache",
|
||||
"Service Worker",
|
||||
"GCM Store",
|
||||
"Crashpad",
|
||||
"BrowserMetrics",
|
||||
"Sessions",
|
||||
"Sessions_Encrypted",
|
||||
}
|
||||
|
||||
VOLATILE_PROFILE_FILE_NAMES = {
|
||||
"BrowserMetrics-spare.pma",
|
||||
"RunningChromeVersion",
|
||||
"DevToolsActivePort",
|
||||
"SingletonCookie",
|
||||
"SingletonLock",
|
||||
"SingletonSocket",
|
||||
"Current Session",
|
||||
"Current Tabs",
|
||||
"Last Session",
|
||||
"Last Tabs",
|
||||
}
|
||||
|
||||
|
||||
@@ -217,7 +211,7 @@ class Persona(ModelWithConfig):
|
||||
cleaned = True
|
||||
|
||||
for path in profile_paths():
|
||||
if path.name not in {"chrome.log", "chrome_debug.log"}:
|
||||
if not path.match("*.log"):
|
||||
continue
|
||||
try:
|
||||
path.unlink()
|
||||
@@ -303,10 +297,6 @@ class Persona(ModelWithConfig):
|
||||
else:
|
||||
runtime_profile_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for filename in ("cookies.txt", "auth.json"):
|
||||
source = self.path / filename
|
||||
if source.is_file():
|
||||
shutil.copy2(source, runtime_root / filename)
|
||||
runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.cleanup_chrome_profile(runtime_profile_dir)
|
||||
|
||||
@@ -323,11 +313,6 @@ class Persona(ModelWithConfig):
|
||||
# derivation centralized in the Chrome plugin helpers.
|
||||
"PERSONAS_DIR": str(runtime_root.parent),
|
||||
"ACTIVE_PERSONA": self.name,
|
||||
**{
|
||||
key: str(runtime_root / filename)
|
||||
for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
|
||||
if (runtime_root / filename).is_file()
|
||||
},
|
||||
}
|
||||
|
||||
def prepare_runtime_for_snapshot(self, snapshot, chrome_binary: str = "") -> dict[str, str]:
|
||||
@@ -344,12 +329,6 @@ class Persona(ModelWithConfig):
|
||||
else:
|
||||
runtime_profile_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for filename in ("cookies.txt", "auth.json"):
|
||||
source = self.runtime_root_for_crawl(snapshot.crawl) / filename
|
||||
if not source.is_file():
|
||||
source = self.path / filename
|
||||
if source.is_file():
|
||||
shutil.copy2(source, runtime_root / filename)
|
||||
runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.cleanup_chrome_profile(runtime_profile_dir)
|
||||
|
||||
@@ -364,11 +343,6 @@ class Persona(ModelWithConfig):
|
||||
# Chrome hooks and ArchiveBox-driven hooks resolve paths the same way.
|
||||
"PERSONAS_DIR": str(runtime_root.parent),
|
||||
"ACTIVE_PERSONA": self.name,
|
||||
**{
|
||||
key: str(runtime_root / filename)
|
||||
for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
|
||||
if (runtime_root / filename).is_file()
|
||||
},
|
||||
}
|
||||
|
||||
def cleanup_runtime_for_crawl(self, crawl) -> None:
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR;
|
||||
const {ensureNodeModuleResolution} = require(path.join(pluginsDir, 'base', 'utils.js'));
|
||||
ensureNodeModuleResolution(module);
|
||||
const installShutdownHandler = require(path.join(pluginsDir, 'base', 'daemon_lifecycle.js')).captureShutdownSignals();
|
||||
const {
|
||||
acquireSessionLock,
|
||||
ensureChromeSession,
|
||||
getChromeSessionOptionsFromConfig,
|
||||
getChromeLaunchPrerequisites,
|
||||
connectToBrowserEndpoint,
|
||||
closeBrowserInChromeSession,
|
||||
} = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
|
||||
|
||||
async function main() {
|
||||
const config = JSON.parse(fs.readFileSync(0, 'utf8'));
|
||||
const outputDir = path.join(path.dirname(config.CHROME_USER_DATA_DIR), '.browser');
|
||||
fs.mkdirSync(outputDir, {recursive: true});
|
||||
const release = await acquireSessionLock(path.join(outputDir, 'launch.lock'));
|
||||
let session;
|
||||
const {puppeteer, binary} = getChromeLaunchPrerequisites();
|
||||
try {
|
||||
session = await ensureChromeSession({
|
||||
...getChromeSessionOptionsFromConfig(config),
|
||||
outputDir, puppeteer, binary,
|
||||
});
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
const browser = await connectToBrowserEndpoint(puppeteer, session.cdpUrl, {defaultViewport: null});
|
||||
installShutdownHandler(async () => {
|
||||
if (!session.reusedExisting) {
|
||||
await closeBrowserInChromeSession({
|
||||
outputDir, puppeteer, cdpUrl: session.cdpUrl, pid: session.pid, processIsLocal: true,
|
||||
});
|
||||
}
|
||||
await browser.disconnect();
|
||||
process.exit(0);
|
||||
});
|
||||
const pages = await browser.pages();
|
||||
const page = pages[0] || await browser.newPage();
|
||||
const windowSession = await page.createCDPSession();
|
||||
const {windowId} = await windowSession.send('Browser.getWindowForTarget');
|
||||
await windowSession.send('Browser.setWindowBounds', {windowId, bounds: {windowState: 'maximized'}});
|
||||
await windowSession.detach();
|
||||
await page.bringToFront();
|
||||
console.log(`Browser ready for persona ${config.ACTIVE_PERSONA || 'Default'}`);
|
||||
if (session.reusedExisting) {
|
||||
await browser.disconnect();
|
||||
return;
|
||||
}
|
||||
await new Promise(resolve => browser.once('disconnected', resolve));
|
||||
}
|
||||
|
||||
main().catch(error => {
|
||||
console.error(error.message);
|
||||
process.exit(1);
|
||||
});
|
||||
+10
-139
@@ -2,9 +2,7 @@ __package__ = "archivebox.plugins"
|
||||
|
||||
from collections.abc import Iterable
|
||||
from functools import lru_cache
|
||||
from importlib import import_module
|
||||
from pathlib import Path, PurePosixPath
|
||||
from types import ModuleType
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol, TypedDict
|
||||
|
||||
from abx_plugins import get_plugins_dir
|
||||
@@ -57,6 +55,10 @@ def get_plugins() -> list[str]:
|
||||
return sorted(get_plugin_catalog())
|
||||
|
||||
|
||||
def get_plugin_models():
|
||||
return get_plugin_catalog().plugins
|
||||
|
||||
|
||||
def get_plugin_name(plugin: str) -> str:
|
||||
"""
|
||||
Get the base plugin name without numeric prefix.
|
||||
@@ -68,141 +70,8 @@ def get_plugin_name(plugin: str) -> str:
|
||||
"""
|
||||
parts = plugin.split("_", 1)
|
||||
if len(parts) == 2 and parts[0].isdigit():
|
||||
plugin = parts[1]
|
||||
return get_archive_result_aliases().get(plugin, plugin)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_archive_result_aliases() -> dict[str, str]:
|
||||
aliases: dict[str, str] = {}
|
||||
for plugin in get_plugin_catalog().values():
|
||||
for alias in plugin.manifest.get("archive_result_aliases") or []:
|
||||
alias = str(alias or "").strip()
|
||||
if alias:
|
||||
aliases[alias] = plugin.name
|
||||
return aliases
|
||||
|
||||
|
||||
def get_archive_result_names(plugin_name: str) -> tuple[str, ...]:
|
||||
"""Return the canonical ArchiveResult name and its declared historic aliases."""
|
||||
canonical = get_plugin_name(plugin_name)
|
||||
aliases = get_archive_result_aliases()
|
||||
return tuple(dict.fromkeys((canonical, plugin_name, *(alias for alias, target in aliases.items() if target == canonical))))
|
||||
|
||||
|
||||
def plugin_card_is_interactive(plugin_name: str) -> bool:
|
||||
plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
|
||||
return bool(plugin and plugin.manifest.get("card_interactive"))
|
||||
|
||||
|
||||
def get_plugin_output_extension_preference(plugin_name: str) -> tuple[tuple[str, ...], ...] | None:
|
||||
plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
|
||||
raw_groups = plugin.manifest.get("output_extension_preference") if plugin else None
|
||||
if not isinstance(raw_groups, list):
|
||||
return None
|
||||
groups = []
|
||||
for raw_group in raw_groups:
|
||||
if not isinstance(raw_group, list):
|
||||
continue
|
||||
group = tuple(str(extension).lower() for extension in raw_group if str(extension).startswith("."))
|
||||
if group:
|
||||
groups.append(group)
|
||||
return tuple(groups) or None
|
||||
|
||||
|
||||
def get_plugin_default_output_path(plugin_name: str) -> str | None:
|
||||
prefix, separator, base_name = plugin_name.partition("_")
|
||||
archive_result_name = base_name if separator and prefix.isdigit() else plugin_name
|
||||
plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
|
||||
raw_path = str(plugin.manifest.get("default_output_path") or "").strip() if plugin else ""
|
||||
path = PurePosixPath(raw_path)
|
||||
if not raw_path or raw_path.startswith("/") or ".." in path.parts or len(path.parts) != 1:
|
||||
return None
|
||||
return f"{archive_result_name}/{raw_path}"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_snapshot_thumbnail_card_order() -> dict[str, int]:
|
||||
"""Return plugin-owned ArchiveResult names eligible for snapshot thumbnails."""
|
||||
ordering: dict[str, int] = {}
|
||||
for plugin in get_plugin_catalog().values():
|
||||
declarations = plugin.manifest.get("snapshot_thumbnail_cards") or []
|
||||
if not isinstance(declarations, list):
|
||||
continue
|
||||
for declaration in declarations:
|
||||
if not isinstance(declaration, dict):
|
||||
continue
|
||||
result_name = str(declaration.get("archive_result_name") or "").strip()
|
||||
order = declaration.get("order")
|
||||
if not result_name or isinstance(order, bool) or not isinstance(order, int):
|
||||
continue
|
||||
ordering[result_name] = order
|
||||
return ordering
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def get_snapshot_role_names(role: str) -> tuple[str, ...]:
|
||||
"""Return ArchiveResult names for a plugin-owned snapshot presentation role."""
|
||||
manifest_key = f"snapshot_{role}"
|
||||
names = []
|
||||
for plugin in get_plugin_catalog().values():
|
||||
if plugin.manifest.get(manifest_key):
|
||||
names.extend(get_archive_result_names(plugin.name))
|
||||
return tuple(dict.fromkeys(names))
|
||||
|
||||
|
||||
def get_snapshot_role_result(results, role: str):
|
||||
"""Select the first successful result declared for a presentation role."""
|
||||
results_by_name = {result.plugin: result for result in results if result.status == "succeeded"}
|
||||
return next((results_by_name[name] for name in get_snapshot_role_names(role) if name in results_by_name), None)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_plugin_presentation_modules() -> tuple[ModuleType, ...]:
|
||||
"""Load optional plugin-owned presentation hooks declared in manifests."""
|
||||
return tuple(module for plugin in get_plugin_catalog().values() if (module := get_plugin_presentation_module(plugin.name)) is not None)
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def get_plugin_presentation_module(plugin_name: str) -> ModuleType | None:
|
||||
"""Load one plugin's optional presentation hooks module."""
|
||||
plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
|
||||
module_name = str(plugin.manifest.get("presentation_module") or "").strip() if plugin else ""
|
||||
return import_module(module_name) if module_name else None
|
||||
|
||||
|
||||
def plugin_has_custom_full_response(plugin_name: str) -> bool:
|
||||
"""Return whether a plugin renders its full template through a response hook."""
|
||||
module = get_plugin_presentation_module(plugin_name)
|
||||
return bool(module and getattr(module, "render_full_response", None))
|
||||
|
||||
|
||||
def get_extra_snapshot_output_cards(outputs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Collect additional cards from plugin-owned presentation hooks."""
|
||||
cards = []
|
||||
for module in get_plugin_presentation_modules():
|
||||
hook = getattr(module, "extra_snapshot_output_cards", None)
|
||||
if hook is not None:
|
||||
cards.extend(hook(outputs) or ())
|
||||
return cards
|
||||
|
||||
|
||||
def serve_plugin_replay_asset(path: str, config, response_class):
|
||||
"""Give plugin presentation hooks the first chance to serve replay assets."""
|
||||
for module in get_plugin_presentation_modules():
|
||||
hook = getattr(module, "serve_replay_asset_response", None)
|
||||
if hook is not None and (response := hook(path, config, response_class)) is not None:
|
||||
return response
|
||||
return None
|
||||
|
||||
|
||||
def render_plugin_full_response(path: str, output_url: str, **kwargs):
|
||||
"""Let plugin presentation hooks render a saved output's full template."""
|
||||
for module in get_plugin_presentation_modules():
|
||||
render = getattr(module, "render_full_response", None)
|
||||
if render is not None and (response := render(path, output_url, **kwargs)) is not None:
|
||||
return response
|
||||
return None
|
||||
return parts[1]
|
||||
return plugin
|
||||
|
||||
|
||||
def get_enabled_plugins(config: ConfigLookup | None = None, **config_kwargs: Any) -> list[str]:
|
||||
@@ -286,10 +155,12 @@ def get_plugin_template(plugin: str, template_name: str, fallback: bool = True)
|
||||
|
||||
Args:
|
||||
plugin: Plugin name (e.g., 'screenshot', '15_singlefile')
|
||||
template_name: A template slot such as 'icon', 'card', or 'full'.
|
||||
template_name: One of 'icon', 'card', 'full'
|
||||
fallback: If True, return default template if plugin template not found
|
||||
"""
|
||||
base_name = get_plugin_name(plugin)
|
||||
if base_name in ("yt-dlp", "youtube-dl"):
|
||||
base_name = "ytdlp"
|
||||
|
||||
catalog = get_plugin_catalog()
|
||||
if base_name in catalog:
|
||||
|
||||
@@ -213,7 +213,7 @@ class PluginConfigFormMixin:
|
||||
"enabled_config_key": enabled_config_key,
|
||||
"description": str(schema.get("description") or "").strip(),
|
||||
"source_url": f"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/{plugin_name}",
|
||||
"docs_url": f"https://plugins.archivebox.io/#{plugin_name}",
|
||||
"docs_url": f"https://archivebox.github.io/abx-plugins/#{plugin_name}",
|
||||
"required_plugins": [str(item) for item in schema.get("required_plugins") or []],
|
||||
"required_binary_links": _build_required_binary_links(
|
||||
schema.get("required_binaries") or [],
|
||||
@@ -276,7 +276,7 @@ class PluginConfigFormMixin:
|
||||
elif is_sensitive:
|
||||
input_type = "password"
|
||||
else:
|
||||
input_value = "" if current_value is None else str(current_value)
|
||||
input_value = "" if is_sensitive else str(current_value)
|
||||
|
||||
return {
|
||||
"key": config_key,
|
||||
@@ -290,7 +290,7 @@ class PluginConfigFormMixin:
|
||||
"default": _jsonish(default_display),
|
||||
"current": "configured"
|
||||
if is_sensitive and current_value
|
||||
else (("" if current_value is None else str(current_value)) if "string" in schema_types else _jsonish(current_value)),
|
||||
else (str(current_value) if "string" in schema_types else _jsonish(current_value)),
|
||||
"current_url": self.plugin_config_binary_urls.get(config_key, "") if str(config_key).endswith("_BINARY") else "",
|
||||
"is_sensitive": is_sensitive,
|
||||
"minimum": prop_schema.get("minimum"),
|
||||
@@ -342,8 +342,6 @@ class PluginConfigFormMixin:
|
||||
continue
|
||||
|
||||
base_value = effective_config.get(config_key, prop_schema.get("default", ""))
|
||||
if base_value is None and coerced_value == "":
|
||||
continue
|
||||
if _same_config_value(coerced_value, base_value):
|
||||
continue
|
||||
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
"""Snapshot output taxonomy. Display preference is independent of hook order."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from archivebox.plugins.discovery import (
|
||||
get_extra_snapshot_output_cards,
|
||||
get_plugin_catalog,
|
||||
get_plugin_name,
|
||||
get_snapshot_thumbnail_card_order,
|
||||
)
|
||||
|
||||
_GROUP_LABELS = (
|
||||
("html", "HTML"),
|
||||
("raster", "Raster"),
|
||||
("article_text", "Article text"),
|
||||
("embedded_media", "Embedded media"),
|
||||
("metadata", "Metadata"),
|
||||
("other", "Other files"),
|
||||
)
|
||||
|
||||
|
||||
def _plugin_presentation() -> dict[str, dict[str, Any]]:
|
||||
presentation = {}
|
||||
for plugin in get_plugin_catalog().values():
|
||||
manifest = plugin.manifest
|
||||
order = manifest.get("snapshot_output_order")
|
||||
presentation[plugin.name] = {
|
||||
"group": manifest.get("snapshot_output_group") or "other",
|
||||
"order": order if isinstance(order, int) and not isinstance(order, bool) else None,
|
||||
"display_name": manifest.get("snapshot_display_name"),
|
||||
"icon_hidden": bool(manifest.get("icon_hidden")),
|
||||
"card_hidden": bool(manifest.get("card_hidden")),
|
||||
}
|
||||
return presentation
|
||||
|
||||
|
||||
PLUGIN_PRESENTATION = _plugin_presentation()
|
||||
OUTPUT_GROUPS = tuple(
|
||||
(
|
||||
group_id,
|
||||
label,
|
||||
tuple(
|
||||
name
|
||||
for name, metadata in sorted(
|
||||
PLUGIN_PRESENTATION.items(),
|
||||
key=lambda item: (item[1]["order"] is None, item[1]["order"] or 0, item[0]),
|
||||
)
|
||||
if metadata["group"] == group_id and metadata["order"] is not None
|
||||
),
|
||||
)
|
||||
for group_id, label in _GROUP_LABELS
|
||||
)
|
||||
UNORDERED_OUTPUTS = {
|
||||
group_id: {name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["group"] == group_id and metadata["order"] is None}
|
||||
for group_id, _label in _GROUP_LABELS
|
||||
}
|
||||
HIDDEN_ICON_PLUGINS = frozenset(name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["icon_hidden"])
|
||||
HIDDEN_CARD_PLUGINS = frozenset(name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["card_hidden"])
|
||||
|
||||
|
||||
def output_group_for_plugin(plugin: str) -> str:
|
||||
"""Return the snapshot-detail stack that contains a plugin output."""
|
||||
plugin = get_plugin_name(plugin)
|
||||
for group_id, _, plugins in OUTPUT_GROUPS:
|
||||
if plugin in plugins or plugin in UNORDERED_OUTPUTS.get(group_id, set()):
|
||||
return group_id
|
||||
return "other"
|
||||
|
||||
|
||||
def plugin_icon_is_hidden(plugin: str) -> bool:
|
||||
return get_plugin_name(plugin) in HIDDEN_ICON_PLUGINS
|
||||
|
||||
|
||||
def plugin_card_is_hidden(plugin: str) -> bool:
|
||||
return get_plugin_name(plugin) in HIDDEN_CARD_PLUGINS
|
||||
|
||||
|
||||
def display_plugin_name(plugin: str) -> str:
|
||||
canonical = get_plugin_name(plugin)
|
||||
return PLUGIN_PRESENTATION.get(canonical, {}).get("display_name") or plugin.replace("_", " ").replace("-", " ").title()
|
||||
|
||||
|
||||
def plugin_output_sizes(results) -> dict[str, int]:
|
||||
"""Count each plugin's files once, even when several hooks report them."""
|
||||
files_by_plugin: dict[str, dict[str, int]] = {}
|
||||
for result in results:
|
||||
files = files_by_plugin.setdefault(result.plugin, {})
|
||||
for path, metadata in result.output_file_map().items():
|
||||
files[path] = result._coerce_output_file_size(metadata.get("size"))
|
||||
return {plugin: sum(files.values()) for plugin, files in files_by_plugin.items() if files}
|
||||
|
||||
|
||||
def order_output_plugins(plugins, sizes: dict[str, int]) -> list[str]:
|
||||
"""Use the detail-card ordering for icon piles, without synthetic cards."""
|
||||
return [output["name"] for output in order_snapshot_outputs([{"name": plugin, "size": sizes.get(plugin, 0)} for plugin in plugins])]
|
||||
|
||||
|
||||
def snapshot_thumbnail_result(results):
|
||||
"""Select the first available card using plugin-owned thumbnail priority."""
|
||||
card_order = get_snapshot_thumbnail_card_order()
|
||||
candidates = []
|
||||
for result in results:
|
||||
if result.plugin not in card_order:
|
||||
continue
|
||||
try:
|
||||
path = result.embed_path()
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
continue
|
||||
if not path:
|
||||
continue
|
||||
candidates.append((card_order[result.plugin], str(result.id), result))
|
||||
return min(candidates, default=(None, None, None))[2]
|
||||
|
||||
|
||||
def order_snapshot_outputs(outputs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Annotate and order available cards; unknown plugins stay visible in Other files."""
|
||||
group_order = {group_id: index for index, (group_id, _, _) in enumerate(OUTPUT_GROUPS)}
|
||||
preferences = {}
|
||||
for group_id, _, plugins in OUTPUT_GROUPS:
|
||||
for rank, plugin in enumerate(plugins):
|
||||
preferences[plugin] = (group_id, rank)
|
||||
for group_id, plugins in UNORDERED_OUTPUTS.items():
|
||||
for plugin in plugins:
|
||||
preferences[plugin] = (group_id, None)
|
||||
|
||||
if cards := get_extra_snapshot_output_cards(outputs):
|
||||
outputs = [*outputs, *cards]
|
||||
|
||||
def sort_key(output):
|
||||
plugin = get_plugin_name(output["name"])
|
||||
declared_group = output.get("output_group")
|
||||
declared_order = output.get("output_order")
|
||||
if declared_group in group_order and isinstance(declared_order, int) and not isinstance(declared_order, bool):
|
||||
group_id, rank = declared_group, declared_order
|
||||
else:
|
||||
group_id, rank = preferences.get(plugin, ("other", None))
|
||||
output["output_group"] = group_id
|
||||
return (group_order[group_id], rank is None, rank if rank is not None else -int(output.get("size") or 0), plugin)
|
||||
|
||||
return sorted(outputs, key=sort_key)
|
||||
Loaded 100 of 352 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user