Compare commits

...
Author SHA1 Message Date
ArchiveBox Release Bot 810a5e4f59 Bump release version to 0.9.35rc559 2026-09-20 11:24:33 +00:00
Nick Sweeting 372d06fe81 Refresh homepage feature cards and mobile share screenshots 2026-09-20 04:24:15 -07:00
ArchiveBox Release Bot 98b3b67f36 Bump release version to 0.9.35rc558 2026-09-20 10:50:58 +00:00
ArchiveBox Release Bot 193b699b83 Bump for abx-dl 1.12.333 2026-09-20 10:50:42 +00:00
ArchiveBox Release Bot 273c5d6f3c Bump release version to 0.9.35rc557 2026-09-20 10:24:52 +00:00
Nick Sweeting 21ed8ea960 Add demo link alongside hero screenshots link 2026-09-20 03:23:44 -07:00
ArchiveBox Release Bot 716178290a Bump release version to 0.9.35rc556 2026-09-20 10:19:00 +00:00
Nick Sweeting 3177596156 Point extension menu name and Safari to the extension site 2026-09-20 03:18:20 -07:00
ArchiveBox Release Bot 0c1d0f561c Bump release version to 0.9.35rc555 2026-09-20 10:16:43 +00:00
Nick Sweeting 746d487390 Document legacy Docker port compatibility in one place 2026-09-20 03:16:26 -07:00
Nick Sweeting be35cb45b0 Merge branch 'dev' of git+ssh://github.com/ArchiveBox/ArchiveBox into dev 2026-09-20 03:16:06 -07:00
ArchiveBox Release Bot afd7576ad8 Bump for abx-dl 1.12.330 2026-09-20 10:15:58 +00:00
Nick Sweeting aba8f1342a Highlight platform download choices for the visitor OS 2026-09-20 03:15:57 -07:00
ArchiveBox Release Bot b50f1e7692 Bump release version to 0.9.35rc554 2026-09-20 10:11:48 +00:00
Nick Sweeting a056d5daf7 Use standard port 5797 in route examples and gallery links 2026-09-20 03:11:28 -07:00
ArchiveBox Release Bot 4384d09e3f Bump release version to 0.9.35rc553 2026-09-20 10:07:41 +00:00
Nick Sweeting 3f3166cfea Show real Android screenshot in Quickstart phone frame 2026-09-20 03:07:20 -07:00
ArchiveBox Release Bot 3eb01ddd25 Bump release version to 0.9.35rc552 2026-09-20 10:00:32 +00:00
Nick Sweeting 4b52eed55a Merge branch 'dev' of git+ssh://github.com/ArchiveBox/ArchiveBox into dev 2026-09-20 02:59:18 -07:00
Nick Sweeting 105c1e793e Merge remote-tracking branch 'origin/dev' into dev 2026-09-20 02:59:06 -07:00
Nick Sweeting f4b9634863 Add pip alternative instructions and fit app CTAs on mobile 2026-09-20 02:59:06 -07:00
ArchiveBox Release Bot 35a2e9c9c6 Bump release version to 0.9.35rc551 2026-09-20 09:57:38 +00:00
Nick Sweeting d76083d26c Polish platform icon colors and app download labels 2026-09-20 02:57:32 -07:00
ArchiveBox Release Bot f5b2f6a030 Bump for abx-dl 1.12.328 2026-09-20 09:56:46 +00:00
ArchiveBox Release Bot 83c83e53c3 Bump release version to 0.9.35rc550 2026-09-20 09:54:52 +00:00
Nick Sweeting 0a4f328fa6 Link Decomposerize from the raw Docker quickstart 2026-09-20 02:54:25 -07:00
Nick Sweeting c4388d2435 Split Quickstart platform tabs and combine Docker install options 2026-09-20 02:53:56 -07:00
ArchiveBox Release Bot 7af15501f8 Bump release version to 0.9.35rc549 2026-09-20 09:42:43 +00:00
Nick Sweeting b9c9daaa65 Improve native anchor positioning with CSS scroll spacing 2026-09-20 02:41:01 -07:00
ArchiveBox Release Bot c3f9c1a9ef Bump release version to 0.9.35rc548 2026-09-20 09:31:15 +00:00
Nick Sweeting e03db54bca Polish shared Apps dropdown with platform and integration badges 2026-09-20 02:30:57 -07:00
ArchiveBox Release Bot 23a5c6bba9 Bump release version to 0.9.35rc547 2026-09-20 08:26:48 +00:00
Nick Sweeting 7a5ad4c99a Scope snapshot caches to effective permissions while preserving artifact caching 2026-09-20 01:26:31 -07:00
ArchiveBox Release Bot 1a93058585 Bump release version to 0.9.35rc546 2026-09-20 08:07:29 +00:00
Nick Sweeting 85033aa622 Show real Linux desktop CI screenshot in quickstart 2026-09-20 01:06:57 -07:00
ArchiveBox Release Bot 922bcc05ba Bump release version to 0.9.35rc545 2026-09-20 08:05:10 +00:00
Nick Sweeting 070b8ae9c5 Reject dot directory names in ArchiveResult uploads 2026-09-20 01:04:34 -07:00
ArchiveBox Release Bot f77ac105a1 Bump release version to 0.9.35rc544 2026-09-20 08:01:41 +00:00
ArchiveBox Release Bot f41f0afa03 Bump for abx-dl 1.12.326 2026-09-20 08:01:20 +00:00
ArchiveBox Release Bot 31062043fc Bump release version to 0.9.35rc543 2026-09-20 07:52:28 +00:00
Nick Sweeting eeb06148ad Serve admin thumbnails without replay auth redirects 2026-09-20 00:52:11 -07:00
ArchiveBox Release Bot 80d3804e56 Bump release version to 0.9.35rc542 2026-09-20 07:43:58 +00:00
Nick Sweeting 3a4696165d Authorize scoped progress from private replay sessions 2026-09-20 00:43:11 -07:00
ArchiveBox Release Bot 28b3bda674 Bump release version to 0.9.35rc541 2026-09-20 07:37:21 +00:00
Nick Sweeting 8de094435e Replace obsolete Electron installers with current downloads 2026-09-20 00:36:54 -07:00
ArchiveBox Release Bot 4c993242dc Bump release version to 0.9.35rc540 2026-09-20 07:35:19 +00:00
Nick Sweeting c3e6f62d0f Link Electron quickstart to desktop setup guide 2026-09-20 00:34:42 -07:00
ArchiveBox Release Bot f141cb841a Bump release version to 0.9.35rc539 2026-09-20 07:27:59 +00:00
Nick Sweeting 10152bce2a Label desktop setup links for the canonical guide 2026-09-20 00:27:36 -07:00
Nick Sweeting d6fe46d327 Prioritize recently queued capture work 2026-09-20 00:27:20 -07:00
ArchiveBox Release Bot a7d4245c36 Bump release version to 0.9.35rc538 2026-09-20 07:26:46 +00:00
Nick Sweeting 8b634f9568 Use electron.archivebox.io as the desktop site canonical URL 2026-09-20 00:26:15 -07:00
ArchiveBox Release Bot 9cddb2b2dc Bump release version to 0.9.35rc537 2026-09-20 07:21:45 +00:00
Nick Sweeting 7e7af50eff Add direct desktop downloads to the quickstart 2026-09-20 00:21:31 -07:00
ArchiveBox Release Bot e019a2b07a Bump release version to 0.9.35rc536 2026-09-20 06:48:10 +00:00
Nick Sweeting 0d2a4bf393 Center snapshot grid dates within each card 2026-09-19 23:47:39 -07:00
ArchiveBox Release Bot 8914d95e83 Bump release version to 0.9.35rc535 2026-09-20 06:37:37 +00:00
Nick Sweeting 0f965972e3 Wait for both Docker server ports 2026-09-19 23:37:22 -07:00
ArchiveBox Release Bot 7a66e8ef6e Bump release version to 0.9.35rc534 2026-09-20 06:26:57 +00:00
Nick Sweeting be857415ad Accept chunked multipart uploads from Safari clients 2026-09-19 23:26:44 -07:00
ArchiveBox Release Bot bfd6b3c2a7 Bump release version to 0.9.35rc533 2026-09-20 06:06:00 +00:00
Nick Sweeting 69667e5cdb Prioritize new submissions and keep live still browser frames visible 2026-09-19 23:05:27 -07:00
ArchiveBox Release Bot fffbdf0226 Bump release version to 0.9.35rc532 2026-09-20 05:58:12 +00:00
ArchiveBox Release Bot d44422fccb Bump for abx-dl 1.12.325 2026-09-20 05:57:58 +00:00
ArchiveBox Release Bot 9c7a3f1f16 Bump release version to 0.9.35rc531 2026-09-20 05:29:59 +00:00
ArchiveBox Release Bot 52328ad841 Bump for abx-dl 1.12.324 2026-09-20 05:29:42 +00:00
ArchiveBox Release Bot 92008db9c4 Bump release version to 0.9.35rc530 2026-09-20 05:14:07 +00:00
Nick Sweeting e2d3b74dcd Keep global runner recovery off historical archive payload storage 2026-09-19 22:13:47 -07:00
Nick Sweeting 0b74322ea3 Require full CI before releases and fix Docker cache permissions 2026-09-19 22:13:03 -07:00
Nick Sweeting 4d888e94d0 Add opt-in resumable filesystem rescan with portable snapshot identities 2026-09-19 22:11:11 -07:00
ArchiveBox Release Bot 43c6dc2ea7 Bump release version to 0.9.35rc529 2026-09-19 06:49:37 +00:00
Nick Sweeting eeb057433a Restore release checks for documentation and setup visibility guidance 2026-09-18 23:49:04 -07:00
ArchiveBox Release Bot f31727e157 Bump release version to 0.9.35rc528 2026-09-19 06:48:12 +00:00
ArchiveBox Release Bot 03c7074f4e Bump for abx-dl 1.12.323 2026-09-19 06:47:51 +00:00
ArchiveBox Release Bot 845822f7ae Bump release version to 0.9.35rc527 2026-09-19 06:40:47 +00:00
Nick Sweeting 592c79a52e Resolve Bash for Docker release candidate port validation 2026-09-18 23:40:22 -07:00
ArchiveBox Release Bot d0e01fe00f Bump release version to 0.9.35rc526 2026-09-19 06:12:07 +00:00
Nick Sweeting e90aa46d73 Migrate ArchiveBox websites and links to custom domains 2026-09-18 23:10:56 -07:00
ArchiveBox Release Bot deddb4d415 Bump release version to 0.9.35rc525 2026-09-19 06:02:20 +00:00
Nick Sweeting 9a6adf146c Improve site social metadata and ArchiveBox icons 2026-09-18 23:01:53 -07:00
ArchiveBox Release Bot 2b568939ec Bump release version to 0.9.35rc524 2026-09-18 23:36:48 +00:00
ArchiveBox Release Bot 8b98009cb0 Bump for abx-dl 1.12.319 2026-09-18 23:36:12 +00:00
ArchiveBox Release Bot 01f2af413a Bump release version to 0.9.35rc523 2026-09-18 23:17:07 +00:00
ArchiveBox Release Bot 27d243742c Bump for abx-dl 1.12.317 2026-09-18 23:15:55 +00:00
ArchiveBox Release Bot de595545ba Bump release version to 0.9.35rc522 2026-09-18 22:57:23 +00:00
ArchiveBox Release Bot 22f0009a79 Bump for abx-dl 1.12.316 2026-09-18 22:56:12 +00:00
ArchiveBox Release Bot ab5458e22f Bump release version to 0.9.35rc521 2026-09-18 22:37:05 +00:00
Nick Sweeting 7eddcd6e3c Preserve legacy Docker port mappings alongside the new default 2026-09-18 15:36:12 -07:00
Nick Sweeting 9b2fc58e5c Use port 5797 consistently 2026-09-18 15:29:02 -07:00
ArchiveBox Release Bot e239984415 Bump release version to 0.9.35rc520 2026-09-18 22:24:39 +00:00
Nick Sweeting a7baeae476 Link snapshot list previews to snapshot details 2026-09-18 15:24:16 -07:00
ArchiveBox Release Bot b4f9633831 Bump release version to 0.9.35rc519 2026-09-18 22:15:06 +00:00
Nick Sweeting 16b76cac0d Remove redundant public index warning from setup wizard 2026-09-18 15:13:55 -07:00
ArchiveBox Release Bot 48ced005a8 Bump release version to 0.9.35rc518 2026-09-18 22:12:54 +00:00
Nick Sweeting 498d7715b2 Add PostgreSQL to homepage integration icons 2026-09-18 15:11:31 -07:00
ArchiveBox Release Bot 736a93cb92 Bump release version to 0.9.35rc517 2026-09-18 22:05:27 +00:00
Nick Sweeting 5c99cdf7c9 Clarify public and private data separation copy 2026-09-18 15:04:59 -07:00
Nick Sweeting 8da338a887 Update homepage archive file access description 2026-09-18 15:03:45 -07:00
ArchiveBox Release Bot 155097ce6f Bump release version to 0.9.35rc516 2026-09-18 22:01:57 +00:00
Nick Sweeting eb9bb069d6 Start macOS quickstart with ArchiveBox app setup wizard 2026-09-18 15:00:08 -07:00
ArchiveBox Release Bot 53523a6a1e Bump release version to 0.9.35rc515 2026-09-18 21:59:37 +00:00
Nick Sweeting ffdc16fc22 Add download icon to macOS quickstart button 2026-09-18 14:58:07 -07:00
ArchiveBox Release Bot 8d14775759 Bump release version to 0.9.35rc514 2026-09-18 21:57:17 +00:00
Nick Sweeting cd5f96c26a Make macOS server download a prominent quickstart button 2026-09-18 14:56:25 -07:00
ArchiveBox Release Bot 877e5f3104 Bump release version to 0.9.35rc513 2026-09-18 21:54:57 +00:00
Nick Sweeting d91ea053a2 Add integration logo stack and TLSNotary authenticity feature 2026-09-18 14:54:41 -07:00
ArchiveBox Release Bot f326097d33 Bump release version to 0.9.35rc512 2026-09-18 21:41:51 +00:00
ArchiveBox Release Bot efa29074f8 Bump for abx-dl 1.12.315 2026-09-18 21:41:30 +00:00
ArchiveBox Release Bot 99fcb02963 Bump release version to 0.9.35rc511 2026-09-18 21:41:16 +00:00
Nick Sweeting e4981957d4 Simplify homepage configuration copy 2026-09-18 14:40:54 -07:00
ArchiveBox Release Bot 8c83947331 Bump release version to 0.9.35rc510 2026-09-18 21:40:50 +00:00
Nick Sweeting 13f9564fa4 Show featured plugin icons beside configuration heading 2026-09-18 14:39:09 -07:00
ArchiveBox Release Bot 4d8b162099 Bump release version to 0.9.35rc509 2026-09-18 21:37:19 +00:00
Nick Sweeting b8d7669c6c Link macOS quickstart artwork to latest Server app download 2026-09-18 14:36:47 -07:00
ArchiveBox Release Bot d08b483c3b Bump release version to 0.9.35rc508 2026-09-18 21:28:21 +00:00
Nick Sweeting f3f7d564e3 Use plus signs in Docker quickstart graphics 2026-09-18 14:27:11 -07:00
Nick Sweeting 38b65fee63 Clarify homepage workflows and link wiki configuration guides 2026-09-18 14:26:44 -07:00
ArchiveBox Release Bot 6580ed01f3 Bump release version to 0.9.35rc507 2026-09-18 21:23:22 +00:00
Nick Sweeting 8cc3d78b3d Illustrate quickstart tabs and add browser extension setup 2026-09-18 14:22:11 -07:00
ArchiveBox Release Bot 1dd74f3ec8 Bump release version to 0.9.35rc506 2026-09-18 21:21:28 +00:00
Nick Sweeting bacd3d053e Link homepage integration items to usage and export guides 2026-09-18 14:20:21 -07:00
ArchiveBox Release Bot 287e792d29 Bump release version to 0.9.35rc505 2026-09-18 21:13:00 +00:00
Nick Sweeting 383adc4e35 Show TLSNotary and crawl options in audience cards 2026-09-18 14:11:21 -07:00
ArchiveBox Release Bot ba740b4568 Bump release version to 0.9.35rc504 2026-09-18 21:09:51 +00:00
Nick Sweeting e7c419ce78 Refine homepage feature headings and card order 2026-09-18 14:08:49 -07:00
Nick Sweeting 363b79e13f Verify OpenCode Home toggles between mounted chats and overview 2026-09-18 14:01:43 -07:00
ArchiveBox Release Bot 05b1c8b829 Bump release version to 0.9.35rc503 2026-09-18 21:00:45 +00:00
ArchiveBox Release Bot cf03d6c8b3 Bump for abx-dl 1.12.314 2026-09-18 20:55:40 +00:00
ArchiveBox Release Bot 6cfdbbd68b Bump release version to 0.9.35rc502 2026-09-18 20:54:41 +00:00
Nick Sweeting 2c0d5584e2 Add compact screenshot thumbnails to homepage cards 2026-09-18 13:54:21 -07:00
ArchiveBox Release Bot 288af543a6 Bump release version to 0.9.35rc501 2026-09-18 20:19:03 +00:00
ArchiveBox Release Bot afeb937db7 Bump for abx-dl 1.12.313 2026-09-18 20:18:01 +00:00
ArchiveBox Release Bot c8a699da0b Bump release version to 0.9.35rc500 2026-09-18 20:08:14 +00:00
Nick Sweeting 6e49cf279f Check OpenCode shutdown releases OAuth and API listeners 2026-09-18 13:07:33 -07:00
ArchiveBox Release Bot 631378a80d Bump release version to 0.9.35rc499 2026-09-18 20:05:50 +00:00
Nick Sweeting 1ef37a02dd Refresh cached homepage styles after mobile layout fix 2026-09-18 13:04:49 -07:00
ArchiveBox Release Bot b618e20fbf Bump release version to 0.9.35rc498 2026-09-18 20:03:23 +00:00
Nick Sweeting 6be2bd85e5 Verify fresh OpenCode drafts promote through real shell submission 2026-09-18 13:02:54 -07:00
Nick Sweeting 8149cc0320 Fix mobile homepage alignment and image overflow 2026-09-18 13:00:44 -07:00
ArchiveBox Release Bot 72b5439137 Bump release version to 0.9.35rc497 2026-09-18 19:45:10 +00:00
Nick Sweeting 8f87aa9e43 Refresh public homepage copy, visuals, and install tabs 2026-09-18 12:44:35 -07:00
ArchiveBox Release Bot e83f2c1808 Bump release version to 0.9.35rc496 2026-09-18 19:30:33 +00:00
ArchiveBox Release Bot a6d56657b2 Bump for abx-dl 1.12.312 2026-09-18 19:30:14 +00:00
ArchiveBox Release Bot 67af2f3a95 Bump release version to 0.9.35rc495 2026-09-18 19:09:45 +00:00
Nick Sweeting b47fc056e3 Install OpenCode only in ArchiveBox application image layers 2026-09-18 12:08:40 -07:00
ArchiveBox Release Bot 007ecaea80 Bump release version to 0.9.35rc494 2026-09-18 19:05:17 +00:00
Nick Sweeting d2eea2e0f1 Allow 900 MiB ArchiveBox images with bundled OpenCode 2026-09-18 12:04:25 -07:00
ArchiveBox Release Bot a88fa72b16 Bump release version to 0.9.35rc493 2026-09-18 19:01:26 +00:00
Nick Sweeting fead132b5c Verify OpenCode provider flows and proxy transports across domain modes 2026-09-18 12:01:10 -07:00
ArchiveBox Release Bot bfef7285f6 Bump release version to 0.9.35rc492 2026-09-18 18:34:44 +00:00
Nick Sweeting 35efcea657 Gate releases on bundled OpenCode and mounted browser tests 2026-09-18 11:34:26 -07:00
ArchiveBox Release Bot 9b58aa30e5 Bump release version to 0.9.35rc491 2026-09-18 18:28:14 +00:00
Nick Sweeting 38dce9b358 Wait for OpenCode draft navigation before checking session links 2026-09-18 11:27:57 -07:00
ArchiveBox Release Bot bc8daf18c0 Bump release version to 0.9.35rc490 2026-09-18 17:52:44 +00:00
ArchiveBox Release Bot 4cd5d170ea Bump for abx-dl 1.12.306 2026-09-18 17:52:28 +00:00
ArchiveBox Release Bot 4ec52a646b Bump release version to 0.9.35rc489 2026-09-18 10:55:18 +00:00
ArchiveBox Release Bot ee78c2c58b Bump for abx-dl 1.12.305 2026-09-18 10:55:03 +00:00
ArchiveBox Release Bot bdfa7f45fc Bump release version to 0.9.35rc488 2026-09-18 10:35:07 +00:00
ArchiveBox Release Bot 1ec5582527 Bump for abx-dl 1.12.304 2026-09-18 10:34:53 +00:00
ArchiveBox Release Bot 75697e4ebf Bump release version to 0.9.35rc487 2026-09-18 10:07:57 +00:00
ArchiveBox Release Bot 7a19792be8 Bump for abx-dl 1.12.302 2026-09-18 10:07:36 +00:00
ArchiveBox Release Bot 601b2db35b Bump release version to 0.9.35rc486 2026-09-18 09:30:14 +00:00
Nick Sweeting 16506ed55a Respect automatic server setup and import private host personas 2026-09-18 02:29:56 -07:00
ArchiveBox Release Bot cc218245c3 Bump release version to 0.9.35rc485 2026-09-18 09:02:15 +00:00
ArchiveBox Release Bot b625cf4958 Bump for abx-dl 1.12.301 2026-09-18 09:01:50 +00:00
ArchiveBox Release Bot 92c0f52324 Bump release version to 0.9.35rc484 2026-09-18 08:43:52 +00:00
Nick Sweeting 880ab87da2 Improve persona import layout and support Chrome Beta profiles 2026-09-18 01:43:19 -07:00
ArchiveBox Release Bot 7e34e43cf5 Bump release version to 0.9.35rc483 2026-09-18 08:31:33 +00:00
Nick Sweeting 4a1e06fda7 Prepare XDG config directory for mounted browser profiles 2026-09-18 01:31:14 -07:00
ArchiveBox Release Bot e1647d8f93 Bump release version to 0.9.35rc482 2026-09-18 08:22:03 +00:00
ArchiveBox Release Bot 0badc70c4d Bump for abx-dl 1.12.300 2026-09-18 08:21:44 +00:00
ArchiveBox Release Bot 9fbddc5b99 Bump release version to 0.9.35rc481 2026-09-18 08:02:08 +00:00
ArchiveBox Release Bot eb7e6f2ba2 Bump for abx-dl 1.12.299 2026-09-18 08:01:47 +00:00
ArchiveBox Release Bot a7eb82ffda Bump release version to 0.9.35rc480 2026-09-18 07:48:43 +00:00
ArchiveBox Release Bot d3f2509c30 Bump for abx-dl 1.12.298 2026-09-18 07:48:16 +00:00
ArchiveBox Release Bot 3e1a31755c Bump release version to 0.9.35rc479 2026-09-18 07:30:56 +00:00
ArchiveBox Release Bot 70bce92bf3 Bump for abx-dl 1.12.297 2026-09-18 07:29:28 +00:00
ArchiveBox Release Bot 80e4a1c4f8 Bump release version to 0.9.35rc478 2026-09-18 06:40:57 +00:00
Nick Sweeting ceb3c04aff Fix explicit DISPLAY handling for Docker noVNC persona setup (#1882)
An explicit `DISPLAY=novnc:0.0` was silently removed by the Docker
entrypoint whenever `xdpyinfo` failed. In the reported `0.9.35rc381`
image, `xdpyinfo` is missing, so `docker compose run --rm` clears the
display even with a healthy noVNC server and Chromium exits with
“Missing X server or $DISPLAY”.

Preserve explicitly supplied `DISPLAY`, including an empty value to opt
out. Probe only the implicit `novnc:0.0` default, retaining automatic
headless fallback when no display was configured. Current dev already
includes `x11-utils`; this change prevents probe failure from overriding
user configuration.

Document the one-off `persona create` / `persona open` Compose flow.
`persona open` resolves Chromium automatically, avoiding the obsolete
`/data/lib/env/bin/chromium` and architecture/version-specific
Playwright paths.

Validation:
- Reproduced persona creation followed by the exact Chromium X11 failure
on the published `0.9.35rc381` image with real, healthy `theasp/novnc`.
- Mounted only the patched entrypoint into that affected image: explicit
DISPLAY survived despite the missing `xdpyinfo`, and the same Chromium
command opened a window visible through noVNC.
- Built the complete branch Docker image using the Dockerfile's default
abx-dl base. Real `docker compose run --rm archivebox persona create
personal` and `persona open personal` succeeded; viewed and controlled
Chromium through noVNC, closed it normally, and reopened in another
one-off container. A bookmark created in the UI persisted across both
runs.
- Added `IMAGE=<built-image> ./bin/validate_docker_display.sh`: six real
Compose checks cover working implicit/explicit displays, unavailable
explicit display, explicit empty opt-out, and implicit/explicit behavior
with noVNC stopped. Passed on the rebuilt image; failed on the unpatched
image at the explicit unavailable-display assertion.
- `bash -n`, `git diff --check`, and pre-commit hooks passed. No mocks
or fake binaries.

Fixes #1872.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the Docker entrypoint silently dropping an explicit `DISPLAY` when
`xdpyinfo` fails, so Chromium can connect to a healthy noVNC server in
images like `0.9.35rc381` where `xdpyinfo` is missing. The entrypoint
previously unset the display on probe failure, causing Chromium to exit
with "Missing X server or $DISPLAY"; now only the implicit `novnc:0.0`
default is probed, an explicit value (including empty to opt out) is
preserved, and headless fallback still applies when no display was
configured. Fixes #1872.

**Validation**
- Adds `bin/validate_docker_display.sh` with six real Compose checks
covering implicit/explicit displays, unavailable display, empty opt-out,
and noVNC stopped behavior.
- Documents the one-off `persona create` / `persona open` Compose flow
and that `persona open` resolves the Chromium binary automatically.

<sup>Written for commit 953f29b65e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1882?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
2026-09-18 02:39:44 -04:00
Nick Sweeting 953f29b65e Preserve explicit Docker DISPLAY for noVNC persona setup 2026-09-17 23:36:08 -07:00
152 changed files with 4090 additions and 845 deletions

No files matched your search

+3 -3
View File
@@ -30,8 +30,8 @@ body:
description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
render: markdown
placeholder: |
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 5797:5797 archivebox/archivebox` in iTerm2
2. Went to the https://127.0.0.1:5797/add/ page in Google Chrome
3. Typed 'https://example.com/xyz' into the 'Add URL' input field
4. Clicked the 'Add+' button
5. Got a 500 error and saw the errors below in terminal
@@ -138,7 +138,7 @@ body:
archivebox:
image: archivebox/archivebox:latest
ports:
- 8000:8000
- 5797:5797
volumes:
- ./data:/data
environment:
+4
View File
@@ -116,6 +116,10 @@
}
],
"docker_binaries": [
{
"name": "{BASH_BINARY}",
"binproviders": "env,brew,apt"
},
{
"name": "{DOCKER_BINARY}",
"binproviders": "env"
+32 -6
View File
@@ -246,6 +246,26 @@ jobs:
if: inputs.push_digests
run: echo ${{ steps.docker_build.outputs.digest }}
- name: Verify OpenCode install with a read-only browser profile
shell: bash
env:
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
run: |
set -Eeuo pipefail
PROFILE_DIR="$(mktemp -d)"
"$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
--volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
"$IMAGE" bash -c '
set -Eeuo pipefail
test -w "$XDG_CONFIG_HOME"
test ! -w "$XDG_CONFIG_HOME/chromium"
# Check the shipped executable before the installer can repair
# an image that accidentally omitted this opt-in dependency.
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
archivebox install opencode
"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
'
- name: Validate exact built image version and commit
if: ${{ !inputs.push_digests }}
shell: bash
@@ -296,8 +316,8 @@ jobs:
MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
fi
COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
LIMIT=$((750 * 1024 * 1024))
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
LIMIT=$((900 * 1024 * 1024))
echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=900 MiB"
(( COMPRESSED_SIZE < LIMIT ))
- name: Resolve real Docker CLI test dependencies through abxpkg
@@ -378,6 +398,12 @@ jobs:
IMAGE="${{ matrix.local_tag }}" \
"$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
- name: Validate Docker port compatibility
shell: bash
env:
IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
run: '"$BASH_BINARY" bin/validate_docker_ports.sh'
- name: Validate Docker CLI add/list and compose behavior
if: inputs.full_tests && matrix.platform == 'linux/amd64'
shell: bash
@@ -401,13 +427,13 @@ jobs:
"$DOCKER_BINARY" compose run --rm archivebox init
"$DOCKER_BINARY" compose up -d --wait
"$DOCKER_BINARY" network connect archivebox_default www-nginx
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/static/admin/js/jquery.init.js')"
[[ "$index_html" == *ArchiveBox* ]]
[[ "$admin_js" == *window.django* ]]
"$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
[[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
"$DOCKER_BINARY" compose down
@@ -423,7 +449,7 @@ jobs:
'${{ runner.temp }}/archivebox-image.tar' \
'${{ steps.docker_build_pr.outputs.digest }}' \
"$LOCAL_IMAGE_ID" \
"$((750 * 1024 * 1024))" <<'PY'
"$((900 * 1024 * 1024))" <<'PY'
from hashlib import sha256
import json
import os
+14
View File
@@ -49,3 +49,17 @@ jobs:
push_digests: true
full_tests: false
secrets: inherit
opencode:
needs: prepare
if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
strategy:
fail-fast: false
matrix:
test_file:
- archivebox/tests/test_opencode_agent.py
- archivebox/tests/test_opencode_browser.py
uses: ./.github/workflows/test-parallel.yml
with:
test_file: ${{ matrix.test_file }}
secrets: inherit
+30 -1
View File
@@ -27,8 +27,11 @@ jobs:
((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
(github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
runs-on: ubuntu-24.04
permissions:
actions: read
contents: read
outputs:
ready: ${{ steps.owner.outputs.ready }}
ready: ${{ steps.verified.outputs.ready }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
@@ -45,6 +48,32 @@ jobs:
TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
[[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
- name: Require successful full CI for this commit
id: verified
if: steps.owner.outputs.ready == 'true'
timeout-minutes: 40
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -Eeuo pipefail
CI_RUN_ID=''
for _ in {1..12}; do
CI_RUN_ID="$(gh run list \
--repo "$GITHUB_REPOSITORY" \
--workflow ci.yml \
--commit "$RELEASE_SHA" \
--event push \
--limit 1 \
--json databaseId \
--jq '.[0].databaseId // empty')"
[[ -n "$CI_RUN_ID" ]] && break
sleep 5
done
[[ -n "$CI_RUN_ID" ]]
gh run watch "$CI_RUN_ID" --repo "$GITHUB_REPOSITORY" --exit-status
echo 'ready=true' >> "$GITHUB_OUTPUT"
python-release:
name: Publish tested Python artifacts
needs: candidate
+1 -1
View File
@@ -43,7 +43,7 @@ jobs:
- name: Build UI screenshot gallery
run: bin/collect_ui_screenshots.sh
env:
BASE_URL: http://archivebox.localhost:8000
BASE_URL: http://archivebox.localhost:5797
- name: Verify gallery was generated from this exact revision
run: |
+11
View File
@@ -2,6 +2,12 @@ name: Parallel Tests
on:
workflow_call:
inputs:
test_file:
description: Run one discovered test file; empty runs the complete matrix
required: false
default: ""
type: string
env:
PYTHONIOENCODING: utf-8
@@ -58,10 +64,15 @@ jobs:
- name: Discover every test exactly once
id: discover
env:
TEST_FILE: ${{ inputs.test_file }}
run: |
set -Eeuo pipefail
mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
test "${#output[@]}" -eq 2
if [[ -n "$TEST_FILE" ]]; then
output[1]="$("$JQ_BINARY" -ce '[.[] | select(.paths == [env.TEST_FILE])] | select(length == 1)' <<< "${output[1]}")"
fi
echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
echo "${output[0]}"
"$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
+22 -7
View File
@@ -1,8 +1,10 @@
# syntax=docker/dockerfile:1.7
# Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
# WHY: abx-dl owns Python, Node, Chromium, and extraction dependencies so the
# standalone downloader stays independent of ArchiveBox's server features.
# ArchiveBox-only dependencies (Sonic, OpenCode, supervisor, Django) belong in
# the layers added by this Dockerfile, never in the shared abx-dl base image.
# Build abx-dl first, then point this file at it:
# docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
# --build-context abxbus=../abxbus \
@@ -10,10 +12,10 @@
# --build-context abx-plugins=../abx-plugins \
# -t archivebox/abx-dl:dev
# docker buildx build . -f Dockerfile \
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.289 \
# --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.333 \
# -t archivebox:multistage
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.289
ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.333
FROM archivebox/sonic:1.4.9 AS sonic
FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -250,6 +252,19 @@ RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777
&& echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
&& echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
# The optional AI service belongs to ArchiveBox, not the downloader base image.
# Always bundle its executable while leaving OPENCODE_ENABLED opt-in.
RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=archivebox-opencode-$TARGETARCH \
chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" /var/tmp/abxpkg-cache \
&& chmod 1777 /var/tmp/abxpkg-cache \
&& env ABXPKG_TMP_CACHE_DIR=/var/tmp/abxpkg-cache XDG_CACHE_HOME=/var/tmp/abxpkg-cache \
setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install opencode \
# pnpm includes musl variants that Debian's glibc runtime cannot use.
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
&& find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} +
RUN --network=none setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install opencode'
WORKDIR "$DATA_DIR"
RUN echo "[+] Initializing image collection..." \
&& find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
@@ -282,10 +297,10 @@ RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully
WORKDIR "$DATA_DIR"
VOLUME "$DATA_DIR"
EXPOSE 8000
EXPOSE 5797
HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:5797/health/' | grep -q 'OK'
ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]
CMD ["archivebox", "server", "--init", "0.0.0.0:5797"]
+39 -42
View File
@@ -73,15 +73,15 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
docker compose pull
docker compose up -d --wait # initializes new collections automatically
# open http://admin.archivebox.localhost:8000 to finish setup
# open http://admin.archivebox.localhost:5797 to finish setup
# docker compose run --rm archivebox add 'https://example.com'
# docker compose run --rm archivebox help
<br/>
<br/>
# Option B: Or use it as a plain Docker container:
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
# open http://admin.archivebox.localhost:8000 to finish setup
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
# open http://admin.archivebox.localhost:5797 to finish setup
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
# docker run -it -v $PWD:/data archivebox/archivebox:dev help
<br/>
@@ -93,14 +93,14 @@ archivebox init
archivebox install
# archivebox add 'https://example.com'
# archivebox help
# archivebox server 0.0.0.0:8000
# archivebox server 0.0.0.0:5797
<br/>
<br/>
# Option D: Or use the uv install shortcut for Option C
curl -fsSL 'https://get.archivebox.io' | bash
</code></pre>
<br/>
<sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
<sub>Open <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI and <a href="http://admin.archivebox.localhost:5797"><code>http://admin.archivebox.localhost:5797</code></a> for the admin UI ➡️</sub><br/>
<sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
</details>
<br/>
@@ -129,7 +129,7 @@ curl -fsSL 'https://get.archivebox.io' | bash
- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
- [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
@@ -152,12 +152,12 @@ curl -fsSL 'https://get.archivebox.io' | bash
#### ✳️&nbsp; Easy Setup
<details>
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
<br/>
<ol>
<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
<li>Set up an admin account and generate an API Key.</li>
<li>Use the Web UI or <a href="https://archivebox.github.io/ios-archivebox/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://archivebox.github.io/ios-archivebox/">mobile app</a>, the <a href="https://archivebox.github.io/archivebox-browser-extension/">browser extension</a>, and other integrations to your server using your API Key.</li>
<li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
</ol>
<br/>
</details>
@@ -177,7 +177,7 @@ docker compose pull
<li>Start the server, which initializes a new collection automatically.
<pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
</code></pre></li>
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
<pre lang="bash"><code style="white-space: pre-line">
# run CLI commands inside the server container started above
docker compose exec archivebox archivebox add 'https://example.com'
@@ -198,10 +198,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
</code></pre>
</li>
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
<pre lang="bash"><code style="white-space: pre-line">
# completely optional, CLI can always be used without running a server
# docker exec archivebox archivebox [subcommand] [--help]
@@ -256,8 +256,8 @@ archivebox init # initialize a new collection
archivebox install # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
</code></pre>
</li>
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
# completely optional, CLI can always be used without running a server
# archivebox [subcommand] [--help]
archivebox help
@@ -292,8 +292,8 @@ archivebox add 'https://example.com'
</code></pre>
<br/>
</li>
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
# completely optional, CLI can always be used without running a server
# archivebox [subcommand] [--help]
archivebox help
@@ -325,8 +325,8 @@ archivebox init
archivebox install
</code></pre>
</li>
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
# completely optional, CLI can always be used without running a server
# archivebox [subcommand] [--help]
archivebox help
@@ -365,17 +365,12 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
<br/>
<ol>
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Download a binary release for your OS or build the native app from source<br/>
<ul>
<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
</ul>
</li>
<li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
<li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
</ol>
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
<br/>
<i>✨ Alpha (contributors wanted!)</i>: for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
<a href="https://electron.archivebox.io/">Desktop app guide</a> · <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
<br/>
</details>
@@ -544,16 +539,16 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html' # inspect snapshot da
<summary><b>🖥&nbsp; Web UI & API Usage</b></summary>
<pre lang="bash"><code style="white-space: pre-line">
# Start the server on bare metal (uv/apt/brew):
archivebox server 0.0.0.0:8000 # start the server
archivebox server 0.0.0.0:5797 # start the server
<br/>
# Or with Docker Compose:
docker compose up # start the server
<br/>
# Or with a Docker container:
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
</code></pre>
<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
<sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI. ➡️</sup><br/>
<sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
<sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
<br/><br/>
@@ -570,18 +565,20 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
# restart the server to apply any config changes
</code></pre>
Snapshot caching follows each snapshot's saved `PERMISSIONS`, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses `Cache-Control: private` so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.
</details>
<br/>
<details>
<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
<br/>
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://archivebox.github.io/ios-archivebox/)
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://archivebox.github.io/ios-archivebox/)
- 🧩 [ArchiveBox Browser Extension](https://archivebox.github.io/archivebox-browser-extension/)
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://github.com/ArchiveBox/electron-archivebox)
- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
- 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
<br/>
</details>
@@ -641,7 +638,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>
<i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
- <img src="https://archivebox.github.io/ios-archivebox/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
- <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://app.archivebox.io/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
<i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
@@ -1346,10 +1343,10 @@ archivebox init
archivebox install # detect and install all extractor dependencies
# Run the development server w/ autoreloading (but no bg workers)
archivebox server --debug --reload 0.0.0.0:8000
archivebox server --debug --reload 0.0.0.0:5797
# Run the production server (with bg workers but no autoreloading)
archivebox server 0.0.0.0:8000
archivebox server 0.0.0.0:5797
```
#### 2. Option B: Build the docker container and use that for development instead
@@ -1361,10 +1358,10 @@ archivebox server 0.0.0.0:8000
./bin/build_docker.sh dev
# Run the development server w/ autoreloading (but no bg workers)
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
# Run the production server (with bg workers but no autoreloading)
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
# (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
# When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
@@ -1386,11 +1383,11 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
archivebox config --set DEBUG=True
# OR you can run a dev server with DEBUG=True in a few ways:
archivebox server --debug --reload 0.0.0.0:8000
archivebox server --debug --reload 0.0.0.0:5797
# or
archivebox server --debug 0.0.0.0:8000
archivebox server --debug 0.0.0.0:5797
# or
env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application
env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
```
https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
@@ -1425,7 +1422,7 @@ services:
# or with plain Docker:
docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
# or with uv:
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
+2 -1
View File
@@ -247,7 +247,7 @@ def get_archiveresult(request: HttpRequest, archiveresult_id: str):
def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
normalized = str(plugin or "").strip().strip("/")
if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
if normalized in (".", "..") or not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
raise HttpError(400, "Invalid ArchiveResult plugin name")
return normalized
@@ -386,6 +386,7 @@ def _write_archiveresult_files(
existing_output_files: dict[str, dict[str, Any]] | None = None,
allow_empty: bool = False,
) -> dict[str, dict[str, Any]]:
plugin_name = _normalize_uploaded_archiveresult_plugin(plugin_name)
files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
+2 -2
View File
@@ -68,7 +68,7 @@ def help() -> None:
[blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
[grey53]# using Docker:[/grey53]
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
[blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 5797:5797[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
"""
if IN_DOCKER
else ""
@@ -109,7 +109,7 @@ def help() -> None:
[dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
[dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
[dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue] [grey53]# Start the Web UI / API server[/grey53]
[dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:5797[/blue] [grey53]# Start the Web UI / API server[/grey53]
"""
print(
Panel(
+1 -1
View File
@@ -134,7 +134,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
print(" > Skipping orphan snapshot import during init.")
print()
print(" [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
print(" archivebox update")
print(" archivebox update --rescan --migrate-only")
print("\n[green]----------------------------------------------------------------------[/green]")
+33 -4
View File
@@ -109,6 +109,7 @@ def create_personas(
profile: str | None = None,
source: str | None = None,
browser_binary: str | None = None,
permissions: str | None = None,
) -> int:
"""
Create Personas from names.
@@ -162,7 +163,10 @@ def create_personas(
rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
continue
persona, created = Persona.objects.get_or_create(name=name)
persona, created = Persona.objects.get_or_create(
name=name,
defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
)
if created:
persona.ensure_dirs()
@@ -441,7 +445,11 @@ def main():
@main.command("create")
@click.argument("names", nargs=-1)
@click.option("--import", "import_from", help="Import from chrome, chromium, brave, edge, a persona directory, or a live CDP URL")
@click.option(
"--import",
"import_from",
help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
)
@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
@click.option(
"--browser-binary",
@@ -449,9 +457,30 @@ def main():
help="Source Chromium browser executable; required for other Chromium-based browsers",
)
@click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
def create_cmd(names: tuple, import_from: str | None, profile: str | None, source: str | None, browser_binary: str | None):
@click.option(
"--permissions",
type=click.Choice(["public", "unlisted", "private"]),
help="Visibility for a newly created persona (existing personas are unchanged)",
)
def create_cmd(
names: tuple,
import_from: str | None,
profile: str | None,
source: str | None,
browser_binary: str | None,
permissions: str | None,
):
"""Create Personas, optionally importing from a browser profile."""
sys.exit(create_personas(names, import_from=import_from, profile=profile, source=source, browser_binary=browser_binary))
sys.exit(
create_personas(
names,
import_from=import_from,
profile=profile,
source=source,
browser_binary=browser_binary,
permissions=permissions,
),
)
@main.command("list")
+7 -7
View File
@@ -46,13 +46,13 @@ def _split_bind_spec(spec: str) -> tuple[str, str]:
"""Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
The empty strings stand in for "not provided"; the caller fills in
defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
defaults. Bracketed IPv6 literals like ``[::1]:5797`` are handled.
"""
spec = (spec or "").strip()
if not spec:
return "", ""
if spec.startswith("["):
# Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
# Bracketed IPv6: ``[::1]`` or ``[::1]:5797``
end = spec.find("]")
if end == -1:
return spec, "" # malformed; let validator reject it
@@ -77,11 +77,11 @@ def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
(normalized to ``127.0.0.1``). Bare hostnames are rejected because the
bind address feeds Daphne, which has to listen on a numeric address;
public hostnames belong in ``BASE_URL`` instead. Empty values fall back
to ``127.0.0.1`` / ``8000``.
to ``127.0.0.1`` / ``5797``.
"""
raw_host, raw_port = _split_bind_spec(spec)
host = raw_host.strip()
port = (raw_port or "").strip() or "8000"
port = (raw_port or "").strip() or "5797"
if host == "" or host.lower() == "localhost":
host = "127.0.0.1"
@@ -198,7 +198,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
"[yellow] routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
)
print(
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
)
if config.USES_SUBDOMAIN_ROUTING:
print(
@@ -221,7 +221,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
"[yellow] a reverse proxy / ingress / public hostname, e.g.[/yellow]",
)
print(
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
"[yellow] BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
)
print()
@@ -254,7 +254,7 @@ def server(
print()
# First non-empty positional arg is the bind spec; otherwise inherit from
# config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
# config (which defaults to "127.0.0.1:5797"). _parse_and_validate_bind_spec
# hard-errors on hostnames so the rest of the server can assume a numeric
# bind host.
bind_spec = next((arg for arg in runserver_args if arg), "")
+7
View File
@@ -44,6 +44,7 @@ SNAPSHOT_LIST_CHUNK_SIZE = 5000
def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
from archivebox.config import VERSION
from archivebox.config.common import redact_sensitive_config
from archivebox.core.models import SnapshotTag
fields = (
@@ -53,6 +54,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
"title",
"bookmarked_at",
"created_at",
"modified_at",
"config",
"notes",
"timestamp",
"depth",
"status",
@@ -80,6 +84,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
"tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
"bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
"created_at": row["created_at"].isoformat() if row["created_at"] else None,
"modified_at": row["modified_at"].isoformat() if row["modified_at"] else None,
"config": redact_sensitive_config(row["config"]),
"notes": row["notes"],
"timestamp": row["timestamp"],
"depth": row["depth"],
"status": row["status"],
+103 -37
View File
@@ -7,6 +7,7 @@ import os
import asyncio
import shlex
import time
from itertools import islice
from typing import TYPE_CHECKING, Any
from collections.abc import Iterable
@@ -52,17 +53,20 @@ def _get_search_indexing_plugins() -> list[str]:
def _build_filtered_snapshots_queryset(
**kwargs,
):
from archivebox.core.models import Snapshot
from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
limit = kwargs.pop("limit", None)
reverse = kwargs.pop("reverse", False)
snapshots = build_snapshot_queryset(**kwargs)
snapshots = snapshots.order_by(kwargs.get("sort") or "-created_at", "-id")
if reverse:
snapshots = snapshots.reverse()
if kwargs.get("resume"):
snapshots = snapshots.filter(timestamp__lte=kwargs["resume"])
snapshots = snapshots.filter(**{"timestamp__gte" if reverse else "timestamp__lte": kwargs["resume"]})
snapshots = snapshots.select_related("crawl")
if limit is not None and limit > 0:
snapshot_ids = list(snapshots.values_list("id", flat=True)[:limit])
snapshots = Snapshot.objects.filter(id__in=snapshot_ids).select_related("crawl")
snapshots = snapshots.filter(id__in=snapshot_ids)
return snapshots
@@ -73,6 +77,7 @@ def reindex_snapshots(
search_plugins: list[str],
batch_size: int,
collect_ids: bool = False,
reverse: bool = False,
wait_for_turn=None,
) -> dict[str, Any]:
from archivebox.cli.archivebox_extract import run_plugins
@@ -94,16 +99,15 @@ def reindex_snapshots(
plugin=plugin_name,
status__in=completed_statuses,
)
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False).order_by("id")
after_id = None
candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False)
candidates = candidates.order_by("created_at", "id") if reverse else candidates.order_by("-created_at", "-id")
remaining = candidates.select_related(None).only("id", "timestamp", "created_at").paged_iterator(chunk_size=batch_size)
while True:
if wait_for_turn:
wait_for_turn()
page = candidates.filter(id__gt=after_id) if after_id is not None else candidates
batch = list(page.select_related(None).only("id", "timestamp")[:batch_size])
batch = list(islice(remaining, batch_size))
if not batch:
break
after_id = batch[-1].id
records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
stats["processed"] += len(batch)
stats["requested"] += len(batch)
@@ -161,18 +165,18 @@ def update(
continuous: bool = False,
index_only: bool = False,
migrate_only: bool = False,
rescan: bool = False,
reverse: bool = False,
stop_daemon_stack: bool = True,
) -> None:
"""
Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
Update the database without forcing a filesystem scan by default.
Three-phase operation (without filters):
- Phase 1: Drain legacy archive/ directories into the current layout
- Phase 2: Select only stale fs_version rows through the indexed column
- Phase 3: Run queued snapshot-level filesystem maintenance until idle
With filters: Only phase 2 (DB query), no filesystem operations.
Without filters: All phases (full update).
--migrate-only: eagerly finish pending filesystem migrations for known rows.
--index-only: reconcile known snapshot assets/metadata and backfill search.
--rescan: discover filesystem orphans first, then repair known snapshots.
Explicit scans default to newest first; --reverse selects oldest first.
Filters select existing database snapshots for targeted updates.
"""
from rich import print
@@ -248,12 +252,28 @@ def update(
exit_code = 0
try:
if rescan and (is_filtered_update or resume or continuous):
raise click.UsageError("--rescan cannot be combined with filters, --resume, or --continuous; rerun it to resume automatically.")
if rescan:
migrate_only = True # Discovery must not schedule new captures.
wait_for_turn()
with foreground_shutdown_signals(), foreground_parent_watchdog():
while True:
do_migrate = migrate_only or not index_only
do_index = index_only or not migrate_only
if rescan:
from archivebox.cli.rescan import rescan_snapshots
rescan_snapshots(
wait_for_turn=wait_for_turn,
reverse=reverse,
scan_legacy=lambda: drain_old_archive_dirs(batch_size=batch_size, reverse=reverse),
)
do_migrate = migrate_only or is_filtered_update and not index_only
do_index = index_only
if not do_migrate and not do_index:
print(
"[*] Collection ready. Filesystem migrations remain lazy; use --migrate-only, --index-only, or --rescan for explicit scans.",
)
if do_migrate:
if (
@@ -285,7 +305,8 @@ def update(
after=after,
resume=resume,
batch_size=batch_size,
queue_for_archiving=True,
queue_for_archiving=not migrate_only,
reverse=reverse,
wait_for_turn=wait_for_turn,
)
print_stats(stats)
@@ -293,17 +314,13 @@ def update(
else:
stats_combined = {"phase1": {}, "phase2": {}}
print("[*] Phase 1: Draining old archive/ directories (0.8.x → 0.9.x migration)...")
stats_combined["phase1"] = drain_old_archive_dirs(
resume_from=resume,
batch_size=batch_size,
)
print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
stats_combined["phase2"] = process_all_db_snapshots(
batch_size=batch_size,
resume=resume,
wait_for_turn=wait_for_turn,
reverse=reverse,
migrate=True,
)
print_combined_stats(stats_combined)
# The due selectors are indexed and cheap when empty, so
@@ -311,7 +328,7 @@ def update(
# whole-table counts merely to decide whether to call it.
print("[*] Phase 3: Running filesystem maintenance until idle...")
if is_filtered_update:
for snapshot_id in sorted(touched_snapshot_ids):
for snapshot_id in sorted(touched_snapshot_ids, reverse=not reverse):
run_scoped_runner("--snapshot-id", snapshot_id)
elif migrate_only:
run_scoped_runner("--maintenance-only")
@@ -319,6 +336,25 @@ def update(
run_scoped_runner()
if do_index:
from archivebox.cli.rescan import reconcile_known_snapshots
snapshots = _build_filtered_snapshots_queryset(
filter_patterns=filter_patterns,
filter_type=filter_type,
status=status,
url__icontains=url__icontains,
url__istartswith=url__istartswith,
tag=tag,
crawl_id=crawl_id,
limit=limit,
sort=sort,
search=search,
before=before,
after=after,
resume=resume,
reverse=reverse,
)
reconcile_known_snapshots(snapshots, wait_for_turn=wait_for_turn)
search_plugins = _get_search_indexing_plugins()
if not search_plugins:
print("[*] No search indexing plugins are available, nothing to backfill.")
@@ -337,12 +373,14 @@ def update(
before=before,
after=after,
resume=resume,
reverse=reverse,
)
stats = reindex_snapshots(
snapshots,
search_plugins=search_plugins,
batch_size=batch_size,
collect_ids=is_filtered_update,
reverse=reverse,
wait_for_turn=wait_for_turn,
)
print_index_stats(stats)
@@ -358,6 +396,10 @@ def update(
exit_code = 130
exact_resume = err.__dict__.get("archivebox_resume")
resume_cmd = ["archivebox", "update"]
if rescan:
resume_cmd.append("--rescan")
if reverse:
resume_cmd.append("--reverse")
if migrate_only:
resume_cmd.append("--migrate-only")
if index_only:
@@ -405,7 +447,7 @@ def update(
stop_own_supervisord_process()
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500) -> dict[str, int]:
def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500, reverse: bool = False) -> dict[str, int]:
"""
Drain old archive/ directories (0.8.x → 0.9.x migration).
@@ -471,18 +513,18 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
# Scan real legacy directories only; these still contain data to migrate.
entries = [
(e.stat().st_mtime, e.path)
(float(e.name), e.path)
for e in all_entries
if e.is_dir(follow_symlinks=False) and Snapshot.is_legacy_archive_dir(Path(e.path)) # Skip symlinks and 0.9.x roots
]
entries.sort(reverse=True) # Newest first
entries.sort(reverse=not reverse) # Newest first
print(f"[*] Found {len(entries)} old directories to drain")
for mtime, entry_path in entries:
entry_path = Path(entry_path)
# Resume from timestamp if specified
if resume_from and entry_path.name > resume_from:
if resume_from and (entry_path.name < resume_from if reverse else entry_path.name > resume_from):
continue
stats["processed"] += 1
@@ -584,7 +626,13 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
return stats
def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
def process_all_db_snapshots(
batch_size: int = 500,
resume: str | None = None,
wait_for_turn=None,
reverse: bool = False,
migrate: bool = False,
) -> dict[str, int]:
"""Queue only snapshots whose indexed filesystem version is stale."""
from archivebox.core.models import Snapshot
from django.db.models import Q
@@ -601,21 +649,27 @@ def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, w
}
queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
if resume:
queryset = queryset.filter(timestamp__lte=resume)
queryset = queryset.filter(**{"timestamp__gte" if reverse else "timestamp__lte": resume})
initial_now = timezone.now()
rows_to_wake = queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
after_id = None
rows_to_wake = queryset if migrate else queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
ordered = rows_to_wake.order_by("created_at", "id") if reverse else rows_to_wake.order_by("-created_at", "-id")
remaining = ordered.select_related("crawl__created_by").paged_iterator(chunk_size=batch_size)
while True:
if wait_for_turn:
wait_for_turn()
page = rows_to_wake.filter(id__gt=after_id) if after_id is not None else rows_to_wake
batch = list(page.only("id", "fs_version", "modified_at").order_by("id")[:batch_size])
batch = list(islice(remaining, batch_size))
if not batch:
break
after_id = batch[-1].id
now = timezone.now()
updated = 0
for snapshot in batch:
if migrate:
from archivebox.services.runner import run_snapshot_maintenance
if wait_for_turn:
wait_for_turn()
run_snapshot_maintenance(str(snapshot.id))
continue
updated += int(
snapshot.safe_update(
{"retry_at": now, "modified_at": now},
@@ -647,6 +701,7 @@ def process_filtered_snapshots(
resume: str | None,
batch_size: int,
queue_for_archiving: bool = True,
reverse: bool = False,
wait_for_turn=None,
) -> dict[str, Any]:
"""Process snapshots matching filters (DB query only)."""
@@ -669,6 +724,7 @@ def process_filtered_snapshots(
before=before,
after=after,
resume=resume,
reverse=reverse,
)
total = snapshots.count()
@@ -687,6 +743,10 @@ def process_filtered_snapshots(
stats["snapshot_ids"].append(str(snapshot.id))
update_values = {}
updated = 0
if not queue_for_archiving and snapshot.fs_migration_needed:
from archivebox.services.runner import run_snapshot_maintenance
run_snapshot_maintenance(str(snapshot.id))
if queue_for_archiving:
update_values.update(
{
@@ -784,6 +844,12 @@ def print_index_stats(stats: dict[str, Any]) -> None:
@click.option("--continuous", is_flag=True, help="Run continuously as background worker")
@click.option("--index-only", is_flag=True, help="Backfill available search indexes from existing archived content")
@click.option("--migrate-only", is_flag=True, help="Only migrate filesystem and update database/index state")
@click.option("--reverse", is_flag=True, help="Process oldest first instead of newest first for explicit scans")
@click.option(
"--rescan",
is_flag=True,
help="Scan all snapshot directories for orphans and metadata repairs (slow, opt-in, safe to restart)",
)
@snapshot_filter_options(default_filter_type="exact")
@docstring(update.__doc__)
def main(**kwargs):
+222
View File
@@ -0,0 +1,222 @@
"""Explicit discovery of portable snapshots; no checkpoint or default tree walk."""
import json
import os
import uuid
from itertools import groupby
from pathlib import Path
from django.contrib.auth import get_user_model
from django.core.exceptions import ValidationError
from django.db import IntegrityError
from rich import print
from archivebox.config import CONSTANTS
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.crawls.models import Crawl
from archivebox.misc.util import parse_date
def _directories(parent: Path):
# Names encode dates / UUIDv7 creation time. Newest first within each
# subtree, with bounded memory and no traversal into plugin payloads.
with os.scandir(parent) as entries:
paths = [Path(entry.path) for entry in entries if entry.is_dir(follow_symlinks=False)]
yield from sorted(paths, key=lambda path: path.name, reverse=True)
def _snapshot_directories(reverse=False):
dates = []
for user in _directories(CONSTANTS.USERS_DIR):
root = user / CONSTANTS.SNAPSHOTS_DIR_NAME
if root.is_symlink() or not root.is_dir():
continue
dates.extend(_directories(root))
dates.sort(key=lambda path: path.name, reverse=not reverse)
for _, date_group in groupby(dates, key=lambda path: path.name):
directories = [directory for date in date_group for domain in _directories(date) for directory in _directories(domain)]
yield from sorted(directories, key=lambda path: path.name, reverse=not reverse)
def _fields(record, names):
return {name: record[name] for name in names.split() if name in record}
def _dates(record):
return {name: parse_date(record[name]) for name in ("created_at", "modified_at") if record.get(name)}
def _restore_dates(obj, record):
dates = _dates(record)
if dates:
type(obj).objects.filter(pk=obj.pk).update(**dates)
for name, value in dates.items():
setattr(obj, name, value)
def _reconcile_directory(directory: Path) -> str:
snapshot_id = uuid.UUID(directory.name)
snapshot = Snapshot.objects.select_related("crawl__created_by").filter(pk=snapshot_id).first()
index = directory / CONSTANTS.JSONL_INDEX_FILENAME
if index.is_symlink():
raise ValueError("metadata must not be a symlink")
if not index.exists():
if snapshot is None or Path(snapshot.output_dir) != directory or snapshot.status != "sealed":
raise ValueError("missing index.jsonl; cannot reconstruct an unknown or active snapshot")
snapshot.write_index_jsonl(output_dir=directory)
snapshot.reconcile_filesystem_links()
return "Repaired"
records = [json.loads(line) for line in index.read_text().splitlines() if line.strip()]
if any(not isinstance(record, dict) for record in records):
raise ValueError("metadata records must be JSON objects")
snapshots = [record for record in records if record.get("type") == "Snapshot"]
if len(snapshots) != 1:
raise ValueError("expected exactly one Snapshot record")
record = snapshots[0]
if uuid.UUID(record["id"]) != snapshot_id:
raise ValueError("snapshot ID conflicts with directory name")
if record.get("status") != "sealed":
raise ValueError("snapshot is not sealed; finish the source capture first")
if record.get("fs_version") not in {Snapshot._fs_current_version(), *Snapshot._FS_VERSION_MIGRATION_PATHS}:
raise ValueError("unsupported filesystem version")
crawl_id = uuid.UUID(record["crawl_id"])
owner = directory.parents[3].name
crawl_records = [item for item in records if item.get("type") == "Crawl" and uuid.UUID(item["id"]) == crawl_id]
crawl_record = crawl_records[0] if crawl_records else {}
if crawl_record.get("created_by", owner) != owner:
raise ValueError("crawl owner conflicts with directory path")
crawl = Crawl.objects.select_related("created_by").filter(pk=crawl_id).first()
imported_crawl = crawl is None
if crawl and crawl.created_by.username != owner:
raise ValueError("existing crawl ID belongs to a different owner")
if snapshot and (snapshot.url != record["url"] or snapshot.crawl_id != crawl_id or snapshot.timestamp != record["timestamp"]):
raise ValueError("existing snapshot ID has conflicting identity; refusing to overwrite")
if snapshot and Path(snapshot.output_dir) != directory:
raise ValueError("existing snapshot has a different storage path")
# Validate identities before inserting anything. Never turn an ID collision
# into a new ID, and never overwrite a newer destination row.
# write_index_jsonl retains historical hook records before appending the
# canonical DB rows. The final record for each hook is authoritative.
results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
pending = []
for result in results:
result_id = uuid.UUID(result["id"])
if uuid.UUID(result["snapshot_id"]) != snapshot_id:
raise ValueError("ArchiveResult belongs to another snapshot")
existing = ArchiveResult.objects.filter(pk=result_id).first()
identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
if existing:
if (existing.snapshot_id, existing.plugin, existing.hook_name) != identity:
raise ValueError("existing ArchiveResult ID has conflicting identity")
elif ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin=identity[1], hook_name=identity[2]).exists():
raise ValueError("existing hook has a different ArchiveResult ID")
else:
pending.append(result)
imported = snapshot is None
if imported:
if Snapshot.objects.filter(timestamp=record["timestamp"]).exists():
raise ValueError("timestamp belongs to another snapshot")
if Snapshot.objects.filter(crawl_id=crawl_id, url=record["url"]).exists():
raise ValueError("crawl URL belongs to another snapshot")
user, _ = get_user_model().objects.get_or_create(
username=owner,
defaults={"is_active": False, "password": "!"},
)
if crawl is None:
# Old exports contain only crawl_id. Keep that identity but report
# that absent crawl metadata cannot be reconstructed exactly.
if not crawl_record:
print(f"[yellow] {directory}: no Crawl record; preserving ID with recovered metadata[/yellow]")
crawl = Crawl(
id=crawl_id,
created_by=user,
status="sealed",
retry_at=None,
**_fields(crawl_record, "urls max_depth config tags_str label"),
)
crawl.save(force_insert=True)
_restore_dates(crawl, crawl_record)
snapshot = Snapshot.create_from_directory(directory, crawl=crawl, preserve_identity=True)
if snapshot is None:
raise ValueError("invalid snapshot metadata")
snapshot.status = "sealed"
snapshot.retry_at = None
if "config" not in record:
snapshot.config = {"PERMISSIONS": "private"}
if Path(snapshot.get_storage_path_for_version(Snapshot._fs_current_version())) != directory:
raise ValueError("metadata does not match its owner/date/domain directory")
snapshot.save(force_insert=True)
_restore_dates(snapshot, record)
# Each insert is durable. An interruption after the Snapshot insert is not
# completion: the next pass still imports missing result IDs and tags.
for result in pending:
snapshot._create_archive_result_if_missing(result, {}, preserve_identity=True)
tags_before = set(snapshot.tags.values_list("name", flat=True))
snapshot._merge_tags_from_index(record)
tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
link = Path(snapshot.crawl.output_dir) / "snapshots" / snapshot.extract_domain_from_url(snapshot.url) / str(snapshot.id)
links_ok = link.is_symlink() and link.resolve() == directory.resolve()
if not links_ok:
snapshot.reconcile_filesystem_links()
if snapshot.fs_migration_needed:
snapshot.migrate_filesystem_to_current_version(source_dir=directory)
if imported:
_restore_dates(snapshot, record)
if imported_crawl:
_restore_dates(crawl, crawl_record)
# Leave source metadata untouched, including process/binary records that
# are historical evidence rather than runnable destination process state.
return "Imported" if imported else "Repaired" if pending or tags_changed or not links_ok else "Unchanged"
def reconcile_known_snapshots(snapshots, wait_for_turn=None):
"""Explicit metadata / asset repair; normal updates never call this scan."""
for snapshot in snapshots.filter(status="sealed").paged_iterator():
if wait_for_turn:
wait_for_turn()
if snapshot.fs_migration_needed:
snapshot.migrate_filesystem_to_current_version()
directory = Path(snapshot.output_dir)
if not directory.is_dir():
raise ValueError(f"Missing snapshot directory: {directory}")
_reconcile_directory(directory)
for result in snapshot.archiveresult_set.all():
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
def rescan_snapshots(wait_for_turn=None, reverse=False, scan_legacy=None):
stats = dict.fromkeys(("Imported", "Repaired", "Unchanged", "Errors"), 0)
print("[*] Rescanning snapshot directories (O(N), orphans first; completed records are reused)...")
known = {str(pk) for pk in Snapshot.objects.values_list("pk", flat=True)}
# Two streaming passes avoid enumerating the entire archive before the
# first import. Rows imported before interruption move to the second pass
# next time; only one day's directory names are held in memory at a time.
for existing in (False, True):
if existing and scan_legacy:
scan_legacy()
for directory in _snapshot_directories(reverse=reverse):
if (directory.name.replace("-", "") in known) != existing:
continue
if wait_for_turn:
wait_for_turn()
try:
outcome = _reconcile_directory(directory)
if directory.name.replace("-", "") in known:
for result in ArchiveResult.objects.filter(snapshot_id=uuid.UUID(directory.name)):
result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
stats[outcome] += 1
if outcome != "Unchanged":
print(f" {outcome}: {directory}")
except (OSError, ValueError, KeyError, TypeError, ValidationError, IntegrityError) as err:
stats["Errors"] += 1
print(f"[yellow] Unresolved: {directory}: {err}[/yellow]")
if sum(stats.values()) % 100 == 0:
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
print(" " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
if stats["Errors"]:
raise SystemExit(1)
return stats
+7 -3
View File
@@ -83,12 +83,16 @@ def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
return candidate
listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
if listen_host and listen_host.lower() != "archivebox.localhost:8000":
if listen_host and listen_host.lower() not in {"archivebox.localhost:8000", "archivebox.localhost:5797"}:
return normalize(listen_host)
csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
csrf_origins = [origin for origin in csrf_origins if origin]
nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
nondefault_csrf_origins = [
origin
for origin in csrf_origins
if origin not in {"http://admin.archivebox.localhost:8000", "http://admin.archivebox.localhost:5797"}
]
if len(nondefault_csrf_origins) == 1:
return nondefault_csrf_origins[0]
@@ -314,7 +318,7 @@ class ServerConfig(BaseConfigSet):
)
SECRET_KEY: str = Field(default_factory=lambda: "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789_") for _ in range(50)))
BIND_ADDR: str = Field(default="127.0.0.1:8000")
BIND_ADDR: str = Field(default="127.0.0.1:5797")
BASE_URL: str = Field(default="")
ALLOWED_HOSTS: str = Field(default="*")
CSRF_TRUSTED_ORIGINS: str = Field(default="")
+38 -2
View File
@@ -5,9 +5,10 @@ from functools import lru_cache
from types import SimpleNamespace
from django.contrib import admin, messages
from django.core.exceptions import PermissionDenied
from django.urls import path, reverse
from django.shortcuts import get_object_or_404, redirect
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed, Http404
from django.utils import timezone
from django.utils.html import format_html, format_html_join
from django.utils.safestring import mark_safe
@@ -20,6 +21,7 @@ from archivebox.config.common import get_config
from archivebox.misc.util import htmldecode, urldecode
from archivebox.misc.paginators import AcceleratedPaginator
from archivebox.misc.logging_util import printable_filesize
from archivebox.misc.serve_static import serve_static_with_byterange_support
from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
from archivebox.search.views import admin_snapshot_search_stream_view
from archivebox.core.routes_util import build_snapshot_url, build_web_url
@@ -46,6 +48,12 @@ from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
GLOBAL_CONTEXT = {}
SNAPSHOT_PERMISSION_META = PERMISSIONS_META
GRID_PREVIEW_OUTPUTS = {
("screenshot", "screenshot.png"),
("chrome_extension_screenshot", "screenshot-1.png"),
("chrome_extension_screenshot", "screenshot.png"),
("favicon", "favicon.ico"),
}
@lru_cache(maxsize=1)
@@ -569,6 +577,11 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
custom_urls = [
path("grid/", self.admin_site.admin_view(self.grid_view), name="grid"),
path("search-stream/", self.admin_site.admin_view(self.search_stream_view), name="core_snapshot_search_stream"),
path(
"<path:object_id>/preview/<str:plugin>/<path:filename>",
self.admin_site.admin_view(self.preview_view, cacheable=True),
name="core_snapshot_preview",
),
path("<path:object_id>/redo-failed/", self.admin_site.admin_view(self.redo_failed_view), name="core_snapshot_redo_failed"),
path(
"<path:object_id>/set-permissions/",
@@ -578,6 +591,26 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
]
return custom_urls + urls
def preview_view(self, request, object_id, plugin, filename):
"""Serve grid thumbnails from the authenticated admin origin."""
if (plugin, filename) not in GRID_PREVIEW_OUTPUTS:
raise Http404("Unsupported snapshot preview")
snapshot = get_object_or_404(Snapshot, pk=object_id)
if not self.has_view_or_change_permission(request, snapshot):
raise PermissionDenied
result = snapshot.archiveresult_set.filter(plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED).first()
file_info = (result.output_files or {}).get(filename) if result else None
if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
raise Http404("Snapshot preview does not exist")
output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
request.archivebox_cache_policy = "private"
response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
raise Http404("Snapshot preview is not an image")
response.headers["Vary"] = "Cookie"
response.headers["X-Content-Type-Options"] = "nosniff"
return response
def search_stream_view(self, request):
return admin_snapshot_search_stream_view(self, request)
@@ -1046,7 +1079,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
return None
return format_html(
'<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">',
'<a href="{}" title="Open snapshot details">'
'<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">'
"</a>",
build_web_url(f"/{obj.archive_path_from_db}/index.html", request=self.request, config=self.request.archivebox_config),
preview["img_url"],
preview["img_alt"],
preview["preview_class"],
+15 -2
View File
@@ -7,13 +7,26 @@ For more information on this file, see
https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
"""
from django.core.asgi import get_asgi_application
from django.core.handlers.asgi import ASGIHandler
from archivebox.config.django import setup_django
setup_django(check_db=True)
django_application = get_asgi_application()
class ArchiveBoxASGIHandler(ASGIHandler):
def create_request(self, scope, body_file):
# ASGI already spools the complete body. Django's multipart parser
# otherwise treats chunked uploads without Content-Length as empty.
if not any(name == b"content-length" for name, _value in scope.get("headers", [])):
body_file.seek(0, 2)
length = body_file.tell()
body_file.seek(0)
scope = {**scope, "headers": [*scope.get("headers", []), (b"content-length", str(length).encode("ascii"))]}
return super().create_request(scope, body_file)
django_application = ArchiveBoxASGIHandler()
async def application(scope, receive, send):
+13 -11
View File
@@ -13,6 +13,7 @@ from django.http import HttpResponseForbidden, HttpResponseNotModified
from django.shortcuts import redirect
from django.utils import timezone
from django.utils.http import http_date
from django.utils.cache import patch_cache_control, patch_vary_headers
from archivebox.config import VERSION
from archivebox.config.common import get_config, get_request_config
@@ -123,13 +124,12 @@ def AdminCookieIsolationMiddleware(get_response):
def CacheControlMiddleware(get_response):
snapshot_path_re = re.compile(r"^/[^/]+/\\d{8}/[^/]+/[0-9a-fA-F-]{8,36}/")
static_cache_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
def middleware(request):
response = get_response(request)
if request.path.startswith("/static/"):
if request.path.startswith("/static/") and response.status_code in (200, 304):
rel_path = request.path[len("/static/") :]
static_path = finders.find(rel_path)
if static_path:
@@ -154,15 +154,17 @@ def CacheControlMiddleware(get_response):
response.headers["Last-Modified"] = http_date(mtime)
return response
if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
"Cache-Control",
):
config = request.__dict__.get("archivebox_config")
if config is None:
config = get_config(resolve_plugins=False)
request.archivebox_config = config
policy = "private" if config.PERMISSIONS == "private" else "public"
response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
# Keep artifact freshness/validators, but never let a shared cache store
# private or unlisted snapshots. Auth redirects and errors stay unstored.
snapshot_policy = getattr(request, "archivebox_cache_policy", None)
if snapshot_policy and response.status_code in (200, 206, 304):
if not response.get("Cache-Control"):
response["Cache-Control"] = f"{snapshot_policy}, max-age=60, stale-while-revalidate=300"
if snapshot_policy == "private":
patch_cache_control(response, private=True)
elif snapshot_policy or not response.get("Cache-Control"):
response["Cache-Control"] = "private, no-store"
patch_vary_headers(response, ("Cookie", "Authorization"))
return response
return middleware
+37 -19
View File
@@ -1394,7 +1394,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
Uses full hostname with sanitized special chars.
Examples:
https://example.com:8080 → example.com_8080
https://example.com:5797 → example.com_5797
https://sub.example.com → sub.example.com
file:///path → localhost
data:text/html → data
@@ -1546,7 +1546,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
return cls.objects.select_related("crawl__created_by").filter(url=url, timestamp=timestamp).first()
@classmethod
def create_from_directory(cls, snapshot_dir: Path) -> Optional["Snapshot"]:
def create_from_directory(cls, snapshot_dir: Path, *, crawl=None, preserve_identity: bool = False) -> Optional["Snapshot"]:
"""
Create new Snapshot from orphaned directory.
@@ -1618,26 +1618,25 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
return None
# Ensure uniqueness (reuses existing logic from create_or_update_from_dict)
timestamp = cls._ensure_unique_timestamp(url, timestamp)
if not preserve_identity:
timestamp = cls._ensure_unique_timestamp(url, timestamp)
# Detect version
fs_version = cls._detect_fs_version_from_index(data)
system_user_id = get_or_create_system_user_pk()
catchall_crawl, _ = Crawl.objects.get_or_create(
label="[migration] orphaned snapshots",
defaults={
"urls": f"# Orphaned snapshot: {url}",
"max_depth": 0,
"created_by_id": system_user_id,
},
)
if cls.objects.filter(crawl=catchall_crawl, url=url).exists():
catchall_crawl = crawl
if catchall_crawl is None:
system_user_id = get_or_create_system_user_pk()
catchall_crawl, _ = Crawl.objects.get_or_create(
label="[migration] orphaned snapshots",
defaults={"urls": f"# Orphaned snapshot: {url}", "max_depth": 0, "created_by_id": system_user_id},
)
if not preserve_identity and cls.objects.filter(crawl=catchall_crawl, url=url).exists():
catchall_crawl = Crawl.objects.create(
label=f"[migration] orphaned snapshot {timestamp}",
urls=url,
max_depth=0,
created_by_id=system_user_id,
created_by_id=catchall_crawl.created_by_id,
)
snapshot_kwargs = {
@@ -1647,6 +1646,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
"fs_version": fs_version,
"crawl": catchall_crawl,
}
if preserve_identity:
snapshot_kwargs.update({key: data[key] for key in ("id", "depth", "config", "notes") if key in data})
try:
bookmarked_at = parse_date(data.get("bookmarked_at") or timestamp)
except (TypeError, ValueError, OSError):
@@ -1847,7 +1848,13 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
result_data["plugin"] = result_data.get("plugin") or result_data.get("extractor") or plugin
self._create_archive_result_if_missing(result_data, existing, update_existing=update_existing)
def _create_archive_result_if_missing(self, result_data: dict, existing: dict, update_existing: bool = True):
def _create_archive_result_if_missing(
self,
result_data: dict,
existing: dict,
update_existing: bool = True,
preserve_identity: bool = False,
):
"""Create ArchiveResult if not already in DB."""
from dateutil import parser
from django.db import transaction
@@ -1925,7 +1932,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
# Machine.current() can probe the host and sanitize config. Do that before
# atomic() so the transaction below only covers the two related row writes.
machine = Machine.current() if cmd or pwd else None
machine = Machine.current() if (cmd or pwd) and not preserve_identity else None
with transaction.atomic():
if machine is not None:
process = Process.objects.create(
@@ -1941,6 +1948,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
)
archiveresult = ArchiveResult.objects.create(
**({"id": result_data["id"]} if preserve_identity else {}),
snapshot=self,
plugin=plugin,
hook_name=hook_name,
@@ -1954,6 +1962,10 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
end_ts=end_ts,
process=process,
)
if preserve_identity:
dates = {key: parse_date(result_data[key]) for key in ("created_at", "modified_at") if result_data.get(key)}
if dates:
ArchiveResult.objects.filter(pk=archiveresult.pk).update(**dates)
existing[(plugin, hook_name)] = archiveresult
def write_index_json(self):
@@ -2010,7 +2022,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
# without a corresponding DB row can be retained byte-for-byte.
binaries_seen = set()
processes_seen = set()
records = [self.to_json()]
records = [self.to_json(), self.crawl.to_json()]
for ar in archive_results:
process = ar.process_record
if process and process.binary and process.binary_id not in binaries_seen:
@@ -2837,6 +2849,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
Includes all fields needed to fully reconstruct/identify this snapshot.
"""
from archivebox.config import VERSION
from archivebox.config.common import redact_sensitive_config
archive_size = self.archive_size
@@ -2850,6 +2863,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
"tags": self.tags_str(),
"bookmarked_at": self.bookmarked_at.isoformat() if self.bookmarked_at else None,
"created_at": self.created_at.isoformat() if self.created_at else None,
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
"config": redact_sensitive_config(self.config),
"notes": self.notes,
"timestamp": self.timestamp,
"depth": self.depth,
"status": self.status,
@@ -4009,6 +4025,8 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
"schema_version": VERSION,
"id": str(self.id),
"snapshot_id": str(self.snapshot_id),
"created_at": self.created_at.isoformat() if self.created_at else None,
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
"plugin": self.plugin,
"hook_name": self.hook_name,
"status": self.status,
@@ -4222,7 +4240,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
def output_file_paths(self) -> list[str]:
return list(self.output_file_map().keys())
def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True) -> bool:
def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True, full_scan: bool = False) -> bool:
from abx_dl.output_files import OutputManifest, output_file_from_path
if self.plugin == "title":
@@ -4262,7 +4280,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
add_file(snapshot_dir / raw_output, raw_output, root_relative=True)
plugin_dir = snapshot_dir / self.plugin
if not output_files and plugin_dir.is_dir():
if (full_scan or not output_files) and plugin_dir.is_dir():
output_files = OutputManifest.scan(plugin_dir, containment_root=snapshot_dir).as_mapping()
if not output_files:
+7
View File
@@ -140,6 +140,13 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
else:
orphaned_hook_processes = orphaned_hook_processes.none()
else:
# Global startup recovery must stay database-only. Process.pwd points
# into archive payload storage, which may be a cold FUSE mount; scanning
# every historical unlinked hook here blocks the sole scheduler before
# it can claim newly queued work. Explicit crawl recovery below remains
# available for the work the caller asked to resume.
orphaned_hook_processes = orphaned_hook_processes.none()
for process in orphaned_hook_processes.only(
"id",
"pwd",
+2 -2
View File
@@ -74,7 +74,7 @@ def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwa
had ``CSRF_TRUSTED_ORIGINS=https://archive.example.com`` set (required for
admin login to work). On upgrade, ``BASE_URL`` is the new knob — but it
defaults to empty, and falling through to ``BIND_ADDR`` produces an
unreachable URL like ``http://0.0.0.0:8000``. If the user has exactly one
unreachable URL like ``http://0.0.0.0:5797``. If the user has exactly one
CSRF origin we treat it as the implicit BASE_URL so links/redirects keep
pointing at the public hostname they already configured.
@@ -133,7 +133,7 @@ def canonical_base_host_for_request(request_host: str) -> str:
"""Strip role subdomains and remap loopback hostnames to ``archivebox.localhost``.
Used by the banner suggestion and the in-browser pin endpoint: when the
user is hitting the server on raw ``localhost:9292`` or ``127.0.0.1:9292``
user is hitting the server on raw ``localhost:5797`` or ``127.0.0.1:5797``
we want to suggest the wildcard-friendly ``archivebox.localhost`` family
instead, so the eventual pinned ``BASE_URL`` plays nicely with subdomain
routing without forcing the user to add a /etc/hosts entry.
+2 -2
View File
@@ -87,6 +87,7 @@ DJANGO_OBJECT_ACTIONS_DEFAULT_HTTP_METHOD = "POST"
MIDDLEWARE = [
"archivebox.core.middleware.TimezoneMiddleware",
"django.middleware.security.SecurityMiddleware",
"archivebox.core.middleware.CacheControlMiddleware",
"archivebox.core.middleware.AdminCookieIsolationMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
@@ -97,7 +98,6 @@ MIDDLEWARE = [
"archivebox.core.middleware.ServerSecurityModeMiddleware",
"archivebox.core.middleware.HostRoutingMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"archivebox.core.middleware.CacheControlMiddleware",
# Additional middlewares from plugins (if any)
]
@@ -390,7 +390,7 @@ SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
# get_base_url() also covers deployments that only set CSRF_TRUSTED_ORIGINS (the
# implicit-BASE_URL fallback used on 0.7.x->0.9.x upgrades), so HTTPS hardening
# isn't lost until BASE_URL is migrated. An explicit plain-http base (e.g. local
# http://archivebox.localhost:8000) disables proxy HTTPS handling.
# http://archivebox.localhost:5797) disables proxy HTTPS handling.
BASE_URL_IS_HTTPS = get_base_url(config=CONFIG).strip().lower().startswith("https://")
if BASE_URL_IS_HTTPS or not CONFIG.BASE_URL:
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
+12 -4
View File
@@ -8,6 +8,7 @@ from urllib.parse import quote, urlparse
from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
from django import template
from django.templatetags.static import static
from django.urls import reverse
from django.utils import timezone
from django.utils.html import escape
from django.utils.safestring import mark_safe
@@ -460,9 +461,9 @@ def system_warnings_banner(context):
"""Render the top-of-page warning banner for one of the conditions below,
in priority order (highest first):
1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
issue: until it's pinned, generated URLs can echo any Host the client
sends, and admin/web/api routing has no canonical anchor.
1. ``mode="unconfigured"``— ``BASE_URL`` is empty and the operator has
not selected safe single-domain routing. That mode intentionally uses
each device's connection address, including LAN and Tailscale IPs.
2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
an origin that differs from the configured canonical ``BASE_URL``.
3. ``mode="unsafe"`` — ``SERVER_SECURITY_MODE`` explicitly enables
@@ -489,7 +490,7 @@ def system_warnings_banner(context):
config = get_config(resolve_plugins=False)
if not config.BASE_URL:
if not config.BASE_URL and config.SERVER_SECURITY_MODE != "safe-onedomain-nojsreplay":
return get_setup_wizard_context(context.get("request"), config)
mismatch = get_base_url_mismatch_context(context.get("request"), config)
if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
@@ -613,6 +614,13 @@ def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) ->
return ""
@register.simple_tag(takes_context=True)
def admin_snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
if not snapshot_archiveresult_url(context, snapshot, plugin, filename):
return ""
return reverse("admin:core_snapshot_preview", args=(snapshot.pk, plugin, filename))
@register.simple_tag(takes_context=True)
def snapshot_index_row(context, link) -> str:
snapshot_base = _snapshot_base_url_for_context(context, link)
+23 -2
View File
@@ -23,6 +23,7 @@ from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidde
from django.shortcuts import redirect, render
from django.utils.decorators import method_decorator
from django.utils.html import format_html, format_html_join
from django.utils.cache import patch_cache_control, patch_vary_headers
from django.utils.safestring import mark_safe
from django.views import View
from django.views.decorators.csrf import csrf_exempt, csrf_protect
@@ -327,6 +328,7 @@ class SnapshotView(View):
@staticmethod
def render_live_index(request, snapshot):
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
return render(
template_name="core/snapshot.html",
request=request,
@@ -350,6 +352,7 @@ class SnapshotView(View):
try:
try:
snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
if not can_view_snapshot(request, snapshot):
return _private_snapshot_auth_redirect(request, snapshot, archivefile or "index.html")
canonical_base = snapshot.url_path
@@ -629,6 +632,7 @@ class SnapshotPathView(View):
status=404,
)
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
canonical_base = snapshot.url_path
if date:
requested_base = f"{username}/{date}/{domain or url or ''}"
@@ -891,7 +895,7 @@ def _serve_responses_path(request, responses_root: Path, rel_path: str, show_ind
return None
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, path: str = ""):
rel_path = path or ""
request_config = get_request_config(
request,
@@ -899,6 +903,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
)
request.archivebox_config = request_config
request.archivebox_snapshot_url = snapshot.url
request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
snapshot._runtime_config = request_config
if rel_path.startswith("replay/") or rel_path == "replay":
@@ -911,7 +916,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
# /progress.json on through to the same view used everywhere else. The caller
# passes snapshot_id explicitly in the query string — we don't read it from the
# subdomain (this keeps the endpoint identical across all security modes).
return live_progress_view(request)
return live_progress_view(request, authorized_snapshot=snapshot)
is_directory_request = bool(path) and path.endswith("/")
show_indexes = bool(request.GET.get("files")) or (request_config.USES_SUBDOMAIN_ROUTING and is_directory_request)
@@ -954,6 +959,20 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
raise Http404
def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
response = _build_snapshot_replay_response(request, snapshot, path)
if path != "progress.json" and response.status_code in (200, 206, 304):
if not response.get("Cache-Control"):
policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
response.headers["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
if snapshot.permissions != PERMISSIONS_PUBLIC:
patch_cache_control(response, private=True)
else:
response.headers["Cache-Control"] = "private, no-store"
patch_vary_headers(response, ("Cookie", "Authorization"))
return response
def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str = ""):
request_config = get_request_config(request, resolve_plugins=False)
request.archivebox_config = request_config
@@ -977,6 +996,8 @@ def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str =
root_match = (match[0], match[1]) if match else _latest_responses_root(snapshots, domain)
responses_root = root_match[1] if root_match else None
if root_match:
request.archivebox_cache_policy = "public" if root_match[0].permissions == PERMISSIONS_PUBLIC else "private"
if request_config.USES_SUBDOMAIN_ROUTING:
snapshot = root_match[0] if root_match else (snapshots[0] if requested_root_index and snapshots else None)
if snapshot:
+2
View File
@@ -467,6 +467,8 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
"tags_str": self.tags_str,
"label": self.label,
"created_at": self.created_at.isoformat() if self.created_at else None,
"modified_at": self.modified_at.isoformat() if self.modified_at else None,
"created_by": self.created_by.username,
}
@staticmethod
+1 -1
View File
@@ -88,7 +88,7 @@ def get_vm_info():
try:
# Getting SMBIOS data from sysfs.
# SMBIOS 2.8 present.
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:8000 │
# argo-1 | 2024-10-01T10:40:51Z ERR error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:5797 │
# Handle 0x0100, DMI type 1, 27 bytes
# System Information
# Manufacturer: DigitalOcean
+27 -18
View File
@@ -129,7 +129,7 @@ def _resolve_archive_path(document_root: str | Path, rel_path: str) -> tuple[Pat
def _cache_policy(config=None, **config_kwargs) -> str:
config = config or get_config(resolve_plugins=False, **config_kwargs)
return "private" if config.PERMISSIONS == "private" else "public"
return "public" if config.PERMISSIONS == "public" else "private"
def _format_direntry_timestamp(stat_result: os.stat_result) -> str:
@@ -605,9 +605,15 @@ def _apply_transformed_html_preview_style(html_text: str) -> str:
return f"{TRANSFORMED_HTML_PREVIEW_STYLE}\n{html_text}"
def _set_transformed_response_headers(response, fullpath: Path, statobj: os.stat_result, encoding: str | None, config) -> None:
def _set_transformed_response_headers(
response,
fullpath: Path,
statobj: os.stat_result,
encoding: str | None,
cache_policy: str,
) -> None:
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
if encoding:
response.headers["Content-Encoding"] = encoding
@@ -858,19 +864,22 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
config = request.__dict__.get("archivebox_config")
if config is None:
config = get_config(resolve_plugins=False)
cache_policy = getattr(request, "archivebox_cache_policy", None) or _cache_policy(config=config)
fullpath, path = _resolve_archive_path(document_root, path)
if os.access(fullpath, os.R_OK) and fullpath.is_dir():
if request.GET.get("download") == "zip" and show_indexes:
return _build_directory_zip_response(
response = _build_directory_zip_response(
fullpath,
path,
is_archive_replay=is_archive_replay,
use_async_stream=_is_asgi_request(request),
config=config,
)
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
return response
if show_indexes:
response = _render_directory_index(request, path, fullpath)
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
response.headers["Last-Modified"] = http_date(fullpath.stat().st_mtime)
return _apply_archive_replay_headers(
response,
@@ -899,7 +908,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
if etag in inm_list or etag.strip('"') in [i.strip('"') for i in inm_list]:
not_modified = HttpResponseNotModified()
not_modified.headers["ETag"] = etag
not_modified.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
not_modified.headers["Last-Modified"] = http_date(statobj.st_mtime)
return _apply_archive_replay_headers(
not_modified,
@@ -940,8 +949,10 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
request.META.get("HTTP_IF_MODIFIED_SINCE"),
statobj.st_mtime,
):
not_modified = HttpResponseNotModified()
not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
return _apply_archive_replay_headers(
HttpResponseNotModified(),
not_modified,
fullpath=fullpath,
content_type=content_type,
is_archive_replay=is_archive_replay,
@@ -956,7 +967,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
decoded = fullpath.read_text(encoding="utf-8", errors="replace")
wrapped = _render_text_preview_document(decoded, fullpath.name)
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
return _apply_archive_replay_headers(
response,
fullpath=fullpath,
@@ -976,7 +987,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
raw_image_url = f"{raw_image_url}?{urlencode(list(preview_query.lists()), doseq=True)}"
wrapped = _render_image_preview_document(raw_image_url, fullpath.name)
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
return _apply_archive_replay_headers(
response,
fullpath=fullpath,
@@ -1002,9 +1013,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
last_modified=http_date(statobj.st_mtime),
etag=etag or "",
cache_control=(
f"{_cache_policy(config=config)}, max-age=31536000, immutable"
if etag
else f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
f"{cache_policy}, max-age=31536000, immutable" if etag else f"{cache_policy}, max-age=60, stale-while-revalidate=300"
),
content_encoding=encoding or "",
)
@@ -1042,7 +1051,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
wrapped, _rewrite_count = _rewrite_html_image_sources_for_request(request, wrapped, document_root, rel_path)
wrapped = _apply_transformed_html_preview_style(wrapped)
response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
return _apply_archive_replay_headers(
response,
fullpath=fullpath,
@@ -1053,7 +1062,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
if rewritten_count:
rewritten_html = _apply_transformed_html_preview_style(rewritten_html)
response = HttpResponse(rewritten_html, content_type=content_type)
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
return _apply_archive_replay_headers(
response,
fullpath=fullpath,
@@ -1064,7 +1073,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
if escaped_count and escaped_count > tag_count * 2:
decoded = _apply_transformed_html_preview_style(decoded)
response = HttpResponse(decoded, content_type=content_type)
_set_transformed_response_headers(response, fullpath, statobj, encoding, config)
_set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
return _apply_archive_replay_headers(
response,
fullpath=fullpath,
@@ -1084,13 +1093,13 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
response.headers["Last-Modified"] = http_date(statobj.st_mtime)
if etag:
response.headers["ETag"] = etag
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
response.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
else:
response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
if is_text_like:
response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
if content_type.startswith("image/"):
response.headers["Cache-Control"] = "public, max-age=604800, immutable"
response.headers["Cache-Control"] = f"{cache_policy}, max-age=604800, immutable"
# handle byte-range requests by serving chunk of file
if stat.S_ISREG(statobj.st_mode):
+2 -2
View File
@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
"Persona",
{
"fields": ("name", "created_by", "permissions"),
"classes": ("card", "persona-card-primary"),
"classes": ("card", "wide", "persona-card-primary"),
},
),
(
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
"import_extract_cookies",
"import_capture_storage",
),
"classes": ("card", "wide"),
"classes": ("card", "wide", "persona-card-import"),
},
),
(
+1 -1
View File
@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
label="Autodiscovered profiles",
widget=forms.RadioSelect,
choices=(),
help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
)
import_source = forms.CharField(
required=False,
+28 -7
View File
@@ -18,6 +18,7 @@ import subprocess
import sys
import tempfile
from dataclasses import dataclass, field
from functools import partial
from pathlib import Path
from typing import TYPE_CHECKING
from urllib.parse import urlparse
@@ -31,6 +32,7 @@ if TYPE_CHECKING:
BROWSER_LABELS = {
"chrome": "Google Chrome",
"chrome-beta": "Google Chrome Beta",
"chromium": "Chromium",
"brave": "Brave",
"edge": "Microsoft Edge",
@@ -216,24 +218,29 @@ class PersonaImportResult:
return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
def get_chrome_user_data_dir() -> Path | None:
def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
"""Get the default Chrome user data directory for the current platform."""
system = platform.system()
home = Path.home()
app_name = "Chrome Beta" if beta else "Chrome"
if system == "Darwin":
candidates = [
home / "Library" / "Application Support" / "Google" / "Chrome",
home / "Library" / "Application Support" / "Google" / app_name,
]
elif system == "Linux":
candidates = [
home / ".config" / "google-chrome",
home / ".config" / "chrome",
]
candidates = (
[home / ".config" / "google-chrome-beta"]
if beta
else [
home / ".config" / "google-chrome",
home / ".config" / "chrome",
]
)
elif system == "Windows":
local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
candidates = [
local_app_data / "Google" / "Chrome" / "User Data",
local_app_data / "Google" / app_name / "User Data",
]
else:
candidates = []
@@ -320,6 +327,7 @@ def get_edge_user_data_dir() -> Path | None:
BROWSER_PROFILE_FINDERS = {
"chrome": get_chrome_user_data_dir,
"chrome-beta": partial(get_chrome_user_data_dir, beta=True),
"chromium": get_chromium_user_data_dir,
"brave": get_brave_user_data_dir,
"edge": get_edge_user_data_dir,
@@ -354,6 +362,12 @@ def validate_persona_name(name: str) -> tuple[bool, str]:
def discover_local_browser_profiles() -> list[PersonaImportSource]:
# The macOS companion exports with the host Keychain before creating personas.
# Raw mounts alone cannot decrypt macOS cookies in the Linux container.
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
host_templates = discover_persona_template_profiles(Path(host_profiles))
host_names = {source.source_name for source in host_templates}
return host_templates + [source for source in discover_persona_template_profiles() if source.source_name not in host_names]
discovered: list[PersonaImportSource] = []
for browser, finder in BROWSER_PROFILE_FINDERS.items():
@@ -430,6 +444,11 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
def resolve_browser_import_source(browser: str, profile_dir: str | None = None) -> PersonaImportSource:
browser = browser.lower().strip()
if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
label = BROWSER_LABELS.get(browser, browser)
exported = Path(host_profiles) / f"{label} - {profile_dir or 'Default'}"
if exported.is_dir():
return resolve_custom_import_source(str(exported))
if browser not in BROWSER_PROFILE_FINDERS:
supported = ", ".join(BROWSER_PROFILE_FINDERS)
raise ValueError(f"Unknown browser: {browser}. Supported browsers: {supported}")
@@ -547,6 +566,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
return str(Path(source.browser_binary).expanduser().resolve())
names = {
"chrome": ("Google Chrome", "google-chrome"),
"chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
"chromium": ("Chromium", "chromium"),
"brave": ("Brave Browser", "brave-browser"),
"edge": ("Microsoft Edge", "microsoft-edge"),
@@ -561,6 +581,7 @@ def resolve_source_browser_binary(source: PersonaImportSource) -> str:
elif platform.system() == "Windows":
relative = {
"chrome": "Google/Chrome/Application/chrome.exe",
"chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
"edge": "Microsoft/Edge/Application/msedge.exe",
"brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
"chromium": "Chromium/Application/chrome.exe",
+1 -1
View File
@@ -213,7 +213,7 @@ class PluginConfigFormMixin:
"enabled_config_key": enabled_config_key,
"description": str(schema.get("description") or "").strip(),
"source_url": f"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/{plugin_name}",
"docs_url": f"https://archivebox.github.io/abx-plugins/#{plugin_name}",
"docs_url": f"https://plugins.archivebox.io/#{plugin_name}",
"required_plugins": [str(item) for item in schema.get("required_plugins") or []],
"required_binary_links": _build_required_binary_links(
schema.get("required_binaries") or [],
+1 -1
View File
@@ -17,7 +17,7 @@ from archivebox.config.common import get_live_config_url
from archivebox.config.views import get_environment_binary_url, is_superuser
from archivebox.plugins.discovery import BUILTIN_PLUGINS_DIR, USER_PLUGINS_DIR, discover_plugin_configs, iter_plugin_dirs
ABX_PLUGINS_DOCS_BASE_URL = "https://archivebox.github.io/abx-plugins/"
ABX_PLUGINS_DOCS_BASE_URL = "https://plugins.archivebox.io/"
ABX_PLUGINS_GITHUB_BASE_URL = "https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/"
LIVE_PLUGIN_BASE_URL = "/admin/environment/plugins/"
+13 -3
View File
@@ -43,7 +43,7 @@ def _live_progress_plugin_names() -> tuple[frozenset[str], frozenset[str]]:
return download_plugin_names, indexing_plugin_names
def live_progress_view(request):
def live_progress_view(request, *, authorized_snapshot=None):
"""Simple JSON endpoint for live progress status - used by admin progress monitor."""
try:
from archivebox.core.models import ArchiveResult, Snapshot
@@ -63,7 +63,10 @@ def live_progress_view(request):
except (TypeError, ValueError):
return JsonResponse({"error": "Invalid snapshot_id"}, status=400)
scoped_snapshot = Snapshot.objects.filter(id=snapshot_id_filter).select_related("crawl").first()
if scoped_snapshot is None or not can_view_snapshot(request, scoped_snapshot):
authorized_snapshot_id = str(getattr(authorized_snapshot, "id", "")).replace("-", "")
if scoped_snapshot is None or (
authorized_snapshot_id != str(scoped_snapshot.id).replace("-", "") and not can_view_snapshot(request, scoped_snapshot)
):
return JsonResponse({"error": "Permission denied"}, status=403)
elif crawl_id_filter:
# Crawl-only scope still requires staff: there's no per-crawl ACL helper,
@@ -221,6 +224,8 @@ def live_progress_view(request):
api_base = get_api_base_url(request=request, config=request_config) if scoped_snapshot is not None else ""
running_screencasts = {}
def screencast_frame_url(crawl_id: str, crawl_dir: Path) -> str:
frame_path = crawl_dir / "chrome_screencast" / "latest.jpg"
try:
@@ -229,7 +234,8 @@ def live_progress_view(request):
return ""
if frame_stat.st_size <= 0:
return ""
if now.timestamp() - frame_stat.st_mtime > 15:
started_at = running_screencasts.get(crawl_id)
if started_at is None or frame_stat.st_mtime < started_at.timestamp():
return ""
rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={frame_stat.st_mtime_ns}"
return f"{api_base}{rel}" if api_base else rel
@@ -552,6 +558,10 @@ def live_progress_view(request):
snapshot_id = str(matched_snapshot["id"])
running_worker_ids.add(str(proc["id"]))
_plugin, _label, phase, _hook_name = process_label(proc["cmd"])
if _plugin == "chrome_screencast" and proc["started_at"] and Process(**proc, machine_id=machine_id).is_running:
# CDP sends frames only when the page changes. Keep a still
# frame visible for its live capture, never for a previous run.
running_screencasts[crawl_id] = max(proc["started_at"], running_screencasts.get(crawl_id, proc["started_at"]))
if crawl_id and proc["pid"]:
crawl_process_pids.setdefault(crawl_id, proc["pid"])
if phase == "snapshot" and snapshot_id and proc["pid"]:
+34 -8
View File
@@ -1643,11 +1643,19 @@ def run_install(*, plugin_names: list[str] | None = None) -> None:
asyncio.run(_run_install(plugin_names=plugin_names))
def _first_due_id(queryset):
return queryset.order_by("retry_at", "created_at").values_list("id", flat=True).first()
def _first_due_id(queryset, *, newest_first: bool = False):
ordering = ("-retry_at", "-created_at") if newest_first else ("retry_at", "created_at")
return queryset.order_by(*ordering).values_list("id", flat=True).first()
def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: int, interactive_interrupts: bool) -> bool:
def _run_due_crawl_status(
status: str,
*,
crawl_id: str | None,
lock_seconds: int,
interactive_interrupts: bool,
newest_first: bool = False,
) -> bool:
from archivebox.crawls.models import Crawl
due_crawls = Crawl.objects.filter(
@@ -1656,7 +1664,7 @@ def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: in
)
if crawl_id:
due_crawls = due_crawls.filter(id=crawl_id)
due_crawl_id = _first_due_id(due_crawls)
due_crawl_id = _first_due_id(due_crawls, newest_first=newest_first)
if due_crawl_id is None:
return False
due_crawl = Crawl.objects.filter(id=due_crawl_id).first()
@@ -1670,8 +1678,15 @@ def _run_due_crawl_status(status: str, *, crawl_id: str | None, lock_seconds: in
return True
def _run_due_snapshot_query(queryset, *, lock_seconds: int, interactive_interrupts: bool, runtime_config) -> bool:
due_snapshot_id = _first_due_id(queryset)
def _run_due_snapshot_query(
queryset,
*,
lock_seconds: int,
interactive_interrupts: bool,
runtime_config,
newest_first: bool = False,
) -> bool:
due_snapshot_id = _first_due_id(queryset, newest_first=newest_first)
return _run_due_snapshot_id(
due_snapshot_id,
lock_seconds=lock_seconds,
@@ -1779,20 +1794,31 @@ def run_pending_crawls(
)
if crawl_id:
active_snapshots = active_snapshots.filter(crawl_id=crawl_id)
# Admit new submissions between captures instead of making them
# wait behind every overdue child of an old, resumed crawl.
if _run_due_snapshot_query(
active_snapshots,
active_snapshots.filter(crawl__status=Crawl.StatusChoices.QUEUED),
lock_seconds=60,
interactive_interrupts=interactive_interrupts,
runtime_config=runtime_config,
newest_first=True,
):
continue
if not maintenance_only:
if _run_due_crawl_status(
Crawl.StatusChoices.QUEUED,
crawl_id=crawl_id,
lock_seconds=crawl_claim_lock_seconds,
interactive_interrupts=interactive_interrupts,
newest_first=True,
):
continue
if _run_due_snapshot_query(
active_snapshots,
lock_seconds=60,
interactive_interrupts=interactive_interrupts,
runtime_config=runtime_config,
):
continue
@@ -38,8 +38,8 @@
border-bottom: 1px dashed #64748b;
}
.field-import_mode ul,
.field-import_discovered_profile ul {
.abx-import-mode,
.abx-profile-picker {
margin: 0;
padding: 0;
list-style: none;
@@ -47,20 +47,26 @@
gap: 12px;
}
.field-import_mode ul {
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
.abx-import-mode {
grid-template-columns: repeat(auto-fit, minmax(min(220px, 100%), 1fr));
}
.field-import_discovered_profile ul {
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
max-height: 460px;
overflow: auto;
padding-right: 4px;
.abx-profile-picker {
grid-template-columns: repeat(auto-fit, minmax(min(260px, 100%), 1fr));
}
.field-import_mode li,
.field-import_discovered_profile li {
.abx-import-mode > div,
.abx-profile-picker > div {
margin: 0;
min-width: 0;
}
/* Import rows grow with their contents instead of sharing a fixed flex
height. Avoid overflow-x:hidden turning overflow-y:visible into auto. */
#content-main #persona_form .persona-card-import > .form-row,
#content-main #persona_form .persona-card-import fieldset > div {
flex: none;
overflow: visible;
}
.field-import_mode label,
@@ -68,7 +74,6 @@
display: flex;
gap: 12px;
align-items: flex-start;
min-height: 100%;
padding: 14px 16px;
border-radius: 14px;
border: 1px solid #dbe4ee;
@@ -167,9 +172,14 @@
color: #991b1b;
}
body.model-persona.change-form #content-main form fieldset.persona-card-primary {
flex: 1 1 520px !important;
max-width: calc(66.66% - 10px) !important;
#content-main #persona_form .field-name .flex-container {
max-width: 100% !important;
}
#persona_form .field-name input {
width: 100%;
max-width: none;
box-sizing: border-box;
}
body.model-persona.change-form #content-main form fieldset.persona-card-timestamps {
@@ -257,6 +267,11 @@ document.addEventListener('DOMContentLoaded', function () {
or by pasting in a CDP url from
<code class="persona-import-copy" data-copy-text="chrome://inspect#remote-debugging" title="Copy chrome://inspect#remote-debugging">chrome://inspect#remote-debugging</code> (e.g. <code>http://127.0.0.1:9222</code>).
</p>
<p>
You can also import cookies into this persona from another browser using the
<a href="https://extension.archivebox.io/" target="_blank" rel="noopener noreferrer">ArchiveBox browser extension</a>.
Create the persona here, then select it in the extension's settings to sync cookies from that browser.
</p>
</div>
</section>
{{ block.super }}
@@ -264,9 +279,9 @@ document.addEventListener('DOMContentLoaded', function () {
{% block after_field_sets %}
{{ block.super }}
<section id="plugin-config" class="module aligned persona-plugin-config">
<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">
<h2>Plugin Config</h2>
<p>These typed controls update the same Persona config JSON shown above. Shared config keys stay synced across plugin sections.</p>
{% include "plugins/plugin_config_grid.html" with plugin_groups=adminform.form.plugin_groups %}
</section>
</fieldset>
{% endblock %}
+16 -6
View File
@@ -147,11 +147,21 @@
}
.cards .card .card-meta {
display: grid;
grid-template-columns: 18px minmax(0, 1fr) auto;
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
gap: 8px;
align-items: center;
text-align: center;
}
.cards .card .card-meta .action-select {
justify-self: start;
}
.cards .card .card-meta .card-date {
justify-self: center;
max-width: 100%;
}
.cards .card .card-meta .card-size {
justify-self: end;
}
.cards .card .card-meta .card-date,
.cards .card .card-meta .card-size {
min-width: 0;
@@ -249,16 +259,16 @@
<code class="card-url" title="{{obj.url}}">{{obj.url}}</code>
<div class="card-media">
<a href="{% snapshot_base_url obj %}/index.html" class="card-thumbnail">
{% snapshot_archiveresult_url obj 'screenshot' 'screenshot.png' as screenshot_url %}
{% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot-1.png' as extension_screenshot_1_url %}
{% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot.png' as extension_screenshot_url %}
{% snapshot_archiveresult_url obj 'favicon' 'favicon.ico' as favicon_url %}
{% admin_snapshot_archiveresult_url obj 'screenshot' 'screenshot.png' as screenshot_url %}
{% admin_snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot-1.png' as extension_screenshot_1_url %}
{% admin_snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot.png' as extension_screenshot_url %}
{% admin_snapshot_archiveresult_url obj 'favicon' 'favicon.ico' as favicon_url %}
{% if screenshot_url or extension_screenshot_1_url or extension_screenshot_url or favicon_url %}
<img
src="{% firstof screenshot_url extension_screenshot_1_url extension_screenshot_url favicon_url %}"
alt="{{obj.title|default:'Not yet archived...'}}"
data-fallbacks="{{ extension_screenshot_1_url }},{{ extension_screenshot_url }},{{ favicon_url }}"
onerror="const fallbacks=(this.dataset.fallbacks || '').split(',').filter(Boolean); if (fallbacks.length) { this.dataset.fallbacks=fallbacks.slice(1).join(','); this.src=fallbacks[0]; } else { this.onerror=null; this.closest('.card-thumbnail').classList.add('missing'); this.remove(); }"
onerror="const current=this.src; const fallbacks=(this.dataset.fallbacks || '').split(',').filter(Boolean).map(candidate => new URL(candidate, document.baseURI).href).filter((candidate, index, all) => candidate !== current && all.indexOf(candidate) === index); if (fallbacks.length) { this.dataset.fallbacks=fallbacks.slice(1).join(','); this.src=fallbacks[0]; } else { this.onerror=null; this.closest('.card-thumbnail').classList.add('missing'); this.remove(); }"
/>
{% elif obj.status == 'started' %}
<img src="{% static 'spinner.gif' %}" class="loading-preview" alt="Archiving in progress"/>
+2 -1
View File
@@ -287,7 +287,8 @@
<code>daily</code> - Run once per day<br/>
<code>weekly</code> - Run once per week<br/>
<code>0 */6 * * *</code> - Every 6 hours (cron format)<br/>
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)
<code>0 0 * * 0</code> - Every Sunday at midnight (cron format)<br/>
<a href="https://crontab.guru/" target="_blank" rel="noopener">Cron Format Helper</a>
</div>
</div>
+1 -1
View File
@@ -50,7 +50,7 @@
<div class="abx-question-options" id="archivebox-setup-tls-options">
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="localhost"> <span><b>⭐ Localhost</b><small>No setup needed. HTTPS is not required for local access; all replay features remain available.</small></span></label>
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="none"> <span><b>⚠️ No TLS, private networks only</b><small>Access ArchiveBox directly on a trusted LAN, VPN, or private network without an ingress service. Public servers require HTTPS. In-browser WARC viewing will be disabled unless using <code>localhost</code> or HTTPS.</small></span></label>
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="single"> <span><b>⚠️ Single-domain HTTPS certificate</b><small>Configure one certificate for the <code>BASE_URL</code> hostname in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your platform's ingress UI, then proxy it to ArchiveBox on port <code>8000</code>. This mode is not allowed unless also using Single-domain DNS. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="single"> <span><b>⚠️ Single-domain HTTPS certificate</b><small>Configure one certificate for the <code>BASE_URL</code> hostname in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your platform's ingress UI, then proxy it to ArchiveBox on port <code>5797</code>. This mode is not allowed unless also using Single-domain DNS. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
<label class="abx-question-option"><input type="radio" name="archivebox-tls-mode" value="wildcard"> <span><b>⭐ Wildcard TLS</b><small>Configure one certificate covering both the <code>BASE_URL</code> hostname and <code>*.BASE_URL</code>, normally through DNS-01. Never enable on-demand TLS or request individual certificates for snapshot subdomains. Restart ArchiveBox after saving an HTTPS <code>BASE_URL</code> for the first time.</small></span></label>
</div>
<div class="abx-question-status" id="archivebox-setup-tls-status">Choose how HTTPS will reach ArchiveBox.</div>
+2 -2
View File
@@ -1987,7 +1987,7 @@ tbody .output-link:hover {opacity: 1;}
}
.model-snapshot.change-list .cards .card .card-info.card-meta {
grid-template-columns: 18px minmax(0, 1fr) auto;
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
}
.model-snapshot.change-list .cards .card .card-info.card-meta > a {
@@ -2070,7 +2070,7 @@ tbody .output-link:hover {opacity: 1;}
}
.model-snapshot.change-list .cards .card .card-info.card-meta {
grid-template-columns: 18px minmax(0, 1fr) auto;
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
}
}
+8 -6
View File
@@ -195,9 +195,11 @@
} else {
risks.push('⚠️ UI, API, and archive replay share one origin. CSP disables risky archived scripts, but a CSP or content-type bypass could expose the archive index, other snapshots, saved headers, admin pages, API data, and canonical-host mutations.');
}
risks.push(publicIndexInput.checked
? '⚠️ Anonymous visitors can enumerate public snapshot URLs and titles; saved URLs may contain private share tokens or other secrets.'
: '✅ Anonymous visitors cannot browse the snapshot index.');
if (!publicIndexInput.checked) {
risks.push('✅ Anonymous visitors cannot browse the snapshot index.');
} else {
risks.push('⚠️ Anonymous visitors can browse the public snapshot index; saved URLs may contain private share tokens or other secrets.');
}
risks.push(!controlPlaneEnabled
? '✅ URL submission and other state-changing requests are disabled for everyone in this replay-only mode.'
: (publicAddInput.checked ? '⚠️ Anonymous visitors can submit malicious or private/intranet URLs. A filtering or per-crawl configuration bypass could expose internal content or threaten the server.' : '✅ Only signed-in admins can submit new URLs.'));
@@ -234,7 +236,7 @@
var exampleBaseUrl = desiredScheme + exampleBaseHost;
var exampleWildcardUrl = desiredScheme + exampleWildcardHost;
var exampleAdminUrl = desiredScheme + (dnsMode === 'wildcard' ? 'admin.' + exampleBaseHost : exampleBaseHost) + '/admin/';
var localhostAdminUrl = 'http://admin.archivebox.localhost:8000/admin/';
var localhostAdminUrl = 'http://admin.archivebox.localhost:5797/admin/';
document.getElementById('archivebox-setup-hosting-status').textContent = hosting === 'localhost'
? '❌ Visit ' + localhostAdminUrl + ' from the same machine to continue setup.'
@@ -253,9 +255,9 @@
document.getElementById('archivebox-setup-tls-status').textContent = tlsMode === 'localhost'
? '❌ Visit ' + localhostAdminUrl + ' from this machine to continue setup. No certificate is needed.'
: (tlsMode === 'wildcard'
? '❌ In your ingress provider, proxy to ArchiveBox on port 8000 and configure one browser-trusted certificate covering ' + exampleBaseHost + ' and ' + exampleWildcardHost + ', normally through DNS-01. Never enable on-demand TLS or request individual snapshot certificates. Visit ' + exampleAdminUrl + ' to continue setup.'
? '❌ In your ingress provider, proxy to ArchiveBox on port 5797 and configure one browser-trusted certificate covering ' + exampleBaseHost + ' and ' + exampleWildcardHost + ', normally through DNS-01. Never enable on-demand TLS or request individual snapshot certificates. Visit ' + exampleAdminUrl + ' to continue setup.'
: (tlsMode === 'single'
? '❌ In your ingress provider, proxy to ArchiveBox on port 8000 and configure one browser-trusted certificate for ' + exampleBaseHost + '. Visit ' + exampleAdminUrl + ' to continue setup.'
? '❌ In your ingress provider, proxy to ArchiveBox on port 5797 and configure one browser-trusted certificate for ' + exampleBaseHost + '. Visit ' + exampleAdminUrl + ' to continue setup.'
: (tlsMode === 'none'
? '❌ Expose this ArchiveBox server directly over HTTP without a separate ingress or SSL termination service. Visit ' + exampleAdminUrl + ' to continue setup. In-browser WARC viewing will remain disabled unless browsing through localhost or HTTPS.'
: '❌ Choose an ingress and TLS mode. Your choice will not be changed automatically.')));
+5 -5
View File
@@ -1164,9 +1164,9 @@ def run_python_cwd(
# Server/API Integration Helpers
# =============================================================================
API_TEST_HOST = "api.archivebox.localhost:8000"
ADMIN_TEST_HOST = "admin.archivebox.localhost:8000"
WEB_TEST_HOST = "web.archivebox.localhost:8000"
API_TEST_HOST = "api.archivebox.localhost:5797"
ADMIN_TEST_HOST = "admin.archivebox.localhost:5797"
WEB_TEST_HOST = "web.archivebox.localhost:5797"
@pytest.fixture
@@ -1794,8 +1794,8 @@ def real_archive_with_example(tmp_path_factory, request):
[
"config",
"--set",
"BIND_ADDR=127.0.0.1:8000",
"BASE_URL=http://archivebox.localhost:8000",
"BIND_ADDR=127.0.0.1:5797",
"BASE_URL=http://archivebox.localhost:5797",
"PUBLIC_INDEX=True",
"PUBLIC_ADD_VIEW=True",
],
@@ -67,3 +67,68 @@ def test_archiveresult_patch_upload_finalizes_queued_result(client, api_admin_us
assert result.output_str == "output.html"
assert snapshot.status == Snapshot.StatusChoices.SEALED
assert snapshot.retry_at is not None
@pytest.mark.parametrize("chunked", [False, True])
@pytest.mark.parametrize("method", ["POST", "PATCH"])
def test_live_archiveresult_multipart_upload(tmp_path, chunked, method):
import requests
from archivebox.tests.conftest import (
api_auth_headers,
cli_env,
create_admin_and_token,
get_free_port,
init_archive,
live_api_request,
stop_archivebox_process,
)
from archivebox.tests.test_api_v1_cli_add import start_api_server_without_runner
init_archive(tmp_path)
token = create_admin_and_token(tmp_path)
port = get_free_port()
server = start_api_server_without_runner(tmp_path, cli_env(port=port, server=True), port)
try:
snapshot = live_api_request(
port,
"post",
"/api/v1/core/snapshots",
api_token=token,
json={"url": "https://example.com/multipart", "status": "sealed"},
)
assert snapshot.status_code == 200, snapshot.text
result = live_api_request(
port,
"post",
"/api/v1/core/archiveresults",
api_token=token,
data={"snapshot_id": snapshot.json()["id"], "plugin": "dom", "status": "started"},
)
assert result.status_code == 200, result.text
payload = b"<html>Real multipart upload</html>"
body = encode_multipart(
BOUNDARY,
{
"files": SimpleUploadedFile("output.html", payload, content_type="text/html"),
"output_paths": "output.html",
"status": "succeeded",
"snapshot_id": snapshot.json()["id"],
"plugin": "dom",
},
)
path = "/api/v1/core/archiveresults" if method == "POST" else f"/api/v1/core/archiveresult/{result.json()['id']}"
response = requests.request(
method,
f"http://127.0.0.1:{port}{path}",
data=iter([body]) if chunked else body,
headers={**api_auth_headers(token, port=port), "Content-Type": MULTIPART_CONTENT},
timeout=30,
)
assert response.status_code == 200, response.text
assert response.json()["status"] == "succeeded"
files = list((tmp_path / "archive").glob("users/*/snapshots/*/*/*/dom/output.html"))
assert len(files) == 1
assert files[0].read_bytes() == payload
finally:
stop_archivebox_process(server)
@@ -15,6 +15,79 @@ from archivebox.tests.conftest import api_client_request
pytestmark = pytest.mark.django_db(transaction=True)
@pytest.mark.parametrize("plugin", [".", "..", "/./", "/../", " . ", " ../ "])
def test_archiveresult_upload_rejects_dot_directory_names(client, api_admin_user, api_headers, plugin):
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
snapshot = Snapshot.objects.create(url="https://example.com/invalid-upload-directory", crawl=crawl)
snapshot_dir = snapshot.output_dir
filename = "rejected-upload.txt"
response = client.post(
"/api/v1/core/archiveresults",
{
"snapshot_id": str(snapshot.id),
"plugin": plugin,
"files": SimpleUploadedFile(filename, b"must not be written", content_type="text/plain"),
},
**api_headers,
)
assert response.status_code == 400, response.content
assert response.json()["detail"] == "Invalid ArchiveResult plugin name"
assert not snapshot.archiveresult_set.exists()
assert not (snapshot_dir / filename).exists()
assert not (snapshot_dir.parent / filename).exists()
@pytest.mark.parametrize("plugin", [".", ".."])
def test_archiveresult_patch_rejects_existing_dot_directory_names(client, api_admin_user, api_headers, plugin):
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
snapshot = Snapshot.objects.create(url="https://example.com/invalid-existing-directory", crawl=crawl)
result = ArchiveResult.objects.create(snapshot=snapshot, plugin=plugin)
snapshot_dir = snapshot.output_dir
filename = "rejected-patch.txt"
response = client.patch(
f"/api/v1/core/archiveresult/{result.id}",
encode_multipart(
BOUNDARY,
{"files": SimpleUploadedFile(filename, b"must not be written", content_type="text/plain")},
),
content_type=MULTIPART_CONTENT,
**api_headers,
)
assert response.status_code == 400, response.content
assert response.json()["detail"] == "Invalid ArchiveResult plugin name"
result.refresh_from_db()
assert result.output_files == {}
assert not (snapshot_dir / filename).exists()
assert not (snapshot_dir.parent / filename).exists()
@pytest.mark.parametrize("plugin", ["custom.files", "custom-files_v1"])
def test_archiveresult_upload_accepts_unregistered_directory_names(client, api_admin_user, api_headers, plugin):
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
snapshot = Snapshot.objects.create(url="https://example.com/custom-upload-directory", crawl=crawl)
contents = b"uploaded original file"
response = client.post(
"/api/v1/core/archiveresults",
{
"snapshot_id": str(snapshot.id),
"plugin": plugin,
"files": SimpleUploadedFile("original.txt", contents, content_type="text/plain"),
},
**api_headers,
)
assert response.status_code == 200, response.content
result = snapshot.archiveresult_set.get()
assert result.plugin == plugin
assert result.output_files["original.txt"]["size"] == len(contents)
assert (snapshot.output_dir / plugin / "original.txt").read_bytes() == contents
def test_archiveresult_upload_upserts_by_snapshot_plugin_and_hook(client, api_admin_user, api_headers):
crawl = Crawl.objects.create(urls="https://example.com", created_by=api_admin_user)
snapshot = Snapshot.objects.create(url="https://example.com/unified-result", crawl=crawl)
@@ -11,7 +11,7 @@ pytestmark = pytest.mark.django_db
User = get_user_model()
ADMIN_HOST = "admin.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:5797"
@pytest.fixture
@@ -12,7 +12,7 @@ def test_basic_success_case_request(client, tmp_path, api_admin_user, api_token)
response = client.get(
"/api/v1/core/tags/autocomplete/",
{"q": "api-basic", "api_key": api_token.token},
HTTP_HOST="api.archivebox.localhost:8000",
HTTP_HOST="api.archivebox.localhost:5797",
)
assert response.status_code == 200, response.content
@@ -33,7 +33,7 @@ from .conftest import (
pytestmark = pytest.mark.django_db(transaction=True)
User = get_user_model()
ADMIN_HOST = "admin.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:5797"
@pytest.fixture
+17
View File
@@ -16,6 +16,23 @@ def test_persona_help_runs_successfully(tmp_path):
assert "list" in result.stdout
def test_persona_create_permissions_preserves_existing_persona(initialized_archive):
from archivebox.tests.conftest import run_python_cwd
result = run_archivebox_cmd(["persona", "create", "--permissions=private", "host-profile"], cwd=initialized_archive)
assert result.returncode == 0, result.stderr
existing = run_archivebox_cmd(["persona", "create", "--permissions=public", "host-profile"], cwd=initialized_archive)
assert existing.returncode == 0, existing.stderr
stdout, stderr, returncode = run_python_cwd(
"import os; os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.core.settings'); "
"import django; django.setup(); from archivebox.personas.models import Persona; "
"p = Persona.objects.get(name='host-profile'); assert p.permissions == 'private'; print('private')",
cwd=initialized_archive,
)
assert returncode == 0, stderr
assert "private" in stdout
def test_persona_import_missing_source_is_actionable_and_does_not_create(initialized_archive, tmp_path):
result = run_archivebox_cmd(
["persona", "create", "--import=brave", "--source", str(tmp_path / "missing"), "broken"],
+134 -1
View File
@@ -31,6 +31,45 @@ RUN_TEST_ENV = {
}
@pytest.mark.django_db(transaction=True)
def test_cli_run_retries_snapshot_after_filesystem_failure(initialized_archive):
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.tests.test_orm_helpers import use_archivebox_db
env = cli_env(live=True, PLUGINS="hashes")
created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
assert created.returncode == 0, created.stdout + created.stderr
snapshot_id = next(record["id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
with use_archivebox_db(initialized_archive):
snapshot = Snapshot.objects.get(pk=snapshot_id)
crawl_id = str(snapshot.crawl_id)
directory = snapshot.output_dir
directory.mkdir(parents=True, exist_ok=True)
(directory / "input.txt").write_text("Capture survives a temporary filesystem failure")
broken_directory = directory / "hashes"
broken_directory.symlink_to("hashes")
failed = run_archivebox_cmd(["run", f"--snapshot-id={snapshot_id}"], cwd=initialized_archive, env=env, timeout=60)
assert failed.returncode == 1, failed.stdout + failed.stderr
assert "FileExistsError: [Errno 17]" in failed.stderr
with use_archivebox_db(initialized_archive):
snapshot = Snapshot.objects.get(pk=snapshot_id)
assert snapshot.status == Snapshot.StatusChoices.STARTED
assert snapshot.retry_at is not None
assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
broken_directory.unlink()
resumed = run_archivebox_cmd(["run", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=60)
assert resumed.returncode == 0, resumed.stdout + resumed.stderr
with use_archivebox_db(initialized_archive):
snapshot = Snapshot.objects.get(pk=snapshot_id)
assert snapshot.status == Snapshot.StatusChoices.SEALED
result = snapshot.archiveresult_set.get(plugin="hashes")
assert result.status == ArchiveResult.StatusChoices.SUCCEEDED
assert result.output_size > 0
assert (directory / "hashes").is_dir()
def _install_real_chrome_for_test(data_dir, env, *, isolation):
env["CHROME_ISOLATION"] = isolation
env["CHROME_HEADLESS"] = "true"
@@ -44,6 +83,52 @@ def _install_real_chrome_for_test(data_dir, env, *, isolation):
assert install_process.returncode == 0, install_process.stderr or install_process.stdout
@pytest.mark.django_db(transaction=True)
@pytest.mark.parametrize("precreate_snapshot", [False, True])
def test_newest_crawl_runs_before_older_queued_and_started_crawls(initialized_archive, precreate_snapshot):
from datetime import timedelta
from django.utils import timezone
from archivebox.core.models import Snapshot
from archivebox.crawls.models import Crawl
from archivebox.tests.test_orm_helpers import use_archivebox_db
env = cli_env(live=True, PLUGINS="hashes")
old_url, queued_url, newest_url = create_test_url(), create_test_url(), create_test_url()
old = run_archivebox_cmd(["snapshot", "create", old_url], cwd=initialized_archive, env=env)
assert old.returncode == 0, old.stdout + old.stderr
with use_archivebox_db(initialized_archive):
snapshot = Snapshot.objects.get(url=old_url)
Crawl.objects.filter(pk=snapshot.crawl_id).update(status=Crawl.StatusChoices.STARTED)
Snapshot.objects.filter(pk=snapshot.pk).update(retry_at=timezone.now() - timedelta(days=1))
queued = run_archivebox_cmd(
["snapshot" if precreate_snapshot else "crawl", "create", queued_url],
cwd=initialized_archive,
env=env,
)
assert queued.returncode == 0, queued.stdout + queued.stderr
newest = run_archivebox_cmd(
["snapshot" if precreate_snapshot else "crawl", "create", newest_url],
cwd=initialized_archive,
env=env,
)
assert newest.returncode == 0, newest.stdout + newest.stderr
result = run_archivebox_cmd(["run", "--no-stdin"], cwd=initialized_archive, env=env, timeout=60)
assert result.returncode == 0, result.stdout + result.stderr
assert all(url in result.stdout for url in (old_url, queued_url, newest_url))
assert result.stdout.index(newest_url) < result.stdout.index(queued_url) < result.stdout.index(old_url), result.stdout
with use_archivebox_db(initialized_archive):
assert (
Snapshot.objects.filter(
url__in=[old_url, queued_url, newest_url],
status=Snapshot.StatusChoices.SEALED,
).count()
== 3
)
@pytest.mark.django_db(transaction=True)
@pytest.mark.timeout(660)
def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archive, recursive_test_site):
@@ -2096,7 +2181,7 @@ class TestRecoverOrchestratorStateRedFailureModes:
stdout="\n".join(json.dumps(record) for record in records),
)
recovered = recover_orchestrator_state()
recovered = recover_orchestrator_state(crawl_id=str(crawl.id))
assert recovered["archiveresults_missing_for_orphaned_hook_processes"] == 1
assert ArchiveResult.objects.filter(snapshot=snapshot, plugin="title", hook_name=hook_name).count() == 1
@@ -2108,6 +2193,54 @@ class TestRecoverOrchestratorStateRedFailureModes:
assert result.end_ts == newer_start + timedelta(seconds=1)
assert result.process.started_at == newer_start
def test_global_recovery_skips_historical_hook_output_scan(self, tmp_path):
from datetime import timedelta
from django.utils import timezone
from archivebox.base_models.models import get_or_create_system_user_pk
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.core.recovery_util import recover_orchestrator_state
from archivebox.crawls.models import Crawl
from archivebox.machine.models import Machine, NetworkInterface, Process
future = timezone.now() + timedelta(days=1)
crawl = Crawl.objects.create(
urls="https://example.com/old\nhttps://example.com/new",
created_by_id=get_or_create_system_user_pk(),
status=Crawl.StatusChoices.STARTED,
retry_at=future,
)
snapshot = Snapshot.objects.create(
url="https://example.com/old",
crawl=crawl,
status=Snapshot.StatusChoices.STARTED,
retry_at=future,
)
historical_plugin_dir = tmp_path / str(snapshot.id) / "title"
historical_plugin_dir.mkdir(parents=True)
(historical_plugin_dir / "title.txt").write_text("preserved historical output")
Process.objects.create(
machine=Machine.current(refresh=True),
iface=NetworkInterface.current(refresh=True),
process_type=Process.TypeChoices.HOOK,
worker_type="archiveresult",
pwd=str(historical_plugin_dir),
cmd=["on_Snapshot__01_title.py"],
status=Process.StatusChoices.EXITED,
retry_at=None,
exit_code=0,
started_at=timezone.now() - timedelta(days=30),
ended_at=timezone.now() - timedelta(days=30) + timedelta(seconds=1),
)
recovered = recover_orchestrator_state()
assert recovered["snapshots_started_without_running_results"] == 1
assert recovered["archiveresults_missing_for_orphaned_hook_processes"] == 0
assert Snapshot.objects.filter(crawl=crawl, retry_at__lte=timezone.now()).count() == 1
assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
def test_recovery_does_not_seal_queued_snapshot_waiting_for_future_retry_even_with_final_results(self):
from datetime import timedelta
+36 -6
View File
@@ -351,12 +351,42 @@ def test_runtime_binary_wins_over_ambient_system_path(tmp_path):
def test_daphne_worker_uses_default_application_close_timeout():
from archivebox.workers.supervisord_util import SERVER_WORKER
command = SERVER_WORKER("127.0.0.1", "8000")["command"]
command = SERVER_WORKER("127.0.0.1", "5797")["command"]
assert "daphne" in command
assert "--application-close-timeout=0" not in command
@pytest.mark.parametrize("in_docker,port", [(False, "5797"), (False, "8000"), (True, "9579")])
@pytest.mark.parametrize("debug", [False, True])
def test_port_compatibility_does_not_add_listeners_outside_docker_defaults(tmp_path, in_docker, port, debug):
result = subprocess.run(
[
sys.executable,
"-c",
"from archivebox.config.common import get_config;"
"from archivebox.workers.supervisord_util import build_server_worker_plan;"
"import json,sys;"
"config=get_config();"
"workers,_,_=build_server_worker_plan(config=config,host='127.0.0.1',"
"port=sys.argv[1],debug=sys.argv[2]=='True',reload=False,nothreading=False);"
"print(json.dumps([worker for worker,lazy in workers]))",
port,
str(debug),
],
cwd=tmp_path,
env=cli_env(IN_DOCKER=str(in_docker), SEARCH_BACKEND_SONIC_ENABLED="False", DISPLAY="", ARCHIVEBOX_VNC_PERSONA=""),
capture_output=True,
text=True,
check=True,
)
workers = json.loads(result.stdout)
assert [worker["name"] for worker in workers] == ["worker_runserver" if debug else "worker_daphne", "worker_runner"]
command = shlex.split(workers[0]["command"])
assert "--endpoint" not in command
assert (f"127.0.0.1:{port}" if debug else f"--port={port}") in command
def test_supervisord_parent_watchdog_does_not_start_another_archivebox_runtime():
from archivebox.workers import supervisord_util
@@ -401,17 +431,17 @@ def test_server_worker_memory_preflight_fails_before_starting_supervisord(capsys
def test_reload_workers_use_active_archivebox_module():
from archivebox.workers.supervisord_util import RUNNER_WATCH_WORKER, RUNSERVER_WORKER, archivebox_cmd
runserver = RUNSERVER_WORKER("127.0.0.1", "8000", reload=True)
watcher = RUNNER_WATCH_WORKER("http://127.0.0.1:8000")
runserver = RUNSERVER_WORKER("127.0.0.1", "5797", reload=True)
watcher = RUNNER_WATCH_WORKER("http://127.0.0.1:5797")
assert runserver["name"] == "worker_runserver"
assert shlex.split(runserver["command"]) == archivebox_cmd("manage", "runserver", "127.0.0.1:8000")
assert shlex.split(runserver["command"]) == archivebox_cmd("manage", "runserver", "127.0.0.1:5797")
assert 'ARCHIVEBOX_RUNSERVER="1"' in runserver["environment"]
assert 'ARCHIVEBOX_AUTORELOAD="1"' in runserver["environment"]
assert 'ARCHIVEBOX_RUNSERVER_BIND_URL="http://127.0.0.1:8000"' in runserver["environment"]
assert 'ARCHIVEBOX_RUNSERVER_BIND_URL="http://127.0.0.1:5797"' in runserver["environment"]
assert watcher["name"] == "worker_runner_watch"
assert shlex.split(watcher["command"]) == archivebox_cmd("manage", "runner_watch", "--bind-url=http://127.0.0.1:8000")
assert shlex.split(watcher["command"]) == archivebox_cmd("manage", "runner_watch", "--bind-url=http://127.0.0.1:5797")
def test_server_daemon_starts_real_plugin_owned_sonic_worker(initialized_archive, archivebox_daemon_server):
+136 -7
View File
@@ -5,8 +5,13 @@ Verify update drains old dirs, reconciles DB, and queues snapshots.
"""
import pytest
import shutil
import signal
import json
import uuid
from archivebox.core.models import Snapshot
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.config import CONSTANTS
from archivebox.tests.conftest import run_queued_crawls, run_archivebox_cmd, cli_env
from archivebox.tests.test_orm_helpers import use_archivebox_db
@@ -14,6 +19,131 @@ from archivebox.tests.test_orm_helpers import use_archivebox_db
pytestmark = pytest.mark.django_db(transaction=True)
def test_rescan_imports_current_layout_and_is_idempotent(initialized_archive, tmp_path):
env = cli_env(disable_extractors=True)
run_archivebox_cmd(["add", "--index-only", "https://example.com/rescan"], cwd=initialized_archive, env=env, check=True)
run_queued_crawls(initialized_archive, env)
run_archivebox_cmd(["update", "--migrate-only"], cwd=initialized_archive, env=env, check=True)
with use_archivebox_db(initialized_archive):
original = Snapshot.objects.get(url="https://example.com/rescan")
original.title = "Imported snapshot"
original.save()
output = original.output_dir / "singlefile" / "singlefile.html"
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text("<html><title>Portable archive fixture</title></html>")
result = ArchiveResult.objects.create(
snapshot=original,
plugin="singlefile",
hook_name="on_Snapshot__singlefile",
status="succeeded",
output_str="singlefile/singlefile.html",
)
result.update_output_metadata_from_filesystem(full_scan=True)
result.refresh_from_db()
expected_result = (result.pk, result.created_at, result.modified_at, result.output_files, result.output_size)
original.refresh_from_db()
original.write_index_jsonl()
index = original.output_dir / "index.jsonl"
historical = {**result.to_json(), "id": uuid.uuid4().hex}
index.write_text(json.dumps(historical) + "\n" + index.read_text())
snapshot_id = original.pk
original_fields = (original.crawl_id, original.created_at, original.modified_at, original.bookmarked_at, original.timestamp)
original.crawl.refresh_from_db()
crawl_dates = (original.crawl.created_at, original.crawl.modified_at)
relative_path = original.output_dir.relative_to(initialized_archive)
destination = tmp_path / "destination"
destination.mkdir()
run_archivebox_cmd(["init", "--quick"], cwd=destination, env=env, check=True)
shutil.copytree(initialized_archive / "archive", destination / "archive", dirs_exist_ok=True, symlinks=True)
result = run_archivebox_cmd(["update", "--migrate-only", "--index-only"], cwd=destination, env=env, timeout=120)
assert result.returncode == 0, result.stdout + result.stderr
with use_archivebox_db(destination):
assert not Snapshot.objects.filter(pk=snapshot_id).exists()
for expected in ("Imported: 1", "Unchanged: 1"):
result = run_archivebox_cmd(["update", "--rescan", "--migrate-only"], cwd=destination, env=env, timeout=120)
assert result.returncode == 0, result.stdout + result.stderr
assert expected in result.stdout
with use_archivebox_db(destination):
imported = Snapshot.objects.get(pk=snapshot_id)
assert imported.title == "Imported snapshot"
assert imported.status == "sealed"
assert imported.output_dir.relative_to(CONSTANTS.DATA_DIR) == relative_path
assert (destination / relative_path / "index.jsonl").is_file()
assert (
imported.crawl_id,
imported.created_at,
imported.modified_at,
imported.bookmarked_at,
imported.timestamp,
) == original_fields
assert (imported.crawl.created_at, imported.crawl.modified_at) == crawl_dates
assert Snapshot.objects.count() == 1
imported_result = imported.archiveresult_set.get()
assert (
imported_result.pk,
imported_result.created_at,
imported_result.modified_at,
imported_result.output_files,
imported_result.output_size,
) == expected_result
assert (destination / relative_path / "singlefile" / "singlefile.html").read_text() == output.read_text()
copied_index = destination / relative_path / "index.jsonl"
records = [json.loads(line) for line in copied_index.read_text().splitlines()]
for record in records:
if record.get("type") == "Snapshot":
record["url"] = "https://example.com/conflicting-identity"
conflicting_bytes = "\n".join(json.dumps(record) for record in records) + "\n"
copied_index.write_text(conflicting_bytes)
result = run_archivebox_cmd(["update", "--rescan"], cwd=destination, env=env)
assert result.returncode == 1, result.stdout + result.stderr
assert "conflicting identity" in result.stdout
assert copied_index.read_text() == conflicting_bytes
with use_archivebox_db(destination):
assert Snapshot.objects.get(pk=snapshot_id).url == "https://example.com/rescan"
@pytest.mark.parametrize("reverse", [False, True])
def test_rescan_resumes_after_real_interrupt_in_requested_order(initialized_archive, tmp_path, reverse):
env = cli_env(disable_extractors=True)
urls = [f"https://example.com/rescan-interrupt/{number}" for number in range(30)]
run_archivebox_cmd(["add", "--index-only", *urls], cwd=initialized_archive, env=env, check=True)
run_queued_crawls(initialized_archive, env)
run_archivebox_cmd(["update", "--migrate-only"], cwd=initialized_archive, env=env, check=True)
with use_archivebox_db(initialized_archive):
originals = list(Snapshot.objects.order_by("id" if reverse else "-id"))
expected = [(snapshot.pk, snapshot.created_at, snapshot.crawl_id) for snapshot in originals]
for snapshot in originals:
snapshot.write_index_jsonl()
destination = tmp_path / "destination"
destination.mkdir()
run_archivebox_cmd(["init", "--quick"], cwd=destination, env=env, check=True)
shutil.copytree(initialized_archive / "archive", destination / "archive", dirs_exist_ok=True, symlinks=True)
args = ["update", "--rescan", *(["--reverse"] if reverse else [])]
process = run_archivebox_cmd(args, cwd=destination, env=env, wait=False)
try:
for line in process.stdout:
if "Imported:" in line:
process.send_signal(signal.SIGINT)
break
stdout, stderr = process.communicate(timeout=60)
assert process.returncode == 130, stdout + stderr
finally:
if process.poll() is None:
process.terminate()
process.wait(timeout=60)
with use_archivebox_db(destination):
partial = list(Snapshot.objects.order_by("id" if reverse else "-id").values_list("pk", "created_at", "crawl_id"))
assert 0 < len(partial) < len(expected)
assert partial == expected[: len(partial)]
result = run_archivebox_cmd(args, cwd=destination, env=env, timeout=120)
assert result.returncode == 0, result.stdout + result.stderr
with use_archivebox_db(destination):
assert list(Snapshot.objects.order_by("id" if reverse else "-id").values_list("pk", "created_at", "crawl_id")) == expected
def test_update_runs_successfully_on_empty_archive(initialized_archive):
"""Test that update runs without error on empty archive."""
result = run_archivebox_cmd(
@@ -23,10 +153,9 @@ def test_update_runs_successfully_on_empty_archive(initialized_archive):
output = result.stdout + result.stderr
assert result.returncode == 0, output
assert "Phase 1: Draining old archive/ directories" in output
assert "Phase 2: Selecting database snapshots with stale filesystem versions" in output
assert "Updated DB rows: 0" in output
assert "Sealed crawls: 0" in output
assert "Filesystem migrations remain lazy" in output
assert "Phase 1" not in output
assert "Phase 2" not in output
with use_archivebox_db(initialized_archive):
assert Snapshot.objects.count() == 0
@@ -52,8 +181,8 @@ def test_update_reconciles_existing_snapshots(initialized_archive):
output = result.stdout + result.stderr
assert result.returncode == 0, output
assert "Phase 2: Selecting database snapshots with stale filesystem versions" in output
assert "Updated DB rows:" in output
assert "Filesystem migrations remain lazy" in output
assert "Updated DB rows:" not in output
with use_archivebox_db(initialized_archive):
assert Snapshot.objects.filter(url="https://example.com", status="sealed").count() == 1
@@ -35,9 +35,9 @@ def test_update_imports_orphaned_snapshots(tmp_path, initialized_archive):
),
)
# Run the migration phase only; default update also runs queued crawl work.
# Filesystem-only discovery is explicit; default update stays lazy.
update_process = run_archivebox_cmd(
["update", "--migrate-only"],
["update", "--rescan", "--migrate-only"],
env=env,
timeout=60,
)
+2 -2
View File
@@ -10,8 +10,8 @@ from archivebox.tests.conftest import run_archivebox_cmd, run_queued_crawls, cli
pytestmark = pytest.mark.django_db(transaction=True)
ADMIN_HOST = "admin.archivebox.localhost:8000"
API_HOST = "api.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:5797"
API_HOST = "api.archivebox.localhost:5797"
def test_delete_after_real_cli_and_orchestrator_paths_cover_all_retained_models(tmp_path, recursive_test_site):
+1 -1
View File
@@ -620,7 +620,7 @@ class TestProcessCurrent:
"""runner_watch should be classified as a worker, not the orchestrator itself."""
old_argv = sys.argv
try:
sys.argv = ["archivebox", "manage", "runner_watch", "--bind-url=http://127.0.0.1:8000"]
sys.argv = ["archivebox", "manage", "runner_watch", "--bind-url=http://127.0.0.1:5797"]
result = Process._detect_process_type()
assert result == Process.TypeChoices.WORKER
finally:
+4 -1
View File
@@ -85,8 +85,11 @@ def test_oldest_django_collection_migrates_end_to_end_without_data_loss(tmp_path
result = run_archivebox_migration_cmd(tmp_path, ["init"], timeout=90)
assert result.returncode == 0, result.stderr
assert "received a naive datetime" not in result.stderr
result = run_archivebox_migration_cmd(tmp_path, ["update"], timeout=60)
assert result.returncode == 0, result.stderr
assert all((tmp_path / "archive" / timestamp).exists() for timestamp in original_trees)
for pass_number in (1, 2):
result = run_archivebox_migration_cmd(tmp_path, ["update"], timeout=180)
result = run_archivebox_migration_cmd(tmp_path, ["update", "--migrate-only"], timeout=180)
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
for command in (["status"], ["list", "--json"]):
+1 -1
View File
@@ -178,7 +178,7 @@ def test_legacy_onedomain_server_upgrade_works_in_auto_mode(archive_07):
"""[SERVER_CONFIG]
LISTEN_HOST = archivebox.mydomain
ALLOWED_HOSTS = archivebox.mydomain
CSRF_TRUSTED_ORIGINS = http://admin.archivebox.localhost:8000
CSRF_TRUSTED_ORIGINS = http://admin.archivebox.localhost:5797
PUBLIC_INDEX = True
PUBLIC_SNAPSHOTS = True
PUBLIC_ADD_VIEW = False
+5 -5
View File
@@ -720,7 +720,7 @@ def test_update_migrates_db_snapshot_when_legacy_index_missing(tmp_path):
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
assert result.returncode == 0, f"Init failed: {result.stderr}"
for pass_number in (1, 2):
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
migrated_files = list((work_dir / "archive" / "users").glob("*/snapshots/*/*/*/screenshot.png"))
@@ -749,7 +749,7 @@ def test_update_recovers_orphan_with_corrupt_index_from_archive_org_url(tmp_path
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
assert result.returncode == 0, f"Init failed: {result.stderr}"
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
assert result.returncode == 0, f"Update failed: {result.stderr}"
conn = sqlite3.connect(str(db_path))
@@ -792,7 +792,7 @@ def test_update_preserves_legacy_folder_timestamp_over_index_float_variant(tmp_p
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
assert result.returncode == 0, f"Init failed: {result.stderr}"
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
assert result.returncode == 0, f"Update failed: {result.stderr}"
conn = sqlite3.connect(str(db_path))
@@ -834,7 +834,7 @@ def test_update_preserves_distinct_legacy_dirs_with_integer_and_float_timestamps
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
assert result.returncode == 0, f"Init failed: {result.stderr}"
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
assert result.returncode == 0, f"Update failed: {result.stderr}"
conn = sqlite3.connect(str(db_path))
@@ -885,7 +885,7 @@ def test_update_preserves_legacy_plugin_directory_without_output_files(migration
result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=60)
assert result.returncode == 0, f"Init failed: {result.stderr}"
for pass_number in (1, 2):
result = run_archivebox_migration_cmd(work_dir, ["update"], timeout=120)
result = run_archivebox_migration_cmd(work_dir, ["update", "--rescan", "--migrate-only"], timeout=120)
assert result.returncode == 0, f"Update pass {pass_number} failed: {result.stderr}"
migrated_outputs = list((work_dir / "archive" / "users").glob("*/snapshots/*/*/*/media/stderr.log"))
+64 -6
View File
@@ -1,9 +1,11 @@
import asyncio
import json
import os
import re
import shutil
import socket
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import TimeoutError as FutureTimeoutError
from pathlib import Path
from types import SimpleNamespace
from urllib.parse import parse_qs, quote, urlsplit
@@ -92,9 +94,9 @@ def installed_opencode(opencode_archive_config):
config = get_config().model_dump(mode="json")
settings = runtime._settings(config, opencode_archive_config.data_dir)
settings["archivebox_base_url"] = "http://admin.archivebox.localhost:8000"
settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:8000/admin"
settings["archivebox_api_url"] = "http://admin.archivebox.localhost:8000/api/"
settings["archivebox_base_url"] = "http://admin.archivebox.localhost:5797"
settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:5797/admin"
settings["archivebox_api_url"] = "http://admin.archivebox.localhost:5797/api/"
binary, binary_env = runtime._resolve_binary(settings["binary"], settings["config"])
version = binary.exec(
cmd=("--version",),
@@ -359,6 +361,45 @@ def test_opencode_proxy_preserves_protocol_headers(admin_client, live_opencode):
assert deleted.status_code == 200, deleted.content
def test_opencode_oauth_callback_waits_for_user_and_preserves_cancellation(live_opencode):
from abx_plugins.plugins.opencode import runtime
settings = {**live_opencode.settings, "timeout": 1}
headers = {"Content-Type": "application/json"}
status, _, body = runtime.proxy(settings, "POST", "provider/openai/oauth/authorize", (), headers, b'{"method":0}')
assert status == 200
authorization = json.loads(body)
assert urlsplit(authorization["url"]).hostname == "auth.openai.com"
redirect = parse_qs(urlsplit(authorization["url"]).query)["redirect_uri"][0]
callback_origin = urlsplit(redirect)
assert callback_origin.hostname == "localhost"
with ThreadPoolExecutor(max_workers=1) as executor:
callback = executor.submit(runtime.proxy, settings, "POST", "provider/openai/oauth/callback", (), headers, b'{"method":0}')
try:
# A real pending authorization must outlive the ordinary API read
# timeout. No tokens or substituted provider responses are used.
with pytest.raises(FutureTimeoutError):
callback.result(timeout=2)
finally:
cancelled = requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=5)
assert cancelled.status_code == 200
assert cancelled.text == "Login cancelled"
status, response_headers, body = callback.result(timeout=5)
assert status == 500
assert response_headers["Content-Type"].startswith("application/json")
error = json.loads(body)
assert error["name"] == "UnknownError"
assert error["data"]["ref"].startswith("err_")
runtime._stop_owned_process()
# The pnpm launcher can exit before its server child. Stopping the owned
# process must release both listeners so another collection can authorize.
with pytest.raises(requests.ConnectionError):
requests.get(settings["origin"] + "/global/health", timeout=2)
with pytest.raises(requests.ConnectionError):
requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=2)
def test_opencode_static_assets_cache_privately_and_revalidate(admin_client, live_opencode):
headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
page = admin_client.get("/admin/agent/opencode/", **headers)
@@ -392,9 +433,26 @@ def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
assert response.headers["Cache-Control"] == "no-store"
def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode):
@pytest.mark.parametrize("path", ["event", "global/event"])
def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode, path):
from abx_plugins.plugins.opencode import runtime
# The real /event endpoint emits heartbeats indefinitely. A bounded
# upstream read exposes accidental buffering without hanging the test.
status, headers, body = runtime.proxy(
{**live_opencode.settings, "timeout": 1},
"GET",
path,
(),
{},
b"",
)
assert status == 200
assert not isinstance(body, bytes)
assert headers["Content-Type"] == "text/event-stream"
asyncio.run(body.aclose())
response = admin_client.get(
"/admin/agent/opencode/global/event",
f"/admin/agent/opencode/{path}",
HTTP_HOST=ADMIN_TEST_HOST,
HTTP_SEC_FETCH_SITE="same-origin",
)
@@ -440,7 +498,7 @@ def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client
(b"sec-fetch-site", b"same-origin"),
],
"client": ("127.0.0.1", 12345),
"server": ("127.0.0.1", 8000),
"server": ("127.0.0.1", 5797),
},
)
runtime._PROCESS_LOCK.acquire()
+127 -6
View File
@@ -20,8 +20,21 @@ from .test_server_security_browser import browser_runtime as browser_runtime
pytestmark = pytest.mark.django_db(transaction=True)
@pytest.mark.parametrize(
"agent_server",
[
("safe-onedomain-nojsreplay", False),
("safe-subdomains-fullreplay", True),
("danger-onedomain-fullreplay", False),
("auto", False),
("auto", True),
],
indirect=True,
)
def test_agent_navigation_stays_inside_mount(agent_server, browser_runtime):
server_url, _, _ = agent_server
server_url, data_dir, _ = agent_server
filename = "proxy acceptance #é.txt"
(data_dir / filename).write_text("Proxy file read: café", encoding="utf-8")
script = r"""
const assert = require('node:assert/strict');
const puppeteer = require('puppeteer');
@@ -31,6 +44,14 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
args: ['--no-sandbox', '--disable-frame-rate-limit']});
try {
const page = await browser.newPage();
const escapedRequests = [];
page.on('request', request => {
const url = new URL(request.url());
if (request.frame()?.parentFrame() && url.origin === config.url &&
!url.pathname.startsWith('/admin/agent/opencode/')) {
escapedRequests.push(request.method() + ' ' + url.pathname);
}
});
await page.setViewport({width: 1440, height: 900});
await page.goto(config.url + '/admin/login/', {waitUntil: 'domcontentloaded'});
await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
@@ -44,7 +65,9 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
const links = await frame.$$eval('a[href*="/session"]', nodes => nodes.map(node => node.getAttribute('href')));
assert.ok(links.length, 'OpenCode must expose session navigation');
for (const href of links) assert.ok(href.startsWith('/admin/agent/opencode/'), href);
const initialSessionUrl = frame.url();
await frame.locator('::-p-aria(New session[role="button"])').click();
await frame.waitForFunction(previous => location.href !== previous, {}, initialSessionUrl);
await frame.waitForSelector('[contenteditable="true"]');
let navigation;
for (const link of await frame.$$('a[href*="/session"]')) {
@@ -61,6 +84,101 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
assert.equal((await frame.goto(sessionUrl, {waitUntil: 'domcontentloaded'})).status(), 200);
await frame.waitForSelector('[contenteditable="true"]');
assert.ok(!(await frame.$eval('body', node => node.innerText)).includes('Something went wrong'));
// Real JSON methods, encoded file paths, and both SSE subscriptions must
// work on the admin origin in every routing mode, without installing or
// authenticating a paid model provider during the test.
const requests = await frame.evaluate(async filename => {
const base = location.origin + '/admin/agent/opencode';
const json = async (path, method = 'GET', body) => {
const response = await fetch(base + path, {
method, headers: {'Content-Type': 'application/json'},
...(body === undefined ? {} : {body: JSON.stringify(body)}),
});
if (!response.ok) throw new Error(method + ' ' + path + ': ' + response.status);
if (!response.headers.get('Content-Type')?.includes('application/json')) {
throw new Error('Non-JSON response: ' + path);
}
return response.json();
};
const providers = await json('/provider');
const auth = await json('/provider/auth');
const file = await json('/file/content?path=' + encodeURIComponent(filename));
const session = await json('/session', 'POST', {title: 'Proxy request verification'});
let renamed;
try {
renamed = await json('/session/' + session.id, 'PATCH', {title: 'Proxy: café & # symbols'});
const readback = await json('/session/' + session.id);
if (readback.title !== renamed.title) throw new Error('Session mutation was not persisted');
} finally { await json('/session/' + session.id, 'DELETE'); }
const streams = [];
for (const path of ['/event', '/global/event']) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
try {
const response = await fetch(base + path, {signal: controller.signal});
if (!response.ok || !response.headers.get('Content-Type')?.includes('text/event-stream')) {
throw new Error('Invalid event stream: ' + path);
}
const reader = response.body.getReader();
const first = await reader.read();
streams.push(new TextDecoder().decode(first.value));
await reader.cancel();
} finally { clearTimeout(timeout); controller.abort(); }
}
return {providers: providers.all.map(provider => provider.id), auth, file, renamed, streams};
}, config.filename);
for (const provider of ['openai', 'anthropic']) assert.ok(requests.providers.includes(provider), provider);
assert.ok(requests.auth.openai.some(method => method.type === 'oauth'));
assert.equal(requests.file.content, 'Proxy file read: café');
assert.equal(requests.renamed.title, 'Proxy: café & # symbols');
for (const event of requests.streams) assert.ok(event.includes('server.connected'), event);
for (const provider of ['OpenAI', 'Anthropic']) {
await frame.locator('[data-action="prompt-model"]').click();
await frame.locator('::-p-aria(' + provider + '[role="button"])').click();
if (provider === 'OpenAI') {
await frame.waitForSelector('::-p-aria(ChatGPT Pro/Plus Headless[role="button"])');
await frame.waitForSelector('::-p-aria(ChatGPT Pro/Plus Browser[role="button"])');
await frame.locator('::-p-aria(API key Browser[role="button"])').click();
}
const input = await frame.waitForSelector('::-p-aria(' + provider + ' API key[role="textbox"])');
assert.equal(await input.evaluate(node => node.value), '');
await frame.waitForSelector('::-p-aria(Continue[role="button"])');
await frame.locator('::-p-aria(Close[role="button"])').click();
}
// Submit a real shell command from a fresh draft. This exercises draft
// promotion and the shell API without needing a paid model or API key.
await frame.locator('::-p-aria(New session[role="button"])').click();
await frame.waitForFunction(() => location.pathname.endsWith('/new-session'));
await (await frame.waitForSelector('[contenteditable="true"]')).focus();
const modifier = await page.evaluate(() => navigator.platform.includes('Mac') ? 'Meta' : 'Control');
await page.keyboard.down(modifier);
await page.keyboard.down('Shift');
await page.keyboard.press('x');
await page.keyboard.up('Shift');
await page.keyboard.up(modifier);
await page.keyboard.type('printf ABX_DRAFT_SHELL_OK');
await frame.locator('::-p-aria(Send[role="button"])').click();
await frame.waitForFunction(() => /\/session\/ses_/.test(location.pathname));
assert.ok(new URL(frame.url()).pathname.startsWith('/admin/agent/opencode/'), frame.url());
await frame.waitForFunction(async () => {
const sessionID = location.pathname.match(/\/session\/(ses_[^/]+)/)[1];
const response = await fetch('/admin/agent/opencode/session/' + sessionID + '/message');
if (!response.ok) throw new Error('Shell message readback: ' + response.status);
const messages = await response.json();
return messages.some(message => message.parts.some(part =>
part.type === 'tool' && part.state.status === 'completed' &&
part.state.output.trim() === 'ABX_DRAFT_SHELL_OK'));
}, {polling: 250});
// The tabbed layout's own route parser must recognize mounted sessions.
// Otherwise Home incorrectly appears selected and clicking it does nothing.
const shellSessionUrl = frame.url();
const home = await frame.waitForSelector('[aria-label="Home"]');
assert.equal(await home.evaluate(node => node.getAttribute('aria-pressed')), 'false');
await home.click();
await frame.waitForFunction(() => location.pathname.replace(/\/+$/, '') === '/admin/agent/opencode');
assert.equal(await home.evaluate(node => node.getAttribute('aria-pressed')), 'true');
await home.click();
await frame.waitForFunction(expected => location.href === expected, {}, shellSessionUrl);
// Exercise the actual public PTY API and native browser WebSocket. No
// intercepted traffic or replacement server: this runs a real shell.
const terminal = await frame.evaluate(async () => {
@@ -97,6 +215,7 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
} finally { await fetch(base + '/pty/' + pty.id, {method: 'DELETE'}); }
});
assert.ok(terminal.includes('ABX_TERMINAL_OK'), terminal);
assert.deepEqual(escapedRequests, [], 'OpenCode requests must stay inside its proxy mount');
assert.equal((await page.goto(config.url + '/add/', {waitUntil: 'domcontentloaded'})).status(), 200);
console.log('AGENT_NAVIGATION_OK');
} finally { await browser.close(); }
@@ -104,7 +223,7 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
"""
result = subprocess.run(
[str(browser_runtime["node_binary"]), "-e", script],
input=json.dumps({"chrome": str(browser_runtime["chrome_binary"]), "url": server_url}),
input=json.dumps({"chrome": str(browser_runtime["chrome_binary"]), "url": server_url, "filename": filename}),
env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
text=True,
capture_output=True,
@@ -115,14 +234,16 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
@pytest.fixture
def agent_server(installed_opencode, browser_runtime):
def agent_server(installed_opencode, browser_runtime, request):
port = get_free_port()
url = f"http://localhost:{port}"
mode, subdomains = getattr(request, "param", ("safe-onedomain-nojsreplay", False))
base_url = f"http://archivebox.localhost:{port}" if subdomains else f"http://localhost:{port}"
url = f"http://admin.archivebox.localhost:{port}" if subdomains else base_url
config = installed_opencode.config
_set_archivebox_config(
config.data_dir,
f"BASE_URL={url}",
"SERVER_SECURITY_MODE=safe-onedomain-nojsreplay",
f"BASE_URL={base_url}",
f"SERVER_SECURITY_MODE={mode}",
)
user = run_archivebox_cmd(
[
+2 -2
View File
@@ -32,8 +32,8 @@ from archivebox.tests.conftest import (
pytestmark = pytest.mark.django_db(transaction=True)
User = get_user_model()
ADMIN_HOST = "admin.archivebox.localhost:8000"
WEB_HOST = "web.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:5797"
WEB_HOST = "web.archivebox.localhost:5797"
@pytest.fixture
@@ -0,0 +1,61 @@
"""Exercise automatic server addressing through real HTTP login pages."""
import re
import requests
from .conftest import cli_env, get_free_port, run_archivebox_cmd, start_archivebox_server, stop_archivebox_process
def test_safe_single_domain_login_does_not_require_base_url(tmp_path):
port = get_free_port()
env = cli_env(
disable_extractors=True,
BASE_URL="",
SERVER_SECURITY_MODE="safe-onedomain-nojsreplay",
ADMIN_USERNAME="automatic-url-test",
ADMIN_PASSWORD="Automatic-url-test-password-93!",
ALLOWED_HOSTS="*",
)
result = run_archivebox_cmd(["init", "--quick"], cwd=tmp_path, env=env, timeout=60)
assert result.returncode == 0, result.stderr or result.stdout
process = start_archivebox_server(tmp_path, port=port, env=env, daemonize=False, log_name="server.log")
try:
for hostname in ("localhost", "archivebox.localhost", "admin.archivebox.localhost", "192.168.1.20", "100.100.10.20"):
session = requests.Session()
response = session.get(
f"http://127.0.0.1:{port}/admin/login/",
headers={"Host": f"{hostname}:{port}"},
timeout=15,
allow_redirects=False,
)
assert response.status_code == 200, (hostname, response.status_code)
assert 'name="password"' in response.text, hostname
assert "base_url not set" not in response.text.lower(), hostname
assert "archivebox-setup-wizard" not in response.text, hostname
csrf = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', response.text)
assert csrf, hostname
login = session.post(
f"http://127.0.0.1:{port}/admin/login/",
headers={"Host": f"{hostname}:{port}"},
data={
"username": env["ADMIN_USERNAME"],
"password": env["ADMIN_PASSWORD"],
"csrfmiddlewaretoken": csrf[1],
"next": "/admin/",
},
timeout=15,
allow_redirects=False,
)
assert login.status_code == 302, (hostname, login.status_code)
admin = session.get(
f"http://127.0.0.1:{port}/admin/",
headers={"Host": f"{hostname}:{port}"},
timeout=15,
allow_redirects=False,
)
assert admin.status_code == 200, (hostname, admin.status_code)
assert "base_url not set" not in admin.text.lower(), hostname
assert "archivebox-setup-wizard" not in admin.text, hostname
finally:
stop_archivebox_process(process)
@@ -749,7 +749,7 @@ def test_unconfigured_public_host_superuser_can_reach_setup_wizard(tmp_path: Pat
assert "BASE_URL" in archivebox_environment
assert "SERVER_SECURITY_MODE" in archivebox_environment
assert all("ARCHIVEBOX_INGRESS_BASE_URL" not in value for value in archivebox_environment)
assert compose["services"]["archivebox"]["ports"] == ["${ARCHIVEBOX_PORT:-8000}:8000"]
assert compose["services"]["archivebox"]["ports"] == ["${ARCHIVEBOX_PORT:-5797}:5797"]
assert set(compose["services"]) == {"archivebox"}
+3 -3
View File
@@ -16,8 +16,8 @@ from archivebox.workers.models import RETRY_AT_MAX
pytestmark = pytest.mark.django_db
User = get_user_model()
WEB_HOST = "web.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:8000"
WEB_HOST = "web.archivebox.localhost:5797"
ADMIN_HOST = "admin.archivebox.localhost:5797"
ADD_FORM_DEFAULTS = {
@@ -99,7 +99,7 @@ def test_add_view_admin_renders_plugin_config_grid(client, admin_user):
assert b">Source</a>" in response.content
assert b">Docs</a>" in response.content
assert b"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/" in response.content
assert b"https://archivebox.github.io/abx-plugins/#" in response.content
assert b"https://plugins.archivebox.io/#" in response.content
assert not any(
field["key"].endswith("_BINARY") for group in form.plugin_groups for card in group["plugins"] for field in card["config_fields"]
)
+11 -11
View File
@@ -99,8 +99,8 @@ def test_unconfigured_superuser_banner_uses_browser_assisted_setup_wizard():
{
"mode": "unconfigured",
"can_configure": True,
"canonical_host": "archivebox.example.test:8000",
"suggested_base_url": "http://archivebox.example.test:8000",
"canonical_host": "archivebox.example.test:5797",
"suggested_base_url": "http://archivebox.example.test:5797",
"machine_admin_url": "/admin/machine/machine/current/change/",
"public_index": True,
"public_add_view": False,
@@ -111,7 +111,7 @@ def test_unconfigured_superuser_banner_uses_browser_assisted_setup_wizard():
assert html.count('id="archivebox-setup-wizard"') == 1
assert 'id="archivebox-system-warning-banner"' not in html
assert "⚠ base_url not set" not in html
assert "Setup access to your ArchiveBox Server: <code>archivebox.example.test:8000</code>" in html
assert "Setup access to your ArchiveBox Server: <code>archivebox.example.test:5797</code>" in html
assert 'id="archivebox-setup-base-url"' in html
assert 'id="archivebox-setup-security-mode"' in html
assert 'id="archivebox-setup-public-index"' in html
@@ -245,9 +245,9 @@ def test_health_check_is_identifiable_across_ingress_origins():
@pytest.mark.parametrize(
("request_host", "secure", "expected_display_host"),
(
("admin.archivebox.localhost:8000", False, "archivebox.localhost:8000"),
("127.0.0.1:8000", False, "archivebox.localhost:8000"),
("0.0.0.0:8000", False, "archivebox.localhost:8000"),
("admin.archivebox.localhost:5797", False, "archivebox.localhost:5797"),
("127.0.0.1:5797", False, "archivebox.localhost:5797"),
("0.0.0.0:5797", False, "archivebox.localhost:5797"),
("archivebox.io:80", False, "archivebox.io"),
("archivebox.io:443", True, "archivebox.io"),
("192.0.2.10:443", True, "192.0.2.10"),
@@ -279,7 +279,7 @@ def test_unconfigured_banner_honors_forwarded_https_from_ingress():
("base_url", "request_host"),
(
("", "archivebox.example.test:18443"),
("http://archivebox.localhost:8000", "admin.archivebox.localhost:18010"),
("http://archivebox.localhost:5797", "admin.archivebox.localhost:18010"),
),
)
def test_admin_cookie_isolation_accepts_first_run_and_external_port_mapping(base_url, request_host):
@@ -302,7 +302,7 @@ def test_unconfigured_banner_does_not_show_setup_wizard_to_non_superusers():
{
"mode": "unconfigured",
"can_configure": False,
"suggested_base_url": "http://archivebox.example.test:8000",
"suggested_base_url": "http://archivebox.example.test:5797",
},
)
@@ -329,13 +329,13 @@ def test_system_warning_modes_share_one_banner(context, expected_text):
def test_configured_base_url_mismatch_banner_shows_both_origins():
config = get_config(include_machine=False).model_copy(update={"BASE_URL": "https://archivebox.example.test"})
request = RequestFactory().get("/admin/", HTTP_HOST="archivebox.internal:8000")
request = RequestFactory().get("/admin/", HTTP_HOST="archivebox.internal:5797")
request.user = AnonymousUser()
context = get_base_url_mismatch_context(request, config)
assert context == {
"mode": "base_url_mismatch",
"browser_url": "http://archivebox.internal:8000",
"browser_url": "http://archivebox.internal:5797",
"configured_base_url": "https://archivebox.example.test",
}
assert system_warnings_banner({"CONFIG": config, "request": request}) == context
@@ -343,7 +343,7 @@ def test_configured_base_url_mismatch_banner_shows_both_origins():
html = render_to_string("core/system_warnings_banner.html", context)
assert "base_url mismatch" in html
assert "Browser URL:" in html
assert "http://archivebox.internal:8000" in html
assert "http://archivebox.internal:5797" in html
assert "Configured BASE_URL:" in html
assert "https://archivebox.example.test" in html
+1 -1
View File
@@ -416,7 +416,7 @@ def test_snapshot_from_json_trims_markdown_suffixes_on_discovered_urls(crawl):
def test_create_snapshots_from_urls_skips_invalid_and_archivebox_internal_urls(admin_user):
crawl = Crawl.objects.create(
urls=("https://example.com/root\nhttp://127.0.0.1:8765/page-001.html\nnot-a-url\nhttp://admin.archivebox.localhost:8000/admin/"),
urls=("https://example.com/root\nhttp://127.0.0.1:8765/page-001.html\nnot-a-url\nhttp://admin.archivebox.localhost:5797/admin/"),
created_by=admin_user,
)
+2 -2
View File
@@ -130,7 +130,7 @@ def _admin_post_request(path):
def _admin_get_request(path="/"):
from archivebox.config.common import get_config
request = RequestFactory().get(path, HTTP_HOST="admin.archivebox.localhost:8000")
request = RequestFactory().get(path, HTTP_HOST="admin.archivebox.localhost:5797")
request.archivebox_config = get_config()
return request
@@ -284,7 +284,7 @@ def test_admin_navigation_hides_agent_link_when_opencode_is_disabled(client, adm
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
client.force_login(admin_user)
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:8000")
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:5797")
assert response.status_code == 200
assert b"/admin/agent" not in response.content
@@ -127,6 +127,10 @@ def test_persona_admin_add_view_renders_import_ui(admin_client):
assert b"auth.json" in response.content
assert b"Plugin Config" in response.content
assert b'name="plugin_config__wget__WGET_TIMEOUT"' in response.content
assert b'<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">' in response.content
assert b'href="https://extension.archivebox.io/"' in response.content
assert b"import cookies into this persona from another browser" in response.content
assert "wide" in response.context["adminform"].fieldsets[0][1]["classes"]
def test_import_persona_from_source_copies_profile_without_browser_export(admin_user):
+106 -7
View File
@@ -13,7 +13,7 @@ import pytest
from django.contrib.auth.models import AnonymousUser
from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
from django.core.paginator import UnorderedObjectListWarning
from django.test import RequestFactory
from django.test import Client, RequestFactory
from django.urls import reverse
from archivebox.core.middleware import ADMIN_LOGIN_HINT_COOKIE
@@ -152,6 +152,8 @@ def test_snapshot_changelist_uses_stable_ordering_without_unordered_paginator_wa
def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client, snapshot, real_hash_projection):
from archivebox.core.models import ArchiveResult
snapshot.title = "Preview destination test"
snapshot.save(update_fields=["title"])
_process, result = real_hash_projection
ArchiveResult.objects.filter(pk=result.pk).update(
plugin="screenshot",
@@ -165,6 +167,103 @@ def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client,
assert response.status_code == 200
assert b'class="snapshot-preview' in response.content
assert b"screenshot.png" in response.content
html = response.content.decode()
preview_cell = re.search(r'<td class="field-preview_icon">(.*?)</td>', html, re.S)
title_cell = re.search(r'<td class="field-title_str">(.*?)</td>', html, re.S)
assert preview_cell and title_cell
preview_link = re.search(r'<a href="([^"]+)"', preview_cell.group(1))
title_link = re.search(r'<a href="([^"]+)"', title_cell.group(1))
assert preview_link and title_link
assert preview_link.group(1) == title_link.group(1)
assert preview_link.group(1).endswith(f"/{snapshot.archive_path_from_db}/index.html")
def test_snapshot_grid_missing_preview_is_inside_snapshot_detail_link(admin_client, snapshot):
snapshot.status = snapshot.StatusChoices.SEALED
snapshot.save(update_fields=["status"])
response = admin_client.get(reverse("admin:grid"), HTTP_HOST=ADMIN_TEST_HOST)
assert response.status_code == 200
html = response.content.decode()
thumbnail_link = re.search(
r'<a href="([^"]+)" class="card-thumbnail">(.*?)</a>',
html,
re.S,
)
title_link = re.search(
r'<div class="card-title"[^>]*>\s*<a href="([^"]+)">',
html,
re.S,
)
assert thumbnail_link and title_link
assert thumbnail_link.group(1) == title_link.group(1)
assert thumbnail_link.group(1).endswith("/index.html")
assert '<span class="missing-preview">No preview captured</span>' in thumbnail_link.group(2)
def test_snapshot_grid_serves_extension_preview_from_authenticated_admin_origin(admin_client, snapshot, real_hash_projection):
_process, result = real_hash_projection
preview_dir = Path(snapshot.output_dir) / "chrome_extension_screenshot"
preview_dir.mkdir(parents=True, exist_ok=True)
preview_bytes = b"\x89PNG\r\n\x1a\nreal-extension-preview"
(preview_dir / "screenshot.png").write_bytes(preview_bytes)
result.plugin = "chrome_extension_screenshot"
result.status = "succeeded"
result.output_files = {"screenshot.png": {"size": len(preview_bytes), "mimetype": "image/png"}}
result.save(update_fields=["plugin", "status", "output_files"])
grid_response = admin_client.get(reverse("admin:grid"), HTTP_HOST=ADMIN_TEST_HOST)
assert grid_response.status_code == 200
html = grid_response.content.decode()
preview_url = reverse(
"admin:core_snapshot_preview",
args=(snapshot.pk, "chrome_extension_screenshot", "screenshot.png"),
)
assert f'src="{preview_url}"' in html
assert "new URL(candidate, document.baseURI).href" in html
assert "candidate !== current" in html
preview_response = admin_client.get(preview_url, HTTP_HOST=ADMIN_TEST_HOST)
assert preview_response.status_code == 200
assert b"".join(preview_response.streaming_content) == preview_bytes
assert preview_response["Content-Type"] == "image/png"
assert {part.strip() for part in preview_response["Cache-Control"].split(",")} == {
"private",
"max-age=604800",
"immutable",
}
assert {part.strip() for part in preview_response["Vary"].split(",")} >= {"Cookie", "Authorization"}
assert preview_response["X-Content-Type-Options"] == "nosniff"
not_modified = admin_client.get(
preview_url,
HTTP_HOST=ADMIN_TEST_HOST,
HTTP_IF_MODIFIED_SINCE=preview_response["Last-Modified"],
)
assert not_modified.status_code == 304
assert {part.strip() for part in not_modified["Cache-Control"].split(",")} == {
"private",
"max-age=60",
"stale-while-revalidate=300",
}
assert {part.strip() for part in not_modified["Vary"].split(",")} >= {"Cookie", "Authorization"}
denied = Client().get(preview_url, HTTP_HOST=ADMIN_TEST_HOST)
assert denied.status_code in (301, 302)
assert {part.strip() for part in denied["Cache-Control"].split(",")} == {"private", "no-store"}
disallowed_dir = Path(snapshot.output_dir) / "singlefile"
disallowed_dir.mkdir(parents=True, exist_ok=True)
(disallowed_dir / "singlefile.html").write_text("<script>document.cookie</script>", encoding="utf-8")
result.plugin = "singlefile"
result.output_files = {"singlefile.html": {"size": 32, "mimetype": "text/html"}}
result.save(update_fields=["plugin", "output_files"])
disallowed_url = reverse("admin:core_snapshot_preview", args=(snapshot.pk, "singlefile", "singlefile.html"))
assert admin_client.get(disallowed_url, HTTP_HOST=ADMIN_TEST_HOST).status_code == 404
def test_snapshot_result_health_filter_uses_live_status_rows(admin_client, snapshot):
@@ -312,7 +411,7 @@ def test_snapshot_admin_preview_uses_extension_screenshot_when_standard_screensh
)
admin = SnapshotAdmin(Snapshot, archivebox_admin)
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:8000")
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:5797")
request.archivebox_config = get_config()
admin.request = request
@@ -411,7 +510,7 @@ class TestSnapshotProgressStats:
prefetched_snapshot = Snapshot.objects.prefetch_related("archiveresult_set").get(pk=snapshot.pk)
admin = SnapshotAdmin(Snapshot, archivebox_admin)
request = RequestFactory().get("/", HTTP_HOST="archivebox.localhost:8000")
request = RequestFactory().get("/", HTTP_HOST="archivebox.localhost:5797")
request.resolver_match = resolve("/")
request.archivebox_config = get_config()
admin.request = request
@@ -777,7 +876,7 @@ class TestSnapshotProgressStats:
assert embed_path in html
assert "?preview=1" not in html
assert html != "http://snap-ffa4215f6d64.archivebox.localhost:8000"
assert html != "http://snap-ffa4215f6d64.archivebox.localhost:5797"
def test_plugin_full_returns_empty_for_none_result(self):
from archivebox.core.templatetags import core_tags
@@ -1279,13 +1378,13 @@ class TestAdminSnapshotListView:
admin = SnapshotAdmin(snapshot.__class__, archivebox_admin)
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:8000")
request = RequestFactory().get("/", HTTP_HOST="admin.archivebox.localhost:5797")
request.archivebox_config = get_config(overrides={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
admin.request = request
assert admin.get_snapshot_view_url(snapshot) == f"http://snap-{str(snapshot.pk).replace('-', '')[-12:]}.archivebox.localhost:8000"
assert admin.get_snapshot_view_url(snapshot) == f"http://snap-{str(snapshot.pk).replace('-', '')[-12:]}.archivebox.localhost:5797"
request.archivebox_config = get_config(overrides={"SERVER_SECURITY_MODE": "safe-onedomain-nojsreplay"})
assert admin.get_snapshot_view_url(snapshot) == f"http://archivebox.localhost:8000/snapshot/{snapshot.pk}"
assert admin.get_snapshot_view_url(snapshot) == f"http://archivebox.localhost:5797/snapshot/{snapshot.pk}"
def test_find_snapshots_for_url_matches_fragment_suffixed_variants(self, crawl, db):
from archivebox.core.models import Snapshot
+1 -1
View File
@@ -132,7 +132,7 @@ def test_plugin_detail_view_renders_shipped_plugin_config_in_dedicated_sections(
"source": "builtin",
}
assert "abx_plugins/plugins/wget" in summary_section["description"]
assert "https://archivebox.github.io/abx-plugins/#wget" in summary_section["description"]
assert "https://plugins.archivebox.io/#wget" in summary_section["description"]
assert hooks_section["name"] == "Hooks"
assert hooks_section["fields"] == {}
+109 -1
View File
@@ -3,9 +3,11 @@
import subprocess
import uuid
from datetime import datetime, timezone as dt_timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from importlib.resources import files
from pathlib import Path
from threading import Thread
from threading import Event, Thread
from types import SimpleNamespace
import pytest
from django.test import override_settings
@@ -19,6 +21,41 @@ from archivebox.tests.test_archive_result_service import _run_shipped_snapshot_h
pytestmark = pytest.mark.django_db(transaction=True)
@pytest.fixture
def painted_http_server():
"""Serve a complete static page so CDP stops emitting changed frames."""
request_started = Event()
class PaintedBlockingHandler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"""<!doctype html>
<html><head><title>Still frame</title></head>
<body style="background:#246;color:white"><h1>Visible page</h1></body></html>
"""
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
self.wfile.flush()
request_started.set()
def log_message(self, format, *args):
pass
server = ThreadingHTTPServer(("127.0.0.1", 0), PaintedBlockingHandler)
thread = Thread(target=server.serve_forever, name="archivebox-test-painted-http-barrier")
thread.start()
yield SimpleNamespace(
url=f"http://127.0.0.1:{server.server_port}/",
request_started=request_started,
)
server.shutdown()
server.server_close()
thread.join()
@pytest.fixture
def real_unscoped_hook_process(tmp_path):
from archivebox.tests.conftest import run_test_hook
@@ -329,6 +366,77 @@ class TestLiveProgressView:
result.refresh_from_db()
assert result.status in (ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS)
def test_live_progress_keeps_still_browser_frame_while_screencast_runs(
self,
client,
admin_user,
crawl,
snapshot,
painted_http_server,
cached_abxpkg_lib_dir,
):
import time
from archivebox.core.models import Snapshot
from archivebox.crawls.models import Crawl
from archivebox.services.runner import run_due_snapshot
now = timezone.now()
Crawl.objects.filter(pk=crawl.pk).update(
status=Crawl.StatusChoices.STARTED,
retry_at=now,
urls=painted_http_server.url,
config={"PLUGINS": "chrome,chrome_screencast", "CHROME_ISOLATION": "snapshot"},
)
Snapshot.objects.filter(pk=snapshot.pk).update(
status=Snapshot.StatusChoices.QUEUED,
retry_at=now,
url=painted_http_server.url,
config={
"PLUGINS": "chrome,chrome_screencast",
"CHROME_HEADLESS": True,
"CHROME_SANDBOX": False,
"CHROME_ISOLATION": "snapshot",
"CHROME_TIMEOUT": 90,
"CHROME_PAGELOAD_TIMEOUT": 90,
"CHROME_DELAY_AFTER_LOAD": 30,
"TIMEOUT": 90,
},
)
snapshot.refresh_from_db()
frame = Path(snapshot.crawl.output_dir) / "chrome_screencast" / "latest.jpg"
errors = []
def run_snapshot():
try:
assert run_due_snapshot(snapshot, lock_seconds=120) is True
except BaseException as err:
errors.append(err)
runner = Thread(target=run_snapshot, name="archivebox-test-screencast-runner")
runner.start()
try:
assert painted_http_server.request_started.wait(timeout=60)
deadline = time.monotonic() + 90
while time.monotonic() < deadline:
assert errors == []
if frame.is_file() and time.time() - frame.stat().st_mtime > 16:
break
time.sleep(0.1)
assert frame.is_file()
assert frame.read_bytes().startswith(b"\xff\xd8\xff")
assert time.time() - frame.stat().st_mtime > 16
client.force_login(admin_user)
response = client.get("/progress.json", HTTP_HOST=ADMIN_TEST_HOST)
assert response.status_code == 200, response.content
active_crawl = next(item for item in response.json()["active_crawls"] if item["id"] == str(crawl.pk))
assert active_crawl["screencast_url"].startswith(f"/api/v1/crawls/crawl/{crawl.pk}/files/chrome_screencast/latest.jpg")
finally:
runner.join(timeout=120)
assert not runner.is_alive()
assert errors == []
assert not frame.exists()
def test_live_progress_hides_finished_cancelled_crawl(self, client, admin_user, crawl, snapshot):
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.crawls.models import Crawl
+5 -5
View File
@@ -8,7 +8,7 @@ from pathlib import Path
from archivebox.tests.conftest import cli_env, create_test_url
ADMIN_HOST = "admin.archivebox.localhost:8000"
ADMIN_HOST = "admin.archivebox.localhost:5797"
def _run_savepagenow_script(
@@ -93,7 +93,7 @@ def _run_savepagenow_not_found_script(initialized_archive: Path, request_url: st
client = Client()
target_url = {request_url!r}
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
assert resp.status_code == 302, resp.status_code
assert resp['Location'] == f'http://{ADMIN_HOST}/web/' + target_url
assert Snapshot.objects.count() == 0
@@ -226,7 +226,7 @@ def _run_savepagenow_via_web_host_redirect_script(initialized_archive: Path, req
target_url = {request_url!r}
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
assert resp.status_code == 302, resp.status_code
assert resp['Location'] == f'http://{ADMIN_HOST}/web/' + target_url
@@ -274,7 +274,7 @@ def _run_savepagenow_existing_snapshot_script(initialized_archive: Path, request
snapshot = Snapshot.objects.create(url=stored_url, crawl=crawl)
client = Client()
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:8000')
resp = client.get('/web/' + target_url, HTTP_HOST='web.archivebox.localhost:5797')
assert resp.status_code == 302, resp.status_code
assert resp['Location'] == f"/{{snapshot.url_path}}"
""",
@@ -310,7 +310,7 @@ def test_web_add_creates_and_reuses_snapshot_public(initialized_archive):
url,
login=False,
public_add_view=True,
host="web.archivebox.localhost:8000",
host="web.archivebox.localhost:5797",
)
assert result.returncode == 0, f"SavePageNow shortcut (public add) test failed.\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
+404 -20
View File
@@ -8,12 +8,51 @@ from pathlib import Path
from threading import Thread
import pytest
from types import SimpleNamespace
from archivebox.tests.conftest import run_archivebox_cmd
REPO_ROOT = Path(__file__).resolve().parents[2]
def test_private_snapshot_image_response_is_not_publicly_cacheable(tmp_path):
from django.test import RequestFactory
from archivebox.misc.serve_static import serve_static_with_byterange_support
(tmp_path / "screenshot.png").write_bytes(b"\x89PNG\r\n\x1a\nprivate")
request = RequestFactory().get("/snapshot/private/screenshot.png")
request.archivebox_config = SimpleNamespace(PERMISSIONS="public")
request.archivebox_cache_policy = "private"
response = serve_static_with_byterange_support(request, "screenshot.png", document_root=tmp_path)
assert response.status_code == 200
assert response["Content-Type"] == "image/png"
assert response["Cache-Control"] == "private, max-age=604800, immutable"
@pytest.mark.django_db
def test_private_snapshot_replay_never_returns_public_cache_headers(snapshot):
from django.test import RequestFactory
from archivebox.core.views import _serve_snapshot_replay
snapshot.config = {**snapshot.config, "PERMISSIONS": "private"}
snapshot.save(update_fields=["config"])
snapshot.output_dir.mkdir(parents=True, exist_ok=True)
(snapshot.output_dir / "screenshot.png").write_bytes(b"\x89PNG\r\n\x1a\nprivate-replay")
request = RequestFactory().get(f"/snapshot/{snapshot.id}/screenshot.png")
response = _serve_snapshot_replay(request, snapshot, "screenshot.png")
assert response.status_code == 200
assert response["Cache-Control"].startswith("private,")
assert "public" not in response["Cache-Control"]
assert "Cookie" in response["Vary"]
assert b"".join(response.streaming_content).endswith(b"private-replay")
@pytest.mark.django_db
def test_archiveresult_relpath_uses_sibling_hook_that_owns_output(admin_user):
from archivebox.core.models import ArchiveResult, Snapshot
@@ -495,16 +534,16 @@ class TestUrlRouting:
bind_host, bind_port = split_host_port(bind_addr)
assert bind_host == "127.0.0.1"
assert bind_port == "8000"
assert bind_port == "5797"
host_only, port = split_host_port(base_host)
assert host_only == "archivebox.localhost"
assert port == "8000"
assert web_host == "web.archivebox.localhost:8000"
assert admin_host == "admin.archivebox.localhost:8000"
assert api_host == "api.archivebox.localhost:8000"
assert port == "5797"
assert web_host == "web.archivebox.localhost:5797"
assert admin_host == "admin.archivebox.localhost:5797"
assert api_host == "api.archivebox.localhost:5797"
assert snapshot_subdomain == f"snap-{snapshot_id[-12:].lower()}"
assert snapshot_host == f"{snapshot_subdomain}.archivebox.localhost:8000"
assert original_host == f"{domain}.archivebox.localhost:8000"
assert snapshot_host == f"{snapshot_subdomain}.archivebox.localhost:5797"
assert original_host == f"{domain}.archivebox.localhost:5797"
assert get_listen_subdomain(web_host) == "web"
assert get_listen_subdomain(admin_host) == "admin"
assert get_listen_subdomain(api_host) == "api"
@@ -562,7 +601,7 @@ class TestUrlRouting:
)
finally:
reset_result = run_archivebox_cmd(
["config", "--set", "BASE_URL=http://archivebox.localhost:8000"],
["config", "--set", "BASE_URL=http://archivebox.localhost:5797"],
cwd=self.data_dir,
)
assert reset_result.returncode == 0, reset_result.stderr
@@ -622,6 +661,351 @@ class TestUrlRouting:
""",
)
def test_private_snapshot_replay_cookie_authorizes_only_its_progress(self) -> None:
self._run(
"""
from urllib.parse import urlsplit
ensure_admin_user()
snapshot = get_snapshot()
original_config = snapshot.config
Snapshot.objects.filter(pk=snapshot.pk).update(config={**snapshot.config, "PERMISSIONS": "private"})
other_snapshot = Snapshot.objects.create(
url="https://other-private-progress.example/",
crawl=snapshot.crawl,
config={"PERMISSIONS": "private"},
)
snapshot_host = get_snapshot_host(str(snapshot.id))
admin_host = get_admin_host()
try:
admin_client = Client()
assert admin_client.login(username="testadmin", password="testpassword")
grant_response = admin_client.get(
f"/admin/core/snapshot/replay-auth/?snapshot={snapshot.id}&next=/index.html",
HTTP_HOST=admin_host,
)
assert grant_response.status_code in (301, 302)
grant_url = urlsplit(grant_response["Location"])
assert grant_url.netloc == snapshot_host
replay_client = Client()
auth_response = replay_client.get(
f"{grant_url.path}?{grant_url.query}",
HTTP_HOST=snapshot_host,
)
assert auth_response.status_code in (301, 302)
snapshot_id = str(snapshot.id).replace("-", "")
progress_response = replay_client.get(
f"/progress.json?snapshot_id={snapshot_id}",
HTTP_HOST=snapshot_host,
)
assert progress_response.status_code == 200, response_body(progress_response)[:500]
assert progress_response.json()["scope"]["snapshot_id"] == snapshot_id
other_snapshot_id = str(other_snapshot.id).replace("-", "")
rejected = replay_client.get(
f"/progress.json?snapshot_id={other_snapshot_id}",
HTTP_HOST=snapshot_host,
)
assert rejected.status_code == 403
finally:
other_snapshot.delete()
Snapshot.objects.filter(pk=snapshot.pk).update(config=original_config)
print("OK")
""",
mode="safe-subdomains-fullreplay",
)
def test_snapshot_payload_cache_policy_tracks_effective_permissions_and_revocation(self) -> None:
self._run(
"""
import hashlib
import json
from urllib.parse import urlsplit
from archivebox.crawls.models import Crawl
admin = ensure_admin_user()
snapshot = get_snapshot()
original_config = snapshot.config
raw_name = "cache-policy.png"
raw_bytes = b"\\x89PNG\\r\\n\\x1a\\ncache-policy-payload"
raw_path = Path(snapshot.output_dir) / raw_name
raw_path.write_bytes(raw_bytes)
metadata_path = Path(snapshot.output_dir) / "index.jsonl"
if not metadata_path.exists():
metadata_path.write_text('{"type":"Snapshot"}\\n', encoding="utf-8")
hashes_dir = Path(snapshot.output_dir) / "hashes"
hashes_dir.mkdir(parents=True, exist_ok=True)
(hashes_dir / "hashes.json").write_text(
json.dumps(
{
"files": [
{"path": raw_name, "hash": hashlib.sha256(raw_bytes).hexdigest()},
{
"path": "index.jsonl",
"hash": hashlib.sha256(metadata_path.read_bytes()).hexdigest(),
},
],
},
),
encoding="utf-8",
)
snapshot_host = get_snapshot_host(str(snapshot.id))
def cache_tokens(response):
return {token.strip().lower() for token in response.headers.get("Cache-Control", "").split(",") if token.strip()}
def assert_cache(response, *expected):
assert cache_tokens(response) == set(expected), (response.status_code, response.headers)
def assert_auth_vary(response):
vary = {token.strip().lower() for token in response.headers.get("Vary", "").split(",") if token.strip()}
assert {"cookie", "authorization"} <= vary, response.headers
try:
snapshot.config = {**snapshot.config, "PERMISSIONS": "public"}
snapshot.save(update_fields=["config"])
snapshot.refresh_from_db()
assert snapshot.permissions == "public"
public_client = Client()
public_index = public_client.get("/index.html", HTTP_HOST=snapshot_host)
assert public_index.status_code == 200
assert_cache(public_index, "public", "max-age=60", "stale-while-revalidate=300")
public_files = public_client.get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
assert public_files.status_code == 200
assert raw_name.encode() in response_body(public_files)
assert_cache(public_files, "public", "max-age=60", "stale-while-revalidate=300")
public_metadata = public_client.get("/index.jsonl", HTTP_HOST=snapshot_host)
assert public_metadata.status_code == 200
assert response_body(public_metadata) == metadata_path.read_bytes()
assert_cache(public_metadata, "public", "max-age=31536000", "immutable")
public_raw = public_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host)
assert public_raw.status_code == 200
assert response_body(public_raw) == raw_bytes
assert public_raw.headers.get("ETag")
assert_cache(public_raw, "public", "max-age=604800", "immutable")
public_preview = public_client.get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
assert public_preview.status_code == 200
assert public_preview["Content-Type"].startswith("text/html")
assert_cache(public_preview, "public", "max-age=60", "stale-while-revalidate=300")
public_range = public_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
assert public_range.status_code == 206
assert response_body(public_range) == raw_bytes[:8]
assert_cache(public_range, "public", "max-age=604800", "immutable")
public_not_modified = public_client.get(
f"/{raw_name}",
HTTP_HOST=snapshot_host,
HTTP_IF_NONE_MATCH=public_raw["ETag"],
)
assert public_not_modified.status_code == 304
assert_cache(public_not_modified, "public", "max-age=31536000", "immutable")
public_progress = public_client.get(
f"/progress.json?snapshot_id={str(snapshot.id).replace('-', '')}",
HTTP_HOST=snapshot_host,
)
assert public_progress.status_code == 200
assert_cache(public_progress, "private", "no-store")
public_missing = public_client.get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
assert public_missing.status_code == 404
assert_cache(public_missing, "private", "no-store")
public_redirect = public_client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
assert public_redirect.status_code in (301, 302)
assert public_redirect["Location"].startswith(f"http://{snapshot_host}")
assert_cache(public_redirect, "private", "no-store")
legacy_redirect = public_client.get(
f"/{snapshot.legacy_archive_path}/index.html",
HTTP_HOST=get_web_host(),
)
assert legacy_redirect.status_code in (301, 302)
assert snapshot.url_path in legacy_redirect["Location"]
assert_cache(legacy_redirect, "private", "no-store")
static_asset = public_client.get("/static/jquery.min.js", HTTP_HOST=snapshot_host)
assert static_asset.status_code == 200
assert static_asset.headers.get("ETag")
assert_cache(static_asset, "public", "max-age=31536000", "immutable")
static_not_modified = public_client.get(
"/static/jquery.min.js",
HTTP_HOST=snapshot_host,
HTTP_IF_NONE_MATCH=static_asset["ETag"],
)
assert static_not_modified.status_code == 304
assert_cache(static_not_modified, "public", "max-age=31536000", "immutable")
# Tightening permissions must still run authorization before honoring
# a stale validator presented in a new request.
snapshot.config = {**snapshot.config, "PERMISSIONS": "private"}
snapshot.save(update_fields=["config"])
snapshot.refresh_from_db()
assert snapshot.permissions == "private"
revoked = Client().get(
f"/{raw_name}",
HTTP_HOST=snapshot_host,
HTTP_IF_NONE_MATCH=public_raw["ETag"],
)
assert revoked.status_code in (301, 302)
assert "/admin/core/snapshot/replay-auth/" in revoked["Location"]
assert_cache(revoked, "private", "no-store")
assert_auth_vary(revoked)
private_files = Client().get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
assert private_files.status_code in (301, 302)
assert "/admin/core/snapshot/replay-auth/" in private_files["Location"]
assert_cache(private_files, "private", "no-store")
assert_auth_vary(private_files)
admin_client = Client()
assert admin_client.login(username=admin.username, password="testpassword")
grant = admin_client.get(
f"/admin/core/snapshot/replay-auth/?snapshot={snapshot.id}&next=/{raw_name}",
HTTP_HOST=get_admin_host(),
)
assert grant.status_code in (301, 302)
grant_url = urlsplit(grant["Location"])
replay_client = Client()
replay_auth = replay_client.get(f"{grant_url.path}?{grant_url.query}", HTTP_HOST=snapshot_host)
assert replay_auth.status_code in (301, 302)
assert_cache(replay_auth, "private", "no-store")
private_raw = replay_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host)
assert private_raw.status_code == 200
assert response_body(private_raw) == raw_bytes
assert_cache(private_raw, "private", "max-age=604800", "immutable")
assert_auth_vary(private_raw)
private_index = replay_client.get("/index.html", HTTP_HOST=snapshot_host)
assert private_index.status_code == 200
assert_cache(private_index, "private", "max-age=60", "stale-while-revalidate=300")
assert_auth_vary(private_index)
private_zip = replay_client.get("/?files=1&download=zip", HTTP_HOST=snapshot_host)
assert private_zip.status_code == 200
assert private_zip["Content-Type"] == "application/zip"
assert response_body(private_zip).startswith(b"PK")
assert_cache(private_zip, "private", "max-age=60", "stale-while-revalidate=300")
private_metadata = replay_client.get("/index.jsonl", HTTP_HOST=snapshot_host)
assert private_metadata.status_code == 200
assert response_body(private_metadata) == metadata_path.read_bytes()
assert_cache(private_metadata, "private", "max-age=31536000", "immutable")
private_preview = replay_client.get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
assert private_preview.status_code == 200
assert private_preview["Content-Type"].startswith("text/html")
assert_cache(private_preview, "private", "max-age=60", "stale-while-revalidate=300")
private_range = replay_client.get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
assert private_range.status_code == 206
assert response_body(private_range) == raw_bytes[:8]
assert_cache(private_range, "private", "max-age=604800", "immutable")
private_not_modified = replay_client.get(
f"/{raw_name}",
HTTP_HOST=snapshot_host,
HTTP_IF_NONE_MATCH=private_raw["ETag"],
)
assert private_not_modified.status_code == 304
assert_cache(private_not_modified, "private", "max-age=31536000", "immutable")
assert_auth_vary(private_not_modified)
private_missing = replay_client.get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
assert private_missing.status_code == 404
assert_cache(private_missing, "private", "no-store")
private_redirect = admin_client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
assert private_redirect.status_code in (301, 302)
assert private_redirect["Location"].startswith(f"http://{snapshot_host}")
assert_cache(private_redirect, "private", "no-store")
snapshot.config = {**snapshot.config, "PERMISSIONS": "unlisted"}
snapshot.save(update_fields=["config"])
snapshot.refresh_from_db()
assert snapshot.permissions == "unlisted"
unlisted_raw = Client().get(f"/{raw_name}", HTTP_HOST=snapshot_host)
assert unlisted_raw.status_code == 200
assert response_body(unlisted_raw) == raw_bytes
assert_cache(unlisted_raw, "private", "max-age=604800", "immutable")
unlisted_index = Client().get("/index.html", HTTP_HOST=snapshot_host)
assert unlisted_index.status_code == 200
assert_cache(unlisted_index, "private", "max-age=60", "stale-while-revalidate=300")
unlisted_files = Client().get(f"/{snapshot.url_path}/index.html?files=1", HTTP_HOST=get_web_host())
assert unlisted_files.status_code == 200
assert raw_name.encode() in response_body(unlisted_files)
assert_cache(unlisted_files, "private", "max-age=60", "stale-while-revalidate=300")
unlisted_zip = Client().get(
f"/{snapshot.url_path}/index.html?files=1&download=zip",
HTTP_HOST=get_web_host(),
)
assert unlisted_zip.status_code == 200
assert unlisted_zip["Content-Type"] == "application/zip"
assert response_body(unlisted_zip).startswith(b"PK")
assert_cache(unlisted_zip, "private", "max-age=60", "stale-while-revalidate=300")
unlisted_metadata = Client().get("/index.jsonl", HTTP_HOST=snapshot_host)
assert unlisted_metadata.status_code == 200
assert response_body(unlisted_metadata) == metadata_path.read_bytes()
assert_cache(unlisted_metadata, "private", "max-age=31536000", "immutable")
unlisted_preview = Client().get(f"/{raw_name}?preview=1", HTTP_HOST=snapshot_host)
assert unlisted_preview.status_code == 200
assert unlisted_preview["Content-Type"].startswith("text/html")
assert_cache(unlisted_preview, "private", "max-age=60", "stale-while-revalidate=300")
unlisted_range = Client().get(f"/{raw_name}", HTTP_HOST=snapshot_host, HTTP_RANGE="bytes=0-7")
assert unlisted_range.status_code == 206
assert response_body(unlisted_range) == raw_bytes[:8]
assert_cache(unlisted_range, "private", "max-age=604800", "immutable")
unlisted_not_modified = Client().get(
f"/{raw_name}",
HTTP_HOST=snapshot_host,
HTTP_IF_NONE_MATCH=unlisted_raw["ETag"],
)
assert unlisted_not_modified.status_code == 304
assert_cache(unlisted_not_modified, "private", "max-age=31536000", "immutable")
unlisted_missing = Client().get("/missing-cache-policy-output", HTTP_HOST=snapshot_host)
assert unlisted_missing.status_code == 404
assert_cache(unlisted_missing, "private", "no-store")
unlisted_redirect = Client().get(f"/{snapshot.url_path}/index.html", HTTP_HOST=get_admin_host())
assert unlisted_redirect.status_code in (301, 302)
assert unlisted_redirect["Location"].startswith(f"http://{snapshot_host}")
assert_cache(unlisted_redirect, "private", "no-store")
# Snapshots materialize their crawl's permission when no explicit
# snapshot override is supplied, and each effective permission drives caching.
for inherited_permission, expected_cache in (
("public", ("public", "max-age=604800", "immutable")),
("unlisted", ("private", "max-age=604800", "immutable")),
("private", ("private", "no-store")),
):
crawl = Crawl.objects.create(
urls=f"https://inherited-{inherited_permission}.example/",
created_by=admin,
config={"PERMISSIONS": inherited_permission},
)
inherited = Snapshot.objects.create(
url=f"https://inherited-{inherited_permission}.example/",
crawl=crawl,
config={},
)
inherited.refresh_from_db()
assert inherited.permissions == inherited_permission
inherited.output_dir.mkdir(parents=True, exist_ok=True)
(inherited.output_dir / raw_name).write_bytes(raw_bytes)
inherited_host = get_snapshot_host(str(inherited.id))
response = Client().get(f"/{raw_name}", HTTP_HOST=inherited_host)
if inherited_permission == "private":
assert response.status_code in (301, 302)
else:
assert response.status_code == 200
assert response_body(response) == raw_bytes
assert_cache(response, *expected_cache)
print("OK")
finally:
Snapshot.objects.filter(pk=snapshot.pk).update(config=original_config)
""",
mode="safe-subdomains-fullreplay",
)
def test_admin_login_next_allows_archivebox_hosts_only(self) -> None:
self._run(
"""
@@ -634,7 +1018,7 @@ class TestUrlRouting:
allowed_next_urls = [
"/admin/core/snapshot/",
f"http://archivebox.localhost:8000/public/",
f"http://archivebox.localhost:5797/public/",
f"http://{web_host}/public/",
f"http://{admin_host}/admin/core/snapshot/",
f"http://{api_host}/api/v1/docs",
@@ -1205,7 +1589,7 @@ class TestUrlRouting:
mode="safe-subdomains-fullreplay",
)
finally:
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
def test_subdomain_replay_assets_route_without_base_url(self) -> None:
lib_dir = self.data_dir / "test-lib"
@@ -1233,7 +1617,7 @@ class TestUrlRouting:
env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
)
finally:
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
def test_subdomain_replay_assets_use_derived_chromewebstore_extensions_dir(self) -> None:
lib_dir = self.data_dir / "test-lib"
@@ -1262,7 +1646,7 @@ class TestUrlRouting:
env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
)
finally:
self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
self._set_config("BIND_ADDR=127.0.0.1:5797", "BASE_URL=http://archivebox.localhost:5797")
def test_onedomain_base_url_overrides_are_preserved_for_external_links(self) -> None:
try:
@@ -1289,7 +1673,7 @@ class TestUrlRouting:
mode="safe-onedomain-nojsreplay",
)
finally:
self._set_config("BASE_URL=http://archivebox.localhost:8000")
self._set_config("BASE_URL=http://archivebox.localhost:5797")
def test_subdomain_snapshot_urls_inherit_https_archive_base_url(self) -> None:
try:
@@ -1310,7 +1694,7 @@ class TestUrlRouting:
mode="safe-subdomains-fullreplay",
)
finally:
self._set_config("BASE_URL=http://archivebox.localhost:8000")
self._set_config("BASE_URL=http://archivebox.localhost:5797")
def test_template_and_admin_links(self) -> None:
self._run(
@@ -1343,7 +1727,7 @@ class TestUrlRouting:
resp = client.get("/public/", HTTP_HOST=web_host)
assert resp.status_code in (301, 302)
assert resp["Location"] == "http://admin.archivebox.localhost:8000/admin/core/snapshot/"
assert resp["Location"] == "http://admin.archivebox.localhost:5797/admin/core/snapshot/"
resp = client.get(f"/{snapshot.url_path}/index.html", HTTP_HOST=web_host)
assert resp.status_code == 200
@@ -1518,12 +1902,12 @@ class TestUrlRouting:
# Exercise the same real captured files with each request-time posture,
# including both auto branches and an HTTPS canonical reverse-proxy URL.
for mode, base in (
("auto", "http://archivebox.localhost:8000"),
("auto", "http://archivebox.example:8000"),
("auto", "http://archivebox.localhost:5797"),
("auto", "http://archivebox.example:5797"),
("safe-subdomains-fullreplay", "https://archivebox.example"),
("safe-onedomain-nojsreplay", "http://archivebox.example:8000"),
("unsafe-onedomain-noadmin", "http://archivebox.example:8000"),
("danger-onedomain-fullreplay", "http://archivebox.example:8000"),
("safe-onedomain-nojsreplay", "http://archivebox.example:5797"),
("unsafe-onedomain-noadmin", "http://archivebox.example:5797"),
("danger-onedomain-fullreplay", "http://archivebox.example:5797"),
):
self._run(
"""
@@ -8,7 +8,7 @@ class Command(BaseCommand):
parser.add_argument(
"--bind-url",
default="",
help="Runserver bind URL to watch, e.g. http://127.0.0.1:8000",
help="Runserver bind URL to watch, e.g. http://127.0.0.1:5797",
)
parser.add_argument(
"--interval",
+18 -4
View File
@@ -330,13 +330,18 @@ def SUPERVISORD_PARENT_WATCHDOG_WORKER(
}
SERVER_WORKER = lambda host, port: {
SERVER_WORKER = lambda host, port, extra_port=None: {
"name": "worker_daphne",
"command": _shell_join(
[
str(resolve_env_binary("daphne")),
f"--bind={host}",
f"--port={port}",
*(
["--endpoint", "tcp:port={}:interface={}".format(extra_port, host.strip("[]").replace(":", r"\:"))]
if extra_port is not None
else []
),
"archivebox.core.asgi:application",
],
),
@@ -1115,7 +1120,7 @@ def format_runtime_components(components: list[str] | tuple[str, ...]) -> str:
def worker_runtime_component(worker_name: str) -> str | None:
if worker_name in {"worker_runner", "worker_runner_watch"} or worker_name.startswith("worker_runner_"):
return "orchestrator"
if worker_name in {"worker_daphne", "worker_runserver"}:
if worker_name in {"worker_daphne", "worker_runserver", "worker_runserver_compatibility"}:
return "server"
if worker_name == "worker_sonic":
return "sonic"
@@ -1140,6 +1145,9 @@ def active_supervisord_runtime_components(*, supervisor=None) -> list[str]:
def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reload: bool, nothreading: bool, supervisor=None):
bind_url = f"http://{host}:{port}"
# Docker does not expose its host port mappings to the container. Keep both
# image defaults reachable so existing mappings survive an image upgrade.
extra_port = {"5797": "8000", "8000": "5797"}.get(str(port)) if config.IN_DOCKER else None
if debug:
server_worker = RUNSERVER_WORKER(host=host, port=port, reload=reload, nothreading=nothreading)
@@ -1150,7 +1158,7 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
if reload:
log_files.insert(1, "logs/worker_runner_watch.log")
else:
server_worker = SERVER_WORKER(host=host, port=port)
server_worker = SERVER_WORKER(host=host, port=port, extra_port=extra_port)
bg_workers = [(RUNNER_WORKER(), False)]
log_files = ["logs/worker_daphne.log", "logs/worker_runner.log"]
@@ -1178,6 +1186,12 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
log_files.append(str(sonic_worker["stdout_logfile"]))
workers = [(server_worker, False), *bg_workers]
if debug and extra_port is not None:
compatibility_worker = RUNSERVER_WORKER(host=host, port=extra_port, reload=reload, nothreading=nothreading)
compatibility_worker["name"] = "worker_runserver_compatibility"
compatibility_worker["stdout_logfile"] = "logs/worker_runserver_compatibility.log"
workers.insert(1, (compatibility_worker, False))
log_files.append(compatibility_worker["stdout_logfile"])
if os.environ.get("DISPLAY") and (config.IN_DOCKER or os.environ.get("ARCHIVEBOX_VNC_PERSONA")):
vnc_persona = os.environ.get("ARCHIVEBOX_VNC_PERSONA") or config.DEFAULT_PERSONA
workers.append(
@@ -1478,7 +1492,7 @@ def stop_stale_sonic_processes(
def start_server_workers(
host="0.0.0.0",
port="8000",
port="5797",
daemonize=False,
debug=False,
reload=False,
+1 -1
View File
@@ -111,7 +111,7 @@ if [[ -n "$DEPLOY_EXPECT_VERSION" && "$VERSION_OUTPUT" != *"ArchiveBox v${DEPLOY
fi
echo "[+] Health check:"
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:8000/health/ </dev/null
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:5797/health/ </dev/null
REMOTE
echo "[√] Demo deploy finished."
+12 -6
View File
@@ -230,11 +230,14 @@ assert_writable_dir "$DATA_DIR/logs"
assert_writable_dir "$DATA_DIR/archive"
assert_writable_dir "$PERSONAS_DIR/Default/chrome_profile"
# check if novnc x11 $DISPLAY is available
export DISPLAY="${DISPLAY:-"novnc:0.0"}"
if ! xdpyinfo > /dev/null 2>&1; then
# cant connect to x11 display, unset it so that chrome doesn't try to connect to it and hang indefinitely
unset DISPLAY
# Preserve an explicit DISPLAY (including an empty value to disable X11).
# Only probe the implicit noVNC default: a missing probe or unavailable server
# must not silently discard the user's display configuration.
if [[ -z "${DISPLAY+x}" ]]; then
export DISPLAY="novnc:0.0"
if ! xdpyinfo > /dev/null 2>&1; then
unset DISPLAY
fi
fi
# Active browser processes do not survive container restarts, but their lock
@@ -250,6 +253,9 @@ find /tmp "$TMP_DIR" -maxdepth 1 -type d -name "archivebox-chrome-profile.*" -mm
ensure_dir "/home/$ARCHIVEBOX_USER"
# A nested read-only browser-profile mount can create this parent as root.
# OpenCode's installer needs to create a sibling config directory as archivebox.
ensure_dir "${XDG_CONFIG_HOME:-/home/$ARCHIVEBOX_USER/.config}"
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR"
ensure_small_runtime_tree "$ABXBUS_CACHE_DIR/semaphores"
ensure_small_runtime_tree "$UV_CACHE_DIR"
@@ -365,7 +371,7 @@ else
# e.g. "docker run archivebox help"
# "docker run archivebox add --depth=1 https://example.com"
# "docker run archivebox manage createsupseruser"
# "docker run archivebox server 0.0.0.0:8000"
# "docker run archivebox server 0.0.0.0:5797"
if [[ "$(id -u)" == "0" ]]; then
exec setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$ARCHIVEBOX_BIN_PATH" "$@"
else
+2 -1
View File
@@ -128,7 +128,8 @@ def render_gallery(gallery_header: str, gallery_sections: str) -> str:
return (
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
"<title>ArchiveBox UI Screenshots</title><style>"
+ (REPO_DIR / "bin/templates/screenshots-head.html").read_text(encoding="utf-8")
+ "<style>"
":root{--ink:#28252a;--muted:#716c74;--accent:#9b2854;--paper:#fcfaf7;--line:#e9e2e3}"
"*{box-sizing:border-box}body{margin:0;font:16px/1.75 system-ui,sans-serif;"
"background:var(--paper);color:var(--ink)}.screenshot-content{max-width:1100px;margin:0 auto;padding:28px}"
+2 -2
View File
@@ -39,8 +39,8 @@ if os.environ.get("GH_TOKEN"):
# Published captures remain available even after Actions artifacts expire.
base, manifest_name = {
"archivebox": ("https://archivebox.io/screenshots/", "build.json"),
"ios": ("https://archivebox.github.io/ios-archivebox/screenshots/", "manifest.json"),
"extension": ("https://archivebox.github.io/archivebox-browser-extension/screenshots/", "manifest.json"),
"ios": ("https://app.archivebox.io/screenshots/", "manifest.json"),
"extension": ("https://extension.archivebox.io/screenshots/", "manifest.json"),
}[args.kind]
+14 -14
View File
@@ -189,9 +189,9 @@ docker_run_archivebox_install() {
docker_run_archivebox_server() {
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
"$DOCKER_BINARY" run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
"$DOCKER_BINARY" run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 5797:5797 --name=archivebox "$ARCHIVEBOX_IMAGE"
else
"$DOCKER_BINARY" run -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
"$DOCKER_BINARY" run -v "$PWD":/data -d -p 5797:5797 --name=archivebox "$ARCHIVEBOX_IMAGE"
fi
}
@@ -204,8 +204,8 @@ docker_compose_run_archivebox() {
}
wait_for_archivebox() {
url="http://127.0.0.1:8000/health/"
host_header="admin.archivebox.localhost:8000"
url="http://127.0.0.1:5797/health/"
host_header="admin.archivebox.localhost:5797"
attempts=60
attempt=1
@@ -224,7 +224,7 @@ wait_for_archivebox() {
open_archivebox() {
if [ -n "$OPEN_BINARY" ] && [ -t 1 ]; then
"$OPEN_BINARY" "http://admin.archivebox.localhost:8000/admin/" || true
"$OPEN_BINARY" "http://admin.archivebox.localhost:5797/admin/" || true
fi
}
@@ -410,13 +410,13 @@ if [ "$DOCKER_IMAGE_READY" = "true" ] && "$DOCKER_BINARY" compose version > /dev
wait_for_archivebox
open_archivebox
echo
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo " cd $ARCHIVEBOX_HOME_DIR"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose ps"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose down"
echo " ${FOLLOWUP_SUDO}env ARCHIVEBOX_IMAGE=$ARCHIVEBOX_IMAGE $CONTAINER_ENGINE_NAME compose pull"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose up"
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose exec archivebox archivebox add 'https://example.com'"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME compose exec archivebox archivebox list"
@@ -438,13 +438,13 @@ elif [ "$DOCKER_IMAGE_READY" = "true" ]; then
wait_for_archivebox
open_archivebox
echo
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo " cd $ARCHIVEBOX_DATA_DIR"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME ps --filter name=archivebox"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME rm -f archivebox"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME pull $ARCHIVEBOX_IMAGE"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -d -p 8000:8000 --name=archivebox $ARCHIVEBOX_IMAGE"
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -d -p 5797:5797 --name=archivebox $ARCHIVEBOX_IMAGE"
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -it $ARCHIVEBOX_IMAGE add 'https://example.com'"
echo " ${FOLLOWUP_SUDO}$CONTAINER_ENGINE_NAME run $DOCKER_PLATFORM_ARGS -v $PWD:/data -it $ARCHIVEBOX_IMAGE list"
@@ -501,14 +501,14 @@ select_archivebox_lib_dir
# init shows version output at the end too
echo
echo "[+] Starting ArchiveBox server using: archivebox server --daemonize..."
"$ARCHIVEBOX_BINARY" server --daemonize 0.0.0.0:8000
"$ARCHIVEBOX_BINARY" server --daemonize 0.0.0.0:5797
wait_for_archivebox
open_archivebox
echo
echo "[√] Server started on http://127.0.0.1:8000 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo "[√] Server started on http://127.0.0.1:5797 and data directory initialized in $ARCHIVEBOX_DATA_DIR. Usage:"
echo " cd $ARCHIVEBOX_DATA_DIR # see your data dir"
echo " ${FOLLOWUP_SUDO}archivebox server 0.0.0.0:8000 # start server process"
echo " Open http://localhost:8000/admin/ to create the first admin and finish web setup."
echo " ${FOLLOWUP_SUDO}archivebox server 0.0.0.0:5797 # start server process"
echo " Open http://localhost:5797/admin/ to create the first admin and finish web setup."
echo " (When running remotely, replace localhost with this server's IP address or hostname.)"
echo " ps aux | grep archivebox # see server process pid"
echo " ${FOLLOWUP_SUDO}pkill -f 'archivebox server' # stop the server"
+1 -1
View File
@@ -4,7 +4,7 @@ const fs = require('fs');
const path = require('path');
const puppeteer = require('puppeteer');
const DEFAULT_URL = 'http://web.archivebox.localhost:8000/add/';
const DEFAULT_URL = 'http://web.archivebox.localhost:5797/add/';
const DEFAULT_OUTPUT = 'tmp/add-page.png';
function usage() {
+1 -1
View File
@@ -1,6 +1,6 @@
<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
<div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
<div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abxpkg.archivebox.io/">Package &ampLine truncated
<div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
</footer>
+25
View File
@@ -0,0 +1,25 @@
<title>ArchiveBox UI Screenshots</title>
<link rel="canonical" href="https://archivebox.io/screenshots/">
<meta name="description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
<meta name="robots" content="index,follow,max-image-preview:large">
<meta name="theme-color" content="#9b2854">
<meta property="og:type" content="website">
<meta property="og:site_name" content="ArchiveBox">
<meta property="og:locale" content="en_US">
<meta property="og:url" content="https://archivebox.io/screenshots/">
<meta property="og:title" content="ArchiveBox UI Screenshots">
<meta property="og:description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
<meta property="og:image" content="https://archivebox.io/assets/social-card.png">
<meta property="og:image:type" content="image/png">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="630">
<meta property="og:image:alt" content="ArchiveBox. ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@ArchiveBoxApp">
<meta name="twitter:title" content="ArchiveBox UI Screenshots">
<meta name="twitter:description" content="ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
<meta name="twitter:image" content="https://archivebox.io/assets/social-card.png">
<meta name="twitter:image:alt" content="ArchiveBox. ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.">
<link rel="icon" type="image/x-icon" href="https://archivebox.io/assets/favicon.ico">
<link rel="icon" type="image/png" sizes="192x192" href="https://archivebox.io/assets/icon.png">
<link rel="apple-touch-icon" sizes="180x180" href="https://archivebox.io/assets/apple-touch-icon.png">
+8 -1
View File
@@ -1,7 +1,14 @@
<header class="abx-header">
<a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">UI Screenshots</span></a>
<nav class="abx-nav" aria-label="Main navigation">
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
<details class="abx-apps"><summary>Apps</summary><div class="abx-app-links">
<div class="abx-app-row"><a class="abx-app-name" href="https://archivebox.io/#install-compose">ArchiveBox Server</a><div class="abx-platforms"><a class="abx-platform abx-platform-docker" href="https://archivebox.io/#install-compose" aria-label="ArchiveBox Server for Docker" title="ArchiveBox Server for Docker"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M349.9 236.3l-66.1 0 0-59.4 66.1 0 0 59.4zm0-204.3l-66.1 0 0 60.7 66.1 0 0-60.7zm78.2 144.8l-66.1 0 0 59.4 66.1 0 0-59.4zM271.8 104.7l-66.1 0 0 60.1 66.1 0 0-60.1zm78.1 0l-66.1 0 0 60.1 66.1 0 0-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7l-434.7 0c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zM115.6 176.8l-66 0 0 59.4 66.1 0 0-59.4-.1 0zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm-78.1-72.1l-66.1 0 0 60.1 66.1 0 0-60.1z"/></svg></a><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/#archivebox-serverapp" aria-label="ArchiveBox Server for macOS" title="ArchiveBox Server for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://electron.archivebox.io/" aria-label="ArchiveBox Server for Windows" title="ArchiveBox Server for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://archivebox.io/#install-apt" aria-label="ArchiveBox Server for Linux" title="ArchiveBox Server for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-1Line truncated
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Desktop</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Desktop for macOS" title="ArchiveBox Desktop for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://windows.archivebox.io/" aria-label="ArchiveBox Desktop for Windows" title="ArchiveBox Desktop for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://linux.archivebox.io/" aria-label="ArchiveBox Desktop for Linux" title="ArchiveBox Desktop for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-10 14.1-10.5 4.7-4.4 13.5-14.2 27.9-14.2 7.1 0 15.6 2.3 25.9 8.9 6.3 4.1 11.3 4.4 22.6 9.3 8.4 3.5 13.7 9.7 10.5 18.2-2.6 7.1-11 14.4-22.7 18.1-11.1 3.6-19.8 16-38.2 14.9-3.9-.2-7-1-9.6-2.1-8-3.5-12.2-10.4-20-15-8.6-4.8-13.2-10.4-14.7-15.3-1.4-4.9 0-9 4.2-12.3zm3.3 334c-2.7 35.1-43.9 34.4-75.3 18-29.9-15.8-68.6-6.5-76.5-21.9-2.4-4.7-2.4-12.7 2.6-26.4l0-.2c2.4-7.6 .6-16-.6-23.9-1.2-7.8-1.8-15 .9-20 3.5-6.7 8.5-9.1 14.8-11.3 10.3-3.7 11.8-3.4 19.6-9.9 5.5-5.7 9.5-12.9 14.3-18 5.1-5.5 10-8.1 17.7-6.9 8.1 1.2 15.1 6.8 21.9 16l19.6 35.6c9.5 19.9 43.1 48.4 41 68.9zm-1.4-25.9c-4.1-6.6-9.6-13.6-14.4-19.6 7.1 0 14.2-2.2 16.7-8.9 2.3-6.2 0-14.9-7.4-24.9-13.5-18.2-38.3-32.5-38.3-32.5-13.5-8.4-21.1-18.7-24.6-29.9s-3-23.3-.3-35.2c5.2-22.9 18.6-45.2 27.2-59.2 2.3-1.7 .8 3.2-8.7 20.8-8.5 16.1-24.4 53.3-2.6 82.4 .6-20.7 5.5-41.8 13.8-61.5 12-27.4 37.3-74.9 39.3-112.7 1.1 .8 4.6 3.2 6.2 4.1 4.6 2.7 8.1 6.7 12.6 10.3 12.4 10 28.5 9.2 42.4 1.2 6.2-3.5 11.2-7.5 15.9-9 9.9-3.1 17.8-8.6 22.3-15 7.7 30.4 25.7 74.3 37.2 95.7 6.1 11.4 18.3 35.5 23.6 64.6 3.3-.1 7 .4 10.9 1.4 13.8-35.7-11.7-74.2-23.3-84.9-4.7-4.6-4.9-6.6-2.6-6.5 12.6 11.2 29.2 33.7 35.2 59 2.8 11.6 3.3 23.7 .4 35.7 16.4 6.8 35.9 17.9 30.7 34.8-2.2-.1-3.2 0-4Line truncated
<div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Mobile</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Mobile for iOS" title="ArchiveBox Mobile for iOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-android" href="https://android.archivebox.io/" aria-label="ArchiveBox Mobile for Android" title="ArchiveBox Mobile for Android"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!-- Font Awesome Free 5.0.13 by Fonticons, Inc. https://fontawesome.com License: https://creativecommons.org/licenses/by/4.0/ --><path d="M89.6 204.5v115.8c0 15.4-12.1 27.7-27.5 27.7-15.3 0-30.1-12.4-30.1-27.7V204.5c0-15.1 14.8-27.5 30.1-27.5 15.1 0 27.5 12.4 27.5 27.5zm10.8 157c0 16.4 13.2 29.6 29.6 29.6h19.9l.3 61.1c0 36.9 55.2 36.6 55.2 0v-61.1h37.2v61.1c0 36.7 55.5 36.8 55.5 0v-61.1h20.2c16.2 0 29.4-13.2 29.4-29.6V182.1H100.4v179.4zm248-189.1H99.3c0-42.8 25.6-80 63.6-99.4l-19.1-35.3c-2.8-4.9 4.3-8 6.7-3.8l19.4 35.6c34.9-15.5 75-14.7 108.3 0L297.5 34c2.5-4.3 9.5-1.1 6.7 3.8L285.1 73c37.7 19.4 63.3 56.6 63.3 99.4zm-170.7-55.5c0-5.7-4.6-10.5-10.5-10.5-5.7 0-10.2 4.8-10.2 10.5s4.6 10.5 10.2 10.5c5.9 0 10.5-4.8 10.5-10.5zm113.4 0c0-5.7-4.6-10.5-10.2-10.5-5.9 0-10.5 4.8-10.5 10.5s4.6 10.5 10.5 10.5c5.6 0 10.2-4.8 10.2-10.5zm94.8 60.1c-15.1 0-27.5 12.1-27.5 27.5v115.8c0 15.4 12.4 27.7 27.5 27.7 15.4 0 30.1-12.4 30.1-27.7V204.5c0-15.4-14.8-27.5-30.1-27.5z"/></svg></a></div></div>
<div class="abx-app-row"><a class="abx-app-name" href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><div class="abx-platforms"><a class="abx-platform abx-platform-chrome" href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj" aria-label="ArchiveBox Browser Extension for Chrome" title="ArchiveBox Browser Extension for Chrome"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2024 Fonticons, Inc. --><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg></a><a class="abx-platform abx-platform-safari" href="https://extension.archivebox.io/" aria-label="ArchiveBox Browser Extension for Safari" title="ArchiveBox Browser Extension for Safari"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M274.7 274.7L237.3 237.3 166 346 274.7 274.7zM256 8a248 248 0 1 0 0 496 248 248 0 1 0 0-496zM411.8 182.8l14.8-6.1c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.8 1 2.6 1.7 1.3 1.6 1.7 2.6 .6 2 .6 3.1-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.3-2.6 1.7L418 197.6c-2 .8-4.2 .8-6.1 0s-3.5-2.4-4.3-4.3-.8-4.2 0-6.1 2.4-3.5 4.3-4.3zM314.4 94l6.1-14.8c.8-2 2.4-3.5 4.3-4.3s4.2-.8 6.1 0 3.5 2.4 4.3 4.3 .8 4.2 0 6.1l-6.1 14.8c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0c-1-.4-1.9-1-2.6-1.7s-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1zM256 60c2.1 0 4.2 .8 5.7 2.3S264 65.9 264 68l0 16c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7S253.9 60 256 60zM181 74.9c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.9 1 2.6 1.7 1.3 1.6 1.7 2.6L197.6 94c.4 1 .7 2 .7 3.1s-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.4-2.6 1.8-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.8-1.3-1.7-1.7-2.7l-6.1-14.8c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3zm-63.6 42.5c1.5-1.5 3.5-2.3 5.7-2.3s4.2 .8 5.7 2.3L140 128.7c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3l-11.3-11.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6l0 0zM60 256c0-2.1 .8-4.2 2.3-5.7S65.9 248 68 248l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3l-16 0c-2.1 0-4.2-.8-5.7-2.3S60 258.1 60 256zm40.2 73.2l-14.8 6.1c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1 1-1.9 1.7-2.6 1.6-1.3 2.6-1.7L94 314.4c2-.8 4.2-.8 6.1 0s3.5 2.4 4.3 4.3 .8 4.2 0 6.1-2.4 3.5-4.3 4.3zm4.3-136c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1 2.4-3.5 4.3-4.3 4.2-.8 6.1 0l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1l0 0zM197.6 418l-6.1 14.8c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1l6.1-14.8c.4-1 1-1.9 1.7-2.7s1.6-1.4 2.6-1.8 2-.6 3.1-.6 2.1 .2 3.1 .6 1.9 1 2.6 1.8 1.3 1.7 1.7 2.6 .6 2 .6 3.1-.2 2.1-.7 3.1zM264 444c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7s3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3 2.3 3.5 2.3 5.7l0 16zm67-6.9c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6L314.4 418c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3 4.2-.8 6.1 0 3.5 2.4 4.3 4.3l6.1 14.8c.8 2 .8 4.2 0 6.1s-2.4 3.5-4.3 4.3zm63.6-42.5c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3L372 383.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6 3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3l11.3 11.3c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6l0 0zM286.2 286.2L110.3 401.7 225.8 225.8 401.7 110.3 286.2 286.2zM437.1 331c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1c.4-1 1-1.9 1.7-2.6s1.6-1.3 2.6-1.7 2-.6 3.1-.6 2.1 .2 3.1 .6l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1zm6.9-67l-16 0c-2.1 0-4.2-.8-5.7-2.3s-2.3-3.5-2.3-5.7 .8-4.2 2.3-5.7 3.5-2.3 5.7-2.3l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3z"/></svg></a><a class="abx-platform abx-platform-firefox" href="https://addons.mozilla.org/firefox/addon/archivebox-exporter/" aria-label="ArchiveBox Browser Extension for Firefox" title="ArchiveBox Browser Extension for Firefox"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512Line truncated
<div class="abx-app-row abx-plugin-row"><a class="abx-app-name" href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><div class="abx-platforms abx-icon-stack"><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Google Drive storage" title="Google Drive storage"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m38.7 419.3 22.6 39c4.7 8.2 11.4 14.7 19.4 19.4l80.6-139.6H0c0 9.1 2.3 18.2 7 26.4z" style="fill:#0066da"/><path d="M256 173.9 175.4 34.3c-7.9 4.7-14.7 11.1-19.4 19.4L7 311.7c-4.6 8-7 17.1-7 26.4h161.3z" style="fill:#00ac47"/><path d="M431.4 477.7c7.9-4.7 14.7-11.1 19.4-19.4l9.4-16.1 44.9-77.7c4.7-8.2 7-17.3 7-26.4H350.7l34.3 67.4z" style="fill:#ea4335"/><path d="m256 173.9 80.6-139.6c-7.9-4.7-17-7-26.4-7H201.8c-9.4 0-18.5 2.6-26.4 7z" style="fill:#00832d"/><path d="M350.7 338.1H161.3L80.6 477.7c7.9 4.7 17 7 26.4 7h298c9.4 0 18.5-2.6 26.4-7z" style="fill:#2684fc"/><path d="M430.5 182.7 356 53.7c-4.7-8.2-11.4-14.7-19.4-19.4L256 173.9l94.7 164.2h161c0-9.1-2.3-18.2-7-26.4z" style="fill:#ffba00"/></svg></a><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Amazon S3 storage" title="Amazon S3 storage"><svg aria-hidden="true" focusable="false" fill="#569A31" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M20.913 13.147l.12-.895c.947.576 1.258.922 1.354 1.071-.16.031-.562.046-1.474-.176zm-2.174 7.988a.547.547 0 0 0-.005.073c0 .084-.207.405-1.124.768a10.28 10.28 0 0 1-1.438.432c-1.405.325-3.128.504-4.853.504-4.612 0-7.412-1.184-7.412-1.704a.547.547 0 0 0-.005-.073L1.81 5.602c.135.078.28.154.432.227.042.02.086.038.128.057.134.062.272.122.417.18l.179.069c.154.058.314.114.478.168.043.013.084.029.13.043.207.065.423.127.646.187l.176.044c.175.044.353.087.534.127a23.414 23.414 0 0 0 .843.17l.121.023c.252.045.508.085.768.122.071.011.144.02.216.03.2.027.4.053.604.077l.24.027c.245.026.49.05.74.07l.081.009c.275.022.552.04.83.056l.233.012c.21.01.422.018.633.025a33.088 33.088 0 0 0 2.795-.026l.232-.011c.278-.016.555-.034.83-.056l.08-.008c.25-.02.497-.045.742-.072l.238-.026c.205-.024.408-.05.609-.077.07-.01.141-.019.211-.03.261-.037.519-.078.772-.122l.111-.02c.215-.04.427-.082.634-.125l.212-.047c.186-.041.368-.085.546-.13l.166-.042c.225-.06.444-.122.654-.189.04-.012.077-.026.115-.038a10.6 10.6 0 0 0 .493-.173c.058-.021.114-.044.17-.066.15-.06.293-.12.43-.185.038-.017.079-.034.116-.052.153-.073.3-.15.436-.228l-.976 7.245c-2.488-.78-5.805-2.292-7.311-3a1.09 1.09 0 0 0-1.088-1.085c-.6 0-1.088.489-1.088 1.088 0 .6.488 1.089 1.088 1.089.196 0 .378-.056.537-.148 1.72.812 5.144 2.367 7.715 3.15zm-7.42-20.047c5.677 0 9.676 1.759 9.75 2.736l-.014.113c-.01.033-.031.067-.048.101-.015.028-.026.057-.047.087-.024.033-.058.068-.09.102-.028.03-.051.06-.084.09-.038.035-.087.07-.133.105-.04.03-.074.06-.119.091-.053.036-.116.071-.177.107-.05.03-.095.06-.15.09-.068.036-.147.073-.222.11-.059.028-.114.057-.177.085-.084.038-.177.074-.268.111-.068.027-.13.054-.203.082-.097.036-.205.072-.31.107-.075.026-.148.053-.228.079-.111.035-.233.069-.35.103-.085.024-.165.05-.253.073-.124.034-.258.065-.389.098-.093.022-.181.046-.278.068-.139.032-.287.061-.433.091-.098.02-.191.041-.293.06-.155.03-.32.057-.482.084-.1.018-.198.036-.302.052-.166.026-.342.048-.515.072-.11.014-.213.03-.325.044-.181.023-.372.041-.56.06-.11.012-.218.025-.332.036-.188.016-.386.029-.58.043-.122.009-.24.02-.364.028-.207.012-.422.02-.635.028-.12.005-.234.012-.354.016a35.605 35.605 0 0 1-2.069 0c-.12-.004-.234-.011-.352-.016-.214-.008-.43-.016-.637-.028-.122-.008-.238-.02-.36-.027-.195-.015-.394-.028-.584-.044-.11-.01-.215-.024-.324-.035-.19-.02-.384-.038-.568-.06l-.315-.044c-.176-.024-.355-.046-.525-.073-.1-.015-.192-.033-.29-.05-.167-.028-.335-.055-.494-.086-.096-.018-.183-.038-.276-.056-.151-.032-.305-.062-.45-.095-.09-.02-.173-.043-.26-.064-.138-.034-.277-.067-.407-.102-.082-.022-.157-.046-.235-.069a11.75 11.75 0 0 1-.368-.108c-.075-.024-.141-.049-.213-.073-.11-.037-.223-.075-.325-.113-.067-.025-.125-.051-.188-.077-.096-.038-.195-.076-.282-.115-.06-.027-.11-.054-.166-.08-.08-.039-.162-.077-.233-.116-.052-.028-.094-.055-.142-.084-.063-.038-.13-.075-.185-.113-.043-.029-.075-.058-.113-.086-.048-.037-.098-.073-.139-.11-.032-.029-.054-.057-.08-.087-.033-.035-.069-.07-.093-.104-.02-.03-.031-.058-.046-.086-.018-.035-.039-.068-.049-.102l-.015-.113c.076-.977 4.074-2.736 9.748-2.736zm12.182 12.124c-.118-.628-.84-1.291-2.31-2.128l.963-7.16a.531.531 0 0 0 .005-.073C22.16 1.581 16.447 0 11.32 0 6.194 0 .482 1.581.482 3.851a.58.58 0 0 0 .005.072L2.819 21.25c.071 2.002 5.236 2.75 8.5 2.75 1.805 0 3.615-.188 5.098-.531.598-.138 1.133-.3 1.592-.48 1.18-.467 1.789-1.053 1.813-1.739l.945-7.018c.557.131 1.016.197 1.389.197.54 0 .902-.137 1.134-.413aLine truncated
<div class="abx-app-row abx-tools-row"><a class="abx-app-name" href="https://abx-dl.archivebox.io/">Tools</a><div class="abx-platforms"><a class="abx-tool" href="https://abx-dl.archivebox.io/"><span>One-shot <code>abx-dl</code> CLI</span></a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/mcp">MCP</a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage">API</a><a class="abx-tool" href="https://web.archivebox.io">Demo</a></div></div>
</div></details>
<a href="https://archivebox.io/screenshots/">Screenshots</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
<a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
<a class="abx-cta" href="https://archivebox.io/#install">Get started</a>
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
# Real Compose regression checks. Build the image under test first, then run:
# IMAGE=archivebox/archivebox:issue-1872 ./bin/validate_docker_display.sh
set -Eeuo pipefail
export IMAGE="${IMAGE:-archivebox/archivebox:dev}"
validation_dir="$(mktemp -d "${TMPDIR:-/tmp}/archivebox-display.XXXXXX")"
compose=(docker compose --project-name "archivebox-display-$$" --file "$validation_dir/compose.yml")
cleanup() {
"${compose[@]}" down --volumes --remove-orphans
echo "Validation files: $validation_dir"
}
trap cleanup EXIT
cat > "$validation_dir/compose.yml" <<'YAML'
services:
archivebox:
image: ${IMAGE}
volumes:
- ./data:/data
- ./data/personas:/data/personas
- tmp:/tmp/archivebox
shm_size: 1gb
novnc:
image: theasp/novnc:latest
environment:
DISPLAY_WIDTH: 1280
DISPLAY_HEIGHT: 800
RUN_XTERM: "no"
healthcheck:
test: ["CMD", "xdpyinfo", "-display", ":0"]
interval: 2s
timeout: 2s
retries: 10
volumes:
tmp:
YAML
"${compose[@]}" up -d --wait novnc
# Check the real runtime dependency as well as successful X11 connectivity.
"${compose[@]}" run --rm archivebox sh -ec '
command -v xdpyinfo
test "$DISPLAY" = novnc:0.0
xdpyinfo >/dev/null
'
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
test "$DISPLAY" = novnc:0.0
xdpyinfo >/dev/null
'
# Explicit values belong to the caller, even when X11 is unavailable.
"${compose[@]}" run --rm -e DISPLAY=unavailable:0 archivebox sh -ec '
test "$DISPLAY" = unavailable:0
'
# An explicit empty value disables automatic display discovery.
"${compose[@]}" run --rm -e DISPLAY= archivebox sh -ec '
test "${DISPLAY+x}" = x
test -z "$DISPLAY"
'
"${compose[@]}" stop novnc
"${compose[@]}" run --rm archivebox sh -ec '
test "${DISPLAY+x}" != x
'
"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
test "$DISPLAY" = novnc:0.0
'
echo 'PASS: Compose DISPLAY discovery, explicit override, empty opt-out, and headless fallback'
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
set -Eeuo pipefail
IMAGE="${IMAGE:-archivebox/archivebox:dev}"
DOCKER_BINARY="${DOCKER_BINARY:-docker}"
CURL_BINARY="${CURL_BINARY:-curl}"
name="archivebox-port-check-$$"
volume="$name-data"
cookies=$(mktemp)
cleanup() {
status=$?
if [[ "$status" != 0 ]]; then "$DOCKER_BINARY" logs "$name" || true; fi
"$DOCKER_BINARY" rm -fv "$name" >/dev/null 2>&1 || true
"$DOCKER_BINARY" volume rm "$volume" >/dev/null
rm -f "$cookies"
exit "$status"
}
"$DOCKER_BINARY" volume create "$volume" >/dev/null
trap cleanup EXIT
"$DOCKER_BINARY" run --rm -v "$volume:/data" "${OLD_IMAGE:-$IMAGE}" init
"$DOCKER_BINARY" run --rm -v "$volume:/data" "${OLD_IMAGE:-$IMAGE}" config --set BIND_ADDR=0.0.0.0:8000
"$DOCKER_BINARY" run --rm -v "$volume:/data" \
-e DJANGO_SUPERUSER_USERNAME=portcheck -e DJANGO_SUPERUSER_EMAIL=portcheck@example.com \
-e DJANGO_SUPERUSER_PASSWORD="$name-password" "${OLD_IMAGE:-$IMAGE}" manage createsuperuser --noinput
for mode in default legacy debug; do
command=()
if [[ "$mode" == legacy ]]; then command=(server 0.0.0.0:8000); fi
if [[ "$mode" == debug ]]; then command=(server --debug 0.0.0.0:5797); fi
"$DOCKER_BINARY" run -d --name "$name" -v "$volume:/data" \
-p 127.0.0.1::5797 -p 127.0.0.1::8000 "$IMAGE" "${command[@]}" >/dev/null
for port in 5797 8000; do
address=$("$DOCKER_BINARY" port "$name" "$port/tcp")
ready=0
for ((attempt=0; attempt<60; attempt++)); do
if "$CURL_BINARY" -fsS --max-time 2 "http://$address/health/" >/dev/null 2>&1; then ready=1; break; fi
[[ $("$DOCKER_BINARY" inspect -f '{{.State.Running}}' "$name") == true ]]
sleep 1
done
[[ "$ready" == 1 ]]
body=$("$CURL_BINARY" -fsS --max-time 10 "http://$address/health/")
[[ "$body" == *OK* ]]
echo "PASS: $mode server responds through Docker mapping $address -> $port"
origin="http://admin.archivebox.localhost:${address##*:}"
if [[ "$port" == 5797 ]]; then
login=$("$CURL_BINARY" -fsS --max-time 10 -c "$cookies" "$origin/admin/login/?next=/admin/")
csrf=$(printf '%s' "$login" | sed -n 's/.*name="csrfmiddlewaretoken" value="\([^"]*\)".*/\1/p')
[[ -n "$csrf" ]]
"$CURL_BINARY" -fsS --max-time 10 -b "$cookies" -c "$cookies" -e "$origin/admin/login/" \
--data-urlencode username=portcheck --data-urlencode "password=$name-password" \
--data-urlencode "csrfmiddlewaretoken=$csrf" --data-urlencode next=/admin/ \
"$origin/admin/login/" >/dev/null
fi
page=$("$CURL_BINARY" -fsS --max-time 10 -b "$cookies" "$origin/admin/crawls/crawl/")
[[ "$page" == *model-crawl* ]]
[[ "$page" != *'name="password"'* ]]
echo "PASS: $mode authenticated Crawls works on $port with the same browser session"
done
"$DOCKER_BINARY" rm -fv "$name" >/dev/null
done
+13 -3
View File
@@ -9,8 +9,12 @@ DEPLOY_IMAGE="${DEPLOY_IMAGE:-archivebox/archivebox:dev}"
DEPLOY_INTERVAL="${DEPLOY_INTERVAL:-60}"
STATE_FILE="${STATE_FILE:-${DEPLOY_PATH}/.archivebox-dev-image.digest}"
OVERRIDE_FILE="${OVERRIDE_FILE:-${DEPLOY_PATH}/.archivebox-deploy.override.yml}"
DEPLOY_BASE_OVERRIDE="${DEPLOY_BASE_OVERRIDE:-}"
DEPLOY_HEALTH_HOST="${DEPLOY_HEALTH_HOST:-admin.archivebox.io}"
DEPLOY_HEALTH_PORT="${DEPLOY_HEALTH_PORT:-5797}"
DEPLOY_HEALTH_PATH="${DEPLOY_HEALTH_PATH:-/health/}"
ABXPKG_LIB_DIR="${ABXPKG_LIB_DIR:-$HOME/.config/archivebox/lib}"
DOCKER_BINARY="$ABXPKG_LIB_DIR/env/bin/docker"
DOCKER_BINARY="${DOCKER_BINARY:-$ABXPKG_LIB_DIR/env/bin/docker}"
test -x "$DOCKER_BINARY"
cd "$DEPLOY_PATH"
@@ -35,7 +39,12 @@ services:
command: null
EOF
COMPOSE=("$DOCKER_BINARY" compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE")
COMPOSE=("$DOCKER_BINARY" compose -f "$COMPOSE_FILE")
if [[ -n "$DEPLOY_BASE_OVERRIDE" ]]; then
test -f "$DEPLOY_BASE_OVERRIDE"
COMPOSE+=(-f "$DEPLOY_BASE_OVERRIDE")
fi
COMPOSE+=(-f "$OVERRIDE_FILE")
remote_digest() {
local line
@@ -71,7 +80,8 @@ deploy_digest() {
printf '%s\n' "$line"
line_count=$((line_count + 1))
done <<<"$version_output"
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:8000/health/ </dev/null
"${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" /opt/archivebox/lib/env/bin/curl -fsS --max-time 10 --connect-timeout 2 \
-H "Host: $DEPLOY_HEALTH_HOST" "http://127.0.0.1:${DEPLOY_HEALTH_PORT}${DEPLOY_HEALTH_PATH}" </dev/null
printf '%s\n' "$digest" > "$STATE_FILE"
}
+2 -2
View File
@@ -3,7 +3,7 @@
# curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
# docker compose pull
# docker compose up -d --wait
# Open /admin/ on the hostname or IP used to reach ArchiveBox (local: http://admin.archivebox.localhost:8000/admin/).
# Open /admin/ on the hostname or IP used to reach ArchiveBox (local: http://admin.archivebox.localhost:5797/admin/).
# docker compose exec archivebox archivebox add --depth=1 'https://news.ycombinator.com'
# docker compose exec -T archivebox archivebox add < ~/Downloads/bookmarks.txt
@@ -20,7 +20,7 @@ services:
# novnc:
# condition: service_healthy
ports:
- "${ARCHIVEBOX_PORT:-8000}:8000"
- "${ARCHIVEBOX_PORT:-5797}:5797"
volumes:
- ./data:/data
# Override abx-dl's nested VOLUME so profiles persist across containers.
+17 -1
View File
@@ -149,10 +149,26 @@ ssh -N -L 8080:127.0.0.1:8080 user@your-server
4. Open a page and confirm you can view it while logged in. For a new persona, log in to the sites you want to archive. Then follow [Save and check the page](#3-save-and-check-the-page), selecting the same persona.
#### One-off Compose setup
To set up a persona without starting the ArchiveBox server, enable the same `novnc` service and healthy `depends_on` block, then run:
```bash
docker compose up -d --wait novnc
docker compose run --rm archivebox persona create personal
docker compose run --rm -e DISPLAY=novnc:0.0 archivebox persona open personal
```
Connect at <http://localhost:8080/vnc.html>, log in, and close the browser when finished. The persona remains in `data/personas/personal` across one-off containers.
`persona open` resolves the installed Chromium binary automatically. Do not hard-code `/data/lib/env/bin/chromium` or a versioned Playwright cache path: the browser location varies by image version and architecture.
An explicitly set `DISPLAY` is preserved. When it is unset, ArchiveBox probes `novnc:0.0` and falls back to headless operation if unavailable. Set `DISPLAY=` to disable this automatic display selection.
## Troubleshooting Chromium Install
<a name="more-info--troubleshooting"></a>
- **Browser missing or unable to start:** see [Chromium troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#chromiumgoogle-chrome).
- **Saved a login screen:** confirm you can view the page in the source browser, re-import the correct profile, and select that persona when saving.
- **Need browser settings:** see the [Chrome plugin configuration](https://archivebox.github.io/abx-plugins/#chrome).
- **Need browser settings:** see the [Chrome plugin configuration](https://plugins.archivebox.io/#chrome).
+26 -26
View File
@@ -68,7 +68,7 @@ archivebox add --no-only-new https://example.com # force a re-archive even
**Possible Values:** [`60`]/`120`/...
Maximum allowed runtime **per-extractor, per-Snapshot** in seconds. If you have a slow network connection or are seeing frequent timeout errors, you can raise this value.
This is a *plugin-shared* setting — each individual extractor can override it with its own `<EXTRACTOR>_TIMEOUT` (e.g. [`WGET_TIMEOUT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_TIMEOUT`](https://archivebox.github.io/abx-plugins/#chrome), [`YTDLP_TIMEOUT`](https://archivebox.github.io/abx-plugins/#ytdlp)). See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for the full list.
This is a *plugin-shared* setting — each individual extractor can override it with its own `<EXTRACTOR>_TIMEOUT` (e.g. [`WGET_TIMEOUT`](https://plugins.archivebox.io/#wget), [`CHROME_TIMEOUT`](https://plugins.archivebox.io/#chrome), [`YTDLP_TIMEOUT`](https://plugins.archivebox.io/#ytdlp)). See the [per-plugin docs](https://plugins.archivebox.io/) for the full list.
> [!NOTE]
> `TIMEOUT` only caps a single extractor invocation. To bound the *total* wall-clock runtime of an entire crawl, use [`CRAWL_TIMEOUT`](#crawl_timeout) instead.
@@ -84,14 +84,14 @@ This is a *plugin-shared* setting — each individual extractor can override it
**Possible Values:** [`1440,2000`]/`1024,768`/...
Default screenshot/PDF viewport resolution in `width,height` pixels. Used as the fallback for `SCREENSHOT_RESOLUTION`, `PDF_RESOLUTION`, and `CHROME_RESOLUTION`.
This is a *plugin-shared* setting — individual extractors override it via `<EXTRACTOR>_RESOLUTION` (e.g. [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`PDF_RESOLUTION`](https://archivebox.github.io/abx-plugins/#pdf), [`CHROME_RESOLUTION`](https://archivebox.github.io/abx-plugins/#chrome)). See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for plugin-specific overrides.
This is a *plugin-shared* setting — individual extractors override it via `<EXTRACTOR>_RESOLUTION` (e.g. [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`PDF_RESOLUTION`](https://plugins.archivebox.io/#pdf), [`CHROME_RESOLUTION`](https://plugins.archivebox.io/#chrome)). See the [per-plugin docs](https://plugins.archivebox.io/) for plugin-specific overrides.
---
#### `CHECK_SSL_VALIDITY`
**Possible Values:** [`True`]/`False`
Whether to enforce HTTPS certificate validity and HSTS chain of trust when archiving sites. Set this to `False` if you want to archive pages even if they have expired or invalid certificates.
This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](https://archivebox.github.io/abx-plugins/#wget), [`yt-dlp`](https://archivebox.github.io/abx-plugins/#ytdlp), [`gallery-dl`](https://archivebox.github.io/abx-plugins/#gallerydl), [`chrome`](https://archivebox.github.io/abx-plugins/#chrome), etc.) honors it, and individual extractors can override with `<EXTRACTOR>_CHECK_SSL_VALIDITY`. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/).
This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](https://plugins.archivebox.io/#wget), [`yt-dlp`](https://plugins.archivebox.io/#ytdlp), [`gallery-dl`](https://plugins.archivebox.io/#gallerydl), [`chrome`](https://plugins.archivebox.io/#chrome), etc.) honors it, and individual extractors can override with `<EXTRACTOR>_CHECK_SSL_VALIDITY`. See the [per-plugin docs](https://plugins.archivebox.io/).
> [!WARNING]
> When `False`, ArchiveBox cannot guarantee that the captured content matches the real site — a man-in-the-middle could substitute responses. Only disable for trusted networks or for archiving legacy/internal sites with expired certs.
@@ -101,7 +101,7 @@ This is a *plugin-shared* setting — every HTTP-fetching extractor ([`wget`](ht
**Possible Values:** [`Mozilla/5.0 ... ArchiveBox/{VERSION} ...`]/`"Mozilla/5.0 ..."`/...
The default `User-Agent` string sent during archiving. The built-in default identifies ArchiveBox and links back to the GitHub repo so site operators can identify and contact archivers if needed.
This is a *plugin-shared* setting — each extractor ([`wget`](https://archivebox.github.io/abx-plugins/#wget), [`chrome`](https://archivebox.github.io/abx-plugins/#chrome), [`yt-dlp`](https://archivebox.github.io/abx-plugins/#ytdlp), [`singlefile`](https://archivebox.github.io/abx-plugins/#singlefile), …) can override it with its own `<EXTRACTOR>_USER_AGENT`, otherwise it falls back to this value. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for per-extractor specifics.
This is a *plugin-shared* setting — each extractor ([`wget`](https://plugins.archivebox.io/#wget), [`chrome`](https://plugins.archivebox.io/#chrome), [`yt-dlp`](https://plugins.archivebox.io/#ytdlp), [`singlefile`](https://plugins.archivebox.io/#singlefile), …) can override it with its own `<EXTRACTOR>_USER_AGENT`, otherwise it falls back to this value. See the [per-plugin docs](https://plugins.archivebox.io/) for per-extractor specifics.
> [!NOTE]
> Some sites block requests that look like bots or that don't match a real browser. If you're getting 403s or empty responses, try setting this to a current Chrome/Firefox UA string.
@@ -115,7 +115,7 @@ This is a *plugin-shared* setting — each extractor ([`wget`](https://archivebo
Path to a [Netscape-format `cookies.txt`](http://www.cookiecentral.com/faq/#3.5) file passed to `wget`, `curl`, `yt-dlp`, and other non-Chrome extractors for authentication. Required when archiving sites behind a login (paywalls, social media feeds, members-only forums, etc.) **if you're not using a persona**.
This is a *plugin-shared* setting — each extractor can override it with `<EXTRACTOR>_COOKIES_FILE` (e.g. [`WGET_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#gallerydl)). [Chrome](https://archivebox.github.io/abx-plugins/#chrome)-based extractors instead read auth state from the persona's `CHROME_USER_DATA_DIR`. See the [per-plugin docs](https://archivebox.github.io/abx-plugins/) for per-extractor variants.
This is a *plugin-shared* setting — each extractor can override it with `<EXTRACTOR>_COOKIES_FILE` (e.g. [`WGET_COOKIES_FILE`](https://plugins.archivebox.io/#wget), [`YTDLP_COOKIES_FILE`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://plugins.archivebox.io/#gallerydl)). [Chrome](https://plugins.archivebox.io/#chrome)-based extractors instead read auth state from the persona's `CHROME_USER_DATA_DIR`. See the [per-plugin docs](https://plugins.archivebox.io/) for per-extractor variants.
You can generate a `cookies.txt` using a [browser extension](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc), or with `wget --save-cookies` + `--user=... --password=...`.
@@ -130,7 +130,7 @@ archivebox add --persona=personal https://members.example.com/feed
> **Use separate burner credentials dedicated to archiving** — don't re-use your normal daily Facebook/Instagram/Youtube/etc. account cookies as server responses often contain your name/email/PII and session tokens, which then get preserved in your snapshots forever!
*Related options:*
[`DEFAULT_PERSONA`](#default_persona), [`CHROME_USER_DATA_DIR`](https://archivebox.github.io/abx-plugins/#chrome)
[`DEFAULT_PERSONA`](#default_persona), [`CHROME_USER_DATA_DIR`](https://plugins.archivebox.io/#chrome)
---
<a id="active_persona"></a>
@@ -353,17 +353,17 @@ Django's secret key, used for cryptographic signing of sessions, CSRF tokens, pa
---
#### `BIND_ADDR`
**Possible Values:** [`127.0.0.1:8000`]/`0.0.0.0:8000`/`[::]:8000`/`0.0.0.0:80`/...
**Possible Values:** [`127.0.0.1:5797`]/`0.0.0.0:5797`/`[::]:5797`/`0.0.0.0:80`/...
The `host:port` socket the ArchiveBox web server actually listens on. **This is the local bind socket, not the public URL** — for the public URL clients see, set [`BASE_URL`](#base_url).
- `127.0.0.1:8000` (default) — listen only on the loopback interface. Safest when you're running a reverse proxy on the same host and don't want the server reachable directly from the network.
- `0.0.0.0:8000` — listen on **all** IPv4 interfaces. Required when running in Docker without `--network=host`, or when you want the server reachable from other machines on your LAN without a reverse proxy.
- `[::]:8000` — listen on all IPv6 interfaces (most modern OSes will accept v4-mapped connections too).
IPv6 literal addresses must be bracketed: `[::1]:8000`, not `::1:8000`.
- `127.0.0.1:5797` (default) — listen only on the loopback interface. Safest when you're running a reverse proxy on the same host and don't want the server reachable directly from the network.
- `0.0.0.0:5797` — listen on **all** IPv4 interfaces. Required when running in Docker without `--network=host`, or when you want the server reachable from other machines on your LAN without a reverse proxy.
- `[::]:5797` — listen on all IPv6 interfaces (most modern OSes will accept v4-mapped connections too).
IPv6 literal addresses must be bracketed: `[::1]:5797`, not `::1:5797`.
> [!NOTE]
> Inside Docker, binding to `127.0.0.1` means the server is unreachable from outside the container — use `0.0.0.0:8000` and let Docker handle the port-forwarding, or publish the port with `-p 127.0.0.1:8000:8000` on the host side instead.
> Inside Docker, binding to `127.0.0.1` means the server is unreachable from outside the container — use `0.0.0.0:5797` and let Docker handle the port-forwarding, or publish the port with `-p 127.0.0.1:5797:5797` on the host side instead.
*Related options:*
[`BASE_URL`](#base_url), [`SERVER_SECURITY_MODE`](#server_security_mode)
@@ -372,7 +372,7 @@ IPv6 literal addresses must be bracketed: `[::1]:8000`, not `::1:8000`.
<a id="allowed_hosts"></a>
<a id="csrf_trusted_origins"></a>
#### `BASE_URL`
**Possible Values:** [`""`]/`https://archive.example.com`/`http://archivebox.localhost:8000`/...
**Possible Values:** [`""`]/`https://archive.example.com`/`http://archivebox.localhost:5797`/...
The canonical public URL of your ArchiveBox instance. Used to build absolute links in templates, redirects (`/admin/login/?next=...`), admin notification emails, OG/meta tags, and — in subdomain security mode — to derive the `admin.`, `web.`, `api.`, and per-snapshot `snap-<id>.` subdomains.
@@ -778,7 +778,7 @@ Lower values retry more aggressively (useful if you expect locks to clear quickl
ArchiveBox can index Snapshot text/HTML output into searchable indexes that power the search bar in the Web UI and the `archivebox search <query>` CLI command. Multiple backend engines can be enabled at once; `SEARCH_BACKEND_ENGINE` selects the default used by the UI and CLI.
> [!NOTE]
> Each backend has its own tuning knobs (e.g. [Sonic](https://archivebox.github.io/abx-plugins/#search_backend_sonic) host/port, [ripgrep](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) flags, [SQLite FTS](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) database path). Those backend-specific options now live with the plugin that implements them — see the [abx-plugins docs](https://archivebox.github.io/abx-plugins/) for the full per-backend schema.
> Each backend has its own tuning knobs (e.g. [Sonic](https://plugins.archivebox.io/#search_backend_sonic) host/port, [ripgrep](https://plugins.archivebox.io/#search_backend_ripgrep) flags, [SQLite FTS](https://plugins.archivebox.io/#search_backend_sqlite) database path). Those backend-specific options now live with the plugin that implements them — see the [abx-plugins docs](https://plugins.archivebox.io/) for the full per-backend schema.
---
#### `SEARCH_BACKEND_ENGINE`
@@ -786,13 +786,13 @@ ArchiveBox can index Snapshot text/HTML output into searchable indexes that powe
Which search backend engine to use when running `archivebox search` and rendering the Web UI search bar.
- **`ripgrep`** — Pure filesystem grep across each Snapshot's archived output (HTML, text, metadata) via the [`search_backend_ripgrep`](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) plugin. No extra daemon, no extra database to maintain — just install `rg` and it works. Slow on very large collections (each query re-scans the disk) but always 100% correct: results reflect what's actually on disk *right now*, no stale index. ArchiveBox keeps it enabled as the fallback when Sonic is unavailable.
- **`ripgrep`** — Pure filesystem grep across each Snapshot's archived output (HTML, text, metadata) via the [`search_backend_ripgrep`](https://plugins.archivebox.io/#search_backend_ripgrep) plugin. No extra daemon, no extra database to maintain — just install `rg` and it works. Slow on very large collections (each query re-scans the disk) but always 100% correct: results reflect what's actually on disk *right now*, no stale index. ArchiveBox keeps it enabled as the fallback when Sonic is unavailable.
- **`sonic`** *(default)* — Fast, suggest-style fuzzy search via a running [Sonic](https://github.com/valeriansaliou/sonic) daemon (configured via the [`search_backend_sonic`](https://archivebox.github.io/abx-plugins/#search_backend_sonic) plugin). ArchiveBox pushes text into Sonic at index time and queries it at search time. Sub-millisecond queries even at very large scale; ArchiveBox starts the managed service automatically when this backend is selected.
- **`sonic`** *(default)* — Fast, suggest-style fuzzy search via a running [Sonic](https://github.com/valeriansaliou/sonic) daemon (configured via the [`search_backend_sonic`](https://plugins.archivebox.io/#search_backend_sonic) plugin). ArchiveBox pushes text into Sonic at index time and queries it at search time. Sub-millisecond queries even at very large scale; ArchiveBox starts the managed service automatically when this backend is selected.
- **`sqlite`** — FTS5 full-text index stored alongside ArchiveBox's main `index.sqlite3`, configured via the [`search_backend_sqlite`](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) plugin. No extra processes, no extra binary — uses the SQLite already shipped with Python. Faster than `ripgrep` on large collections, slightly slower than `sonic`, but no daemon to babysit. Good middle ground for users who want a real index without operational overhead.
- **`sqlite`** — FTS5 full-text index stored alongside ArchiveBox's main `index.sqlite3`, configured via the [`search_backend_sqlite`](https://plugins.archivebox.io/#search_backend_sqlite) plugin. No extra processes, no extra binary — uses the SQLite already shipped with Python. Faster than `ripgrep` on large collections, slightly slower than `sonic`, but no daemon to babysit. Good middle ground for users who want a real index without operational overhead.
*Note: Backend-specific tuning ([Sonic](https://archivebox.github.io/abx-plugins/#search_backend_sonic) host/port/password, [ripgrep](https://archivebox.github.io/abx-plugins/#search_backend_ripgrep) flag overrides, [SQLite FTS](https://archivebox.github.io/abx-plugins/#search_backend_sqlite) database path, indexer batch size, etc.) lives in each search-backend plugin's own config schema — see the [abx-plugins docs](https://archivebox.github.io/abx-plugins/) for the full per-backend option list.*
*Note: Backend-specific tuning ([Sonic](https://plugins.archivebox.io/#search_backend_sonic) host/port/password, [ripgrep](https://plugins.archivebox.io/#search_backend_ripgrep) flag overrides, [SQLite FTS](https://plugins.archivebox.io/#search_backend_sqlite) database path, indexer batch size, etc.) lives in each search-backend plugin's own config schema — see the [abx-plugins docs](https://plugins.archivebox.io/) for the full per-backend option list.*
---
@@ -1087,21 +1087,21 @@ SHOW_PROGRESS=False archivebox add < urls.txt
> **Per-plugin configuration has moved to its own documentation site.**
> This `Configuration.md` doc covers only ArchiveBox's *core* settings. For everything that lives inside a plugin — extractor toggles, binary paths, timeouts, args, user agents, cookies, persona scoping, etc. — see:
>
> ## ➡️ **<https://archivebox.github.io/abx-plugins/>**
> ## ➡️ **<https://plugins.archivebox.io/>**
That site is regenerated from each plugin's `config.json` schema on every release, so it stays in sync with the code. Looking for [`WGET_ARGS`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_USER_DATA_DIR`](https://archivebox.github.io/abx-plugins/#chrome), [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`YTDLP_EXTRA_ARGS`](https://archivebox.github.io/abx-plugins/#ytdlp), [`SINGLEFILE_*`](https://archivebox.github.io/abx-plugins/#singlefile), [`SONIC_HOST`](https://archivebox.github.io/abx-plugins/#search_backend_sonic), etc.? They all live there now.
That site is regenerated from each plugin's `config.json` schema on every release, so it stays in sync with the code. Looking for [`WGET_ARGS`](https://plugins.archivebox.io/#wget), [`CHROME_USER_DATA_DIR`](https://plugins.archivebox.io/#chrome), [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`YTDLP_EXTRA_ARGS`](https://plugins.archivebox.io/#ytdlp), [`SINGLEFILE_*`](https://plugins.archivebox.io/#singlefile), [`SONIC_HOST`](https://plugins.archivebox.io/#search_backend_sonic), etc.? They all live there now.
### Shared core options that plugins fall back to
A handful of *core* options (documented above on this page) act as the **fallback default** for every plugin that has a matching per-extractor override. If you set the core option, every plugin honors it; if you also set the plugin-specific override, that wins for just that one plugin.
| Core option (this doc) | Plugin-level overrides (see [abx-plugins](https://archivebox.github.io/abx-plugins/)) |
| Core option (this doc) | Plugin-level overrides (see [abx-plugins](https://plugins.archivebox.io/)) |
|---|---|
| [`TIMEOUT`](#timeout) | [`WGET_TIMEOUT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_TIMEOUT`](https://archivebox.github.io/abx-plugins/#chrome), [`YTDLP_TIMEOUT`](https://archivebox.github.io/abx-plugins/#ytdlp), [`SINGLEFILE_TIMEOUT`](https://archivebox.github.io/abx-plugins/#singlefile), [`TITLE_TIMEOUT`](https://archivebox.github.io/abx-plugins/#title), [`FAVICON_TIMEOUT`](https://archivebox.github.io/abx-plugins/#favicon), ... |
| [`CHECK_SSL_VALIDITY`](#check_ssl_validity) | [`WGET_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#gallerydl), [`CHROME_CHECK_SSL_VALIDITY`](https://archivebox.github.io/abx-plugins/#chrome), ... |
| [`USER_AGENT`](#user_agent) | [`WGET_USER_AGENT`](https://archivebox.github.io/abx-plugins/#wget), [`CHROME_USER_AGENT`](https://archivebox.github.io/abx-plugins/#chrome), ... |
| [`COOKIES_FILE`](#cookies_file) | [`WGET_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#wget), [`YTDLP_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://archivebox.github.io/abx-plugins/#gallerydl), ... |
| [`RESOLUTION`](#resolution) | [`SCREENSHOT_RESOLUTION`](https://archivebox.github.io/abx-plugins/#screenshot), [`PDF_RESOLUTION`](https://archivebox.github.io/abx-plugins/#pdf), [`CHROME_RESOLUTION`](https://archivebox.github.io/abx-plugins/#chrome) |
| [`TIMEOUT`](#timeout) | [`WGET_TIMEOUT`](https://plugins.archivebox.io/#wget), [`CHROME_TIMEOUT`](https://plugins.archivebox.io/#chrome), [`YTDLP_TIMEOUT`](https://plugins.archivebox.io/#ytdlp), [`SINGLEFILE_TIMEOUT`](https://plugins.archivebox.io/#singlefile), [`TITLE_TIMEOUT`](https://plugins.archivebox.io/#title), [`FAVICON_TIMEOUT`](https://plugins.archivebox.io/#favicon), ... |
| [`CHECK_SSL_VALIDITY`](#check_ssl_validity) | [`WGET_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#wget), [`YTDLP_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#gallerydl), [`CHROME_CHECK_SSL_VALIDITY`](https://plugins.archivebox.io/#chrome), ... |
| [`USER_AGENT`](#user_agent) | [`WGET_USER_AGENT`](https://plugins.archivebox.io/#wget), [`CHROME_USER_AGENT`](https://plugins.archivebox.io/#chrome), ... |
| [`COOKIES_FILE`](#cookies_file) | [`WGET_COOKIES_FILE`](https://plugins.archivebox.io/#wget), [`YTDLP_COOKIES_FILE`](https://plugins.archivebox.io/#ytdlp), [`GALLERYDL_COOKIES_FILE`](https://plugins.archivebox.io/#gallerydl), ... |
| [`RESOLUTION`](#resolution) | [`SCREENSHOT_RESOLUTION`](https://plugins.archivebox.io/#screenshot), [`PDF_RESOLUTION`](https://plugins.archivebox.io/#pdf), [`CHROME_RESOLUTION`](https://plugins.archivebox.io/#chrome) |
> [!TIP]
> The resolution order for any plugin-tunable option is always:
+12 -10
View File
@@ -71,7 +71,7 @@ docker compose pull
docker compose up -d --wait
```
Open <http://admin.archivebox.localhost:8000> and follow the setup wizard to create the first admin and configure web access. Existing `BASE_URL` and security settings are used as-is, so configured servers skip the web-access wizard.
Open <http://admin.archivebox.localhost:5797> and follow the setup wizard to create the first admin and configure web access. Existing `BASE_URL` and security settings are used as-is, so configured servers skip the web-access wizard.
ArchiveBox installs and enables both ripgrep and [Sonic](https://github.com/valeriansaliou/sonic). Sonic is selected by default in the UI, while ripgrep remains available as the fallback. To select ripgrep explicitly:
```bash
@@ -148,7 +148,9 @@ Alternatively, to use the web UI, start the server with:
docker compose up # add -d to run in the background
```
Then open [`http://web.archivebox.localhost:8000`](http://web.archivebox.localhost:8000) for the public UI or [`http://admin.archivebox.localhost:8000`](http://admin.archivebox.localhost:8000) for the admin UI.
Then open [`http://web.archivebox.localhost:5797`](http://web.archivebox.localhost:5797) for the public UI or [`http://admin.archivebox.localhost:5797`](http://admin.archivebox.localhost:5797) for the admin UI.
Use port **5797** for new deployments. The Docker container also listens on the old port **8000** so existing port mappings remain compatible.
<br/>
@@ -162,7 +164,7 @@ docker compose exec archivebox archivebox config --set TIMEOUT=120
# OR edit ./data/ArchiveBox.conf and add this under its existing [ARCHIVING_CONFIG] section:
TIMEOUT=120
# plugin-specific options work the same way (see https://archivebox.github.io/abx-plugins/)
# plugin-specific options work the same way (see https://plugins.archivebox.io/)
docker compose exec archivebox archivebox config --set YTDLP_MAX_SIZE=750m
```
This will apply the config to all containers or archivebox instances that access the collection.
@@ -184,11 +186,11 @@ services:
...
```
For public HTTPS, start the default stack with `docker compose up -d`, use port `8000` only as the temporary setup/upstream endpoint, and follow the first-run wizard. It gives the DNS, upstream, and certificate settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI, then verifies the public HTTPS URLs before saving `BASE_URL` and `SERVER_SECURITY_MODE`.
For public HTTPS, start the default stack with `docker compose up -d`, use port `5797` only as the temporary setup/upstream endpoint, and follow the first-run wizard. It gives the DNS, upstream, and certificate settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI, then verifies the public HTTPS URLs before saving `BASE_URL` and `SERVER_SECURITY_MODE`.
Use exactly one of these certificate layouts:
- **Single-domain mode:** one certificate for the `BASE_URL` hostname, proxied to ArchiveBox port `8000`.
- **Single-domain mode:** one certificate for the `BASE_URL` hostname, proxied to ArchiveBox port `5797`.
- **Isolated-subdomain mode:** one certificate covering both the `BASE_URL` hostname and `*.BASE_URL`, normally obtained through DNS-01.
Never enable on-demand TLS or request individual certificates for `snap-*` hostnames.
@@ -211,10 +213,10 @@ Fetch and run the ArchiveBox Docker image. Starting the server creates the initi
docker pull archivebox/archivebox:dev
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
```
Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local example: <http://admin.archivebox.localhost:8000/admin/>) to create the first admin. If `BASE_URL` is not configured yet, continue through the web setup wizard.
Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local example: <http://admin.archivebox.localhost:5797/admin/>) to create the first admin. If `BASE_URL` is not configured yet, continue through the web setup wizard.
*(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
@@ -270,8 +272,8 @@ docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox:d
Then to view your data, you can look in the folder on the host `/media/USB-DRIVE/archivebox/data`, or use the Web UI:
```bash
docker run -it -v /media/USB-DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox:dev
# then open http://web.archivebox.localhost:8000
docker run -it -v /media/USB-DRIVE/archivebox/data:/data -p 5797:5797 archivebox/archivebox:dev
# then open http://web.archivebox.localhost:5797
```
<br/>
@@ -287,7 +289,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev config --set TIMEOUT=120
TIMEOUT=120
```
ArchiveBox in Docker also accepts config as environment variables, see more on the [[Configuration]] page (and the [abx-plugins config reference](https://archivebox.github.io/abx-plugins/) for per-plugin options).
ArchiveBox in Docker also accepts config as environment variables, see more on the [[Configuration]] page (and the [abx-plugins config reference](https://plugins.archivebox.io/) for per-plugin options).
For example, this disables the screenshot extractor for a single run (without persisting for other runs):
```bash
+5 -5
View File
@@ -33,7 +33,7 @@ Other operating systems are not tested or supported for this release.
<br/>
You can reduce crawl-time memory pressure by disabling all Chrome-based archiving methods with [`CHROME_ENABLED=False`](https://archivebox.github.io/abx-plugins/#chrome) (or its `USE_CHROME` alias).
You can reduce crawl-time memory pressure by disabling all Chrome-based archiving methods with [`CHROME_ENABLED=False`](https://plugins.archivebox.io/#chrome) (or its `USE_CHROME` alias).
It's also recommended to use a filesystem with compression and/or [deduplication](https://www.ixsystems.com/blog/ixsystems-and-klara-systems-celebrate-valentines-day-with-a-heartfelt-donation-of-fast-dedupe-to-openzfs-and-truenas/) (e.g. [ZFS](https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html) or BTRFS) for maximum efficiency.
@@ -94,13 +94,13 @@ After running the setup script, continue with the [Quickstart](https://github.co
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [`uv` install shortcut](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `uv`, `apt`, or Homebrew.
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://archivebox.github.io/abx-plugins/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://archivebox.github.io/abx-plugins/#pdf), [`SCREENSHOT_ENABLED=False`](https://archivebox.github.io/abx-plugins/#screenshot), and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom) you don't need `chromium`.
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://plugins.archivebox.io/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://plugins.archivebox.io/#pdf), [`SCREENSHOT_ENABLED=False`](https://plugins.archivebox.io/#screenshot), and [`DOM_ENABLED=False`](https://plugins.archivebox.io/#dom) you don't need `chromium`.
<img src="https://avatars0.githubusercontent.com/u/1503512?s=200&v=4" width="100px" align="right"/>
**More info:**
- For help installing these, see the [Manual Setup](#manual-setup), [[Troubleshooting]] and [[Chromium Install]] pages.
- For per-plugin binary and enable/disable options (CHROME_BINARY, RIPGREP_BINARY, `<plugin>_ENABLED`, etc.) see the [abx-plugins config reference](https://archivebox.github.io/abx-plugins/).
- For per-plugin binary and enable/disable options (CHROME_BINARY, RIPGREP_BINARY, `<plugin>_ENABLED`, etc.) see the [abx-plugins config reference](https://plugins.archivebox.io/).
@@ -251,8 +251,8 @@ archivebox status
```
```bash
# OR start the webserver and view them in the Web UI
archivebox server 0.0.0.0:8000
# Visit http://admin.archivebox.localhost:8000/admin/ in a browser to finish setup
archivebox server 0.0.0.0:5797
# Visit http://admin.archivebox.localhost:5797/admin/ in a browser to finish setup
```
See our [[Usage]] Wiki documentation page for more examples.
+22 -43
View File
@@ -1,56 +1,35 @@
# Merging Collections
Current ArchiveBox collections cannot be merged safely by copying their `archive/users/...` trees: the database owns the Crawl, Snapshot, user, permission, and state-machine records, and `archivebox init` intentionally does not import orphaned current-layout directories. For current collections, use a database-aware migration or export the source URLs and re-archive them into the destination collection. Copying current Snapshot directories alone is a backup operation, not a merge.
Copy snapshot directories into the destination collection, then explicitly rescan them. Ordinary `archivebox init` and `archivebox update` intentionally avoid discovering current-layout orphan directories: that would make every startup/update scan the entire collection.
The workflow below is retained for **legacy collections whose real Snapshot directories are `archive/<timestamp>/`**. `archivebox update` can import those legacy directories into a fresh index.
1. Back up both collections and finish active captures before copying. Upgrade legacy collections using the normal [[Upgrading]] instructions.
2. Initialize the destination collection with `archivebox init`.
3. Copy or drag the contents of each source `archive/` into the destination `archive/`, preserving the `users/<username>/snapshots/<date>/<domain>/<uuid>/` hierarchy and every snapshot's `index.jsonl`. Inspect collisions; do not overwrite different files at the same path. Do not replace the destination SQLite database.
4. From the destination data directory, run:
> [!WARNING]
> Back up every collection before merging. Confirm that the source entries are real legacy timestamp directories containing data, and inspect path conflicts instead of allowing one collection to overwrite another.
```bash
archivebox init
archivebox update --rescan --migrate-only --index-only
archivebox status
```
1. Upgrade both old collections to the most recent ArchiveBox version (following instructions above)
```bash
cd /path/to/archivebox1/data
archivebox init
archivebox status
With Docker Compose:
cd /path/to/archivebox2/data
archivebox init
archivebox status
```bash
docker compose run --rm archivebox init
docker compose run --rm archivebox update --rescan --migrate-only --index-only
docker compose run --rm archivebox status
```
# ... repeat the same for each collection if merging more than two
```
`--rescan` deliberately performs an O(N) scan of snapshot directories. It imports sealed orphan snapshots, restores missing archive-result records and tags, and repairs missing metadata files and crawl links for known snapshots. It does not recover missing payload bytes or guess the identity of malformed metadata. Unresolved directories are reported, retained in place, and cause a nonzero exit status. Real legacy `archive/<timestamp>/` directories still use the existing filesystem migration flow.
2. Create a new empty archivebox collection in a new folder somewhere, this will hold the new merged collection
```bash
mkdir -p /path/to/archivebox_new/data
cd /path/to/archivebox_new/data
archivebox init
```
The scan preserves Snapshot, Crawl, and ArchiveResult IDs and the timestamps present in their metadata. ID/URL/timestamp conflicts are reported rather than assigned replacement IDs. Existing destination records take precedence over imported metadata. Missing owners are created as inactive accounts, without importing passwords or granting administrator privileges. Older metadata without permissions imports privately; exports without Crawl records or creation timestamps cannot reproduce those missing fields exactly. Machine-local processes, credentials, schedules, and full user accounts are not synchronized.
3. Copy the real legacy `archive/<timestamp>/` directories from each old collection into the new collection's `archive/` folder.
```bash
rsync --archive --info=progress2 /path/to/archivebox1/data/archive/ /path/to/archivebox_new/data/archive/
rsync --archive --info=progress2 /path/to/archivebox2/data/archive/ /path/to/archivebox_new/data/archive/
# ...repeat the same for each collection if merging more than two
```
Stop with Ctrl+C and rerun the same command to resume. There is no checkpoint file or new database state: each successful record is durable, and the scan compares existing IDs/results/tags/links before doing repair work. It discovers orphans before repairing known snapshots, with newest date/UUID names first in each group. Completed imports move to the known group on restart. Add `--reverse` to process oldest first instead. Known snapshot asset checks reuse the existing output manifest scanner. A restart still enumerates directories and reads metadata to find incomplete work; timestamps alone cannot prove completeness because copied files may retain old dates. Filters, `--resume`, and `--continuous` cannot be combined with `--rescan`.
4. Run `archivebox update` in the new collection to import the legacy directories
```bash
cd /path/to/archivebox_new/data
archivebox update
```
`--rescan` discovers filesystem-only snapshots and implies filesystem maintenance without new captures. `--migrate-only` eagerly finishes pending migrations for known database snapshots; ordinary updates leave them lazy. `--index-only` checks known snapshots, rebuilds output metadata, and backfills missing search indexes from archived content. All three explicit modes default to newest first; `--reverse` flips the order. `archivebox update --rescan` is enough when search backfill is unnecessary.
5. The new collection should now contain all the entries from the old collections combined
```bash
cd /path/to/archivebox_new/data
archivebox status
# optionally force an update of the snapshot index files (normally done lazily)
archivebox update --index-only
```
For more information about why Snapshot index files are usually updated lazily, see: https://github.com/ArchiveBox/ArchiveBox/issues/962
After you've confirmed your Snapshots are present in the new index, the old `index.sqlite3`, `index.json`, `index.html`, etc. main index files from the old archives can be safely deleted. You can optionally merge the contents of `ArchiveBox.conf` (your ArchiveBox config options), `sources/` (copies of all URLs imported in their original format), `logs/` (ArchiveBox error logs and debug info), and other root-level items yourself if that data is important to you.
For shared remote mounts, first make sure newly written objects have uploaded and the destination's directory cache can see them. Sharing `archive/` does not synchronize independent databases automatically. Confirm imported entries and representative archived files before removing any source or backup copies.
---
@@ -100,7 +79,7 @@ VALUES ('GENERATED_PASSWORD_HASH', NULL, 0, 'someUsername', '', '', 'someEmail@e
Replace the values above with the desired username, email, and password hash from python output^.
3. Log in using the new generated user to confirm it works
http://admin.archivebox.localhost:8000/admin/login/ user: `someUsername` pass:`somePasswordHere`
http://admin.archivebox.localhost:5797/admin/login/ user: `someUsername` pass:`somePasswordHere`
More info:
- https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage
+4 -4
View File
@@ -8,10 +8,10 @@ There are two ways to publish your archive: using the `archivebox server` or by
```bash
# start the webserver
archivebox server 0.0.0.0:8000
archivebox server 0.0.0.0:5797
```
Open <http://admin.archivebox.localhost:8000/admin/> in a browser to create the first admin and finish web setup. To create additional accounts from the CLI, use `archivebox manage createsuperuser`.
Open <http://admin.archivebox.localhost:5797/admin/> in a browser to create the first admin and finish web setup. To create additional accounts from the CLI, use `archivebox manage createsuperuser`.
This server is enabled out-of-the-box if you're using `docker-compose` to run ArchiveBox.
If `BASE_URL` is not configured yet, complete the first-run wizard before exposing ArchiveBox. It guides you through choosing a security mode and shows the settings to enter in Cloudflare, Nginx Proxy Manager, Caddy, Traefik, Tailscale, or your hosting platform's ingress UI. Configure HTTPS there, then open the canonical URL you chose in the wizard. Existing `BASE_URL` and security settings are trusted, so configured servers skip this wizard.
@@ -39,8 +39,8 @@ archivebox list --json --with-headers > index.json
# e.g. github pages or another static hosting provider
# you can also serve it with the simple python HTTP server
python3 -m http.server --bind 0.0.0.0 --directory . 8000
open http://127.0.0.1:8000
python3 -m http.server --bind 0.0.0.0 --directory . 5797
open http://127.0.0.1:5797
```
Here's a sample nginx configuration that works to serve your static archive folder:
+2 -2
View File
@@ -63,7 +63,7 @@ Open `./archive` to view your archive data in the filesystem.
You can also use the interactive Web UI to view/manage/add links to your archive:
```bash
# with plain Docker:
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
# with Docker Compose:
docker compose up -d
@@ -72,7 +72,7 @@ docker compose up -d
archivebox server
# Open /admin/ on the hostname or IP used to reach ArchiveBox.
# Local example: http://admin.archivebox.localhost:8000/admin/
# Local example: http://admin.archivebox.localhost:5797/admin/
# If BASE_URL is not configured yet, continue through the web setup wizard.
```
+1 -1
View File
@@ -38,7 +38,7 @@ ArchiveBox v0.9 is our biggest release since the 0.7.x series. It makes everyday
- 🔌 **A substantially expanded automation surface.** The Django Ninja REST API now exposes more collection, crawl, Snapshot, result, tag, user, token, and operational workflows. API tokens, outbound webhooks, and browser-extension uploads make ArchiveBox easier to connect to other tools.
- 🤖 **Optional AI agents and Claude-powered archiving.** Disabled by default, the built-in `archivebox mcp` server lets a dedicated AI agent perform crawls, search and manage your archive, prune duplicate results, and more. New optional plugins add [Claude-powered browser interaction](https://archivebox.github.io/abx-plugins/#claudechrome), [custom content extraction](https://archivebox.github.io/abx-plugins/#claudecodeextract), and [duplicate-result cleanup](https://archivebox.github.io/abx-plugins/#claudecodecleanup).
- 🤖 **Optional AI agents and Claude-powered archiving.** Disabled by default, the built-in `archivebox mcp` server lets a dedicated AI agent perform crawls, search and manage your archive, prune duplicate results, and more. New optional plugins add [Claude-powered browser interaction](https://plugins.archivebox.io/#claudechrome), [custom content extraction](https://plugins.archivebox.io/#claudecodeextract), and [duplicate-result cleanup](https://plugins.archivebox.io/#claudecodecleanup).
- 📦 **Simpler dependency and installation management.** `archivebox install` can discover and install required browser and command-line dependencies, and `uv tool install archivebox` provides a cleaner native installation path with isolated Python dependencies.
+1 -1
View File
@@ -43,7 +43,7 @@ With the new orchestrator flow, you only need the main `archivebox` service:
services:
archivebox:
image: archivebox/archivebox:dev
command: server --init 0.0.0.0:8000
command: server --init 0.0.0.0:5797
volumes:
- ./data:/data
```
+72 -72
View File
@@ -6,7 +6,7 @@ These desktop, tablet, and mobile screenshots cover ArchiveBox's major public an
## Login
View: [`/admin/login/`](http://admin.archivebox.localhost:9292/admin/login/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/login.html)
View: [`/admin/login/`](http://admin.archivebox.localhost:5797/admin/login/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/login.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -16,7 +16,7 @@ View: [`/admin/login/`](http://admin.archivebox.localhost:9292/admin/login/) ·
## Public snapshot list
View: [`/public/`](http://web.archivebox.localhost:9292/public/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
View: [`/public/`](http://web.archivebox.localhost:5797/public/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -26,7 +26,7 @@ View: [`/public/`](http://web.archivebox.localhost:9292/public/) · [View code](
## Snapshot View (capture in progress)
View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:9292) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:5797) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -36,7 +36,7 @@ View: [`/`](http://snap-8adbc4d6be2e.archivebox.localhost:9292) · [View code](h
## Add URLs
View: [`/add/`](http://admin.archivebox.localhost:9292/add/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
View: [`/add/`](http://admin.archivebox.localhost:5797/add/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -46,7 +46,7 @@ View: [`/add/`](http://admin.archivebox.localhost:9292/add/) · [View code](http
## Admin dashboard
View: [`/admin/`](http://admin.archivebox.localhost:9292/admin/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_site.py)
View: [`/admin/`](http://admin.archivebox.localhost:5797/admin/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_site.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -56,7 +56,7 @@ View: [`/admin/`](http://admin.archivebox.localhost:9292/admin/) · [View code](
## AI agent
View: [`/admin/agent/`](http://admin.archivebox.localhost:9292/admin/agent/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/abx_plugins/plugins/opencode/views.py)
View: [`/admin/agent/`](http://admin.archivebox.localhost:5797/admin/agent/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/abx_plugins/plugins/opencode/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -66,7 +66,7 @@ View: [`/admin/agent/`](http://admin.archivebox.localhost:9292/admin/agent/) ·
## Snapshots table
View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -76,7 +76,7 @@ View: [`/admin/core/snapshot/`](http://admin.archivebox.localhost:9292/admin/cor
## Snapshots grid
View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/grid/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/snapshots_grid.html)
View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/grid/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/admin/snapshots_grid.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -86,7 +86,7 @@ View: [`/admin/core/snapshot/grid/`](http://admin.archivebox.localhost:9292/admi
## Snapshot admin detail
View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://admin.archivebox.localhost:9292/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://admin.archivebox.localhost:5797/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_snapshots.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -96,7 +96,7 @@ View: [`/admin/core/snapshot/06a628b5939c74da8000f9864c2b39b4/change/`](http://a
## Snapshot files
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292/?files=1) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/static_index.html)
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:5797/?files=1) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/static_index.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -106,7 +106,7 @@ View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292/?files=1) · [Vie
## Archive results
View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:9292/admin/core/archiveresult/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:5797/admin/core/archiveresult/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -116,7 +116,7 @@ View: [`/admin/core/archiveresult/`](http://admin.archivebox.localhost:9292/admi
## Archive result detail
View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](http://admin.archivebox.localhost:9292/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](http://admin.archivebox.localhost:5797/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_archiveresults.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -126,7 +126,7 @@ View: [`/admin/core/archiveresult/06a628b7a41d7052800070cd8ff47301/change/`](htt
## Tags
View: [`/admin/core/tag/`](http://admin.archivebox.localhost:9292/admin/core/tag/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
View: [`/admin/core/tag/`](http://admin.archivebox.localhost:5797/admin/core/tag/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -136,7 +136,7 @@ View: [`/admin/core/tag/`](http://admin.archivebox.localhost:9292/admin/core/tag
## Tag detail
View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:9292/admin/core/tag/10129/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:5797/admin/core/tag/10129/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_tags.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -146,7 +146,7 @@ View: [`/admin/core/tag/10129/change/`](http://admin.archivebox.localhost:9292/a
## Users
View: [`/admin/auth/user/`](http://admin.archivebox.localhost:9292/admin/auth/user/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
View: [`/admin/auth/user/`](http://admin.archivebox.localhost:5797/admin/auth/user/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -156,7 +156,7 @@ View: [`/admin/auth/user/`](http://admin.archivebox.localhost:9292/admin/auth/us
## User detail
View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:9292/admin/auth/user/49/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:5797/admin/auth/user/49/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/admin_users.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -166,7 +166,7 @@ View: [`/admin/auth/user/49/change/`](http://admin.archivebox.localhost:9292/adm
## Crawls
View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:9292/admin/crawls/crawl/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:5797/admin/crawls/crawl/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -176,7 +176,7 @@ View: [`/admin/crawls/crawl/`](http://admin.archivebox.localhost:9292/admin/craw
## Crawl detail
View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://admin.archivebox.localhost:9292/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://admin.archivebox.localhost:5797/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -186,7 +186,7 @@ View: [`/admin/crawls/crawl/06a628b548e575088000fd924686ae24/change/`](http://ad
## Crawl schedules
View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:9292/admin/crawls/crawlschedule/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:5797/admin/crawls/crawlschedule/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -196,7 +196,7 @@ View: [`/admin/crawls/crawlschedule/`](http://admin.archivebox.localhost:9292/ad
## Crawl schedule detail
View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](http://admin.archivebox.localhost:9292/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](http://admin.archivebox.localhost:5797/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/crawls/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -206,7 +206,7 @@ View: [`/admin/crawls/crawlschedule/06a5fc0c9ca870e080005b6823701cca/change/`](h
## Personas
View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:9292/admin/personas/persona/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:5797/admin/personas/persona/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -216,7 +216,7 @@ View: [`/admin/personas/persona/`](http://admin.archivebox.localhost:9292/admin/
## Persona detail
View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http://admin.archivebox.localhost:9292/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http://admin.archivebox.localhost:5797/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/personas/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -226,7 +226,7 @@ View: [`/admin/personas/persona/019e84a0ffd376d98b410dfe1a409567/change/`](http:
## Machines
View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:9292/admin/machine/machine/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:5797/admin/machine/machine/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -236,7 +236,7 @@ View: [`/admin/machine/machine/`](http://admin.archivebox.localhost:9292/admin/m
## Machine detail
View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http://admin.archivebox.localhost:9292/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http://admin.archivebox.localhost:5797/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -246,7 +246,7 @@ View: [`/admin/machine/machine/019bd48b941f77fcafb2de79f2d2851e/change/`](http:/
## Network interfaces
View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:9292/admin/machine/networkinterface/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:5797/admin/machine/networkinterface/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -256,7 +256,7 @@ View: [`/admin/machine/networkinterface/`](http://admin.archivebox.localhost:929
## Network interface detail
View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/`](http://admin.archivebox.localhost:9292/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/`](http://admin.archivebox.localhost:5797/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -266,7 +266,7 @@ View: [`/admin/machine/networkinterface/06a2ee559ef570b080003f12b381ab42/change/
## Binaries
View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:9292/admin/machine/binary/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:5797/admin/machine/binary/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -276,7 +276,7 @@ View: [`/admin/machine/binary/`](http://admin.archivebox.localhost:9292/admin/ma
## Binary detail
View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://admin.archivebox.localhost:9292/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://admin.archivebox.localhost:5797/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -286,7 +286,7 @@ View: [`/admin/machine/binary/06a14cf28b4a7b1d800008b92ca5b94a/change/`](http://
## Processes
View: [`/admin/machine/process/`](http://admin.archivebox.localhost:9292/admin/machine/process/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/process/`](http://admin.archivebox.localhost:5797/admin/machine/process/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -296,7 +296,7 @@ View: [`/admin/machine/process/`](http://admin.archivebox.localhost:9292/admin/m
## Process detail
View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http://admin.archivebox.localhost:9292/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http://admin.archivebox.localhost:5797/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/machine/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -306,7 +306,7 @@ View: [`/admin/machine/process/06a628ec064b72d980002aac0a1bedd1/change/`](http:/
## API tokens
View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:9292/admin/api/apitoken/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:5797/admin/api/apitoken/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -316,7 +316,7 @@ View: [`/admin/api/apitoken/`](http://admin.archivebox.localhost:9292/admin/api/
## API token detail
View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://admin.archivebox.localhost:9292/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://admin.archivebox.localhost:5797/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -326,7 +326,7 @@ View: [`/admin/api/apitoken/06a628eae1477aae80008621afbbc237/change/`](http://ad
## Webhooks
View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:9292/admin/api/outboundwebhook/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:5797/admin/api/outboundwebhook/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -336,7 +336,7 @@ View: [`/admin/api/outboundwebhook/`](http://admin.archivebox.localhost:9292/adm
## Webhook detail
View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](http://admin.archivebox.localhost:9292/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](http://admin.archivebox.localhost:5797/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api/admin.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -346,7 +346,7 @@ View: [`/admin/api/outboundwebhook/06a628eb2801724880009c7991a4a522/change/`](ht
## Environment
View: [`/admin/environment/`](http://admin.archivebox.localhost:9292/admin/environment/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/settings.py)
View: [`/admin/environment/`](http://admin.archivebox.localhost:5797/admin/environment/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/settings.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -356,7 +356,7 @@ View: [`/admin/environment/`](http://admin.archivebox.localhost:9292/admin/envir
## Configuration
View: [`/admin/environment/config/`](http://admin.archivebox.localhost:9292/admin/environment/config/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
View: [`/admin/environment/config/`](http://admin.archivebox.localhost:5797/admin/environment/config/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -366,7 +366,7 @@ View: [`/admin/environment/config/`](http://admin.archivebox.localhost:9292/admi
## Configuration detail
View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:9292/admin/environment/config/BASE_URL/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:5797/admin/environment/config/BASE_URL/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -376,7 +376,7 @@ View: [`/admin/environment/config/BASE_URL/`](http://admin.archivebox.localhost:
## Dependencies
View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:9292/admin/environment/binaries/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:5797/admin/environment/binaries/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -386,7 +386,7 @@ View: [`/admin/environment/binaries/`](http://admin.archivebox.localhost:9292/ad
## Dependency detail
View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:9292/admin/environment/binaries/abxbus/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:5797/admin/environment/binaries/abxbus/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -396,7 +396,7 @@ View: [`/admin/environment/binaries/abxbus/`](http://admin.archivebox.localhost:
## Plugins
View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:9292/admin/environment/plugins/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/plugins/views.py)
View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:5797/admin/environment/plugins/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/plugins/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -406,7 +406,7 @@ View: [`/admin/environment/plugins/`](http://admin.archivebox.localhost:9292/adm
## Workers
View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:9292/admin/environment/workers/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:5797/admin/environment/workers/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -416,7 +416,7 @@ View: [`/admin/environment/workers/`](http://admin.archivebox.localhost:9292/adm
## Worker detail
View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localhost:9292/admin/environment/workers/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localhost:5797/admin/environment/workers/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -426,7 +426,7 @@ View: [`/admin/environment/workers/supervisord/`](http://admin.archivebox.localh
## Logs
View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:9292/admin/environment/logs/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:5797/admin/environment/logs/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -436,7 +436,7 @@ View: [`/admin/environment/logs/`](http://admin.archivebox.localhost:9292/admin/
## Log detail
View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost:9292/admin/environment/logs/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost:5797/admin/environment/logs/supervisord/) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/views.py)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -446,7 +446,7 @@ View: [`/admin/environment/logs/supervisord/`](http://admin.archivebox.localhost
## Snapshot View (singlefile)
View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:9292#singlefile) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:5797#singlefile) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -456,7 +456,7 @@ View: [`/#singlefile`](http://snap-f9864c2b39b4.archivebox.localhost:9292#single
## Snapshot View (screenshot)
View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:9292#screenshot) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:5797#screenshot) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -466,7 +466,7 @@ View: [`/#screenshot`](http://snap-f9864c2b39b4.archivebox.localhost:9292#screen
## Snapshot View (wget)
View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:9292#wget) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:5797#wget) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -476,7 +476,7 @@ View: [`/#wget`](http://snap-f9864c2b39b4.archivebox.localhost:9292#wget) · [Vi
## Snapshot View (dom)
View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dom) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:5797#dom) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -486,7 +486,7 @@ View: [`/#dom`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dom) · [View
## Snapshot View (pdf)
View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pdf) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:5797#pdf) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -496,7 +496,7 @@ View: [`/#pdf`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pdf) · [View
## Snapshot View (readability)
View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:9292#readability) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:5797#readability) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -506,7 +506,7 @@ View: [`/#readability`](http://snap-f9864c2b39b4.archivebox.localhost:9292#reada
## Snapshot View (ytdlp)
View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ytdlp) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:5797#ytdlp) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -516,7 +516,7 @@ View: [`/#ytdlp`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ytdlp) · [
## Snapshot View (responses)
View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:9292#responses) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:5797#responses) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -526,7 +526,7 @@ View: [`/#responses`](http://snap-f9864c2b39b4.archivebox.localhost:9292#respons
## Snapshot View (chrome_mhtml)
View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome_mhtml) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:5797#chrome_mhtml) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -536,7 +536,7 @@ View: [`/#chrome_mhtml`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chro
## Snapshot View (defuddle)
View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:9292#defuddle) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:5797#defuddle) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -546,7 +546,7 @@ View: [`/#defuddle`](http://snap-f9864c2b39b4.archivebox.localhost:9292#defuddle
## Snapshot View (mercury)
View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:9292#mercury) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:5797#mercury) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -556,7 +556,7 @@ View: [`/#mercury`](http://snap-f9864c2b39b4.archivebox.localhost:9292#mercury)
## Snapshot View (archivewebpage)
View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:9292#archivewebpage) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:5797#archivewebpage) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -566,7 +566,7 @@ View: [`/#archivewebpage`](http://snap-f9864c2b39b4.archivebox.localhost:9292#ar
## Snapshot View (chrome)
View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:5797#chrome) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -576,7 +576,7 @@ View: [`/#chrome`](http://snap-f9864c2b39b4.archivebox.localhost:9292#chrome) ·
## Snapshot View (consolelog)
View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:9292#consolelog) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:5797#consolelog) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -586,7 +586,7 @@ View: [`/#consolelog`](http://snap-f9864c2b39b4.archivebox.localhost:9292#consol
## Snapshot View (dns)
View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dns) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:5797#dns) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -596,7 +596,7 @@ View: [`/#dns`](http://snap-f9864c2b39b4.archivebox.localhost:9292#dns) · [View
## Snapshot View (sslcerts)
View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:9292#sslcerts) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:5797#sslcerts) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -606,7 +606,7 @@ View: [`/#sslcerts`](http://snap-f9864c2b39b4.archivebox.localhost:9292#sslcerts
## Snapshot View (redirects)
View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:9292#redirects) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:5797#redirects) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -616,7 +616,7 @@ View: [`/#redirects`](http://snap-f9864c2b39b4.archivebox.localhost:9292#redirec
## Snapshot View (headers)
View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:9292#headers) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:5797#headers) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -626,7 +626,7 @@ View: [`/#headers`](http://snap-f9864c2b39b4.archivebox.localhost:9292#headers)
## Snapshot View (seo)
View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:9292#seo) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:5797#seo) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -636,7 +636,7 @@ View: [`/#seo`](http://snap-f9864c2b39b4.archivebox.localhost:9292#seo) · [View
## Snapshot View (accessibility)
View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:9292#accessibility) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:5797#accessibility) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -646,7 +646,7 @@ View: [`/#accessibility`](http://snap-f9864c2b39b4.archivebox.localhost:9292#acc
## Snapshot View (htmltotext)
View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:9292#htmltotext) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:5797#htmltotext) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -656,7 +656,7 @@ View: [`/#htmltotext`](http://snap-f9864c2b39b4.archivebox.localhost:9292#htmlto
## Snapshot View (trafilatura)
View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:9292#trafilatura) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:5797#trafilatura) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -666,7 +666,7 @@ View: [`/#trafilatura`](http://snap-f9864c2b39b4.archivebox.localhost:9292#trafi
## Snapshot View (liteparse)
View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:9292#liteparse) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:5797#liteparse) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -676,7 +676,7 @@ View: [`/#liteparse`](http://snap-f9864c2b39b4.archivebox.localhost:9292#litepar
## Snapshot View (parse_html_urls)
View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_html_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_html_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -686,7 +686,7 @@ View: [`/#parse_html_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#p
## Snapshot View (parse_txt_urls)
View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_txt_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_txt_urls) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -696,7 +696,7 @@ View: [`/#parse_txt_urls`](http://snap-f9864c2b39b4.archivebox.localhost:9292#pa
## Snapshot View (parse_dom_outlinks)
View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:9292#parse_dom_outlinks) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:5797#parse_dom_outlinks) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -706,7 +706,7 @@ View: [`/#parse_dom_outlinks`](http://snap-f9864c2b39b4.archivebox.localhost:929
## Snapshot View (hashes)
View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:9292#hashes) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:5797#hashes) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
@@ -716,7 +716,7 @@ View: [`/#hashes`](http://snap-f9864c2b39b4.archivebox.localhost:9292#hashes) ·
## Snapshot View (header collapsed)
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:9292) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
View: [`/`](http://snap-f9864c2b39b4.archivebox.localhost:5797) · [View code](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/core/snapshot.html)
<table><thead><tr>
<th>Desktop</th><th>Tablet</th><th>Mobile</th>
+5 -5
View File
@@ -28,7 +28,7 @@ The default mode should not be used for archiving entire browser history or auth
```bash
# (these are the defaults)
archivebox config --set ARCHIVEDOTORG_ENABLED=True # see https://archivebox.github.io/abx-plugins/#archivedotorg
archivebox config --set ARCHIVEDOTORG_ENABLED=True # see https://plugins.archivebox.io/#archivedotorg
archivebox persona create public
archivebox add --persona=public 'https://example.com'
```
@@ -54,10 +54,10 @@ If you're importing private links or authenticated content, you probably don't w
#### ⚠️ Things to watch out for: ⚠️
- any cookies / secret state present in a Chrome user profile or `cookies.txt` file may be reflected in server responses and saved in the Snapshot output (e.g. in [`headers`](https://archivebox.github.io/abx-plugins/#headers) extractor output) — visible in cleartext to anyone viewing the Snapshot. **Don't use your personal Chrome profile for archiving** or people viewing your archive can then authenticate as you.
- any secret tokens embedded in URLs (e.g. secret invite links, Google Doc URLs, etc.) will be visible on `archive.org` as the URLs are not filtered when saving to it. Disable submitting to Archive.org entirely with [`ARCHIVEDOTORG_ENABLED=False`](https://archivebox.github.io/abx-plugins/#archivedotorg).
- the domain portion in archived URLs is sent to a favicon service in order to retrieve an icon more reliably than a janky internal implementation would be able to (if leaking domains is a concern, you can change the [`FAVICON_PROVIDER`](https://archivebox.github.io/abx-plugins/#favicon) or disable favicon fetching entirely with [`FAVICON_ENABLED=False`](https://archivebox.github.io/abx-plugins/#favicon)).
- [viewing malicious archived JS could allow an attacker to access your other archive items + the admin interface](https://github.com/ArchiveBox/ArchiveBox/issues/239) — use the default [`SERVER_SECURITY_MODE=auto`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#server_security_mode) (`*.localhost` uses isolated replay subdomains and other hosts use one-domain/no-JS raw replay), or disable risky extractors entirely with [`WGET_ENABLED=False`](https://archivebox.github.io/abx-plugins/#wget) and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom). Plugins that need their own trusted viewer can opt in with an explicit `full.html` preview template; ArchiveBox applies the no-JS policy to all other raw archived HTML without maintaining a plugin whitelist.
- any cookies / secret state present in a Chrome user profile or `cookies.txt` file may be reflected in server responses and saved in the Snapshot output (e.g. in [`headers`](https://plugins.archivebox.io/#headers) extractor output) — visible in cleartext to anyone viewing the Snapshot. **Don't use your personal Chrome profile for archiving** or people viewing your archive can then authenticate as you.
- any secret tokens embedded in URLs (e.g. secret invite links, Google Doc URLs, etc.) will be visible on `archive.org` as the URLs are not filtered when saving to it. Disable submitting to Archive.org entirely with [`ARCHIVEDOTORG_ENABLED=False`](https://plugins.archivebox.io/#archivedotorg).
- the domain portion in archived URLs is sent to a favicon service in order to retrieve an icon more reliably than a janky internal implementation would be able to (if leaking domains is a concern, you can change the [`FAVICON_PROVIDER`](https://plugins.archivebox.io/#favicon) or disable favicon fetching entirely with [`FAVICON_ENABLED=False`](https://plugins.archivebox.io/#favicon)).
- [viewing malicious archived JS could allow an attacker to access your other archive items + the admin interface](https://github.com/ArchiveBox/ArchiveBox/issues/239) — use the default [`SERVER_SECURITY_MODE=auto`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#server_security_mode) (`*.localhost` uses isolated replay subdomains and other hosts use one-domain/no-JS raw replay), or disable risky extractors entirely with [`WGET_ENABLED=False`](https://plugins.archivebox.io/#wget) and [`DOM_ENABLED=False`](https://plugins.archivebox.io/#dom). Plugins that need their own trusted viewer can opt in with an explicit `full.html` preview template; ArchiveBox applies the no-JS policy to all other raw archived HTML without maintaining a plugin whitelist.
<br/>
<img src="https://imgur.zervice.io/Jszo4h2.png" width="400px"/>
Loaded 100 of 152 files, more files were not shown because too many files have changed in this diff. Show more