Preserve historical code blocks and pin the complete docs build

.dockerignore

@@ -1,31 +0,0 @@
-.DS_Store
-._*
-*.pyc
-__pycache__/
-.mypy_cache/
-.pytest_cache/
-.github/
-.pdm-build/
-.pdm-python
-.eggs/
-.git/
-!.git/HEAD
-!.git/refs/heads/*
-
-venv/
-.venv/
-.venv-old/
-.docker-venv/
-node_modules/
-
-docs/
-build/
-dist/
-brew_dist/
-deb_dist/
-pip_dist/
-assets/
-docker/
-
-data/
-output/

.github/.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: 2
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+  jobs:
+    build:
+      html:
+        - uv run --no-project --with-requirements .github/docs/requirements.txt sphinx-build -W --keep-going -b html -c .github/docs . "$READTHEDOCS_OUTPUT/html"
+python:
+  install:
+    - method: uv
+      command: pip
+      requirements: .github/docs/requirements.txt
+sphinx:
+  configuration: .github/docs/conf.py
+  fail_on_warning: true

.github/CODE_OF_CONDUCT.md

@@ -1,76 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, sex characteristics, gender identity and expression,
-level of experience, education, socio-economic status, nationality, personal
-appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
- advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
- address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at archivebox-coc@sweeting.me. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
-
-[homepage]: https://www.contributor-covenant.org
-
-For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq

.github/CONTRIBUTING.md

@@ -1,38 +0,0 @@
-# Contribution Process
-
-1. Confirm your desired features fit into our bigger project goals [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap).
-2. Open an issue with your planned implementation to discuss
-3. Check in with me before starting development to make sure your work wont conflict with or duplicate existing work
-4. Setup your dev environment, make some changes, and test using the test input files
-5. Commit, push, and submit a PR and wait for review feedback
-6. Have patience, don't abandon your PR! We love contributors but we all have day jobs and don't always have time to respond to notifications instantly. If you want a faster response, ping @theSquashSH on twitter or Patreon.
-
-**Useful links:**
-
-- https://github.com/ArchiveBox/ArchiveBox/issues
-- https://github.com/ArchiveBox/ArchiveBox/pulls
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
-
-### Development Setup
-
-```bash
-git clone https://github.com/ArchiveBox/ArchiveBox
-cd ArchiveBox
-# Ideally do this in a virtualenv
-pip install -e '.[dev]'  # or use: pipenv install --dev
-```
-
-### Running Tests
-
-```bash
-./bin/lint.sh
-./bin/test.sh
-./bin/build.sh
-```
-
-For more common tasks see the `Development` section at the bottom of the README.
-
-### Getting Help
-
-Open issues on Github or message me https://sweeting.me/#contact.

.github/FUNDING.yml

@@ -1,3 +0,0 @@
-github: pirate
-patreon: theSquashSH
-custom: ["https://twitter.com/ArchiveBoxApp", "https://paypal.me/NicholasSweeting", "https://www.blockchain.com/eth/address/0x5D4c34D4a121Fe08d1dDB7969F07550f2dB9f471", "https://www.blockchain.com/btc/address/1HuxXriPE2Bbnag3jJrqa3bkNHrs297dYH"]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,46 +0,0 @@
----
-name: 🐞 Bug report
-about: Create a report to help us improve
-title: 'Bug: ...'
-labels: 'bug'
-assignees: ''
-
----
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you.
-(the only required section is the version information)
--->
-
-#### Describe the bug
-<!--
-A description of what the bug is, 
-what you expected to happen, 
-and any relevant context about issue.
--->
-
-#### Steps to reproduce
-<!--
-For example:
-1. Ran ArchiveBox with the following config '...'
-2. Saw this output during archiving '....'
-3. UI didn't show the thing I was expecting '....'
--->
-
-#### Screenshots or log output
-
-<!--
-If applicable, post any relevant screenshots or copy/pasted terminal output from ArchiveBox.
-If you're reporting a parsing / importing error, **you must paste a copy of your redacted import file here**.
--->
-
-#### ArchiveBox version
-
-<!-- Run the `archivebox version` command locally then copy paste the result here: -->
-```logs
-replace this line with the *full*, unshortened output of running `archivebox version`
-```
-<!-- Tickets without full version info will closed until it is provided,
-we need the full output here to help you solve your issue -->

.github/ISSUE_TEMPLATE/documentation_change.md

@@ -1,16 +0,0 @@
----
-name: 📑 Documentation change
-about: Submit a suggestion for the Wiki documentation
-title: 'Documentation: Improvement request ...'
-labels: ''
-assignees: ''
-
----
-
-## Wiki Page URL
-<!-- e.g. https://github.com/pirate/ArchiveBox/wiki/Configuration#use_color -->
-
-
-## Suggested Edit
-<!-- e.g. Please add more example usages, or please fix `xyz` typo to be `abc`. -->
-

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,50 +0,0 @@
----
-name: 💡 Feature request
-about: Suggest an idea for this project
-title: 'Feature Request: ...'
-labels: 'changes: behavior,status: idea phase'
-assignees: ''
-
----
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you :)
--->
-
-## Type 
-
- - [ ] General question or discussion
- - [ ] Propose a brand new feature
- - [ ] Request modification of existing behavior or design
-
-## What is the problem that your feature request solves
-<!--
-e.g. I need to be able to archive spanish and french subtitle files  
-from a particular <example.com> movie site that's going down soon.
--->
-
-## Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes
-<!--
-e.g. I specifically need a new archive method to look for multilingual subtitle files related to pages.  
-The bigger picture solution is the ability for custom user scripts to be run in a puppeteer context during archiving.
--->
-
-## What hacks or alternative solutions have you tried to solve the problem?
-<!--
-A clear and concise description of any alternative solutions, 
-workarounds, or other software you've considered using to fix the problem.
--->
-
-## How badly do you want this new feature?
-
- - [ ] It's an urgent deal-breaker, I can't live without it
- - [ ] It's important to add it in the near-mid term future
- - [ ] It would be nice to have eventually
-
----
-
- - [ ] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue
- - [ ] I like ArchiveBox so far / would recommend it to a friend
- - [ ] I've had a lot of difficulty getting ArchiveBox set up

.github/ISSUE_TEMPLATE/question_or_discussion.md

@@ -1,9 +0,0 @@
----
-name: 💬 Question, discussion, or support request
-about: Start a discussion or ask a question about ArchiveBox
-title: 'Question: ...'
-labels: ''
-assignees: ''
-
----
-

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,18 +0,0 @@
-<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes. -->
-
-# Summary
-
-<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->
-
-# Related issues
-
-<!-- e.g. #123 or Roadmap goal # https://github.com/pirate/ArchiveBox/wiki/Roadmap -->
-
-# Changes these areas
-
-- [ ] Bugfixes
-- [ ] Feature behavior
-- [ ] Command line interface
-- [ ] Configuration options
-- [ ] Internal architecture
-- [ ] Snapshot data layout on disk

.github/docs/README.md

@@ -0,0 +1,25 @@
+# Historical 0.1.0 documentation
+
+This documentation-only branch starts at the immutable `v0.1.0` tag.
+The original README, application files, and pinned wiki (when present) remain unchanged.
+Sphinx reads them without installing or importing the historical application.
+The render-time adapter fixes historical link and heading syntax only; the visible warning
+explains that these old installation commands and external services may no longer work.
+
+Prerequisites: [uv](https://docs.astral.sh/uv/getting-started/installation/) 0.11.3 and Python 3.13, matching CI and Read the Docs. `uv` can install Python when needed.
+
+```sh
+READTHEDOCS=True uv run --no-project --python 3.13 --with-requirements .github/docs/requirements.txt \
+  sphinx-build -E -a -W --keep-going -b html -c .github/docs . /tmp/archivebox-historical-docs
+```
+
+Read the Docs must use `.github/.readthedocs.yaml`. The pinned theme supplies
+its version selector and listens for Read the Docs addon version data.
+
+`requirements.txt` pins the complete documentation dependency set. To update it,
+edit `requirements.in` and run:
+
+```sh
+uv pip compile --universal --python-version 3.13 .github/docs/requirements.in \
+  --output-file .github/docs/requirements.txt
+```

.github/docs/conf.py

@@ -0,0 +1,131 @@
+"""Render this release's historical documentation without importing the application."""
+
+import os
+import re
+from pathlib import Path
+from urllib.parse import urlsplit
+
+from markdown_it import MarkdownIt
+
+ROOT = Path(__file__).resolve().parents[2]
+release = "0.1.0"
+version = release
+project = "ArchiveBox"
+author = "ArchiveBox contributors"
+copyright = "ArchiveBox contributors"
+extensions = ["myst_parser", "sphinx_rtd_theme"]
+source_suffix = {".md": "markdown"}
+root_doc = "index"
+exclude_patterns = [
+    p.name for p in ROOT.iterdir() if p.name not in {"index.md", "docs"}
+]
+exclude_patterns += ["docs/_Sidebar.md", "docs/_Footer.md", "docs/.git"]
+html_theme = "sphinx_rtd_theme"
+html_theme_options = {
+    "navigation_depth": 3,
+    "collapse_navigation": False,
+    "version_selector": True,
+    "language_selector": False,
+}
+html_context = {"current_version": os.environ.get("READTHEDOCS_VERSION", release)}
+html_title = f"ArchiveBox {release} historical documentation"
+html_show_sphinx = False
+myst_heading_anchors = 6
+
+
+def rewrite_prose(text, transform):
+    """Rewrite navigation outside fenced and indented code blocks."""
+    lines = text.splitlines(keepends=True)
+    output = []
+    start = 0
+    for token in MarkdownIt().parse(text):
+        if token.type not in {"fence", "code_block"}:
+            continue
+        first, last = token.map
+        output.append(transform("".join(lines[start:first])))
+        output.append("".join(lines[first:last]))
+        start = last
+    output.append(transform("".join(lines[start:])))
+    return "".join(output)
+
+
+def prepare_source(app, docname, source):
+    text = (ROOT / "README.md").read_text() if docname == "index" else source[0]
+    directory = ROOT if docname == "index" else ROOT / "docs"
+
+    def wiki(match):
+        parts = match.group(1).split("|", 1)
+        return f"[{parts[0]}]({parts[-1].replace(' ', '-')}.md)"
+
+    text = rewrite_prose(text, lambda prose: re.sub(r"\[\[([^\]\n]+)\]\]", wiki, prose))
+
+    def resource(target):
+        # These README images were deleted just before 0.1.0 without fixing its links.
+        # Preserve the original screenshots from the last preceding commit containing them.
+        if target in {"screenshot.png", "screenshot_mobile.png"}:
+            return (
+                "https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/62e33c011b2928ba166a8059d445b8aa7429da96/"
+                + target
+            )
+        if urlsplit(target).scheme or target.startswith(("#", "//")):
+            return target
+        page, sep, anchor = target.partition("#")
+        path = directory / page
+        if path.suffix == ".md" and path.parent == ROOT / "docs":
+            return target
+        if path.is_file():
+            return (
+                f"https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/v{release}/{path.relative_to(ROOT)}"
+                + ("#" + anchor if sep else "")
+            )
+        return target
+
+    def rewrite_links(prose):
+        prose = re.sub(
+            r'((?:src|href)=["\'])([^"\']+)', lambda m: m[1] + resource(m[2]), prose
+        )
+        prose = re.sub(r"\]\(([^)\s]+)\)", lambda m: "](" + resource(m[1]) + ")", prose)
+        return prose
+
+    text = rewrite_prose(text, rewrite_links)
+    text = (
+        f"# ArchiveBox {release}"
+        + (
+            " historical guide"
+            if docname == "index"
+            else " — " + Path(docname).name.replace("-", " ")
+        )
+        + "\n\n"
+        + text
+    )
+    lines = text.splitlines()
+    stack = []
+    for token in MarkdownIt().parse(text):
+        if token.type != "heading_open":
+            continue
+        original = int(token.tag[1])
+        while stack and stack[-1] >= original:
+            stack.pop()
+        stack.append(original)
+        lines[token.map[0]] = re.sub(
+            r"^#{1,6}(?=\s)", "#" * len(stack), lines[token.map[0]]
+        )
+    text = "\n".join(lines) + "\n"
+    if docname == "index":
+        notice = f"\n```{{warning}}\nThis is the original guide shipped with ArchiveBox {release}, preserved for historical reference. Commands, dependencies, and external services describe that old release and may no longer work. Use the version selector for current documentation.\n```\n"
+        first, rest = text.split("\n", 1)
+        text = first + "\n" + notice + rest
+        pages = sorted(
+            p.stem for p in (ROOT / "docs").glob("*.md") if not p.name.startswith("_")
+        )
+        if pages:
+            text += (
+                "\n```{toctree}\n:maxdepth: 2\n\n"
+                + "\n".join("docs/" + p for p in pages)
+                + "\n```\n"
+            )
+    source[0] = text
+
+
+def setup(app):
+    app.connect("source-read", prepare_source)

.github/docs/requirements.in

@@ -0,0 +1,3 @@
+Sphinx==8.2.3
+myst-parser==4.0.1
+sphinx-rtd-theme==3.1.0

.github/docs/requirements.txt

@@ -0,0 +1,75 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --universal --python-version 3.13 .github/docs/requirements.in --output-file .github/docs/requirements.txt
+alabaster==1.0.0
+    # via sphinx
+babel==2.18.0
+    # via sphinx
+certifi==2026.7.22
+    # via requests
+charset-normalizer==3.5.1
+    # via requests
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via sphinx
+docutils==0.21.2
+    # via
+    #   myst-parser
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.20
+    # via requests
+imagesize==2.0.1
+    # via sphinx
+jinja2==3.1.6
+    # via
+    #   myst-parser
+    #   sphinx
+markdown-it-py==3.0.0
+    # via
+    #   mdit-py-plugins
+    #   myst-parser
+markupsafe==3.0.3
+    # via jinja2
+mdit-py-plugins==0.6.1
+    # via myst-parser
+mdurl==0.1.2
+    # via markdown-it-py
+myst-parser==4.0.1
+    # via -r .github/docs/requirements.in
+packaging==26.3
+    # via sphinx
+pygments==2.21.0
+    # via sphinx
+pyyaml==6.0.3
+    # via myst-parser
+requests==2.34.2
+    # via sphinx
+roman-numerals==4.1.0
+    # via roman-numerals-py
+roman-numerals-py==4.1.0
+    # via sphinx
+snowballstemmer==3.1.1
+    # via sphinx
+sphinx==8.2.3
+    # via
+    #   -r .github/docs/requirements.in
+    #   myst-parser
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-jquery
+sphinx-rtd-theme==3.1.0
+    # via -r .github/docs/requirements.in
+sphinxcontrib-applehelp==2.0.0
+    # via sphinx
+sphinxcontrib-devhelp==2.0.0
+    # via sphinx
+sphinxcontrib-htmlhelp==2.1.0
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==2.0.0
+    # via sphinx
+sphinxcontrib-serializinghtml==2.0.0
+    # via sphinx
+urllib3==2.8.0
+    # via requests

.github/workflows/codeql-analysis.yml

@@ -1,32 +0,0 @@
-name: "CodeQL"
-
-on:
-  push:
-    branches: [ dev ]
-  pull_request:
-    branches: [ dev ]
-  schedule:
-    - cron: '43 1 * * 2'
-
-jobs:
-  analyze:
-    name: Analyze
-    runs-on: ubuntu-latest
-
-    strategy:
-      fail-fast: false
-      matrix:
-        language: [ 'python' ]
-
-    steps:
-    - name: Checkout repository
-      uses: actions/checkout@v2
-
-    - name: Initialize CodeQL
-      uses: github/codeql-action/init@v1
-      with:
-        languages: ${{ matrix.language }}
-        queries: security-extended
-
-    - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v1

.github/workflows/debian.yml

@@ -1,64 +0,0 @@
-name: Build Debian package
-
-on:
-  workflow_dispatch:
-  push:
-
-env:
-  DEB_BUILD_OPTIONS: nocheck
-
-jobs:
-  build:
-    runs-on: ubuntu-22.04
-
-    steps:
-      - uses: actions/checkout@v3
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Install packaging dependencies
-        run: |
-          sudo apt-get update -qq
-          sudo apt-get install -y \
-            python3 python3-dev python3-pip python3-venv python3-all \
-            dh-python debhelper devscripts dput software-properties-common \
-            python3-distutils python3-setuptools python3-wheel python3-stdeb
-
-      # - name: Build Debian/Apt sdist_dsc
-      #   run: |
-      #     ./bin/build_pip.sh
-
-      # - name: Check ArchiveBox version
-      #   run: |
-      #     # must create dir needed for snaps to run as non-root on github actions
-      #     sudo mkdir -p /run/user/1001 && sudo chmod -R 777 /run/user/1001
-      #     mkdir "${{ github.workspace }}/data" && cd "${{ github.workspace }}/data"
-      #     archivebox --version
-      #     archivebox init --setup
-
-      # - name: Add some links to test
-      #   run: |
-      #     cd "${{ github.workspace }}/data"
-      #     archivebox add 'https://example.com'
-      #     archivebox status
-
-      # - name: Commit built package
-      #   run: |
-      #     cd deb_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Debian package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/debian-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: deb_dist
-
-      # - name: Push build to Launchpad PPA
-      #   run: |
-      #     debsign -k "$PGP_KEY_ID" "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"
-      #     dput archivebox "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"

.github/workflows/docker.yml

@@ -1,93 +0,0 @@
-name: Build Docker image
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - '**'
-    tags:
-      - 'v*'
-  # pull_request:
-
-env:
-  DOCKER_IMAGE: archivebox-ci
-      
-jobs:
-  buildx:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
-      
-      - name: Set up Docker Buildx
-        id: buildx
-        uses: docker/setup-buildx-action@v3
-        with:
-          version: latest
-          install: true
-          platforms: linux/amd64,linux/arm64,linux/arm/v7
-      
-      - name: Builder instance name
-        run: echo ${{ steps.buildx.outputs.name }}
-      
-      - name: Available platforms
-        run: echo ${{ steps.buildx.outputs.platforms }}
-      
-      - name: Cache Docker layers
-        uses: actions/cache@v3
-        with:
-          path: /tmp/.buildx-cache
-          key: ${{ runner.os }}-buildx-${{ github.sha }}
-          restore-keys: |
-            ${{ runner.os }}-buildx-
-
-      - name: Docker Login
-        uses: docker/login-action@v3
-        if: github.event_name != 'pull_request'
-        with:
-           username: ${{ secrets.DOCKER_USERNAME }}
-           password: ${{ secrets.DOCKER_PASSWORD }}
-      
-      - name: Collect Docker tags
-        id: docker_meta
-        uses: docker/metadata-action@v5
-        with:
-          images: archivebox/archivebox,nikisweeting/archivebox
-          tags: |
-              type=ref,event=branch
-              type=semver,pattern={{version}}
-              type=semver,pattern={{major}}.{{minor}}
-              type=sha
-              type=raw,value=latest,enable={{is_default_branch}}
-      
-      - name: Build and push
-        id: docker_build
-        uses: docker/build-push-action@v5
-        with:
-          context: ./
-          file: ./Dockerfile
-          builder: ${{ steps.buildx.outputs.name }}
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.docker_meta.outputs.tags }}
-          cache-from: type=local,src=/tmp/.buildx-cache
-          cache-to: type=local,dest=/tmp/.buildx-cache-new
-          platforms: linux/amd64,linux/arm64,linux/arm/v7
-
-      - name: Image digest
-        run: echo ${{ steps.docker_build.outputs.digest }}
-       
-      # This ugly bit is necessary if you don't want your cache to grow forever
-      # until it hits GitHub's limit of 5GB.
-      # Temp fix
-      # https://github.com/docker/build-push-action/issues/252
-      # https://github.com/moby/buildkit/issues/1896
-      - name: Move cache
-        run: |
-          rm -rf /tmp/.buildx-cache
-          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

.github/workflows/homebrew.yml

@@ -1,51 +0,0 @@
-name: Build Homebrew package
-
-on:
-  workflow_dispatch:
-  push:
-
-
-jobs:
-  build:
-    runs-on: macos-latest
-
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      # TODO: modify archivebox.rb to update src url, hashes, and dependencies
-
-      - name: Build Homebrew Bottle
-        run: |
-          pip3 install --upgrade pip setuptools wheel
-          cd brew_dist/
-          brew install --build-bottle ./archivebox.rb
-          # brew bottle archivebox
-          archivebox version
-
-      - name: Add some links to test
-        run: |
-          mkdir data && cd data
-          archivebox init --setup
-          archivebox add 'https://example.com'
-          archivebox version
-          archivebox status
-
-      # - name: Commit built package
-      #   run: |
-      #     cd brew_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Homebrew package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/homebrew-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: brew_dist
-
-      # TODO: push bottle homebrew core PR with latest changes

.github/workflows/jekyll-gh-pages.yml

@@ -1,58 +0,0 @@
-# Sample workflow for building and deploying a Jekyll site to GitHub Pages
-name: Build GitHub Pages website
-
-on:
-  # Runs on pushes targeting the default branch
-  push:
-    branches: ["dev"]
-
-  # Allows you to run this workflow manually from the Actions tab
-  workflow_dispatch:
-
-# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-
-# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
-# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
-concurrency:
-  group: "pages"
-  cancel-in-progress: true
-
-jobs:
-  # Build job
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-      - name: Copy README.md into place
-        run: |
-          rm ./website/README.md
-          cp ./README.md ./website/README.md
-      - name: Setup Pages
-        uses: actions/configure-pages@v3
-      - name: Build with Jekyll
-        uses: actions/jekyll-build-pages@v1
-        with:
-          source: ./website
-          destination: ./_site
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@v2
-
-  # Deployment job
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
-    needs: build
-    steps:
-      - name: Deploy to GitHub Pages
-        id: deployment
-        uses: actions/deploy-pages@v2

.github/workflows/lint.yml

@@ -1,35 +0,0 @@
-name: Run linters
-
-on:
-  workflow_dispatch:
-  push:
-
-env:
-  MAX_LINE_LENGTH: 110
-
-jobs:
-  lint:
-    runs-on: ubuntu-20.04
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.9
-          architecture: x64
-
-      - name: Install flake8
-        run: |
-          pip install flake8
-
-      - name: Lint with flake8
-        run: |
-          cd archivebox
-          # one pass for show-stopper syntax errors or undefined names
-          flake8 . --count --show-source --statistics
-          # one pass for small stylistic things
-          flake8 . --count --max-line-length="$MAX_LINE_LENGTH" --statistics

.github/workflows/pip.yml

@@ -1,78 +0,0 @@
-name: Build Pip package
-
-on:
-  workflow_dispatch:
-  push:
-
-
-jobs:
-  build:
-    runs-on: ubuntu-22.04
-
-    permissions:
-      id-token: write
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: '3.11'
-          architecture: x64
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
-
-      - uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: '3.11'
-          cache: true
-
-      - name: Install dependencies
-        run: pdm install --fail-fast --no-lock --group :all --no-self
-
-      - name: Build package
-        run: |
-          pdm build
-
-      - name: Install from build
-        run: pip install ./dist/archivebox-*.whl
-
-      - name: Add some links to test
-        run: |
-          mkdir data && cd data
-          archivebox init
-          archivebox add 'https://example.com'
-          archivebox version
-          archivebox status
-
-      #- name: Publish package distributions to PyPI
-      #  run: pdm publish --no-build
-
-      # - name: Push build to PyPI
-      #   run: |
-      #     cd pip_dist/
-      #     python3 -m twine upload --repository testpypi pip_dist/*.{whl,tar.gz}
-      #     python3 -m twine upload --repository pypi pip_dist/*.{whl,tar.gz}
-
-      # - name: Commit built package
-      #   run: |
-      #     cd pip_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Pip package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/pip-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: pip_dist
-

.github/workflows/test.yml

@@ -1,158 +0,0 @@
-name: Run tests
-on: [push]
-
-env:
-  DOCKER_IMAGE: archivebox-ci
-  PYTHONIOENCODING: utf-8
-  PYTHONLEGACYWINDOWSSTDIO: utf-8
-  USE_COLOR: False
-
-jobs:
-  python_tests:
-    runs-on: ${{ matrix.os }}
-
-    strategy:
-      matrix:
-        os: [ubuntu-22.04]
-        # os: [ubuntu-22.04, macos-latest, windows-latest]
-        python: [3.11]
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      ### Setup Python & JS Languages
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ matrix.python }}
-          architecture: x64
-
-      - name: Set up Node JS
-        uses: actions/setup-node@v4
-        with:
-          node-version: 20.10.0
-
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: '3.11'
-          cache: true
-
-      ### Install Python & JS Dependencies
-      - name: Get pip cache dir
-        id: pip-cache
-        run: |
-          echo "::set-output name=dir::$(pip cache dir)"
-
-      - name: Cache pip
-        uses: actions/cache@v3
-        id: cache-pip
-        with:
-          path: ${{ steps.pip-cache.outputs.dir }}
-          key: ${{ runner.os }}-${{ matrix.python }}-venv-${{ hashFiles('setup.py') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.python }}-venv-
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 python3-minimal gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
-
-      - name: Install pip dependencies
-        run: |
-          python -m pip install --upgrade pip setuptools wheel pytest bottle build
-          python -m pip install -r requirements.txt
-          python -m pip install -e .[sonic,ldap]
-
-      - name: Get npm cache dir
-        id: npm-cache
-        run: |
-          echo "::set-output name=dir::$GITHUB_WORKSPACE/node_modules"
-
-      - name: Cache npm
-        uses: actions/cache@v3
-        id: cache-npm
-        with:
-          path: ${{ steps.npm-cache.outputs.dir }}
-          key: ${{ runner.os }}-node_modules-${{ hashFiles('package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-node_modules
-
-      - name: Install npm requirements
-        run: |
-          npm install
-          echo "SINGLEFILE_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/single-file" >> $GITHUB_ENV
-          echo "READABILITY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/readability-extractor" >> $GITHUB_ENV
-          echo "MERCURY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/mercury-parser" >> $GITHUB_ENV
-
-      ### Run the tests
-      - name: Directory listing for debugging
-        run: |
-          pwd
-          ls
-
-      - name: Archivebox version
-        run: |
-          archivebox version
-
-      - name: Test built package with pytest
-        # TODO: remove this exception for windows once we get tests passing on that platform
-        if: ${{ !contains(matrix.os, 'windows') }}
-        run: |
-          python -m pytest -s --basetemp=tests/out --ignore=archivebox/vendor --ignore=deb_dist --ignore=pip_dist --ignore=brew_dist
-
-  docker_tests:
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      # TODO: as of 2020-11 this helper layer broke, upgrade and re-enable this once it's usable again
-      # - uses: satackey/action-docker-layer-caching@v0.0.8
-
-      - name: Build image
-        run: |
-          docker build . -t "$DOCKER_IMAGE"
-
-      - name: Init data dir
-        run: |
-          mkdir "${{ github.workspace }}/data"
-          docker run -v "${{ github.workspace }}/data":/data "$DOCKER_IMAGE" init
-
-      - name: Run test server
-        run: |
-          sudo bash -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
-          docker run --name www-nginx -p 80:80 -d nginx
-
-      - name: Add link
-        run: |
-          docker run -v "$PWD"/data:/data --network host "$DOCKER_IMAGE" add http://www.test-nginx-1.local
-
-      - name: Add stdin link
-        run: |
-          echo "http://www.test-nginx-2.local" | docker run -i --network host -v "$PWD"/data:/data "$DOCKER_IMAGE" add
-
-      - name: List links
-        run: |
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
-
-      - name: Start docker-compose stack
-        run: |
-          docker-compose run archivebox init
-          docker-compose up -d
-          sleep 5
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'ArchiveBox'
-          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'window.django'
-
-      - name: Check added urls show up in index
-        run: |
-          docker-compose run archivebox add 'http://example.com/#test_docker' --index-only
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'http://example.com/#test_docker'
-          docker-compose down || true

.gitignore

@@ -1,31 +1,4 @@
-.DS_Store
-._*
-*.pyc
-__pycache__/
-.mypy_cache/
-.eggs/
-tests/out/
-
-# Python and Node dependencies
-venv/
-.venv/
-.docker-venv/
-node_modules/
-
-# Packaging artifacts
-.pdm-python
-.pdm-build
-archivebox.egg-info
-archivebox-*.tar.gz
-build/
-dist/
-
-# Data folders
-data/
-data1/
-data2/
-data3/
 output/
-
-# vim
-*.sw?
+.DS_Store
+__pycache__/
+archiver/venv

.gitmodules

@@ -1,28 +0,0 @@
-[submodule "docs"]
-    path = docs
-    url = https://github.com/ArchiveBox/ArchiveBox.wiki.git
-
-[submodule "deb_dist"]
-    path = deb_dist
-    url = https://github.com/ArchiveBox/debian-archivebox.git
-[submodule "brew_dist"]
-	path = brew_dist
-	url = https://github.com/ArchiveBox/homebrew-archivebox.git
-[submodule "pip_dist"]
-	path = pip_dist
-	url = https://github.com/ArchiveBox/pip-archivebox.git
-[submodule "docker"]
-	path = docker
-	url = https://github.com/ArchiveBox/docker-archivebox.git
-[submodule "archivebox/vendor/base32-crockford"]
-	path = archivebox/vendor/base32-crockford
-	url = https://github.com/jbittel/base32-crockford
-[submodule "archivebox/vendor/pocket"]
-	path = archivebox/vendor/pocket
-	url = https://github.com/tapanpandita/pocket
-[submodule "archivebox/vendor/django-taggit"]
-	path = archivebox/vendor/django-taggit
-	url = https://github.com/jazzband/django-taggit
-[submodule "archivebox/vendor/python-atomicwrites"]
-	path = archivebox/vendor/python-atomicwrites
-	url = https://github.com/untitaker/python-atomicwrites

.npmignore

@@ -1,19 +0,0 @@
-tests/
-archivebox/
-archivebox.egg-info/
-build/
-dist/
-docs/
-etc/
-.github
-.gitmodules
-.dockerignore
-.flake8
-CNAME
-_config.yml
-docker-compose.yaml
-docker-compose.yml
-Dockerfile
-MANIFEST.in
-Pipfile
-setup.py

.readthedocs.yaml

@@ -1,34 +0,0 @@
-# Read the Docs configuration file for Sphinx projects
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-version: 2
-
-submodules:
-  include: all
-  recursive: true
-
-# Set the OS, Python version and other tools you might need
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.12"
-    nodejs: "20"
-
-# Build documentation in the "docs/" directory with Sphinx
-sphinx:
-  configuration: docs/conf.py
-  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
-  # builder: "dirhtml"
-
-# Optionally build your docs in additional formats such as PDF and ePub
-formats:
-  - pdf
-  - epub
-
-# Optional but recommended, declare the Python requirements required
-# to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
-  install:
-    - requirements: requirements.txt
-    - requirements: docs/requirements.txt

Dockerfile

@@ -1,297 +0,0 @@
-# This is the Dockerfile for ArchiveBox, it bundles the following dependencies:
-#     python3, ArchiveBox, curl, wget, git, chromium, youtube-dl, yt-dlp, single-file
-# Usage:
-#     git submodule update --init --recursive
-#     git pull --recurse-submodules
-#     docker build . -t archivebox --no-cache
-#     docker run -v "$PWD/data":/data archivebox init
-#     docker run -v "$PWD/data":/data archivebox add 'https://example.com'
-#     docker run -v "$PWD/data":/data -it archivebox manage createsuperuser
-#     docker run -v "$PWD/data":/data -p 8000:8000 archivebox server
-# Multi-arch build:
-#     docker buildx create --use
-#     docker buildx build . --platform=linux/amd64,linux/arm64,linux/arm/v7 --push -t archivebox/archivebox:latest -t archivebox/archivebox:dev
-#
-# Read more about [developing Archivebox](https://github.com/ArchiveBox/ArchiveBox#archivebox-development).
-
-
-# Use Debian 12 w/ faster package updates: https://packages.debian.org/bookworm-backports/
-FROM python:3.11-slim-bookworm
-
-LABEL name="archivebox" \
-    maintainer="Nick Sweeting <dockerfile@archivebox.io>" \
-    description="All-in-one personal internet archiving container" \
-    homepage="https://github.com/ArchiveBox/ArchiveBox" \
-    documentation="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker"
-
-ARG TARGETPLATFORM
-ARG TARGETOS
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-######### Environment Variables #################################
-
-# Global system-level config
-ENV TZ=UTC \
-    LANGUAGE=en_US:en \
-    LC_ALL=C.UTF-8 \
-    LANG=C.UTF-8 \
-    DEBIAN_FRONTEND=noninteractive \
-    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 \
-    PYTHONIOENCODING=UTF-8 \
-    PYTHONUNBUFFERED=1 \
-    PIP_DISABLE_PIP_VERSION_CHECK=1 \
-    npm_config_loglevel=error
-
-# Version config
-ENV PYTHON_VERSION=3.11 \
-    NODE_VERSION=20
-
-# User config
-ENV ARCHIVEBOX_USER="archivebox" \
-    DEFAULT_PUID=911 \
-    DEFAULT_PGID=911
-
-# Global paths
-ENV CODE_DIR=/app \
-    DATA_DIR=/data \
-    GLOBAL_VENV=/venv \
-    PLAYWRIGHT_BROWSERS_PATH=/browsers
-
-# Application-level paths
-ENV APP_VENV=/app/.venv \
-    NODE_MODULES=/app/node_modules
-
-# Build shell config
-ENV PATH="$PATH:$GLOBAL_VENV/bin:$APP_VENV/bin:$NODE_MODULES/.bin"
-SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "errtrace", "-o", "nounset", "-c"] 
-
-######### System Environment ####################################
-
-# Detect ArchiveBox version number by reading package.json
-COPY --chown=root:root --chmod=755 package.json "$CODE_DIR/"
-RUN grep '"version": ' "${CODE_DIR}/package.json" | awk -F'"' '{print $4}' > /VERSION.txt
-
-# Force apt to leave downloaded binaries in /var/cache/apt (massively speeds up Docker builds)
-RUN echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache \
-    && rm -f /etc/apt/apt.conf.d/docker-clean
-
-# Print debug info about build and save it to disk, for human eyes only, not used by anything else
-RUN (echo "[i] Docker build for ArchiveBox $(cat /VERSION.txt) starting..." \
-    && echo "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})" \
-    && echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
-    && echo \
-    && echo "GLOBAL_VENV=${GLOBAL_VENV} APP_VENV=${APP_VENV} NODE_MODULES=${NODE_MODULES}" \
-    && echo "PYTHON=${PYTHON_VERSION} NODE=${NODE_VERSION} PATH=${PATH}" \
-    && echo "CODE_DIR=${CODE_DIR} DATA_DIR=${DATA_DIR}" \
-    && echo \
-    && uname -a \
-    && cat /etc/os-release | head -n7 \
-    && which bash && bash --version | head -n1 \
-    && which dpkg && dpkg --version | head -n1 \
-    && echo -e '\n\n' && env && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Create non-privileged user for archivebox and chrome
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_PUID}..." \
-    && groupadd --system $ARCHIVEBOX_USER \
-    && useradd --system --create-home --gid $ARCHIVEBOX_USER --groups audio,video $ARCHIVEBOX_USER \
-    && usermod -u "$DEFAULT_PUID" "$ARCHIVEBOX_USER" \
-    && groupmod -g "$DEFAULT_PGID" "$ARCHIVEBOX_USER" \
-    && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER PUID=$(id -u $ARCHIVEBOX_USER) PGID=$(id -g $ARCHIVEBOX_USER)\n\n" \
-    | tee -a /VERSION.txt
-    # DEFAULT_PUID and DEFAULT_PID are overriden by PUID and PGID in /bin/docker_entrypoint.sh at runtime
-    # https://docs.linuxserver.io/general/understanding-puid-and-pgid
-
-# Install system apt dependencies (adding backports to access more recent apt updates)
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing APT base system dependencies for $TARGETPLATFORM..." \
-    && echo 'deb https://deb.debian.org/debian bookworm-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list \
-    && mkdir -p /etc/apt/keyrings \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        # 1. packaging dependencies
-        apt-transport-https ca-certificates apt-utils gnupg2 curl wget \
-        # 2. docker and init system dependencies
-        zlib1g-dev dumb-init gosu cron unzip grep \
-        # 3. frivolous CLI helpers to make debugging failed archiving easier
-        # nano iputils-ping dnsutils htop procps jq yq
-    && rm -rf /var/lib/apt/lists/*
-
-######### Language Environments ####################################
-
-# Install Node environment
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.npm,sharing=locked,id=npm-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing Node $NODE_VERSION environment in $NODE_MODULES..." \
-    && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_VERSION}.x nodistro main" >> /etc/apt/sources.list.d/nodejs.list \
-    && curl -fsSL "https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key" | gpg --dearmor | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        nodejs libatomic1 python3-minimal \
-    && rm -rf /var/lib/apt/lists/* \
-    # Update NPM to latest version
-    && npm i -g npm --cache /root/.npm \
-    # Save version info
-    && ( \
-        which node && node --version \
-        && which npm && npm --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install Python environment
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Setting up Python $PYTHON_VERSION runtime..." \
-    # tell PDM to allow using global system python site packages
-    # && rm /usr/lib/python3*/EXTERNALLY-MANAGED \
-    # create global virtual environment GLOBAL_VENV to use (better than using pip install --global)
-    # && python3 -m venv --system-site-packages --symlinks $GLOBAL_VENV \
-    # && python3 -m venv --system-site-packages $GLOBAL_VENV \
-    # && python3 -m venv $GLOBAL_VENV \
-    # install global dependencies / python build dependencies in GLOBAL_VENV
-    # && pip install --upgrade pip setuptools wheel \
-    # Save version info
-    && ( \
-        which python3 && python3 --version | grep " $PYTHON_VERSION" \
-        && which pip && pip --version \
-        # && which pdm && pdm --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-######### Extractor Dependencies ##################################
-
-# Install apt dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing APT extractor dependencies globally using apt..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        curl wget git yt-dlp ffmpeg ripgrep \
-        # Packages we have also needed in the past:
-        # youtube-dl wget2 aria2 python3-pyxattr rtmpdump libfribidi-bin mpv \
-        # fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
-    && rm -rf /var/lib/apt/lists/* \
-    # Save version info
-    && ( \
-        which curl && curl --version | head -n1 \
-        && which wget && wget --version 2>&1 | head -n1 \
-        && which yt-dlp && yt-dlp --version 2>&1 | head -n1 \
-        && which git && git --version 2>&1 | head -n1 \
-        && which rg && rg --version 2>&1 | head -n1 \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install chromium browser using playwright
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/ms-playwright,sharing=locked,id=browsers-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing Browser binary dependencies to $PLAYWRIGHT_BROWSERS_PATH..." \
-    && apt-get update -qq \
-    && if [[ "$TARGETPLATFORM" == *amd64* || "$TARGETPLATFORM" == *arm64* ]]; then \
-        # install Chromium using playwright
-        pip install playwright \
-        && cp -r /root/.cache/ms-playwright "$PLAYWRIGHT_BROWSERS_PATH" \
-        && playwright install --with-deps chromium \
-        && export CHROME_BINARY="$(python -c 'from playwright.sync_api import sync_playwright; print(sync_playwright().start().chromium.executable_path)')"; \
-    else \
-        # fall back to installing Chromium via apt-get on platforms not supported by playwright (e.g. risc, ARMv7, etc.) 
-        apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-            chromium fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
-        && export CHROME_BINARY="$(which chromium)"; \
-    fi \
-    && rm -rf /var/lib/apt/lists/* \
-    && ln -s "$CHROME_BINARY" /usr/bin/chromium-browser \
-    && mkdir -p "/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/" \
-    && chown -R $ARCHIVEBOX_USER "/home/${ARCHIVEBOX_USER}/.config" \
-    && mkdir -p "$PLAYWRIGHT_BROWSERS_PATH" \
-    && chown -R $ARCHIVEBOX_USER "$PLAYWRIGHT_BROWSERS_PATH" \
-    # Save version info
-    && ( \
-        which chromium-browser && /usr/bin/chromium-browser --version || /usr/lib/chromium/chromium --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install Node dependencies
-WORKDIR "$CODE_DIR"
-COPY --chown=root:root --chmod=755 "package.json" "package-lock.json" "$CODE_DIR"/
-RUN --mount=type=cache,target=/root/.npm,sharing=locked,id=npm-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing NPM extractor dependencies from package.json into $NODE_MODULES..." \
-    && npm ci --prefer-offline --no-audit --cache /root/.npm \
-    && ( \
-        which node && node --version \
-        && which npm && npm version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-######### Build Dependencies ####################################
-
-# Install ArchiveBox Python dependencies
-WORKDIR "$CODE_DIR"
-COPY --chown=root:root --chmod=755 "./pyproject.toml" "requirements.txt" "$CODE_DIR"/
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing PIP ArchiveBox dependencies from requirements.txt for ${TARGETPLATFORM}..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        build-essential \
-        libssl-dev libldap2-dev libsasl2-dev \
-        python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps \
-    # && ln -s "$GLOBAL_VENV" "$APP_VENV" \
-    # && pdm use --venv in-project \
-    # && pdm run python -m ensurepip \
-    # && pdm sync --fail-fast --no-editable --group :all --no-self \
-    # && pdm export -o requirements.txt --without-hashes \
-    # && source $GLOBAL_VENV/bin/activate \
-    && pip install -r requirements.txt \
-    && apt-get purge -y \
-        build-essential \
-    && apt-get autoremove -y \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install ArchiveBox Python package from source
-COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[*] Installing PIP ArchiveBox package from $CODE_DIR..." \
-    # && apt-get update -qq \
-    # install C compiler to build deps on platforms that dont have 32-bit wheels available on pypi
-    # && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-    #     build-essential  \
-    # INSTALL ARCHIVEBOX python package globally from CODE_DIR, with all optional dependencies
-    && pip install -e "$CODE_DIR"[sonic,ldap] \
-    # save docker image size and always remove compilers / build tools after building is complete
-    # && apt-get purge -y build-essential \
-    # && apt-get autoremove -y \
-    && rm -rf /var/lib/apt/lists/*
-
-####################################################
-
-# Setup ArchiveBox runtime config
-WORKDIR "$DATA_DIR"
-ENV IN_DOCKER=True
-    ## No need to set explicitly, these values will be autodetected by archivebox in docker:
-    # CHROME_SANDBOX=False \
-    # WGET_BINARY="wget" \
-    # YOUTUBEDL_BINARY="yt-dlp" \
-    # CHROME_BINARY="/usr/bin/chromium-browser" \
-    # USE_SINGLEFILE=True \
-    # SINGLEFILE_BINARY="$NODE_MODULES/.bin/single-file" \
-    # USE_READABILITY=True \
-    # READABILITY_BINARY="$NODE_MODULES/.bin/readability-extractor" \
-    # USE_MERCURY=True \
-    # MERCURY_BINARY="$NODE_MODULES/.bin/postlight-parser"
-
-# Print version for nice docker finish summary
-RUN (echo -e "\n\n[√] Finished Docker build succesfully. Saving build summary in: /VERSION.txt" \
-    && echo -e "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})\n" \
-    && echo -e "BUILD_END_TIME=$(date +"%Y-%m-%d %H:%M:%S %s")\n\n" \
-    ) | tee -a /VERSION.txt
-RUN "$CODE_DIR"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt
-
-####################################################
-
-# Open up the interfaces to the outside world
-WORKDIR "$DATA_DIR"
-VOLUME "$DATA_DIR"
-EXPOSE 8000
-
-# Optional:
-# HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
-#     CMD curl --silent 'http://localhost:8000/admin/login/' || exit 1
-
-ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
-CMD ["archivebox", "server", "--quick-init", "0.0.0.0:8000"]

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 Nick Sweeting
+Copyright (c) 2017 Nick Sweeting
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

SECURITY.md

@@ -1,34 +0,0 @@
-# Security Policy
-
----
-
-## Security Information
-
-Please see this wiki page for important notices about ArchiveBox security, publishing your archives securely, and the dangers of executing archived JS:
-
-https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
-
-Also see this section of the README about important caveats when running ArchiveBox:
-
-https://github.com/ArchiveBox/ArchiveBox?tab=readme-ov-file#caveats
-
-You can also read these pages for more information about ArchiveBox's internals, development environment, DB schema, and more:
-
-- https://github.com/ArchiveBox/ArchiveBox#archive-layout
-- https://github.com/ArchiveBox/ArchiveBox#archivebox-development
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting
-
----
-
-## Reporting a Vulnerability
-
-We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.
-
-1. Go to the Security tab on our Github repo: https://github.com/ArchiveBox/ArchiveBox/security
-
-2. Click the ["Report a Vulnerability"](https://github.com/ArchiveBox/ArchiveBox/security/advisories/new) button
-
-3. Fill out the form to submit the details of the report and it will be securely sent to the maintainers
-
-You can also contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp).

archive

@@ -0,0 +1 @@
+bin/bookmark-archiver

archivebox/.flake8

@@ -1,6 +0,0 @@
-[flake8]
-ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391
-select = F,E9,W
-max-line-length = 130
-max-complexity = 10
-exclude = migrations,tests,node_modules,vendor,venv,.venv,.venv2,.docker-venv

archivebox/LICENSE

@@ -1 +0,0 @@
-../LICENSE

archivebox/README.md

@@ -1 +0,0 @@
-../README.md

archivebox/__init__.py

@@ -1 +0,0 @@
-__package__ = 'archivebox'

archivebox/__main__.py

@@ -1,11 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox'
-
-import sys
-
-from .cli import main
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/__init__.py

@@ -1,155 +0,0 @@
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox'
-
-import os
-import sys
-import argparse
-
-from typing import Optional, Dict, List, IO, Union
-from pathlib import Path
-
-from ..config import OUTPUT_DIR, check_data_folder, check_migrations
-
-from importlib import import_module
-
-CLI_DIR = Path(__file__).resolve().parent
-
-# these common commands will appear sorted before any others for ease-of-use
-meta_cmds = ('help', 'version')                               # dont require valid data folder at all
-main_cmds = ('init', 'config', 'setup')                       # dont require existing db present
-archive_cmds = ('add', 'remove', 'update', 'list', 'status')  # require existing db present
-fake_db = ("oneshot",)                                        # use fake in-memory db
-
-display_first = (*meta_cmds, *main_cmds, *archive_cmds)
-
-# every imported command module must have these properties in order to be valid
-required_attrs = ('__package__', '__command__', 'main')
-
-# basic checks to make sure imported files are valid subcommands
-is_cli_module = lambda fname: fname.startswith('archivebox_') and fname.endswith('.py')
-is_valid_cli_module = lambda module, subcommand: (
-    all(hasattr(module, attr) for attr in required_attrs)
-    and module.__command__.split(' ')[-1] == subcommand
-)
-
-
-def list_subcommands() -> Dict[str, str]:
-    """find and import all valid archivebox_<subcommand>.py files in CLI_DIR"""
-
-    COMMANDS = []
-    for filename in os.listdir(CLI_DIR):
-        if is_cli_module(filename):
-            subcommand = filename.replace('archivebox_', '').replace('.py', '')
-            module = import_module('.archivebox_{}'.format(subcommand), __package__)
-            assert is_valid_cli_module(module, subcommand)
-            COMMANDS.append((subcommand, module.main.__doc__))
-            globals()[subcommand] = module.main
-
-    display_order = lambda cmd: (
-        display_first.index(cmd[0])
-        if cmd[0] in display_first else
-        100 + len(cmd[0])
-    )
-
-    return dict(sorted(COMMANDS, key=display_order))
-
-
-def run_subcommand(subcommand: str,
-                   subcommand_args: List[str]=None,
-                   stdin: Optional[IO]=None,
-                   pwd: Union[Path, str, None]=None) -> None:
-    """Run a given ArchiveBox subcommand with the given list of args"""
-
-    subcommand_args = subcommand_args or []
-
-    if subcommand not in meta_cmds:
-        from ..config import setup_django
-
-        cmd_requires_db = subcommand in archive_cmds
-        init_pending = '--init' in subcommand_args or '--quick-init' in subcommand_args
-
-        if cmd_requires_db:
-            check_data_folder(pwd)
-
-        setup_django(in_memory_db=subcommand in fake_db, check_db=cmd_requires_db and not init_pending)
-
-        if cmd_requires_db:
-            check_migrations()
-
-    module = import_module('.archivebox_{}'.format(subcommand), __package__)
-    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
-
-
-SUBCOMMANDS = list_subcommands()
-
-class NotProvided:
-    pass
-
-
-def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided, pwd: Optional[str]=None) -> None:
-    args = sys.argv[1:] if args is NotProvided else args
-    stdin = sys.stdin if stdin is NotProvided else stdin
-
-    subcommands = list_subcommands()
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description='ArchiveBox: The self-hosted internet archive',
-        add_help=False,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--help', '-h',
-        action='store_true',
-        help=subcommands['help'],
-    )
-    group.add_argument(
-        '--version',
-        action='store_true',
-        help=subcommands['version'],
-    )
-    group.add_argument(
-        "subcommand",
-        type=str,
-        help= "The name of the subcommand to run",
-        nargs='?',
-        choices=subcommands.keys(),
-        default=None,
-    )
-    parser.add_argument(
-        "subcommand_args",
-        help="Arguments for the subcommand",
-        nargs=argparse.REMAINDER,
-    )
-    command = parser.parse_args(args or ())
-
-    if command.version:
-        command.subcommand = 'version'
-    elif command.help or command.subcommand is None:
-        command.subcommand = 'help'
-
-    if command.subcommand not in ('help', 'version', 'status'):
-        from ..logging_util import log_cli_command
-
-        log_cli_command(
-            subcommand=command.subcommand,
-            subcommand_args=command.subcommand_args,
-            stdin=stdin,
-            pwd=pwd or OUTPUT_DIR
-        )
-
-    run_subcommand(
-        subcommand=command.subcommand,
-        subcommand_args=command.subcommand_args,
-        stdin=stdin,
-        pwd=pwd or OUTPUT_DIR,
-    )
-
-
-__all__ = (
-    'SUBCOMMANDS',
-    'list_subcommands',
-    'run_subcommand',
-    *SUBCOMMANDS.keys(),
-)
-
-

archivebox/cli/archivebox_add.py

@@ -1,140 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox add'
-
-import sys
-import argparse
-
-from typing import List, Optional, IO
-
-from ..main import add
-from ..util import docstring
-from ..parsers import PARSERS
-from ..config import OUTPUT_DIR, ONLY_NEW
-from ..logging_util import SmartFormatter, accept_stdin, stderr
-
-
-@docstring(add.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=add.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--tag', '-t',
-        type=str,
-        default='',
-        help="Tag the added URLs with the provided tags e.g. --tag=tag1,tag2,tag3",
-    )
-    parser.add_argument(
-        '--update', #'-u',
-        action='store_true',
-        default=not ONLY_NEW,  # when ONLY_NEW=True we skip updating old links
-        help="Also retry previously skipped/failed links when adding new links",
-    )
-    parser.add_argument(
-        '--update-all', #'-n',
-        action='store_true',
-        default=False, 
-        help="Also update ALL links in index when finished adding new links",
-    )
-    parser.add_argument(
-        '--index-only', #'-o',
-        action='store_true',
-        help="Add the links to the main index without archiving them",
-    )
-    parser.add_argument(
-        'urls',
-        nargs='*',
-        type=str,
-        default=None,
-        help=(
-            'URLs or paths to archive e.g.:\n'
-            '    https://getpocket.com/users/USERNAME/feed/all\n'
-            '    https://example.com/some/rss/feed.xml\n'
-            '    https://example.com\n'
-            '    ~/Downloads/firefox_bookmarks_export.html\n'
-            '    ~/Desktop/sites_list.csv\n'
-        )
-    )
-    parser.add_argument(
-        "--depth",
-        action="store",
-        default=0,
-        choices=[0, 1],
-        type=int,
-        help="Recursively archive all linked pages up to this many hops away"
-    )
-    parser.add_argument(
-        "--overwrite",
-        default=False,
-        action="store_true",
-        help="Re-archive URLs from scratch, overwriting any existing files"
-    )
-    parser.add_argument(
-        "--init", #'-i',
-        action='store_true',
-        help="Init/upgrade the curent data directory before adding",
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    parser.add_argument(
-        "--parser",
-        type=str,
-        help="Parser used to read inputted URLs.",
-        default="auto",
-        choices=["auto", *PARSERS.keys()],
-    )
-    command = parser.parse_args(args or ())
-    urls = command.urls
-
-    stdin_urls = ''
-    if not urls:
-        stdin_urls = accept_stdin(stdin)
-
-    if (stdin_urls and urls) or (not stdin and not urls):
-        stderr(
-            '[X] You must pass URLs/paths to add via stdin or CLI arguments.\n',
-            color='red',
-        )
-        raise SystemExit(2)
-    add(
-        urls=stdin_urls or urls,
-        depth=command.depth,
-        tag=command.tag,
-        update=command.update,
-        update_all=command.update_all,
-        index_only=command.index_only,
-        overwrite=command.overwrite,
-        init=command.init,
-        extractors=command.extract,
-        parser=command.parser,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)
-
-
-# TODO: Implement these
-#
-# parser.add_argument(
-#     '--mirror', #'-m',
-#     action='store_true',
-#     help='Archive an entire site (finding all linked pages below it on the same domain)',
-# )
-# parser.add_argument(
-#     '--crawler', #'-r',
-#     choices=('depth_first', 'breadth_first'),
-#     help='Controls which crawler to use in order to find outlinks in a given page',
-#     default=None,
-# )

archivebox/cli/archivebox_config.py

@@ -1,64 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox config'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import config
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(config.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=config.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--get', #'-g',
-        action='store_true',
-        help="Get the value for the given config KEYs",
-    )
-    group.add_argument(
-        '--set', #'-s',
-        action='store_true',
-        help="Set the given KEY=VALUE config values",
-    )
-    group.add_argument(
-        '--reset', #'-s',
-        action='store_true',
-        help="Reset the given KEY config values to their defaults",
-    )
-    parser.add_argument(
-        'config_options',
-        nargs='*',
-        type=str,
-        help='KEY or KEY=VALUE formatted config values to get or set',
-    )
-    command = parser.parse_args(args or ())
-
-    config_options_str = ''
-    if not command.config_options:
-        config_options_str = accept_stdin(stdin)
-
-    config(
-        config_options_str=config_options_str,
-        config_options=command.config_options,
-        get=command.get,
-        set=command.set,
-        reset=command.reset,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_help.py

@@ -1,32 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox help'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import help
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(help.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=help.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    help(out_dir=pwd or OUTPUT_DIR)
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_init.py

@@ -1,52 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox init'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import init
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(init.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=init.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--force', # '-f',
-        action='store_true',
-        help='Ignore unrecognized files in current directory and initialize anyway',
-    )
-    parser.add_argument(
-        '--quick', '-q',
-        action='store_true',
-        help='Run any updates or migrations without rechecking all snapshot dirs',
-    )
-    parser.add_argument(
-        '--setup', #'-s',
-        action='store_true',
-        help='Automatically install dependencies and extras used for archiving',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    init(
-        force=command.force,
-        quick=command.quick,
-        setup=command.setup,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_list.py

@@ -1,139 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox list'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import list_all
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..index import (
-    LINK_FILTERS,
-    get_indexed_folders,
-    get_archived_folders,
-    get_unarchived_folders,
-    get_present_folders,
-    get_valid_folders,
-    get_invalid_folders,
-    get_duplicate_folders,
-    get_orphaned_folders,
-    get_corrupted_folders,
-    get_unrecognized_folders,
-)
-from ..logging_util import SmartFormatter, reject_stdin, stderr
-
-
-@docstring(list_all.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=list_all.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--csv', #'-c',
-        type=str,
-        help="Print the output in CSV format with the given columns, e.g.: timestamp,url,extension",
-        default=None,
-    )
-    group.add_argument(
-        '--json', #'-j',
-        action='store_true',
-        help="Print the output in JSON format with all columns included",
-    )
-    group.add_argument(
-        '--html',
-        action='store_true',
-        help="Print the output in HTML format"
-    )
-    parser.add_argument(
-        '--with-headers',
-        action='store_true',
-        help='Include the headers in the output document' 
-    )
-    parser.add_argument(
-        '--sort', #'-s',
-        type=str,
-        help="List the links sorted using the given key, e.g. timestamp or updated",
-        default=None,
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="List only links bookmarked before (less than) the given timestamp",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="List only links bookmarked after (greater than or equal to) the given timestamp",
-        default=None,
-    )
-    parser.add_argument(
-        '--status',
-        type=str,
-        choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'),
-        default='indexed',
-        help=(
-            'List only links or data directories that have the given status\n'
-            f'    indexed       {get_indexed_folders.__doc__} (the default)\n'
-            f'    archived      {get_archived_folders.__doc__}\n'
-            f'    unarchived    {get_unarchived_folders.__doc__}\n'
-            '\n'
-            f'    present       {get_present_folders.__doc__}\n'
-            f'    valid         {get_valid_folders.__doc__}\n'
-            f'    invalid       {get_invalid_folders.__doc__}\n'
-            '\n'
-            f'    duplicate     {get_duplicate_folders.__doc__}\n'
-            f'    orphaned      {get_orphaned_folders.__doc__}\n'
-            f'    corrupted     {get_corrupted_folders.__doc__}\n'
-            f'    unrecognized  {get_unrecognized_folders.__doc__}\n'
-        )
-    )
-    parser.add_argument(
-        '--filter-type', '-t',
-        type=str,
-        choices=(*LINK_FILTERS.keys(), 'search'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        default=None,
-        help='List only URLs matching these filter patterns'
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(stdin)
-
-    if command.with_headers and not (command.json or command.html or command.csv):
-        stderr(
-            '[X] --with-headers can only be used with --json, --html or --csv options\n',
-            color='red',
-        )
-        raise SystemExit(2)
-
-    matching_folders = list_all(
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        status=command.status,
-        after=command.after,
-        before=command.before,
-        sort=command.sort,
-        csv=command.csv,
-        json=command.json,
-        html=command.html,
-        with_headers=command.with_headers,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    raise SystemExit(not matching_folders)
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_manage.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox manage'
-
-import sys
-
-from typing import Optional, List, IO
-
-from ..main import manage
-from ..util import docstring
-from ..config import OUTPUT_DIR
-
-
-@docstring(manage.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    manage(
-        args=args,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_oneshot.py

@@ -1,73 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox oneshot'
-
-import sys
-import argparse
-
-from pathlib import Path
-from typing import List, Optional, IO
-
-from ..main import oneshot
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin, stderr
-
-
-@docstring(oneshot.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=oneshot.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        'url',
-        type=str,
-        default=None,
-        help=(
-            'URLs or paths to archive e.g.:\n'
-            '    https://getpocket.com/users/USERNAME/feed/all\n'
-            '    https://example.com/some/rss/feed.xml\n'
-            '    https://example.com\n'
-            '    ~/Downloads/firefox_bookmarks_export.html\n'
-            '    ~/Desktop/sites_list.csv\n'
-        )
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    parser.add_argument(
-        '--out-dir',
-        type=str,
-        default=OUTPUT_DIR,
-        help= "Path to save the single archive folder to, e.g. ./example.com_archive"
-    )
-    command = parser.parse_args(args or ())
-    stdin_url = None
-    url = command.url
-    if not url:
-        stdin_url = accept_stdin(stdin)
-
-    if (stdin_url and url) or (not stdin and not url):
-        stderr(
-            '[X] You must pass a URL/path to add via stdin or CLI arguments.\n',
-            color='red',
-        )
-        raise SystemExit(2)
-    
-    oneshot(
-        url=stdin_url or url,
-        out_dir=Path(command.out_dir).resolve(),
-        extractors=command.extract,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_remove.py

@@ -1,82 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox remove'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import remove
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(remove.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=remove.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--yes', # '-y',
-        action='store_true',
-        help='Remove links instantly without prompting to confirm.',
-    )
-    parser.add_argument(
-        '--delete', # '-r',
-        action='store_true',
-        help=(
-            "In addition to removing the link from the index, "
-            "also delete its archived content and metadata folder."
-        ),
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="List only URLs bookmarked before the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="List only URLs bookmarked after the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--filter-type',
-        type=str,
-        choices=('exact', 'substring', 'domain', 'regex','tag'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        help='URLs matching this filter pattern will be removed from the index.'
-    )
-    command = parser.parse_args(args or ())
-    
-    filter_str = None
-    if not command.filter_patterns:
-        filter_str = accept_stdin(stdin)
-
-    remove(
-        filter_str=filter_str,
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        before=command.before,
-        after=command.after,
-        yes=command.yes,
-        delete=command.delete,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_schedule.py

@@ -1,116 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox schedule'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import schedule
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(schedule.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=schedule.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--quiet', '-q',
-        action='store_true',
-        help=("Don't warn about storage space."),
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--add', # '-a',
-        action='store_true',
-        help='Add a new scheduled ArchiveBox update job to cron',
-    )
-    parser.add_argument(
-        '--every', # '-e',
-        type=str,
-        default=None,
-        help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")',
-    )
-    parser.add_argument(
-        '--tag', '-t',
-        type=str,
-        default='',
-        help="Tag the added URLs with the provided tags e.g. --tag=tag1,tag2,tag3",
-    )
-    parser.add_argument(
-        '--depth', # '-d',
-        type=int,
-        choices=[0, 1],
-        default=0,
-        help='Depth to archive to [0] or 1, see "add" command help for more info',
-    )
-    parser.add_argument(
-        '--overwrite',
-        action='store_true',
-        help='Re-archive any URLs that have been previously archived, overwriting existing Snapshots',
-    )
-    parser.add_argument(
-        '--update',
-        action='store_true',
-        help='Re-pull any URLs that have been previously added, as needed to fill missing ArchiveResults',
-    )
-    group.add_argument(
-        '--clear', # '-c'
-        action='store_true',
-        help=("Stop all ArchiveBox scheduled runs (remove cron jobs)"),
-    )
-    group.add_argument(
-        '--show', # '-s'
-        action='store_true',
-        help=("Print a list of currently active ArchiveBox cron jobs"),
-    )
-    group.add_argument(
-        '--foreground', '-f',
-        action='store_true',
-        help=("Launch ArchiveBox scheduler as a long-running foreground task "
-              "instead of using cron."),
-    )
-    group.add_argument(
-        '--run-all', # '-a',
-        action='store_true',
-        help=("Run all the scheduled jobs once immediately, independent of "
-              "their configured schedules, can be used together with --foreground"),
-    )
-    parser.add_argument(
-        'import_path',
-        nargs='?',
-        type=str,
-        default=None,
-        help=("Check this path and import any new links on every run "
-              "(can be either local file or remote URL)"),
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    schedule(
-        add=command.add,
-        show=command.show,
-        clear=command.clear,
-        foreground=command.foreground,
-        run_all=command.run_all,
-        quiet=command.quiet,
-        every=command.every,
-        tag=command.tag,
-        depth=command.depth,
-        overwrite=command.overwrite,
-        update=command.update,
-        import_path=command.import_path,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_server.py

@@ -1,76 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox server'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import server
-from ..util import docstring
-from ..config import OUTPUT_DIR, BIND_ADDR
-from ..logging_util import SmartFormatter, reject_stdin
-
-@docstring(server.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=server.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        'runserver_args',
-        nargs='*',
-        type=str,
-        default=[BIND_ADDR],
-        help='Arguments to pass to Django runserver'
-    )
-    parser.add_argument(
-        '--reload',
-        action='store_true',
-        help='Enable auto-reloading when code or templates change',
-    )
-    parser.add_argument(
-        '--debug',
-        action='store_true',
-        help='Enable DEBUG=True mode with more verbose errors',
-    )
-    parser.add_argument(
-        '--nothreading',
-        action='store_true',
-        help='Force runserver to run in single-threaded mode',
-    )
-    parser.add_argument(
-        '--init',
-        action='store_true',
-        help='Run a full archivebox init/upgrade before starting the server',
-    )
-    parser.add_argument(
-        '--quick-init', '-i',
-        action='store_true',
-        help='Run quick archivebox init/upgrade before starting the server',
-    )
-    parser.add_argument(
-        '--createsuperuser',
-        action='store_true',
-        help='Run archivebox manage createsuperuser before starting the server',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    server(
-        runserver_args=command.runserver_args + (['--nothreading'] if command.nothreading else []),
-        reload=command.reload,
-        debug=command.debug,
-        init=command.init,
-        quick_init=command.quick_init,
-        createsuperuser=command.createsuperuser,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_setup.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox setup'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import setup
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(setup.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=setup.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    # parser.add_argument(
-    #     '--force', # '-f',
-    #     action='store_true',
-    #     help='Overwrite any existing packages that conflict with the ones ArchiveBox is trying to install',
-    # )
-    command = parser.parse_args(args or ())   # noqa
-    reject_stdin(__command__, stdin)
-
-    setup(
-        # force=command.force,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_shell.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox shell'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import shell
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(shell.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=shell.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    shell(
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_status.py

@@ -1,32 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox status'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import status
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(status.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=status.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    status(out_dir=pwd or OUTPUT_DIR)
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_update.py

@@ -1,136 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox update'
-
-import sys
-import argparse
-
-from typing import List, Optional, IO
-
-from ..main import update
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..index import (
-    LINK_FILTERS,
-    get_indexed_folders,
-    get_archived_folders,
-    get_unarchived_folders,
-    get_present_folders,
-    get_valid_folders,
-    get_invalid_folders,
-    get_duplicate_folders,
-    get_orphaned_folders,
-    get_corrupted_folders,
-    get_unrecognized_folders,
-)
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(update.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=update.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--only-new', #'-n',
-        action='store_true',
-        help="Don't attempt to retry previously skipped/failed links when updating",
-    )
-    parser.add_argument(
-        '--index-only', #'-o',
-        action='store_true',
-        help="Update the main index without archiving any content",
-    )
-    parser.add_argument(
-        '--resume', #'-r',
-        type=float,
-        help='Resume the update process from a given timestamp',
-        default=None,
-    )
-    parser.add_argument(
-        '--overwrite', #'-x',
-        action='store_true',
-        help='Ignore existing archived content and overwrite with new versions (DANGEROUS)',
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="Update only links bookmarked before the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="Update only links bookmarked after the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--status',
-        type=str,
-        choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'),
-        default='indexed',
-        help=(
-            'Update only links or data directories that have the given status\n'
-            f'    indexed       {get_indexed_folders.__doc__} (the default)\n'
-            f'    archived      {get_archived_folders.__doc__}\n'
-            f'    unarchived    {get_unarchived_folders.__doc__}\n'
-            '\n'
-            f'    present       {get_present_folders.__doc__}\n'
-            f'    valid         {get_valid_folders.__doc__}\n'
-            f'    invalid       {get_invalid_folders.__doc__}\n'
-            '\n'
-            f'    duplicate     {get_duplicate_folders.__doc__}\n'
-            f'    orphaned      {get_orphaned_folders.__doc__}\n'
-            f'    corrupted     {get_corrupted_folders.__doc__}\n'
-            f'    unrecognized  {get_unrecognized_folders.__doc__}\n'
-        )
-    )
-    parser.add_argument(
-        '--filter-type', '-t',
-        type=str,
-        choices=(*LINK_FILTERS.keys(), 'search'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        default=None,
-        help='Update only URLs matching these filter patterns.'
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    command = parser.parse_args(args or ())
-
-    filter_patterns_str = None
-    if not command.filter_patterns:
-        filter_patterns_str = accept_stdin(stdin)
-
-    update(
-        resume=command.resume,
-        only_new=command.only_new,
-        index_only=command.index_only,
-        overwrite=command.overwrite,
-        filter_patterns_str=filter_patterns_str,
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        status=command.status,
-        after=command.after,
-        before=command.before,
-        out_dir=pwd or OUTPUT_DIR,
-        extractors=command.extract,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_version.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox version'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import version
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(version.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=version.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--quiet', '-q',
-        action='store_true',
-        help='Only print ArchiveBox version number and nothing else.',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    version(
-        quiet=command.quiet,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/tests.py

@@ -1,227 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-
-
-import os
-import sys
-import shutil
-import unittest
-from pathlib import Path
-
-from contextlib import contextmanager
-
-TEST_CONFIG = {
-    'USE_COLOR': 'False',
-    'SHOW_PROGRESS': 'False',
-
-    'OUTPUT_DIR': 'data.tests',
-    
-    'SAVE_ARCHIVE_DOT_ORG': 'False',
-    'SAVE_TITLE': 'False',
-    
-    'USE_CURL': 'False',
-    'USE_WGET': 'False',
-    'USE_GIT': 'False',
-    'USE_CHROME': 'False',
-    'USE_YOUTUBEDL': 'False',
-}
-
-OUTPUT_DIR = 'data.tests'
-os.environ.update(TEST_CONFIG)
-
-from ..main import init
-from ..index import load_main_index
-from ..config import (
-    SQL_INDEX_FILENAME,
-    JSON_INDEX_FILENAME,
-    HTML_INDEX_FILENAME,
-)
-
-from . import (
-    archivebox_init,
-    archivebox_add,
-    archivebox_remove,
-)
-
-HIDE_CLI_OUTPUT = True
-
-test_urls = '''
-https://example1.com/what/is/happening.html?what=1#how-about-this=1
-https://example2.com/what/is/happening/?what=1#how-about-this=1
-HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f
-https://example4.com/what/is/happening.html
-https://example5.com/
-https://example6.com
-
-<test>http://example7.com</test>
-[https://example8.com/what/is/this.php?what=1]
-[and http://example9.com?what=1&other=3#and-thing=2]
-<what>https://example10.com#and-thing=2 "</about>
-abc<this["https://subb.example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def
-sdflkf[what](https://subb.example12.com/who/what.php?whoami=1#whatami=2)?am=hi
-example13.bada
-and example14.badb
-<or>htt://example15.badc</that>
-'''
-
-stdout = sys.stdout
-stderr = sys.stderr
-
-
-@contextmanager
-def output_hidden(show_failing=True):
-    if not HIDE_CLI_OUTPUT:
-        yield
-        return
-
-    sys.stdout = open('stdout.txt', 'w+', encoding='utf-8')
-    sys.stderr = open('stderr.txt', 'w+', encoding='utf-8')
-    try:
-        yield
-        sys.stdout.close()
-        sys.stderr.close()
-        sys.stdout = stdout
-        sys.stderr = stderr
-    except Exception:
-        sys.stdout.close()
-        sys.stderr.close()
-        sys.stdout = stdout
-        sys.stderr = stderr
-        if show_failing:
-            with open('stdout.txt', 'r', encoding='utf-8') as f:
-                print(f.read())
-            with open('stderr.txt', 'r', encoding='utf-8') as f:
-                print(f.read())
-        raise
-    finally:
-        os.remove('stdout.txt')
-        os.remove('stderr.txt')
-
-
-class TestInit(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-
-    def tearDown(self):
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-    def test_basic_init(self):
-        with output_hidden():
-            archivebox_init.main([])
-
-        assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
-        assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
-        assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
-        assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0
-
-    def test_conflicting_init(self):
-        with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+', encoding='utf-8') as f:
-            f.write('test')
-
-        try:
-            with output_hidden(show_failing=False):
-                archivebox_init.main([])
-            assert False, 'Init should have exited with an exception'
-        except SystemExit:
-            pass
-
-        assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
-        assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
-        assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
-        try:
-            load_main_index(out_dir=OUTPUT_DIR)
-            assert False, 'load_main_index should raise an exception when no index is present'
-        except Exception:
-            pass
-
-    def test_no_dirty_state(self):
-        with output_hidden():
-            init()
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-        with output_hidden():
-            init()
-
-
-class TestAdd(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-        with output_hidden():
-            init()
-
-    def tearDown(self):
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-    def test_add_arg_url(self):
-        with output_hidden():
-            archivebox_add.main(['https://getpocket.com/users/nikisweeting/feed/all'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 30
-
-    def test_add_arg_file(self):
-        test_file = Path(OUTPUT_DIR) / 'test.txt'
-        with open(test_file, 'w+', encoding='utf') as f:
-            f.write(test_urls)
-
-        with output_hidden():
-            archivebox_add.main([test_file])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 12
-        os.remove(test_file)
-
-    def test_add_stdin_url(self):
-        with output_hidden():
-            archivebox_add.main([], stdin=test_urls)
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 12
-
-
-class TestRemove(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-        with output_hidden():
-            init()
-            archivebox_add.main([], stdin=test_urls)
-
-    # def tearDown(self):
-        # shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-
-    def test_remove_exact(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', 'https://example5.com/'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 11
-
-    def test_remove_regex(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', '--filter-type=regex', r'http(s)?:\/\/(.+\.)?(example\d\.com)'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 4
-
-    def test_remove_domain(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', '--filter-type=domain', 'example5.com', 'example6.com'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 10
-
-    def test_remove_none(self):
-        try:
-            with output_hidden(show_failing=False):
-                archivebox_remove.main(['--yes', '--delete', 'https://doesntexist.com'])
-            assert False, 'Should raise if no URLs match'
-        except Exception:
-            pass
-
-
-if __name__ == '__main__':
-    if '--verbose' in sys.argv or '-v' in sys.argv:
-        HIDE_CLI_OUTPUT = False
-    
-    unittest.main()

archivebox/config_stubs.py

@@ -1,114 +0,0 @@
-from pathlib import Path
-from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List
-from mypy_extensions import TypedDict
-
-
-
-SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]]
-SimpleConfigValueDict = Dict[str, SimpleConfigValue]
-SimpleConfigValueGetter = Callable[[], SimpleConfigValue]
-ConfigValue = Union[SimpleConfigValue, SimpleConfigValueDict, SimpleConfigValueGetter]
-
-
-class BaseConfig(TypedDict):
-    pass
-
-class ConfigDict(BaseConfig, total=False):
-    """
-    # Regenerate by pasting this quine into `archivebox shell` 🥚
-    from archivebox.config import ConfigDict, CONFIG_DEFAULTS
-    print('class ConfigDict(BaseConfig, total=False):')
-    print('    ' + '"'*3 + ConfigDict.__doc__ + '"'*3)
-    for section, configs in CONFIG_DEFAULTS.items():
-        for key, attrs in configs.items():
-            Type, default = attrs['type'], attrs['default']
-            if default is None:
-                print(f'    {key}: Optional[{Type.__name__}]')
-            else:
-                print(f'    {key}: {Type.__name__}')
-        print()
-    """
-    IS_TTY: bool
-    USE_COLOR: bool
-    SHOW_PROGRESS: bool
-    IN_DOCKER: bool
-
-    PACKAGE_DIR: Path
-    OUTPUT_DIR: Path
-    CONFIG_FILE: Path
-    ONLY_NEW: bool
-    TIMEOUT: int
-    MEDIA_TIMEOUT: int
-    OUTPUT_PERMISSIONS: str
-    RESTRICT_FILE_NAMES: str
-    URL_DENYLIST: str
-
-    SECRET_KEY: Optional[str]
-    BIND_ADDR: str
-    ALLOWED_HOSTS: str
-    DEBUG: bool
-    PUBLIC_INDEX: bool
-    PUBLIC_SNAPSHOTS: bool
-    FOOTER_INFO: str
-
-    SAVE_TITLE: bool
-    SAVE_FAVICON: bool
-    SAVE_WGET: bool
-    SAVE_WGET_REQUISITES: bool
-    SAVE_SINGLEFILE: bool
-    SAVE_READABILITY: bool
-    SAVE_MERCURY: bool
-    SAVE_PDF: bool
-    SAVE_SCREENSHOT: bool
-    SAVE_DOM: bool
-    SAVE_WARC: bool
-    SAVE_GIT: bool
-    SAVE_MEDIA: bool
-    SAVE_ARCHIVE_DOT_ORG: bool
-
-    RESOLUTION: str
-    GIT_DOMAINS: str
-    CHECK_SSL_VALIDITY: bool
-    CURL_USER_AGENT: str
-    WGET_USER_AGENT: str
-    CHROME_USER_AGENT: str
-    COOKIES_FILE: Union[str, Path, None]
-    CHROME_USER_DATA_DIR: Union[str, Path, None]
-    CHROME_TIMEOUT: int
-    CHROME_HEADLESS: bool
-    CHROME_SANDBOX: bool
-
-    USE_CURL: bool
-    USE_WGET: bool
-    USE_SINGLEFILE: bool
-    USE_READABILITY: bool
-    USE_MERCURY: bool
-    USE_GIT: bool
-    USE_CHROME: bool
-    USE_YOUTUBEDL: bool
-    CURL_BINARY: str
-    GIT_BINARY: str
-    WGET_BINARY: str
-    SINGLEFILE_BINARY: str
-    READABILITY_BINARY: str
-    MERCURY_BINARY: str
-    YOUTUBEDL_BINARY: str
-    CHROME_BINARY: Optional[str]
-
-    YOUTUBEDL_ARGS: List[str]
-    WGET_ARGS: List[str]
-    CURL_ARGS: List[str]
-    GIT_ARGS: List[str]
-    TAG_SEPARATOR_PATTERN: str
-
-
-ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue]
-ConfigDefaultValue = Union[ConfigValue, ConfigDefaultValueGetter]
-
-ConfigDefault = TypedDict('ConfigDefault', {
-    'default': ConfigDefaultValue,
-    'type': Optional[Type],
-    'aliases': Optional[Tuple[str, ...]],
-}, total=False)
-
-ConfigDefaultDict = Dict[str, ConfigDefault]

archivebox/core/__init__.py

@@ -1 +0,0 @@
-__package__ = 'archivebox.core'

archivebox/core/admin.py

@@ -1,429 +0,0 @@
-__package__ = 'archivebox.core'
-
-from io import StringIO
-from pathlib import Path
-from contextlib import redirect_stdout
-from datetime import datetime, timezone
-
-from django.contrib import admin
-from django.urls import path
-from django.utils.html import format_html
-from django.utils.safestring import mark_safe
-from django.shortcuts import render, redirect
-from django.contrib.auth import get_user_model
-from django import forms
-
-from ..util import htmldecode, urldecode, ansi_to_html
-
-from core.models import Snapshot, ArchiveResult, Tag
-from core.forms import AddLinkForm
-
-from core.mixins import SearchResultsAdminMixin
-
-from index.html import snapshot_icons
-from logging_util import printable_filesize
-from main import add, remove
-from config import OUTPUT_DIR, SNAPSHOTS_PER_PAGE
-from extractors import archive_links
-
-# Admin URLs
-# /admin/
-# /admin/login/
-# /admin/core/
-# /admin/core/snapshot/
-# /admin/core/snapshot/:uuid/
-# /admin/core/tag/
-# /admin/core/tag/:uuid/
-
-
-# TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel
-
-
-class ArchiveResultInline(admin.TabularInline):
-    model = ArchiveResult
-
-class TagInline(admin.TabularInline):
-    model = Snapshot.tags.through
-
-from django.contrib.admin.helpers import ActionForm
-from django.contrib.admin.widgets import AutocompleteSelectMultiple
-
-# WIP: broken by Django 3.1.2 -> 4.0 migration
-class AutocompleteTags:
-    model = Tag
-    search_fields = ['name']
-    name = 'tags'
-
-class AutocompleteTagsAdminStub:
-    name = 'admin'
-
-
-class SnapshotActionForm(ActionForm):
-    tags = forms.ModelMultipleChoiceField(
-        queryset=Tag.objects.all(),
-        required=False,
-        # WIP: broken by Django 3.1.2 -> 4.0 migration
-        widget=AutocompleteSelectMultiple(
-            AutocompleteTags(),
-            AutocompleteTagsAdminStub(),
-        ),
-    )
-
-    # TODO: allow selecting actions for specific extractors? is this useful?
-    # EXTRACTOR_CHOICES = [
-    #     (name, name.title())
-    #     for name, _, _ in get_default_archive_methods()
-    # ]
-    # extractor = forms.ChoiceField(
-    #     choices=EXTRACTOR_CHOICES,
-    #     required=False,
-    #     widget=forms.MultileChoiceField(attrs={'class': "form-control"})
-    # )
-
-
-class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin):
-    list_display = ('added', 'title_str', 'files', 'size', 'url_str')
-    sort_fields = ('title_str', 'url_str', 'added', 'files')
-    readonly_fields = ('info', 'bookmarked', 'added', 'updated')
-    search_fields = ('id', 'url', 'timestamp', 'title', 'tags__name')
-    fields = ('timestamp', 'url', 'title', 'tags', *readonly_fields)
-    list_filter = ('added', 'updated', 'tags', 'archiveresult__status')
-    ordering = ['-added']
-    actions = ['add_tags', 'remove_tags', 'update_titles', 'update_snapshots', 'resnapshot_snapshot', 'overwrite_snapshots', 'delete_snapshots']
-    autocomplete_fields = ['tags']
-    inlines = [ArchiveResultInline]
-    list_per_page = SNAPSHOTS_PER_PAGE
-
-    action_form = SnapshotActionForm
-
-    def get_urls(self):
-        urls = super().get_urls()
-        custom_urls = [
-            path('grid/', self.admin_site.admin_view(self.grid_view), name='grid')
-        ]
-        return custom_urls + urls
-
-    def get_queryset(self, request):
-        self.request = request
-        return super().get_queryset(request).prefetch_related('tags')
-
-    def tag_list(self, obj):
-        return ', '.join(obj.tags.values_list('name', flat=True))
-
-    # TODO: figure out a different way to do this, you cant nest forms so this doenst work
-    # def action(self, obj):
-    #     # csrfmiddlewaretoken: Wa8UcQ4fD3FJibzxqHN3IYrrjLo4VguWynmbzzcPYoebfVUnDovon7GEMYFRgsh0
-    #     # action: update_snapshots
-    #     # select_across: 0
-    #     # _selected_action: 76d29b26-2a88-439e-877c-a7cca1b72bb3
-    #     return format_html(
-    #         '''
-    #             <form action="/admin/core/snapshot/" method="post" onsubmit="e => e.stopPropagation()">
-    #                 <input type="hidden" name="csrfmiddlewaretoken" value="{}">
-    #                 <input type="hidden" name="_selected_action" value="{}">
-    #                 <button name="update_snapshots">Check</button>
-    #                 <button name="update_titles">Pull title + favicon</button>
-    #                 <button name="update_snapshots">Update</button>
-    #                 <button name="overwrite_snapshots">Re-Archive (overwrite)</button>
-    #                 <button name="delete_snapshots">Permanently delete</button>
-    #             </form>
-    #         ''',
-    #         csrf.get_token(self.request),
-    #         obj.id,
-    #     )
-
-    def info(self, obj):
-        return format_html(
-            '''
-            UUID: <code style="font-size: 10px; user-select: all">{}</code> &nbsp; &nbsp;
-            Timestamp: <code style="font-size: 10px; user-select: all">{}</code> &nbsp; &nbsp;
-            URL Hash: <code style="font-size: 10px; user-select: all">{}</code><br/>
-            Archived: {} ({} files {}) &nbsp; &nbsp;
-            Favicon: <img src="{}" style="height: 20px"/> &nbsp; &nbsp;
-            Status code: {} &nbsp; &nbsp;
-            Server: {} &nbsp; &nbsp;
-            Content type: {} &nbsp; &nbsp;
-            Extension: {} &nbsp; &nbsp;
-            <br/><br/>
-            <a href="/archive/{}">View Snapshot index ➡️</a> &nbsp; &nbsp;
-            <a href="/admin/core/snapshot/?id__exact={}">View actions ⚙️</a>
-            ''',
-            obj.id,
-            obj.timestamp,
-            obj.url_hash,
-            '✅' if obj.is_archived else '❌',
-            obj.num_outputs,
-            self.size(obj),
-            f'/archive/{obj.timestamp}/favicon.ico',
-            obj.status_code or '?',
-            obj.headers and obj.headers.get('Server') or '?',
-            obj.headers and obj.headers.get('Content-Type') or '?',
-            obj.extension or '?',
-            obj.timestamp,
-            obj.id,
-        )
-
-    def title_str(self, obj):
-        canon = obj.as_link().canonical_outputs()
-        tags = ''.join(
-            format_html('<a href="/admin/core/snapshot/?tags__id__exact={}"><span class="tag">{}</span></a> ', tag.id, tag)
-            for tag in obj.tags.all()
-            if str(tag).strip()
-        )
-        return format_html(
-            '<a href="/{}">'
-                '<img src="/{}/{}" class="favicon" onerror="this.remove()">'
-            '</a>'
-            '<a href="/{}/index.html">'
-                '<b class="status-{}">{}</b>'
-            '</a>',
-            obj.archive_path,
-            obj.archive_path, canon['favicon_path'],
-            obj.archive_path,
-            'fetched' if obj.latest_title or obj.title else 'pending',
-            urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...'
-        ) + mark_safe(f' <span class="tags">{tags}</span>')
-
-    def files(self, obj):
-        return snapshot_icons(obj)
-
-    files.admin_order_field = 'updated'
-    files.short_description = 'Files Saved'
-
-    def size(self, obj):
-        archive_size = (Path(obj.link_dir) / 'index.html').exists() and obj.archive_size
-        if archive_size:
-            size_txt = printable_filesize(archive_size)
-            if archive_size > 52428800:
-                size_txt = mark_safe(f'<b>{size_txt}</b>')
-        else:
-            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
-        return format_html(
-            '<a href="/{}" title="View all files">{}</a>',
-            obj.archive_path,
-            size_txt,
-        )
-
-    size.admin_order_field = 'archiveresult__count'
-
-    def url_str(self, obj):
-        return format_html(
-            '<a href="{}"><code style="user-select: all;">{}</code></a>',
-            obj.url,
-            obj.url,
-        )
-
-    def grid_view(self, request, extra_context=None):
-
-        # cl = self.get_changelist_instance(request)
-
-        # Save before monkey patching to restore for changelist list view
-        saved_change_list_template = self.change_list_template
-        saved_list_per_page = self.list_per_page
-        saved_list_max_show_all = self.list_max_show_all
-
-        # Monkey patch here plus core_tags.py
-        self.change_list_template = 'private_index_grid.html'
-        self.list_per_page = SNAPSHOTS_PER_PAGE
-        self.list_max_show_all = self.list_per_page
-
-        # Call monkey patched view
-        rendered_response = self.changelist_view(request, extra_context=extra_context)
-
-        # Restore values
-        self.change_list_template = saved_change_list_template
-        self.list_per_page = saved_list_per_page
-        self.list_max_show_all = saved_list_max_show_all
-
-        return rendered_response
-
-    # for debugging, uncomment this to print all requests:
-    # def changelist_view(self, request, extra_context=None):
-    #     print('[*] Got request', request.method, request.POST)
-    #     return super().changelist_view(request, extra_context=None)
-
-    def update_snapshots(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], out_dir=OUTPUT_DIR)
-    update_snapshots.short_description = "Pull"
-
-    def update_titles(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR)
-    update_titles.short_description = "⬇️ Title"
-
-    def resnapshot_snapshot(self, request, queryset):
-        for snapshot in queryset:
-            timestamp = datetime.now(timezone.utc).isoformat('T', 'seconds')
-            new_url = snapshot.url.split('#')[0] + f'#{timestamp}'
-            add(new_url, tag=snapshot.tags_str())
-    resnapshot_snapshot.short_description = "Re-Snapshot"
-
-    def overwrite_snapshots(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], overwrite=True, out_dir=OUTPUT_DIR)
-    overwrite_snapshots.short_description = "Reset"
-
-    def delete_snapshots(self, request, queryset):
-        remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR)
-
-    delete_snapshots.short_description = "Delete"
-
-    def add_tags(self, request, queryset):
-        tags = request.POST.getlist('tags')
-        print('[+] Adding tags', tags, 'to Snapshots', queryset)
-        for obj in queryset:
-            obj.tags.add(*tags)
-
-    add_tags.short_description = "+"
-
-    def remove_tags(self, request, queryset):
-        tags = request.POST.getlist('tags')
-        print('[-] Removing tags', tags, 'to Snapshots', queryset)
-        for obj in queryset:
-            obj.tags.remove(*tags)
-
-    remove_tags.short_description = "–"
-
-        
-
-    title_str.short_description = 'Title'
-    url_str.short_description = 'Original URL'
-
-    title_str.admin_order_field = 'title'
-    url_str.admin_order_field = 'url'
-
-
-
-class TagAdmin(admin.ModelAdmin):
-    list_display = ('slug', 'name', 'num_snapshots', 'snapshots', 'id')
-    sort_fields = ('id', 'name', 'slug')
-    readonly_fields = ('id', 'num_snapshots', 'snapshots')
-    search_fields = ('id', 'name', 'slug')
-    fields = (*readonly_fields, 'name', 'slug')
-    actions = ['delete_selected']
-    ordering = ['-id']
-
-    def num_snapshots(self, obj):
-        return format_html(
-            '<a href="/admin/core/snapshot/?tags__id__exact={}">{} total</a>',
-            obj.id,
-            obj.snapshot_set.count(),
-        )
-
-    def snapshots(self, obj):
-        total_count = obj.snapshot_set.count()
-        return mark_safe('<br/>'.join(
-            format_html(
-                '{} <code><a href="/admin/core/snapshot/{}/change"><b>[{}]</b></a> {}</code>',
-                snap.updated.strftime('%Y-%m-%d %H:%M') if snap.updated else 'pending...',
-                snap.id,
-                snap.timestamp,
-                snap.url,
-            )
-            for snap in obj.snapshot_set.order_by('-updated')[:10]
-        ) + (f'<br/><a href="/admin/core/snapshot/?tags__id__exact={obj.id}">and {total_count-10} more...<a>' if obj.snapshot_set.count() > 10 else ''))
-
-
-class ArchiveResultAdmin(admin.ModelAdmin):
-    list_display = ('id', 'start_ts', 'extractor', 'snapshot_str', 'tags_str', 'cmd_str', 'status', 'output_str')
-    sort_fields = ('start_ts', 'extractor', 'status')
-    readonly_fields = ('id', 'uuid', 'snapshot_str', 'tags_str')
-    search_fields = ('id', 'uuid', 'snapshot__url', 'extractor', 'output', 'cmd_version', 'cmd', 'snapshot__timestamp')
-    fields = (*readonly_fields, 'snapshot', 'extractor', 'status', 'start_ts', 'end_ts', 'output', 'pwd', 'cmd', 'cmd_version')
-    autocomplete_fields = ['snapshot']
-
-    list_filter = ('status', 'extractor', 'start_ts', 'cmd_version')
-    ordering = ['-start_ts']
-    list_per_page = SNAPSHOTS_PER_PAGE
-
-    def snapshot_str(self, obj):
-        return format_html(
-            '<a href="/archive/{}/index.html"><b><code>[{}]</code></b></a><br/>'
-            '<small>{}</small>',
-            obj.snapshot.timestamp,
-            obj.snapshot.timestamp,
-            obj.snapshot.url[:128],
-        )
-
-    def tags_str(self, obj):
-        return obj.snapshot.tags_str()
-
-    def cmd_str(self, obj):
-        return format_html(
-            '<pre>{}</pre>',
-            ' '.join(obj.cmd) if isinstance(obj.cmd, list) else str(obj.cmd),
-        )
-
-    def output_str(self, obj):
-        return format_html(
-            '<a href="/archive/{}/{}" class="output-link">↗️</a><pre>{}</pre>',
-            obj.snapshot.timestamp,
-            obj.output if (obj.status == 'succeeded') and obj.extractor not in ('title', 'archive_org') else 'index.html',
-            obj.output,
-        )
-
-    tags_str.short_description = 'tags'
-    snapshot_str.short_description = 'snapshot'
-
-class ArchiveBoxAdmin(admin.AdminSite):
-    site_header = 'ArchiveBox'
-    index_title = 'Links'
-    site_title = 'Index'
-
-    def get_urls(self):
-        return [
-            path('core/snapshot/add/', self.add_view, name='Add'),
-        ] + super().get_urls()
-
-    def add_view(self, request):
-        if not request.user.is_authenticated:
-            return redirect(f'/admin/login/?next={request.path}')
-
-        request.current_app = self.name
-        context = {
-            **self.each_context(request),
-            'title': 'Add URLs',
-        }
-
-        if request.method == 'GET':
-            context['form'] = AddLinkForm()
-
-        elif request.method == 'POST':
-            form = AddLinkForm(request.POST)
-            if form.is_valid():
-                url = form.cleaned_data["url"]
-                print(f'[+] Adding URL: {url}')
-                depth = 0 if form.cleaned_data["depth"] == "0" else 1
-                input_kwargs = {
-                    "urls": url,
-                    "depth": depth,
-                    "update_all": False,
-                    "out_dir": OUTPUT_DIR,
-                }
-                add_stdout = StringIO()
-                with redirect_stdout(add_stdout):
-                   add(**input_kwargs)
-                print(add_stdout.getvalue())
-
-                context.update({
-                    "stdout": ansi_to_html(add_stdout.getvalue().strip()),
-                    "form": AddLinkForm()
-                })
-            else:
-                context["form"] = form
-
-        return render(template_name='add.html', request=request, context=context)
-
-admin.site = ArchiveBoxAdmin()
-admin.site.register(get_user_model())
-admin.site.register(Snapshot, SnapshotAdmin)
-admin.site.register(Tag, TagAdmin)
-admin.site.register(ArchiveResult, ArchiveResultAdmin)
-admin.site.disable_action('delete_selected')

archivebox/core/apps.py

@@ -1,7 +0,0 @@
-from django.apps import AppConfig
-
-
-class CoreConfig(AppConfig):
-    name = 'core'
-    # WIP: broken by Django 3.1.2 -> 4.0 migration
-    default_auto_field = 'django.db.models.UUIDField'

archivebox/core/forms.py

@@ -1,96 +0,0 @@
-__package__ = 'archivebox.core'
-
-from django import forms
-
-from ..util import URL_REGEX
-from ..parsers import PARSERS
-from ..vendor.taggit_utils import edit_string_for_tags, parse_tags
-
-PARSER_CHOICES = [
-    (parser_key, parser[0])
-    for parser_key, parser in PARSERS.items()
-]
-DEPTH_CHOICES = (
-    ('0', 'depth = 0 (archive just these URLs)'),
-    ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'),
-)
-
-from ..extractors import get_default_archive_methods
-
-ARCHIVE_METHODS = [
-    (name, name)
-    for name, _, _ in get_default_archive_methods()
-]
-
-
-class AddLinkForm(forms.Form):
-    url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True)
-    parser = forms.ChoiceField(label="URLs format", choices=[('auto', 'Auto-detect parser'), *PARSER_CHOICES], initial='auto')
-    tag = forms.CharField(label="Tags (comma separated tag1,tag2,tag3)", strip=True, required=False)
-    depth = forms.ChoiceField(label="Archive depth", choices=DEPTH_CHOICES, initial='0', widget=forms.RadioSelect(attrs={"class": "depth-selection"}))
-    archive_methods = forms.MultipleChoiceField(
-        label="Archive methods (select at least 1, otherwise all will be used by default)",
-        required=False,
-        widget=forms.SelectMultiple,
-        choices=ARCHIVE_METHODS,
-    )
-    # TODO: hook these up to the view and put them 
-    # in a collapsible UI section labeled "Advanced"
-    #
-    # exclude_patterns = forms.CharField(
-    #     label="Exclude patterns",
-    #     min_length='1',
-    #     required=False,
-    #     initial=URL_DENYLIST,
-    # )
-    # timeout = forms.IntegerField(
-    #     initial=TIMEOUT,
-    # )
-    # overwrite = forms.BooleanField(
-    #     label="Overwrite any existing Snapshots",
-    #     initial=False,
-    # )
-    # index_only = forms.BooleanField(
-    #     label="Add URLs to index without Snapshotting",
-    #     initial=False,
-    # )
-
-class TagWidgetMixin:
-    def format_value(self, value):
-        if value is not None and not isinstance(value, str):
-            value = edit_string_for_tags(value)
-        return super().format_value(value)
-
-class TagWidget(TagWidgetMixin, forms.TextInput):
-    pass
-
-class TagField(forms.CharField):
-    widget = TagWidget
-
-    def clean(self, value):
-        value = super().clean(value)
-        try:
-            return parse_tags(value)
-        except ValueError:
-            raise forms.ValidationError(
-                "Please provide a comma-separated list of tags."
-            )
-
-    def has_changed(self, initial_value, data_value):
-        # Always return False if the field is disabled since self.bound_data
-        # always uses the initial value in this case.
-        if self.disabled:
-            return False
-
-        try:
-            data_value = self.clean(data_value)
-        except forms.ValidationError:
-            pass
-
-        if initial_value is None:
-            initial_value = []
-
-        initial_value = [tag.name for tag in initial_value]
-        initial_value.sort()
-
-        return initial_value != data_value

archivebox/core/management/commands/archivebox.py

@@ -1,18 +0,0 @@
-__package__ = 'archivebox'
-
-from django.core.management.base import BaseCommand
-
-
-from .cli import run_subcommand
-
-
-class Command(BaseCommand):
-    help = 'Run an ArchiveBox CLI subcommand (e.g. add, remove, list, etc)'
-
-    def add_arguments(self, parser):
-        parser.add_argument('subcommand', type=str, help='The subcommand you want to run')
-        parser.add_argument('command_args', nargs='*', help='Arguments to pass to the subcommand')
-
-
-    def handle(self, *args, **kwargs):
-        run_subcommand(kwargs['subcommand'], args=kwargs['command_args'])

archivebox/core/middleware.py

@@ -1,60 +0,0 @@
-__package__ = 'archivebox.core'
-
-import ipaddress
-from django.utils import timezone
-from django.contrib.auth.middleware import RemoteUserMiddleware
-from django.core.exceptions import ImproperlyConfigured
-
-from ..config import PUBLIC_SNAPSHOTS, REVERSE_PROXY_USER_HEADER, REVERSE_PROXY_WHITELIST
-
-
-def detect_timezone(request, activate: bool=True):
-    gmt_offset = (request.COOKIES.get('GMT_OFFSET') or '').strip()
-    tz = None
-    if gmt_offset.replace('-', '').isdigit():
-        tz = timezone.get_fixed_timezone(int(gmt_offset))
-        if activate:
-            timezone.activate(tz)
-    # print('GMT_OFFSET', gmt_offset, tz)
-    return tz
-
-
-def TimezoneMiddleware(get_response):
-    def middleware(request):
-        detect_timezone(request, activate=True)
-        return get_response(request)
-
-    return middleware
-
-
-def CacheControlMiddleware(get_response):
-    def middleware(request):
-        response = get_response(request)
-
-        if '/archive/' in request.path or '/static/' in request.path:
-            policy = 'public' if PUBLIC_SNAPSHOTS else 'private'
-            response['Cache-Control'] = f'{policy}, max-age=60, stale-while-revalidate=300'
-            # print('Set Cache-Control header to', response['Cache-Control'])
-        return response
-
-    return middleware
-
-class ReverseProxyAuthMiddleware(RemoteUserMiddleware):
-    header = 'HTTP_{normalized}'.format(normalized=REVERSE_PROXY_USER_HEADER.replace('-', '_').upper())
-
-    def process_request(self, request):
-        if REVERSE_PROXY_WHITELIST == '':
-            return
-
-        ip = request.META.get('REMOTE_ADDR')
-
-        for cidr in REVERSE_PROXY_WHITELIST.split(','):
-            try:
-                network = ipaddress.ip_network(cidr)
-            except ValueError:
-                raise ImproperlyConfigured(
-                    "The REVERSE_PROXY_WHITELIST config paramater is in invalid format, or "
-                    "contains invalid CIDR. Correct format is a coma-separated list of IPv4/IPv6 CIDRs.")
-
-            if ipaddress.ip_address(ip) in network:
-                return super().process_request(request)

archivebox/core/migrations/0001_initial.py

@@ -1,27 +0,0 @@
-# Generated by Django 2.2 on 2019-05-01 03:27
-
-from django.db import migrations, models
-import uuid
-
-
-class Migration(migrations.Migration):
-
-    initial = True
-
-    dependencies = [
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='Snapshot',
-            fields=[
-                ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
-                ('url', models.URLField(unique=True)),
-                ('timestamp', models.CharField(default=None, max_length=32, null=True, unique=True)),
-                ('title', models.CharField(default=None, max_length=128, null=True)),
-                ('tags', models.CharField(default=None, max_length=256, null=True)),
-                ('added', models.DateTimeField(auto_now_add=True)),
-                ('updated', models.DateTimeField(default=None, null=True)),
-            ],
-        ),
-    ]

archivebox/core/migrations/0002_auto_20200625_1521.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.0.7 on 2020-06-25 15:21
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0001_initial'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(default=None, max_length=32, null=True),
-        ),
-    ]

archivebox/core/migrations/0003_auto_20200630_1034.py

@@ -1,38 +0,0 @@
-# Generated by Django 3.0.7 on 2020-06-30 10:34
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0002_auto_20200625_1521'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='added',
-            field=models.DateTimeField(auto_now_add=True, db_index=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.CharField(db_index=True, default=None, max_length=256, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(db_index=True, default=None, max_length=32, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(db_index=True, default=None, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(db_index=True, default=None, null=True),
-        ),
-    ]

archivebox/core/migrations/0004_auto_20200713_1552.py

@@ -1,19 +0,0 @@
-# Generated by Django 3.0.7 on 2020-07-13 15:52
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0003_auto_20200630_1034'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(db_index=True, default=None, max_length=32, unique=True),
-            preserve_default=False,
-        ),
-    ]

archivebox/core/migrations/0005_auto_20200728_0326.py

@@ -1,28 +0,0 @@
-# Generated by Django 3.0.7 on 2020-07-28 03:26
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0004_auto_20200713_1552'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.CharField(blank=True, db_index=True, max_length=256, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(blank=True, db_index=True, null=True),
-        ),
-    ]

archivebox/core/migrations/0006_auto_20201012_1520.py

@@ -1,70 +0,0 @@
-# Generated by Django 3.0.8 on 2020-10-12 15:20
-
-from django.db import migrations, models
-from django.utils.text import slugify
-
-def forwards_func(apps, schema_editor):
-    SnapshotModel = apps.get_model("core", "Snapshot")
-    TagModel = apps.get_model("core", "Tag")
-
-    db_alias = schema_editor.connection.alias
-    snapshots = SnapshotModel.objects.all()
-    for snapshot in snapshots:
-        tags = snapshot.tags
-        tag_set = (
-            set(tag.strip() for tag in (snapshot.tags_old or '').split(','))
-        )
-        tag_set.discard("")
-
-        for tag in tag_set:
-            to_add, _ = TagModel.objects.get_or_create(name=tag, defaults={'slug': slugify(tag)})
-            snapshot.tags.add(to_add)
-
-
-def reverse_func(apps, schema_editor):
-    SnapshotModel = apps.get_model("core", "Snapshot")
-    TagModel = apps.get_model("core", "Tag")
-
-    db_alias = schema_editor.connection.alias
-    snapshots = SnapshotModel.objects.all()
-    for snapshot in snapshots:
-        tags = snapshot.tags.values_list("name", flat=True)
-        snapshot.tags_old = ",".join([tag for tag in tags])
-        snapshot.save()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0005_auto_20200728_0326'),
-    ]
-
-    operations = [
-        migrations.RenameField(
-            model_name='snapshot',
-            old_name='tags',
-            new_name='tags_old',
-        ),
-        migrations.CreateModel(
-            name='Tag',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('name', models.CharField(max_length=100, unique=True, verbose_name='name')),
-                ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')),
-            ],
-            options={
-                'verbose_name': 'Tag',
-                'verbose_name_plural': 'Tags',
-            },
-        ),
-        migrations.AddField(
-            model_name='snapshot',
-            name='tags',
-            field=models.ManyToManyField(to='core.Tag'),
-        ),
-        migrations.RunPython(forwards_func, reverse_func),
-        migrations.RemoveField(
-            model_name='snapshot',
-            name='tags_old',
-        ),
-    ]

archivebox/core/migrations/0007_archiveresult.py

@@ -1,114 +0,0 @@
-# Generated by Django 3.0.8 on 2020-11-04 12:25
-
-import json
-from pathlib import Path
-
-from django.db import migrations, models
-import django.db.models.deletion
-
-from config import CONFIG
-from index.json import to_json
-
-try:
-    JSONField = models.JSONField
-except AttributeError:
-    import jsonfield
-    JSONField = jsonfield.JSONField
-
-
-def forwards_func(apps, schema_editor):
-    from core.models import EXTRACTORS
-
-    Snapshot = apps.get_model("core", "Snapshot")
-    ArchiveResult = apps.get_model("core", "ArchiveResult")
-
-    snapshots = Snapshot.objects.all()
-    for snapshot in snapshots:
-        out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
-
-        try:
-            with open(out_dir / "index.json", "r") as f:
-                fs_index = json.load(f)
-        except Exception as e:
-            continue
-
-        history = fs_index["history"]
-
-        for extractor in history:
-            for result in history[extractor]:
-                try:
-                    ArchiveResult.objects.create(
-                        extractor=extractor,
-                        snapshot=snapshot,
-                        pwd=result["pwd"],
-                        cmd=result.get("cmd") or [],
-                        cmd_version=result.get("cmd_version") or 'unknown',
-                        start_ts=result["start_ts"],
-                        end_ts=result["end_ts"],
-                        status=result["status"],
-                        output=result.get("output") or 'null',
-                    )
-                except Exception as e:
-                    print(
-                        '    ! Skipping import due to missing/invalid index.json:',
-                        out_dir,
-                        e,
-                        '(open an issue with this index.json for help)',
-                    )
-
-
-def verify_json_index_integrity(snapshot):
-    results = snapshot.archiveresult_set.all()
-    out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
-    with open(out_dir / "index.json", "r") as f:
-        index = json.load(f)
-
-    history = index["history"]
-    index_results = [result for extractor in history for result in history[extractor]]
-    flattened_results = [result["start_ts"] for result in index_results]
-    
-    missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results]
-
-    for missing in missing_results:
-        index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(),
-                                                    "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output,
-                                                    "schema": "ArchiveResult", "status": missing.status})
-
-    json_index = to_json(index)
-    with open(out_dir / "index.json", "w") as f:
-        f.write(json_index)
-
-
-def reverse_func(apps, schema_editor):
-    Snapshot = apps.get_model("core", "Snapshot")
-    ArchiveResult = apps.get_model("core", "ArchiveResult")
-    for snapshot in Snapshot.objects.all():
-        verify_json_index_integrity(snapshot)
-
-    ArchiveResult.objects.all().delete()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0006_auto_20201012_1520'),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='ArchiveResult',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('cmd', JSONField()),
-                ('pwd', models.CharField(max_length=256)),
-                ('cmd_version', models.CharField(max_length=32)),
-                ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)),
-                ('output', models.CharField(max_length=512)),
-                ('start_ts', models.DateTimeField()),
-                ('end_ts', models.DateTimeField()),
-                ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)),
-                ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')),
-            ],
-        ),
-        migrations.RunPython(forwards_func, reverse_func),
-    ]

archivebox/core/migrations/0008_auto_20210105_1421.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-01-05 14:21
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0007_archiveresult'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='cmd_version',
-            field=models.CharField(blank=True, default=None, max_length=32, null=True),
-        ),
-    ]

archivebox/core/migrations/0009_auto_20210216_1038.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 10:38
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0008_auto_20210105_1421'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(auto_now=True, db_index=True, null=True),
-        ),
-    ]

archivebox/core/migrations/0010_auto_20210216_1055.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 10:55
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0009_auto_20210216_1038'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='start_ts',
-            field=models.DateTimeField(db_index=True),
-        ),
-    ]

archivebox/core/migrations/0011_auto_20210216_1331.py

@@ -1,24 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 13:31
-
-from django.db import migrations, models
-import uuid
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0010_auto_20210216_1055'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='archiveresult',
-            name='uuid',
-            field=models.UUIDField(default=uuid.uuid4, editable=False),
-        ),
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/migrations/0012_auto_20210216_1425.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 14:25
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0011_auto_20210216_1331'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='cmd_version',
-            field=models.CharField(blank=True, default=None, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='output',
-            field=models.CharField(max_length=1024),
-        ),
-    ]

archivebox/core/migrations/0013_auto_20210218_0729.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:29
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0012_auto_20210216_1425'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=256, null=True),
-        ),
-    ]

archivebox/core/migrations/0014_auto_20210218_0729.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:29
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0013_auto_20210218_0729'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=1024, null=True),
-        ),
-    ]

archivebox/core/migrations/0015_auto_20210218_0730.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:30
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0014_auto_20210218_0729'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=512, null=True),
-        ),
-    ]

archivebox/core/migrations/0016_auto_20210218_1204.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 12:04
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0015_auto_20210218_0730'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.ManyToManyField(blank=True, to='core.Tag'),
-        ),
-    ]

archivebox/core/migrations/0017_auto_20210219_0211.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-19 02:11
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0016_auto_20210218_1204'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='tag',
-            name='slug',
-            field=models.SlugField(blank=True, max_length=100, unique=True, verbose_name='slug'),
-        ),
-    ]

archivebox/core/migrations/0018_auto_20210327_0952.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.3 on 2021-03-27 09:52
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0017_auto_20210219_0211'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='tag',
-            name='name',
-            field=models.CharField(max_length=100, unique=True),
-        ),
-        migrations.AlterField(
-            model_name='tag',
-            name='slug',
-            field=models.SlugField(blank=True, max_length=100, unique=True),
-        ),
-    ]

archivebox/core/migrations/0019_auto_20210401_0654.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-04-01 06:54
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0018_auto_20210327_0952'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='url',
-            field=models.URLField(db_index=True, unique=True),
-        ),
-    ]

archivebox/core/migrations/0020_auto_20210410_1031.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.8 on 2021-04-10 10:31
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0019_auto_20210401_0654'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='id',
-            field=models.AutoField(primary_key=True, serialize=False, verbose_name='ID'),
-        ),
-        migrations.AlterField(
-            model_name='tag',
-            name='id',
-            field=models.AutoField(primary_key=True, serialize=False, verbose_name='ID'),
-        ),
-    ]

archivebox/core/migrations/0021_auto_20220914_0934.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.14 on 2022-09-14 09:34
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0020_auto_20210410_1031'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('title', 'title'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/migrations/0022_auto_20231023_2008.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.14 on 2023-10-23 20:08
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0021_auto_20220914_0934'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('title', 'title'), ('readability', 'readability'), ('mercury', 'mercury'), ('htmltotext', 'htmltotext'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/mixins.py

@@ -1,21 +0,0 @@
-from django.contrib import messages
-
-from archivebox.search import query_search_index
-
-class SearchResultsAdminMixin:
-    def get_search_results(self, request, queryset, search_term: str):
-        """Enhances the search queryset with results from the search backend"""
-        
-        qs, use_distinct = super().get_search_results(request, queryset, search_term)
-
-        search_term = search_term.strip()
-        if not search_term:
-            return qs, use_distinct
-        try:
-            qsearch = query_search_index(search_term)
-            qs = qs | qsearch
-        except Exception as err:
-            print(f'[!] Error while using search backend: {err.__class__.__name__} {err}')
-            messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}')
-        
-        return qs, use_distinct

archivebox/core/models.py

@@ -1,286 +0,0 @@
-__package__ = 'archivebox.core'
-
-
-import uuid
-import json
-
-from pathlib import Path
-from typing import Optional, List
-
-from django.db import models
-from django.utils.functional import cached_property
-from django.utils.text import slugify
-from django.core.cache import cache
-from django.urls import reverse
-from django.db.models import Case, When, Value, IntegerField
-from django.contrib.auth.models import User   # noqa
-
-from ..config import ARCHIVE_DIR, ARCHIVE_DIR_NAME
-from ..system import get_dir_size
-from ..util import parse_date, base_url, hashurl
-from ..index.schema import Link
-from ..index.html import snapshot_icons
-from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE
-
-EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()]
-STATUS_CHOICES = [
-    ("succeeded", "succeeded"),
-    ("failed", "failed"),
-    ("skipped", "skipped")
-]
-
-try:
-    JSONField = models.JSONField
-except AttributeError:
-    import jsonfield
-    JSONField = jsonfield.JSONField
-
-
-class Tag(models.Model):
-    """
-    Based on django-taggit model
-    """
-    id = models.AutoField(primary_key=True, serialize=False, verbose_name='ID')
-
-    name = models.CharField(unique=True, blank=False, max_length=100)
-
-    # slug is autoset on save from name, never set it manually
-    slug = models.SlugField(unique=True, blank=True, max_length=100)
-
-
-    class Meta:
-        verbose_name = "Tag"
-        verbose_name_plural = "Tags"
-
-    def __str__(self):
-        return self.name
-
-    def slugify(self, tag, i=None):
-        slug = slugify(tag)
-        if i is not None:
-            slug += "_%d" % i
-        return slug
-
-    def save(self, *args, **kwargs):
-        if self._state.adding and not self.slug:
-            self.slug = self.slugify(self.name)
-
-            # if name is different but slug conficts with another tags slug, append a counter
-            # with transaction.atomic():
-            slugs = set(
-                type(self)
-                ._default_manager.filter(slug__startswith=self.slug)
-                .values_list("slug", flat=True)
-            )
-
-            i = None
-            while True:
-                slug = self.slugify(self.name, i)
-                if slug not in slugs:
-                    self.slug = slug
-                    return super().save(*args, **kwargs)
-                i = 1 if i is None else i+1
-        else:
-            return super().save(*args, **kwargs)
-
-
-class Snapshot(models.Model):
-    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
-
-    url = models.URLField(unique=True, db_index=True)
-    timestamp = models.CharField(max_length=32, unique=True, db_index=True)
-
-    title = models.CharField(max_length=512, null=True, blank=True, db_index=True)
-
-    added = models.DateTimeField(auto_now_add=True, db_index=True)
-    updated = models.DateTimeField(auto_now=True, blank=True, null=True, db_index=True)
-    tags = models.ManyToManyField(Tag, blank=True)
-
-    keys = ('url', 'timestamp', 'title', 'tags', 'updated')
-
-    def __repr__(self) -> str:
-        title = self.title or '-'
-        return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})'
-
-    def __str__(self) -> str:
-        title = self.title or '-'
-        return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})'
-
-    @classmethod
-    def from_json(cls, info: dict):
-        info = {k: v for k, v in info.items() if k in cls.keys}
-        return cls(**info)
-
-    def as_json(self, *args) -> dict:
-        args = args or self.keys
-        return {
-            key: getattr(self, key)
-            if key != 'tags' else self.tags_str()
-            for key in args
-        }
-
-    def as_link(self) -> Link:
-        return Link.from_json(self.as_json())
-
-    def as_link_with_details(self) -> Link:
-        from ..index import load_link_details
-        return load_link_details(self.as_link())
-
-    def tags_str(self, nocache=True) -> str:
-        cache_key = f'{self.id}-{(self.updated or self.added).timestamp()}-tags'
-        calc_tags_str = lambda: ','.join(self.tags.order_by('name').values_list('name', flat=True))
-        if nocache:
-            tags_str = calc_tags_str()
-            cache.set(cache_key, tags_str)
-            return tags_str
-        return cache.get_or_set(cache_key, calc_tags_str)
-
-    def icons(self) -> str:
-        return snapshot_icons(self)
-
-    @cached_property
-    def extension(self) -> str:
-        from ..util import extension
-        return extension(self.url)
-
-    @cached_property
-    def bookmarked(self):
-        return parse_date(self.timestamp)
-
-    @cached_property
-    def bookmarked_date(self):
-        # TODO: remove this
-        return self.bookmarked
-
-    @cached_property
-    def is_archived(self):
-        return self.as_link().is_archived
-
-    @cached_property
-    def num_outputs(self):
-        return self.archiveresult_set.filter(status='succeeded').count()
-
-    @cached_property
-    def url_hash(self):
-        return hashurl(self.url)
-
-    @cached_property
-    def base_url(self):
-        return base_url(self.url)
-
-    @cached_property
-    def link_dir(self):
-        return str(ARCHIVE_DIR / self.timestamp)
-
-    @cached_property
-    def archive_path(self):
-        return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp)
-
-    @cached_property
-    def archive_size(self):
-        cache_key = f'{str(self.id)[:12]}-{(self.updated or self.added).timestamp()}-size'
-
-        def calc_dir_size():
-            try:
-                return get_dir_size(self.link_dir)[0]
-            except Exception:
-                return 0
-
-        return cache.get_or_set(cache_key, calc_dir_size)
-
-    @cached_property
-    def thumbnail_url(self) -> Optional[str]:
-        result = self.archiveresult_set.filter(
-            extractor='screenshot',
-            status='succeeded'
-        ).only('output').last()
-        if result:
-            return reverse('Snapshot', args=[f'{str(self.timestamp)}/{result.output}'])
-        return None
-
-    @cached_property
-    def headers(self) -> Optional[dict]:
-        try:
-            return json.loads((Path(self.link_dir) / 'headers.json').read_text(encoding='utf-8').strip())
-        except Exception:
-            pass
-        return None
-
-    @cached_property
-    def status_code(self) -> Optional[str]:
-        return self.headers and self.headers.get('Status-Code')
-
-    @cached_property
-    def history(self) -> dict:
-        # TODO: use ArchiveResult for this instead of json
-        return self.as_link_with_details().history
-
-    @cached_property
-    def latest_title(self) -> Optional[str]:
-        if self.title:
-            return self.title   # whoopdedoo that was easy
-        
-        try:
-            # take longest successful title from ArchiveResult db history
-            return sorted(
-                self.archiveresult_set\
-                    .filter(extractor='title', status='succeeded', output__isnull=False)\
-                    .values_list('output', flat=True),
-                key=lambda r: len(r),
-            )[-1]
-        except IndexError:
-            pass
-
-        try:
-            # take longest successful title from Link json index file history
-            return sorted(
-                (
-                    result.output.strip()
-                    for result in self.history['title']
-                    if result.status == 'succeeded' and result.output.strip()
-                ),
-                key=lambda r: len(r),
-            )[-1]
-        except (KeyError, IndexError):
-            pass
-
-        return None
-
-    def save_tags(self, tags: List[str]=()) -> None:
-        tags_id = []
-        for tag in tags:
-            if tag.strip():
-                tags_id.append(Tag.objects.get_or_create(name=tag)[0].id)
-        self.tags.clear()
-        self.tags.add(*tags_id)
-
-
-class ArchiveResultManager(models.Manager):
-    def indexable(self, sorted: bool = True):
-        INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
-        qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded')
-
-        if sorted:
-            precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
-            qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence')
-        return qs
-
-
-class ArchiveResult(models.Model):
-    id = models.AutoField(primary_key=True, serialize=False, verbose_name='ID')
-    uuid = models.UUIDField(default=uuid.uuid4, editable=False)
-
-    snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE)
-    extractor = models.CharField(choices=EXTRACTORS, max_length=32)
-    cmd = JSONField()
-    pwd = models.CharField(max_length=256)
-    cmd_version = models.CharField(max_length=128, default=None, null=True, blank=True)
-    output = models.CharField(max_length=1024)
-    start_ts = models.DateTimeField(db_index=True)
-    end_ts = models.DateTimeField()
-    status = models.CharField(max_length=16, choices=STATUS_CHOICES)
-
-    objects = ArchiveResultManager()
-
-    def __str__(self):
-        return self.extractor

archivebox/core/settings.py

@@ -1,373 +0,0 @@
-__package__ = 'archivebox.core'
-
-import os
-import sys
-import re
-import logging
-import tempfile
-
-from pathlib import Path
-from django.utils.crypto import get_random_string
-
-from ..config import (
-    DEBUG,
-    SECRET_KEY,
-    ALLOWED_HOSTS,
-    PACKAGE_DIR,
-    TEMPLATES_DIR_NAME,
-    CUSTOM_TEMPLATES_DIR,
-    SQL_INDEX_FILENAME,
-    OUTPUT_DIR,
-    LOGS_DIR,
-    TIMEZONE,
-
-    LDAP,
-    LDAP_SERVER_URI,
-    LDAP_BIND_DN,
-    LDAP_BIND_PASSWORD,
-    LDAP_USER_BASE,
-    LDAP_USER_FILTER,
-    LDAP_USERNAME_ATTR,
-    LDAP_FIRSTNAME_ATTR,
-    LDAP_LASTNAME_ATTR,
-    LDAP_EMAIL_ATTR,
-)
-
-IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3]
-IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ
-IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3]
-
-################################################################################
-### Django Core Settings
-################################################################################
-
-WSGI_APPLICATION = 'core.wsgi.application'
-ROOT_URLCONF = 'core.urls'
-
-LOGIN_URL = '/accounts/login/'
-LOGOUT_REDIRECT_URL = os.environ.get('LOGOUT_REDIRECT_URL', '/')
-
-PASSWORD_RESET_URL = '/accounts/password_reset/'
-APPEND_SLASH = True
-
-DEBUG = DEBUG or ('--debug' in sys.argv)
-
-INSTALLED_APPS = [
-    'django.contrib.auth',
-    'django.contrib.contenttypes',
-    'django.contrib.sessions',
-    'django.contrib.messages',
-    'django.contrib.staticfiles',
-    'django.contrib.admin',
-
-    'core',
-
-    'django_extensions',
-]
-
-
-# For usage with https://www.jetadmin.io/integrations/django
-# INSTALLED_APPS += ['jet_django']
-# JET_PROJECT = 'archivebox'
-# JET_TOKEN = 'some-api-token-here'
-
-
-MIDDLEWARE = [
-    'core.middleware.TimezoneMiddleware',
-    'django.middleware.security.SecurityMiddleware',
-    'django.contrib.sessions.middleware.SessionMiddleware',
-    'django.middleware.common.CommonMiddleware',
-    'django.middleware.csrf.CsrfViewMiddleware',
-    'django.contrib.auth.middleware.AuthenticationMiddleware',
-    'core.middleware.ReverseProxyAuthMiddleware',
-    'django.contrib.messages.middleware.MessageMiddleware',
-    'core.middleware.CacheControlMiddleware',
-]
-
-################################################################################
-### Authentication Settings
-################################################################################
-
-AUTHENTICATION_BACKENDS = [
-    'django.contrib.auth.backends.RemoteUserBackend',
-    'django.contrib.auth.backends.ModelBackend',
-]
-
-if LDAP:
-    try:
-        import ldap
-        from django_auth_ldap.config import LDAPSearch
-
-        global AUTH_LDAP_SERVER_URI
-        global AUTH_LDAP_BIND_DN
-        global AUTH_LDAP_BIND_PASSWORD
-        global AUTH_LDAP_USER_SEARCH
-        global AUTH_LDAP_USER_ATTR_MAP
-
-        AUTH_LDAP_SERVER_URI = LDAP_SERVER_URI
-        AUTH_LDAP_BIND_DN = LDAP_BIND_DN
-        AUTH_LDAP_BIND_PASSWORD = LDAP_BIND_PASSWORD
-
-        assert AUTH_LDAP_SERVER_URI and LDAP_USERNAME_ATTR and LDAP_USER_FILTER, 'LDAP_* config options must all be set if LDAP=True'
-
-        AUTH_LDAP_USER_SEARCH = LDAPSearch(
-            LDAP_USER_BASE,
-            ldap.SCOPE_SUBTREE,
-            '(&(' + LDAP_USERNAME_ATTR + '=%(user)s)' + LDAP_USER_FILTER + ')',
-        )
-
-        AUTH_LDAP_USER_ATTR_MAP = {
-            'username': LDAP_USERNAME_ATTR,
-            'first_name': LDAP_FIRSTNAME_ATTR,
-            'last_name': LDAP_LASTNAME_ATTR,
-            'email': LDAP_EMAIL_ATTR,
-        }
-
-        AUTHENTICATION_BACKENDS = [
-            'django.contrib.auth.backends.ModelBackend',
-            'django_auth_ldap.backend.LDAPBackend',
-        ]
-    except ModuleNotFoundError:
-        sys.stderr.write('[X] Error: Found LDAP=True config but LDAP packages not installed. You may need to run: pip install archivebox[ldap]\n\n')
-        # dont hard exit here. in case the user is just running "archivebox version" or "archivebox help", we still want those to work despite broken ldap
-        # sys.exit(1)
-
-
-################################################################################
-### Debug Settings
-################################################################################
-
-# only enable debug toolbar when in DEBUG mode with --nothreading (it doesnt work in multithreaded mode)
-DEBUG_TOOLBAR = DEBUG and ('--nothreading' in sys.argv) and ('--reload' not in sys.argv)
-if DEBUG_TOOLBAR:
-    try:
-        import debug_toolbar   # noqa
-        DEBUG_TOOLBAR = True
-    except ImportError:
-        DEBUG_TOOLBAR = False
-
-if DEBUG_TOOLBAR:
-    INSTALLED_APPS = [*INSTALLED_APPS, 'debug_toolbar']
-    INTERNAL_IPS = ['0.0.0.0', '127.0.0.1', '*']
-    DEBUG_TOOLBAR_CONFIG = {
-        "SHOW_TOOLBAR_CALLBACK": lambda request: True,
-        "RENDER_PANELS": True,
-    }
-    DEBUG_TOOLBAR_PANELS = [
-        'debug_toolbar.panels.history.HistoryPanel',
-        'debug_toolbar.panels.versions.VersionsPanel',
-        'debug_toolbar.panels.timer.TimerPanel',
-        'debug_toolbar.panels.settings.SettingsPanel',
-        'debug_toolbar.panels.headers.HeadersPanel',
-        'debug_toolbar.panels.request.RequestPanel',
-        'debug_toolbar.panels.sql.SQLPanel',
-        'debug_toolbar.panels.staticfiles.StaticFilesPanel',
-        # 'debug_toolbar.panels.templates.TemplatesPanel',
-        'debug_toolbar.panels.cache.CachePanel',
-        'debug_toolbar.panels.signals.SignalsPanel',
-        'debug_toolbar.panels.logging.LoggingPanel',
-        'debug_toolbar.panels.redirects.RedirectsPanel',
-        'debug_toolbar.panels.profiling.ProfilingPanel',
-        'djdt_flamegraph.FlamegraphPanel',
-    ]
-    MIDDLEWARE = [*MIDDLEWARE, 'debug_toolbar.middleware.DebugToolbarMiddleware']
-
-################################################################################
-### Staticfile and Template Settings
-################################################################################
-
-STATIC_URL = '/static/'
-
-STATICFILES_DIRS = [
-    *([str(CUSTOM_TEMPLATES_DIR / 'static')] if CUSTOM_TEMPLATES_DIR else []),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'static'),
-]
-
-TEMPLATE_DIRS = [
-    *([str(CUSTOM_TEMPLATES_DIR)] if CUSTOM_TEMPLATES_DIR else []),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'core'),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'admin'),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME),
-]
-
-TEMPLATES = [
-    {
-        'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': TEMPLATE_DIRS,
-        'APP_DIRS': True,
-        'OPTIONS': {
-            'context_processors': [
-                'django.template.context_processors.debug',
-                'django.template.context_processors.request',
-                'django.contrib.auth.context_processors.auth',
-                'django.contrib.messages.context_processors.messages',
-            ],
-        },
-    },
-]
-
-
-################################################################################
-### External Service Settings
-################################################################################
-
-DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME
-DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", str(DATABASE_FILE))
-
-DATABASES = {
-    'default': {
-        'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': DATABASE_NAME,
-        'OPTIONS': {
-            'timeout': 60,
-            'check_same_thread': False,
-        },
-        'TIME_ZONE': TIMEZONE,
-        # DB setup is sometimes modified at runtime by setup_django() in config.py
-    }
-}
-
-CACHE_BACKEND = 'django.core.cache.backends.locmem.LocMemCache'
-# CACHE_BACKEND = 'django.core.cache.backends.db.DatabaseCache'
-# CACHE_BACKEND = 'django.core.cache.backends.dummy.DummyCache'
-
-CACHES = {
-    'default': {
-        'BACKEND': CACHE_BACKEND,
-        'LOCATION': 'django_cache_default',
-    }
-}
-
-EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
-
-
-################################################################################
-### Security Settings
-################################################################################
-
-SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789_')
-
-ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
-
-SECURE_BROWSER_XSS_FILTER = True
-SECURE_CONTENT_TYPE_NOSNIFF = True
-SECURE_REFERRER_POLICY = 'strict-origin-when-cross-origin'
-
-CSRF_COOKIE_SECURE = False
-SESSION_COOKIE_SECURE = False
-SESSION_COOKIE_DOMAIN = None
-SESSION_COOKIE_AGE = 1209600  # 2 weeks
-SESSION_EXPIRE_AT_BROWSER_CLOSE = False
-SESSION_SAVE_EVERY_REQUEST = True
-
-SESSION_ENGINE = "django.contrib.sessions.backends.db"
-
-AUTH_PASSWORD_VALIDATORS = [
-    {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'},
-]
-
-# WIP: broken by Django 3.1.2 -> 4.0 migration
-DEFAULT_AUTO_FIELD = 'django.db.models.UUIDField'
-
-################################################################################
-### Shell Settings
-################################################################################
-
-SHELL_PLUS = 'ipython'
-SHELL_PLUS_PRINT_SQL = False
-IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner']
-IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell'
-if IS_SHELL:
-    os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py')
-
-
-################################################################################
-### Internationalization & Localization Settings
-################################################################################
-
-LANGUAGE_CODE = 'en-us'
-USE_I18N = True
-USE_L10N = True
-USE_TZ = True
-DATETIME_FORMAT = 'Y-m-d g:iA'
-SHORT_DATETIME_FORMAT = 'Y-m-d h:iA'
-TIME_ZONE = TIMEZONE        # django convention is TIME_ZONE, archivebox config uses TIMEZONE, they are equivalent
-
-
-from django.conf.locale.en import formats as en_formats
-
-en_formats.DATETIME_FORMAT = DATETIME_FORMAT
-en_formats.SHORT_DATETIME_FORMAT = SHORT_DATETIME_FORMAT
-
-
-################################################################################
-### Logging Settings
-################################################################################
-
-IGNORABLE_404_URLS = [
-    re.compile(r'apple-touch-icon.*\.png$'),
-    re.compile(r'favicon\.ico$'),
-    re.compile(r'robots\.txt$'),
-    re.compile(r'.*\.(css|js)\.map$'),
-]
-
-class NoisyRequestsFilter(logging.Filter):
-    def filter(self, record):
-        logline = record.getMessage()
-
-        # ignore harmless 404s for the patterns in IGNORABLE_404_URLS
-        for ignorable_url_pattern in IGNORABLE_404_URLS:
-            ignorable_log_pattern = re.compile(f'^"GET /.*/?{ignorable_url_pattern.pattern[:-1]} HTTP/.*" (200|30.|404) .+$', re.I | re.M)
-            if ignorable_log_pattern.match(logline):
-                return 0
-
-        # ignore staticfile requests that 200 or 30*
-        ignoreable_200_log_pattern = re.compile(r'"GET /static/.* HTTP/.*" (200|30.) .+', re.I | re.M)
-        if ignoreable_200_log_pattern.match(logline):
-            return 0
-
-        return 1
-
-if LOGS_DIR.exists():
-    ERROR_LOG = (LOGS_DIR / 'errors.log')
-else:
-    # historically too many edge cases here around creating log dir w/ correct permissions early on
-    # if there's an issue on startup, we trash the log and let user figure it out via stdout/stderr
-    ERROR_LOG = tempfile.NamedTemporaryFile().name
-
-LOGGING = {
-    'version': 1,
-    'disable_existing_loggers': False,
-    'handlers': {
-        'console': {
-            'class': 'logging.StreamHandler',
-        },
-        'logfile': {
-            'level': 'ERROR',
-            'class': 'logging.handlers.RotatingFileHandler',
-            'filename': ERROR_LOG,
-            'maxBytes': 1024 * 1024 * 25,  # 25 MB
-            'backupCount': 10,
-        },
-    },
-    'filters': {
-        'noisyrequestsfilter': {
-            '()': NoisyRequestsFilter,
-        }
-    },
-    'loggers': {
-        'django': {
-            'handlers': ['console', 'logfile'],
-            'level': 'INFO',
-            'filters': ['noisyrequestsfilter'],
-        },
-        'django.server': {
-            'handlers': ['console', 'logfile'],
-            'level': 'INFO',
-            'filters': ['noisyrequestsfilter'],
-        }
-    },
-}

archivebox/core/templatetags/core_tags.py

@@ -1,40 +0,0 @@
-from django import template
-from django.contrib.admin.templatetags.base import InclusionAdminNode
-
-
-from typing import Union
-
-
-register = template.Library()
-
-@register.filter(name='split')
-def split(value, separator: str=','):
-    return (value or '').split(separator)
-
-@register.filter
-def file_size(num_bytes: Union[int, float]) -> str:
-    for count in ['Bytes','KB','MB','GB']:
-        if num_bytes > -1024.0 and num_bytes < 1024.0:
-            return '%3.1f %s' % (num_bytes, count)
-        num_bytes /= 1024.0
-    return '%3.1f %s' % (num_bytes, 'TB')
-
-def result_list(cl):
-    """
-    Monkey patched result
-    """
-    num_sorted_fields = 0
-    return {
-        'cl': cl,
-        'num_sorted_fields': num_sorted_fields,
-        'results': cl.result_list,
-    }
-
-@register.tag(name='snapshots_grid')
-def result_list_tag(parser, token):
-    return InclusionAdminNode(
-        parser, token,
-        func=result_list,
-        template_name='snapshots_grid.html',
-        takes_context=False,
-    )

archivebox/core/tests.py

@@ -1,3 +0,0 @@
-#from django.test import TestCase
-
-# Create your tests here.

archivebox/core/urls.py

@@ -1,90 +0,0 @@
-from django.contrib import admin
-
-from django.urls import path, include
-from django.views import static
-from django.contrib.staticfiles.urls import staticfiles_urlpatterns
-from django.conf import settings
-from django.views.generic.base import RedirectView
-
-from core.views import HomepageView, SnapshotView, PublicIndexView, AddView, HealthCheckView
-
-# GLOBAL_CONTEXT doesn't work as-is, disabled for now: https://github.com/ArchiveBox/ArchiveBox/discussions/1306
-# from config import VERSION, VERSIONS_AVAILABLE, CAN_UPGRADE
-# GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': VERSIONS_AVAILABLE, 'CAN_UPGRADE': CAN_UPGRADE}
-
-
-# print('DEBUG', settings.DEBUG)
-
-urlpatterns = [
-    path('public/', PublicIndexView.as_view(), name='public-index'),
-
-    path('robots.txt', static.serve, {'document_root': settings.STATICFILES_DIRS[0], 'path': 'robots.txt'}),
-    path('favicon.ico', static.serve, {'document_root': settings.STATICFILES_DIRS[0], 'path': 'favicon.ico'}),
-
-    path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'),
-
-    path('archive/', RedirectView.as_view(url='/')),
-    path('archive/<path:path>', SnapshotView.as_view(), name='Snapshot'),
-
-    path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')),
-    path('add/', AddView.as_view(), name='add'),
-
-    path('accounts/login/', RedirectView.as_view(url='/admin/login/')),
-    path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')),
-
-
-    path('accounts/', include('django.contrib.auth.urls')),
-    path('admin/', admin.site.urls),
-    
-    # do not add extra_context like this as not all admin views (e.g. ModelAdmin.autocomplete_view accept extra kwargs)
-    # path('admin/', admin.site.urls, {'extra_context': GLOBAL_CONTEXT}),
-
-    path('health/', HealthCheckView.as_view(), name='healthcheck'),
-    path('error/', lambda _: 1/0),
-
-    # path('jet_api/', include('jet_django.urls')),  Enable to use https://www.jetadmin.io/integrations/django
-
-    path('index.html', RedirectView.as_view(url='/')),
-    path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}),
-    path('', HomepageView.as_view(), name='Home'),
-]
-urlpatterns += staticfiles_urlpatterns()
-
-if settings.DEBUG_TOOLBAR:
-    import debug_toolbar
-    urlpatterns += [
-        path('__debug__/', include(debug_toolbar.urls)),
-    ]
-
-
-# # Proposed FUTURE URLs spec
-# path('',                 HomepageView)
-# path('/add',             AddView)
-# path('/public',          PublicIndexView)
-# path('/snapshot/:slug',  SnapshotView)
-
-# path('/admin',           admin.site.urls)
-# path('/accounts',        django.contrib.auth.urls)
-
-# # Prposed REST API spec
-# # :slugs can be uuid, short_uuid, or any of the unique index_fields
-# path('api/v1/'),
-# path('api/v1/core/'                      [GET])
-# path('api/v1/core/snapshot/',            [GET, POST, PUT]),
-# path('api/v1/core/snapshot/:slug',       [GET, PATCH, DELETE]),
-# path('api/v1/core/archiveresult',        [GET, POST, PUT]),
-# path('api/v1/core/archiveresult/:slug',  [GET, PATCH, DELETE]),
-# path('api/v1/core/tag/',                 [GET, POST, PUT]),
-# path('api/v1/core/tag/:slug',            [GET, PATCH, DELETE]),
-
-# path('api/v1/cli/',                      [GET])
-# path('api/v1/cli/{add,list,config,...}', [POST]),  # pass query as kwargs directly to `run_subcommand` and return stdout, stderr, exitcode
-
-# path('api/v1/extractors/',                    [GET])
-# path('api/v1/extractors/:extractor/',         [GET]),
-# path('api/v1/extractors/:extractor/:func',    [GET, POST]),  # pass query as args directly to chosen function
-
-# future, just an idea:
-# path('api/v1/scheduler/',                [GET])
-# path('api/v1/scheduler/task/',           [GET, POST, PUT]),
-# path('api/v1/scheduler/task/:slug',      [GET, PATCH, DELETE]),

archivebox/core/views.py

@@ -1,314 +0,0 @@
-__package__ = 'archivebox.core'
-
-from io import StringIO
-from contextlib import redirect_stdout
-
-from django.shortcuts import render, redirect
-from django.http import HttpResponse, Http404
-from django.utils.html import format_html, mark_safe
-from django.views import View, static
-from django.views.generic.list import ListView
-from django.views.generic import FormView
-from django.db.models import Q
-from django.contrib.auth.mixins import UserPassesTestMixin
-from django.views.decorators.csrf import csrf_exempt
-from django.utils.decorators import method_decorator
-
-from core.models import Snapshot
-from core.forms import AddLinkForm
-
-from ..config import (
-    OUTPUT_DIR,
-    PUBLIC_INDEX,
-    PUBLIC_SNAPSHOTS,
-    PUBLIC_ADD_VIEW,
-    VERSION,
-    COMMIT_HASH,
-    FOOTER_INFO,
-    SNAPSHOTS_PER_PAGE,
-)
-from ..main import add
-from ..util import base_url, ansi_to_html
-from ..search import query_search_index
-
-
-class HomepageView(View):
-    def get(self, request):
-        if request.user.is_authenticated:
-            return redirect('/admin/core/snapshot/')
-
-        if PUBLIC_INDEX:
-            return redirect('/public')
-
-        return redirect(f'/admin/login/?next={request.path}')
-
-
-class SnapshotView(View):
-    # render static html index from filesystem archive/<timestamp>/index.html
-
-    def get(self, request, path):
-        if not request.user.is_authenticated and not PUBLIC_SNAPSHOTS:
-            return redirect(f'/admin/login/?next={request.path}')
-
-        try:
-            slug, archivefile = path.split('/', 1)
-        except (IndexError, ValueError):
-            slug, archivefile = path.split('/', 1)[0], 'index.html'
-
-        # slug is a timestamp
-        if slug.replace('.','').isdigit():
-
-            # missing trailing slash -> redirect to index
-            if '/' not in path:
-                return redirect(f'{path}/index.html')
-
-            try:
-                try:
-                    snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
-                    response = static.serve(request, archivefile, document_root=snapshot.link_dir, show_indexes=True)
-                    response["Link"] = f'<{snapshot.url}>; rel="canonical"'
-                    return response
-                except Snapshot.DoesNotExist:
-                    if Snapshot.objects.filter(timestamp__startswith=slug).exists():
-                        raise Snapshot.MultipleObjectsReturned
-                    else:
-                        raise
-            except Snapshot.DoesNotExist:
-                # Snapshot does not exist
-                return HttpResponse(
-                    format_html(
-                        (
-                            '<center><br/><br/><br/>'
-                            'No Snapshot directories match the given timestamp or UUID: <code>{}</code><br/><br/>'
-                            'You can <a href="/add/" target="_top">add a new Snapshot</a>, or return to the <a href="/" target="_top">Main Index</a>'
-                            '</center>'
-                        ),
-                        slug,
-                        path,
-                    ),
-                    content_type="text/html",
-                    status=404,
-                )
-            except Snapshot.MultipleObjectsReturned:
-                snapshot_hrefs = mark_safe('<br/>').join(
-                    format_html(
-                        '{} <a href="/archive/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
-                        snap.added.strftime('%Y-%m-%d %H:%M:%S'),
-                        snap.timestamp,
-                        snap.timestamp,
-                        snap.url,
-                        snap.title or '',
-                    )
-                    for snap in Snapshot.objects.filter(timestamp__startswith=slug).only('url', 'timestamp', 'title', 'added').order_by('-added')
-                )
-                return HttpResponse(
-                    format_html(
-                        (
-                            'Multiple Snapshots match the given timestamp/UUID <code>{}</code><br/><pre>'
-                        ),
-                        slug,
-                    ) + snapshot_hrefs + format_html(
-                        (
-                            '</pre><br/>'
-                            'Choose a Snapshot to proceed or go back to the <a href="/" target="_top">Main Index</a>'
-                        )
-                    ),
-                    content_type="text/html",
-                    status=404,
-                )
-            except Http404:
-                # Snapshot dir exists but file within does not e.g. 124235.324234/screenshot.png
-                return HttpResponse(
-                    format_html(
-                        (
-                            '<center><br/><br/><br/>'
-                            f'Snapshot <a href="/archive/{snapshot.timestamp}/index.html" target="_top"><b><code>[{snapshot.timestamp}]</code></b></a> exists in DB, but resource <b><code>{snapshot.timestamp}/'
-                            '{}'
-                            f'</code></b> does not exist in <a href="/archive/{snapshot.timestamp}/" target="_top">snapshot dir</a> yet.<br/><br/>'
-                            'Maybe this resource type is not availabe for this Snapshot,<br/>or the archiving process has not completed yet?<br/>'
-                            f'<pre><code># run this cmd to finish archiving this Snapshot<br/>archivebox update -t timestamp {snapshot.timestamp}</code></pre><br/><br/>'
-                            '<div class="text-align: left; width: 100%; max-width: 400px">'
-                            '<i><b>Next steps:</i></b><br/>'
-                            f'- list all the <a href="/archive/{snapshot.timestamp}/" target="_top">Snapshot files <code>.*</code></a><br/>'
-                            f'- view the <a href="/archive/{snapshot.timestamp}/index.html" target="_top">Snapshot <code>./index.html</code></a><br/>'
-                            f'- go to the <a href="/admin/core/snapshot/{snapshot.id}/change/" target="_top">Snapshot admin</a> to edit<br/>'
-                            f'- go to the <a href="/admin/core/snapshot/?id__startswith={snapshot.id}" target="_top">Snapshot actions</a> to re-archive<br/>'
-                            '- or return to <a href="/" target="_top">the main index...</a></div>'
-                            '</center>'
-                        ),
-                        archivefile,
-                    ),
-                    content_type="text/html",
-                    status=404,
-                )
-        # slug is a URL
-        try:
-            try:
-                # try exact match on full url first
-                snapshot = Snapshot.objects.get(
-                    Q(url='http://' + path) | Q(url='https://' + path) | Q(id__startswith=path)
-                )
-            except Snapshot.DoesNotExist:
-                # fall back to match on exact base_url
-                try:
-                    snapshot = Snapshot.objects.get(
-                        Q(url='http://' + base_url(path)) | Q(url='https://' + base_url(path))
-                    )
-                except Snapshot.DoesNotExist:
-                    # fall back to matching base_url as prefix
-                    snapshot = Snapshot.objects.get(
-                        Q(url__startswith='http://' + base_url(path)) | Q(url__startswith='https://' + base_url(path))
-                    )
-            return redirect(f'/archive/{snapshot.timestamp}/index.html')
-        except Snapshot.DoesNotExist:
-            return HttpResponse(
-                format_html(
-                    (
-                        '<center><br/><br/><br/>'
-                        'No Snapshots match the given url: <code>{}</code><br/><br/><br/>'
-                        'Return to the <a href="/" target="_top">Main Index</a>, or:<br/><br/>'
-                        '+ <i><a href="/add/?url={}" target="_top">Add a new Snapshot for <code>{}</code></a><br/><br/></i>'
-                        '</center>'
-                    ),
-                    base_url(path),
-                    path if '://' in path else f'https://{path}',
-                    path,
-                ),
-                content_type="text/html",
-                status=404,
-            )
-        except Snapshot.MultipleObjectsReturned:
-            snapshot_hrefs = mark_safe('<br/>').join(
-                format_html(
-                    '{} <a href="/archive/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
-                    snap.added.strftime('%Y-%m-%d %H:%M:%S'),
-                    snap.timestamp,
-                    snap.timestamp,
-                    snap.url,
-                    snap.title or '',
-                )
-                for snap in Snapshot.objects.filter(
-                    Q(url__startswith='http://' + base_url(path)) | Q(url__startswith='https://' + base_url(path))
-                ).only('url', 'timestamp', 'title', 'added').order_by('-added')
-            )
-            return HttpResponse(
-                format_html(
-                    (
-                        'Multiple Snapshots match the given URL <code>{}</code><br/><pre>'
-                    ),
-                    base_url(path),
-                ) + snapshot_hrefs + format_html(
-                    (
-                        '</pre><br/>'
-                        'Choose a Snapshot to proceed or go back to the <a href="/" target="_top">Main Index</a>'
-                    )
-                ),
-                content_type="text/html",
-                status=404,
-            )
-
-
-class PublicIndexView(ListView):
-    template_name = 'public_index.html'
-    model = Snapshot
-    paginate_by = SNAPSHOTS_PER_PAGE
-    ordering = ['-added']
-
-    def get_context_data(self, **kwargs):
-        return {
-            **super().get_context_data(**kwargs),
-            'VERSION': VERSION,
-            'COMMIT_HASH': COMMIT_HASH,
-            'FOOTER_INFO': FOOTER_INFO,
-        }
-
-    def get_queryset(self, **kwargs):
-        qs = super().get_queryset(**kwargs)
-        query = self.request.GET.get('q')
-        if query and query.strip():
-            qs = qs.filter(Q(title__icontains=query) | Q(url__icontains=query) | Q(timestamp__icontains=query) | Q(tags__name__icontains=query))
-            try:
-                qs = qs | query_search_index(query)
-            except Exception as err:
-                print(f'[!] Error while using search backend: {err.__class__.__name__} {err}')
-        return qs
-
-    def get(self, *args, **kwargs):
-        if PUBLIC_INDEX or self.request.user.is_authenticated:
-            response = super().get(*args, **kwargs)
-            return response
-        else:
-            return redirect(f'/admin/login/?next={self.request.path}')
-
-@method_decorator(csrf_exempt, name='dispatch')
-class AddView(UserPassesTestMixin, FormView):
-    template_name = "add.html"
-    form_class = AddLinkForm
-
-    def get_initial(self):
-        """Prefill the AddLinkForm with the 'url' GET parameter"""
-        if self.request.method == 'GET':
-            url = self.request.GET.get('url', None)
-            if url:
-                return {'url': url if '://' in url else f'https://{url}'}
-
-        return super().get_initial()
-
-    def test_func(self):
-        return PUBLIC_ADD_VIEW or self.request.user.is_authenticated
-
-    def get_context_data(self, **kwargs):
-        return {
-            **super().get_context_data(**kwargs),
-            'title': "Add URLs",
-            # We can't just call request.build_absolute_uri in the template, because it would include query parameters
-            'absolute_add_path': self.request.build_absolute_uri(self.request.path),
-            'VERSION': VERSION,
-            'FOOTER_INFO': FOOTER_INFO,
-            'stdout': '',
-        }
-
-    def form_valid(self, form):
-        url = form.cleaned_data["url"]
-        print(f'[+] Adding URL: {url}')
-        parser = form.cleaned_data["parser"]
-        tag = form.cleaned_data["tag"]
-        depth = 0 if form.cleaned_data["depth"] == "0" else 1
-        extractors = ','.join(form.cleaned_data["archive_methods"])
-        input_kwargs = {
-            "urls": url,
-            "tag": tag,
-            "depth": depth,
-            "parser": parser,
-            "update_all": False,
-            "out_dir": OUTPUT_DIR,
-        }
-        if extractors:
-            input_kwargs.update({"extractors": extractors})
-        add_stdout = StringIO()
-        with redirect_stdout(add_stdout):
-            add(**input_kwargs)
-            print(add_stdout.getvalue())
-
-        context = self.get_context_data()
-
-        context.update({
-            "stdout": ansi_to_html(add_stdout.getvalue().strip()),
-            "form": AddLinkForm()
-        })
-        return render(template_name=self.template_name, request=self.request, context=context)
-
-
-class HealthCheckView(View):
-    """
-    A Django view that renders plain text "OK" for service discovery tools
-    """
-    def get(self, request):
-        """
-        Handle a GET request
-        """
-        return HttpResponse(
-            'OK',
-            content_type='text/plain',
-            status=200
-        )

archivebox/core/welcome_message.py

@@ -1,5 +0,0 @@
-from archivebox.logging_util import log_shell_welcome_msg
-
-
-if __name__ == '__main__':
-    log_shell_welcome_msg()

archivebox/core/wsgi.py

@@ -1,16 +0,0 @@
-"""
-WSGI config for archivebox project.
-
-It exposes the WSGI callable as a module-level variable named ``application``.
-
-For more information on this file, see
-https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/
-"""
-
-
-from archivebox.config import setup_django
-setup_django(in_memory_db=False, check_db=True)
-
-from django.core.wsgi import get_wsgi_application
-
-application = get_wsgi_application()

archivebox/extractors/__init__.py

@@ -1,245 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-import os
-import sys
-from pathlib import Path
-
-from typing import Callable, Optional, List, Iterable, Union
-from datetime import datetime, timezone
-from django.db.models import QuerySet
-
-from ..config import (
-    SAVE_ALLOWLIST_PTN,
-    SAVE_DENYLIST_PTN,
-)
-from ..core.settings import ERROR_LOG
-from ..index.schema import ArchiveResult, Link
-from ..index.sql import write_link_to_sql_index
-from ..index import (
-    load_link_details,
-    write_link_details,
-)
-from ..util import enforce_types
-from ..logging_util import (
-    log_archiving_started,
-    log_archiving_paused,
-    log_archiving_finished,
-    log_link_archiving_started,
-    log_link_archiving_finished,
-    log_archive_method_started,
-    log_archive_method_finished,
-)
-from ..search import write_search_index
-
-from .title import should_save_title, save_title
-from .favicon import should_save_favicon, save_favicon
-from .wget import should_save_wget, save_wget
-from .singlefile import should_save_singlefile, save_singlefile
-from .readability import should_save_readability, save_readability
-from .mercury import should_save_mercury, save_mercury
-from .htmltotext import should_save_htmltotext, save_htmltotext
-from .pdf import should_save_pdf, save_pdf
-from .screenshot import should_save_screenshot, save_screenshot
-from .dom import should_save_dom, save_dom
-from .git import should_save_git, save_git
-from .media import should_save_media, save_media
-from .archive_org import should_save_archive_dot_org, save_archive_dot_org
-from .headers import should_save_headers, save_headers
-
-
-ShouldSaveFunction = Callable[[Link, Optional[Path], Optional[bool]], bool]
-SaveFunction = Callable[[Link, Optional[Path], int], ArchiveResult]
-ArchiveMethodEntry = tuple[str, ShouldSaveFunction, SaveFunction]
-
-def get_default_archive_methods() -> List[ArchiveMethodEntry]:
-    return [
-        ('favicon', should_save_favicon, save_favicon),
-        ('headers', should_save_headers, save_headers),
-        ('singlefile', should_save_singlefile, save_singlefile),
-        ('pdf', should_save_pdf, save_pdf),
-        ('screenshot', should_save_screenshot, save_screenshot),
-        ('dom', should_save_dom, save_dom),
-        ('wget', should_save_wget, save_wget),
-        # keep title, readability, and htmltotext below wget and singlefile, as they depend on them
-        ('title', should_save_title, save_title),
-        ('readability', should_save_readability, save_readability),
-        ('mercury', should_save_mercury, save_mercury),
-        ('htmltotext', should_save_htmltotext, save_htmltotext),
-        ('git', should_save_git, save_git),
-        ('media', should_save_media, save_media),
-        ('archive_org', should_save_archive_dot_org, save_archive_dot_org),
-    ]
-
-ARCHIVE_METHODS_INDEXING_PRECEDENCE = [
-    ('readability', 1),
-    ('mercury', 2),
-    ('htmltotext', 3),
-    ('singlefile', 4),
-    ('dom', 5),
-    ('wget', 6)
-]
-
-
-@enforce_types
-def get_archive_methods_for_link(link: Link) -> Iterable[ArchiveMethodEntry]:
-    DEFAULT_METHODS = get_default_archive_methods()
-    allowed_methods = {
-        m for pat, methods in
-        SAVE_ALLOWLIST_PTN.items()
-        if pat.search(link.url)
-        for m in methods
-    } or { m[0] for m in DEFAULT_METHODS }
-    denied_methods = {
-        m for pat, methods in
-        SAVE_DENYLIST_PTN.items()
-        if pat.search(link.url)
-        for m in methods
-    }
-    allowed_methods -= denied_methods
-
-    return (m for m in DEFAULT_METHODS if m[0] in allowed_methods)
-
-@enforce_types
-def ignore_methods(to_ignore: List[str]) -> Iterable[str]:
-    ARCHIVE_METHODS = get_default_archive_methods()
-    return [x[0] for x in ARCHIVE_METHODS if x[0] not in to_ignore]
-
-@enforce_types
-def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Link:
-    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
-
-    # TODO: Remove when the input is changed to be a snapshot. Suboptimal approach.
-    from core.models import Snapshot, ArchiveResult
-    try:
-        snapshot = Snapshot.objects.get(url=link.url) # TODO: This will be unnecessary once everything is a snapshot
-    except Snapshot.DoesNotExist:
-        snapshot = write_link_to_sql_index(link)
-
-    active_methods = get_archive_methods_for_link(link)
-    
-    if methods:
-        active_methods = [
-            method for method in active_methods
-            if method[0] in methods
-        ]
-
-    out_dir = out_dir or Path(link.link_dir)
-    try:
-        is_new = not Path(out_dir).exists()
-        if is_new:
-            os.makedirs(out_dir)
-
-        link = load_link_details(link, out_dir=out_dir)
-        write_link_details(link, out_dir=out_dir, skip_sql_index=False)
-        log_link_archiving_started(link, out_dir, is_new)
-        link = link.overwrite(updated=datetime.now(timezone.utc))
-        stats = {'skipped': 0, 'succeeded': 0, 'failed': 0}
-        start_ts = datetime.now(timezone.utc)
-
-        for method_name, should_run, method_function in active_methods:
-            try:
-                if method_name not in link.history:
-                    link.history[method_name] = []
-
-                if should_run(link, out_dir, overwrite):
-                    log_archive_method_started(method_name)
-
-                    result = method_function(link=link, out_dir=out_dir)
-
-                    link.history[method_name].append(result)
-
-                    stats[result.status] += 1
-                    log_archive_method_finished(result)
-                    write_search_index(link=link, texts=result.index_texts)
-                    ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version,
-                                                 output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status)
-
-
-                    # bump the updated time on the main Snapshot here, this is critical
-                    # to be able to cache summaries of the ArchiveResults for a given
-                    # snapshot without having to load all the results from the DB each time.
-                    # (we use {Snapshot.id}-{Snapshot.updated} as the cache key and assume
-                    # ArchiveResults are unchanged as long as the updated timestamp is unchanged)
-                    snapshot.save()
-                else:
-                    # print('{black}      X {}{reset}'.format(method_name, **ANSI))
-                    stats['skipped'] += 1
-            except Exception as e:
-                # Disabled until https://github.com/ArchiveBox/ArchiveBox/issues/984
-                # and https://github.com/ArchiveBox/ArchiveBox/issues/1014
-                # are fixed.
-                """
-                raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format(
-                    method_name,
-                    link.url,
-                )) from e
-                """
-                # Instead, use the kludgy workaround from
-                # https://github.com/ArchiveBox/ArchiveBox/issues/984#issuecomment-1150541627
-                with open(ERROR_LOG, "a", encoding='utf-8') as f:
-                    command = ' '.join(sys.argv)
-                    ts = datetime.now(timezone.utc).strftime('%Y-%m-%d__%H:%M:%S')
-                    f.write(("\n" + 'Exception in archive_methods.save_{}(Link(url={})) command={}; ts={}'.format(
-                        method_name,
-                        link.url,
-                        command,
-                        ts
-                    ) + "\n" + str(e) + "\n"))
-                    #f.write(f"\n> {command}; ts={ts} version={config['VERSION']} docker={config['IN_DOCKER']} is_tty={config['IS_TTY']}\n")
-
-        # print('    ', stats)
-
-        try:
-            latest_title = link.history['title'][-1].output.strip()
-            if latest_title and len(latest_title) >= len(link.title or ''):
-                link = link.overwrite(title=latest_title)
-        except Exception:
-            pass
-
-        write_link_details(link, out_dir=out_dir, skip_sql_index=False)
-
-        log_link_archiving_finished(link, out_dir, is_new, stats, start_ts)
-
-    except KeyboardInterrupt:
-        try:
-            write_link_details(link, out_dir=link.link_dir)
-        except:
-            pass
-        raise
-
-    except Exception as err:
-        print('    ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err))
-        raise
-
-    return link
-
-@enforce_types
-def archive_links(all_links: Union[Iterable[Link], QuerySet], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> List[Link]:
-
-    if type(all_links) is QuerySet:
-        num_links: int = all_links.count()
-        get_link = lambda x: x.as_link()
-        all_links = all_links.iterator()
-    else:
-        num_links: int = len(all_links)
-        get_link = lambda x: x
-
-    if num_links == 0:
-        return []
-
-    log_archiving_started(num_links)
-    idx: int = 0
-    try:
-        for link in all_links:
-            idx += 1
-            to_archive = get_link(link)
-            archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir))
-    except KeyboardInterrupt:
-        log_archiving_paused(num_links, idx, link.timestamp)
-        raise SystemExit(0)
-    except BaseException:
-        print()
-        raise
-
-    log_archiving_finished(num_links)
-    return all_links

archivebox/extractors/archive_org.py

@@ -1,112 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-
-from pathlib import Path
-from typing import Optional, List, Dict, Tuple
-from collections import defaultdict
-
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
-from ..system import run, chmod_file
-from ..util import (
-    enforce_types,
-    is_static_file,
-)
-from ..config import (
-    TIMEOUT,
-    CURL_ARGS,
-    CHECK_SSL_VALIDITY,
-    SAVE_ARCHIVE_DOT_ORG,
-    CURL_BINARY,
-    CURL_VERSION,
-    CURL_USER_AGENT,
-)
-from ..logging_util import TimedProgress
-
-
-
-@enforce_types
-def should_save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
-    if is_static_file(link.url):
-        return False
-
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'archive.org.txt').exists():
-        # if open(path, 'r', encoding='utf-8').read().strip() != 'None':
-        return False
-
-    return SAVE_ARCHIVE_DOT_ORG
-
-@enforce_types
-def save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """submit site to archive.org for archiving via their service, save returned archive url"""
-
-    out_dir = out_dir or Path(link.link_dir)
-    output: ArchiveOutput = 'archive.org.txt'
-    archive_org_url = None
-    submit_url = 'https://web.archive.org/save/{}'.format(link.url)
-    cmd = [
-        CURL_BINARY,
-        *CURL_ARGS,
-        '--head',
-        '--max-time', str(timeout),
-        *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
-        *([] if CHECK_SSL_VALIDITY else ['--insecure']),
-        submit_url,
-    ]
-    status = 'succeeded'
-    timer = TimedProgress(timeout, prefix='      ')
-    try:
-        result = run(cmd, cwd=str(out_dir), timeout=timeout)
-        content_location, errors = parse_archive_dot_org_response(result.stdout)
-        if content_location:
-            archive_org_url = content_location[0]
-        elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]:
-            archive_org_url = None
-            # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url)))
-        elif errors:
-            raise ArchiveError(', '.join(errors))
-        else:
-            raise ArchiveError('Failed to find "content-location" URL header in Archive.org response.')
-    except Exception as err:
-        status = 'failed'
-        output = err
-    finally:
-        timer.end()
-
-    if output and not isinstance(output, Exception):
-        # instead of writing None when archive.org rejects the url write the
-        # url to resubmit it to archive.org. This is so when the user visits
-        # the URL in person, it will attempt to re-archive it, and it'll show the
-        # nicer error message explaining why the url was rejected if it fails.
-        archive_org_url = archive_org_url or submit_url
-        with open(str(out_dir / output), 'w', encoding='utf-8') as f:
-            f.write(archive_org_url)
-        chmod_file('archive.org.txt', cwd=str(out_dir))
-        output = archive_org_url
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=CURL_VERSION,
-        output=output,
-        status=status,
-        **timer.stats,
-    )
-
-@enforce_types
-def parse_archive_dot_org_response(response: bytes) -> Tuple[List[str], List[str]]:
-    # Parse archive.org response headers
-    headers: Dict[str, List[str]] = defaultdict(list)
-
-    # lowercase all the header names and store in dict
-    for header in response.splitlines():
-        if b':' not in header or not header.strip():
-            continue
-        name, val = header.decode().split(':', 1)
-        headers[name.lower().strip()].append(val.strip())
-
-    # Get successful archive url in "content-location" header or any errors
-    content_location = headers.get('content-location', headers['location'])
-    errors = headers['x-archive-wayback-runtime-error']
-    return content_location, errors
-

archivebox/extractors/dom.py

@@ -1,72 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-from pathlib import Path
-from typing import Optional
-
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
-from ..system import run, chmod_file, atomic_write
-from ..util import (
-    enforce_types,
-    is_static_file,
-    chrome_args,
-    chrome_cleanup,
-)
-from ..config import (
-    TIMEOUT,
-    SAVE_DOM,
-    CHROME_VERSION,
-)
-from ..logging_util import TimedProgress
-
-
-
-@enforce_types
-def should_save_dom(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
-    if is_static_file(link.url):
-        return False
-
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'output.html').exists():
-        if (out_dir / 'output.html').stat().st_size > 1:
-            return False
-
-    return SAVE_DOM
-
-@enforce_types
-def save_dom(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """print HTML of site to file using chrome --dump-html"""
-
-    out_dir = out_dir or Path(link.link_dir)
-    output: ArchiveOutput = 'output.html'
-    output_path = out_dir / output
-    cmd = [
-        *chrome_args(),
-        '--dump-dom',
-        link.url
-    ]
-    status = 'succeeded'
-    timer = TimedProgress(timeout, prefix='      ')
-    try:
-        result = run(cmd, cwd=str(out_dir), timeout=timeout)
-        atomic_write(output_path, result.stdout)
-
-        if result.returncode:
-            hints = result.stderr.decode()
-            raise ArchiveError('Failed to save DOM', hints)
-
-        chmod_file(output, cwd=str(out_dir))
-    except Exception as err:
-        status = 'failed'
-        output = err
-        chrome_cleanup()
-    finally:
-        timer.end()
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=CHROME_VERSION,
-        output=output,
-        status=status,
-        **timer.stats,
-    )

archivebox/extractors/favicon.py

@@ -1,64 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-from pathlib import Path
-
-from typing import Optional
-
-from ..index.schema import Link, ArchiveResult, ArchiveOutput
-from ..system import chmod_file, run
-from ..util import enforce_types, domain
-from ..config import (
-    TIMEOUT,
-    SAVE_FAVICON,
-    FAVICON_PROVIDER,
-    CURL_BINARY,
-    CURL_ARGS,
-    CURL_VERSION,
-    CHECK_SSL_VALIDITY,
-    CURL_USER_AGENT,
-)
-from ..logging_util import TimedProgress
-
-
-@enforce_types
-def should_save_favicon(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'favicon.ico').exists():
-        return False
-
-    return SAVE_FAVICON
-
-@enforce_types
-def save_favicon(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """download site favicon from google's favicon api"""
-
-    out_dir = out_dir or link.link_dir
-    output: ArchiveOutput = 'favicon.ico'
-    cmd = [
-        CURL_BINARY,
-        *CURL_ARGS,
-        '--max-time', str(timeout),
-        '--output', str(output),
-        *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
-        *([] if CHECK_SSL_VALIDITY else ['--insecure']),
-        FAVICON_PROVIDER.format(domain(link.url)),
-    ]
-    status = 'failed'
-    timer = TimedProgress(timeout, prefix='      ')
-    try:
-        run(cmd, cwd=str(out_dir), timeout=timeout)
-        chmod_file(output, cwd=str(out_dir))
-        status = 'succeeded'
-    except Exception as err:
-        output = err
-    finally:
-        timer.end()
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=CURL_VERSION,
-        output=output,
-        status=status,
-        **timer.stats,
-    )

archivebox/extractors/git.py

@@ -1,90 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-
-from pathlib import Path
-from typing import Optional
-
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
-from ..system import run, chmod_file
-from ..util import (
-    enforce_types,
-    is_static_file,
-    domain,
-    extension,
-    without_query,
-    without_fragment,
-)
-from ..config import (
-    TIMEOUT,
-    SAVE_GIT,
-    GIT_BINARY,
-    GIT_ARGS,
-    GIT_VERSION,
-    GIT_DOMAINS,
-    CHECK_SSL_VALIDITY
-)
-from ..logging_util import TimedProgress
-
-
-
-@enforce_types
-def should_save_git(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
-    if is_static_file(link.url):
-        return False
-
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'git').exists():
-        return False
-
-    is_clonable_url = (
-        (domain(link.url) in GIT_DOMAINS)
-        or (extension(link.url) == 'git')
-    )
-    if not is_clonable_url:
-        return False
-
-    return SAVE_GIT
-
-
-@enforce_types
-def save_git(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """download full site using git"""
-
-    out_dir = out_dir or Path(link.link_dir)
-    output: ArchiveOutput = 'git'
-    output_path = out_dir / output
-    output_path.mkdir(exist_ok=True)
-    cmd = [
-        GIT_BINARY,
-        'clone',
-        *GIT_ARGS,
-        *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']),
-        without_query(without_fragment(link.url)),
-    ]
-    status = 'succeeded'
-    timer = TimedProgress(timeout, prefix='      ')
-    try:
-        result = run(cmd, cwd=str(output_path), timeout=timeout + 1)
-        if result.returncode == 128:
-            # ignore failed re-download when the folder already exists
-            pass
-        elif result.returncode > 0:
-            hints = 'Got git response code: {}.'.format(result.returncode)
-            raise ArchiveError('Failed to save git clone', hints)
-
-        chmod_file(output, cwd=str(out_dir))
-
-    except Exception as err:
-        status = 'failed'
-        output = err
-    finally:
-        timer.end()
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=GIT_VERSION,
-        output=output,
-        status=status,
-        **timer.stats,
-    )

archivebox/extractors/headers.py

@@ -1,70 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-from pathlib import Path
-
-from typing import Optional
-
-from ..index.schema import Link, ArchiveResult, ArchiveOutput
-from ..system import atomic_write
-from ..util import (
-    enforce_types,
-    get_headers,
-)
-from ..config import (
-    TIMEOUT,
-    CURL_BINARY,
-    CURL_ARGS,
-    CURL_USER_AGENT,
-    CURL_VERSION,
-    CHECK_SSL_VALIDITY,
-    SAVE_HEADERS
-)
-from ..logging_util import TimedProgress
-
-@enforce_types
-def should_save_headers(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'headers.json').exists():
-        return False
-
-    return SAVE_HEADERS
-
-
-@enforce_types
-def save_headers(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """Download site headers"""
-
-    out_dir = Path(out_dir or link.link_dir)
-    output_folder = out_dir.absolute()
-    output: ArchiveOutput = 'headers.json'
-
-    status = 'succeeded'
-    timer = TimedProgress(timeout, prefix='      ')
-
-    cmd = [
-        CURL_BINARY,
-        *CURL_ARGS,
-        '--head',
-        '--max-time', str(timeout),
-        *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
-        *([] if CHECK_SSL_VALIDITY else ['--insecure']),
-        link.url,
-    ]
-    try:
-        json_headers = get_headers(link.url, timeout=timeout)
-        output_folder.mkdir(exist_ok=True)
-        atomic_write(str(output_folder / "headers.json"), json_headers)
-    except (Exception, OSError) as err:
-        status = 'failed'
-        output = err
-    finally:
-        timer.end()
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=CURL_VERSION,
-        output=output,
-        status=status,
-        **timer.stats,
-    )

archivebox/extractors/htmltotext.py

@@ -1,154 +0,0 @@
-__package__ = 'archivebox.extractors'
-
-from html.parser import HTMLParser
-import io
-from pathlib import Path
-from typing import Optional
-
-from ..config import (
-    SAVE_HTMLTOTEXT,
-    TIMEOUT,
-    VERSION,
-)
-from ..index.schema import Link, ArchiveResult, ArchiveError
-from ..logging_util import TimedProgress
-from ..system import atomic_write
-from ..util import (
-    enforce_types,
-    is_static_file,
-)
-from .title import get_html
-
-class HTMLTextExtractor(HTMLParser):
-    TEXT_ATTRS = [
-        "alt", "cite", "href", "label",
-        "list", "placeholder", "title", "value"
-    ]
-    NOTEXT_TAGS = ["script", "style", "template"]
-    NOTEXT_HREF = ["data:", "javascript:", "#"]
-
-    def __init__(self):
-        super().__init__()
-
-        self.output = io.StringIO()
-        self._tag_stack = []
-
-    def _is_text_attr(self, name, value):
-        if not isinstance(value, str):
-            return False
-        if name == "href" and any(map(lambda p: value.startswith(p), self.NOTEXT_HREF)):
-            return False
-
-        if name in self.TEXT_ATTRS:
-            return True
-
-        return False
-
-    def _parent_tag(self):
-        try:
-            return self._tag_stack[-1]
-        except IndexError:
-            return None
-
-    def _in_notext_tag(self):
-        return any([t in self._tag_stack for t in self.NOTEXT_TAGS])
-
-    def handle_starttag(self, tag, attrs):
-        self._tag_stack.append(tag)
-
-        # Don't write out attribute values if any ancestor
-        # is in NOTEXT_TAGS
-        if self._in_notext_tag():
-            return
-
-        for name, value in attrs:
-            if self._is_text_attr(name, value):
-                self.output.write(f"({value.strip()}) ")
-
-    def handle_endtag(self, tag):
-        orig_stack = self._tag_stack.copy()
-        try:
-            # Keep popping tags until we find the nearest
-            # ancestor matching this end tag
-            while tag != self._tag_stack.pop():
-                pass
-            # Write a space after every tag, to ensure that tokens
-            # in tag text aren't concatenated. This may result in
-            # excess spaces, which should be ignored by search tokenizers.
-            if not self._in_notext_tag() and tag not in self.NOTEXT_TAGS:
-                self.output.write(" ")
-        except IndexError:
-            # Got to the top of the stack, but somehow missed
-            # this end tag -- maybe malformed markup -- restore the
-            # stack
-            self._tag_stack = orig_stack
-
-    def handle_data(self, data):
-        # Don't output text data if any ancestor is in NOTEXT_TAGS
-        if self._in_notext_tag():
-            return
-
-        data = data.lstrip()
-        len_before_rstrip = len(data)
-        data = data.rstrip()
-        spaces_rstripped = len_before_rstrip - len(data)
-        if data:
-            self.output.write(data)
-            if spaces_rstripped:
-                # Add back a single space if 1 or more
-                # whitespace characters were stripped
-                self.output.write(' ')
-
-    def __str__(self):
-        return self.output.getvalue()
-
-
-@enforce_types
-def should_save_htmltotext(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
-    if is_static_file(link.url):
-        return False
-
-    out_dir = out_dir or Path(link.link_dir)
-    if not overwrite and (out_dir / 'htmltotext.txt').exists():
-        return False
-
-    return SAVE_HTMLTOTEXT
-
-
-@enforce_types
-def save_htmltotext(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
-    """extract search-indexing-friendly text from an HTML document"""
-
-    out_dir = Path(out_dir or link.link_dir)
-    output = "htmltotext.txt"
-
-    timer = TimedProgress(timeout, prefix='      ')
-    extracted_text = None
-    try:
-        extractor = HTMLTextExtractor()
-        document = get_html(link, out_dir)
-
-        if not document:
-            raise ArchiveError('htmltotext could not find HTML to parse for article text')
-
-        extractor.feed(document)
-        extractor.close()
-        extracted_text = str(extractor)
-
-        atomic_write(str(out_dir / output), extracted_text)
-    except (Exception, OSError) as err:
-        status = 'failed'
-        output = err
-        cmd = ['(internal) archivebox.extractors.htmltotext', './{singlefile,dom}.html']
-    finally:
-        timer.end()
-
-    return ArchiveResult(
-        cmd=cmd,
-        pwd=str(out_dir),
-        cmd_version=VERSION,
-        output=output,
-        status=status,
-        index_texts=[extracted_text] if extracted_text else [],
-        **timer.stats,  
-    )