Preserve historical code blocks and pin the complete docs build

.dockerignore

@@ -1,32 +0,0 @@
-.DS_Store
-._*
-*.pyc
-__pycache__/
-.mypy_cache/
-.pytest_cache/
-.github/
-.pdm-build/
-.pdm-python
-.eggs/
-.git/
-!.git/HEAD
-!.git/refs/heads/*
-
-venv/
-.venv/
-.venv-old/
-.docker-venv/
-node_modules/
-
-docs/
-build/
-dist/
-brew_dist/
-deb_dist/
-pip_dist/
-assets/
-docker/
-
-data/
-data*/
-output/

.github/.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: 2
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+  jobs:
+    build:
+      html:
+        - uv run --no-project --with-requirements .github/docs/requirements.txt sphinx-build -W --keep-going -b html -c .github/docs . "$READTHEDOCS_OUTPUT/html"
+python:
+  install:
+    - method: uv
+      command: pip
+      requirements: .github/docs/requirements.txt
+sphinx:
+  configuration: .github/docs/conf.py
+  fail_on_warning: true

.github/CODE_OF_CONDUCT.md

@@ -1,76 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, sex characteristics, gender identity and expression,
-level of experience, education, socio-economic status, nationality, personal
-appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
- advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
- address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at archivebox-coc@sweeting.me. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
-
-[homepage]: https://www.contributor-covenant.org
-
-For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq

.github/CONTRIBUTING.md

@@ -1,38 +0,0 @@
-# Contribution Process
-
-1. Confirm your desired features fit into our bigger project goals [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap).
-2. Open an issue with your planned implementation to discuss
-3. Check in with me before starting development to make sure your work wont conflict with or duplicate existing work
-4. Setup your dev environment, make some changes, and test using the test input files
-5. Commit, push, and submit a PR and wait for review feedback
-6. Have patience, don't abandon your PR! We love contributors but we all have day jobs and don't always have time to respond to notifications instantly. If you want a faster response, ping @theSquashSH on twitter or Patreon.
-
-**Useful links:**
-
-- https://github.com/ArchiveBox/ArchiveBox/issues
-- https://github.com/ArchiveBox/ArchiveBox/pulls
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
-
-### Development Setup
-
-```bash
-git clone https://github.com/ArchiveBox/ArchiveBox
-cd ArchiveBox
-# Ideally do this in a virtualenv
-pip install -e '.[dev]'  # or use: pipenv install --dev
-```
-
-### Running Tests
-
-```bash
-./bin/lint.sh
-./bin/test.sh
-./bin/build.sh
-```
-
-For more common tasks see the `Development` section at the bottom of the README.
-
-### Getting Help
-
-Open issues on Github or message me https://sweeting.me/#contact.

.github/FUNDING.yml

@@ -1,3 +0,0 @@
-github: pirate
-patreon: theSquashSH
-custom: ["https://hcb.hackclub.com/donations/start/archivebox", "https://paypal.me/NicholasSweeting"]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,46 +0,0 @@
----
-name: 🐞 Bug report
-about: Create a report to help us improve
-title: 'Bug: ...'
-labels: 'bug'
-assignees: ''
-
----
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you.
-(the only required section is the version information)
--->
-
-#### Describe the bug
-<!--
-A description of what the bug is, 
-what you expected to happen, 
-and any relevant context about issue.
--->
-
-#### Steps to reproduce
-<!--
-For example:
-1. Ran ArchiveBox with the following config '...'
-2. Saw this output during archiving '....'
-3. UI didn't show the thing I was expecting '....'
--->
-
-#### Screenshots or log output
-
-<!--
-If applicable, post any relevant screenshots or copy/pasted terminal output from ArchiveBox.
-If you're reporting a parsing / importing error, **you must paste a copy of your redacted import file here**.
--->
-
-#### ArchiveBox version
-
-<!-- Run the `archivebox version` command locally then copy paste the result here: -->
-```logs
-replace this line with the *full*, unshortened output of running `archivebox version`
-```
-<!-- Tickets without full version info will closed until it is provided,
-we need the full output here to help you solve your issue -->

.github/ISSUE_TEMPLATE/documentation_change.md

@@ -1,17 +0,0 @@
----
-name: 📑 Documentation change
-about: Submit a suggestion for the Wiki documentation
-title: 'Documentation: Improvement request ...'
-labels: ''
-assignees: ''
-
----
-<!-- If you perfer, you can make a PR to https://github.com/ArchiveBox/docs instead of opening an issue -->
-
-## Wiki Page URL
-<!-- e.g. https://github.com/pirate/ArchiveBox/wiki/Configuration#use_color -->
-
-
-## Suggested Edit
-<!-- e.g. Please add more example usages, or please fix `xyz` typo to be `abc`. -->
-

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,50 +0,0 @@
----
-name: 💡 Feature request
-about: Suggest an idea for this project
-title: 'Feature Request: ...'
-labels: 'changes: behavior,status: idea phase'
-assignees: ''
-
----
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you :)
--->
-
-## Type 
-
- - [ ] General question or discussion
- - [ ] Propose a brand new feature
- - [ ] Request modification of existing behavior or design
-
-## What is the problem that your feature request solves
-<!--
-e.g. I need to be able to archive spanish and french subtitle files  
-from a particular <example.com> movie site that's going down soon.
--->
-
-## Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes
-<!--
-e.g. I specifically need a new archive method to look for multilingual subtitle files related to pages.  
-The bigger picture solution is the ability for custom user scripts to be run in a puppeteer context during archiving.
--->
-
-## What hacks or alternative solutions have you tried to solve the problem?
-<!--
-A clear and concise description of any alternative solutions, 
-workarounds, or other software you've considered using to fix the problem.
--->
-
-## How badly do you want this new feature?
-
- - [ ] It's an urgent deal-breaker, I can't live without it
- - [ ] It's important to add it in the near-mid term future
- - [ ] It would be nice to have eventually
-
----
-
- - [ ] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue
- - [ ] I like ArchiveBox so far / would recommend it to a friend
- - [ ] I've had a lot of difficulty getting ArchiveBox set up

.github/ISSUE_TEMPLATE/question_or_discussion.md

@@ -1,9 +0,0 @@
----
-name: 💬 Question, discussion, or support request
-about: Start a discussion or ask a question about ArchiveBox
-title: 'Question: ...'
-labels: ''
-assignees: ''
-
----
-

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,18 +0,0 @@
-<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes. -->
-
-# Summary
-
-<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->
-
-# Related issues
-
-<!-- e.g. #123 or Roadmap goal # https://github.com/pirate/ArchiveBox/wiki/Roadmap -->
-
-# Changes these areas
-
-- [ ] Bugfixes
-- [ ] Feature behavior
-- [ ] Command line interface
-- [ ] Configuration options
-- [ ] Internal architecture
-- [ ] Snapshot data layout on disk

.github/dependabot.yml

@@ -1,12 +0,0 @@
-# To get started with Dependabot version updates, you'll need to specify which
-# package ecosystems to update and where the package manifests are located.
-# Please see the documentation for all configuration options:
-# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
-
-version: 2
-updates:
-  - package-ecosystem: "pip" # See documentation for possible values
-    directory: "/"
-    target-branch: "dev"
-    schedule:
-      interval: "weekly"

.github/docs/README.md

@@ -0,0 +1,25 @@
+# Historical 0.0.3 documentation
+
+This documentation-only branch starts at the immutable `v0.0.3` tag.
+The original README, application files, and pinned wiki (when present) remain unchanged.
+Sphinx reads them without installing or importing the historical application.
+The render-time adapter fixes historical link and heading syntax only; the visible warning
+explains that these old installation commands and external services may no longer work.
+
+Prerequisites: [uv](https://docs.astral.sh/uv/getting-started/installation/) 0.11.3 and Python 3.13, matching CI and Read the Docs. `uv` can install Python when needed.
+
+```sh
+READTHEDOCS=True uv run --no-project --python 3.13 --with-requirements .github/docs/requirements.txt \
+  sphinx-build -E -a -W --keep-going -b html -c .github/docs . /tmp/archivebox-historical-docs
+```
+
+Read the Docs must use `.github/.readthedocs.yaml`. The pinned theme supplies
+its version selector and listens for Read the Docs addon version data.
+
+`requirements.txt` pins the complete documentation dependency set. To update it,
+edit `requirements.in` and run:
+
+```sh
+uv pip compile --universal --python-version 3.13 .github/docs/requirements.in \
+  --output-file .github/docs/requirements.txt
+```

.github/docs/conf.py

@@ -0,0 +1,124 @@
+"""Render this release's historical documentation without importing the application."""
+
+import os
+import re
+from pathlib import Path
+from urllib.parse import urlsplit
+
+from markdown_it import MarkdownIt
+
+ROOT = Path(__file__).resolve().parents[2]
+release = "0.0.3"
+version = release
+project = "ArchiveBox"
+author = "ArchiveBox contributors"
+copyright = "ArchiveBox contributors"
+extensions = ["myst_parser", "sphinx_rtd_theme"]
+source_suffix = {".md": "markdown"}
+root_doc = "index"
+exclude_patterns = [
+    p.name for p in ROOT.iterdir() if p.name not in {"index.md", "docs"}
+]
+exclude_patterns += ["docs/_Sidebar.md", "docs/_Footer.md", "docs/.git"]
+html_theme = "sphinx_rtd_theme"
+html_theme_options = {
+    "navigation_depth": 3,
+    "collapse_navigation": False,
+    "version_selector": True,
+    "language_selector": False,
+}
+html_context = {"current_version": os.environ.get("READTHEDOCS_VERSION", release)}
+html_title = f"ArchiveBox {release} historical documentation"
+html_show_sphinx = False
+myst_heading_anchors = 6
+
+
+def rewrite_prose(text, transform):
+    """Rewrite navigation outside fenced and indented code blocks."""
+    lines = text.splitlines(keepends=True)
+    output = []
+    start = 0
+    for token in MarkdownIt().parse(text):
+        if token.type not in {"fence", "code_block"}:
+            continue
+        first, last = token.map
+        output.append(transform("".join(lines[start:first])))
+        output.append("".join(lines[first:last]))
+        start = last
+    output.append(transform("".join(lines[start:])))
+    return "".join(output)
+
+
+def prepare_source(app, docname, source):
+    text = (ROOT / "README.md").read_text() if docname == "index" else source[0]
+    directory = ROOT if docname == "index" else ROOT / "docs"
+
+    def wiki(match):
+        parts = match.group(1).split("|", 1)
+        return f"[{parts[0]}]({parts[-1].replace(' ', '-')}.md)"
+
+    text = rewrite_prose(text, lambda prose: re.sub(r"\[\[([^\]\n]+)\]\]", wiki, prose))
+
+    def resource(target):
+        if urlsplit(target).scheme or target.startswith(("#", "//")):
+            return target
+        page, sep, anchor = target.partition("#")
+        path = directory / page
+        if path.suffix == ".md" and path.parent == ROOT / "docs":
+            return target
+        if path.is_file():
+            return (
+                f"https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/v{release}/{path.relative_to(ROOT)}"
+                + ("#" + anchor if sep else "")
+            )
+        return target
+
+    def rewrite_links(prose):
+        prose = re.sub(
+            r'((?:src|href)=["\'])([^"\']+)', lambda m: m[1] + resource(m[2]), prose
+        )
+        prose = re.sub(r"\]\(([^)\s]+)\)", lambda m: "](" + resource(m[1]) + ")", prose)
+        return prose
+
+    text = rewrite_prose(text, rewrite_links)
+    text = (
+        f"# ArchiveBox {release}"
+        + (
+            " historical guide"
+            if docname == "index"
+            else " — " + Path(docname).name.replace("-", " ")
+        )
+        + "\n\n"
+        + text
+    )
+    lines = text.splitlines()
+    stack = []
+    for token in MarkdownIt().parse(text):
+        if token.type != "heading_open":
+            continue
+        original = int(token.tag[1])
+        while stack and stack[-1] >= original:
+            stack.pop()
+        stack.append(original)
+        lines[token.map[0]] = re.sub(
+            r"^#{1,6}(?=\s)", "#" * len(stack), lines[token.map[0]]
+        )
+    text = "\n".join(lines) + "\n"
+    if docname == "index":
+        notice = f"\n```{{warning}}\nThis is the original guide shipped with ArchiveBox {release}, preserved for historical reference. Commands, dependencies, and external services describe that old release and may no longer work. Use the version selector for current documentation.\n```\n"
+        first, rest = text.split("\n", 1)
+        text = first + "\n" + notice + rest
+        pages = sorted(
+            p.stem for p in (ROOT / "docs").glob("*.md") if not p.name.startswith("_")
+        )
+        if pages:
+            text += (
+                "\n```{toctree}\n:maxdepth: 2\n\n"
+                + "\n".join("docs/" + p for p in pages)
+                + "\n```\n"
+            )
+    source[0] = text
+
+
+def setup(app):
+    app.connect("source-read", prepare_source)

.github/docs/requirements.in

@@ -0,0 +1,3 @@
+Sphinx==8.2.3
+myst-parser==4.0.1
+sphinx-rtd-theme==3.1.0

.github/docs/requirements.txt

@@ -0,0 +1,75 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --universal --python-version 3.13 .github/docs/requirements.in --output-file .github/docs/requirements.txt
+alabaster==1.0.0
+    # via sphinx
+babel==2.18.0
+    # via sphinx
+certifi==2026.7.22
+    # via requests
+charset-normalizer==3.5.1
+    # via requests
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via sphinx
+docutils==0.21.2
+    # via
+    #   myst-parser
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.20
+    # via requests
+imagesize==2.0.1
+    # via sphinx
+jinja2==3.1.6
+    # via
+    #   myst-parser
+    #   sphinx
+markdown-it-py==3.0.0
+    # via
+    #   mdit-py-plugins
+    #   myst-parser
+markupsafe==3.0.3
+    # via jinja2
+mdit-py-plugins==0.6.1
+    # via myst-parser
+mdurl==0.1.2
+    # via markdown-it-py
+myst-parser==4.0.1
+    # via -r .github/docs/requirements.in
+packaging==26.3
+    # via sphinx
+pygments==2.21.0
+    # via sphinx
+pyyaml==6.0.3
+    # via myst-parser
+requests==2.34.2
+    # via sphinx
+roman-numerals==4.1.0
+    # via roman-numerals-py
+roman-numerals-py==4.1.0
+    # via sphinx
+snowballstemmer==3.1.1
+    # via sphinx
+sphinx==8.2.3
+    # via
+    #   -r .github/docs/requirements.in
+    #   myst-parser
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-jquery
+sphinx-rtd-theme==3.1.0
+    # via -r .github/docs/requirements.in
+sphinxcontrib-applehelp==2.0.0
+    # via sphinx
+sphinxcontrib-devhelp==2.0.0
+    # via sphinx
+sphinxcontrib-htmlhelp==2.1.0
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==2.0.0
+    # via sphinx
+sphinxcontrib-serializinghtml==2.0.0
+    # via sphinx
+urllib3==2.8.0
+    # via requests

.github/workflows/codeql.yml

@@ -1,92 +0,0 @@
-# For most projects, this workflow file will not need changing; you simply need
-# to commit it to your repository.
-#
-# You may wish to alter this file to override the set of languages analyzed,
-# or to provide custom queries or build logic.
-#
-# ******** NOTE ********
-# We have attempted to detect the languages in your repository. Please check
-# the `language` matrix defined below to confirm you have the correct set of
-# supported CodeQL languages.
-#
-name: "CodeQL"
-
-on:
-  push:
-    branches: [ "dev" ]
-  pull_request:
-    branches: [ "dev" ]
-  schedule:
-    - cron: '33 17 * * 6'
-
-jobs:
-  analyze:
-    name: Analyze (${{ matrix.language }})
-    # Runner size impacts CodeQL analysis time. To learn more, please see:
-    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
-    #   - https://gh.io/supported-runners-and-hardware-resources
-    #   - https://gh.io/using-larger-runners (GitHub.com only)
-    # Consider using larger runners or machines with greater resources for possible analysis time improvements.
-    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
-    timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
-    permissions:
-      # required for all workflows
-      security-events: write
-
-      # required to fetch internal or private CodeQL packs
-      packages: read
-
-      # only required for workflows in private repositories
-      actions: read
-      contents: read
-
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-        - language: python
-          build-mode: none
-        # CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
-        # Use `c-cpp` to analyze code written in C, C++ or both
-        # Use 'java-kotlin' to analyze code written in Java, Kotlin or both
-        # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
-        # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
-        # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
-        # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
-        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
-    steps:
-    - name: Checkout repository
-      uses: actions/checkout@v4
-
-    # Initializes the CodeQL tools for scanning.
-    - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
-      with:
-        languages: ${{ matrix.language }}
-        build-mode: ${{ matrix.build-mode }}
-        # If you wish to specify custom queries, you can do so here or in a config file.
-        # By default, queries listed here will override any specified in a config file.
-        # Prefix the list here with "+" to use these queries and those in the config file.
-
-        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
-        # queries: security-extended,security-and-quality
-
-    # If the analyze step fails for one of the languages you are analyzing with
-    # "We were unable to automatically build your code", modify the matrix above
-    # to set the build mode to "manual" for that language. Then modify this step
-    # to build your code.
-    # ℹ️ Command-line programs to run using the OS shell.
-    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
-    - if: matrix.build-mode == 'manual'
-      run: |
-        echo 'If you are using a "manual" build mode for one or more of the' \
-          'languages you are analyzing, replace this with the commands to build' \
-          'your code, for example:'
-        echo '  make bootstrap'
-        echo '  make release'
-        exit 1
-
-    - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
-      with:
-        category: "/language:${{matrix.language}}"

.github/workflows/debian.yml

@@ -1,64 +0,0 @@
-name: Build Debian package
-
-on:
-  workflow_dispatch:
-  push:
-
-env:
-  DEB_BUILD_OPTIONS: nocheck
-
-jobs:
-  build:
-    runs-on: ubuntu-22.04
-
-    steps:
-      - uses: actions/checkout@v3
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Install packaging dependencies
-        run: |
-          sudo apt-get update -qq
-          sudo apt-get install -y \
-            python3 python3-dev python3-pip python3-venv python3-all \
-            dh-python debhelper devscripts dput software-properties-common \
-            python3-distutils python3-setuptools python3-wheel python3-stdeb
-
-      # - name: Build Debian/Apt sdist_dsc
-      #   run: |
-      #     ./bin/build_pip.sh
-
-      # - name: Check ArchiveBox version
-      #   run: |
-      #     # must create dir needed for snaps to run as non-root on github actions
-      #     sudo mkdir -p /run/user/1001 && sudo chmod -R 777 /run/user/1001
-      #     mkdir "${{ github.workspace }}/data" && cd "${{ github.workspace }}/data"
-      #     archivebox --version
-      #     archivebox init --setup
-
-      # - name: Add some links to test
-      #   run: |
-      #     cd "${{ github.workspace }}/data"
-      #     archivebox add 'https://example.com'
-      #     archivebox status
-
-      # - name: Commit built package
-      #   run: |
-      #     cd deb_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Debian package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/debian-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: deb_dist
-
-      # - name: Push build to Launchpad PPA
-      #   run: |
-      #     debsign -k "$PGP_KEY_ID" "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"
-      #     dput archivebox "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"

.github/workflows/docker.yml

@@ -1,106 +0,0 @@
-name: Build Docker image
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - '**'
-    tags:
-      - 'v*'
-  # pull_request:
-
-env:
-  DOCKER_IMAGE: archivebox-ci
-
-jobs:
-  buildx:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
-
-      - name: Set up Docker Buildx
-        id: buildx
-        uses: docker/setup-buildx-action@v3
-        with:
-          version: latest
-          install: true
-          platforms: linux/amd64,linux/arm64
-
-      - name: Builder instance name
-        run: echo ${{ steps.buildx.outputs.name }}
-
-      - name: Available platforms
-        run: echo ${{ steps.buildx.outputs.platforms }}
-
-      - name: Cache Docker layers
-        uses: actions/cache@v3
-        with:
-          path: /tmp/.buildx-cache
-          key: ${{ runner.os }}-buildx-${{ github.sha }}
-          restore-keys: |
-            ${{ runner.os }}-buildx-
-
-      - name: Docker Login
-        uses: docker/login-action@v3
-        if: github.event_name != 'pull_request'
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-
-      - name: Collect Docker tags
-        # https://github.com/docker/metadata-action
-        id: docker_meta
-        uses: docker/metadata-action@v5
-        with:
-          images: archivebox/archivebox,nikisweeting/archivebox
-          tags: |
-              # :stable
-              type=ref,event=branch
-              # :0.7.3
-              type=semver,pattern={{version}}
-              # :0.7
-              type=semver,pattern={{major}}.{{minor}}
-              # :sha-463ea54
-              type=sha
-              # :latest
-              type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'stable') }}
-
-      - name: Build and push
-        id: docker_build
-        uses: docker/build-push-action@v5
-        with:
-          context: ./
-          file: ./Dockerfile
-          builder: ${{ steps.buildx.outputs.name }}
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.docker_meta.outputs.tags }}
-          cache-from: type=local,src=/tmp/.buildx-cache
-          cache-to: type=local,dest=/tmp/.buildx-cache-new
-          platforms: linux/amd64,linux/arm64
-
-      - name: Image digest
-        run: echo ${{ steps.docker_build.outputs.digest }}
-
-      - name: Update README
-        uses: peter-evans/dockerhub-description@v4
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-          repository: archivebox/archivebox
-
-      # This ugly bit is necessary if you don't want your cache to grow forever
-      # until it hits GitHub's limit of 5GB.
-      # Temp fix
-      # https://github.com/docker/build-push-action/issues/252
-      # https://github.com/moby/buildkit/issues/1896
-      - name: Move cache
-        run: |
-          rm -rf /tmp/.buildx-cache
-          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

.github/workflows/homebrew.yml

@@ -1,51 +0,0 @@
-name: Build Homebrew package
-
-on:
-  workflow_dispatch:
-  push:
-
-
-jobs:
-  build:
-    runs-on: macos-latest
-
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      # TODO: modify archivebox.rb to update src url, hashes, and dependencies
-
-      - name: Build Homebrew Bottle
-        run: |
-          pip3 install --upgrade pip setuptools wheel
-          cd brew_dist/
-          brew install --build-bottle ./archivebox.rb
-          # brew bottle archivebox
-          archivebox version
-
-      - name: Add some links to test
-        run: |
-          mkdir data && cd data
-          archivebox init --setup
-          archivebox add 'https://example.com'
-          archivebox version
-          archivebox status
-
-      # - name: Commit built package
-      #   run: |
-      #     cd brew_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Homebrew package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/homebrew-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: brew_dist
-
-      # TODO: push bottle homebrew core PR with latest changes

.github/workflows/jekyll-gh-pages.yml

@@ -1,58 +0,0 @@
-# Sample workflow for building and deploying a Jekyll site to GitHub Pages
-name: Build GitHub Pages website
-
-on:
-  # Runs on pushes targeting the default branch
-  push:
-    branches: ["dev"]
-
-  # Allows you to run this workflow manually from the Actions tab
-  workflow_dispatch:
-
-# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-
-# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
-# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
-concurrency:
-  group: "pages"
-  cancel-in-progress: true
-
-jobs:
-  # Build job
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-      - name: Copy README.md into place
-        run: |
-          rm ./website/README.md
-          cp ./README.md ./website/README.md
-      - name: Setup Pages
-        uses: actions/configure-pages@v3
-      - name: Build with Jekyll
-        uses: actions/jekyll-build-pages@v1
-        with:
-          source: ./website
-          destination: ./_site
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@v2
-
-  # Deployment job
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
-    needs: build
-    steps:
-      - name: Deploy to GitHub Pages
-        id: deployment
-        uses: actions/deploy-pages@v2

.github/workflows/lint.yml

@@ -1,35 +0,0 @@
-name: Run linters
-
-on:
-  workflow_dispatch:
-  push:
-
-env:
-  MAX_LINE_LENGTH: 110
-
-jobs:
-  lint:
-    runs-on: ubuntu-20.04
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.9
-          architecture: x64
-
-      - name: Install flake8
-        run: |
-          pip install flake8
-
-      - name: Lint with flake8
-        run: |
-          cd archivebox
-          # one pass for show-stopper syntax errors or undefined names
-          flake8 . --count --show-source --statistics
-          # one pass for small stylistic things
-          flake8 . --count --max-line-length="$MAX_LINE_LENGTH" --statistics

.github/workflows/pip.yml

@@ -1,78 +0,0 @@
-name: Build Pip package
-
-on:
-  workflow_dispatch:
-  push:
-
-
-jobs:
-  build:
-    runs-on: ubuntu-22.04
-
-    permissions:
-      id-token: write
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: '3.11'
-          architecture: x64
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
-
-      - uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: '3.11'
-          cache: true
-
-      - name: Install dependencies
-        run: pdm install --fail-fast --no-lock --dev --group=':all' --no-self
-
-      - name: Build package
-        run: |
-          pdm build
-
-      - name: Install from build
-        run: pip install ./dist/archivebox-*.whl
-
-      - name: Add some links to test
-        run: |
-          mkdir data && cd data
-          archivebox init
-          archivebox add 'https://example.com'
-          archivebox version
-          archivebox status
-
-      #- name: Publish package distributions to PyPI
-      #  run: pdm publish --no-build
-
-      # - name: Push build to PyPI
-      #   run: |
-      #     cd pip_dist/
-      #     python3 -m twine upload --repository testpypi pip_dist/*.{whl,tar.gz}
-      #     python3 -m twine upload --repository pypi pip_dist/*.{whl,tar.gz}
-
-      # - name: Commit built package
-      #   run: |
-      #     cd pip_dist/
-      #     git config --local user.email "action@github.com"
-      #     git config --local user.name "GitHub Action"
-      #     git commit -m "Pip package autobuild" -a
-      
-      # - name: Push build to Github
-      #   uses: ad-m/github-push-action@master
-      #   with:
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
-      #     repository: ArchiveBox/pip-archivebox
-      #     branch: ${{ github.ref }}
-      #     directory: pip_dist
-

.github/workflows/test.yml

@@ -1,158 +0,0 @@
-name: Run tests
-on: [push]
-
-env:
-  DOCKER_IMAGE: archivebox-ci
-  PYTHONIOENCODING: utf-8
-  PYTHONLEGACYWINDOWSSTDIO: utf-8
-  USE_COLOR: False
-
-jobs:
-  python_tests:
-    runs-on: ${{ matrix.os }}
-
-    strategy:
-      matrix:
-        os: [ubuntu-22.04]
-        # os: [ubuntu-22.04, macos-latest, windows-latest]
-        python: [3.11]
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      ### Setup Python & JS Languages
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ matrix.python }}
-          architecture: x64
-
-      - name: Set up Node JS
-        uses: actions/setup-node@v4
-        with:
-          node-version: 20.10.0
-
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: '3.11'
-          cache: true
-
-      ### Install Python & JS Dependencies
-      - name: Get pip cache dir
-        id: pip-cache
-        run: |
-          echo "::set-output name=dir::$(pip cache dir)"
-
-      - name: Cache pip
-        uses: actions/cache@v3
-        id: cache-pip
-        with:
-          path: ${{ steps.pip-cache.outputs.dir }}
-          key: ${{ runner.os }}-${{ matrix.python }}-venv-${{ hashFiles('setup.py') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.python }}-venv-
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 python3-minimal gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
-
-      - name: Install pip dependencies
-        run: |
-          python -m pip install --upgrade pip setuptools wheel pytest bottle build
-          python -m pip install -r requirements.txt
-          python -m pip install -e .[sonic,ldap]
-
-      - name: Get npm cache dir
-        id: npm-cache
-        run: |
-          echo "::set-output name=dir::$GITHUB_WORKSPACE/node_modules"
-
-      - name: Cache npm
-        uses: actions/cache@v3
-        id: cache-npm
-        with:
-          path: ${{ steps.npm-cache.outputs.dir }}
-          key: ${{ runner.os }}-node_modules-${{ hashFiles('package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-node_modules
-
-      - name: Install npm requirements
-        run: |
-          npm install
-          echo "SINGLEFILE_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/single-file" >> $GITHUB_ENV
-          echo "READABILITY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/readability-extractor" >> $GITHUB_ENV
-          echo "MERCURY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/mercury-parser" >> $GITHUB_ENV
-
-      ### Run the tests
-      - name: Directory listing for debugging
-        run: |
-          pwd
-          ls
-
-      - name: Archivebox version
-        run: |
-          archivebox version
-
-      - name: Test built package with pytest
-        # TODO: remove this exception for windows once we get tests passing on that platform
-        if: ${{ !contains(matrix.os, 'windows') }}
-        run: |
-          python -m pytest -s --basetemp=tests/out --ignore=archivebox/vendor --ignore=deb_dist --ignore=pip_dist --ignore=brew_dist
-
-  docker_tests:
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      # TODO: as of 2020-11 this helper layer broke, upgrade and re-enable this once it's usable again
-      # - uses: satackey/action-docker-layer-caching@v0.0.8
-
-      - name: Build image
-        run: |
-          docker build . -t "$DOCKER_IMAGE"
-
-      - name: Init data dir
-        run: |
-          mkdir "${{ github.workspace }}/data"
-          docker run -v "${{ github.workspace }}/data":/data "$DOCKER_IMAGE" init
-
-      - name: Run test server
-        run: |
-          sudo bash -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
-          docker run --name www-nginx -p 80:80 -d nginx
-
-      - name: Add link
-        run: |
-          docker run -v "$PWD"/data:/data --network host "$DOCKER_IMAGE" add http://www.test-nginx-1.local
-
-      - name: Add stdin link
-        run: |
-          echo "http://www.test-nginx-2.local" | docker run -i --network host -v "$PWD"/data:/data "$DOCKER_IMAGE" add
-
-      - name: List links
-        run: |
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
-
-      - name: Start docker-compose stack
-        run: |
-          docker-compose run archivebox init
-          docker-compose up -d
-          sleep 5
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'ArchiveBox'
-          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'window.django'
-
-      - name: Check added urls show up in index
-        run: |
-          docker-compose run archivebox add 'http://example.com/#test_docker' --index-only
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'http://example.com/#test_docker'
-          docker-compose down || true

.gitignore

@@ -1,34 +1,19 @@
-.DS_Store
-._*
-*.pyc
+# Pocket archive output folder
+pocket/
+bookmarks/
+pinboard/
+html/
+downloads/
+
+# Byte-compiled / optimized / DLL files
 __pycache__/
-.mypy_cache/
-.eggs/
-tests/out/
 
-# Python and Node dependencies
+# Virtualenv
+env/
+ENV/
 venv/
+VENV/
 .venv/
-.docker-venv/
-node_modules/
-
-# Ignore dev lockfiles (should always be built fresh)
-requirements-dev.txt
-pdm.dev.lock
-
-# Packaging artifacts
-.pdm-python
-.pdm-build
-archivebox.egg-info
-archivebox-*.tar.gz
-build/
-dist/
-
-# Data folders
-data/
-data*/
-output/
-
-# vim
-*.sw?
-.vscode/
+.env/
+.python-version
+.env

.gitmodules

@@ -1,28 +0,0 @@
-[submodule "docs"]
-    path = docs
-    url = https://github.com/ArchiveBox/ArchiveBox.wiki.git
-
-[submodule "deb_dist"]
-    path = deb_dist
-    url = https://github.com/ArchiveBox/debian-archivebox.git
-[submodule "brew_dist"]
-	path = brew_dist
-	url = https://github.com/ArchiveBox/homebrew-archivebox.git
-[submodule "pip_dist"]
-	path = pip_dist
-	url = https://github.com/ArchiveBox/pip-archivebox.git
-[submodule "docker"]
-	path = docker
-	url = https://github.com/ArchiveBox/docker-archivebox.git
-[submodule "archivebox/vendor/base32-crockford"]
-	path = archivebox/vendor/base32-crockford
-	url = https://github.com/jbittel/base32-crockford
-[submodule "archivebox/vendor/pocket"]
-	path = archivebox/vendor/pocket
-	url = https://github.com/tapanpandita/pocket
-[submodule "archivebox/vendor/django-taggit"]
-	path = archivebox/vendor/django-taggit
-	url = https://github.com/jazzband/django-taggit
-[submodule "archivebox/vendor/python-atomicwrites"]
-	path = archivebox/vendor/python-atomicwrites
-	url = https://github.com/untitaker/python-atomicwrites

.npmignore

@@ -1,19 +0,0 @@
-tests/
-archivebox/
-archivebox.egg-info/
-build/
-dist/
-docs/
-etc/
-.github
-.gitmodules
-.dockerignore
-.flake8
-CNAME
-_config.yml
-docker-compose.yaml
-docker-compose.yml
-Dockerfile
-MANIFEST.in
-Pipfile
-setup.py

.readthedocs.yaml

@@ -1,33 +0,0 @@
-# Read the Docs configuration file for Sphinx projects
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-version: 2
-
-submodules:
-  include: all
-  recursive: true
-
-# Set the OS, Python version and other tools you might need
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.12"
-    nodejs: "20"
-
-# Build documentation in the "docs/" directory with Sphinx
-sphinx:
-  configuration: docs/conf.py
-  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
-  # builder: "dirhtml"
-
-# Optionally build your docs in additional formats such as PDF and ePub
-formats:
-  - pdf
-  - epub
-
-# Optional but recommended, declare the Python requirements required
-# to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
-  install:
-    - requirements: docs/requirements.txt

CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at git@nicksweeting.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

Dockerfile

@@ -1,322 +0,0 @@
-# This is the Dockerfile for ArchiveBox, it bundles the following dependencies:
-#     python3, ArchiveBox, curl, wget, git, chromium, youtube-dl, yt-dlp, single-file
-# Usage:
-#     git submodule update --init --recursive
-#     git pull --recurse-submodules
-#     docker build . -t archivebox --no-cache
-#     docker run -v "$PWD/data":/data archivebox init
-#     docker run -v "$PWD/data":/data archivebox add 'https://example.com'
-#     docker run -v "$PWD/data":/data -it archivebox manage createsuperuser
-#     docker run -v "$PWD/data":/data -p 8000:8000 archivebox server
-# Multi-arch build:
-#     docker buildx create --use
-#     docker buildx build . --platform=linux/amd64,linux/arm64--push -t archivebox/archivebox:latest -t archivebox/archivebox:dev
-#
-# Read more about [developing Archivebox](https://github.com/ArchiveBox/ArchiveBox#archivebox-development).
-
-
-FROM python:3.11-slim-bookworm
-# Uses Debian 12 w/ faster-updating apt-lists added below: https://packages.debian.org/bookworm-backports/
-
-LABEL name="archivebox" \
-    maintainer="Nick Sweeting <dockerfile@archivebox.io>" \
-    description="All-in-one self-hosted internet archiving solution" \
-    homepage="https://github.com/ArchiveBox/ArchiveBox" \
-    documentation="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker" \
-    org.opencontainers.image.title="ArchiveBox" \
-    org.opencontainers.image.vendor="ArchiveBox" \
-    org.opencontainers.image.description="All-in-one self-hosted internet archiving solution" \
-    org.opencontainers.image.source="https://github.com/ArchiveBox/ArchiveBox" \
-    com.docker.image.source.entrypoint="Dockerfile" \
-    # TODO: release ArchiveBox as a Docker Desktop extension (requires these labels):
-    # https://docs.docker.com/desktop/extensions-sdk/architecture/metadata/
-    com.docker.desktop.extension.api.version=">= 1.4.7" \
-    com.docker.desktop.extension.icon="https://archivebox.io/icon.png" \
-    com.docker.extension.publisher-url="https://archivebox.io" \
-    com.docker.extension.screenshots='[{"alt": "Screenshot of Admin UI", "url": "https://github.com/ArchiveBox/ArchiveBox/assets/511499/e8e0b6f8-8fdf-4b7f-8124-c10d8699bdb2"}]' \
-    com.docker.extension.detailed-description='See here for detailed documentation: https://wiki.archivebox.io' \
-    com.docker.extension.changelog='See here for release notes: https://github.com/ArchiveBox/ArchiveBox/releases' \
-    com.docker.extension.categories='database,utility-tools'
-    
-ARG TARGETPLATFORM
-ARG TARGETOS
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-######### Environment Variables #################################
-
-# Global system-level config
-ENV TZ=UTC \
-    LANGUAGE=en_US:en \
-    LC_ALL=C.UTF-8 \
-    LANG=C.UTF-8 \
-    DEBIAN_FRONTEND=noninteractive \
-    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 \
-    PYTHONIOENCODING=UTF-8 \
-    PYTHONUNBUFFERED=1 \
-    PIP_DISABLE_PIP_VERSION_CHECK=1 \
-    npm_config_loglevel=error
-
-# Version config
-ENV PYTHON_VERSION=3.11 \
-    NODE_VERSION=20
-
-# User config
-ENV ARCHIVEBOX_USER="archivebox" \
-    DEFAULT_PUID=911 \
-    DEFAULT_PGID=911
-
-# Global paths
-ENV CODE_DIR=/app \
-    DATA_DIR=/data \
-    GLOBAL_VENV=/venv \
-    PLAYWRIGHT_BROWSERS_PATH=/browsers
-
-# Application-level paths
-ENV APP_VENV=/app/.venv \
-    NODE_MODULES=/app/node_modules
-
-# Build shell config
-ENV PATH="$PATH:$GLOBAL_VENV/bin:$APP_VENV/bin:$NODE_MODULES/.bin"
-SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "errtrace", "-o", "nounset", "-c"] 
-
-######### System Environment ####################################
-
-# Detect ArchiveBox version number by reading package.json
-COPY --chown=root:root --chmod=755 package.json "$CODE_DIR/"
-RUN grep '"version": ' "${CODE_DIR}/package.json" | awk -F'"' '{print $4}' > /VERSION.txt
-
-# Force apt to leave downloaded binaries in /var/cache/apt (massively speeds up Docker builds)
-RUN echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache \
-    && rm -f /etc/apt/apt.conf.d/docker-clean
-
-# Print debug info about build and save it to disk, for human eyes only, not used by anything else
-RUN (echo "[i] Docker build for ArchiveBox $(cat /VERSION.txt) starting..." \
-    && echo "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})" \
-    && echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
-    && echo \
-    && echo "GLOBAL_VENV=${GLOBAL_VENV} APP_VENV=${APP_VENV} NODE_MODULES=${NODE_MODULES}" \
-    && echo "PYTHON=${PYTHON_VERSION} NODE=${NODE_VERSION} PATH=${PATH}" \
-    && echo "CODE_DIR=${CODE_DIR} DATA_DIR=${DATA_DIR}" \
-    && echo \
-    && uname -a \
-    && cat /etc/os-release | head -n7 \
-    && which bash && bash --version | head -n1 \
-    && which dpkg && dpkg --version | head -n1 \
-    && echo -e '\n\n' && env && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Create non-privileged user for archivebox and chrome
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_PUID}..." \
-    && groupadd --system $ARCHIVEBOX_USER \
-    && useradd --system --create-home --gid $ARCHIVEBOX_USER --groups audio,video $ARCHIVEBOX_USER \
-    && usermod -u "$DEFAULT_PUID" "$ARCHIVEBOX_USER" \
-    && groupmod -g "$DEFAULT_PGID" "$ARCHIVEBOX_USER" \
-    && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER PUID=$(id -u $ARCHIVEBOX_USER) PGID=$(id -g $ARCHIVEBOX_USER)\n\n" \
-    | tee -a /VERSION.txt
-    # DEFAULT_PUID and DEFAULT_PID are overriden by PUID and PGID in /bin/docker_entrypoint.sh at runtime
-    # https://docs.linuxserver.io/general/understanding-puid-and-pgid
-
-# Install system apt dependencies (adding backports to access more recent apt updates)
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing APT base system dependencies for $TARGETPLATFORM..." \
-    && echo 'deb https://deb.debian.org/debian bookworm-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list \
-    && mkdir -p /etc/apt/keyrings \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        # 1. packaging dependencies
-        apt-transport-https ca-certificates apt-utils gnupg2 curl wget \
-        # 2. docker and init system dependencies
-        zlib1g-dev dumb-init gosu cron unzip grep ncat \
-        # 3. frivolous CLI helpers to make debugging failed archiving easier
-        # nano iputils-ping dnsutils htop procps jq yq \
-    && rm -rf /var/lib/apt/lists/*
-
-######### Language Environments ####################################
-
-# Install Node environment
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.npm,sharing=locked,id=npm-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing Node $NODE_VERSION environment in $NODE_MODULES..." \
-    && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_VERSION}.x nodistro main" >> /etc/apt/sources.list.d/nodejs.list \
-    && curl -fsSL "https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key" | gpg --dearmor | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        nodejs libatomic1 python3-minimal \
-    && rm -rf /var/lib/apt/lists/* \
-    # Update NPM to latest version
-    && npm i -g npm --cache /root/.npm \
-    # Save version info
-    && ( \
-        which node && node --version \
-        && which npm && npm --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install Python environment
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Setting up Python $PYTHON_VERSION runtime..." \
-    # tell PDM to allow using global system python site packages
-    # && rm /usr/lib/python3*/EXTERNALLY-MANAGED \
-    # create global virtual environment GLOBAL_VENV to use (better than using pip install --global)
-    # && python3 -m venv --system-site-packages --symlinks $GLOBAL_VENV \
-    # && python3 -m venv --system-site-packages $GLOBAL_VENV \
-    # && python3 -m venv $GLOBAL_VENV \
-    # install global dependencies / python build dependencies in GLOBAL_VENV
-    # && pip install --upgrade pip setuptools wheel \
-    # Save version info
-    && ( \
-        which python3 && python3 --version | grep " $PYTHON_VERSION" \
-        && which pip && pip --version \
-        # && which pdm && pdm --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-######### Extractor Dependencies ##################################
-
-# Install apt dependencies
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing APT extractor dependencies globally using apt..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        curl wget git yt-dlp ffmpeg ripgrep \
-        # Packages we have also needed in the past:
-        # youtube-dl wget2 aria2 python3-pyxattr rtmpdump libfribidi-bin mpv \
-    && rm -rf /var/lib/apt/lists/* \
-    # Save version info
-    && ( \
-        which curl && curl --version | head -n1 \
-        && which wget && wget --version 2>&1 | head -n1 \
-        && which yt-dlp && yt-dlp --version 2>&1 | head -n1 \
-        && which git && git --version 2>&1 | head -n1 \
-        && which rg && rg --version 2>&1 | head -n1 \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install chromium browser using playwright
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/ms-playwright,sharing=locked,id=browsers-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing Browser binary dependencies to $PLAYWRIGHT_BROWSERS_PATH..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-khmeros fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
-        # chrome can run without dbus/upower technically, it complains about missing dbus but should run ok anyway
-        # libxss1 dbus dbus-x11 upower \
-    # && service dbus start \
-    && if [[ "$TARGETPLATFORM" == *amd64* || "$TARGETPLATFORM" == *arm64* ]]; then \
-        # install Chromium using playwright
-        pip install playwright \
-        && cp -r /root/.cache/ms-playwright "$PLAYWRIGHT_BROWSERS_PATH" \
-        && playwright install --with-deps chromium \
-        && export CHROME_BINARY="$(python -c 'from playwright.sync_api import sync_playwright; print(sync_playwright().start().chromium.executable_path)')"; \
-    else \
-        # fall back to installing Chromium via apt-get on platforms not supported by playwright (e.g. risc, ARMv7, etc.)
-        # apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        #     chromium \
-        # && export CHROME_BINARY="$(which chromium)"; \
-        echo 'armv7 no longer supported in versions after v0.7.3' \
-        exit 1; \
-    fi \
-    && rm -rf /var/lib/apt/lists/* \
-    && ln -s "$CHROME_BINARY" /usr/bin/chromium-browser \
-    && mkdir -p "/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/" \
-    && chown -R $ARCHIVEBOX_USER "/home/${ARCHIVEBOX_USER}/.config" \
-    && mkdir -p "$PLAYWRIGHT_BROWSERS_PATH" \
-    && chown -R $ARCHIVEBOX_USER "$PLAYWRIGHT_BROWSERS_PATH" \
-    # Save version info
-    && ( \
-        which chromium-browser && /usr/bin/chromium-browser --version || /usr/lib/chromium/chromium --version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-# Install Node dependencies
-WORKDIR "$CODE_DIR"
-COPY --chown=root:root --chmod=755 "package.json" "package-lock.json" "$CODE_DIR"/
-RUN --mount=type=cache,target=/root/.npm,sharing=locked,id=npm-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing NPM extractor dependencies from package.json into $NODE_MODULES..." \
-    && npm ci --prefer-offline --no-audit --cache /root/.npm \
-    && ( \
-        which node && node --version \
-        && which npm && npm version \
-        && echo -e '\n\n' \
-    ) | tee -a /VERSION.txt
-
-######### Build Dependencies ####################################
-
-# Install ArchiveBox Python dependencies
-WORKDIR "$CODE_DIR"
-COPY --chown=root:root --chmod=755 "./pyproject.toml" "requirements.txt" "$CODE_DIR"/
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[+] Installing PIP ArchiveBox dependencies from requirements.txt for ${TARGETPLATFORM}..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-        build-essential \
-        libssl-dev libldap2-dev libsasl2-dev \
-        python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps \
-    # && ln -s "$GLOBAL_VENV" "$APP_VENV" \
-    # && pdm use --venv in-project \
-    # && pdm run python -m ensurepip \
-    # && pdm sync --fail-fast --no-editable --group :all --no-self \
-    # && pdm export -o requirements.txt --without-hashes \
-    # && source $GLOBAL_VENV/bin/activate \
-    && pip install -r requirements.txt \
-    && apt-get purge -y \
-        build-essential \
-    && apt-get autoremove -y \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install ArchiveBox Python package from source
-COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-$TARGETARCH$TARGETVARIANT \
-    echo "[*] Installing PIP ArchiveBox package from $CODE_DIR..." \
-    # && apt-get update -qq \
-    # install C compiler to build deps on platforms that dont have 32-bit wheels available on pypi
-    # && apt-get install -qq -y -t bookworm-backports --no-install-recommends \
-    #     build-essential  \
-    # INSTALL ARCHIVEBOX python package globally from CODE_DIR, with all optional dependencies
-    && pip install -e "$CODE_DIR"[sonic,ldap] \
-    # save docker image size and always remove compilers / build tools after building is complete
-    # && apt-get purge -y build-essential \
-    # && apt-get autoremove -y \
-    && rm -rf /var/lib/apt/lists/*
-
-####################################################
-
-# Setup ArchiveBox runtime config
-WORKDIR "$DATA_DIR"
-ENV IN_DOCKER=True \
-    DISPLAY=novnc:0.0 \
-    CUSTOM_TEMPLATES_DIR=/data/templates \
-    CHROME_USER_DATA_DIR=/data/personas/Default/chromium \
-    GOOGLE_API_KEY=no \
-    GOOGLE_DEFAULT_CLIENT_ID=no \
-    GOOGLE_DEFAULT_CLIENT_SECRET=no \
-    ALLOWED_HOSTS=*
-    ## No need to set explicitly, these values will be autodetected by archivebox in docker:
-    # WGET_BINARY="wget" \
-    # YOUTUBEDL_BINARY="yt-dlp" \
-    # CHROME_BINARY="/usr/bin/chromium-browser" \
-    # USE_SINGLEFILE=True \
-    # SINGLEFILE_BINARY="$NODE_MODULES/.bin/single-file" \
-    # USE_READABILITY=True \
-    # READABILITY_BINARY="$NODE_MODULES/.bin/readability-extractor" \
-    # USE_MERCURY=True \
-    # MERCURY_BINARY="$NODE_MODULES/.bin/postlight-parser"
-
-# Print version for nice docker finish summary
-RUN (echo -e "\n\n[√] Finished Docker build succesfully. Saving build summary in: /VERSION.txt" \
-    && echo -e "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})\n" \
-    && echo -e "BUILD_END_TIME=$(date +"%Y-%m-%d %H:%M:%S %s")\n\n" \
-    ) | tee -a /VERSION.txt
-RUN "$CODE_DIR"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt
-
-####################################################
-
-# Open up the interfaces to the outside world
-WORKDIR "$DATA_DIR"
-VOLUME "$DATA_DIR"
-EXPOSE 8000
-
-HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
-    CMD curl --silent 'http://localhost:8000/health/' | grep -q 'OK'
-
-ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
-CMD ["archivebox", "server", "--quick-init", "0.0.0.0:8000"]

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 Nick Sweeting
+Copyright (c) 2017 Nick Sweeting
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

SECURITY.md

@@ -1,34 +0,0 @@
-# Security Policy
-
----
-
-## Security Information
-
-Please see this wiki page for important notices about ArchiveBox security, publishing your archives securely, and the dangers of executing archived JS:
-
-https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
-
-Also see this section of the README about important caveats when running ArchiveBox:
-
-https://github.com/ArchiveBox/ArchiveBox?tab=readme-ov-file#caveats
-
-You can also read these pages for more information about ArchiveBox's internals, development environment, DB schema, and more:
-
-- https://github.com/ArchiveBox/ArchiveBox#archive-layout
-- https://github.com/ArchiveBox/ArchiveBox#archivebox-development
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting
-
----
-
-## Reporting a Vulnerability
-
-We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.
-
-1. Go to the Security tab on our Github repo: https://github.com/ArchiveBox/ArchiveBox/security
-
-2. Click the ["Report a Vulnerability"](https://github.com/ArchiveBox/ArchiveBox/security/advisories/new) button
-
-3. Fill out the form to submit the details of the report and it will be securely sent to the maintainers
-
-You can also contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp).

archive.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# Bookmark Archiver
+# Nick Sweeting 2017 | MIT License
+# https://github.com/pirate/bookmark-archiver
+
+import sys
+
+from datetime import datetime
+
+from parse import parse_links
+from links import validate_links
+from archive_methods import archive_links, _RESULTS_TOTALS
+from index import (
+    write_links_index,
+    write_link_index,
+    parse_json_links_index,
+    parse_json_link_index,
+)
+from config import (
+    ARCHIVE_PERMISSIONS,
+    HTML_FOLDER,
+    ARCHIVE_FOLDER,
+    ANSI,
+    TIMEOUT,
+)
+from util import (
+    download_url,
+    check_dependencies,
+    progress,
+    cleanup_archive,
+)
+
+__DESCRIPTION__ = 'Bookmark Archiver: Create a browsable html archive of a list of links.'
+__DOCUMENTATION__ = 'https://github.com/pirate/bookmark-archiver'
+
+
+def update_archive(export_path, links, resume=None, append=True):
+    """update or create index.html+json given a path to an export file containing new links"""
+
+    start_ts = datetime.now().timestamp()
+
+    # loop over links and archive them
+    archive_links(ARCHIVE_FOLDER, links, export_path, resume=resume)
+
+    # print timing information & summary
+    end_ts = datetime.now().timestamp()
+    seconds = round(end_ts - start_ts, 1)
+    duration = '{} min'.format(seconds / 60) if seconds > 60 else '{} sec'.format(seconds)
+    print('{}[√] [{}] Archive update complete ({}){}'.format(
+        ANSI['green'],
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        duration,
+        ANSI['reset'],
+    ))
+    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
+    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
+    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
+
+
+def update_index(export_path, resume=None, append=True):
+    """handling parsing new links into the json index, returns a set of clean links"""
+
+    # parse an validate the export file
+    new_links = validate_links(parse_links(export_path))
+
+    # load existing links if archive folder is present
+    existing_links = []
+    if append:
+        existing_links = parse_json_links_index(HTML_FOLDER)
+        links = validate_links(existing_links + new_links)
+        
+
+    # merge existing links and new links
+    num_new_links = len(links) - len(existing_links)
+    print('[*] [{}] Adding {} new links from {} to index'.format(
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        num_new_links,
+        export_path,
+    ))
+
+    # write link index html & json
+    write_links_index(HTML_FOLDER, links)
+
+    return links
+
+
+if __name__ == '__main__':
+    argc = len(sys.argv)
+
+    if argc < 2 or sys.argv[1] in ('-h', '--help', 'help'):
+        print(__DESCRIPTION__)
+        print("Documentation:     {}".format(__DOCUMENTATION__))
+        print("")
+        print("Usage:")
+        print("    ./archive.py ~/Downloads/bookmarks_export.html")
+        print("")
+        raise SystemExit(0)
+
+    export_path = sys.argv[1]                        # path to export file
+    resume_from = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
+
+    if any(export_path.startswith(s) for s in ('http://', 'https://', 'ftp://')):
+        export_path = download_url(export_path)
+
+    links = update_index(export_path, resume=resume_from, append=True)
+
+    # make sure folder structure is sane
+    cleanup_archive(ARCHIVE_FOLDER, links)
+    update_archive(export_path, links, resume=resume_from, append=True)

archive_methods.py

@@ -0,0 +1,427 @@
+import os
+
+from functools import wraps
+from datetime import datetime
+from subprocess import run, PIPE, DEVNULL
+
+from index import html_appended_url, parse_json_link_index, write_link_index
+from links import links_after_timestamp
+from config import (
+    ARCHIVE_DIR,
+    CHROME_BINARY,
+    FETCH_WGET,
+    FETCH_WGET_REQUISITES,
+    FETCH_PDF,
+    FETCH_SCREENSHOT,
+    RESOLUTION,
+    SUBMIT_ARCHIVE_DOT_ORG,
+    FETCH_AUDIO,
+    FETCH_VIDEO,
+    FETCH_FAVICON,
+    WGET_USER_AGENT,
+    CHROME_USER_DATA_DIR,
+    TIMEOUT,
+    ANSI,
+)
+from util import (
+    check_dependencies,
+    progress,
+    chmod_file,
+)
+
+
+_RESULTS_TOTALS = {   # globals are bad, mmkay
+    'skipped': 0,
+    'succeded': 0,
+    'failed': 0,
+}
+
+def archive_links(out_dir, links, export_path, resume=None):
+    check_dependencies()
+
+    to_archive = links_after_timestamp(links, resume)
+    try:
+        for idx, link in enumerate(to_archive):
+            link_dir = os.path.join(out_dir, link['timestamp'])
+            archive_link(link_dir, link)
+    
+    except (KeyboardInterrupt, SystemExit, Exception) as e:
+        print('{red}[X] Archive update stopped on #{idx} out of {total} links{reset}'.format(
+            **ANSI,
+            idx=idx,
+            total=len(list(to_archive)),
+        ))
+        print('    Continue where you left off by running:')
+        print('       ./archive.py {} {}'.format(
+            export_path,
+            link['timestamp'],
+        ))
+        if not isinstance(e, KeyboardInterrupt):
+            raise e
+        raise SystemExit(1)
+
+
+def archive_link(out_dir, link, overwrite=False):
+    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+
+    link = {**parse_json_link_index(out_dir), **link}
+    log_link_archive(out_dir, link)
+
+    if FETCH_WGET:
+        link = fetch_wget(out_dir, link, overwrite=overwrite)
+
+    if FETCH_PDF:
+        link = fetch_pdf(out_dir, link, overwrite=overwrite)
+
+    if FETCH_SCREENSHOT:
+        link = fetch_screenshot(out_dir, link, overwrite=overwrite)
+
+    if SUBMIT_ARCHIVE_DOT_ORG:
+        link = archive_dot_org(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_AUDIO:
+    #     link = fetch_audio(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_VIDEO:
+    #     link = fetch_video(out_dir, link, overwrite=overwrite)
+
+    if FETCH_FAVICON:
+        link = fetch_favicon(out_dir, link, overwrite=overwrite)
+
+    write_link_index(out_dir, link)
+    
+    return link
+
+def log_link_archive(out_dir, link):
+    update_existing = os.path.exists(out_dir)
+    if not update_existing:
+        os.makedirs(out_dir)
+    
+    print('[{symbol_color}{symbol}{reset}] [{timestamp}] "{title}": {blue}{base_url}{reset}'.format(
+        symbol='*' if update_existing else '+',
+        symbol_color=ANSI['black' if update_existing else 'green'],
+        **link,
+        **ANSI,
+    ))
+    if link['type']:
+        print('    i Type: {}'.format(link['type']))
+
+    print('    {} ({})'.format(out_dir, 'updating' if update_existing else 'creating'))
+
+
+
+def attach_result_to_link(method):
+    """
+    Instead of returning a result={output:'...', status:'success'} object,
+    attach that result to the links's history & latest fields, then return
+    the updated link object.
+    """
+    def decorator(fetch_func):
+        @wraps(fetch_func)
+        def timed_fetch_func(out_dir, link, overwrite=False, **kwargs):
+            # initialize methods and history json field on link
+            link['latest'] = link.get('latest') or {}
+            link['latest'][method] = link['latest'].get(method) or None
+            link['history'] = link.get('history') or {}
+            link['history'][method] = link['history'].get(method) or []
+
+            start_ts = datetime.now().timestamp()
+
+            # if a valid method output is already present, dont run the fetch function
+            if link['latest'][method] and not overwrite:
+                print('    √ Skipping: {}'.format(method))
+                result = None
+            else:
+                print('    - Fetching: {}'.format(method))
+                result = fetch_func(out_dir, link, **kwargs)
+
+            end_ts = datetime.now().timestamp()
+            duration = str(end_ts * 1000 - start_ts * 1000).split('.')[0]
+
+            # append a history item recording fail/success
+            history_entry = {
+                'timestamp': str(start_ts).split('.')[0],
+            }
+            if result is None:
+                history_entry['status'] = 'skipped'
+            elif isinstance(result.get('output'), Exception):
+                history_entry['status'] = 'failed'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+            else:
+                history_entry['status'] = 'succeded'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+                link['latest'][method] = result['output']
+            
+            _RESULTS_TOTALS[history_entry['status']] += 1
+            
+            return link
+        return timed_fetch_func
+    return decorator
+
+
+@attach_result_to_link('wget')
+def fetch_wget(out_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
+    """download full site using wget"""
+
+    if os.path.exists(os.path.join(out_dir, link['domain'])):
+        return {'output': html_appended_url(link), 'status': 'skipped'}
+
+    CMD = [
+        *'wget --timestamping --adjust-extension --no-parent'.split(' '),                # Docs: https://www.gnu.org/software/wget/manual/wget.html
+        *(('--page-requisites', '--convert-links') if FETCH_WGET_REQUISITES else ()),
+        *(('--user-agent="{}"'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
+        link['url'],
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # index.html
+        end()
+        output = html_appended_url(link)
+        if result.returncode > 0:
+            print('       got wget response code {}:'.format(result.returncode))
+            print('\n'.join('         ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
+            # raise Exception('Failed to wget download')
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('pdf')
+def fetch_pdf(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
+    """print PDF of site to file using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'output.pdf')):
+        return {'output': 'output.pdf', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --print-to-pdf'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # output.pdf
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to print PDF')
+        output = 'output.pdf'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('screenshot')
+def fetch_screenshot(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR, resolution=RESOLUTION):
+    """take screenshot of site using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'screenshot.png')):
+        return {'output': 'screenshot.png', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --screenshot'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        '--window-size={}'.format(resolution),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # sreenshot.png
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to take screenshot')
+        chmod_file('screenshot.png', cwd=out_dir)
+        output = 'screenshot.png'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+    
+
+@attach_result_to_link('archive_org')
+def archive_dot_org(out_dir, link, timeout=TIMEOUT):
+    """submit site to archive.org for archiving via their service, save returned archive url"""
+
+    path = os.path.join(out_dir, 'archive.org.txt')
+    if os.path.exists(path):
+        archive_org_url = open(path, 'r').read().strip()
+        return {'output': archive_org_url, 'status': 'skipped'}
+
+    submit_url = 'https://web.archive.org/save/{}'.format(link['url'].split('?', 1)[0])
+
+    success = False
+    CMD = ['curl', '-I', submit_url]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # archive.org.txt
+        end()
+
+        # Parse archive.org response headers
+        headers = result.stdout.splitlines()
+        content_location = [h for h in headers if b'Content-Location: ' in h]
+        errors = [h for h in headers if h and b'X-Archive-Wayback-Runtime-Error: ' in h]
+
+        if content_location:
+            archive_path = content_location[0].split(b'Content-Location: ', 1)[-1].decode('utf-8')
+            saved_url = 'https://web.archive.org{}'.format(archive_path)
+            success = True
+
+        elif len(errors) == 1 and b'RobotAccessControlException' in errors[0]:
+            output = submit_url
+            # raise Exception('Archive.org denied by {}/robots.txt'.format(link['domain']))
+        elif errors:
+            raise Exception(', '.join(e.decode() for e in errors))
+        else:
+            raise Exception('Failed to find "Content-Location" URL header in Archive.org response.')
+    except Exception as e:
+        end()
+        print('       Visit url to see output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    if success:
+        with open(os.path.join(out_dir, 'archive.org.txt'), 'w', encoding='utf-8') as f:
+            f.write(saved_url)
+        chmod_file('archive.org.txt', cwd=out_dir)
+        output = saved_url
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('favicon')
+def fetch_favicon(out_dir, link, timeout=TIMEOUT):
+    """download site favicon from google's favicon api"""
+
+    if os.path.exists(os.path.join(out_dir, 'favicon.ico')):
+        return {'output': 'favicon.ico', 'status': 'skipped'}
+
+    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
+    fout = open('{}/favicon.ico'.format(out_dir), 'w')
+    end = progress(timeout, prefix='      ')
+    try:
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # favicon.ico
+        fout.close()
+        end()
+        chmod_file('favicon.ico', cwd=out_dir)
+        output = 'favicon.ico'
+    except Exception as e:
+        fout.close()
+        end()
+        print('       Run to see full output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+# @attach_result_to_link('audio')
+# def fetch_audio(out_dir, link, timeout=TIMEOUT):
+#     """Download audio rip using youtube-dl"""
+
+#     if link['type'] not in ('soundcloud',)\
+#        and 'audio' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'audio')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading audio')
+#         CMD = [
+#             "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # audio/audio.mp3
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download audio')
+#             chmod_file('audio.mp3', cwd=out_dir)
+#             return 'audio.mp3'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping audio download')
+
+# @attach_result_to_link('video')
+# def fetch_video(out_dir, link, timeout=TIMEOUT):
+#     """Download video rip using youtube-dl"""
+
+#     if link['type'] not in ('youtube', 'youku', 'vimeo')\
+#        and 'video' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'video')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading video')
+#         CMD = [
+#             "youtube-dl -x --video-format mp4 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # video/movie.mp4
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download video')
+#             chmod_file('video.mp4', cwd=out_dir)
+#             return 'video.mp4'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping video download')
+
+
+def chrome_data_dir_args(user_data_dir=CHROME_USER_DATA_DIR):
+    default = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
+    if user_data_dir:
+        return ('--user-data-dir={}'.format(user_data_dir),)
+    elif os.path.exists(default):
+        return ('--user-data-dir={}'.format(default),)
+    return ()

archivebox/.flake8

@@ -1,6 +0,0 @@
-[flake8]
-ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391
-select = F,E9,W
-max-line-length = 130
-max-complexity = 10
-exclude = migrations,tests,node_modules,vendor,venv,.venv,.venv2,.docker-venv

archivebox/LICENSE

@@ -1 +0,0 @@
-../LICENSE

archivebox/README.md

@@ -1 +0,0 @@
-../README.md

archivebox/__init__.py

@@ -1 +0,0 @@
-__package__ = 'archivebox'

archivebox/__main__.py

@@ -1,11 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox'
-
-import sys
-
-from .cli import main
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/api/apps.py

@@ -1,5 +0,0 @@
-from django.apps import AppConfig
-
-
-class APIConfig(AppConfig):
-    name = 'api'

archivebox/api/archive.py

@@ -1,184 +0,0 @@
-# archivebox_api.py
-from typing import List, Optional
-from enum import Enum
-from pydantic import BaseModel
-from ninja import Router
-from main import (
-    add,
-    remove,
-    update,
-    list_all,
-    ONLY_NEW,
-)  # Assuming these functions are defined in main.py
-
-
-# Schemas
-
-class StatusChoices(str, Enum):
-    indexed = 'indexed'
-    archived = 'archived'
-    unarchived = 'unarchived'
-    present = 'present'
-    valid = 'valid'
-    invalid = 'invalid'
-    duplicate = 'duplicate'
-    orphaned = 'orphaned'
-    corrupted = 'corrupted'
-    unrecognized = 'unrecognized'
-
-
-class AddURLSchema(BaseModel):
-    urls: List[str]
-    tag: str = ""
-    depth: int = 0
-    update: bool = not ONLY_NEW  # Default to the opposite of ONLY_NEW
-    update_all: bool = False
-    index_only: bool = False
-    overwrite: bool = False
-    init: bool = False
-    extractors: str = ""
-    parser: str = "auto"
-
-
-class RemoveURLSchema(BaseModel):
-    yes: bool = False
-    delete: bool = False
-    before: Optional[float] = None
-    after: Optional[float] = None
-    filter_type: str = "exact"
-    filter_patterns: Optional[List[str]] = None
-
-
-class UpdateSchema(BaseModel):
-    resume: Optional[float] = None
-    only_new: Optional[bool] = None
-    index_only: Optional[bool] = False
-    overwrite: Optional[bool] = False
-    before: Optional[float] = None
-    after: Optional[float] = None
-    status: Optional[StatusChoices] = None
-    filter_type: Optional[str] = 'exact'
-    filter_patterns: Optional[List[str]] = None
-    extractors: Optional[str] = ""
-
-
-class ListAllSchema(BaseModel):
-    filter_patterns: Optional[List[str]] = None
-    filter_type: str = 'exact'
-    status: Optional[StatusChoices] = None
-    after: Optional[float] = None
-    before: Optional[float] = None
-    sort: Optional[str] = None
-    csv: Optional[str] = None
-    json: bool = False
-    html: bool = False
-    with_headers: bool = False
-
-
-# API Router
-router = Router()
-
-
-@router.post("/add", response={200: dict})
-def api_add(request, payload: AddURLSchema):
-    try:
-        result = add(
-            urls=payload.urls,
-            tag=payload.tag,
-            depth=payload.depth,
-            update=payload.update,
-            update_all=payload.update_all,
-            index_only=payload.index_only,
-            overwrite=payload.overwrite,
-            init=payload.init,
-            extractors=payload.extractors,
-            parser=payload.parser,
-        )
-        # Currently the add function returns a list of ALL items in the DB, ideally only return new items
-        return {
-            "status": "success",
-            "message": "URLs added successfully.",
-            "result": str(result),
-        }
-    except Exception as e:
-        # Handle exceptions raised by the add function or during processing
-        return {"status": "error", "message": str(e)}
-
-
-@router.post("/remove", response={200: dict})
-def api_remove(request, payload: RemoveURLSchema):
-    try:
-        result = remove(
-            yes=payload.yes,
-            delete=payload.delete,
-            before=payload.before,
-            after=payload.after,
-            filter_type=payload.filter_type,
-            filter_patterns=payload.filter_patterns,
-        )
-        return {
-            "status": "success",
-            "message": "URLs removed successfully.",
-            "result": result,
-        }
-    except Exception as e:
-        # Handle exceptions raised by the remove function or during processing
-        return {"status": "error", "message": str(e)}
-
-
-@router.post("/update", response={200: dict})
-def api_update(request, payload: UpdateSchema):
-    try:
-        result = update(
-            resume=payload.resume,
-            only_new=payload.only_new,
-            index_only=payload.index_only,
-            overwrite=payload.overwrite,
-            before=payload.before,
-            after=payload.after,
-            status=payload.status,
-            filter_type=payload.filter_type,
-            filter_patterns=payload.filter_patterns,
-            extractors=payload.extractors,
-        )
-        return {
-            "status": "success",
-            "message": "Archive updated successfully.",
-            "result": result,
-        }
-    except Exception as e:
-        # Handle exceptions raised by the update function or during processing
-        return {"status": "error", "message": str(e)}
-
-
-@router.post("/list_all", response={200: dict})
-def api_list_all(request, payload: ListAllSchema):
-    try:
-        result = list_all(
-            filter_patterns=payload.filter_patterns,
-            filter_type=payload.filter_type,
-            status=payload.status,
-            after=payload.after,
-            before=payload.before,
-            sort=payload.sort,
-            csv=payload.csv,
-            json=payload.json,
-            html=payload.html,
-            with_headers=payload.with_headers,
-        )
-        # TODO: This is kind of bad, make the format a choice field
-        if payload.json:
-            return {"status": "success", "format": "json", "data": result}
-        elif payload.html:
-            return {"status": "success", "format": "html", "data": result}
-        elif payload.csv:
-            return {"status": "success", "format": "csv", "data": result}
-        else:
-            return {
-                "status": "success",
-                "message": "List generated successfully.",
-                "data": result,
-            }
-    except Exception as e:
-        # Handle exceptions raised by the list_all function or during processing
-        return {"status": "error", "message": str(e)}

archivebox/api/auth.py

@@ -1,48 +0,0 @@
-from django.contrib.auth import authenticate
-from ninja import Form, Router, Schema
-from ninja.security import HttpBearer
-
-from api.models import Token
-
-router = Router()
-
-
-class GlobalAuth(HttpBearer):
-    def authenticate(self, request, token):
-        try:
-            return Token.objects.get(token=token).user
-        except Token.DoesNotExist:
-            pass
-
-
-class AuthSchema(Schema):
-    email: str
-    password: str
-
-
-@router.post("/authenticate", auth=None)  # overriding global auth
-def get_token(request, auth_data: AuthSchema):
-    user = authenticate(username=auth_data.email, password=auth_data.password)
-    if user:
-        # Assuming a user can have multiple tokens and you want to create a new one every time
-        new_token = Token.objects.create(user=user)
-        return {"token": new_token.token, "expires": new_token.expiry_as_iso8601}
-    else:
-        return {"error": "Invalid credentials"}
-
-
-class TokenValidationSchema(Schema):
-    token: str
-
-
-@router.post("/validate_token", auth=None) # No authentication required for this endpoint
-def validate_token(request, token_data: TokenValidationSchema):
-    try:
-        # Attempt to authenticate using the provided token
-        user = GlobalAuth().authenticate(request, token_data.token)
-        if user:
-            return {"status": "valid"}
-        else:
-            return {"status": "invalid"}
-    except Token.DoesNotExist:
-        return {"status": "invalid"}

archivebox/api/migrations/0001_initial.py

@@ -1,28 +0,0 @@
-# Generated by Django 3.1.14 on 2024-04-09 18:52
-
-import api.models
-from django.conf import settings
-from django.db import migrations, models
-import django.db.models.deletion
-
-
-class Migration(migrations.Migration):
-
-    initial = True
-
-    dependencies = [
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='Token',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('token', models.CharField(default=auth.models.hex_uuid, max_length=32, unique=True)),
-                ('created', models.DateTimeField(auto_now_add=True)),
-                ('expiry', models.DateTimeField(blank=True, null=True)),
-                ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='tokens', to=settings.AUTH_USER_MODEL)),
-            ],
-        ),
-    ]

archivebox/api/models.py

@@ -1,30 +0,0 @@
-import uuid
-from datetime import timedelta
-
-from django.conf import settings
-from django.db import models
-from django.utils import timezone
-from django.utils.translation import gettext_lazy as _
-
-def hex_uuid():
-    return uuid.uuid4().hex
-
-
-class Token(models.Model):
-    user = models.ForeignKey(
-        settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name="tokens"
-    )
-    token = models.CharField(max_length=32, default=hex_uuid, unique=True)
-    created = models.DateTimeField(auto_now_add=True)
-    expiry = models.DateTimeField(null=True, blank=True)
-
-    @property
-    def expiry_as_iso8601(self):
-        """Returns the expiry date of the token in ISO 8601 format or a date 100 years in the future if none."""
-        expiry_date = (
-            self.expiry if self.expiry else timezone.now() + timedelta(days=365 * 100)
-        )
-        return expiry_date.isoformat()
-
-    def __str__(self):
-        return self.token

archivebox/api/tests.py

@@ -1,27 +0,0 @@
-from django.test import TestCase
-from ninja.testing import TestClient
-from archivebox.api.archive import router as archive_router
-
-class ArchiveBoxAPITestCase(TestCase):
-    def setUp(self):
-        self.client = TestClient(archive_router)
-
-    def test_add_endpoint(self):
-        response = self.client.post("/add", json={"urls": ["http://example.com"], "tag": "test"})
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.json()["status"], "success")
-
-    def test_remove_endpoint(self):
-        response = self.client.post("/remove", json={"filter_patterns": ["http://example.com"]})
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.json()["status"], "success")
-
-    def test_update_endpoint(self):
-        response = self.client.post("/update", json={})
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.json()["status"], "success")
-
-    def test_list_all_endpoint(self):
-        response = self.client.post("/list_all", json={})
-        self.assertEqual(response.status_code, 200)
-        self.assertTrue("success" in response.json()["status"])

archivebox/cli/__init__.py

@@ -1,155 +0,0 @@
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox'
-
-import os
-import sys
-import argparse
-
-from typing import Optional, Dict, List, IO, Union
-from pathlib import Path
-
-from ..config import OUTPUT_DIR, check_data_folder, check_migrations
-
-from importlib import import_module
-
-CLI_DIR = Path(__file__).resolve().parent
-
-# these common commands will appear sorted before any others for ease-of-use
-meta_cmds = ('help', 'version')                               # dont require valid data folder at all
-main_cmds = ('init', 'config', 'setup')                       # dont require existing db present
-archive_cmds = ('add', 'remove', 'update', 'list', 'status')  # require existing db present
-fake_db = ("oneshot",)                                        # use fake in-memory db
-
-display_first = (*meta_cmds, *main_cmds, *archive_cmds)
-
-# every imported command module must have these properties in order to be valid
-required_attrs = ('__package__', '__command__', 'main')
-
-# basic checks to make sure imported files are valid subcommands
-is_cli_module = lambda fname: fname.startswith('archivebox_') and fname.endswith('.py')
-is_valid_cli_module = lambda module, subcommand: (
-    all(hasattr(module, attr) for attr in required_attrs)
-    and module.__command__.split(' ')[-1] == subcommand
-)
-
-
-def list_subcommands() -> Dict[str, str]:
-    """find and import all valid archivebox_<subcommand>.py files in CLI_DIR"""
-
-    COMMANDS = []
-    for filename in os.listdir(CLI_DIR):
-        if is_cli_module(filename):
-            subcommand = filename.replace('archivebox_', '').replace('.py', '')
-            module = import_module('.archivebox_{}'.format(subcommand), __package__)
-            assert is_valid_cli_module(module, subcommand)
-            COMMANDS.append((subcommand, module.main.__doc__))
-            globals()[subcommand] = module.main
-
-    display_order = lambda cmd: (
-        display_first.index(cmd[0])
-        if cmd[0] in display_first else
-        100 + len(cmd[0])
-    )
-
-    return dict(sorted(COMMANDS, key=display_order))
-
-
-def run_subcommand(subcommand: str,
-                   subcommand_args: List[str]=None,
-                   stdin: Optional[IO]=None,
-                   pwd: Union[Path, str, None]=None) -> None:
-    """Run a given ArchiveBox subcommand with the given list of args"""
-
-    subcommand_args = subcommand_args or []
-
-    if subcommand not in meta_cmds:
-        from ..config import setup_django
-
-        cmd_requires_db = subcommand in archive_cmds
-        init_pending = '--init' in subcommand_args or '--quick-init' in subcommand_args
-
-        if cmd_requires_db:
-            check_data_folder(pwd)
-
-        setup_django(in_memory_db=subcommand in fake_db, check_db=cmd_requires_db and not init_pending)
-
-        if cmd_requires_db:
-            check_migrations()
-
-    module = import_module('.archivebox_{}'.format(subcommand), __package__)
-    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
-
-
-SUBCOMMANDS = list_subcommands()
-
-class NotProvided:
-    pass
-
-
-def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided, pwd: Optional[str]=None) -> None:
-    args = sys.argv[1:] if args is NotProvided else args
-    stdin = sys.stdin if stdin is NotProvided else stdin
-
-    subcommands = list_subcommands()
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description='ArchiveBox: The self-hosted internet archive',
-        add_help=False,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--help', '-h',
-        action='store_true',
-        help=subcommands['help'],
-    )
-    group.add_argument(
-        '--version',
-        action='store_true',
-        help=subcommands['version'],
-    )
-    group.add_argument(
-        "subcommand",
-        type=str,
-        help= "The name of the subcommand to run",
-        nargs='?',
-        choices=subcommands.keys(),
-        default=None,
-    )
-    parser.add_argument(
-        "subcommand_args",
-        help="Arguments for the subcommand",
-        nargs=argparse.REMAINDER,
-    )
-    command = parser.parse_args(args or ())
-
-    if command.version:
-        command.subcommand = 'version'
-    elif command.help or command.subcommand is None:
-        command.subcommand = 'help'
-
-    if command.subcommand not in ('help', 'version', 'status'):
-        from ..logging_util import log_cli_command
-
-        log_cli_command(
-            subcommand=command.subcommand,
-            subcommand_args=command.subcommand_args,
-            stdin=stdin,
-            pwd=pwd or OUTPUT_DIR
-        )
-
-    run_subcommand(
-        subcommand=command.subcommand,
-        subcommand_args=command.subcommand_args,
-        stdin=stdin,
-        pwd=pwd or OUTPUT_DIR,
-    )
-
-
-__all__ = (
-    'SUBCOMMANDS',
-    'list_subcommands',
-    'run_subcommand',
-    *SUBCOMMANDS.keys(),
-)
-
-

archivebox/cli/archivebox_add.py

@@ -1,140 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox add'
-
-import sys
-import argparse
-
-from typing import List, Optional, IO
-
-from ..main import add
-from ..util import docstring
-from ..parsers import PARSERS
-from ..config import OUTPUT_DIR, ONLY_NEW
-from ..logging_util import SmartFormatter, accept_stdin, stderr
-
-
-@docstring(add.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=add.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--tag', '-t',
-        type=str,
-        default='',
-        help="Tag the added URLs with the provided tags e.g. --tag=tag1,tag2,tag3",
-    )
-    parser.add_argument(
-        '--update', #'-u',
-        action='store_true',
-        default=not ONLY_NEW,  # when ONLY_NEW=True we skip updating old links
-        help="Also retry previously skipped/failed links when adding new links",
-    )
-    parser.add_argument(
-        '--update-all', #'-n',
-        action='store_true',
-        default=False, 
-        help="Also update ALL links in index when finished adding new links",
-    )
-    parser.add_argument(
-        '--index-only', #'-o',
-        action='store_true',
-        help="Add the links to the main index without archiving them",
-    )
-    parser.add_argument(
-        'urls',
-        nargs='*',
-        type=str,
-        default=None,
-        help=(
-            'URLs or paths to archive e.g.:\n'
-            '    https://getpocket.com/users/USERNAME/feed/all\n'
-            '    https://example.com/some/rss/feed.xml\n'
-            '    https://example.com\n'
-            '    ~/Downloads/firefox_bookmarks_export.html\n'
-            '    ~/Desktop/sites_list.csv\n'
-        )
-    )
-    parser.add_argument(
-        "--depth",
-        action="store",
-        default=0,
-        choices=[0, 1],
-        type=int,
-        help="Recursively archive all linked pages up to this many hops away"
-    )
-    parser.add_argument(
-        "--overwrite",
-        default=False,
-        action="store_true",
-        help="Re-archive URLs from scratch, overwriting any existing files"
-    )
-    parser.add_argument(
-        "--init", #'-i',
-        action='store_true',
-        help="Init/upgrade the curent data directory before adding",
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    parser.add_argument(
-        "--parser",
-        type=str,
-        help="Parser used to read inputted URLs.",
-        default="auto",
-        choices=["auto", *PARSERS.keys()],
-    )
-    command = parser.parse_args(args or ())
-    urls = command.urls
-
-    stdin_urls = ''
-    if not urls:
-        stdin_urls = accept_stdin(stdin)
-
-    if (stdin_urls and urls) or (not stdin and not urls):
-        stderr(
-            '[X] You must pass URLs/paths to add via stdin or CLI arguments.\n',
-            color='red',
-        )
-        raise SystemExit(2)
-    add(
-        urls=stdin_urls or urls,
-        depth=command.depth,
-        tag=command.tag,
-        update=command.update,
-        update_all=command.update_all,
-        index_only=command.index_only,
-        overwrite=command.overwrite,
-        init=command.init,
-        extractors=command.extract,
-        parser=command.parser,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)
-
-
-# TODO: Implement these
-#
-# parser.add_argument(
-#     '--mirror', #'-m',
-#     action='store_true',
-#     help='Archive an entire site (finding all linked pages below it on the same domain)',
-# )
-# parser.add_argument(
-#     '--crawler', #'-r',
-#     choices=('depth_first', 'breadth_first'),
-#     help='Controls which crawler to use in order to find outlinks in a given page',
-#     default=None,
-# )

archivebox/cli/archivebox_config.py

@@ -1,64 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox config'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import config
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(config.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=config.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--get', #'-g',
-        action='store_true',
-        help="Get the value for the given config KEYs",
-    )
-    group.add_argument(
-        '--set', #'-s',
-        action='store_true',
-        help="Set the given KEY=VALUE config values",
-    )
-    group.add_argument(
-        '--reset', #'-s',
-        action='store_true',
-        help="Reset the given KEY config values to their defaults",
-    )
-    parser.add_argument(
-        'config_options',
-        nargs='*',
-        type=str,
-        help='KEY or KEY=VALUE formatted config values to get or set',
-    )
-    command = parser.parse_args(args or ())
-
-    config_options_str = ''
-    if not command.config_options:
-        config_options_str = accept_stdin(stdin)
-
-    config(
-        config_options_str=config_options_str,
-        config_options=command.config_options,
-        get=command.get,
-        set=command.set,
-        reset=command.reset,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_help.py

@@ -1,32 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox help'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import help
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(help.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=help.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    help(out_dir=pwd or OUTPUT_DIR)
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_init.py

@@ -1,52 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox init'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import init
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(init.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=init.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--force', # '-f',
-        action='store_true',
-        help='Ignore unrecognized files in current directory and initialize anyway',
-    )
-    parser.add_argument(
-        '--quick', '-q',
-        action='store_true',
-        help='Run any updates or migrations without rechecking all snapshot dirs',
-    )
-    parser.add_argument(
-        '--setup', #'-s',
-        action='store_true',
-        help='Automatically install dependencies and extras used for archiving',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    init(
-        force=command.force,
-        quick=command.quick,
-        setup=command.setup,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_list.py

@@ -1,139 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox list'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import list_all
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..index import (
-    LINK_FILTERS,
-    get_indexed_folders,
-    get_archived_folders,
-    get_unarchived_folders,
-    get_present_folders,
-    get_valid_folders,
-    get_invalid_folders,
-    get_duplicate_folders,
-    get_orphaned_folders,
-    get_corrupted_folders,
-    get_unrecognized_folders,
-)
-from ..logging_util import SmartFormatter, reject_stdin, stderr
-
-
-@docstring(list_all.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=list_all.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--csv', #'-c',
-        type=str,
-        help="Print the output in CSV format with the given columns, e.g.: timestamp,url,extension",
-        default=None,
-    )
-    group.add_argument(
-        '--json', #'-j',
-        action='store_true',
-        help="Print the output in JSON format with all columns included",
-    )
-    group.add_argument(
-        '--html',
-        action='store_true',
-        help="Print the output in HTML format"
-    )
-    parser.add_argument(
-        '--with-headers',
-        action='store_true',
-        help='Include the headers in the output document' 
-    )
-    parser.add_argument(
-        '--sort', #'-s',
-        type=str,
-        help="List the links sorted using the given key, e.g. timestamp or updated",
-        default=None,
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="List only links bookmarked before (less than) the given timestamp",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="List only links bookmarked after (greater than or equal to) the given timestamp",
-        default=None,
-    )
-    parser.add_argument(
-        '--status',
-        type=str,
-        choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'),
-        default='indexed',
-        help=(
-            'List only links or data directories that have the given status\n'
-            f'    indexed       {get_indexed_folders.__doc__} (the default)\n'
-            f'    archived      {get_archived_folders.__doc__}\n'
-            f'    unarchived    {get_unarchived_folders.__doc__}\n'
-            '\n'
-            f'    present       {get_present_folders.__doc__}\n'
-            f'    valid         {get_valid_folders.__doc__}\n'
-            f'    invalid       {get_invalid_folders.__doc__}\n'
-            '\n'
-            f'    duplicate     {get_duplicate_folders.__doc__}\n'
-            f'    orphaned      {get_orphaned_folders.__doc__}\n'
-            f'    corrupted     {get_corrupted_folders.__doc__}\n'
-            f'    unrecognized  {get_unrecognized_folders.__doc__}\n'
-        )
-    )
-    parser.add_argument(
-        '--filter-type', '-t',
-        type=str,
-        choices=(*LINK_FILTERS.keys(), 'search'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        default=None,
-        help='List only URLs matching these filter patterns'
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(stdin)
-
-    if command.with_headers and not (command.json or command.html or command.csv):
-        stderr(
-            '[X] --with-headers can only be used with --json, --html or --csv options\n',
-            color='red',
-        )
-        raise SystemExit(2)
-
-    matching_folders = list_all(
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        status=command.status,
-        after=command.after,
-        before=command.before,
-        sort=command.sort,
-        csv=command.csv,
-        json=command.json,
-        html=command.html,
-        with_headers=command.with_headers,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    raise SystemExit(not matching_folders)
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_manage.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox manage'
-
-import sys
-
-from typing import Optional, List, IO
-
-from ..main import manage
-from ..util import docstring
-from ..config import OUTPUT_DIR
-
-
-@docstring(manage.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    manage(
-        args=args,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_oneshot.py

@@ -1,73 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox oneshot'
-
-import sys
-import argparse
-
-from pathlib import Path
-from typing import List, Optional, IO
-
-from ..main import oneshot
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin, stderr
-
-
-@docstring(oneshot.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=oneshot.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        'url',
-        type=str,
-        default=None,
-        help=(
-            'URLs or paths to archive e.g.:\n'
-            '    https://getpocket.com/users/USERNAME/feed/all\n'
-            '    https://example.com/some/rss/feed.xml\n'
-            '    https://example.com\n'
-            '    ~/Downloads/firefox_bookmarks_export.html\n'
-            '    ~/Desktop/sites_list.csv\n'
-        )
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    parser.add_argument(
-        '--out-dir',
-        type=str,
-        default=OUTPUT_DIR,
-        help= "Path to save the single archive folder to, e.g. ./example.com_archive"
-    )
-    command = parser.parse_args(args or ())
-    stdin_url = None
-    url = command.url
-    if not url:
-        stdin_url = accept_stdin(stdin)
-
-    if (stdin_url and url) or (not stdin and not url):
-        stderr(
-            '[X] You must pass a URL/path to add via stdin or CLI arguments.\n',
-            color='red',
-        )
-        raise SystemExit(2)
-    
-    oneshot(
-        url=stdin_url or url,
-        out_dir=Path(command.out_dir).resolve(),
-        extractors=command.extract,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_remove.py

@@ -1,82 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox remove'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import remove
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(remove.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=remove.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--yes', # '-y',
-        action='store_true',
-        help='Remove links instantly without prompting to confirm.',
-    )
-    parser.add_argument(
-        '--delete', # '-r',
-        action='store_true',
-        help=(
-            "In addition to removing the link from the index, "
-            "also delete its archived content and metadata folder."
-        ),
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="List only URLs bookmarked before the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="List only URLs bookmarked after the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--filter-type',
-        type=str,
-        choices=('exact', 'substring', 'domain', 'regex','tag'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        help='URLs matching this filter pattern will be removed from the index.'
-    )
-    command = parser.parse_args(args or ())
-    
-    filter_str = None
-    if not command.filter_patterns:
-        filter_str = accept_stdin(stdin)
-
-    remove(
-        filter_str=filter_str,
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        before=command.before,
-        after=command.after,
-        yes=command.yes,
-        delete=command.delete,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_schedule.py

@@ -1,116 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox schedule'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import schedule
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(schedule.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=schedule.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--quiet', '-q',
-        action='store_true',
-        help=("Don't warn about storage space."),
-    )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
-        '--add', # '-a',
-        action='store_true',
-        help='Add a new scheduled ArchiveBox update job to cron',
-    )
-    parser.add_argument(
-        '--every', # '-e',
-        type=str,
-        default=None,
-        help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")',
-    )
-    parser.add_argument(
-        '--tag', '-t',
-        type=str,
-        default='',
-        help="Tag the added URLs with the provided tags e.g. --tag=tag1,tag2,tag3",
-    )
-    parser.add_argument(
-        '--depth', # '-d',
-        type=int,
-        choices=[0, 1],
-        default=0,
-        help='Depth to archive to [0] or 1, see "add" command help for more info',
-    )
-    parser.add_argument(
-        '--overwrite',
-        action='store_true',
-        help='Re-archive any URLs that have been previously archived, overwriting existing Snapshots',
-    )
-    parser.add_argument(
-        '--update',
-        action='store_true',
-        help='Re-pull any URLs that have been previously added, as needed to fill missing ArchiveResults',
-    )
-    group.add_argument(
-        '--clear', # '-c'
-        action='store_true',
-        help=("Stop all ArchiveBox scheduled runs (remove cron jobs)"),
-    )
-    group.add_argument(
-        '--show', # '-s'
-        action='store_true',
-        help=("Print a list of currently active ArchiveBox cron jobs"),
-    )
-    group.add_argument(
-        '--foreground', '-f',
-        action='store_true',
-        help=("Launch ArchiveBox scheduler as a long-running foreground task "
-              "instead of using cron."),
-    )
-    group.add_argument(
-        '--run-all', # '-a',
-        action='store_true',
-        help=("Run all the scheduled jobs once immediately, independent of "
-              "their configured schedules, can be used together with --foreground"),
-    )
-    parser.add_argument(
-        'import_path',
-        nargs='?',
-        type=str,
-        default=None,
-        help=("Check this path and import any new links on every run "
-              "(can be either local file or remote URL)"),
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    schedule(
-        add=command.add,
-        show=command.show,
-        clear=command.clear,
-        foreground=command.foreground,
-        run_all=command.run_all,
-        quiet=command.quiet,
-        every=command.every,
-        tag=command.tag,
-        depth=command.depth,
-        overwrite=command.overwrite,
-        update=command.update,
-        import_path=command.import_path,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_server.py

@@ -1,76 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox server'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import server
-from ..util import docstring
-from ..config import OUTPUT_DIR, BIND_ADDR
-from ..logging_util import SmartFormatter, reject_stdin
-
-@docstring(server.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=server.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        'runserver_args',
-        nargs='*',
-        type=str,
-        default=[BIND_ADDR],
-        help='Arguments to pass to Django runserver'
-    )
-    parser.add_argument(
-        '--reload',
-        action='store_true',
-        help='Enable auto-reloading when code or templates change',
-    )
-    parser.add_argument(
-        '--debug',
-        action='store_true',
-        help='Enable DEBUG=True mode with more verbose errors',
-    )
-    parser.add_argument(
-        '--nothreading',
-        action='store_true',
-        help='Force runserver to run in single-threaded mode',
-    )
-    parser.add_argument(
-        '--init',
-        action='store_true',
-        help='Run a full archivebox init/upgrade before starting the server',
-    )
-    parser.add_argument(
-        '--quick-init', '-i',
-        action='store_true',
-        help='Run quick archivebox init/upgrade before starting the server',
-    )
-    parser.add_argument(
-        '--createsuperuser',
-        action='store_true',
-        help='Run archivebox manage createsuperuser before starting the server',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    server(
-        runserver_args=command.runserver_args + (['--nothreading'] if command.nothreading else []),
-        reload=command.reload,
-        debug=command.debug,
-        init=command.init,
-        quick_init=command.quick_init,
-        createsuperuser=command.createsuperuser,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_setup.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox setup'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import setup
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(setup.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=setup.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    # parser.add_argument(
-    #     '--force', # '-f',
-    #     action='store_true',
-    #     help='Overwrite any existing packages that conflict with the ones ArchiveBox is trying to install',
-    # )
-    command = parser.parse_args(args or ())   # noqa
-    reject_stdin(__command__, stdin)
-
-    setup(
-        # force=command.force,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_shell.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox shell'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import shell
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(shell.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=shell.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    shell(
-        out_dir=pwd or OUTPUT_DIR,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_status.py

@@ -1,32 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox status'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import status
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(status.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=status.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-
-    status(out_dir=pwd or OUTPUT_DIR)
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_update.py

@@ -1,136 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox update'
-
-import sys
-import argparse
-
-from typing import List, Optional, IO
-
-from ..main import update
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..index import (
-    LINK_FILTERS,
-    get_indexed_folders,
-    get_archived_folders,
-    get_unarchived_folders,
-    get_present_folders,
-    get_valid_folders,
-    get_invalid_folders,
-    get_duplicate_folders,
-    get_orphaned_folders,
-    get_corrupted_folders,
-    get_unrecognized_folders,
-)
-from ..logging_util import SmartFormatter, accept_stdin
-
-
-@docstring(update.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=update.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--only-new', #'-n',
-        action='store_true',
-        help="Don't attempt to retry previously skipped/failed links when updating",
-    )
-    parser.add_argument(
-        '--index-only', #'-o',
-        action='store_true',
-        help="Update the main index without archiving any content",
-    )
-    parser.add_argument(
-        '--resume', #'-r',
-        type=float,
-        help='Resume the update process from a given timestamp',
-        default=None,
-    )
-    parser.add_argument(
-        '--overwrite', #'-x',
-        action='store_true',
-        help='Ignore existing archived content and overwrite with new versions (DANGEROUS)',
-    )
-    parser.add_argument(
-        '--before', #'-b',
-        type=float,
-        help="Update only links bookmarked before the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--after', #'-a',
-        type=float,
-        help="Update only links bookmarked after the given timestamp.",
-        default=None,
-    )
-    parser.add_argument(
-        '--status',
-        type=str,
-        choices=('indexed', 'archived', 'unarchived', 'present', 'valid', 'invalid', 'duplicate', 'orphaned', 'corrupted', 'unrecognized'),
-        default='indexed',
-        help=(
-            'Update only links or data directories that have the given status\n'
-            f'    indexed       {get_indexed_folders.__doc__} (the default)\n'
-            f'    archived      {get_archived_folders.__doc__}\n'
-            f'    unarchived    {get_unarchived_folders.__doc__}\n'
-            '\n'
-            f'    present       {get_present_folders.__doc__}\n'
-            f'    valid         {get_valid_folders.__doc__}\n'
-            f'    invalid       {get_invalid_folders.__doc__}\n'
-            '\n'
-            f'    duplicate     {get_duplicate_folders.__doc__}\n'
-            f'    orphaned      {get_orphaned_folders.__doc__}\n'
-            f'    corrupted     {get_corrupted_folders.__doc__}\n'
-            f'    unrecognized  {get_unrecognized_folders.__doc__}\n'
-        )
-    )
-    parser.add_argument(
-        '--filter-type', '-t',
-        type=str,
-        choices=(*LINK_FILTERS.keys(), 'search'),
-        default='exact',
-        help='Type of pattern matching to use when filtering URLs',
-    )
-    parser.add_argument(
-        'filter_patterns',
-        nargs='*',
-        type=str,
-        default=None,
-        help='Update only URLs matching these filter patterns.'
-    )
-    parser.add_argument(
-        "--extract",
-        type=str,
-        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
-              This does not take precedence over the configuration",
-        default=""
-    )
-    command = parser.parse_args(args or ())
-
-    filter_patterns_str = None
-    if not command.filter_patterns:
-        filter_patterns_str = accept_stdin(stdin)
-
-    update(
-        resume=command.resume,
-        only_new=command.only_new,
-        index_only=command.index_only,
-        overwrite=command.overwrite,
-        filter_patterns_str=filter_patterns_str,
-        filter_patterns=command.filter_patterns,
-        filter_type=command.filter_type,
-        status=command.status,
-        after=command.after,
-        before=command.before,
-        out_dir=pwd or OUTPUT_DIR,
-        extractors=command.extract,
-    )
-    
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_version.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-__command__ = 'archivebox version'
-
-import sys
-import argparse
-
-from typing import Optional, List, IO
-
-from ..main import version
-from ..util import docstring
-from ..config import OUTPUT_DIR
-from ..logging_util import SmartFormatter, reject_stdin
-
-
-@docstring(version.__doc__)
-def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
-    parser = argparse.ArgumentParser(
-        prog=__command__,
-        description=version.__doc__,
-        add_help=True,
-        formatter_class=SmartFormatter,
-    )
-    parser.add_argument(
-        '--quiet', '-q',
-        action='store_true',
-        help='Only print ArchiveBox version number and nothing else.',
-    )
-    command = parser.parse_args(args or ())
-    reject_stdin(__command__, stdin)
-    
-    version(
-        quiet=command.quiet,
-        out_dir=pwd or OUTPUT_DIR,
-    )
-
-
-if __name__ == '__main__':
-    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/tests.py

@@ -1,227 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = 'archivebox.cli'
-
-
-import os
-import sys
-import shutil
-import unittest
-from pathlib import Path
-
-from contextlib import contextmanager
-
-TEST_CONFIG = {
-    'USE_COLOR': 'False',
-    'SHOW_PROGRESS': 'False',
-
-    'OUTPUT_DIR': 'data.tests',
-    
-    'SAVE_ARCHIVE_DOT_ORG': 'False',
-    'SAVE_TITLE': 'False',
-    
-    'USE_CURL': 'False',
-    'USE_WGET': 'False',
-    'USE_GIT': 'False',
-    'USE_CHROME': 'False',
-    'USE_YOUTUBEDL': 'False',
-}
-
-OUTPUT_DIR = 'data.tests'
-os.environ.update(TEST_CONFIG)
-
-from ..main import init
-from ..index import load_main_index
-from ..config import (
-    SQL_INDEX_FILENAME,
-    JSON_INDEX_FILENAME,
-    HTML_INDEX_FILENAME,
-)
-
-from . import (
-    archivebox_init,
-    archivebox_add,
-    archivebox_remove,
-)
-
-HIDE_CLI_OUTPUT = True
-
-test_urls = '''
-https://example1.com/what/is/happening.html?what=1#how-about-this=1
-https://example2.com/what/is/happening/?what=1#how-about-this=1
-HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f
-https://example4.com/what/is/happening.html
-https://example5.com/
-https://example6.com
-
-<test>http://example7.com</test>
-[https://example8.com/what/is/this.php?what=1]
-[and http://example9.com?what=1&other=3#and-thing=2]
-<what>https://example10.com#and-thing=2 "</about>
-abc<this["https://subb.example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def
-sdflkf[what](https://subb.example12.com/who/what.php?whoami=1#whatami=2)?am=hi
-example13.bada
-and example14.badb
-<or>htt://example15.badc</that>
-'''
-
-stdout = sys.stdout
-stderr = sys.stderr
-
-
-@contextmanager
-def output_hidden(show_failing=True):
-    if not HIDE_CLI_OUTPUT:
-        yield
-        return
-
-    sys.stdout = open('stdout.txt', 'w+', encoding='utf-8')
-    sys.stderr = open('stderr.txt', 'w+', encoding='utf-8')
-    try:
-        yield
-        sys.stdout.close()
-        sys.stderr.close()
-        sys.stdout = stdout
-        sys.stderr = stderr
-    except Exception:
-        sys.stdout.close()
-        sys.stderr.close()
-        sys.stdout = stdout
-        sys.stderr = stderr
-        if show_failing:
-            with open('stdout.txt', 'r', encoding='utf-8') as f:
-                print(f.read())
-            with open('stderr.txt', 'r', encoding='utf-8') as f:
-                print(f.read())
-        raise
-    finally:
-        os.remove('stdout.txt')
-        os.remove('stderr.txt')
-
-
-class TestInit(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-
-    def tearDown(self):
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-    def test_basic_init(self):
-        with output_hidden():
-            archivebox_init.main([])
-
-        assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
-        assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
-        assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
-        assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0
-
-    def test_conflicting_init(self):
-        with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+', encoding='utf-8') as f:
-            f.write('test')
-
-        try:
-            with output_hidden(show_failing=False):
-                archivebox_init.main([])
-            assert False, 'Init should have exited with an exception'
-        except SystemExit:
-            pass
-
-        assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
-        assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
-        assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
-        try:
-            load_main_index(out_dir=OUTPUT_DIR)
-            assert False, 'load_main_index should raise an exception when no index is present'
-        except Exception:
-            pass
-
-    def test_no_dirty_state(self):
-        with output_hidden():
-            init()
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-        with output_hidden():
-            init()
-
-
-class TestAdd(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-        with output_hidden():
-            init()
-
-    def tearDown(self):
-        shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-    def test_add_arg_url(self):
-        with output_hidden():
-            archivebox_add.main(['https://getpocket.com/users/nikisweeting/feed/all'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 30
-
-    def test_add_arg_file(self):
-        test_file = Path(OUTPUT_DIR) / 'test.txt'
-        with open(test_file, 'w+', encoding='utf') as f:
-            f.write(test_urls)
-
-        with output_hidden():
-            archivebox_add.main([test_file])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 12
-        os.remove(test_file)
-
-    def test_add_stdin_url(self):
-        with output_hidden():
-            archivebox_add.main([], stdin=test_urls)
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 12
-
-
-class TestRemove(unittest.TestCase):
-    def setUp(self):
-        os.makedirs(OUTPUT_DIR, exist_ok=True)
-        with output_hidden():
-            init()
-            archivebox_add.main([], stdin=test_urls)
-
-    # def tearDown(self):
-        # shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
-
-
-    def test_remove_exact(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', 'https://example5.com/'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 11
-
-    def test_remove_regex(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', '--filter-type=regex', r'http(s)?:\/\/(.+\.)?(example\d\.com)'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 4
-
-    def test_remove_domain(self):
-        with output_hidden():
-            archivebox_remove.main(['--yes', '--delete', '--filter-type=domain', 'example5.com', 'example6.com'])
-
-        all_links = load_main_index(out_dir=OUTPUT_DIR)
-        assert len(all_links) == 10
-
-    def test_remove_none(self):
-        try:
-            with output_hidden(show_failing=False):
-                archivebox_remove.main(['--yes', '--delete', 'https://doesntexist.com'])
-            assert False, 'Should raise if no URLs match'
-        except Exception:
-            pass
-
-
-if __name__ == '__main__':
-    if '--verbose' in sys.argv or '-v' in sys.argv:
-        HIDE_CLI_OUTPUT = False
-    
-    unittest.main()

archivebox/config_stubs.py

@@ -1,169 +0,0 @@
-from pathlib import Path
-from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List
-from mypy_extensions import TypedDict
-
-
-
-SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]]
-SimpleConfigValueDict = Dict[str, SimpleConfigValue]
-SimpleConfigValueGetter = Callable[[], SimpleConfigValue]
-ConfigValue = Union[SimpleConfigValue, SimpleConfigValueDict, SimpleConfigValueGetter]
-
-SHArgs = List[str]   # shell command args list e.g. ["--something=1", "--someotherarg"]
-
-
-class BaseConfig(TypedDict):
-    pass
-
-class ConfigDict(BaseConfig, total=False):
-    """
-    # Regenerate by pasting this quine into `archivebox shell` 🥚
-    from archivebox.config import ConfigDict, CONFIG_SCHEMA
-    print('class ConfigDict(BaseConfig, total=False):')
-    print('    ' + '"'*3 + ConfigDict.__doc__ + '"'*3)
-    for section, configs in CONFIG_SCHEMA.items():
-        for key, attrs in configs.items():
-            Type, default = attrs['type'], attrs['default']
-            if default is None:
-                print(f'    {key}: Optional[{Type.__name__}]')
-            else:
-                print(f'    {key}: {Type.__name__}')
-        print()
-    """
-    IS_TTY: bool
-    USE_COLOR: bool
-    SHOW_PROGRESS: bool
-    IN_DOCKER: bool
-    IN_QEMU: bool
-    PUID: int
-    PGID: int
-
-    OUTPUT_DIR: Optional[str]
-    CONFIG_FILE: Optional[str]
-    ONLY_NEW: bool
-    TIMEOUT: int
-    MEDIA_TIMEOUT: int
-    OUTPUT_PERMISSIONS: str
-    RESTRICT_FILE_NAMES: str
-    URL_DENYLIST: str
-    URL_ALLOWLIST: Optional[str]
-    ADMIN_USERNAME: Optional[str]
-    ADMIN_PASSWORD: Optional[str]
-    ENFORCE_ATOMIC_WRITES: bool
-    TAG_SEPARATOR_PATTERN: str
-
-    SECRET_KEY: Optional[str]
-    BIND_ADDR: str
-    ALLOWED_HOSTS: str
-    DEBUG: bool
-    PUBLIC_INDEX: bool
-    PUBLIC_SNAPSHOTS: bool
-    PUBLIC_ADD_VIEW: bool
-    FOOTER_INFO: str
-    SNAPSHOTS_PER_PAGE: int
-    CUSTOM_TEMPLATES_DIR: Optional[str]
-    TIME_ZONE: str
-    TIMEZONE: str
-    REVERSE_PROXY_USER_HEADER: str
-    REVERSE_PROXY_WHITELIST: str
-    LOGOUT_REDIRECT_URL: str
-    PREVIEW_ORIGINALS: bool
-    LDAP: bool
-    LDAP_SERVER_URI: Optional[str]
-    LDAP_BIND_DN: Optional[str]
-    LDAP_BIND_PASSWORD: Optional[str]
-    LDAP_USER_BASE: Optional[str]
-    LDAP_USER_FILTER: Optional[str]
-    LDAP_USERNAME_ATTR: Optional[str]
-    LDAP_FIRSTNAME_ATTR: Optional[str]
-    LDAP_LASTNAME_ATTR: Optional[str]
-    LDAP_EMAIL_ATTR: Optional[str]
-    LDAP_CREATE_SUPERUSER: bool
-
-    SAVE_TITLE: bool
-    SAVE_FAVICON: bool
-    SAVE_WGET: bool
-    SAVE_WGET_REQUISITES: bool
-    SAVE_SINGLEFILE: bool
-    SAVE_READABILITY: bool
-    SAVE_MERCURY: bool
-    SAVE_HTMLTOTEXT: bool
-    SAVE_PDF: bool
-    SAVE_SCREENSHOT: bool
-    SAVE_DOM: bool
-    SAVE_HEADERS: bool
-    SAVE_WARC: bool
-    SAVE_GIT: bool
-    SAVE_MEDIA: bool
-    SAVE_ARCHIVE_DOT_ORG: bool
-    SAVE_ALLOWLIST: dict
-    SAVE_DENYLIST: dict
-
-    RESOLUTION: str
-    GIT_DOMAINS: str
-    CHECK_SSL_VALIDITY: bool
-    MEDIA_MAX_SIZE: str
-    CURL_USER_AGENT: str
-    WGET_USER_AGENT: str
-    CHROME_USER_AGENT: str
-    COOKIES_FILE: Optional[str]
-    CHROME_USER_DATA_DIR: Optional[str]
-    CHROME_TIMEOUT: int
-    CHROME_HEADLESS: bool
-    CHROME_SANDBOX: bool
-    YOUTUBEDL_ARGS: list
-    WGET_ARGS: list
-    CURL_ARGS: list
-    GIT_ARGS: list
-    SINGLEFILE_ARGS: Optional[list]
-    FAVICON_PROVIDER: str
-
-    USE_INDEXING_BACKEND: bool
-    USE_SEARCHING_BACKEND: bool
-    SEARCH_BACKEND_ENGINE: str
-    SEARCH_BACKEND_HOST_NAME: str
-    SEARCH_BACKEND_PORT: int
-    SEARCH_BACKEND_PASSWORD: str
-    SEARCH_PROCESS_HTML: bool
-    SONIC_COLLECTION: str
-    SONIC_BUCKET: str
-    SEARCH_BACKEND_TIMEOUT: int
-    FTS_SEPARATE_DATABASE: bool
-    FTS_TOKENIZERS: str
-    FTS_SQLITE_MAX_LENGTH: int
-
-    USE_CURL: bool
-    USE_WGET: bool
-    USE_SINGLEFILE: bool
-    USE_READABILITY: bool
-    USE_MERCURY: bool
-    USE_GIT: bool
-    USE_CHROME: bool
-    USE_NODE: bool
-    USE_YOUTUBEDL: bool
-    USE_RIPGREP: bool
-    CURL_BINARY: str
-    GIT_BINARY: str
-    WGET_BINARY: str
-    SINGLEFILE_BINARY: str
-    READABILITY_BINARY: str
-    MERCURY_BINARY: str
-    YOUTUBEDL_BINARY: str
-    NODE_BINARY: str
-    RIPGREP_BINARY: str
-    CHROME_BINARY: Optional[str]
-    POCKET_CONSUMER_KEY: Optional[str]
-    POCKET_ACCESS_TOKENS: dict
-    READWISE_READER_TOKENS: dict
-
-
-ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue]
-ConfigDefaultValue = Union[ConfigValue, ConfigDefaultValueGetter]
-
-ConfigDefault = TypedDict('ConfigDefault', {
-    'default': ConfigDefaultValue,
-    'type': Optional[Type],
-    'aliases': Optional[Tuple[str, ...]],
-}, total=False)
-
-ConfigDefaultDict = Dict[str, ConfigDefault]

archivebox/core/__init__.py

@@ -1,3 +0,0 @@
-__package__ = 'archivebox.core'
-
-# default_app_config = 'core.apps.CoreAppConfig'

archivebox/core/admin.py

@@ -1,469 +0,0 @@
-__package__ = 'archivebox.core'
-
-from io import StringIO
-from pathlib import Path
-from contextlib import redirect_stdout
-from datetime import datetime, timezone
-
-from django.contrib import admin
-from django.db.models import Count
-from django.urls import path
-from django.utils.html import format_html
-from django.utils.safestring import mark_safe
-from django.shortcuts import render, redirect
-from django.contrib.auth import get_user_model
-from django.contrib.auth.models import Group, Permission
-from django import forms
-
-from ..util import htmldecode, urldecode, ansi_to_html
-
-from core.models import Snapshot, ArchiveResult, Tag
-from core.forms import AddLinkForm
-
-from core.mixins import SearchResultsAdminMixin
-
-from index.html import snapshot_icons
-from logging_util import printable_filesize
-from main import add, remove
-from extractors import archive_links
-from config import (
-    OUTPUT_DIR,
-    SNAPSHOTS_PER_PAGE,
-    VERSION,
-    VERSIONS_AVAILABLE,
-    CAN_UPGRADE
-)
-
-GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': VERSIONS_AVAILABLE, 'CAN_UPGRADE': CAN_UPGRADE}
-
-# Admin URLs
-# /admin/
-# /admin/login/
-# /admin/core/
-# /admin/core/snapshot/
-# /admin/core/snapshot/:uuid/
-# /admin/core/tag/
-# /admin/core/tag/:uuid/
-
-
-# TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel
-
-
-class ArchiveBoxAdmin(admin.AdminSite):
-    site_header = 'ArchiveBox'
-    index_title = 'Links'
-    site_title = 'Index'
-    namespace = 'admin'
-
-    def get_urls(self):
-        return [
-            path('core/snapshot/add/', self.add_view, name='Add'),
-        ] + super().get_urls()
-
-    def add_view(self, request):
-        if not request.user.is_authenticated:
-            return redirect(f'/admin/login/?next={request.path}')
-
-        request.current_app = self.name
-        context = {
-            **self.each_context(request),
-            'title': 'Add URLs',
-        }
-
-        if request.method == 'GET':
-            context['form'] = AddLinkForm()
-
-        elif request.method == 'POST':
-            form = AddLinkForm(request.POST)
-            if form.is_valid():
-                url = form.cleaned_data["url"]
-                print(f'[+] Adding URL: {url}')
-                depth = 0 if form.cleaned_data["depth"] == "0" else 1
-                input_kwargs = {
-                    "urls": url,
-                    "depth": depth,
-                    "update_all": False,
-                    "out_dir": OUTPUT_DIR,
-                }
-                add_stdout = StringIO()
-                with redirect_stdout(add_stdout):
-                   add(**input_kwargs)
-                print(add_stdout.getvalue())
-
-                context.update({
-                    "stdout": ansi_to_html(add_stdout.getvalue().strip()),
-                    "form": AddLinkForm()
-                })
-            else:
-                context["form"] = form
-
-        return render(template_name='add.html', request=request, context=context)
-
-archivebox_admin = ArchiveBoxAdmin()
-archivebox_admin.register(get_user_model())
-archivebox_admin.disable_action('delete_selected')
-
-class ArchiveResultInline(admin.TabularInline):
-    model = ArchiveResult
-
-class TagInline(admin.TabularInline):
-    model = Snapshot.tags.through
-
-from django.contrib.admin.helpers import ActionForm
-from django.contrib.admin.widgets import AutocompleteSelectMultiple
-
-class AutocompleteTags:
-    model = Tag
-    search_fields = ['name']
-    name = 'tags'
-    remote_field = TagInline
-
-class AutocompleteTagsAdminStub:
-    name = 'admin'
-
-
-class SnapshotActionForm(ActionForm):
-    tags = forms.ModelMultipleChoiceField(
-        queryset=Tag.objects.all(),
-        required=False,
-        widget=AutocompleteSelectMultiple(
-            AutocompleteTags(),
-            AutocompleteTagsAdminStub(),
-        ),
-    )
-
-    # TODO: allow selecting actions for specific extractors? is this useful?
-    # EXTRACTOR_CHOICES = [
-    #     (name, name.title())
-    #     for name, _, _ in get_default_archive_methods()
-    # ]
-    # extractor = forms.ChoiceField(
-    #     choices=EXTRACTOR_CHOICES,
-    #     required=False,
-    #     widget=forms.MultileChoiceField(attrs={'class': "form-control"})
-    # )
-
-
-@admin.register(Snapshot, site=archivebox_admin)
-class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin):
-    list_display = ('added', 'title_str', 'files', 'size', 'url_str')
-    sort_fields = ('title_str', 'url_str', 'added', 'files')
-    readonly_fields = ('info', 'bookmarked', 'added', 'updated')
-    search_fields = ('id', 'url', 'timestamp', 'title', 'tags__name')
-    fields = ('timestamp', 'url', 'title', 'tags', *readonly_fields)
-    list_filter = ('added', 'updated', 'tags', 'archiveresult__status')
-    ordering = ['-added']
-    actions = ['add_tags', 'remove_tags', 'update_titles', 'update_snapshots', 'resnapshot_snapshot', 'overwrite_snapshots', 'delete_snapshots']
-    autocomplete_fields = ['tags']
-    inlines = [ArchiveResultInline]
-    list_per_page = SNAPSHOTS_PER_PAGE
-
-    action_form = SnapshotActionForm
-
-
-    def changelist_view(self, request, extra_context=None):
-        extra_context = extra_context or {}
-        return super().changelist_view(request, extra_context | GLOBAL_CONTEXT)
-
-    def get_urls(self):
-        urls = super().get_urls()
-        custom_urls = [
-            path('grid/', self.admin_site.admin_view(self.grid_view), name='grid')
-        ]
-        return custom_urls + urls
-
-    def get_queryset(self, request):
-        self.request = request
-        return super().get_queryset(request).prefetch_related('tags').annotate(archiveresult_count=Count('archiveresult'))
-
-    def tag_list(self, obj):
-        return ', '.join(obj.tags.values_list('name', flat=True))
-
-    # TODO: figure out a different way to do this, you cant nest forms so this doenst work
-    # def action(self, obj):
-    #     # csrfmiddlewaretoken: Wa8UcQ4fD3FJibzxqHN3IYrrjLo4VguWynmbzzcPYoebfVUnDovon7GEMYFRgsh0
-    #     # action: update_snapshots
-    #     # select_across: 0
-    #     # _selected_action: 76d29b26-2a88-439e-877c-a7cca1b72bb3
-    #     return format_html(
-    #         '''
-    #             <form action="/admin/core/snapshot/" method="post" onsubmit="e => e.stopPropagation()">
-    #                 <input type="hidden" name="csrfmiddlewaretoken" value="{}">
-    #                 <input type="hidden" name="_selected_action" value="{}">
-    #                 <button name="update_snapshots">Check</button>
-    #                 <button name="update_titles">Pull title + favicon</button>
-    #                 <button name="update_snapshots">Update</button>
-    #                 <button name="overwrite_snapshots">Re-Archive (overwrite)</button>
-    #                 <button name="delete_snapshots">Permanently delete</button>
-    #             </form>
-    #         ''',
-    #         csrf.get_token(self.request),
-    #         obj.id,
-    #     )
-
-    def info(self, obj):
-        return format_html(
-            '''
-            UUID: <code style="font-size: 10px; user-select: all">{}</code> &nbsp; &nbsp;
-            Timestamp: <code style="font-size: 10px; user-select: all">{}</code> &nbsp; &nbsp;
-            URL Hash: <code style="font-size: 10px; user-select: all">{}</code><br/>
-            Archived: {} ({} files {}) &nbsp; &nbsp;
-            Favicon: <img src="{}" style="height: 20px"/> &nbsp; &nbsp;
-            Status code: {} &nbsp; &nbsp;
-            Server: {} &nbsp; &nbsp;
-            Content type: {} &nbsp; &nbsp;
-            Extension: {} &nbsp; &nbsp;
-            <br/><br/>
-            <a href="/archive/{}">View Snapshot index ➡️</a> &nbsp; &nbsp;
-            <a href="/admin/core/snapshot/?id__exact={}">View actions ⚙️</a>
-            ''',
-            obj.id,
-            obj.timestamp,
-            obj.url_hash,
-            '✅' if obj.is_archived else '❌',
-            obj.num_outputs,
-            self.size(obj),
-            f'/archive/{obj.timestamp}/favicon.ico',
-            obj.status_code or '?',
-            obj.headers and obj.headers.get('Server') or '?',
-            obj.headers and obj.headers.get('Content-Type') or '?',
-            obj.extension or '?',
-            obj.timestamp,
-            obj.id,
-        )
-
-    @admin.display(
-        description='Title',
-        ordering='title',
-    )
-    def title_str(self, obj):
-        canon = obj.as_link().canonical_outputs()
-        tags = ''.join(
-            format_html('<a href="/admin/core/snapshot/?tags__id__exact={}"><span class="tag">{}</span></a> ', tag.id, tag)
-            for tag in obj.tags.all()
-            if str(tag).strip()
-        )
-        return format_html(
-            '<a href="/{}">'
-                '<img src="/{}/{}" class="favicon" onerror="this.remove()">'
-            '</a>'
-            '<a href="/{}/index.html">'
-                '<b class="status-{}">{}</b>'
-            '</a>',
-            obj.archive_path,
-            obj.archive_path, canon['favicon_path'],
-            obj.archive_path,
-            'fetched' if obj.latest_title or obj.title else 'pending',
-            urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...'
-        ) + mark_safe(f' <span class="tags">{tags}</span>')
-
-    @admin.display(
-        description='Files Saved',
-        ordering='archiveresult_count',
-    )
-    def files(self, obj):
-        return snapshot_icons(obj)
-
-
-    @admin.display(
-        ordering='archiveresult_count'
-    )
-    def size(self, obj):
-        archive_size = (Path(obj.link_dir) / 'index.html').exists() and obj.archive_size
-        if archive_size:
-            size_txt = printable_filesize(archive_size)
-            if archive_size > 52428800:
-                size_txt = mark_safe(f'<b>{size_txt}</b>')
-        else:
-            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
-        return format_html(
-            '<a href="/{}" title="View all files">{}</a>',
-            obj.archive_path,
-            size_txt,
-        )
-
-
-    @admin.display(
-        description='Original URL',
-        ordering='url',
-    )
-    def url_str(self, obj):
-        return format_html(
-            '<a href="{}"><code style="user-select: all;">{}</code></a>',
-            obj.url,
-            obj.url,
-        )
-
-    def grid_view(self, request, extra_context=None):
-
-        # cl = self.get_changelist_instance(request)
-
-        # Save before monkey patching to restore for changelist list view
-        saved_change_list_template = self.change_list_template
-        saved_list_per_page = self.list_per_page
-        saved_list_max_show_all = self.list_max_show_all
-
-        # Monkey patch here plus core_tags.py
-        self.change_list_template = 'private_index_grid.html'
-        self.list_per_page = SNAPSHOTS_PER_PAGE
-        self.list_max_show_all = self.list_per_page
-
-        # Call monkey patched view
-        rendered_response = self.changelist_view(request, extra_context=extra_context)
-
-        # Restore values
-        self.change_list_template = saved_change_list_template
-        self.list_per_page = saved_list_per_page
-        self.list_max_show_all = saved_list_max_show_all
-
-        return rendered_response
-
-    # for debugging, uncomment this to print all requests:
-    # def changelist_view(self, request, extra_context=None):
-    #     print('[*] Got request', request.method, request.POST)
-    #     return super().changelist_view(request, extra_context=None)
-
-    @admin.action(
-        description="Pull"
-    )
-    def update_snapshots(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], out_dir=OUTPUT_DIR)
-
-    @admin.action(
-        description="⬇️ Title"
-    )
-    def update_titles(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR)
-
-    @admin.action(
-        description="Re-Snapshot"
-    )
-    def resnapshot_snapshot(self, request, queryset):
-        for snapshot in queryset:
-            timestamp = datetime.now(timezone.utc).isoformat('T', 'seconds')
-            new_url = snapshot.url.split('#')[0] + f'#{timestamp}'
-            add(new_url, tag=snapshot.tags_str())
-
-    @admin.action(
-        description="Reset"
-    )
-    def overwrite_snapshots(self, request, queryset):
-        archive_links([
-            snapshot.as_link()
-            for snapshot in queryset
-        ], overwrite=True, out_dir=OUTPUT_DIR)
-
-    @admin.action(
-        description="Delete"
-    )
-    def delete_snapshots(self, request, queryset):
-        remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR)
-
-
-    @admin.action(
-        description="+"
-    )
-    def add_tags(self, request, queryset):
-        tags = request.POST.getlist('tags')
-        print('[+] Adding tags', tags, 'to Snapshots', queryset)
-        for obj in queryset:
-            obj.tags.add(*tags)
-
-
-    @admin.action(
-        description="–"
-    )
-    def remove_tags(self, request, queryset):
-        tags = request.POST.getlist('tags')
-        print('[-] Removing tags', tags, 'to Snapshots', queryset)
-        for obj in queryset:
-            obj.tags.remove(*tags)
-
-
-        
-
-
-
-
-
-@admin.register(Tag, site=archivebox_admin)
-class TagAdmin(admin.ModelAdmin):
-    list_display = ('slug', 'name', 'num_snapshots', 'snapshots', 'id')
-    sort_fields = ('id', 'name', 'slug')
-    readonly_fields = ('id', 'num_snapshots', 'snapshots')
-    search_fields = ('id', 'name', 'slug')
-    fields = (*readonly_fields, 'name', 'slug')
-    actions = ['delete_selected']
-    ordering = ['-id']
-
-    def num_snapshots(self, obj):
-        return format_html(
-            '<a href="/admin/core/snapshot/?tags__id__exact={}">{} total</a>',
-            obj.id,
-            obj.snapshot_set.count(),
-        )
-
-    def snapshots(self, obj):
-        total_count = obj.snapshot_set.count()
-        return mark_safe('<br/>'.join(
-            format_html(
-                '{} <code><a href="/admin/core/snapshot/{}/change"><b>[{}]</b></a> {}</code>',
-                snap.updated.strftime('%Y-%m-%d %H:%M') if snap.updated else 'pending...',
-                snap.id,
-                snap.timestamp,
-                snap.url,
-            )
-            for snap in obj.snapshot_set.order_by('-updated')[:10]
-        ) + (f'<br/><a href="/admin/core/snapshot/?tags__id__exact={obj.id}">and {total_count-10} more...<a>' if obj.snapshot_set.count() > 10 else ''))
-
-
-@admin.register(ArchiveResult, site=archivebox_admin)
-class ArchiveResultAdmin(admin.ModelAdmin):
-    list_display = ('id', 'start_ts', 'extractor', 'snapshot_str', 'tags_str', 'cmd_str', 'status', 'output_str')
-    sort_fields = ('start_ts', 'extractor', 'status')
-    readonly_fields = ('id', 'uuid', 'snapshot_str', 'tags_str')
-    search_fields = ('id', 'uuid', 'snapshot__url', 'extractor', 'output', 'cmd_version', 'cmd', 'snapshot__timestamp')
-    fields = (*readonly_fields, 'snapshot', 'extractor', 'status', 'start_ts', 'end_ts', 'output', 'pwd', 'cmd', 'cmd_version')
-    autocomplete_fields = ['snapshot']
-
-    list_filter = ('status', 'extractor', 'start_ts', 'cmd_version')
-    ordering = ['-start_ts']
-    list_per_page = SNAPSHOTS_PER_PAGE
-
-    @admin.display(
-        description='snapshot'
-    )
-    def snapshot_str(self, obj):
-        return format_html(
-            '<a href="/archive/{}/index.html"><b><code>[{}]</code></b></a><br/>'
-            '<small>{}</small>',
-            obj.snapshot.timestamp,
-            obj.snapshot.timestamp,
-            obj.snapshot.url[:128],
-        )
-
-    @admin.display(
-        description='tags'
-    )
-    def tags_str(self, obj):
-        return obj.snapshot.tags_str()
-
-    def cmd_str(self, obj):
-        return format_html(
-            '<pre>{}</pre>',
-            ' '.join(obj.cmd) if isinstance(obj.cmd, list) else str(obj.cmd),
-        )
-
-    def output_str(self, obj):
-        return format_html(
-            '<a href="/archive/{}/{}" class="output-link">↗️</a><pre>{}</pre>',
-            obj.snapshot.timestamp,
-            obj.output if (obj.status == 'succeeded') and obj.extractor not in ('title', 'archive_org') else 'index.html',
-            obj.output,
-        )

archivebox/core/apps.py

@@ -1,17 +0,0 @@
-from django.apps import AppConfig
-
-
-class CoreAppConfig(AppConfig):
-    name = 'core'
-
-    # label = 'Archive Data'
-    verbose_name = "Archive Data"
-
-    # WIP: broken by Django 3.1.2 -> 4.0 migration
-    # default_auto_field = 'django.db.models.UUIDField'
-
-
-    def ready(self):
-        from .auth import register_signals
-
-        register_signals()

archivebox/core/auth.py

@@ -1,15 +0,0 @@
-__package__ = 'archivebox.core'
-
-
-
-from ..config import (
-    LDAP
-)
-
-def register_signals():
-
-    if LDAP:
-        import django_auth_ldap.backend
-        from .auth_ldap import create_user
-
-        django_auth_ldap.backend.populate_user.connect(create_user)

archivebox/core/auth_ldap.py

@@ -1,12 +0,0 @@
-from django.conf import settings
-from ..config import (
-    LDAP_CREATE_SUPERUSER
-)
-
-def create_user(sender, user=None, ldap_user=None, **kwargs):
-
-    if not user.id and LDAP_CREATE_SUPERUSER:
-        user.is_superuser = True
-
-    user.is_staff = True
-    print(f'[!] WARNING: Creating new user {user} based on LDAP user {ldap_user} (is_staff={user.is_staff}, is_superuser={user.is_superuser})')

archivebox/core/forms.py

@@ -1,96 +0,0 @@
-__package__ = 'archivebox.core'
-
-from django import forms
-
-from ..util import URL_REGEX
-from ..parsers import PARSERS
-from ..vendor.taggit_utils import edit_string_for_tags, parse_tags
-
-PARSER_CHOICES = [
-    (parser_key, parser[0])
-    for parser_key, parser in PARSERS.items()
-]
-DEPTH_CHOICES = (
-    ('0', 'depth = 0 (archive just these URLs)'),
-    ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'),
-)
-
-from ..extractors import get_default_archive_methods
-
-ARCHIVE_METHODS = [
-    (name, name)
-    for name, _, _ in get_default_archive_methods()
-]
-
-
-class AddLinkForm(forms.Form):
-    url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True)
-    parser = forms.ChoiceField(label="URLs format", choices=[('auto', 'Auto-detect parser'), *PARSER_CHOICES], initial='auto')
-    tag = forms.CharField(label="Tags (comma separated tag1,tag2,tag3)", strip=True, required=False)
-    depth = forms.ChoiceField(label="Archive depth", choices=DEPTH_CHOICES, initial='0', widget=forms.RadioSelect(attrs={"class": "depth-selection"}))
-    archive_methods = forms.MultipleChoiceField(
-        label="Archive methods (select at least 1, otherwise all will be used by default)",
-        required=False,
-        widget=forms.SelectMultiple,
-        choices=ARCHIVE_METHODS,
-    )
-    # TODO: hook these up to the view and put them 
-    # in a collapsible UI section labeled "Advanced"
-    #
-    # exclude_patterns = forms.CharField(
-    #     label="Exclude patterns",
-    #     min_length='1',
-    #     required=False,
-    #     initial=URL_DENYLIST,
-    # )
-    # timeout = forms.IntegerField(
-    #     initial=TIMEOUT,
-    # )
-    # overwrite = forms.BooleanField(
-    #     label="Overwrite any existing Snapshots",
-    #     initial=False,
-    # )
-    # index_only = forms.BooleanField(
-    #     label="Add URLs to index without Snapshotting",
-    #     initial=False,
-    # )
-
-class TagWidgetMixin:
-    def format_value(self, value):
-        if value is not None and not isinstance(value, str):
-            value = edit_string_for_tags(value)
-        return super().format_value(value)
-
-class TagWidget(TagWidgetMixin, forms.TextInput):
-    pass
-
-class TagField(forms.CharField):
-    widget = TagWidget
-
-    def clean(self, value):
-        value = super().clean(value)
-        try:
-            return parse_tags(value)
-        except ValueError:
-            raise forms.ValidationError(
-                "Please provide a comma-separated list of tags."
-            )
-
-    def has_changed(self, initial_value, data_value):
-        # Always return False if the field is disabled since self.bound_data
-        # always uses the initial value in this case.
-        if self.disabled:
-            return False
-
-        try:
-            data_value = self.clean(data_value)
-        except forms.ValidationError:
-            pass
-
-        if initial_value is None:
-            initial_value = []
-
-        initial_value = [tag.name for tag in initial_value]
-        initial_value.sort()
-
-        return initial_value != data_value

archivebox/core/management/commands/archivebox.py

@@ -1,18 +0,0 @@
-__package__ = 'archivebox'
-
-from django.core.management.base import BaseCommand
-
-
-from .cli import run_subcommand
-
-
-class Command(BaseCommand):
-    help = 'Run an ArchiveBox CLI subcommand (e.g. add, remove, list, etc)'
-
-    def add_arguments(self, parser):
-        parser.add_argument('subcommand', type=str, help='The subcommand you want to run')
-        parser.add_argument('command_args', nargs='*', help='Arguments to pass to the subcommand')
-
-
-    def handle(self, *args, **kwargs):
-        run_subcommand(kwargs['subcommand'], args=kwargs['command_args'])

archivebox/core/middleware.py

@@ -1,60 +0,0 @@
-__package__ = 'archivebox.core'
-
-import ipaddress
-from django.utils import timezone
-from django.contrib.auth.middleware import RemoteUserMiddleware
-from django.core.exceptions import ImproperlyConfigured
-
-from ..config import PUBLIC_SNAPSHOTS, REVERSE_PROXY_USER_HEADER, REVERSE_PROXY_WHITELIST
-
-
-def detect_timezone(request, activate: bool=True):
-    gmt_offset = (request.COOKIES.get('GMT_OFFSET') or '').strip()
-    tz = None
-    if gmt_offset.replace('-', '').isdigit():
-        tz = timezone.get_fixed_timezone(int(gmt_offset))
-        if activate:
-            timezone.activate(tz)
-    # print('GMT_OFFSET', gmt_offset, tz)
-    return tz
-
-
-def TimezoneMiddleware(get_response):
-    def middleware(request):
-        detect_timezone(request, activate=True)
-        return get_response(request)
-
-    return middleware
-
-
-def CacheControlMiddleware(get_response):
-    def middleware(request):
-        response = get_response(request)
-
-        if '/archive/' in request.path or '/static/' in request.path:
-            policy = 'public' if PUBLIC_SNAPSHOTS else 'private'
-            response['Cache-Control'] = f'{policy}, max-age=60, stale-while-revalidate=300'
-            # print('Set Cache-Control header to', response['Cache-Control'])
-        return response
-
-    return middleware
-
-class ReverseProxyAuthMiddleware(RemoteUserMiddleware):
-    header = 'HTTP_{normalized}'.format(normalized=REVERSE_PROXY_USER_HEADER.replace('-', '_').upper())
-
-    def process_request(self, request):
-        if REVERSE_PROXY_WHITELIST == '':
-            return
-
-        ip = request.META.get('REMOTE_ADDR')
-
-        for cidr in REVERSE_PROXY_WHITELIST.split(','):
-            try:
-                network = ipaddress.ip_network(cidr)
-            except ValueError:
-                raise ImproperlyConfigured(
-                    "The REVERSE_PROXY_WHITELIST config paramater is in invalid format, or "
-                    "contains invalid CIDR. Correct format is a coma-separated list of IPv4/IPv6 CIDRs.")
-
-            if ipaddress.ip_address(ip) in network:
-                return super().process_request(request)

archivebox/core/migrations/0001_initial.py

@@ -1,27 +0,0 @@
-# Generated by Django 2.2 on 2019-05-01 03:27
-
-from django.db import migrations, models
-import uuid
-
-
-class Migration(migrations.Migration):
-
-    initial = True
-
-    dependencies = [
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='Snapshot',
-            fields=[
-                ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
-                ('url', models.URLField(unique=True)),
-                ('timestamp', models.CharField(default=None, max_length=32, null=True, unique=True)),
-                ('title', models.CharField(default=None, max_length=128, null=True)),
-                ('tags', models.CharField(default=None, max_length=256, null=True)),
-                ('added', models.DateTimeField(auto_now_add=True)),
-                ('updated', models.DateTimeField(default=None, null=True)),
-            ],
-        ),
-    ]

archivebox/core/migrations/0002_auto_20200625_1521.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.0.7 on 2020-06-25 15:21
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0001_initial'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(default=None, max_length=32, null=True),
-        ),
-    ]

archivebox/core/migrations/0003_auto_20200630_1034.py

@@ -1,38 +0,0 @@
-# Generated by Django 3.0.7 on 2020-06-30 10:34
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0002_auto_20200625_1521'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='added',
-            field=models.DateTimeField(auto_now_add=True, db_index=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.CharField(db_index=True, default=None, max_length=256, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(db_index=True, default=None, max_length=32, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(db_index=True, default=None, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(db_index=True, default=None, null=True),
-        ),
-    ]

archivebox/core/migrations/0004_auto_20200713_1552.py

@@ -1,19 +0,0 @@
-# Generated by Django 3.0.7 on 2020-07-13 15:52
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0003_auto_20200630_1034'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='timestamp',
-            field=models.CharField(db_index=True, default=None, max_length=32, unique=True),
-            preserve_default=False,
-        ),
-    ]

archivebox/core/migrations/0005_auto_20200728_0326.py

@@ -1,28 +0,0 @@
-# Generated by Django 3.0.7 on 2020-07-28 03:26
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0004_auto_20200713_1552'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.CharField(blank=True, db_index=True, max_length=256, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(blank=True, db_index=True, null=True),
-        ),
-    ]

archivebox/core/migrations/0006_auto_20201012_1520.py

@@ -1,70 +0,0 @@
-# Generated by Django 3.0.8 on 2020-10-12 15:20
-
-from django.db import migrations, models
-from django.utils.text import slugify
-
-def forwards_func(apps, schema_editor):
-    SnapshotModel = apps.get_model("core", "Snapshot")
-    TagModel = apps.get_model("core", "Tag")
-
-    db_alias = schema_editor.connection.alias
-    snapshots = SnapshotModel.objects.all()
-    for snapshot in snapshots:
-        tags = snapshot.tags
-        tag_set = (
-            set(tag.strip() for tag in (snapshot.tags_old or '').split(','))
-        )
-        tag_set.discard("")
-
-        for tag in tag_set:
-            to_add, _ = TagModel.objects.get_or_create(name=tag, defaults={'slug': slugify(tag)})
-            snapshot.tags.add(to_add)
-
-
-def reverse_func(apps, schema_editor):
-    SnapshotModel = apps.get_model("core", "Snapshot")
-    TagModel = apps.get_model("core", "Tag")
-
-    db_alias = schema_editor.connection.alias
-    snapshots = SnapshotModel.objects.all()
-    for snapshot in snapshots:
-        tags = snapshot.tags.values_list("name", flat=True)
-        snapshot.tags_old = ",".join([tag for tag in tags])
-        snapshot.save()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0005_auto_20200728_0326'),
-    ]
-
-    operations = [
-        migrations.RenameField(
-            model_name='snapshot',
-            old_name='tags',
-            new_name='tags_old',
-        ),
-        migrations.CreateModel(
-            name='Tag',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('name', models.CharField(max_length=100, unique=True, verbose_name='name')),
-                ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')),
-            ],
-            options={
-                'verbose_name': 'Tag',
-                'verbose_name_plural': 'Tags',
-            },
-        ),
-        migrations.AddField(
-            model_name='snapshot',
-            name='tags',
-            field=models.ManyToManyField(to='core.Tag'),
-        ),
-        migrations.RunPython(forwards_func, reverse_func),
-        migrations.RemoveField(
-            model_name='snapshot',
-            name='tags_old',
-        ),
-    ]

archivebox/core/migrations/0007_archiveresult.py

@@ -1,114 +0,0 @@
-# Generated by Django 3.0.8 on 2020-11-04 12:25
-
-import json
-from pathlib import Path
-
-from django.db import migrations, models
-import django.db.models.deletion
-
-from config import CONFIG
-from index.json import to_json
-
-try:
-    JSONField = models.JSONField
-except AttributeError:
-    import jsonfield
-    JSONField = jsonfield.JSONField
-
-
-def forwards_func(apps, schema_editor):
-    from core.models import EXTRACTORS
-
-    Snapshot = apps.get_model("core", "Snapshot")
-    ArchiveResult = apps.get_model("core", "ArchiveResult")
-
-    snapshots = Snapshot.objects.all()
-    for snapshot in snapshots:
-        out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
-
-        try:
-            with open(out_dir / "index.json", "r") as f:
-                fs_index = json.load(f)
-        except Exception as e:
-            continue
-
-        history = fs_index["history"]
-
-        for extractor in history:
-            for result in history[extractor]:
-                try:
-                    ArchiveResult.objects.create(
-                        extractor=extractor,
-                        snapshot=snapshot,
-                        pwd=result["pwd"],
-                        cmd=result.get("cmd") or [],
-                        cmd_version=result.get("cmd_version") or 'unknown',
-                        start_ts=result["start_ts"],
-                        end_ts=result["end_ts"],
-                        status=result["status"],
-                        output=result.get("output") or 'null',
-                    )
-                except Exception as e:
-                    print(
-                        '    ! Skipping import due to missing/invalid index.json:',
-                        out_dir,
-                        e,
-                        '(open an issue with this index.json for help)',
-                    )
-
-
-def verify_json_index_integrity(snapshot):
-    results = snapshot.archiveresult_set.all()
-    out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
-    with open(out_dir / "index.json", "r") as f:
-        index = json.load(f)
-
-    history = index["history"]
-    index_results = [result for extractor in history for result in history[extractor]]
-    flattened_results = [result["start_ts"] for result in index_results]
-    
-    missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results]
-
-    for missing in missing_results:
-        index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(),
-                                                    "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output,
-                                                    "schema": "ArchiveResult", "status": missing.status})
-
-    json_index = to_json(index)
-    with open(out_dir / "index.json", "w") as f:
-        f.write(json_index)
-
-
-def reverse_func(apps, schema_editor):
-    Snapshot = apps.get_model("core", "Snapshot")
-    ArchiveResult = apps.get_model("core", "ArchiveResult")
-    for snapshot in Snapshot.objects.all():
-        verify_json_index_integrity(snapshot)
-
-    ArchiveResult.objects.all().delete()
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0006_auto_20201012_1520'),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='ArchiveResult',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('cmd', JSONField()),
-                ('pwd', models.CharField(max_length=256)),
-                ('cmd_version', models.CharField(max_length=32)),
-                ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)),
-                ('output', models.CharField(max_length=512)),
-                ('start_ts', models.DateTimeField()),
-                ('end_ts', models.DateTimeField()),
-                ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)),
-                ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')),
-            ],
-        ),
-        migrations.RunPython(forwards_func, reverse_func),
-    ]

archivebox/core/migrations/0008_auto_20210105_1421.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-01-05 14:21
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0007_archiveresult'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='cmd_version',
-            field=models.CharField(blank=True, default=None, max_length=32, null=True),
-        ),
-    ]

archivebox/core/migrations/0009_auto_20210216_1038.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 10:38
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0008_auto_20210105_1421'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='updated',
-            field=models.DateTimeField(auto_now=True, db_index=True, null=True),
-        ),
-    ]

archivebox/core/migrations/0010_auto_20210216_1055.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 10:55
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0009_auto_20210216_1038'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='start_ts',
-            field=models.DateTimeField(db_index=True),
-        ),
-    ]

archivebox/core/migrations/0011_auto_20210216_1331.py

@@ -1,24 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 13:31
-
-from django.db import migrations, models
-import uuid
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0010_auto_20210216_1055'),
-    ]
-
-    operations = [
-        migrations.AddField(
-            model_name='archiveresult',
-            name='uuid',
-            field=models.UUIDField(default=uuid.uuid4, editable=False),
-        ),
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/migrations/0012_auto_20210216_1425.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-16 14:25
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0011_auto_20210216_1331'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='cmd_version',
-            field=models.CharField(blank=True, default=None, max_length=128, null=True),
-        ),
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='output',
-            field=models.CharField(max_length=1024),
-        ),
-    ]

archivebox/core/migrations/0013_auto_20210218_0729.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:29
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0012_auto_20210216_1425'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=256, null=True),
-        ),
-    ]

archivebox/core/migrations/0014_auto_20210218_0729.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:29
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0013_auto_20210218_0729'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=1024, null=True),
-        ),
-    ]

archivebox/core/migrations/0015_auto_20210218_0730.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 07:30
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0014_auto_20210218_0729'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='title',
-            field=models.CharField(blank=True, db_index=True, max_length=512, null=True),
-        ),
-    ]

archivebox/core/migrations/0016_auto_20210218_1204.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-18 12:04
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0015_auto_20210218_0730'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='tags',
-            field=models.ManyToManyField(blank=True, to='core.Tag'),
-        ),
-    ]

archivebox/core/migrations/0017_auto_20210219_0211.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-02-19 02:11
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0016_auto_20210218_1204'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='tag',
-            name='slug',
-            field=models.SlugField(blank=True, max_length=100, unique=True, verbose_name='slug'),
-        ),
-    ]

archivebox/core/migrations/0018_auto_20210327_0952.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.3 on 2021-03-27 09:52
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0017_auto_20210219_0211'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='tag',
-            name='name',
-            field=models.CharField(max_length=100, unique=True),
-        ),
-        migrations.AlterField(
-            model_name='tag',
-            name='slug',
-            field=models.SlugField(blank=True, max_length=100, unique=True),
-        ),
-    ]

archivebox/core/migrations/0019_auto_20210401_0654.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.3 on 2021-04-01 06:54
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0018_auto_20210327_0952'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='snapshot',
-            name='url',
-            field=models.URLField(db_index=True, unique=True),
-        ),
-    ]

archivebox/core/migrations/0020_auto_20210410_1031.py

@@ -1,23 +0,0 @@
-# Generated by Django 3.1.8 on 2021-04-10 10:31
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0019_auto_20210401_0654'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='id',
-            field=models.AutoField(primary_key=True, serialize=False, verbose_name='ID'),
-        ),
-        migrations.AlterField(
-            model_name='tag',
-            name='id',
-            field=models.AutoField(primary_key=True, serialize=False, verbose_name='ID'),
-        ),
-    ]

archivebox/core/migrations/0021_auto_20220914_0934.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.14 on 2022-09-14 09:34
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0020_auto_20210410_1031'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('title', 'title'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/migrations/0022_auto_20231023_2008.py

@@ -1,18 +0,0 @@
-# Generated by Django 3.1.14 on 2023-10-23 20:08
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('core', '0021_auto_20220914_0934'),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name='archiveresult',
-            name='extractor',
-            field=models.CharField(choices=[('favicon', 'favicon'), ('headers', 'headers'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('wget', 'wget'), ('title', 'title'), ('readability', 'readability'), ('mercury', 'mercury'), ('htmltotext', 'htmltotext'), ('git', 'git'), ('media', 'media'), ('archive_org', 'archive_org')], max_length=32),
-        ),
-    ]

archivebox/core/mixins.py

@@ -1,21 +0,0 @@
-from django.contrib import messages
-
-from archivebox.search import query_search_index
-
-class SearchResultsAdminMixin:
-    def get_search_results(self, request, queryset, search_term: str):
-        """Enhances the search queryset with results from the search backend"""
-        
-        qs, use_distinct = super().get_search_results(request, queryset, search_term)
-
-        search_term = search_term.strip()
-        if not search_term:
-            return qs, use_distinct
-        try:
-            qsearch = query_search_index(search_term)
-            qs = qs | qsearch
-        except Exception as err:
-            print(f'[!] Error while using search backend: {err.__class__.__name__} {err}')
-            messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}')
-        
-        return qs.distinct(), use_distinct

archivebox/core/models.py

@@ -1,294 +0,0 @@
-__package__ = 'archivebox.core'
-
-
-import uuid
-import json
-
-from pathlib import Path
-from typing import Optional, List
-
-from django.db import models
-from django.utils.functional import cached_property
-from django.utils.text import slugify
-from django.core.cache import cache
-from django.urls import reverse
-from django.db.models import Case, When, Value, IntegerField
-from django.contrib.auth.models import User   # noqa
-
-from ..config import ARCHIVE_DIR, ARCHIVE_DIR_NAME
-from ..system import get_dir_size
-from ..util import parse_date, base_url, hashurl
-from ..index.schema import Link
-from ..index.html import snapshot_icons
-from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE
-
-EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()]
-STATUS_CHOICES = [
-    ("succeeded", "succeeded"),
-    ("failed", "failed"),
-    ("skipped", "skipped")
-]
-
-try:
-    JSONField = models.JSONField
-except AttributeError:
-    import jsonfield
-    JSONField = jsonfield.JSONField
-
-
-class Tag(models.Model):
-    """
-    Based on django-taggit model
-    """
-    id = models.AutoField(primary_key=True, serialize=False, verbose_name='ID')
-
-    name = models.CharField(unique=True, blank=False, max_length=100)
-
-    # slug is autoset on save from name, never set it manually
-    slug = models.SlugField(unique=True, blank=True, max_length=100)
-
-
-    class Meta:
-        verbose_name = "Tag"
-        verbose_name_plural = "🏷️ Tags"
-
-    def __str__(self):
-        return self.name
-
-    def slugify(self, tag, i=None):
-        slug = slugify(tag)
-        if i is not None:
-            slug += "_%d" % i
-        return slug
-
-    def save(self, *args, **kwargs):
-        if self._state.adding and not self.slug:
-            self.slug = self.slugify(self.name)
-
-            # if name is different but slug conficts with another tags slug, append a counter
-            # with transaction.atomic():
-            slugs = set(
-                type(self)
-                ._default_manager.filter(slug__startswith=self.slug)
-                .values_list("slug", flat=True)
-            )
-
-            i = None
-            while True:
-                slug = self.slugify(self.name, i)
-                if slug not in slugs:
-                    self.slug = slug
-                    return super().save(*args, **kwargs)
-                i = 1 if i is None else i+1
-        else:
-            return super().save(*args, **kwargs)
-
-
-class Snapshot(models.Model):
-    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
-
-    url = models.URLField(unique=True, db_index=True)
-    timestamp = models.CharField(max_length=32, unique=True, db_index=True)
-
-    title = models.CharField(max_length=512, null=True, blank=True, db_index=True)
-
-    added = models.DateTimeField(auto_now_add=True, db_index=True)
-    updated = models.DateTimeField(auto_now=True, blank=True, null=True, db_index=True)
-    tags = models.ManyToManyField(Tag, blank=True)
-
-    keys = ('url', 'timestamp', 'title', 'tags', 'updated')
-
-    class Meta:
-        verbose_name = "Snapshot"
-        verbose_name_plural = "⭐️ Archived Webpages (Snapshots)"
-
-    def __repr__(self) -> str:
-        title = self.title or '-'
-        return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})'
-
-    def __str__(self) -> str:
-        title = self.title or '-'
-        return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})'
-
-    @classmethod
-    def from_json(cls, info: dict):
-        info = {k: v for k, v in info.items() if k in cls.keys}
-        return cls(**info)
-
-    def as_json(self, *args) -> dict:
-        args = args or self.keys
-        return {
-            key: getattr(self, key)
-            if key != 'tags' else self.tags_str()
-            for key in args
-        }
-
-    def as_link(self) -> Link:
-        return Link.from_json(self.as_json())
-
-    def as_link_with_details(self) -> Link:
-        from ..index import load_link_details
-        return load_link_details(self.as_link())
-
-    def tags_str(self, nocache=True) -> str:
-        cache_key = f'{self.id}-{(self.updated or self.added).timestamp()}-tags'
-        calc_tags_str = lambda: ','.join(self.tags.order_by('name').values_list('name', flat=True))
-        if nocache:
-            tags_str = calc_tags_str()
-            cache.set(cache_key, tags_str)
-            return tags_str
-        return cache.get_or_set(cache_key, calc_tags_str)
-
-    def icons(self) -> str:
-        return snapshot_icons(self)
-
-    @cached_property
-    def extension(self) -> str:
-        from ..util import extension
-        return extension(self.url)
-
-    @cached_property
-    def bookmarked(self):
-        return parse_date(self.timestamp)
-
-    @cached_property
-    def bookmarked_date(self):
-        # TODO: remove this
-        return self.bookmarked
-
-    @cached_property
-    def is_archived(self):
-        return self.as_link().is_archived
-
-    @cached_property
-    def num_outputs(self):
-        return self.archiveresult_set.filter(status='succeeded').count()
-
-    @cached_property
-    def url_hash(self):
-        return hashurl(self.url)
-
-    @cached_property
-    def base_url(self):
-        return base_url(self.url)
-
-    @cached_property
-    def link_dir(self):
-        return str(ARCHIVE_DIR / self.timestamp)
-
-    @cached_property
-    def archive_path(self):
-        return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp)
-
-    @cached_property
-    def archive_size(self):
-        cache_key = f'{str(self.id)[:12]}-{(self.updated or self.added).timestamp()}-size'
-
-        def calc_dir_size():
-            try:
-                return get_dir_size(self.link_dir)[0]
-            except Exception:
-                return 0
-
-        return cache.get_or_set(cache_key, calc_dir_size)
-
-    @cached_property
-    def thumbnail_url(self) -> Optional[str]:
-        result = self.archiveresult_set.filter(
-            extractor='screenshot',
-            status='succeeded'
-        ).only('output').last()
-        if result:
-            return reverse('Snapshot', args=[f'{str(self.timestamp)}/{result.output}'])
-        return None
-
-    @cached_property
-    def headers(self) -> Optional[dict]:
-        try:
-            return json.loads((Path(self.link_dir) / 'headers.json').read_text(encoding='utf-8').strip())
-        except Exception:
-            pass
-        return None
-
-    @cached_property
-    def status_code(self) -> Optional[str]:
-        return self.headers and self.headers.get('Status-Code')
-
-    @cached_property
-    def history(self) -> dict:
-        # TODO: use ArchiveResult for this instead of json
-        return self.as_link_with_details().history
-
-    @cached_property
-    def latest_title(self) -> Optional[str]:
-        if self.title:
-            return self.title   # whoopdedoo that was easy
-        
-        try:
-            # take longest successful title from ArchiveResult db history
-            return sorted(
-                self.archiveresult_set\
-                    .filter(extractor='title', status='succeeded', output__isnull=False)\
-                    .values_list('output', flat=True),
-                key=lambda r: len(r),
-            )[-1]
-        except IndexError:
-            pass
-
-        try:
-            # take longest successful title from Link json index file history
-            return sorted(
-                (
-                    result.output.strip()
-                    for result in self.history['title']
-                    if result.status == 'succeeded' and result.output.strip()
-                ),
-                key=lambda r: len(r),
-            )[-1]
-        except (KeyError, IndexError):
-            pass
-
-        return None
-
-    def save_tags(self, tags: List[str]=()) -> None:
-        tags_id = []
-        for tag in tags:
-            if tag.strip():
-                tags_id.append(Tag.objects.get_or_create(name=tag)[0].id)
-        self.tags.clear()
-        self.tags.add(*tags_id)
-
-
-class ArchiveResultManager(models.Manager):
-    def indexable(self, sorted: bool = True):
-        INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
-        qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded')
-
-        if sorted:
-            precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
-            qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence')
-        return qs
-
-
-class ArchiveResult(models.Model):
-    id = models.AutoField(primary_key=True, serialize=False, verbose_name='ID')
-    uuid = models.UUIDField(default=uuid.uuid4, editable=False)
-
-    snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE)
-    extractor = models.CharField(choices=EXTRACTORS, max_length=32)
-    cmd = JSONField()
-    pwd = models.CharField(max_length=256)
-    cmd_version = models.CharField(max_length=128, default=None, null=True, blank=True)
-    output = models.CharField(max_length=1024)
-    start_ts = models.DateTimeField(db_index=True)
-    end_ts = models.DateTimeField()
-    status = models.CharField(max_length=16, choices=STATUS_CHOICES)
-
-    objects = ArchiveResultManager()
-
-    class Meta:
-        verbose_name = "ArchiveResult"
-        verbose_name_plural = "📑 Logs (ArchiveResults)"
-
-    def __str__(self):
-        return self.extractor

archivebox/core/settings.py

@@ -1,417 +0,0 @@
-__package__ = 'archivebox.core'
-
-# TODO: add this after we upgrade to Django >=3.2
-# https://github.com/typeddjango/django-stubs
-# import django_stubs_ext
-# django_stubs_ext.monkeypatch()
-
-import os
-import sys
-import re
-import logging
-import tempfile
-
-from pathlib import Path
-from django.utils.crypto import get_random_string
-
-from ..config import (
-    DEBUG,
-    SECRET_KEY,
-    ALLOWED_HOSTS,
-    PACKAGE_DIR,
-    TEMPLATES_DIR_NAME,
-    CUSTOM_TEMPLATES_DIR,
-    SQL_INDEX_FILENAME,
-    OUTPUT_DIR,
-    LOGS_DIR,
-    TIMEZONE,
-
-    LDAP,
-    LDAP_SERVER_URI,
-    LDAP_BIND_DN,
-    LDAP_BIND_PASSWORD,
-    LDAP_USER_BASE,
-    LDAP_USER_FILTER,
-    LDAP_USERNAME_ATTR,
-    LDAP_FIRSTNAME_ATTR,
-    LDAP_LASTNAME_ATTR,
-    LDAP_EMAIL_ATTR,
-)
-
-IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3]
-IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ
-IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3]
-
-################################################################################
-### Django Core Settings
-################################################################################
-
-WSGI_APPLICATION = 'core.wsgi.application'
-ROOT_URLCONF = 'core.urls'
-
-LOGIN_URL = '/accounts/login/'
-LOGOUT_REDIRECT_URL = os.environ.get('LOGOUT_REDIRECT_URL', '/')
-
-PASSWORD_RESET_URL = '/accounts/password_reset/'
-APPEND_SLASH = True
-
-DEBUG = DEBUG or ('--debug' in sys.argv)
-
-INSTALLED_APPS = [
-    'django.contrib.auth',
-    'django.contrib.contenttypes',
-    'django.contrib.sessions',
-    'django.contrib.messages',
-    'django.contrib.staticfiles',
-    'django.contrib.admin',
-    'solo',
-
-
-    'core',
-    'api',
-    
-    # Plugins
-    
-    'plugins.defaults',
-    'plugins.system',
-    # 'plugins.replaywebpage',           # provides UI to view WARC files
-    # 'plugins.gallerydl',               # provides gallerydl dependency + extractor
-    # 'plugins.browsertrix',             # provides browsertrix dependency + extractor
-    # 'plugins.playwright',              # provides playwright dependency
-    # ...
-    # someday we may have enough plugins to justify dynamic loading:
-    # *(path.parent.name for path in (Path(PACKAGE_DIR) / 'plugins').glob('*/apps.py')),,
-
-    'django_extensions',
-]
-
-################################################################################
-### Staticfile and Template Settings
-################################################################################
-
-STATIC_URL = '/static/'
-
-STATIC_ROOT = Path(PACKAGE_DIR) / 'collected_static'
-
-STATICFILES_DIRS = [
-    *([str(CUSTOM_TEMPLATES_DIR / 'static')] if CUSTOM_TEMPLATES_DIR else []),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'static'),
-
-    # Plugins
-    # str(Path(PACKAGE_DIR) / 'plugins/defaults/static'),
-    # str(Path(PACKAGE_DIR) / 'plugins/replaywebpage/static'),
-    # str(Path(PACKAGE_DIR) / 'plugins/gallerydl/static'),
-    # str(Path(PACKAGE_DIR) / 'plugins/browsertrix/static'),
-    # str(Path(PACKAGE_DIR) / 'plugins/playwright/static'),
-    # ...
-    # someday if there are many more plugins / user-addable plugins:
-    # *(str(path) for path in (Path(PACKAGE_DIR) / 'plugins').glob('*/static')),
-]
-
-MEDIA_URL = '/archive/'
-MEDIA_ROOT = OUTPUT_DIR / 'archive'
-
-
-TEMPLATE_DIRS = [
-    *([str(CUSTOM_TEMPLATES_DIR)] if CUSTOM_TEMPLATES_DIR else []),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'core'),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'admin'),
-    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME),
-
-    # Plugins
-    # added by plugins.<PluginName>.apps.<AppName>.ready -> .settings.register_plugin_settings
-    # str(Path(PACKAGE_DIR) / 'plugins/defaults/templates'),
-    # str(Path(PACKAGE_DIR) / 'plugins/replaywebpage/templates'),
-    # str(Path(PACKAGE_DIR) / 'plugins/gallerydl/templates'),
-    # str(Path(PACKAGE_DIR) / 'plugins/browsertrix/templates'),
-    # str(Path(PACKAGE_DIR) / 'plugins/playwright/templates'),
-    # ...
-    #
-    # someday if there are many more plugins / user-addable plugins:
-    # *(str(path) for path in (Path(PACKAGE_DIR) / 'plugins').glob('*/templates')),
-]
-
-TEMPLATES = [
-    {
-        'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': TEMPLATE_DIRS,
-        'APP_DIRS': True,
-        'OPTIONS': {
-            'context_processors': [
-                'django.template.context_processors.debug',
-                'django.template.context_processors.request',
-                'django.contrib.auth.context_processors.auth',
-                'django.contrib.messages.context_processors.messages',
-            ],
-        },
-    },
-]
-
-# For usage with https://www.jetadmin.io/integrations/django
-# INSTALLED_APPS += ['jet_django']
-# JET_PROJECT = 'archivebox'
-# JET_TOKEN = 'some-api-token-here'
-
-
-MIDDLEWARE = [
-    'core.middleware.TimezoneMiddleware',
-    'django.middleware.security.SecurityMiddleware',
-    'django.contrib.sessions.middleware.SessionMiddleware',
-    'django.middleware.common.CommonMiddleware',
-    'django.middleware.csrf.CsrfViewMiddleware',
-    'django.contrib.auth.middleware.AuthenticationMiddleware',
-    'core.middleware.ReverseProxyAuthMiddleware',
-    'django.contrib.messages.middleware.MessageMiddleware',
-    'core.middleware.CacheControlMiddleware',
-]
-
-################################################################################
-### Authentication Settings
-################################################################################
-
-AUTHENTICATION_BACKENDS = [
-    'django.contrib.auth.backends.RemoteUserBackend',
-    'django.contrib.auth.backends.ModelBackend',
-]
-
-if LDAP:
-    try:
-        import ldap
-        from django_auth_ldap.config import LDAPSearch
-
-        global AUTH_LDAP_SERVER_URI
-        global AUTH_LDAP_BIND_DN
-        global AUTH_LDAP_BIND_PASSWORD
-        global AUTH_LDAP_USER_SEARCH
-        global AUTH_LDAP_USER_ATTR_MAP
-
-        AUTH_LDAP_SERVER_URI = LDAP_SERVER_URI
-        AUTH_LDAP_BIND_DN = LDAP_BIND_DN
-        AUTH_LDAP_BIND_PASSWORD = LDAP_BIND_PASSWORD
-
-        assert AUTH_LDAP_SERVER_URI and LDAP_USERNAME_ATTR and LDAP_USER_FILTER, 'LDAP_* config options must all be set if LDAP=True'
-
-        AUTH_LDAP_USER_SEARCH = LDAPSearch(
-            LDAP_USER_BASE,
-            ldap.SCOPE_SUBTREE,
-            '(&(' + LDAP_USERNAME_ATTR + '=%(user)s)' + LDAP_USER_FILTER + ')',
-        )
-
-        AUTH_LDAP_USER_ATTR_MAP = {
-            'username': LDAP_USERNAME_ATTR,
-            'first_name': LDAP_FIRSTNAME_ATTR,
-            'last_name': LDAP_LASTNAME_ATTR,
-            'email': LDAP_EMAIL_ATTR,
-        }
-
-        AUTHENTICATION_BACKENDS = [
-            'django.contrib.auth.backends.ModelBackend',
-            'django_auth_ldap.backend.LDAPBackend',
-        ]
-    except ModuleNotFoundError:
-        sys.stderr.write('[X] Error: Found LDAP=True config but LDAP packages not installed. You may need to run: pip install archivebox[ldap]\n\n')
-        # dont hard exit here. in case the user is just running "archivebox version" or "archivebox help", we still want those to work despite broken ldap
-        # sys.exit(1)
-
-
-################################################################################
-### Debug Settings
-################################################################################
-
-# only enable debug toolbar when in DEBUG mode with --nothreading (it doesnt work in multithreaded mode)
-DEBUG_TOOLBAR = DEBUG and ('--nothreading' in sys.argv) and ('--reload' not in sys.argv)
-if DEBUG_TOOLBAR:
-    try:
-        import debug_toolbar   # noqa
-        DEBUG_TOOLBAR = True
-    except ImportError:
-        DEBUG_TOOLBAR = False
-
-if DEBUG_TOOLBAR:
-    INSTALLED_APPS = [*INSTALLED_APPS, 'debug_toolbar']
-    INTERNAL_IPS = ['0.0.0.0', '127.0.0.1', '*']
-    DEBUG_TOOLBAR_CONFIG = {
-        "SHOW_TOOLBAR_CALLBACK": lambda request: True,
-        "RENDER_PANELS": True,
-    }
-    DEBUG_TOOLBAR_PANELS = [
-        'debug_toolbar.panels.history.HistoryPanel',
-        'debug_toolbar.panels.versions.VersionsPanel',
-        'debug_toolbar.panels.timer.TimerPanel',
-        'debug_toolbar.panels.settings.SettingsPanel',
-        'debug_toolbar.panels.headers.HeadersPanel',
-        'debug_toolbar.panels.request.RequestPanel',
-        'debug_toolbar.panels.sql.SQLPanel',
-        'debug_toolbar.panels.staticfiles.StaticFilesPanel',
-        # 'debug_toolbar.panels.templates.TemplatesPanel',    # buggy/slow
-        'debug_toolbar.panels.cache.CachePanel',
-        'debug_toolbar.panels.signals.SignalsPanel',
-        'debug_toolbar.panels.logging.LoggingPanel',
-        'debug_toolbar.panels.redirects.RedirectsPanel',
-        'debug_toolbar.panels.profiling.ProfilingPanel',
-        'djdt_flamegraph.FlamegraphPanel',
-    ]
-    MIDDLEWARE = [*MIDDLEWARE, 'debug_toolbar.middleware.DebugToolbarMiddleware']
-
-
-
-################################################################################
-### External Service Settings
-################################################################################
-
-DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME
-DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", str(DATABASE_FILE))
-
-DATABASES = {
-    'default': {
-        'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': DATABASE_NAME,
-        'OPTIONS': {
-            'timeout': 60,
-            'check_same_thread': False,
-        },
-        'TIME_ZONE': TIMEZONE,
-        # DB setup is sometimes modified at runtime by setup_django() in config.py
-    }
-}
-
-CACHE_BACKEND = 'django.core.cache.backends.locmem.LocMemCache'
-# CACHE_BACKEND = 'django.core.cache.backends.db.DatabaseCache'
-# CACHE_BACKEND = 'django.core.cache.backends.dummy.DummyCache'
-
-CACHES = {
-    'default': {
-        'BACKEND': CACHE_BACKEND,
-        'LOCATION': 'django_cache_default',
-    }
-}
-
-EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
-
-
-################################################################################
-### Security Settings
-################################################################################
-
-SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789_')
-
-ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
-
-SECURE_BROWSER_XSS_FILTER = True
-SECURE_CONTENT_TYPE_NOSNIFF = True
-SECURE_REFERRER_POLICY = 'strict-origin-when-cross-origin'
-
-CSRF_COOKIE_SECURE = False
-SESSION_COOKIE_SECURE = False
-SESSION_COOKIE_DOMAIN = None
-SESSION_COOKIE_AGE = 1209600  # 2 weeks
-SESSION_EXPIRE_AT_BROWSER_CLOSE = False
-SESSION_SAVE_EVERY_REQUEST = True
-
-SESSION_ENGINE = "django.contrib.sessions.backends.db"
-
-AUTH_PASSWORD_VALIDATORS = [
-    {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator'},
-    {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'},
-]
-
-################################################################################
-### Shell Settings
-################################################################################
-
-SHELL_PLUS = 'ipython'
-SHELL_PLUS_PRINT_SQL = False
-IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner']
-IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell'
-if IS_SHELL:
-    os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py')
-
-
-################################################################################
-### Internationalization & Localization Settings
-################################################################################
-
-LANGUAGE_CODE = 'en-us'
-USE_I18N = True
-USE_TZ = True
-DATETIME_FORMAT = 'Y-m-d g:iA'
-SHORT_DATETIME_FORMAT = 'Y-m-d h:iA'
-TIME_ZONE = TIMEZONE        # django convention is TIME_ZONE, archivebox config uses TIMEZONE, they are equivalent
-
-
-from django.conf.locale.en import formats as en_formats
-
-en_formats.DATETIME_FORMAT = DATETIME_FORMAT
-en_formats.SHORT_DATETIME_FORMAT = SHORT_DATETIME_FORMAT
-
-
-################################################################################
-### Logging Settings
-################################################################################
-
-IGNORABLE_404_URLS = [
-    re.compile(r'apple-touch-icon.*\.png$'),
-    re.compile(r'favicon\.ico$'),
-    re.compile(r'robots\.txt$'),
-    re.compile(r'.*\.(css|js)\.map$'),
-]
-
-class NoisyRequestsFilter(logging.Filter):
-    def filter(self, record) -> bool:
-        logline = record.getMessage()
-
-        # ignore harmless 404s for the patterns in IGNORABLE_404_URLS
-        for ignorable_url_pattern in IGNORABLE_404_URLS:
-            ignorable_log_pattern = re.compile(f'^"GET /.*/?{ignorable_url_pattern.pattern[:-1]} HTTP/.*" (200|30.|404) .+$', re.I | re.M)
-            if ignorable_log_pattern.match(logline):
-                return False
-
-        # ignore staticfile requests that 200 or 30*
-        ignoreable_200_log_pattern = re.compile(r'"GET /static/.* HTTP/.*" (200|30.) .+', re.I | re.M)
-        if ignoreable_200_log_pattern.match(logline):
-            return False
-
-        return True
-
-if LOGS_DIR.exists():
-    ERROR_LOG = (LOGS_DIR / 'errors.log')
-else:
-    # historically too many edge cases here around creating log dir w/ correct permissions early on
-    # if there's an issue on startup, we trash the log and let user figure it out via stdout/stderr
-    ERROR_LOG = tempfile.NamedTemporaryFile().name
-
-LOGGING = {
-    'version': 1,
-    'disable_existing_loggers': False,
-    'handlers': {
-        'console': {
-            'class': 'logging.StreamHandler',
-        },
-        'logfile': {
-            'level': 'ERROR',
-            'class': 'logging.handlers.RotatingFileHandler',
-            'filename': ERROR_LOG,
-            'maxBytes': 1024 * 1024 * 25,  # 25 MB
-            'backupCount': 10,
-        },
-    },
-    'filters': {
-        'noisyrequestsfilter': {
-            '()': NoisyRequestsFilter,
-        }
-    },
-    'loggers': {
-        'django': {
-            'handlers': ['console', 'logfile'],
-            'level': 'INFO',
-            'filters': ['noisyrequestsfilter'],
-        },
-        'django.server': {
-            'handlers': ['console', 'logfile'],
-            'level': 'INFO',
-            'filters': ['noisyrequestsfilter'],
-        }
-    },
-}