Merge branch 'dev' into link-removal2

mark_safe is dangerous, as the URL's filename could have malicious HTML fragments in it.

Co-authored-by: Nick Sweeting <git@sweeting.me>

.dockerignore

@@ -3,6 +3,8 @@
 *.pyc
 __pycache__/
 .mypy_cache/
+.pytest_cache/
+.github/
 
 venv/
 .venv/
@@ -10,6 +12,10 @@ venv/
 
 build/
 dist/
+pip_dist/
+!pip_dist/archivebox.egg-info/requires.txt
+brew_dist/
+assets/
 
 data/
 output/

.github/CONTRIBUTING.md

@@ -9,32 +9,30 @@
 
 **Useful links:**
 
-- https://github.com/pirate/ArchiveBox/issues
-- https://github.com/pirate/ArchiveBox/pulls
-- https://github.com/pirate/ArchiveBox/wiki/Roadmap
-- https://github.com/pirate/ArchiveBox/wiki/Install#manual-setup
+- https://github.com/ArchiveBox/ArchiveBox/issues
+- https://github.com/ArchiveBox/ArchiveBox/pulls
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
 
 ### Development Setup
 
 ```bash
-git clone https://github.com/pirate/ArchiveBox
+git clone https://github.com/ArchiveBox/ArchiveBox
 cd ArchiveBox
-# Optionally create a virtualenv
-pip install -r requirements.txt
-pip install -e .
+# Ideally do this in a virtualenv
+pip install -e '.[dev]'  # or use: pipenv install --dev
 ```
 
 ### Running Tests
 
 ```bash
-./bin/archive tests/*
-# look for errors in stdout/stderr
-# then confirm output html looks right
-
-# if on >v0.4 run the django test suite:
-archivebox manage test
+./bin/lint.sh
+./bin/test.sh
+./bin/build.sh
 ```
 
+For more common tasks see the `Development` section at the bottom of the README.
+
 ### Getting Help
 
-Open issues on Github or contact me https://sweeting.me/#contact.
+Open issues on Github or message me https://sweeting.me/#contact.

.github/FUNDING.yml

@@ -1,3 +1,3 @@
 github: pirate
 patreon: theSquashSH
-custom: ["https://paypal.me/NicholasSweeting", "https://www.blockchain.com/eth/address/0x5D4c34D4a121Fe08d1dDB7969F07550f2dB9f471", "https://www.blockchain.com/btc/address/1HuxXriPE2Bbnag3jJrqa3bkNHrs297dYH"]
+custom: ["https://twitter.com/ArchiveBoxApp", "https://paypal.me/NicholasSweeting", "https://www.blockchain.com/eth/address/0x5D4c34D4a121Fe08d1dDB7969F07550f2dB9f471", "https://www.blockchain.com/btc/address/1HuxXriPE2Bbnag3jJrqa3bkNHrs297dYH"]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,7 +10,8 @@ assignees: ''
 <!--
 Please fill out the following information, 
 feel free to delete sections if they're not applicable 
-or if long issue templates annoy you :)
+or if long issue templates annoy you.
+(the only required section is the version information)
 -->
 
 #### Describe the bug
@@ -35,9 +36,11 @@ If applicable, post any relevant screenshots or copy/pasted terminal output from
 If you're reporting a parsing / importing error, **you must paste a copy of your redacted import file here**.
 -->
 
-#### Software versions
+#### ArchiveBox version
 
- - OS:                        ([e.g. macOS 10.14] the operating system you're running ArchiveBox on)
- - ArchiveBox version:        (`git rev-parse HEAD | head -c7` [e.g. d798117] commit ID of the version you're running)
- - Python version:            (`python3 --version` [e.g. 3.7.0])
- - Chrome version:            (`chromium-browser --version` [e.g. 73.1.2.3] if relevant to bug)
+<!-- Run the `archivebox version` command locally then copy paste the result here: -->
+```logs
+replace this line with the *full*, unshortened output of running `archivebox version`
+```
+<!-- Tickets without full version info will closed until it is provided,
+we need the full output here to help you solve your issue -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -45,6 +45,6 @@ workarounds, or other software you've considered using to fix the problem.
 
 ---
 
- - [ ] I'm willing to contribute dev time / money to fix this issue
+ - [ ] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue
  - [ ] I like ArchiveBox so far / would recommend it to a friend
  - [ ] I've had a lot of difficulty getting ArchiveBox set up

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,10 +1,12 @@
-**IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes, I will close them with great prejudice.  The PEP8 checks I don't follow are intentional. PRs for minor bugfixes, typos, etc are fine.**
+<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes. -->
 
 # Summary
 
-e.g. This PR fixes ABC or adds the ability to do XYZ...
+<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->
 
-**Related issues: #XYZ** (delete this line if there are no related issues)
+# Related issues
+
+<!-- e.g. #123 or Roadmap goal # https://github.com/pirate/ArchiveBox/wiki/Roadmap -->
 
 # Changes these areas
 
@@ -13,9 +15,4 @@ e.g. This PR fixes ABC or adds the ability to do XYZ...
 - [ ] Command line interface
 - [ ] Configuration options
 - [ ] Internal architecture
-- [ ] Archived data layout on disk
-
-# Roadmap Goals
-
-This PR helps us move towards xyz roadmap goal, as outlined here: https://github.com/pirate/ArchiveBox/wiki/Roadmap
-(delete this section if it's just a bugfix / simple PR)
+- [ ] Snapshot data layout on disk

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,32 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ dev ]
+  pull_request:
+    branches: [ dev ]
+  schedule:
+    - cron: '43 1 * * 2'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        queries: security-extended
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

.github/workflows/debian.yml

@@ -0,0 +1,77 @@
+name: Build Debian package
+
+on:
+  workflow_dispatch:
+  push:
+
+env:
+  DEB_BUILD_OPTIONS: nocheck
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Install packaging dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            python3 python3-dev python3-pip python3-venv python3-all \
+            dh-python debhelper devscripts dput software-properties-common \
+            python3-distutils python3-setuptools python3-wheel python3-stdeb
+
+      - name: Build Debian/Apt sdist_dsc
+        run: |
+          rm -Rf deb_dist/*
+          python3 setup.py --command-packages=stdeb.command sdist_dsc
+
+      - name: Build Debian/Apt bdist_deb
+        run: |
+          python3 setup.py --command-packages=stdeb.command bdist_deb
+
+      - name: Install archivebox from deb
+        run: |
+          cd deb_dist/
+          sudo apt-get install ./archivebox*.deb
+
+      - name: Check ArchiveBox version
+        run: |
+          # must create dir needed for snaps to run as non-root on github actions
+          sudo mkdir -p /run/user/1001 && sudo chmod -R 777 /run/user/1001
+          mkdir "${{ github.workspace }}/data" && cd "${{ github.workspace }}/data"
+          archivebox init
+          archivebox config --set SAVE_READABILITY=False
+          archivebox config --set SAVE_MERCURY=False
+          archivebox config --set SAVE_SINGLEFILE=False
+          archivebox --version
+
+      - name: Add some links to test
+        run: |
+          cd "${{ github.workspace }}/data"
+          archivebox add 'https://example.com'
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd deb_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Debian package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/debian-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: deb_dist
+
+      # - name: Push build to Launchpad PPA
+      #   run: |
+      #     debsign -k "$PGP_KEY_ID" "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"
+      #     dput archivebox "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"

.github/workflows/docker.yml

@@ -0,0 +1,78 @@
+name: Build Docker image
+
+on:
+  on: workflow_dispatch
+  release:
+    types:
+      - created
+
+env:
+  DOCKER_IMAGE: archivebox-ci
+
+
+jobs:
+  buildx:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+      
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v1
+        with:
+          version: latest
+          install: true
+      
+      - name: Builder instance name
+        run: echo ${{ steps.buildx.outputs.name }}
+      
+      - name: Available platforms
+        run: echo ${{ steps.buildx.outputs.platforms }}
+      
+      - name: Cache Docker layers
+        uses: actions/cache@v2
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Docker Login
+        uses: docker/login-action@v1
+        if: github.event_name != 'pull_request'
+        with:
+           username: ${{ secrets.DOCKER_USERNAME }}
+           password: ${{ secrets.DOCKER_PASSWORD }}
+      
+      - name: Collect Docker tags
+        id: docker_meta
+        uses: crazy-max/ghaction-docker-meta@v1
+        with:
+          images: archivebox/archivebox,nikisweeting/archivebox
+          tag-sha: true
+          tag-semver: |
+            {{version}}
+            {{major}}.{{minor}}
+      
+      - name: Build and push
+        id: docker_build
+        uses: docker/build-push-action@v2
+        with:
+          context: ./
+          file: ./Dockerfile
+          builder: ${{ steps.buildx.outputs.name }}
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.docker_meta.outputs.tags }}
+          cache-from: type=local,src=/tmp/.buildx-cache
+          cache-to: type=local,dest=/tmp/.buildx-cache
+          platforms: linux/amd64,linux/386,linux/arm64,linux/arm/v7
+
+      - name: Image digest
+        run: echo ${{ steps.docker_build.outputs.digest }}

.github/workflows/homebrew.yml

@@ -0,0 +1,50 @@
+name: Build Homebrew package
+
+on:
+  workflow_dispatch:
+  push:
+
+
+jobs:
+  build:
+    runs-on: macos-latest
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      # TODO: modify archivebox.rb to update src url, hashes, and dependencies
+
+      - name: Build Homebrew Bottle
+        run: |
+          pip3 install --upgrade pip setuptools wheel
+          cd brew_dist/
+          brew install --build-bottle ./archivebox.rb
+          # brew bottle archivebox
+
+      - name: Add some links to test
+        run: |
+          mkdir data && cd data
+          archivebox init
+          archivebox add 'https://example.com'
+          archivebox version
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd brew_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Homebrew package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/homebrew-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: brew_dist
+
+      # TODO: push bottle homebrew core PR with latest changes

.github/workflows/lint.yml

@@ -0,0 +1,34 @@
+name: Run linters
+
+on:
+  workflow_dispatch:
+  push:
+
+env:
+  MAX_LINE_LENGTH: 110
+
+jobs:
+  lint:
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+
+      - name: Install flake8
+        run: |
+          pip install flake8
+
+      - name: Lint with flake8
+        run: |
+          # one pass for show-stopper syntax errors or undefined names
+          flake8 archivebox --count --show-source --statistics
+          # one pass for small stylistic things
+          flake8 archivebox --count --max-line-length="$MAX_LINE_LENGTH" --statistics

.github/workflows/pip.yml

@@ -0,0 +1,61 @@
+name: Build Pip package
+
+on:
+  workflow_dispatch:
+  push:
+
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+
+      - name: Build Python Package
+        run: |
+          pip3 install --upgrade pip setuptools wheel
+          rm -Rf pip_dist/*.whl
+          python3 setup.py \
+            sdist --dist-dir=./pip_dist \
+            bdist_wheel --dist-dir=./pip_dist \
+            egg_info --egg-base=./pip_dist
+          pip install pip_dist/archivebox-*.whl
+
+      - name: Add some links to test
+        run: |
+          mkdir data && cd data
+          archivebox init
+          archivebox add 'https://example.com'
+          archivebox version
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd pip_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Pip package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/pip-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: pip_dist
+
+      # - name: Push build to PyPI
+      #   run: |
+      #     cd pip_dist/
+      #     python3 -m twine upload --repository testpypi pip_dist/*.{whl,tar.gz}
+      #     python3 -m twine upload --repository pypi pip_dist/*.{whl,tar.gz}

.github/workflows/test.yml

@@ -1,43 +1,25 @@
-name: Test workflow
+name: Run tests
 on: [push]
 
 env:
-  MAX_LINE_LENGTH: 110
+  DOCKER_IMAGE: archivebox-ci
+  PYTHONIOENCODING: utf-8
+  PYTHONLEGACYWINDOWSSTDIO: utf-8
+  USE_COLOR: False
 
 jobs:
-  lint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Set up Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-          architecture: x64
-
-      - name: Install flake8
-        run: |
-          pip install flake8
-
-      - name: Lint with flake8
-        run: |
-          # one pass for show-stopper syntax errors or undefined names
-          flake8 archivebox --count --show-source --statistics
-          # one pass for small stylistic things
-          flake8 archivebox --count --max-line-length="$MAX_LINE_LENGTH" --statistics
-
-  test:
+  python_tests:
     runs-on: ${{ matrix.os }}
 
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
-        python: [3.7, 3.8]
+        os: [ubuntu-20.04, macos-latest, windows-latest]
+        python: [3.7]
 
     steps:
       - uses: actions/checkout@v2
         with:
+          submodules: true
           fetch-depth: 1
 
       ### Setup Python & JS Languages
@@ -69,9 +51,9 @@ jobs:
 
       - name: Install pip dependencies
         run: |
+          python -m pip install --upgrade pip setuptools wheel pytest bottle
+          ./bin/build_pip.sh
           python -m pip install .
-          python -m pip install pytest bottle
-
 
       - name: Get npm cache dir
         id: npm-cache
@@ -90,39 +72,46 @@ jobs:
       - name: Install npm requirements
         run: |
           npm install
-          echo "::set-env name=SINGLEFILE_BINARY::$GITHUB_WORKSPACE/node_modules/.bin/single-file"
-          echo "::set-env name=READABILITY_BINARY::$GITHUB_WORKSPACE/node_modules/.bin/readability-extractor"
-
+          echo "SINGLEFILE_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/single-file" >> $GITHUB_ENV
+          echo "READABILITY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/readability-extractor" >> $GITHUB_ENV
+          echo "MERCURY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/mercury-parser" >> $GITHUB_ENV
 
       ### Run the tests
       - name: Directory listing for debugging
         run: |
           pwd
-          ls -a ./
+          ls
+
+      - name: Archivebox version
+        run: |
           archivebox version
 
       - name: Test built package with pytest
+        # TODO: remove this exception for windows once we get tests passing on that platform
+        if: ${{ !contains(matrix.os, 'windows') }}
         run: |
-          python -m pytest -s
+          python -m pytest -s --ignore=archivebox/vendor
 
-  docker-test:
+  docker_tests:
     runs-on: ubuntu-latest
 
     steps:
       - uses: actions/checkout@v2
         with:
+          submodules: true
           fetch-depth: 1
 
-      - uses: satackey/action-docker-layer-caching@v0.0.4
+      # TODO: as of 2020-11 this helper layer broke, upgrade and re-enable this once it's usable again
+      # - uses: satackey/action-docker-layer-caching@v0.0.8
 
       - name: Build image
         run: |
-          docker build . -t archivebox
+          docker build . -t "$DOCKER_IMAGE"
 
       - name: Init data dir
         run: |
-          mkdir data
-          docker run -v "$PWD"/data:/data archivebox init
+          mkdir "${{ github.workspace }}/data"
+          docker run -v "${{ github.workspace }}/data":/data "$DOCKER_IMAGE" init
 
       - name: Run test server
         run: |
@@ -131,16 +120,16 @@ jobs:
 
       - name: Add link
         run: |
-          docker run -v "$PWD"/data:/data --network host archivebox add http://www.test-nginx-1.local
+          docker run -v "$PWD"/data:/data --network host "$DOCKER_IMAGE" add http://www.test-nginx-1.local
 
       - name: Add stdin link
         run: |
-          echo "http://www.test-nginx-2.local" | docker run -i -v "$PWD"/data:/data archivebox add
+          echo "http://www.test-nginx-2.local" | docker run -i --network host -v "$PWD"/data:/data "$DOCKER_IMAGE" add
 
       - name: List links
         run: |
-          docker run -v "$PWD"/data:/data archivebox list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
-          docker run -v "$PWD"/data:/data archivebox list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
+          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
+          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
 
       - name: Start docker-compose stack
         run: |
@@ -148,7 +137,7 @@ jobs:
           docker-compose up -d
           sleep 5
           curl --silent --location 'http://127.0.0.1:8000' | grep 'ArchiveBox'
-          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'django.jQuery'
+          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'window.django'
 
       - name: Check added urls show up in index
         run: |

.gitignore

@@ -3,13 +3,24 @@
 *.pyc
 __pycache__/
 .mypy_cache/
+.eggs/
+tests/out/
 
+# Python and Node dependencies
 venv/
 .venv/
 .docker-venv/
+node_modules/
 
+# Packaging artifacts
+archivebox.egg-info
+archivebox-*.tar.gz
 build/
 dist/
 
+# Data folders
 data/
+data1/
+data2/
+data3/
 output/

.gitmodules

@@ -1,3 +1,25 @@
 [submodule "docs"]
-	path = docs
-	url = https://github.com/pirate/ArchiveBox.wiki.git
+    path = docs
+    url = https://github.com/ArchiveBox/ArchiveBox.wiki.git
+
+[submodule "deb_dist"]
+    path = deb_dist
+    url = https://github.com/ArchiveBox/debian-archivebox.git
+[submodule "brew_dist"]
+	path = brew_dist
+	url = https://github.com/ArchiveBox/homebrew-archivebox.git
+[submodule "pip_dist"]
+	path = pip_dist
+	url = https://github.com/ArchiveBox/pip-archivebox.git
+[submodule "docker"]
+	path = docker
+	url = https://github.com/ArchiveBox/docker-archivebox.git
+[submodule "archivebox/vendor/base32-crockford"]
+	path = archivebox/vendor/base32-crockford
+	url = https://github.com/jbittel/base32-crockford
+[submodule "archivebox/vendor/pocket"]
+	path = archivebox/vendor/pocket
+	url = https://github.com/tapanpandita/pocket
+[submodule "archivebox/vendor/django-taggit"]
+	path = archivebox/vendor/django-taggit
+	url = https://github.com/jazzband/django-taggit

.npmignore

@@ -0,0 +1,19 @@
+tests/
+archivebox/
+archivebox.egg-info/
+build/
+dist/
+docs/
+etc/
+.github
+.gitmodules
+.dockerignore
+.flake8
+CNAME
+_config.yml
+docker-compose.yaml
+docker-compose.yml
+Dockerfile
+MANIFEST.in
+Pipfile
+setup.py

Dockerfile

@@ -7,13 +7,13 @@
 #     docker run -v "$PWD/data":/data -it archivebox manage createsuperuser
 #     docker run -v "$PWD/data":/data -p 8000:8000 archivebox server
 
-FROM python:3.8-slim-buster
+FROM python:3.9-slim-buster
 
 LABEL name="archivebox" \
     maintainer="Nick Sweeting <archivebox-docker@sweeting.me>" \
     description="All-in-one personal internet archiving container" \
-    homepage="https://github.com/pirate/ArchiveBox" \
-    documentation="https://github.com/pirate/ArchiveBox/wiki/Docker#docker"
+    homepage="https://github.com/ArchiveBox/ArchiveBox" \
+    documentation="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker"
 
 # System-level base config
 ENV TZ=UTC \
@@ -46,41 +46,49 @@ RUN apt-get update -qq \
 # Install apt dependencies
 RUN apt-get update -qq \
     && apt-get install -qq -y --no-install-recommends \
-        wget curl chromium git ffmpeg youtube-dl \
+        wget curl chromium git ffmpeg youtube-dl ripgrep \
         fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
     && rm -rf /var/lib/apt/lists/*
 
+# Install apt development dependencies
+# RUN apt-get install -qq \
+#     && apt-get install -qq -y --no-install-recommends \
+#         python3 python3-dev python3-pip python3-venv python3-all \
+#         dh-python debhelper devscripts dput software-properties-common \
+#         python3-distutils python3-setuptools python3-wheel python3-stdeb
+
 # Install Node environment
 RUN curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - \
-    && echo 'deb https://deb.nodesource.com/node_14.x buster main' >> /etc/apt/sources.list \
+    && echo 'deb https://deb.nodesource.com/node_15.x buster main' >> /etc/apt/sources.list \
     && apt-get update -qq \
     && apt-get install -qq -y --no-install-recommends \
         nodejs \
     && rm -rf /var/lib/apt/lists/*
 
-# Install Python dependencies
-WORKDIR "$CODE_DIR"
-ENV PATH="${PATH}:$VENV_PATH/bin"
-RUN python -m venv --clear --symlinks "$VENV_PATH" \
-    && pip install --upgrade --quiet pip setuptools
-ADD ./archivebox.egg-info/requires.txt "$CODE_DIR/archivebox.egg-info/requires.txt"
-RUN apt-get update -qq \
-    && apt-get install -qq -y --no-install-recommends \
-        build-essential python-dev python3-dev \
-    && grep -B 1000 -E '^$' "$CODE_DIR/archivebox.egg-info/requires.txt" | pip install --quiet -r /dev/stdin \
-    && apt-get purge -y build-essential python-dev python3-dev \
-    && apt-get autoremove -y \
-    && rm -rf /var/lib/apt/lists/*
-
 # Install Node dependencies
 WORKDIR "$NODE_DIR"
 ENV PATH="${PATH}:$NODE_DIR/node_modules/.bin" \
     npm_config_loglevel=error
-RUN npm install -g npm
 ADD ./package.json ./package.json
-RUN npm install
+ADD ./package-lock.json ./package-lock.json
+RUN npm ci
 
-# Install ArchiveBox Python package
+# Install Python dependencies
+WORKDIR "$CODE_DIR"
+ENV PATH="${PATH}:$VENV_PATH/bin"
+RUN python -m venv --clear --symlinks "$VENV_PATH" \
+    && pip install --upgrade --quiet pip setuptools
+ADD ./pip_dist/archivebox.egg-info/requires.txt "$CODE_DIR/pip_dist/archivebox.egg-info/requires.txt"
+RUN apt-get update -qq \
+    && apt-get install -qq -y --no-install-recommends \
+        build-essential python-dev python3-dev \
+    && grep -B 1000 -E '^$' "$CODE_DIR/pip_dist/archivebox.egg-info/requires.txt" | pip install --quiet -r /dev/stdin \
+    && pip install --quiet "sonic-client==0.0.5" \
+    && apt-get purge -y build-essential python-dev python3-dev \
+    && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install ArchiveBox Python package and its dependencies
 WORKDIR "$CODE_DIR"
 ADD . "$CODE_DIR"
 RUN pip install -e .
@@ -93,10 +101,13 @@ ENV IN_DOCKER=True \
     USE_SINGLEFILE=True \
     SINGLEFILE_BINARY="$NODE_DIR/node_modules/.bin/single-file" \
     USE_READABILITY=True \
-    READABILITY_BINARY="$NODE_DIR/node_modules/.bin/readability-extractor"
+    READABILITY_BINARY="$NODE_DIR/node_modules/.bin/readability-extractor" \
+    USE_MERCURY=True \
+    MERCURY_BINARY="$NODE_DIR/node_modules/.bin/mercury-parser"
 
 # Print version for nice docker finish summary
-RUN archivebox version
+# RUN archivebox version
+RUN /app/bin/docker_entrypoint.sh archivebox version
 
 # Open up the interfaces to the outside world
 VOLUME "$DATA_DIR"

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 Nick Sweeting
+Copyright (c) 2020 Nick Sweeting
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

MANIFEST.in

@@ -1,5 +1,4 @@
-include LICENSE
-include README.md
-include package.json
-include package-lock.json
-recursive-include archivebox/themes *
+graft archivebox
+global-exclude .DS_Store
+global-exclude __pycache__
+global-exclude *.pyc

README.md

@@ -1,14 +1,14 @@
 <div align="center">
-<img src="https://i.imgur.com/4nkFjdv.png" height="80px">
-<h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>
+<em><img src="https://i.imgur.com/5B48E3N.png" height="90px"></em>
+<h1>ArchiveBox<br/><sub>Open-source self-hosted web archiving.</sub></h1>
 
-▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
-<a href="https://archivebox.zervice.io/">Demo</a> | 
-<a href="https://github.com/pirate/ArchiveBox">Github</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
-<a href="#background--motivation">Info & Motivation</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community">Community</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>
+▶️ <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> |
+<a href="https://archivebox.zervice.io/">Demo</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox">Github</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Documentation</a> |
+<a href="#background--motivation">Info & Motivation</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap">Roadmap</a>
 
 <pre>
 "Your own personal internet archive" (网站存档 / 爬虫)
@@ -16,169 +16,452 @@
 
 <!--<a href="http://webchat.freenode.net?channels=ArchiveBox&uio=d4"><img src="https://img.shields.io/badge/Community_chat-IRC-%2328A745.svg"/></a>-->
 
-<a href="https://github.com/pirate/ArchiveBox/blob/master/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a>
-<a href="https://github.com/pirate/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a>
-<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a>
 <a href="https://test.pypi.org/project/archivebox/"><img src="https://img.shields.io/badge/Python-%3E%3D3.7-yellow.svg?logo=python&logoColor=yellow"/></a>
-<a href="https://github.com/pirate/ArchiveBox/wiki/Install#dependencies"><img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/></a>
-<a href="https://hub.docker.com/r/nikisweeting/archivebox"><img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies"><img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/></a>
+<a href="https://hub.docker.com/r/archivebox/archivebox"><img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/></a><br/>
+<a href="https://github.com/ArchiveBox/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/ArchiveBox/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=active"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/context:python"><img alt="Language grade: Python" src="https://img.shields.io/lgtm/grade/python/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/context:javascript"><img alt="Language grade: JavaScript" src="https://img.shields.io/lgtm/grade/javascript/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/alerts/"><img alt="Total alerts" src="https://img.shields.io/lgtm/alerts/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+
+
 <hr/>
 </div>
 
-**ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).**
+ArchiveBox is a powerful self-hosted internet archiving solution written in Python. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on setup and content within.
 
-You can use it to preserve access to websites you care about by storing them locally offline. ArchiveBox imports lists of URLs, renders the pages in a headless, authenticated, user-scriptable browser, and then archives the content in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the originals disappear off the internet. It automatically extracts assets and media from pages and saves them in easily-accessible folders, with out-of-the-box support for extracting git repositories, audio, video, subtitles, images, PDFs, and more.
-
-#### How does it work?
+**🔢&nbsp; Run ArchiveBox via [Docker Compose (recommended)](#Quickstart), Docker, Apt, Brew, or Pip ([see below](#Quickstart)).**
 
 ```bash
-mkdir data && cd data
-archivebox init
-archivebox add 'https://example.com'
-archivebox add 'https://getpocket.com/users/USERNAME/feed/all' --depth=1
-archivebox server
+apt/brew/pip3 install archivebox
+
+archivebox init                       # run this in an empty folder
+archivebox add 'https://example.com'  # start adding URLs to archive
+curl https://example.com/rss.xml | archivebox add  # or add via stdin
+archivebox schedule --every=day https://example.com/rss.xml
 ```
 
-After installing archivebox, just pass some new links to the `archivebox add` command to start your collection.
+For each URL added, ArchiveBox saves several types of HTML snapshot (wget, Chrome headless, singlefile), a PDF, a screenshot, a WARC archive, any git repositories, images, audio, video, subtitles, article text, [and more...](#output-formats).
 
-ArchiveBox is written in Python 3.7 and uses wget, Chrome headless, youtube-dl, pywb, and other common UNIX tools to save each page you add in multiple redundant formats. It doesn't require a constantly running server or backend (though it does include an optional one), just open the generated `data/index.html` in a browser to view the archive or run `archivebox server` to use the interactive Web UI. It can import and export links as JSON (among other formats), so it's easy to script or hook up to other APIs. If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
+```bash
+archivebox server --createsuperuser 0.0.0.0:8000   # use the interactive web UI
+archivebox list 'https://example.com'  # use the CLI commands (--help for more)
+ls ./archive/*/index.json              # or browse directly via the filesystem
+```
+
+You can then manage your snapshots via the [filesystem](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout), [CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [SQLite DB](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py) (`./index.sqlite3`), [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (alpha), or [desktop app](https://github.com/ArchiveBox/electron-archivebox) (alpha).
+
+At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
 
 <div align="center">
+<br/><br/>
+<img src="https://i.imgur.com/PAzXZE8.png" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://i.imgur.com/asPNk8n.png" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://i.imgur.com/PAzXZE8.png" height="70px" alt="bookshelf graphic">
+<br/><br/>
+</div>
 
-<img src="https://i.imgur.com/3tBL7PU.png" width="22%" alt="CLI Screenshot" align="top">
-<img src="https://i.imgur.com/viklZNG.png" width="22%" alt="Desktop index screenshot" align="top">
-<img src="https://i.imgur.com/RefWsXB.jpg" width="22%" alt="Desktop details page Screenshot"/>
-<img src="https://i.imgur.com/M6HhzVx.png" width="22%" alt="Desktop details page Screenshot"/><br/>
-<sup><a href="https://archive.sweeting.me/">Demo</a> | <a href="https://github.com/pirate/ArchiveBox/wiki/Usage">Usage</a> | <a href="#screenshots">Screenshots</a></sup>
+#### ⚡️&nbsp; CLI Usage
+
+```bash
+# archivebox [subcommand] [--args]
+archivebox --version
+archivebox help
+```
+
+- `archivebox init/version/status/config/manage` to administer your collection
+- `archivebox add/remove/update/list` to manage Snapshots in the archive
+- `archivebox schedule` to pull in fresh URLs in regularly from [boorkmarks/history/Pocket/Pinboard/RSS/etc.](#input-formats)
+- `archivebox oneshot` archive single URLs without starting a whole collection
+- `archivebox shell/manage dbshell` open a REPL to use the [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha), or SQL API
+
+<div align="center">
+<br/>
+<sup><a href="https://archivebox.zervice.io/">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></sup>
 <br/>
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
-</div><br/>
+<br/><br/>
+<img src="https://i.imgur.com/njxgSbl.png" width="22%" alt="cli init screenshot" align="top">
+<img src="https://i.imgur.com/p6wK6KM.png" width="22%" alt="server snapshot admin screenshot" align="top">
+<img src="https://i.imgur.com/RefWsXB.jpg" width="28.6%" alt="server snapshot details page screenshot" align="top"/>
+<br/>
+<br/>
+<img src="https://i.imgur.com/T2UAGUD.png" width="49%" alt="grass"/><img src="https://i.imgur.com/T2UAGUD.png" width="49%" alt="grass"/>
+</div>
 
-## Quickstart
 
-ArchiveBox is written in `python3.7` and has [3 main binary dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies): `wget`, `chromium`, and `youtube-dl`.
-To get started, you can [install them manually](https://github.com/pirate/ArchiveBox/wiki/Install) using your system's package manager, use the [automated helper script](https://github.com/pirate/ArchiveBox/wiki/Quickstart), or use the official [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker) container. All three dependencies are optional if [disabled](https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles) in settings.
+### Quickstart
 
-```bash
-# Docker
-mkdir data && cd data
-docker run -v $PWD:/data nikisweeting/archivebox init
-docker run -v $PWD:/data nikisweeting/archivebox add 'https://example.com'
-docker run -v $PWD:/data -it nikisweeting/archivebox manage createsuperuser
-docker run -v $PWD:/data -p 8000:8000 nikisweeting/archivebox server 0.0.0.0:8000
-open http://127.0.0.1:8000
-```
+**🖥&nbsp; Supported OSs:** Linux/BSD, macOS, Windows  &nbsp; &nbsp; **🎮&nbsp; CPU Architectures:** x86, amd64, arm7, arm8 (raspi >=3)
+**📦&nbsp; Distributions:** `docker`/`apt`/`brew`/`pip3`/`npm` (in order of completeness)
 
-```bash
-# Docker Compose
-# first download: https://github.com/pirate/ArchiveBox/blob/master/docker-compose.yml
+*(click to expand your preferred **► `distribution`** below for full setup instructions)*
+
+<details>
+<summary><b>Get ArchiveBox with <code>docker-compose</code> on any platform (recommended, everything included out-of-the-box)</b></summary>
+
+<i>First make sure you have Docker installed: https://docs.docker.com/get-docker/</i>
+
+<pre lang="bash"><code>
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+curl -O 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml'
 docker-compose run archivebox init
-docker-compose run archivebox add 'https://example.com'
+docker-compose run archivebox --version
+
+# start the webserver and open the UI (optional)
 docker-compose run archivebox manage createsuperuser
-docker-compose up
+docker-compose up -d
+open 'http://127.0.0.1:8000'
+
+# you can also add links and manage your archive via the CLI:
+docker-compose run archivebox add 'https://example.com'
+docker-compose run archivebox status
+docker-compose run archivebox help  # to see more options
+</code></pre>
+
+This is the recommended way to run ArchiveBox because it includes <i>all</i> the extractors like:<br/>
+chrome, wget, youtube-dl, git, etc., full-text search w/ sonic, and many other great features.
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>docker</code> on any platform</b></summary>
+
+<i>First make sure you have Docker installed: https://docs.docker.com/get-docker/</i>
+
+<pre lang="bash"><code>
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+docker run -v $PWD:/data -it archivebox/archivebox init
+docker run -v $PWD:/data -it archivebox/archivebox --version
+
+# start the webserver and open the UI (optional)
+docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox server --createsuperuser 0.0.0.0:8000
 open http://127.0.0.1:8000
-```
 
-```bash
-# Bare Metal
-# Use apt on Ubuntu/Debian, brew on mac, or pkg on BSD
-apt install python3 python3-pip git curl wget youtube-dl chromium-browser
+# you can also add links and manage your archive via the CLI:
+docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
+docker run -v $PWD:/data -it archivebox/archivebox status
+docker run -v $PWD:/data -it archivebox/archivebox help  # to see more options
+</code></pre>
 
-pip install archivebox      # install archivebox
+</details>
 
-mkdir data && cd data       # (doesn't have to be called data)
+<details>
+<summary><b>Get ArchiveBox with <code>apt</code> on Ubuntu >=20.04</b></summary>
+
+<i>First make sure you're on Ubuntu >= 20.04, or scroll down for older/non-Ubuntu instructions.</i>
+
+<pre lang="bash"><code>
+# add the repo to your sources and install the archivebox package using apt
+sudo apt install software-properties-common
+sudo add-apt-repository -u ppa:archivebox/archivebox
+sudo apt install archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
 archivebox init
-archivebox add 'https://example.com'  # add URLs via args or stdin
+archivebox --version
 
-# or import an RSS/JSON/XML/TXT feed/list of links
-archivebox add https://getpocket.com/users/USERNAME/feed/all --depth=1
-```
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
 
-Once you've added your first links, open `data/index.html` in a browser to view the static archive.
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
 
-You can also start it as a server with a full web UI to manage your links:
-```bash
-archivebox manage createsuperuser
-archivebox server
-```
+For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`:
 
-You can visit `http://127.0.0.1:8000` in your browser to access it.
+<pre lang="bash"><code>
+deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main
+deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main
+</code></pre>
 
-[DEMO: archivebox.zervice.io/](https://archivebox.zervice.io)  
-For more information, see the [full Quickstart guide](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.
+Then run `apt update; apt install archivebox; archivebox --version`.
+
+(you may need to install some other dependencies manually however)
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>brew</code> on macOS >=10.13</b></summary>
+
+<i>First make sure you have Homebrew installed: https://brew.sh/#install</i>
+
+<pre lang="bash"><code>
+# install the archivebox package using homebrew
+brew install archivebox/archivebox/archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
+archivebox init
+archivebox --version
+
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
+
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>pip</code> on any platform</b></summary>
+
+<i>First make sure you have Python >= 3.7 installed: https://realpython.com/installing-python/</i>
+
+<pre lang="bash"><code>
+# install the archivebox package using pip3
+pip3 install archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
+archivebox init
+archivebox --version
+# Install any missing extras like wget/git/chrome/etc. manually as needed
+
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
+
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
+
+</details>
+
+No matter which install method you choose, they all roughly follow this 3-step process and all provide the same CLI, Web UI, and on-disk data format.
+
+<small>
+
+1. Install ArchiveBox: `apt/brew/pip3 install archivebox`
+2. Start a collection: `archivebox init`
+3. Start archiving: `archivebox add 'https://example.com'`
+
+</small>
+
+<br/>
+<div align="center">
+<img src="https://i.imgur.com/6AmOGJT.png" width="49%" alt="grass"/><img src="https://i.imgur.com/6AmOGJT.png" width="49%" alt="grass"/>
+</div>
+<br/>
+
+<div align="center">
+<img src="https://i.imgur.com/lUuicew.png" width="22.4%" align="top">
+<img src="https://i.imgur.com/p6wK6KM.png" width="35.9%" align="top">
+<img src="https://i.imgur.com/pzq4uXq.png" width="29.7%" align="top">
+<br/><br/>
+<sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
+<br/><br/>
+<a href="https://archivebox.zervice.io">DEMO: <code>https://archivebox.zervice.io</code></a><br/>
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a>
+<br/>
+</div>
+
+## Key Features
+
+- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally
+- [**Powerful, intuitive command line interface**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) with [modular optional dependencies](#dependencies) 
+- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (youtube-dl), articles (readability), code (git), etc.](#output-formats)
+- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
+- [**Uses standard, durable, long-term formats**](#saves-lots-of-useful-stuff-for-each-imported-link) like HTML, JSON, PDF, PNG, and WARC
+- [**Usable as a oneshot CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/latest/modules.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox) (ALPHA)
+- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
+- Planned: support for archiving [content requiring a login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (working, but ill-advised until some pending fixes are released)
+- Planned: support for running [JS scripts during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hiding](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expander](https://github.com/ArchiveBox/ArchiveBox/issues/345), etc.
+
+<br/>
 
 ---
 
 <div align="center">
-<img src="https://i.imgur.com/PVO88AZ.png" width="80%"/>
+<img src="https://i.imgur.com/OUmgdlH.png" width="96%" alt="lego">
 </div>
 
-# Overview
+## Input formats
 
-Because modern websites are complicated and often rely on dynamic content,
-ArchiveBox archives the sites in **several different formats** beyond what public
-archiving services like Archive.org and Archive.is are capable of saving. Using multiple
-methods and the market-dominant browser to execute JS ensures we can save even the most
-complex, finicky websites in at least a few high-quality, long-term data formats.
-
-ArchiveBox imports a list of URLs from stdin, remote URL, or file, then adds the pages to a local archive folder using wget to create a browsable HTML clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...
-
-Running `archivebox add` adds only new, unique links into your collection on each run. Because it will ignore duplicates and only archive each link the first time you add it, you can schedule it to [run on a timer](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) and re-import all your feeds multiple times a day. It will run quickly even if the feeds are large, because it's only archiving the newest links since the last run. For each link, it runs through all the archive methods. Methods that fail will save `None` and be automatically retried on the next run, methods that succeed save their output into the data folder and are never retried/overwritten by subsequent runs. Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/pirate/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs).
-
-All the archived links are stored by date bookmarked in `./archive/<timestamp>`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM.
-
-#### Can import links from many formats:
+ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more!
 
 ```bash
 echo 'http://example.com' | archivebox add
 archivebox add 'https://example.com/some/page'
 archivebox add < ~/Downloads/firefox_bookmarks_export.html
-archivebox add --depth=1 'https://example.com/some/rss/feed.xml'
+archivebox add < any_text_with_urls_in_it.txt
+archivebox add --depth=1 'https://example.com/some/downloads.html'
 archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
 ```
 
-- <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more)
-- <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format
-- <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more
 
-See the [Usage: CLI](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
+- <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> TXT, RSS, XML, JSON, CSV, SQL, HTML, Markdown, or [any other text-based format...](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file)
+- <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> [Browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (see instructions for: [Chrome](https://support.google.com/chrome/answer/96816?hl=en), [Firefox](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer), [Safari](http://i.imgur.com/AtcvUZA.png), [IE](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows), [Opera](http://help.opera.com/Windows/12.10/en/importexport.html), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive))
+- <img src="https://getpocket.com/favicon.ico" height="22px"/> [Pocket](https://getpocket.com/export), [Pinboard](https://pinboard.in/export/), [Instapaper](https://www.instapaper.com/user/export), [Shaarli](https://shaarli.readthedocs.io/en/master/Usage/#importexport), [Delicious](https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/), [Reddit Saved](https://github.com/csu/export-saved-reddit), [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html), [Unmark.it](http://help.unmark.it/import-export), [OneTab](https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
 
-#### Saves lots of useful stuff for each imported link:
+See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
+
+It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand.
+
+## Output formats
+
+All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`.
+
+The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.)
 
 ```bash
  ls ./archive/<timestamp>/
 ```
 
 - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details
-- **Title:** `title` title of the site
-- **Favicon:** `favicon.ico` favicon of the site
-- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present
-- **WARC:** `warc/<timestamp>.gz` gzipped WARC of all the resources fetched while archiving
-- **PDF:** `output.pdf` Printed PDF of site using headless chrome
-- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome
-- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome
-- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org
+- **Title**, **Favicon**, **Headers** Response headers, site favicon, and parsed site title
+- **Wget Clone:** `example.com/page-name.html` wget clone of the site with  `warc/<timestamp>.gz`
+- Chrome Headless
+  - **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile
+  - **PDF:** `output.pdf` Printed PDF of site using headless chrome
+  - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome
+  - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome
+  - **Readability:** `article.html/json` Article text extraction using Readability
+- **Archive.org Permalink:** `archive.org.txt` A link to the saved site on archive.org
 - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl
 - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links
-- _More coming soon! See the [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)..._
+- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._
 
-It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/pirate/ArchiveBox/wiki/Configuration) via environment variables or config file.
+It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file.
 
-If you're importing URLs with secret tokens in them (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of these methods to avoid leaking private URLs to 3rd party APIs during the archiving process. See the [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details.
+<div align="center">
+<img src="https://i.imgur.com/ucyimDX.png" width="96%" alt="lego graphic">
+</div>
 
-## Key Features
+<br/>
 
-- [**Free & open source**](https://github.com/pirate/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally
-- [**Few dependencies**](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage)
-- [**Comprehensive documentation**](https://github.com/pirate/ArchiveBox/wiki), [active development](https://github.com/pirate/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-- **Doesn't require a constantly-running server**, proxy, or native app
-- Easy to set up **[scheduled importing](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources**
-- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC
-- ~~**Suitable for paywalled / [authenticated content](https://github.com/pirate/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes)
-- Can [**run scripts during archiving**](https://github.com/pirate/ArchiveBox/issues/51) to [scroll pages](https://github.com/pirate/ArchiveBox/issues/80), [close modals](https://github.com/pirate/ArchiveBox/issues/175), expand comment threads, etc.
-- Can also [**mirror content to 3rd-party archiving services**](https://github.com/pirate/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy
+---
 
-## Background & Motivation
+<br/>
+
+## Dependencies
+
+You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled.
+
+If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) or by running the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart).
+
+ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more.
+
+<br/>
+
+---
+
+<div align="center">
+<img src="https://docs.monadical.com/uploads/upload_b6900afc422ae699bfefa2dcda3306f3.png" width="100%" alt="security graphic"/>
+</div>
+
+## Caveats
+
+If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process.
+
+```bash
+# don't do this:
+archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere'
+archivebox add 'https://example.com/any/url/you/want/to/keep/secret/'
+
+# without first disabling share the URL with 3rd party APIs:
+archivebox config --set SAVE_ARCHIVE_DOT_ORG=False   # disable saving all URLs in Archive.org
+archivebox config --set SAVE_FAVICON=False      # optional: only the domain is leaked, not full URL
+archivebox config --set CHROME_BINARY=chromium  # optional: switch to chromium to avoid Chrome phoning home to Google
+```
+
+Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details.
+
+```bash
+# visiting an archived page with malicious JS:
+https://127.0.0.1:8000/archive/1602401954/example.com/index.html
+
+# example.com/index.js can now make a request to read everything:
+https://127.0.0.1:8000/index.html
+https://127.0.0.1:8000/archive/*
+# then example.com/index.js can send it off to some evil server
+```
+
+Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash:
+
+```bash
+archivebox add 'https://example.com#2020-10-24'
+...
+archivebox add 'https://example.com#2020-10-25'
+```
+
+<br/>
+
+---
+
+<br/>
+
+## Screenshots
+
+<div align="center" width="80%">
+<img src="https://i.imgur.com/PVO88AZ.png" width="80%"/>
+<table>
+<tbody>
+<tr>
+<td>
+<img src="https://i.imgur.com/npareKG.png" alt="brew install archivebox"><br/>
+<img src="https://i.imgur.com/5vSBO2R.png" alt="archivebox version">
+</td>
+<td>
+<img src="https://i.imgur.com/JXXxFzB.png" alt="archivebox init"><br/>
+</td>
+<td>
+<img src="https://i.imgur.com/wNYtV3v.jpg" alt="archivebox add">
+</td>
+<td>
+<img src="https://i.imgur.com/uZcIOn9.png" alt="archivebox data dir">
+</td>
+</tr>
+<tr>
+<td>
+<img src="https://i.imgur.com/H08eaia.png" alt="archivebox server">
+</td>
+<td>
+<img src="https://i.imgur.com/zM4z1aU.png" alt="archivebox server add">
+</td>
+<td>
+<img src="https://i.imgur.com/p6wK6KM.png" alt="archivebox server list">
+</td>
+<td>
+<img src="https://i.imgur.com/pzq4uXq.png" alt="archivebox server detail">
+</td>
+</tr>
+</tbody>
+</table>
+</div>
+<br/>
+
+---
+
+<br/>
+
+<div align="center">
+<img src="https://i.imgur.com/ZSUm9mr.png" width="100%" alt="paisley graphic">
+</div>
+
+# Background & Motivation
 
 Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.
 
@@ -194,15 +477,20 @@ archive internet content enables to you save the stuff you care most about befor
 The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful.
 I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.
 
+Because modern websites are complicated and often rely on dynamic content,
+ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.
+
+All the archived links are stored by date bookmarked in `./archive/<timestamp>`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM.
+
 ## Comparison to Other Projects
 
-▶ **Check out our [community page](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.**
+▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.**
 
-<img src="https://i.imgur.com/4nkFjdv.png" width="10%" align="left"/> The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
+<img src="https://i.imgur.com/4nkFjdv.png" width="10%" align="left" alt="comparison"/> The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
 
 #### User Interface & Intended Purpose
 
-ArchiveBox differentiates itself from [similar projects](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
+ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
 
 #### Private Local Archives vs Centralized Public Archives
 
@@ -212,73 +500,206 @@ Unlike crawler software that starts from a seed URL and works outwards, or publi
 
 Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files.
 
-## Learn more
+<div align="center">
+<br/>
+<img src="https://i.imgur.com/q0Oe36M.png" width="100%" alt="dependencies graphic">
+</div>
 
-<!--▶ **Join out our [community chat](http://webchat.freenode.net?channels=ArchiveBox&uio=d4) hosted on IRC freenode.net:`#ArchiveBox`!**-->
+## Learn more
 
 Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web!
 
 <img src="https://i.imgur.com/0ZOmOvN.png" width="14%" align="right"/>
 
-- [Community Wiki](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-  - [The Master Lists](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists)  
+- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+  - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#the-master-lists)  
     _Community-maintained indexes of archiving tools and institutions._
-  - [Web Archiving Software](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects)  
+  - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects)  
     _Open source tools and projects in the internet archiving space._
-  - [Reading List](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Reading-List)  
+  - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#reading-list)  
     _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
-  - [Communities](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Communities)  
+  - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#communities)  
     _A collection of the most active internet archiving communities and initiatives._
-- Check out the ArchiveBox [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
+- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
 - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
-- Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+
+<br/>
 
 ---
 
+<div align="center">
+<img src="https://i.imgur.com/SMkGW0L.png" width="100%" alt="documentation graphic">
+</div>
+
 # Documentation
 
 <img src="https://read-the-docs-guidelines.readthedocs-hosted.com/_images/logo-dark.png" width="13%" align="right"/>
 
-We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) for documentation.
+We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation.
 
-You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.
-
-You can build the docs by running:
-
-```python
-cd ArchiveBox
-pipenv install --dev
-sphinx-apidoc -o docs archivebox
-cd docs/
-make html
-# then open docs/_build/html/index.html
-```
+You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder.
 
 ## Getting Started
 
-- [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
-- [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
-- [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)
+- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart)
+- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install)
+- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker)
 
 ## Reference
 
-- [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
-- [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
-- [Supported Sources](https://github.com/pirate/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
-- [Supported Outputs](https://github.com/pirate/ArchiveBox/wiki#can-save-these-things-for-each-site)
-- [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
-- [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
-- [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Install-Chromium)
-- [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview)
-- [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)
+- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage)
+- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)
+- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
+- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site)
+- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving)
+- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive)
+- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install)
+- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview)
+- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
+- [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha)
+- [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (alpha)
 
 ## More Info
 
-- [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
-- [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
-- [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
-- [Background & Motivation](https://github.com/pirate/ArchiveBox#background--motivation)
-- [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
+- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues)
+- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)
+- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
+- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation)
+- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+
+<br/>
+
+---
+
+<div align="center">
+<img src="https://i.imgur.com/EGWjbD4.png" width="100%" alt="development">
+</div>
+
+# ArchiveBox Development
+
+All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap.
+
+Low hanging fruit / easy first tickets:<br/>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/alerts/"><img alt="Total alerts" src="https://img.shields.io/lgtm/alerts/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+
+### Setup the dev environment
+
+#### 1. Clone the main code repo (making sure to pull the submodules as well)
+
+```bash
+git clone --recurse-submodules https://github.com/ArchiveBox/ArchiveBox
+cd ArchiveBox
+git checkout dev  # or the branch you want to test
+git submodule update --init --recursive
+git pull --recurse-submodules
+```
+
+#### 2. Option A: Install the Python, JS, and system dependencies directly on your machine
+
+```bash
+# Install ArchiveBox + python dependencies
+python3 -m venv .venv && source .venv/bin/activate && pip install -e '.[dev]'
+# or: pipenv install --dev && pipenv shell
+
+# Install node dependencies
+npm install
+
+# Check to see if anything is missing
+archivebox --version
+# install any missing dependencies manually, or use the helper script:
+./bin/setup.sh
+```
+
+#### 2. Option B: Build the docker container and use that for development instead
+
+```bash
+# Optional: develop via docker by mounting the code dir into the container
+# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
+# inside the container will reload and pick up your changes
+docker build . -t archivebox
+docker run -it --rm archivebox version
+docker run -it --rm -p 8000:8000 \
+    -v $PWD/data:/data \
+    -v $PWD/archivebox:/app/archivebox \
+    archivebox server 0.0.0.0:8000 --debug --reload
+```
+
+### Common development tasks
+
+See the `./bin/` folder and read the source of the bash scripts within.
+You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`.
+
+#### Run in DEBUG mode
+
+```bash
+archivebox config --set DEBUG=True
+# or
+archivebox server --debug ...
+```
+
+#### Build and run a Github branch
+
+```bash
+docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
+docker run -it -v $PWD:/data archivebox:dev ...
+```
+
+#### Run the linters
+
+```bash
+./bin/lint.sh
+```
+(uses `flake8` and `mypy`)
+
+#### Run the integration tests
+
+```bash
+./bin/test.sh
+```
+(uses `pytest -s`)
+
+#### Make migrations or enter a django shell
+
+Make sure to run this whenever you change things in `models.py`.
+```bash
+cd archivebox/
+./manage.py makemigrations
+
+cd path/to/test/data/
+archivebox shell
+archivebox manage dbshell
+```
+(uses `pytest -s`)
+
+#### Build the docs, pip package, and docker image
+
+(Normally CI takes care of this, but these scripts can be run to do it manually)
+```bash
+./bin/build.sh
+
+# or individually:
+./bin/build_docs.sh
+./bin/build_pip.sh
+./bin/build_deb.sh
+./bin/build_brew.sh
+./bin/build_docker.sh
+```
+
+#### Roll a release
+
+(Normally CI takes care of this, but these scripts can be run to do it manually)
+```bash
+./bin/release.sh
+
+# or individually:
+./bin/release_docs.sh
+./bin/release_pip.sh
+./bin/release_deb.sh
+./bin/release_brew.sh
+./bin/release_docker.sh
+```
 
 ---
 
@@ -296,8 +717,8 @@ make html
 <a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
 <br/>
 
-<a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
-<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
+<a href="https://twitter.com/ArchiveBoxApp"><img src="https://img.shields.io/badge/Tweet-%40ArchiveBoxApp-blue.svg?style=flat"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
 
 <br/><br/>
 

_config.yml

@@ -1 +1 @@
-theme: jekyll-theme-minimal
+theme: jekyll-theme-merlot

archivebox.egg-info/PKG-INFO

@@ -1,352 +0,0 @@
-Metadata-Version: 2.1
-Name: archivebox
-Version: 0.4.18
-Summary: The self-hosted internet archive.
-Home-page: https://github.com/pirate/ArchiveBox
-Author: Nick Sweeting
-Author-email: git@nicksweeting.com
-License: MIT
-Project-URL: Source, https://github.com/pirate/ArchiveBox
-Project-URL: Documentation, https://github.com/pirate/ArchiveBox/wiki
-Project-URL: Bug Tracker, https://github.com/pirate/ArchiveBox/issues
-Project-URL: Changelog, https://github.com/pirate/ArchiveBox/wiki/Changelog
-Project-URL: Roadmap, https://github.com/pirate/ArchiveBox/wiki/Roadmap
-Project-URL: Community, https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community
-Project-URL: Donate, https://github.com/pirate/ArchiveBox/wiki/Donations
-Description: <div align="center">
-        <img src="https://i.imgur.com/4nkFjdv.png" height="80px">
-        <h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>
-        
-        ▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
-        <a href="https://archivebox.zervice.io/">Demo</a> | 
-        <a href="https://github.com/pirate/ArchiveBox">Github</a> | 
-        <a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
-        <a href="#background--motivation">Info & Motivation</a> | 
-        <a href="https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community">Community</a> | 
-        <a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>
-        
-        <pre>
-        "Your own personal internet archive" (网站存档 / 爬虫)
-        </pre>
-        
-        <!--<a href="http://webchat.freenode.net?channels=ArchiveBox&uio=d4"><img src="https://img.shields.io/badge/Community_chat-IRC-%2328A745.svg"/></a>-->
-        
-        <a href="https://github.com/pirate/ArchiveBox/blob/master/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a>
-        <a href="https://github.com/pirate/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a>
-        <a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a>
-        <a href="https://test.pypi.org/project/archivebox/"><img src="https://img.shields.io/badge/Python-%3E%3D3.7-yellow.svg?logo=python&logoColor=yellow"/></a>
-        <a href="https://github.com/pirate/ArchiveBox/wiki/Install#dependencies"><img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/></a>
-        <a href="https://hub.docker.com/r/nikisweeting/archivebox"><img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/></a>
-        <hr/>
-        </div>
-        
-        **ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).**
-        
-        You can use it to preserve access to websites you care about by storing them locally offline. ArchiveBox imports lists of URLs, renders the pages in a headless, authenticated, user-scriptable browser, and then archives the content in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the originals disappear off the internet. It automatically extracts assets and media from pages and saves them in easily-accessible folders, with out-of-the-box support for extracting git repositories, audio, video, subtitles, images, PDFs, and more.
-        
-        #### How does it work?
-        
-        ```bash
-        mkdir data && cd data
-        archivebox init
-        archivebox add 'https://example.com'
-        archivebox add 'https://getpocket.com/users/USERNAME/feed/all' --depth=1
-        archivebox server
-        ```
-        
-        After installing archivebox, just pass some new links to the `archivebox add` command to start your collection.
-        
-        ArchiveBox is written in Python 3.7 and uses wget, Chrome headless, youtube-dl, pywb, and other common UNIX tools to save each page you add in multiple redundant formats. It doesn't require a constantly running server or backend (though it does include an optional one), just open the generated `data/index.html` in a browser to view the archive or run `archivebox server` to use the interactive Web UI. It can import and export links as JSON (among other formats), so it's easy to script or hook up to other APIs. If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
-        
-        <div align="center">
-        
-        <img src="https://i.imgur.com/3tBL7PU.png" width="22%" alt="CLI Screenshot" align="top">
-        <img src="https://i.imgur.com/viklZNG.png" width="22%" alt="Desktop index screenshot" align="top">
-        <img src="https://i.imgur.com/RefWsXB.jpg" width="22%" alt="Desktop details page Screenshot"/>
-        <img src="https://i.imgur.com/M6HhzVx.png" width="22%" alt="Desktop details page Screenshot"/><br/>
-        <sup><a href="https://archive.sweeting.me/">Demo</a> | <a href="https://github.com/pirate/ArchiveBox/wiki/Usage">Usage</a> | <a href="#screenshots">Screenshots</a></sup>
-        <br/>
-        <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
-        </div><br/>
-        
-        ## Quickstart
-        
-        ArchiveBox is written in `python3.7` and has [3 main binary dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies): `wget`, `chromium`, and `youtube-dl`.
-        To get started, you can [install them manually](https://github.com/pirate/ArchiveBox/wiki/Install) using your system's package manager, use the [automated helper script](https://github.com/pirate/ArchiveBox/wiki/Quickstart), or use the official [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker) container. All three dependencies are optional if [disabled](https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles) in settings.
-        
-        ```bash
-        # Docker
-        mkdir data && cd data
-        docker run -v $PWD:/data nikisweeting/archivebox init
-        docker run -v $PWD:/data nikisweeting/archivebox add 'https://example.com'
-        docker run -v $PWD:/data -it nikisweeting/archivebox manage createsuperuser
-        docker run -v $PWD:/data -p 8000:8000 nikisweeting/archivebox server 0.0.0.0:8000
-        open http://127.0.0.1:8000
-        ```
-        
-        ```bash
-        # Docker Compose
-        # first download: https://github.com/pirate/ArchiveBox/blob/master/docker-compose.yml
-        docker-compose run archivebox init
-        docker-compose run archivebox add 'https://example.com'
-        docker-compose run archivebox manage createsuperuser
-        docker-compose up
-        open http://127.0.0.1:8000
-        ```
-        
-        ```bash
-        # Bare Metal
-        # Use apt on Ubuntu/Debian, brew on mac, or pkg on BSD
-        apt install python3 python3-pip git curl wget youtube-dl chromium-browser
-        
-        pip install archivebox      # install archivebox
-        
-        mkdir data && cd data       # (doesn't have to be called data)
-        archivebox init
-        archivebox add 'https://example.com'  # add URLs via args or stdin
-        
-        # or import an RSS/JSON/XML/TXT feed/list of links
-        archivebox add https://getpocket.com/users/USERNAME/feed/all --depth=1
-        ```
-        
-        Once you've added your first links, open `data/index.html` in a browser to view the static archive.
-        
-        You can also start it as a server with a full web UI to manage your links:
-        ```bash
-        archivebox manage createsuperuser
-        archivebox server
-        ```
-        
-        You can visit `http://127.0.0.1:8000` in your browser to access it.
-        
-        [DEMO: archivebox.zervice.io/](https://archivebox.zervice.io)  
-        For more information, see the [full Quickstart guide](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.
-        
-        ---
-        
-        <div align="center">
-        <img src="https://i.imgur.com/PVO88AZ.png" width="80%"/>
-        </div>
-        
-        # Overview
-        
-        Because modern websites are complicated and often rely on dynamic content,
-        ArchiveBox archives the sites in **several different formats** beyond what public
-        archiving services like Archive.org and Archive.is are capable of saving. Using multiple
-        methods and the market-dominant browser to execute JS ensures we can save even the most
-        complex, finicky websites in at least a few high-quality, long-term data formats.
-        
-        ArchiveBox imports a list of URLs from stdin, remote URL, or file, then adds the pages to a local archive folder using wget to create a browsable HTML clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...
-        
-        Running `archivebox add` adds only new, unique links into your collection on each run. Because it will ignore duplicates and only archive each link the first time you add it, you can schedule it to [run on a timer](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) and re-import all your feeds multiple times a day. It will run quickly even if the feeds are large, because it's only archiving the newest links since the last run. For each link, it runs through all the archive methods. Methods that fail will save `None` and be automatically retried on the next run, methods that succeed save their output into the data folder and are never retried/overwritten by subsequent runs. Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/pirate/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs).
-        
-        All the archived links are stored by date bookmarked in `./archive/<timestamp>`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM.
-        
-        #### Can import links from many formats:
-        
-        ```bash
-        echo 'http://example.com' | archivebox add
-        archivebox add 'https://example.com/some/page'
-        archivebox add < ~/Downloads/firefox_bookmarks_export.html
-        archivebox add --depth=1 'https://example.com/some/rss/feed.xml'
-        archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
-        ```
-        
-        - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more)
-        - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format
-        - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more
-        
-        See the [Usage: CLI](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
-        
-        #### Saves lots of useful stuff for each imported link:
-        
-        ```bash
-         ls ./archive/<timestamp>/
-        ```
-        
-        - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details
-        - **Title:** `title` title of the site
-        - **Favicon:** `favicon.ico` favicon of the site
-        - **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present
-        - **WARC:** `warc/<timestamp>.gz` gzipped WARC of all the resources fetched while archiving
-        - **PDF:** `output.pdf` Printed PDF of site using headless chrome
-        - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome
-        - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome
-        - **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org
-        - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl
-        - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links
-        - _More coming soon! See the [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)..._
-        
-        It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/pirate/ArchiveBox/wiki/Configuration) via environment variables or config file.
-        
-        If you're importing URLs with secret tokens in them (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of these methods to avoid leaking private URLs to 3rd party APIs during the archiving process. See the [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details.
-        
-        ## Key Features
-        
-        - [**Free & open source**](https://github.com/pirate/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally
-        - [**Few dependencies**](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage)
-        - [**Comprehensive documentation**](https://github.com/pirate/ArchiveBox/wiki), [active development](https://github.com/pirate/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-        - **Doesn't require a constantly-running server**, proxy, or native app
-        - Easy to set up **[scheduled importing](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources**
-        - Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC
-        - ~~**Suitable for paywalled / [authenticated content](https://github.com/pirate/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes)
-        - Can [**run scripts during archiving**](https://github.com/pirate/ArchiveBox/issues/51) to [scroll pages](https://github.com/pirate/ArchiveBox/issues/80), [close modals](https://github.com/pirate/ArchiveBox/issues/175), expand comment threads, etc.
-        - Can also [**mirror content to 3rd-party archiving services**](https://github.com/pirate/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy
-        
-        ## Background & Motivation
-        
-        Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.
-        
-        Whether it's to resist censorship by saving articles before they get taken down or edited, or
-        just to save a collection of early 2010's flash games you love to play, having the tools to
-        archive internet content enables to you save the stuff you care most about before it disappears.
-        
-        <div align="center">
-        <img src="https://i.imgur.com/bC6eZcV.png" width="50%"/><br/>
-         <sup><i>Image from <a href="https://digiday.com/media/wtf-link-rot/">WTF is Link Rot?</a>...</i><br/></sup>
-        </div>
-        
-        The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful.
-        I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.
-        
-        ## Comparison to Other Projects
-        
-        ▶ **Check out our [community page](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.**
-        
-        <img src="https://i.imgur.com/4nkFjdv.png" width="10%" align="left"/> The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
-        
-        #### User Interface & Intended Purpose
-        
-        ArchiveBox differentiates itself from [similar projects](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
-        
-        #### Private Local Archives vs Centralized Public Archives
-        
-        Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle.
-        
-        #### Storage Requirements
-        
-        Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files.
-        
-        ## Learn more
-        
-        <!--▶ **Join out our [community chat](http://webchat.freenode.net?channels=ArchiveBox&uio=d4) hosted on IRC freenode.net:`#ArchiveBox`!**-->
-        
-        Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web!
-        
-        <img src="https://i.imgur.com/0ZOmOvN.png" width="14%" align="right"/>
-        
-        - [Community Wiki](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-          - [The Master Lists](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists)  
-            _Community-maintained indexes of archiving tools and institutions._
-          - [Web Archiving Software](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects)  
-            _Open source tools and projects in the internet archiving space._
-          - [Reading List](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Reading-List)  
-            _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
-          - [Communities](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Communities)  
-            _A collection of the most active internet archiving communities and initiatives._
-        - Check out the ArchiveBox [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
-        - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
-        - Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
-        
-        ---
-        
-        # Documentation
-        
-        <img src="https://read-the-docs-guidelines.readthedocs-hosted.com/_images/logo-dark.png" width="13%" align="right"/>
-        
-        We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) for documentation.
-        
-        You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.
-        
-        You can build the docs by running:
-        
-        ```python
-        cd ArchiveBox
-        pipenv install --dev
-        sphinx-apidoc -o docs archivebox
-        cd docs/
-        make html
-        # then open docs/_build/html/index.html
-        ```
-        
-        ## Getting Started
-        
-        - [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
-        - [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
-        - [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)
-        
-        ## Reference
-        
-        - [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
-        - [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
-        - [Supported Sources](https://github.com/pirate/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
-        - [Supported Outputs](https://github.com/pirate/ArchiveBox/wiki#can-save-these-things-for-each-site)
-        - [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
-        - [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
-        - [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Install-Chromium)
-        - [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview)
-        - [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)
-        
-        ## More Info
-        
-        - [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
-        - [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
-        - [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
-        - [Background & Motivation](https://github.com/pirate/ArchiveBox#background--motivation)
-        - [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-        
-        ---
-        
-        <div align="center">
-        <br/><br/>
-        <img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" height="40px"/>
-        <br/>
-        <sub><i>This project is maintained mostly in <a href="https://nicksweeting.com/blog#About">my spare time</a> with the help from generous contributors and Monadical.com.</i></sub>
-        <br/><br/>
-        
-        <br/>
-        <a href="https://github.com/sponsors/pirate">Sponsor us on Github</a>
-        <br>
-        <br>
-        <a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
-        <br/>
-        
-        <a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
-        <a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
-        
-        <br/><br/>
-        
-        </div>
-        
-Platform: UNKNOWN
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Natural Language :: English
-Classifier: Operating System :: OS Independent
-Classifier: Development Status :: 4 - Beta
-Classifier: Topic :: Utilities
-Classifier: Topic :: System :: Archiving
-Classifier: Topic :: System :: Archiving :: Backup
-Classifier: Topic :: System :: Recovery Tools
-Classifier: Topic :: Sociology :: History
-Classifier: Topic :: Internet :: WWW/HTTP
-Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
-Classifier: Topic :: Internet :: WWW/HTTP :: WSGI
-Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Application
-Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Server
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Education
-Classifier: Intended Audience :: End Users/Desktop
-Classifier: Intended Audience :: Information Technology
-Classifier: Intended Audience :: Legal Industry
-Classifier: Intended Audience :: System Administrators
-Classifier: Environment :: Console
-Classifier: Environment :: Web Environment
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.7
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Framework :: Django
-Classifier: Typing :: Typed
-Requires-Python: >=3.7
-Description-Content-Type: text/markdown
-Provides-Extra: dev

archivebox.egg-info/SOURCES.txt

@@ -1,126 +0,0 @@
-LICENSE
-MANIFEST.in
-README.md
-package-lock.json
-package.json
-setup.py
-archivebox/__init__.py
-archivebox/__main__.py
-archivebox/logging_util.py
-archivebox/main.py
-archivebox/manage.py
-archivebox/system.py
-archivebox/util.py
-archivebox.egg-info/PKG-INFO
-archivebox.egg-info/SOURCES.txt
-archivebox.egg-info/dependency_links.txt
-archivebox.egg-info/entry_points.txt
-archivebox.egg-info/requires.txt
-archivebox.egg-info/top_level.txt
-archivebox/cli/__init__.py
-archivebox/cli/archivebox_add.py
-archivebox/cli/archivebox_config.py
-archivebox/cli/archivebox_help.py
-archivebox/cli/archivebox_init.py
-archivebox/cli/archivebox_list.py
-archivebox/cli/archivebox_manage.py
-archivebox/cli/archivebox_oneshot.py
-archivebox/cli/archivebox_remove.py
-archivebox/cli/archivebox_schedule.py
-archivebox/cli/archivebox_server.py
-archivebox/cli/archivebox_shell.py
-archivebox/cli/archivebox_status.py
-archivebox/cli/archivebox_update.py
-archivebox/cli/archivebox_version.py
-archivebox/cli/tests.py
-archivebox/config/__init__.py
-archivebox/config/stubs.py
-archivebox/core/__init__.py
-archivebox/core/admin.py
-archivebox/core/apps.py
-archivebox/core/forms.py
-archivebox/core/models.py
-archivebox/core/settings.py
-archivebox/core/tests.py
-archivebox/core/urls.py
-archivebox/core/views.py
-archivebox/core/welcome_message.py
-archivebox/core/wsgi.py
-archivebox/core/migrations/0001_initial.py
-archivebox/core/migrations/0002_auto_20200625_1521.py
-archivebox/core/migrations/0003_auto_20200630_1034.py
-archivebox/core/migrations/0004_auto_20200713_1552.py
-archivebox/core/migrations/0005_auto_20200728_0326.py
-archivebox/core/migrations/__init__.py
-archivebox/extractors/__init__.py
-archivebox/extractors/archive_org.py
-archivebox/extractors/dom.py
-archivebox/extractors/favicon.py
-archivebox/extractors/git.py
-archivebox/extractors/media.py
-archivebox/extractors/pdf.py
-archivebox/extractors/readability.py
-archivebox/extractors/screenshot.py
-archivebox/extractors/singlefile.py
-archivebox/extractors/title.py
-archivebox/extractors/wget.py
-archivebox/index/__init__.py
-archivebox/index/csv.py
-archivebox/index/html.py
-archivebox/index/json.py
-archivebox/index/schema.py
-archivebox/index/sql.py
-archivebox/parsers/__init__.py
-archivebox/parsers/generic_html.py
-archivebox/parsers/generic_json.py
-archivebox/parsers/generic_rss.py
-archivebox/parsers/generic_txt.py
-archivebox/parsers/medium_rss.py
-archivebox/parsers/netscape_html.py
-archivebox/parsers/pinboard_rss.py
-archivebox/parsers/pocket_html.py
-archivebox/parsers/shaarli_rss.py
-archivebox/themes/admin/actions_as_select.html
-archivebox/themes/admin/app_index.html
-archivebox/themes/admin/base.html
-archivebox/themes/admin/login.html
-archivebox/themes/default/add_links.html
-archivebox/themes/default/main_index.html
-archivebox/themes/default/static/admin.css
-archivebox/themes/default/static/archive.png
-archivebox/themes/default/static/bootstrap.min.css
-archivebox/themes/default/static/external.png
-archivebox/themes/default/static/jquery.dataTables.min.css
-archivebox/themes/default/static/jquery.dataTables.min.js
-archivebox/themes/default/static/jquery.min.js
-archivebox/themes/default/static/sort_asc.png
-archivebox/themes/default/static/sort_both.png
-archivebox/themes/default/static/sort_desc.png
-archivebox/themes/default/static/spinner.gif
-archivebox/themes/legacy/favicon.ico
-archivebox/themes/legacy/link_details.html
-archivebox/themes/legacy/main_index.html
-archivebox/themes/legacy/main_index_row.html
-archivebox/themes/legacy/robots.txt
-archivebox/themes/legacy/static/archive.png
-archivebox/themes/legacy/static/bootstrap.min.css
-archivebox/themes/legacy/static/external.png
-archivebox/themes/legacy/static/jquery.dataTables.min.css
-archivebox/themes/legacy/static/jquery.dataTables.min.js
-archivebox/themes/legacy/static/jquery.min.js
-archivebox/themes/legacy/static/sort_asc.png
-archivebox/themes/legacy/static/sort_both.png
-archivebox/themes/legacy/static/sort_desc.png
-archivebox/themes/legacy/static/spinner.gif
-tests/__init__.py
-tests/conftest.py
-tests/fixtures.py
-tests/test_args.py
-tests/test_extractors.py
-tests/test_init.py
-tests/test_oneshot.py
-tests/test_remove.py
-tests/test_title.py
-tests/test_util.py
-tests/mock_server/__init__.py
-tests/mock_server/server.py

archivebox.egg-info/dependency_links.txt

@@ -1 +0,0 @@
-

archivebox.egg-info/entry_points.txt

@@ -1,3 +0,0 @@
-[console_scripts]
-archivebox = archivebox.cli:main
-

archivebox.egg-info/requires.txt

@@ -1,26 +0,0 @@
-requests==2.24.0
-atomicwrites==1.4.0
-mypy-extensions==0.4.3
-base32-crockford==0.3.0
-django==3.0.8
-django-extensions==3.0.3
-dateparser
-ipython
-youtube-dl
-python-crontab==2.5.1
-croniter==0.3.34
-w3lib==1.22.0
-
-[dev]
-setuptools
-wheel
-twine
-flake8
-ipdb
-mypy
-django-stubs
-sphinx
-sphinx-rtd-theme
-recommonmark
-pytest
-bottle

archivebox.egg-info/top_level.txt

@@ -1,2 +0,0 @@
-archivebox
-tests

archivebox/LICENSE

@@ -0,0 +1 @@
+../LICENSE

archivebox/README.md

@@ -0,0 +1 @@
+../README.md

archivebox/cli/__init__.py

@@ -5,18 +5,21 @@ import os
 import sys
 import argparse
 
-from typing import Optional, Dict, List, IO
+from typing import Optional, Dict, List, IO, Union
+from pathlib import Path
 
 from ..config import OUTPUT_DIR
 
 from importlib import import_module
 
-CLI_DIR = os.path.dirname(os.path.abspath(__file__))
+CLI_DIR = Path(__file__).resolve().parent
 
 # these common commands will appear sorted before any others for ease-of-use
 meta_cmds = ('help', 'version')
 main_cmds = ('init', 'info', 'config')
-archive_cmds = ('add', 'remove', 'update', 'list')
+archive_cmds = ('add', 'remove', 'update', 'list', 'status')
+
+fake_db = ("oneshot",)
 
 display_first = (*meta_cmds, *main_cmds, *archive_cmds)
 
@@ -55,9 +58,13 @@ def list_subcommands() -> Dict[str, str]:
 def run_subcommand(subcommand: str,
                    subcommand_args: List[str]=None,
                    stdin: Optional[IO]=None,
-                   pwd: Optional[str]=None) -> None:
+                   pwd: Union[Path, str, None]=None) -> None:
     """Run a given ArchiveBox subcommand with the given list of args"""
 
+    if subcommand not in meta_cmds:
+        from ..config import setup_django
+        setup_django(in_memory_db=subcommand in fake_db, check_db=subcommand in archive_cmds)
+
     module = import_module('.archivebox_{}'.format(subcommand), __package__)
     module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
 
@@ -104,11 +111,11 @@ def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided,
     )
     command = parser.parse_args(args or ())
 
-    if command.help or command.subcommand is None:
-        command.subcommand = 'help'
-    elif command.version:
+    if command.version:
         command.subcommand = 'version'
-    
+    elif command.help or command.subcommand is None:
+        command.subcommand = 'help'
+
     if command.subcommand not in ('help', 'version', 'status'):
         from ..logging_util import log_cli_command
 
@@ -133,3 +140,5 @@ __all__ = (
     'run_subcommand',
     *SUBCOMMANDS.keys(),
 )
+
+

archivebox/cli/archivebox_add.py

@@ -62,10 +62,17 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         help="Re-archive URLs from scratch, overwriting any existing files"
     )
     parser.add_argument(
-        '--init', #'-i',
+        "--init", #'-i',
         action='store_true',
         help="Init/upgrade the curent data directory before adding",
     )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
     command = parser.parse_args(args or ())
     urls = command.urls
     stdin_urls = accept_stdin(stdin)
@@ -82,6 +89,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         index_only=command.index_only,
         overwrite=command.overwrite,
         init=command.init,
+        extractors=command.extract,
         out_dir=pwd or OUTPUT_DIR,
     )
 

archivebox/cli/archivebox_list.py

@@ -23,7 +23,7 @@ from ..index import (
     get_corrupted_folders,
     get_unrecognized_folders,
 )
-from ..logging_util import SmartFormatter, accept_stdin
+from ..logging_util import SmartFormatter, accept_stdin, stderr
 
 
 @docstring(list_all.__doc__)
@@ -46,6 +46,16 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         action='store_true',
         help="Print the output in JSON format with all columns included.",
     )
+    group.add_argument(
+        '--html',
+        action='store_true',
+        help="Print the output in HTML format"
+    )
+    parser.add_argument(
+        '--with-headers',
+        action='store_true',
+        help='Include the headers in the output document' 
+    )
     parser.add_argument(
         '--sort', #'-s',
         type=str,
@@ -88,7 +98,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )
@@ -102,6 +112,13 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     command = parser.parse_args(args or ())
     filter_patterns_str = accept_stdin(stdin)
 
+    if command.with_headers and not (command.json or command.html or command.csv):
+        stderr(
+            '[X] --with-headers can only be used with --json, --html or --csv options.\n',
+            color='red',
+        )
+        raise SystemExit(2)
+
     matching_folders = list_all(
         filter_patterns_str=filter_patterns_str,
         filter_patterns=command.filter_patterns,
@@ -112,6 +129,8 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         sort=command.sort,
         csv=command.csv,
         json=command.json,
+        html=command.html,
+        with_headers=command.with_headers,
         out_dir=pwd or OUTPUT_DIR,
     )
     raise SystemExit(not matching_folders)

archivebox/cli/archivebox_oneshot.py

@@ -36,6 +36,13 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
             '    ~/Desktop/sites_list.csv\n'
         )
     )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
     parser.add_argument(
         '--out-dir',
         type=str,
@@ -54,7 +61,8 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     
     oneshot(
         url=stdin_url or url,
-        out_dir=str(Path(command.out_dir).absolute()),
+        out_dir=Path(command.out_dir).resolve(),
+        extractors=command.extract,
     )
 
 

archivebox/cli/archivebox_remove.py

@@ -50,7 +50,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex','tag'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )

archivebox/cli/archivebox_schedule.py

@@ -42,6 +42,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--depth', # '-d',
         type=int,
+        choices=[0, 1],
         default=0,
         help='Depth to archive to [0] or 1, see "add" command help for more info.',
     )

archivebox/cli/archivebox_server.py

@@ -43,6 +43,11 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         action='store_true',
         help='Run archivebox init before starting the server',
     )
+    parser.add_argument(
+        '--createsuperuser',
+        action='store_true',
+        help='Run archivebox manage createsuperuser before starting the server',
+    )
     command = parser.parse_args(args or ())
     reject_stdin(__command__, stdin)
     
@@ -51,6 +56,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         reload=command.reload,
         debug=command.debug,
         init=command.init,
+        createsuperuser=command.createsuperuser,
         out_dir=pwd or OUTPUT_DIR,
     )
 

archivebox/cli/archivebox_update.py

@@ -91,7 +91,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )
@@ -102,6 +102,13 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         default=None,
         help='Update only URLs matching these filter patterns.'
     )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
     command = parser.parse_args(args or ())
     filter_patterns_str = accept_stdin(stdin)
 
@@ -117,6 +124,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         after=command.after,
         before=command.before,
         out_dir=pwd or OUTPUT_DIR,
+        extractors=command.extract,
     )
     
 

archivebox/cli/tests.py

@@ -7,6 +7,7 @@ import os
 import sys
 import shutil
 import unittest
+from pathlib import Path
 
 from contextlib import contextmanager
 
@@ -109,13 +110,13 @@ class TestInit(unittest.TestCase):
         with output_hidden():
             archivebox_init.main([])
 
-        assert os.path.exists(os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME))
-        assert os.path.exists(os.path.join(OUTPUT_DIR, JSON_INDEX_FILENAME))
-        assert os.path.exists(os.path.join(OUTPUT_DIR, HTML_INDEX_FILENAME))
+        assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
+        assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
+        assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
         assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0
 
     def test_conflicting_init(self):
-        with open(os.path.join(OUTPUT_DIR, 'test_conflict.txt'), 'w+') as f:
+        with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+') as f:
             f.write('test')
 
         try:
@@ -125,9 +126,9 @@ class TestInit(unittest.TestCase):
         except SystemExit:
             pass
 
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME))
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, JSON_INDEX_FILENAME))
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, HTML_INDEX_FILENAME))
+        assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
+        assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
+        assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
         try:
             load_main_index(out_dir=OUTPUT_DIR)
             assert False, 'load_main_index should raise an exception when no index is present'
@@ -159,7 +160,7 @@ class TestAdd(unittest.TestCase):
         assert len(all_links) == 30
 
     def test_add_arg_file(self):
-        test_file = os.path.join(OUTPUT_DIR, 'test.txt')
+        test_file = Path(OUTPUT_DIR) / 'test.txt'
         with open(test_file, 'w+') as f:
             f.write(test_urls)
 

archivebox/config.py

@@ -1,4 +1,25 @@
-__package__ = 'archivebox.config'
+"""
+ArchiveBox config definitons (including defaults and dynamic config options).
+
+Config Usage Example:
+
+    archivebox config --set MEDIA_TIMEOUT=600
+    env MEDIA_TIMEOUT=600 USE_COLOR=False ... archivebox [subcommand] ...
+
+Config Precedence Order:
+
+  1. cli args                 (--update-all / --index-only / etc.)
+  2. shell environment vars   (env USE_COLOR=False archivebox add '...')
+  3. config file              (echo "SAVE_FAVICON=False" >> ArchiveBox.conf)
+  4. defaults                 (defined below in Python)
+
+Documentation:
+
+  https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration
+
+"""
+
+__package__ = 'archivebox'
 
 import os
 import io
@@ -6,8 +27,8 @@ import re
 import sys
 import json
 import getpass
-import shutil
 import platform
+import shutil
 import django
 
 from hashlib import md5
@@ -17,7 +38,7 @@ from subprocess import run, PIPE, DEVNULL
 from configparser import ConfigParser
 from collections import defaultdict
 
-from .stubs import (
+from .config_stubs import (
     SimpleConfigValueDict,
     ConfigValue,
     ConfigDict,
@@ -25,28 +46,13 @@ from .stubs import (
     ConfigDefaultDict,
 )
 
-# precedence order for config:
-# 1. cli args
-# 2. shell environment vars
-# 3. config file
-# 4. defaults
+############################### Config Schema ##################################
 
-# env USE_COLO=false archivebox add '...'
-# env SHOW_PROGRESS=1 archivebox add '...'
-
-# ******************************************************************************
-# Documentation: https://github.com/pirate/ArchiveBox/wiki/Configuration
-# Use the 'env' command to pass config options to ArchiveBox.  e.g.:
-#     env USE_COLOR=True CHROME_BINARY=chromium archivebox add < example.html
-# ******************************************************************************
-
-################################# User Config ##################################
-
-CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
+CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = {
     'SHELL_CONFIG': {
         'IS_TTY':                   {'type': bool,  'default': lambda _: sys.stdout.isatty()},
         'USE_COLOR':                {'type': bool,  'default': lambda c: c['IS_TTY']},
-        'SHOW_PROGRESS':            {'type': bool,  'default': lambda c: False if platform.system() == 'Darwin' else c['IS_TTY']},  # TODO: remove this temporary hack once progress bars are fixed on macOS
+        'SHOW_PROGRESS':            {'type': bool,  'default': lambda c: (c['IS_TTY'] and platform.system() != 'Darwin')},  # progress bars are buggy on mac, disable for now
         'IN_DOCKER':                {'type': bool,  'default': False},
         # TODO: 'SHOW_HINTS':       {'type:  bool,  'default': True},
     },
@@ -59,7 +65,7 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'MEDIA_TIMEOUT':            {'type': int,   'default': 3600},
         'OUTPUT_PERMISSIONS':       {'type': str,   'default': '755'},
         'RESTRICT_FILE_NAMES':      {'type': str,   'default': 'windows'},
-        'URL_BLACKLIST':            {'type': str,   'default': r'\.(css|js|otf|ttf|woff|woff2)(\?.*)?$'},  # to avoid downloading code assets as their own pages
+        'URL_BLACKLIST':            {'type': str,   'default': r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$'},  # to avoid downloading code assets as their own pages
     },
 
     'SERVER_CONFIG': {
@@ -69,8 +75,8 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'DEBUG':                    {'type': bool,  'default': False},
         'PUBLIC_INDEX':             {'type': bool,  'default': True},
         'PUBLIC_SNAPSHOTS':         {'type': bool,  'default': True},
+        'PUBLIC_ADD_VIEW':          {'type': bool,  'default': False},
         'FOOTER_INFO':              {'type': str,   'default': 'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.'},
-        'ACTIVE_THEME':             {'type': str,   'default': 'default'},
     },
 
     'ARCHIVE_METHOD_TOGGLES': {
@@ -80,13 +86,14 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'SAVE_WGET_REQUISITES':     {'type': bool,  'default': True, 'aliases': ('FETCH_WGET_REQUISITES',)},
         'SAVE_SINGLEFILE':          {'type': bool,  'default': True, 'aliases': ('FETCH_SINGLEFILE',)},
         'SAVE_READABILITY':         {'type': bool,  'default': True, 'aliases': ('FETCH_READABILITY',)},
+        'SAVE_MERCURY':             {'type': bool,  'default': True, 'aliases': ('FETCH_MERCURY',)},
         'SAVE_PDF':                 {'type': bool,  'default': True, 'aliases': ('FETCH_PDF',)},
         'SAVE_SCREENSHOT':          {'type': bool,  'default': True, 'aliases': ('FETCH_SCREENSHOT',)},
         'SAVE_DOM':                 {'type': bool,  'default': True, 'aliases': ('FETCH_DOM',)},
+        'SAVE_HEADERS':             {'type': bool,  'default': True, 'aliases': ('FETCH_HEADERS',)},
         'SAVE_WARC':                {'type': bool,  'default': True, 'aliases': ('FETCH_WARC',)},
         'SAVE_GIT':                 {'type': bool,  'default': True, 'aliases': ('FETCH_GIT',)},
         'SAVE_MEDIA':               {'type': bool,  'default': True, 'aliases': ('FETCH_MEDIA',)},
-        'SAVE_PLAYLISTS':           {'type': bool,  'default': True, 'aliases': ('FETCH_PLAYLISTS',)},
         'SAVE_ARCHIVE_DOT_ORG':     {'type': bool,  'default': True, 'aliases': ('SUBMIT_ARCHIVE_DOT_ORG',)},
     },
 
@@ -95,8 +102,8 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'GIT_DOMAINS':              {'type': str,   'default': 'github.com,bitbucket.org,gitlab.com'},
         'CHECK_SSL_VALIDITY':       {'type': bool,  'default': True},
 
-        'CURL_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/pirate/ArchiveBox/) curl/{CURL_VERSION}'},
-        'WGET_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/pirate/ArchiveBox/) wget/{WGET_VERSION}'},
+        'CURL_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION}'},
+        'WGET_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION}'},
         'CHROME_USER_AGENT':        {'type': str,   'default': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'},
 
         'COOKIES_FILE':             {'type': str,   'default': None},
@@ -104,6 +111,47 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
 
         'CHROME_HEADLESS':          {'type': bool,  'default': True},
         'CHROME_SANDBOX':           {'type': bool,  'default': lambda c: not c['IN_DOCKER']},
+        'YOUTUBEDL_ARGS':           {'type': list,  'default': ['--write-description',
+                                                                '--write-info-json',
+                                                                '--write-annotations',
+                                                                '--write-thumbnail',
+                                                                '--no-call-home',
+                                                                '--all-subs',
+                                                                '--yes-playlist',
+                                                                '--continue',
+                                                                '--ignore-errors',
+                                                                '--geo-bypass',
+                                                                '--add-metadata',
+                                                                '--max-filesize=750m',
+                                                                ]},
+                                                                    
+
+        'WGET_ARGS':                {'type': list,  'default': ['--no-verbose',
+                                                                '--adjust-extension',
+                                                                '--convert-links',
+                                                                '--force-directories',
+                                                                '--backup-converted',
+                                                                '--span-hosts',
+                                                                '--no-parent',
+                                                                '-e', 'robots=off',
+                                                                ]},
+        'CURL_ARGS':                {'type': list,  'default': ['--silent',
+                                                                '--location',
+                                                                '--compressed'
+                                                               ]},
+        'GIT_ARGS':                 {'type': list,  'default': ['--recursive']},
+    },
+
+    'SEARCH_BACKEND_CONFIG' : {
+        'USE_INDEXING_BACKEND':     {'type': bool,  'default': True},
+        'USE_SEARCHING_BACKEND':    {'type': bool,  'default': True},
+        'SEARCH_BACKEND_ENGINE':    {'type': str,   'default': 'ripgrep'},
+        'SEARCH_BACKEND_HOST_NAME': {'type': str,   'default': 'localhost'},
+        'SEARCH_BACKEND_PORT':      {'type': int,   'default': 1491},
+        'SEARCH_BACKEND_PASSWORD':  {'type': str,   'default': 'SecretPassword'},
+        # SONIC
+        'SONIC_COLLECTION':         {'type': str,   'default': 'archivebox'},
+        'SONIC_BUCKET':             {'type': str,   'default': 'snapshots'},
     },
 
     'DEPENDENCY_CONFIG': {
@@ -111,33 +159,62 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'USE_WGET':                 {'type': bool,  'default': True},
         'USE_SINGLEFILE':           {'type': bool,  'default': True},
         'USE_READABILITY':          {'type': bool,  'default': True},
+        'USE_MERCURY':              {'type': bool,  'default': True},
         'USE_GIT':                  {'type': bool,  'default': True},
         'USE_CHROME':               {'type': bool,  'default': True},
+        'USE_NODE':                 {'type': bool,  'default': True},
         'USE_YOUTUBEDL':            {'type': bool,  'default': True},
-
+        'USE_RIPGREP':              {'type': bool,  'default': True},
+        
         'CURL_BINARY':              {'type': str,   'default': 'curl'},
         'GIT_BINARY':               {'type': str,   'default': 'git'},
         'WGET_BINARY':              {'type': str,   'default': 'wget'},
         'SINGLEFILE_BINARY':        {'type': str,   'default': 'single-file'},
         'READABILITY_BINARY':       {'type': str,   'default': 'readability-extractor'},
+        'MERCURY_BINARY':           {'type': str,   'default': 'mercury-parser'},
         'YOUTUBEDL_BINARY':         {'type': str,   'default': 'youtube-dl'},
+        'NODE_BINARY':              {'type': str,   'default': 'node'},
+        'RIPGREP_BINARY':           {'type': str,   'default': 'rg'},
         'CHROME_BINARY':            {'type': str,   'default': None},
+
+        'POCKET_CONSUMER_KEY':      {'type': str,   'default': None},
+        'POCKET_ACCESS_TOKENS':     {'type': dict,  'default': {}},
     },
 }
 
+
+########################## Backwards-Compatibility #############################
+
+
+# for backwards compatibility with old config files, check old/deprecated names for each key
 CONFIG_ALIASES = {
     alias: key
-    for section in CONFIG_DEFAULTS.values()
+    for section in CONFIG_SCHEMA.values()
         for key, default in section.items()
             for alias in default.get('aliases', ())
 }
-USER_CONFIG = {key for section in CONFIG_DEFAULTS.values() for key in section.keys()}
+USER_CONFIG = {key for section in CONFIG_SCHEMA.values() for key in section.keys()}
+
 def get_real_name(key: str) -> str:
+    """get the current canonical name for a given deprecated config key"""
     return CONFIG_ALIASES.get(key.upper().strip(), key.upper().strip())
 
-############################## Derived Config ##############################
 
-# Constants
+
+################################ Constants #####################################
+
+PACKAGE_DIR_NAME = 'archivebox'
+TEMPLATES_DIR_NAME = 'templates'
+
+ARCHIVE_DIR_NAME = 'archive'
+SOURCES_DIR_NAME = 'sources'
+LOGS_DIR_NAME = 'logs'
+SQL_INDEX_FILENAME = 'index.sqlite3'
+JSON_INDEX_FILENAME = 'index.json'
+HTML_INDEX_FILENAME = 'index.html'
+ROBOTS_TXT_FILENAME = 'robots.txt'
+FAVICON_FILENAME = 'favicon.ico'
+CONFIG_FILENAME = 'ArchiveBox.conf'
 
 DEFAULT_CLI_COLORS = {
     'reset': '\033[00;00m',
@@ -186,55 +263,30 @@ STATICFILE_EXTENSIONS = {
     # html, htm, shtml, xhtml, xml, aspx, php, cgi
 }
 
-PYTHON_DIR_NAME = 'archivebox'
-TEMPLATES_DIR_NAME = 'themes'
-
-ARCHIVE_DIR_NAME = 'archive'
-SOURCES_DIR_NAME = 'sources'
-LOGS_DIR_NAME = 'logs'
-STATIC_DIR_NAME = 'static'
-SQL_INDEX_FILENAME = 'index.sqlite3'
-JSON_INDEX_FILENAME = 'index.json'
-HTML_INDEX_FILENAME = 'index.html'
-ROBOTS_TXT_FILENAME = 'robots.txt'
-FAVICON_FILENAME = 'favicon.ico'
-CONFIG_FILENAME = 'ArchiveBox.conf'
-
-CONFIG_HEADER = (
-"""# This is the config file for your ArchiveBox collection.
-#
-# You can add options here manually in INI format, or automatically by running:
-#    archivebox config --set KEY=VALUE
-# 
-# If you modify this file manually, make sure to update your archive after by running:
-#    archivebox init
-#
-# A list of all possible config with documentation and examples can be found here:
-#    https://github.com/pirate/ArchiveBox/wiki/Configuration
-
-""")
 
 
-DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
+############################## Derived Config ##################################
+
+
+DYNAMIC_CONFIG_SCHEMA: ConfigDefaultDict = {
     'TERM_WIDTH':               {'default': lambda c: lambda: shutil.get_terminal_size((100, 10)).columns},
     'USER':                     {'default': lambda c: getpass.getuser() or os.getlogin()},
     'ANSI':                     {'default': lambda c: DEFAULT_CLI_COLORS if c['USE_COLOR'] else {k: '' for k in DEFAULT_CLI_COLORS.keys()}},
 
-    'REPO_DIR':                 {'default': lambda c: os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..'))},
-    'PYTHON_DIR':               {'default': lambda c: os.path.join(c['REPO_DIR'], PYTHON_DIR_NAME)},
-    'TEMPLATES_DIR':            {'default': lambda c: os.path.join(c['PYTHON_DIR'], TEMPLATES_DIR_NAME, 'legacy')},
+    'PACKAGE_DIR':              {'default': lambda c: Path(__file__).resolve().parent},
+    'TEMPLATES_DIR':            {'default': lambda c: c['PACKAGE_DIR'] / TEMPLATES_DIR_NAME},
 
-    'OUTPUT_DIR':               {'default': lambda c: os.path.abspath(os.path.expanduser(c['OUTPUT_DIR'])) if c['OUTPUT_DIR'] else os.path.abspath(os.curdir)},
-    'ARCHIVE_DIR':              {'default': lambda c: os.path.join(c['OUTPUT_DIR'], ARCHIVE_DIR_NAME)},
-    'SOURCES_DIR':              {'default': lambda c: os.path.join(c['OUTPUT_DIR'], SOURCES_DIR_NAME)},
-    'LOGS_DIR':                 {'default': lambda c: os.path.join(c['OUTPUT_DIR'], LOGS_DIR_NAME)},
-    'CONFIG_FILE':              {'default': lambda c: os.path.abspath(os.path.expanduser(c['CONFIG_FILE'])) if c['CONFIG_FILE'] else os.path.join(c['OUTPUT_DIR'], CONFIG_FILENAME)},
-    'COOKIES_FILE':             {'default': lambda c: c['COOKIES_FILE'] and os.path.abspath(os.path.expanduser(c['COOKIES_FILE']))},
-    'CHROME_USER_DATA_DIR':     {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (os.path.abspath(os.path.expanduser(c['CHROME_USER_DATA_DIR'])) or None)},
+    'OUTPUT_DIR':               {'default': lambda c: Path(c['OUTPUT_DIR']).resolve() if c['OUTPUT_DIR'] else Path(os.curdir).resolve()},
+    'ARCHIVE_DIR':              {'default': lambda c: c['OUTPUT_DIR'] / ARCHIVE_DIR_NAME},
+    'SOURCES_DIR':              {'default': lambda c: c['OUTPUT_DIR'] / SOURCES_DIR_NAME},
+    'LOGS_DIR':                 {'default': lambda c: c['OUTPUT_DIR'] / LOGS_DIR_NAME},
+    'CONFIG_FILE':              {'default': lambda c: Path(c['CONFIG_FILE']).resolve() if c['CONFIG_FILE'] else c['OUTPUT_DIR'] / CONFIG_FILENAME},
+    'COOKIES_FILE':             {'default': lambda c: c['COOKIES_FILE'] and Path(c['COOKIES_FILE']).resolve()},
+    'CHROME_USER_DATA_DIR':     {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (Path(c['CHROME_USER_DATA_DIR']).resolve() if c['CHROME_USER_DATA_DIR'] else None)},   # None means unset, so we autodetect it with find_chrome_Data_dir(), but emptystring '' means user manually set it to '', and we should store it as None
     'URL_BLACKLIST_PTN':        {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'] or '', re.IGNORECASE | re.UNICODE | re.MULTILINE)},
 
     'ARCHIVEBOX_BINARY':        {'default': lambda c: sys.argv[0]},
-    'VERSION':                  {'default': lambda c: json.loads((Path(c['REPO_DIR']) / 'package.json').read_text().strip())['version']},
+    'VERSION':                  {'default': lambda c: json.loads((Path(c['PACKAGE_DIR']) / 'package.json').read_text().strip())['version']},
     'GIT_SHA':                  {'default': lambda c: c['VERSION'].split('+')[-1] or 'unknown'},
 
     'PYTHON_BINARY':            {'default': lambda c: sys.executable},
@@ -247,6 +299,7 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'USE_CURL':                 {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['SAVE_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])},
     'CURL_VERSION':             {'default': lambda c: bin_version(c['CURL_BINARY']) if c['USE_CURL'] else None},
     'CURL_USER_AGENT':          {'default': lambda c: c['CURL_USER_AGENT'].format(**c)},
+    'CURL_ARGS':                {'default': lambda c: c['CURL_ARGS'] or []},
     'SAVE_FAVICON':             {'default': lambda c: c['USE_CURL'] and c['SAVE_FAVICON']},
     'SAVE_ARCHIVE_DOT_ORG':     {'default': lambda c: c['USE_CURL'] and c['SAVE_ARCHIVE_DOT_ORG']},
 
@@ -256,6 +309,9 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'WGET_USER_AGENT':          {'default': lambda c: c['WGET_USER_AGENT'].format(**c)},
     'SAVE_WGET':                {'default': lambda c: c['USE_WGET'] and c['SAVE_WGET']},
     'SAVE_WARC':                {'default': lambda c: c['USE_WGET'] and c['SAVE_WARC']},
+    'WGET_ARGS':                {'default': lambda c: c['WGET_ARGS'] or []},
+
+    'RIPGREP_VERSION':          {'default': lambda c: bin_version(c['RIPGREP_BINARY']) if c['USE_RIPGREP'] else None},
 
     'USE_SINGLEFILE':           {'default': lambda c: c['USE_SINGLEFILE'] and c['SAVE_SINGLEFILE']},
     'SINGLEFILE_VERSION':       {'default': lambda c: bin_version(c['SINGLEFILE_BINARY']) if c['USE_SINGLEFILE'] else None},
@@ -263,6 +319,9 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'USE_READABILITY':          {'default': lambda c: c['USE_READABILITY'] and c['SAVE_READABILITY']},
     'READABILITY_VERSION':      {'default': lambda c: bin_version(c['READABILITY_BINARY']) if c['USE_READABILITY'] else None},
 
+    'USE_MERCURY':              {'default': lambda c: c['USE_MERCURY'] and c['SAVE_MERCURY']},
+    'MERCURY_VERSION':          {'default': lambda c: '1.0.0' if shutil.which(str(bin_path(c['MERCURY_BINARY']))) else None},  # mercury is unversioned
+
     'USE_GIT':                  {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']},
     'GIT_VERSION':              {'default': lambda c: bin_version(c['GIT_BINARY']) if c['USE_GIT'] else None},
     'SAVE_GIT':                 {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']},
@@ -270,16 +329,21 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'USE_YOUTUBEDL':            {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']},
     'YOUTUBEDL_VERSION':        {'default': lambda c: bin_version(c['YOUTUBEDL_BINARY']) if c['USE_YOUTUBEDL'] else None},
     'SAVE_MEDIA':               {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']},
-    'SAVE_PLAYLISTS':           {'default': lambda c: c['SAVE_PLAYLISTS'] and c['SAVE_MEDIA']},
+    'YOUTUBEDL_ARGS':           {'default': lambda c: c['YOUTUBEDL_ARGS'] or []},
 
     'USE_CHROME':               {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'] or c['SAVE_SINGLEFILE'])},
     'CHROME_BINARY':            {'default': lambda c: c['CHROME_BINARY'] if c['CHROME_BINARY'] else find_chrome_binary()},
     'CHROME_VERSION':           {'default': lambda c: bin_version(c['CHROME_BINARY']) if c['USE_CHROME'] else None},
+    
     'SAVE_PDF':                 {'default': lambda c: c['USE_CHROME'] and c['SAVE_PDF']},
     'SAVE_SCREENSHOT':          {'default': lambda c: c['USE_CHROME'] and c['SAVE_SCREENSHOT']},
     'SAVE_DOM':                 {'default': lambda c: c['USE_CHROME'] and c['SAVE_DOM']},
-    'SAVE_SINGLEFILE':          {'default': lambda c: c['USE_CHROME'] and c['USE_SINGLEFILE']},
-    'SAVE_READABILITY':         {'default': lambda c: c['USE_READABILITY']},
+    'SAVE_SINGLEFILE':          {'default': lambda c: c['USE_CHROME'] and c['SAVE_SINGLEFILE'] and c['USE_NODE']},
+    'SAVE_READABILITY':         {'default': lambda c: c['USE_READABILITY'] and c['USE_NODE']},
+    'SAVE_MERCURY':             {'default': lambda c: c['USE_MERCURY'] and c['USE_NODE']},
+    
+    'USE_NODE':                 {'default': lambda c: c['USE_NODE'] and (c['SAVE_READABILITY'] or c['SAVE_SINGLEFILE'] or c['SAVE_MERCURY'])},
+    'NODE_VERSION':             {'default': lambda c: bin_version(c['NODE_BINARY']) if c['USE_NODE'] else None},
 
     'DEPENDENCIES':             {'default': lambda c: get_dependency_info(c)},
     'CODE_LOCATIONS':           {'default': lambda c: get_code_locations(c)},
@@ -292,6 +356,7 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
 
 ################################### Helpers ####################################
 
+
 def load_config_val(key: str,
                     default: ConfigDefaultValue=None,
                     type: Optional[Type]=None,
@@ -338,15 +403,18 @@ def load_config_val(key: str,
             raise ValueError(f'Invalid configuration option {key}={val} (expected an integer)')
         return int(val)
 
-    raise Exception('Config values can only be str, bool, or int')
+    elif type is list or type is dict:
+        return json.loads(val)
+
+    raise Exception('Config values can only be str, bool, int or json')
 
 
 def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]:
     """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf"""
 
-    out_dir = out_dir or os.path.abspath(os.getenv('OUTPUT_DIR', '.'))
-    config_path = os.path.join(out_dir, CONFIG_FILENAME)
-    if os.path.exists(config_path):
+    out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve()
+    config_path = Path(out_dir) / CONFIG_FILENAME
+    if config_path.exists():
         config_file = ConfigParser()
         config_file.optionxform = str 
         config_file.read(config_path)
@@ -365,12 +433,26 @@ def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]:
 def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
     """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf"""
 
-    from ..system import atomic_write
+    from .system import atomic_write
 
-    out_dir = out_dir or os.path.abspath(os.getenv('OUTPUT_DIR', '.'))
-    config_path = os.path.join(out_dir, CONFIG_FILENAME)
+    CONFIG_HEADER = (
+    """# This is the config file for your ArchiveBox collection.
+    #
+    # You can add options here manually in INI format, or automatically by running:
+    #    archivebox config --set KEY=VALUE
+    # 
+    # If you modify this file manually, make sure to update your archive after by running:
+    #    archivebox init
+    #
+    # A list of all possible config with documentation and examples can be found here:
+    #    https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration
+
+    """)
+
+    out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve()
+    config_path = Path(out_dir) /  CONFIG_FILENAME
     
-    if not os.path.exists(config_path):
+    if not config_path.exists():
         atomic_write(config_path, CONFIG_HEADER)
 
     config_file = ConfigParser()
@@ -380,7 +462,7 @@ def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
     with open(config_path, 'r') as old:
         atomic_write(f'{config_path}.bak', old.read())
 
-    find_section = lambda key: [name for name, opts in CONFIG_DEFAULTS.items() if key in opts][0]
+    find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0]
 
     # Set up sections in empty config file
     for key, val in config.items():
@@ -420,7 +502,7 @@ def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
         with open(f'{config_path}.bak', 'r') as old:
             atomic_write(config_path, old.read())
 
-    if os.path.exists(f'{config_path}.bak'):
+    if Path(f'{config_path}.bak').exists():
         os.remove(f'{config_path}.bak')
 
     return {}
@@ -458,7 +540,7 @@ def load_config(defaults: ConfigDefaultDict,
             stderr('    Check your config for mistakes and try again (your archive data is unaffected).')
             stderr()
             stderr('    For config documentation and examples see:')
-            stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration')
+            stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration')
             stderr()
             raise
             raise SystemExit(2)
@@ -469,6 +551,8 @@ def load_config(defaults: ConfigDefaultDict,
 
 #     with open(os.path.join(config['OUTPUT_DIR'], CONFIG_FILENAME), 'w+') as f:
 
+
+# Logging Helpers
 def stdout(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None:
     ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI
 
@@ -500,6 +584,7 @@ def hint(text: Union[Tuple[str, ...], List[str], str], prefix='    ', config: Op
             stderr('{}      {}'.format(prefix, line))
 
 
+# Dependency Metadata Helpers
 def bin_version(binary: Optional[str]) -> Optional[str]:
     """check the presence and return valid version line of a specified binary"""
 
@@ -518,14 +603,18 @@ def bin_version(binary: Optional[str]) -> Optional[str]:
         # stderr(f'        {binary} --version')
         # stderr()
         # stderr('    If you don\'t want to install it, you can disable it via config. See here for more info:')
-        # stderr('        https://github.com/pirate/ArchiveBox/wiki/Install')
+        # stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Install')
     return None
 
 def bin_path(binary: Optional[str]) -> Optional[str]:
     if binary is None:
         return None
 
-    return shutil.which(os.path.expanduser(binary)) or binary
+    node_modules_bin = Path('.') / 'node_modules' / '.bin' / binary
+    if node_modules_bin.exists():
+        return str(node_modules_bin.resolve())
+
+    return shutil.which(str(Path(binary).expanduser())) or shutil.which(str(binary)) or binary
 
 def bin_hash(binary: Optional[str]) -> Optional[str]:
     if binary is None:
@@ -564,8 +653,6 @@ def find_chrome_binary() -> Optional[str]:
         if full_path_exists:
             return name
     
-    stderr('[X] Unable to find a working version of Chrome/Chromium, is it installed and in your $PATH?', color='red')
-    stderr()
     return None
 
 def find_chrome_data_dir() -> Optional[str]:
@@ -588,105 +675,105 @@ def find_chrome_data_dir() -> Optional[str]:
         '~/.config/google-chrome-dev',
     )
     for path in default_profile_paths:
-        full_path = os.path.expanduser(path)
-        if os.path.exists(full_path):
+        full_path = Path(path).resolve()
+        if full_path.exists():
             return full_path
     return None
 
 def wget_supports_compression(config):
-    cmd = [
-        config['WGET_BINARY'],
-        "--compression=auto",
-        "--help",
-    ]
-    return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode
+    try:
+        cmd = [
+            config['WGET_BINARY'],
+            "--compression=auto",
+            "--help",
+        ]
+        return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode
+    except (FileNotFoundError, OSError):
+        return False
 
 def get_code_locations(config: ConfigDict) -> SimpleConfigValueDict:
     return {
-        'REPO_DIR': {
-            'path': os.path.abspath(config['REPO_DIR']),
+        'PACKAGE_DIR': {
+            'path': (config['PACKAGE_DIR']).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['REPO_DIR'], 'archivebox')),
-        },
-        'PYTHON_DIR': {
-            'path': os.path.abspath(config['PYTHON_DIR']),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['PYTHON_DIR'], '__main__.py')),
+            'is_valid': (config['PACKAGE_DIR'] / '__main__.py').exists(),
         },
         'TEMPLATES_DIR': {
-            'path': os.path.abspath(config['TEMPLATES_DIR']),
+            'path': (config['TEMPLATES_DIR']).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['TEMPLATES_DIR'], 'static')),
+            'is_valid': (config['TEMPLATES_DIR'] / 'static').exists(),
         },
+        # 'NODE_MODULES_DIR': {
+        #     'path': ,
+        #     'enabled': ,
+        #     'is_valid': (...).exists(),
+        # },
     }
 
 def get_external_locations(config: ConfigDict) -> ConfigValue:
-    abspath = lambda path: None if path is None else os.path.abspath(path)
+    abspath = lambda path: None if path is None else Path(path).resolve()
     return {
         'CHROME_USER_DATA_DIR': {
             'path': abspath(config['CHROME_USER_DATA_DIR']),
             'enabled': config['USE_CHROME'] and config['CHROME_USER_DATA_DIR'],
-            'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else os.path.exists(os.path.join(config['CHROME_USER_DATA_DIR'], 'Default')),
+            'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(),
         },
         'COOKIES_FILE': {
             'path': abspath(config['COOKIES_FILE']),
             'enabled': config['USE_WGET'] and config['COOKIES_FILE'],
-            'is_valid': False if config['COOKIES_FILE'] is None else os.path.exists(config['COOKIES_FILE']),
+            'is_valid': False if config['COOKIES_FILE'] is None else Path(config['COOKIES_FILE']).exists(),
         },
     }
 
 def get_data_locations(config: ConfigDict) -> ConfigValue:
     return {
         'OUTPUT_DIR': {
-            'path': os.path.abspath(config['OUTPUT_DIR']),
+            'path': config['OUTPUT_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
+            'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(),
         },
         'SOURCES_DIR': {
-            'path': os.path.abspath(config['SOURCES_DIR']),
+            'path': config['SOURCES_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['SOURCES_DIR']),
+            'is_valid': config['SOURCES_DIR'].exists(),
         },
         'LOGS_DIR': {
-            'path': os.path.abspath(config['LOGS_DIR']),
+            'path': config['LOGS_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['LOGS_DIR']),
+            'is_valid': config['LOGS_DIR'].exists(),
         },
         'ARCHIVE_DIR': {
-            'path': os.path.abspath(config['ARCHIVE_DIR']),
+            'path': config['ARCHIVE_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['ARCHIVE_DIR']),
+            'is_valid': config['ARCHIVE_DIR'].exists(),
         },
         'CONFIG_FILE': {
-            'path': os.path.abspath(config['CONFIG_FILE']),
+            'path': config['CONFIG_FILE'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['CONFIG_FILE']),
+            'is_valid': config['CONFIG_FILE'].exists(),
         },
         'SQL_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], SQL_INDEX_FILENAME)),
+            'path': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], SQL_INDEX_FILENAME)),
-        },
-        'JSON_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
-        },
-        'HTML_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], HTML_INDEX_FILENAME)),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], HTML_INDEX_FILENAME)),
+            'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(),
         },
     }
 
 def get_dependency_info(config: ConfigDict) -> ConfigValue:
     return {
+        'ARCHIVEBOX_BINARY': {
+            'path': bin_path(config['ARCHIVEBOX_BINARY']),
+            'version': config['VERSION'],
+            'hash': bin_hash(config['ARCHIVEBOX_BINARY']),
+            'enabled': True,
+            'is_valid': True,
+        },
         'PYTHON_BINARY': {
             'path': bin_path(config['PYTHON_BINARY']),
             'version': config['PYTHON_VERSION'],
             'hash': bin_hash(config['PYTHON_BINARY']),
             'enabled': True,
-            'is_valid': bool(config['DJANGO_VERSION']),
+            'is_valid': bool(config['PYTHON_VERSION']),
         },
         'DJANGO_BINARY': {
             'path': bin_path(config['DJANGO_BINARY']),
@@ -698,7 +785,7 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
         'CURL_BINARY': {
             'path': bin_path(config['CURL_BINARY']),
             'version': config['CURL_VERSION'],
-            'hash': bin_hash(config['PYTHON_BINARY']),
+            'hash': bin_hash(config['CURL_BINARY']),
             'enabled': config['USE_CURL'],
             'is_valid': bool(config['CURL_VERSION']),
         },
@@ -709,6 +796,13 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
             'enabled': config['USE_WGET'],
             'is_valid': bool(config['WGET_VERSION']),
         },
+        'NODE_BINARY': {
+            'path': bin_path(config['NODE_BINARY']),
+            'version': config['NODE_VERSION'],
+            'hash': bin_hash(config['NODE_BINARY']),
+            'enabled': config['USE_NODE'],
+            'is_valid': bool(config['NODE_VERSION']),
+        },
         'SINGLEFILE_BINARY': {
             'path': bin_path(config['SINGLEFILE_BINARY']),
             'version': config['SINGLEFILE_VERSION'],
@@ -723,6 +817,13 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
             'enabled': config['USE_READABILITY'],
             'is_valid': bool(config['READABILITY_VERSION']),
         },
+        'MERCURY_BINARY': {
+            'path': bin_path(config['MERCURY_BINARY']),
+            'version': config['MERCURY_VERSION'],
+            'hash': bin_hash(config['MERCURY_BINARY']),
+            'enabled': config['USE_MERCURY'],
+            'is_valid': bool(config['MERCURY_VERSION']),
+        },
         'GIT_BINARY': {
             'path': bin_path(config['GIT_BINARY']),
             'version': config['GIT_VERSION'],
@@ -744,6 +845,21 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
             'enabled': config['USE_CHROME'],
             'is_valid': bool(config['CHROME_VERSION']),
         },
+        'RIPGREP_BINARY': {
+            'path': bin_path(config['RIPGREP_BINARY']),
+            'version': config['RIPGREP_VERSION'],
+            'hash': bin_hash(config['RIPGREP_BINARY']),
+            'enabled': config['USE_RIPGREP'],
+            'is_valid': bool(config['RIPGREP_VERSION']),
+        },
+        # TODO: add an entry for the sonic search backend?
+        # 'SONIC_BINARY': {
+        #     'path': bin_path(config['SONIC_BINARY']),
+        #     'version': config['SONIC_VERSION'],
+        #     'hash': bin_hash(config['SONIC_BINARY']),
+        #     'enabled': config['USE_SONIC'],
+        #     'is_valid': bool(config['SONIC_VERSION']),
+        # },
     }
 
 def get_chrome_info(config: ConfigDict) -> ConfigValue:
@@ -759,37 +875,69 @@ def get_chrome_info(config: ConfigDict) -> ConfigValue:
     }
 
 
-################################## Load Config #################################
+# ******************************************************************************
+# ******************************************************************************
+# ******************************** Load Config *********************************
+# ******* (compile the defaults, configs, and metadata all into CONFIG) ********
+# ******************************************************************************
+# ******************************************************************************
 
 
 def load_all_config():
     CONFIG: ConfigDict = {}
-    for section_name, section_config in CONFIG_DEFAULTS.items():
+    for section_name, section_config in CONFIG_SCHEMA.items():
         CONFIG = load_config(section_config, CONFIG)
 
-    return load_config(DERIVED_CONFIG_DEFAULTS, CONFIG)
+    return load_config(DYNAMIC_CONFIG_SCHEMA, CONFIG)
 
+# add all final config values in CONFIG to globals in this file
 CONFIG = load_all_config()
 globals().update(CONFIG)
+# this lets us do:  from .config import DEBUG, MEDIA_TIMEOUT, ...
 
-# Timezone set as UTC
+
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+
+
+
+########################### System Environment Setup ###########################
+
+
+# Set timezone to UTC and umask to OUTPUT_PERMISSIONS
 os.environ["TZ"] = 'UTC'
+os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8))  # noqa: F821
+
+# add ./node_modules/.bin to $PATH so we can use node scripts in extractors
+NODE_BIN_PATH = str((Path(CONFIG["OUTPUT_DIR"]).absolute() / 'node_modules' / '.bin'))
+sys.path.append(NODE_BIN_PATH)
+
+# disable stderr "you really shouldnt disable ssl" warnings with library config
+if not CONFIG['CHECK_SSL_VALIDITY']:
+    import urllib3
+    import requests
+    requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)
+    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
 
-############################## Importable Checkers #############################
+########################### Config Validity Checkers ###########################
+
 
 def check_system_config(config: ConfigDict=CONFIG) -> None:
     ### Check system environment
     if config['USER'] == 'root':
         stderr('[!] ArchiveBox should never be run as root!', color='red')
         stderr('    For more information, see the security overview documentation:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Security-Overview#do-not-run-as-root')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root')
         raise SystemExit(2)
 
     ### Check Python environment
     if sys.version_info[:3] < (3, 6, 0):
         stderr(f'[X] Python version is not new enough: {config["PYTHON_VERSION"]} (>3.6 is required)', color='red')
-        stderr('    See https://github.com/pirate/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.')
+        stderr('    See https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.')
         raise SystemExit(2)
 
     if config['PYTHON_ENCODING'] not in ('UTF-8', 'UTF8'):
@@ -804,28 +952,18 @@ def check_system_config(config: ConfigDict=CONFIG) -> None:
     # stderr('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY))
     # stderr('[i] Using Chrome data dir: {}'.format(os.path.abspath(CHROME_USER_DATA_DIR)))
     if config['CHROME_USER_DATA_DIR'] is not None:
-        if not os.path.exists(os.path.join(config['CHROME_USER_DATA_DIR'], 'Default')):
+        if not (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists():
             stderr('[X] Could not find profile "Default" in CHROME_USER_DATA_DIR.', color='red')
             stderr(f'    {config["CHROME_USER_DATA_DIR"]}')
             stderr('    Make sure you set it to a Chrome user data directory containing a Default profile folder.')
             stderr('    For more info see:')
-            stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR')
-            if 'Default' in config['CHROME_USER_DATA_DIR']:
+            stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR')
+            if '/Default' in str(config['CHROME_USER_DATA_DIR']):
                 stderr()
                 stderr('    Try removing /Default from the end e.g.:')
                 stderr('        CHROME_USER_DATA_DIR="{}"'.format(config['CHROME_USER_DATA_DIR'].split('/Default')[0]))
             raise SystemExit(2)
 
-def print_dependency_additional_info(dependency: str) -> None:
-    if dependency == "SINGLEFILE_BINARY":
-        hint(('npm install -g git+https://github.com/gildas-lormeau/SingleFile.git"',
-              'or set SAVE_SINGLEFILE=False to silence this warning',
-              ''))
-    if dependency == "READABILITY_BINARY":
-        hint(('npm install -g git+https://github.com/pirate/readability-extractor.git"',
-              'or set SAVE_READABILITY=False to silence this warning',
-              ''))
-
 
 def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None:
     invalid_dependencies = [
@@ -842,45 +980,45 @@ def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None:
                     info['version'] or 'unable to detect version',
                 )
             )
-            print_dependency_additional_info(dependency)
-        stderr('    {lightred}Hint:{reset} To get more info on dependencies run:'.format(**ANSI))
-        stderr('          archivebox --version')
+            if dependency in ('SINGLEFILE_BINARY', 'READABILITY_BINARY', 'MERCURY_BINARY'):
+                hint(('npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"',
+                    f'or archivebox config --set SAVE_{dependency.rsplit("_", 1)[0]}=False to silence this warning',
+                    ''), prefix='      ')
         stderr('')
 
     if config['TIMEOUT'] < 5:
-        stderr()
         stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red')
         stderr('    You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.')
         stderr('    (Setting it to somewhere between 30 and 3000 seconds is recommended)')
         stderr()
         stderr('    If you want to make ArchiveBox run faster, disable specific archive methods instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr()
 
     elif config['USE_CHROME'] and config['TIMEOUT'] < 15:
-        stderr()
         stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red')
         stderr('    Chrome will fail to archive all sites if set to less than ~15 seconds.')
         stderr('    (Setting it to somewhere between 30 and 300 seconds is recommended)')
         stderr()
         stderr('    If you want to make ArchiveBox run faster, disable specific archive methods instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr()
 
     if config['USE_YOUTUBEDL'] and config['MEDIA_TIMEOUT'] < 20:
-        stderr()
         stderr(f'[!] Warning: MEDIA_TIMEOUT is set too low! (currently set to MEDIA_TIMEOUT={config["MEDIA_TIMEOUT"]} seconds)', color='red')
         stderr('    Youtube-dl will fail to archive all media if set to less than ~20 seconds.')
         stderr('    (Setting it somewhere over 60 seconds is recommended)')
         stderr()
         stderr('    If you want to disable media archiving entirely, set SAVE_MEDIA=False instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#save_media')
-
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_media')
+        stderr()
         
-def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) -> None:
+def check_data_folder(out_dir: Union[str, Path, None]=None, config: ConfigDict=CONFIG) -> None:
     output_dir = out_dir or config['OUTPUT_DIR']
-    assert isinstance(output_dir, str)
+    assert isinstance(output_dir, (str, Path))
 
-    json_index_exists = os.path.exists(os.path.join(output_dir, JSON_INDEX_FILENAME))
-    if not json_index_exists:
+    sql_index_exists = (Path(output_dir) / SQL_INDEX_FILENAME).exists()
+    if not sql_index_exists:
         stderr('[X] No archivebox index found in the current directory.', color='red')
         stderr(f'    {output_dir}', color='lightyellow')
         stderr()
@@ -892,8 +1030,7 @@ def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) ->
         stderr('        archivebox init')
         raise SystemExit(2)
 
-    sql_index_exists = os.path.exists(os.path.join(output_dir, SQL_INDEX_FILENAME))
-    from ..index.sql import list_migrations
+    from .index.sql import list_migrations
 
     pending_migrations = [name for status, name in list_migrations() if not status]
 
@@ -910,31 +1047,38 @@ def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) ->
         stderr('        archivebox init')
         raise SystemExit(3)
 
-    sources_dir = os.path.join(output_dir, SOURCES_DIR_NAME)
-    if not os.path.exists(sources_dir):
-        os.makedirs(sources_dir)
+    sources_dir = Path(output_dir) / SOURCES_DIR_NAME
+    if not sources_dir.exists():
+        sources_dir.mkdir()
 
 
 
-def setup_django(out_dir: str=None, check_db=False, config: ConfigDict=CONFIG) -> None:
+def setup_django(out_dir: Path=None, check_db=False, config: ConfigDict=CONFIG, in_memory_db=False) -> None:
     check_system_config()
     
-    output_dir = out_dir or config['OUTPUT_DIR']
+    output_dir = out_dir or Path(config['OUTPUT_DIR'])
 
-    assert isinstance(output_dir, str) and isinstance(config['PYTHON_DIR'], str)
+    assert isinstance(output_dir, Path) and isinstance(config['PACKAGE_DIR'], Path)
 
     try:
         import django
-        sys.path.append(config['PYTHON_DIR'])
-        os.environ.setdefault('OUTPUT_DIR', output_dir)
+        sys.path.append(str(config['PACKAGE_DIR']))
+        os.environ.setdefault('OUTPUT_DIR', str(output_dir))
+        assert (config['PACKAGE_DIR'] / 'core' / 'settings.py').exists(), 'settings.py was not found at archivebox/core/settings.py'
         os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings')
-        django.setup()
+
+        if in_memory_db:
+            # Put the db in memory and run migrations in case any command requires it
+            from django.core.management import call_command
+            os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:")
+            django.setup()
+            call_command("migrate", interactive=False, verbosity=0)
+        else:
+            django.setup()
 
         if check_db:
-            sql_index_path = os.path.join(output_dir, SQL_INDEX_FILENAME)
-            assert os.path.exists(sql_index_path), (
+            sql_index_path = Path(output_dir) / SQL_INDEX_FILENAME
+            assert sql_index_path.exists(), (
                 f'No database file {SQL_INDEX_FILENAME} found in OUTPUT_DIR: {config["OUTPUT_DIR"]}')
     except KeyboardInterrupt:
         raise SystemExit(2)
-
-os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8))  # noqa: F821

archivebox/config_stubs.py

@@ -1,7 +1,9 @@
-from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any
+from pathlib import Path
+from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List
 from mypy_extensions import TypedDict
 
 
+
 SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]]
 SimpleConfigValueDict = Dict[str, SimpleConfigValue]
 SimpleConfigValueGetter = Callable[[], SimpleConfigValue]
@@ -31,35 +33,37 @@ class ConfigDict(BaseConfig, total=False):
     SHOW_PROGRESS: bool
     IN_DOCKER: bool
 
-    OUTPUT_DIR: str
-    CONFIG_FILE: str
+    PACKAGE_DIR: Path
+    OUTPUT_DIR: Path
+    CONFIG_FILE: Path
     ONLY_NEW: bool
     TIMEOUT: int
     MEDIA_TIMEOUT: int
     OUTPUT_PERMISSIONS: str
-    URL_BLACKLIST: Optional[str]
+    RESTRICT_FILE_NAMES: str
+    URL_BLACKLIST: str
 
-    SECRET_KEY: str
+    SECRET_KEY: Optional[str]
     BIND_ADDR: str
     ALLOWED_HOSTS: str
     DEBUG: bool
     PUBLIC_INDEX: bool
     PUBLIC_SNAPSHOTS: bool
     FOOTER_INFO: str
-    ACTIVE_THEME: str
 
     SAVE_TITLE: bool
     SAVE_FAVICON: bool
     SAVE_WGET: bool
     SAVE_WGET_REQUISITES: bool
+    SAVE_SINGLEFILE: bool
+    SAVE_READABILITY: bool
+    SAVE_MERCURY: bool
     SAVE_PDF: bool
     SAVE_SCREENSHOT: bool
     SAVE_DOM: bool
-    SAVE_SINGLEFILE: bool
     SAVE_WARC: bool
     SAVE_GIT: bool
     SAVE_MEDIA: bool
-    SAVE_PLAYLISTS: bool
     SAVE_ARCHIVE_DOT_ORG: bool
 
     RESOLUTION: str
@@ -68,60 +72,32 @@ class ConfigDict(BaseConfig, total=False):
     CURL_USER_AGENT: str
     WGET_USER_AGENT: str
     CHROME_USER_AGENT: str
-    COOKIES_FILE: Optional[str]
-    CHROME_USER_DATA_DIR: Optional[str]
+    COOKIES_FILE: Union[str, Path, None]
+    CHROME_USER_DATA_DIR: Union[str, Path, None]
     CHROME_HEADLESS: bool
     CHROME_SANDBOX: bool
 
     USE_CURL: bool
     USE_WGET: bool
+    USE_SINGLEFILE: bool
+    USE_READABILITY: bool
+    USE_MERCURY: bool
     USE_GIT: bool
     USE_CHROME: bool
     USE_YOUTUBEDL: bool
-    USE_SINGLEFILE: bool
-
-    CURL_BINARY: Optional[str]
-    GIT_BINARY: Optional[str]
-    WGET_BINARY: Optional[str]
-    YOUTUBEDL_BINARY: Optional[str]
+    CURL_BINARY: str
+    GIT_BINARY: str
+    WGET_BINARY: str
+    SINGLEFILE_BINARY: str
+    READABILITY_BINARY: str
+    MERCURY_BINARY: str
+    YOUTUBEDL_BINARY: str
     CHROME_BINARY: Optional[str]
-    SINGLEFILE_BINARY: Optional[str]
 
-    TERM_WIDTH: Callable[[], int]
-    USER: str
-    ANSI: Dict[str, str]
-    REPO_DIR: str
-    PYTHON_DIR: str
-    TEMPLATES_DIR: str
-    ARCHIVE_DIR: str
-    SOURCES_DIR: str
-    LOGS_DIR: str
-
-    URL_BLACKLIST_PTN: Optional[Pattern]
-    WGET_AUTO_COMPRESSION: bool
-
-    ARCHIVEBOX_BINARY: str
-    VERSION: str
-    GIT_SHA: str
-
-    PYTHON_BINARY: str
-    PYTHON_ENCODING: str
-    PYTHON_VERSION: str
-
-    DJANGO_BINARY: str
-    DJANGO_VERSION: str
-
-    CURL_VERSION: str
-    WGET_VERSION: str
-    YOUTUBEDL_VERSION: str
-    GIT_VERSION: str
-    CHROME_VERSION: str
-
-    DEPENDENCIES: Dict[str, SimpleConfigValueDict]
-    CODE_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    CONFIG_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    DATA_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    CHROME_OPTIONS: Dict[str, SimpleConfigValue]
+    YOUTUBEDL_ARGS: List[str]
+    WGET_ARGS: List[str]
+    CURL_ARGS: List[str]
+    GIT_ARGS: List[str]
 
 
 ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue]

archivebox/core/admin.py

@@ -2,7 +2,6 @@ __package__ = 'archivebox.core'
 
 from io import StringIO
 from contextlib import redirect_stdout
-from pathlib import Path
 
 from django.contrib import admin
 from django.urls import path
@@ -10,15 +9,30 @@ from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.shortcuts import render, redirect
 from django.contrib.auth import get_user_model
+from django import forms
 
-from core.models import Snapshot
-from core.forms import AddLinkForm
+from ..util import htmldecode, urldecode, ansi_to_html
 
-from util import htmldecode, urldecode, ansi_to_html
+from core.models import Snapshot, Tag
+from core.forms import AddLinkForm, TagField
+
+from core.mixins import SearchResultsAdminMixin
+
+from index.html import snapshot_icons
 from logging_util import printable_filesize
 from main import add, remove
 from config import OUTPUT_DIR
-from extractors import archive_links
+from extractors import archive_snapshots
+
+# Admin URLs
+# /admin/
+# /admin/login/
+# /admin/core/
+# /admin/core/snapshot/
+# /admin/core/snapshot/:uuid/
+# /admin/core/tag/
+# /admin/core/tag/:uuid/
+
 
 # TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel
 
@@ -50,21 +64,62 @@ def verify_snapshots(modeladmin, request, queryset):
 verify_snapshots.short_description = "Check"
 
 def delete_snapshots(modeladmin, request, queryset):
-    remove(links=[snapshot.as_link() for snapshot in queryset], yes=True, delete=True, out_dir=OUTPUT_DIR)
+    remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR)
 
 delete_snapshots.short_description = "Delete"
 
 
-class SnapshotAdmin(admin.ModelAdmin):
+class SnapshotAdminForm(forms.ModelForm):
+    tags = TagField(required=False)
+
+    class Meta:
+        model = Snapshot
+        fields = "__all__"
+
+    def save(self, commit=True):
+        # Based on: https://stackoverflow.com/a/49933068/3509554
+
+        # Get the unsave instance
+        instance = forms.ModelForm.save(self, False)
+        tags = self.cleaned_data.pop("tags")
+
+        #update save_m2m
+        def new_save_m2m():
+            instance.save_tags(tags)
+
+        # Do we need to save all changes now?
+        self.save_m2m = new_save_m2m
+        if commit:
+            instance.save()
+
+        return instance
+
+
+class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin):
     list_display = ('added', 'title_str', 'url_str', 'files', 'size')
     sort_fields = ('title_str', 'url_str', 'added')
-    readonly_fields = ('id', 'url', 'timestamp', 'title', 'tags', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated')
-    search_fields = ('url', 'timestamp', 'title', 'tags')
-    fields = (*readonly_fields,)
+    readonly_fields = ('id', 'url', 'timestamp', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated')
+    search_fields = ['url__icontains', 'timestamp', 'title', 'tags__name']
+    fields = (*readonly_fields, 'title', 'tags')
     list_filter = ('added', 'updated', 'tags')
     ordering = ['-added']
     actions = [delete_snapshots, overwrite_snapshots, update_snapshots, update_titles, verify_snapshots]
     actions_template = 'admin/actions_as_select.html'
+    form = SnapshotAdminForm
+    list_per_page = 40
+
+    def get_urls(self):
+        urls = super().get_urls()
+        custom_urls = [
+            path('grid/', self.admin_site.admin_view(self.grid_view),name='grid')
+        ]
+        return custom_urls + urls
+
+    def get_queryset(self, request):
+        return super().get_queryset(request).prefetch_related('tags')
+
+    def tag_list(self, obj):
+        return ', '.join(obj.tags.values_list('name', flat=True))
 
     def id_str(self, obj):
         return format_html(
@@ -75,9 +130,10 @@ class SnapshotAdmin(admin.ModelAdmin):
     def title_str(self, obj):
         canon = obj.as_link().canonical_outputs()
         tags = ''.join(
-            format_html('<span>{}</span>', tag.strip())
-            for tag in obj.tags.split(',')
-        ) if obj.tags else ''
+            format_html('<a href="/admin/core/snapshot/?tags__id__exact={}"><span class="tag">{}</span></a> ', tag.id, tag)
+            for tag in obj.tags.all()
+            if str(tag).strip()
+        )
         return format_html(
             '<a href="/{}">'
                 '<img src="/{}/{}" class="favicon" onerror="this.remove()">'
@@ -90,52 +146,57 @@ class SnapshotAdmin(admin.ModelAdmin):
             obj.archive_path,
             'fetched' if obj.latest_title or obj.title else 'pending',
             urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...'
-        ) + mark_safe(f'<span class="tags">{tags}</span>')
+        ) + mark_safe(f' <span class="tags">{tags}</span>')
 
     def files(self, obj):
-        link = obj.as_link()
-        canon = link.canonical_outputs()
-        out_dir = Path(link.link_dir)
-
-        link_tuple = lambda link, method: (link.archive_path, canon[method] or '', canon[method] and (out_dir / (canon[method] or 'notdone')).exists())
-
-        return format_html(
-            '<span class="files-icons" style="font-size: 1.2em; opacity: 0.8">'
-                '<a href="/{}/{}/" class="exists-{}" title="Wget clone">🌐 </a> '
-                '<a href="/{}/{}" class="exists-{}" title="PDF">📄</a> '
-                '<a href="/{}/{}" class="exists-{}" title="Screenshot">🖥 </a> '
-                '<a href="/{}/{}" class="exists-{}" title="HTML dump">🅷 </a> '
-                '<a href="/{}/{}/" class="exists-{}" title="WARC">🆆 </a> '
-                '<a href="/{}/{}" class="exists-{}" title="SingleFile">&#128476; </a>'
-                '<a href="/{}/{}/" class="exists-{}" title="Media files">📼 </a> '
-                '<a href="/{}/{}/" class="exists-{}" title="Git repos">📦 </a> '
-                '<a href="{}" class="exists-{}" title="Archive.org snapshot">🏛 </a> '
-            '</span>',
-            *link_tuple(link, 'wget_path'),
-            *link_tuple(link, 'pdf_path'),
-            *link_tuple(link, 'screenshot_path'),
-            *link_tuple(link, 'dom_path'),
-            *link_tuple(link, 'warc_path')[:2], any((out_dir / canon['warc_path']).glob('*.warc.gz')),
-            *link_tuple(link, 'singlefile_path'),
-            *link_tuple(link, 'media_path')[:2], any((out_dir / canon['media_path']).glob('*')),
-            *link_tuple(link, 'git_path')[:2], any((out_dir / canon['git_path']).glob('*')),
-            canon['archive_org_path'], (out_dir / 'archive.org.txt').exists(),
-        )
+        return snapshot_icons(obj)
 
     def size(self, obj):
+        archive_size = obj.archive_size
+        if archive_size:
+            size_txt = printable_filesize(archive_size)
+            if archive_size > 52428800:
+                size_txt = mark_safe(f'<b>{size_txt}</b>')
+        else:
+            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
         return format_html(
             '<a href="/{}" title="View all files">{}</a>',
             obj.archive_path,
-            printable_filesize(obj.archive_size) if obj.archive_size else 'pending',
+            size_txt,
         )
 
     def url_str(self, obj):
         return format_html(
-            '<a href="{}">{}</a>',
+            '<a href="{}"><code>{}</code></a>',
             obj.url,
             obj.url.split('://www.', 1)[-1].split('://', 1)[-1][:64],
         )
 
+    def grid_view(self, request):
+
+        # cl = self.get_changelist_instance(request)
+
+        # Save before monkey patching to restore for changelist list view
+        saved_change_list_template = self.change_list_template
+        saved_list_per_page = self.list_per_page
+        saved_list_max_show_all = self.list_max_show_all
+
+        # Monkey patch here plus core_tags.py
+        self.change_list_template = 'private_index_grid.html'
+        self.list_per_page = 20
+        self.list_max_show_all = self.list_per_page
+
+        # Call monkey patched view
+        rendered_response = self.changelist_view(request)
+
+        # Restore values
+        self.change_list_template =  saved_change_list_template
+        self.list_per_page = saved_list_per_page
+        self.list_max_show_all = saved_list_max_show_all
+
+        return rendered_response
+        
+
     id_str.short_description = 'ID'
     title_str.short_description = 'Title'
     url_str.short_description = 'Original URL'
@@ -144,6 +205,12 @@ class SnapshotAdmin(admin.ModelAdmin):
     title_str.admin_order_field = 'title'
     url_str.admin_order_field = 'url'
 
+class TagAdmin(admin.ModelAdmin):
+    list_display = ('slug', 'name', 'id')
+    sort_fields = ('id', 'name', 'slug')
+    readonly_fields = ('id',)
+    search_fields = ('id', 'name', 'slug')
+    fields = (*readonly_fields, 'name', 'slug')
 
 
 class ArchiveBoxAdmin(admin.AdminSite):
@@ -193,10 +260,10 @@ class ArchiveBoxAdmin(admin.AdminSite):
             else:
                 context["form"] = form
 
-        return render(template_name='add_links.html', request=request, context=context)
-
+        return render(template_name='add.html', request=request, context=context)
 
 admin.site = ArchiveBoxAdmin()
 admin.site.register(get_user_model())
 admin.site.register(Snapshot, SnapshotAdmin)
+admin.site.register(Tag, TagAdmin)
 admin.site.disable_action('delete_selected')

archivebox/core/forms.py

@@ -3,12 +3,87 @@ __package__ = 'archivebox.core'
 from django import forms
 
 from ..util import URL_REGEX
+from ..vendor.taggit_utils import edit_string_for_tags, parse_tags
 
 CHOICES = (
     ('0', 'depth = 0 (archive just these URLs)'),
     ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'),
 )
 
+from ..extractors import get_default_archive_methods
+
+ARCHIVE_METHODS = [
+    (name, name)
+    for name, _, _ in get_default_archive_methods()
+]
+
+
 class AddLinkForm(forms.Form):
     url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True)
     depth = forms.ChoiceField(label="Archive depth", choices=CHOICES, widget=forms.RadioSelect, initial='0')
+    archive_methods = forms.MultipleChoiceField(
+        label="Archive methods (select at least 1, otherwise all will be used by default)",
+        required=False,
+        widget=forms.SelectMultiple,
+        choices=ARCHIVE_METHODS,
+    )
+    # TODO: hook these up to the view and put them 
+    # in a collapsible UI section labeled "Advanced"
+    #
+    # exclude_patterns = forms.CharField(
+    #     label="Exclude patterns",
+    #     min_length='1',
+    #     required=False,
+    #     initial=URL_BLACKLIST,
+    # )
+    # timeout = forms.IntegerField(
+    #     initial=TIMEOUT,
+    # )
+    # overwrite = forms.BooleanField(
+    #     label="Overwrite any existing Snapshots",
+    #     initial=False,
+    # )
+    # index_only = forms.BooleanField(
+    #     label="Add URLs to index without Snapshotting",
+    #     initial=False,
+    # )
+
+class TagWidgetMixin:
+    def format_value(self, value):
+        if value is not None and not isinstance(value, str):
+            value = edit_string_for_tags(value)
+        return super().format_value(value)
+
+class TagWidget(TagWidgetMixin, forms.TextInput):
+    pass
+
+class TagField(forms.CharField):
+    widget = TagWidget
+
+    def clean(self, value):
+        value = super().clean(value)
+        try:
+            return parse_tags(value)
+        except ValueError:
+            raise forms.ValidationError(
+                "Please provide a comma-separated list of tags."
+            )
+
+    def has_changed(self, initial_value, data_value):
+        # Always return False if the field is disabled since self.bound_data
+        # always uses the initial value in this case.
+        if self.disabled:
+            return False
+
+        try:
+            data_value = self.clean(data_value)
+        except forms.ValidationError:
+            pass
+
+        if initial_value is None:
+            initial_value = []
+
+        initial_value = [tag.name for tag in initial_value]
+        initial_value.sort()
+
+        return initial_value != data_value

archivebox/core/migrations/0006_auto_20201012_1520.py

@@ -0,0 +1,70 @@
+# Generated by Django 3.0.8 on 2020-10-12 15:20
+
+from django.db import migrations, models
+from django.utils.text import slugify
+
+def forwards_func(apps, schema_editor):
+    SnapshotModel = apps.get_model("core", "Snapshot")
+    TagModel = apps.get_model("core", "Tag")
+
+    db_alias = schema_editor.connection.alias
+    snapshots = SnapshotModel.objects.all()
+    for snapshot in snapshots:
+        tags = snapshot.tags
+        tag_set = (
+            set(tag.strip() for tag in (snapshot.tags_old or '').split(','))
+        )
+        tag_set.discard("")
+
+        for tag in tag_set:
+            to_add, _ = TagModel.objects.get_or_create(name=tag, slug=slugify(tag))
+            snapshot.tags.add(to_add)
+
+
+def reverse_func(apps, schema_editor):
+    SnapshotModel = apps.get_model("core", "Snapshot")
+    TagModel = apps.get_model("core", "Tag")
+
+    db_alias = schema_editor.connection.alias
+    snapshots = SnapshotModel.objects.all()
+    for snapshot in snapshots:
+        tags = snapshot.tags.values_list("name", flat=True)
+        snapshot.tags_old = ",".join([tag for tag in tags])
+        snapshot.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0005_auto_20200728_0326'),
+    ]
+
+    operations = [
+        migrations.RenameField(
+            model_name='snapshot',
+            old_name='tags',
+            new_name='tags_old',
+        ),
+        migrations.CreateModel(
+            name='Tag',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('name', models.CharField(max_length=100, unique=True, verbose_name='name')),
+                ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')),
+            ],
+            options={
+                'verbose_name': 'Tag',
+                'verbose_name_plural': 'Tags',
+            },
+        ),
+        migrations.AddField(
+            model_name='snapshot',
+            name='tags',
+            field=models.ManyToManyField(to='core.Tag'),
+        ),
+        migrations.RunPython(forwards_func, reverse_func),
+        migrations.RemoveField(
+            model_name='snapshot',
+            name='tags_old',
+        ),
+    ]

archivebox/core/migrations/0007_archiveresult.py

@@ -0,0 +1,97 @@
+# Generated by Django 3.0.8 on 2020-11-04 12:25
+
+import json
+from pathlib import Path
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+from config import CONFIG
+from index.json import to_json
+
+try:
+    JSONField = models.JSONField
+except AttributeError:
+    import jsonfield
+    JSONField = jsonfield.JSONField
+
+
+def forwards_func(apps, schema_editor):
+    from core.models import EXTRACTORS
+
+    Snapshot = apps.get_model("core", "Snapshot")
+    ArchiveResult = apps.get_model("core", "ArchiveResult")
+
+    snapshots = Snapshot.objects.all()
+    for snapshot in snapshots:
+        out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
+
+        try:
+            with open(out_dir / "index.json", "r") as f:
+                fs_index = json.load(f)
+        except Exception as e:
+            continue
+
+        history = fs_index["history"]
+
+        for extractor in history:
+            for result in history[extractor]:
+                ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"] or 'unknown', 
+                start_ts=result["start_ts"], end_ts=result["end_ts"], status=result["status"], pwd=result["pwd"], output=result["output"])
+
+
+def verify_json_index_integrity(snapshot):
+    results = snapshot.archiveresult_set.all()
+    out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
+    with open(out_dir / "index.json", "r") as f:
+        index = json.load(f)
+
+    history = index["history"]
+    index_results = [result for extractor in history for result in history[extractor]]
+    flattened_results = [result["start_ts"] for result in index_results]
+    
+    missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results]
+
+    for missing in missing_results:
+        index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(),
+                                                    "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output,
+                                                    "schema": "ArchiveResult", "status": missing.status})
+
+    json_index = to_json(index)
+    with open(out_dir / "index.json", "w") as f:
+        f.write(json_index)
+
+
+def reverse_func(apps, schema_editor):
+    Snapshot = apps.get_model("core", "Snapshot")
+    ArchiveResult = apps.get_model("core", "ArchiveResult")
+    for snapshot in Snapshot.objects.all():
+        verify_json_index_integrity(snapshot)
+
+    ArchiveResult.objects.all().delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0006_auto_20201012_1520'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ArchiveResult',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('cmd', JSONField()),
+                ('pwd', models.CharField(max_length=256)),
+                ('cmd_version', models.CharField(max_length=32)),
+                ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)),
+                ('output', models.CharField(max_length=512)),
+                ('start_ts', models.DateTimeField()),
+                ('end_ts', models.DateTimeField()),
+                ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)),
+                ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')),
+            ],
+        ),
+        migrations.RunPython(forwards_func, reverse_func),
+    ]

archivebox/core/migrations/0008_auto_20210105_1421.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.3 on 2021-01-05 14:21
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0007_archiveresult'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='cmd_version',
+            field=models.CharField(blank=True, default=None, max_length=32, null=True),
+        ),
+    ]

archivebox/core/mixins.py

@@ -0,0 +1,23 @@
+from django.contrib import messages
+
+from archivebox.search import query_search_index
+
+class SearchResultsAdminMixin(object):
+    def get_search_results(self, request, queryset, search_term):
+        ''' Enhances the search queryset with results from the search backend.
+        '''
+        qs, use_distinct = \
+            super(SearchResultsAdminMixin, self).get_search_results(
+                request, queryset, search_term)
+
+        search_term = search_term.strip()
+        if not search_term:
+            return qs, use_distinct
+        try:
+            qsearch = query_search_index(search_term)
+        except Exception as err:
+            messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}')
+        else:
+            qs = queryset & qsearch
+        finally:
+            return qs, use_distinct

archivebox/core/models.py

@@ -1,12 +1,75 @@
 __package__ = 'archivebox.core'
 
 import uuid
+from pathlib import Path
+from typing import Dict, Optional, List
+from datetime import datetime, timedelta
+from collections import defaultdict
 
-from django.db import models
+from django.db import models, transaction
 from django.utils.functional import cached_property
+from django.utils.text import slugify
+from django.db.models import Case, When, Value, IntegerField
 
 from ..util import parse_date
 from ..index.schema import Link
+from ..config import CONFIG
+from ..system import get_dir_size
+
+#EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()]
+EXTRACTORS = [("title", "title"), ("wget", "wget")]
+STATUS_CHOICES = [
+    ("succeeded", "succeeded"),
+    ("failed", "failed"),
+    ("skipped", "skipped")
+]
+
+try:
+    JSONField = models.JSONField
+except AttributeError:
+    import jsonfield
+    JSONField = jsonfield.JSONField
+
+class Tag(models.Model):
+    """
+    Based on django-taggit model
+    """
+    name = models.CharField(verbose_name="name", unique=True, blank=False, max_length=100)
+    slug = models.SlugField(verbose_name="slug", unique=True, max_length=100)
+
+    class Meta:
+        verbose_name = "Tag"
+        verbose_name_plural = "Tags"
+
+    def __str__(self):
+        return self.name
+
+    def slugify(self, tag, i=None):
+        slug = slugify(tag)
+        if i is not None:
+            slug += "_%d" % i
+        return slug
+
+    def save(self, *args, **kwargs):
+        if self._state.adding and not self.slug:
+            self.slug = self.slugify(self.name)
+
+            with transaction.atomic():
+                slugs = set(
+                    type(self)
+                    ._default_manager.filter(slug__startswith=self.slug)
+                    .values_list("slug", flat=True)
+                )
+
+                i = None
+                while True:
+                    slug = self.slugify(self.name, i)
+                    if slug not in slugs:
+                        self.slug = slug
+                        return super().save(*args, **kwargs)
+                    i = 1 if i is None else i+1
+        else:
+            return super().save(*args, **kwargs)
 
 
 class Snapshot(models.Model):
@@ -16,13 +79,12 @@ class Snapshot(models.Model):
     timestamp = models.CharField(max_length=32, unique=True, db_index=True)
 
     title = models.CharField(max_length=128, null=True, blank=True, db_index=True)
-    tags = models.CharField(max_length=256, null=True, blank=True, db_index=True)
 
     added = models.DateTimeField(auto_now_add=True, db_index=True)
     updated = models.DateTimeField(null=True, blank=True, db_index=True)
-    # bookmarked = models.DateTimeField()
+    tags = models.ManyToManyField(Tag)
 
-    keys = ('url', 'timestamp', 'title', 'tags', 'updated')
+    keys = ('id', 'url', 'timestamp', 'title', 'tags', 'updated', 'base_url')
 
     def __repr__(self) -> str:
         title = self.title or '-'
@@ -32,57 +94,157 @@ class Snapshot(models.Model):
         title = self.title or '-'
         return f'[{self.timestamp}] {self.url[:64]} ({title[:64]})'
 
+    def field_names():
+        fields = self._meta.get_field_names()
+        exclude = ["tags", "archiveresult"] # Exclude relationships for now
+        return [field.name for field in fields if field.name not in exclude]
+
+
     @classmethod
     def from_json(cls, info: dict):
         info = {k: v for k, v in info.items() if k in cls.keys}
+        if "tags" in info:
+            # TODO: Handle tags
+            info.pop("tags")
+        info.pop("base_url", None) 
         return cls(**info)
 
+    def get_history(self) -> dict:
+        """
+        Generates the history dictionary out of the stored ArchiveResults
+        """
+        history_list = self.archiveresult_set.all()
+        history = defaultdict(list)
+        for history_item in history_list:
+            history[history_item.extractor].append(
+                {
+                    "cmd": history_item.cmd,
+                    "cmd_version": history_item.cmd_version,
+                    "end_ts": history_item.end_ts.isoformat(),
+                    "start_ts": history_item.start_ts.isoformat(),
+                    "pwd": history_item.pwd,
+                    "output": history_item.output,
+                    "schema": "ArchiveResult",
+                    "status": history_item.status
+                }
+            )
+        return dict(history)
+
     def as_json(self, *args) -> dict:
+        """
+        Returns the snapshot in json format.
+        id is converted to str
+        history is extracted from ArchiveResult
+        """
         args = args or self.keys
-        return {
+        output = {
             key: getattr(self, key)
+            if key != 'tags' else self.tags_str()
             for key in args
         }
+        if "id" in output.keys():
+            output["id"] = str(output["id"])
+
+        output["history"] = self.get_history()
+        return output
+
+    def as_csv(self, cols: Optional[List[str]]=None, separator: str=',', ljust: int=0) -> str:
+        from ..index.csv import to_csv
+        return to_csv(self, cols=cols or self.field_names(), separator=separator, ljust=ljust)
 
     def as_link(self) -> Link:
         return Link.from_json(self.as_json())
 
+    def as_link_with_details(self) -> Link:
+        from ..index import load_link_details
+        return load_link_details(self.as_link())
+
+    def tags_str(self) -> str:
+        return ','.join(self.tags.order_by('name').values_list('name', flat=True))
+
     @cached_property
     def bookmarked(self):
         return parse_date(self.timestamp)
 
     @cached_property
-    def is_archived(self):
-        return self.as_link().is_archived
+    def bookmarked_date(self) -> Optional[str]:
+        from ..util import ts_to_date
+
+        max_ts = (datetime.now() + timedelta(days=30)).timestamp()
+
+        if self.timestamp and self.timestamp.replace('.', '').isdigit():
+            if 0 < float(self.timestamp) < max_ts:
+                return ts_to_date(datetime.fromtimestamp(float(self.timestamp)))
+            else:
+                return str(self.timestamp)
+        return None
+
+    @cached_property
+    def is_archived(self) -> bool:
+        from ..config import ARCHIVE_DIR
+        from ..util import domain
+
+        output_paths = (
+            domain(self.url),
+            'output.pdf',
+            'screenshot.png',
+            'output.html',
+            'media',
+            'singlefile.html'
+        )
+
+        return any(
+            (Path(ARCHIVE_DIR) / self.timestamp / path).exists()
+            for path in output_paths
+        )
+
+    @cached_property
+    def archive_dates(self) -> List[datetime]:
+        return [
+            result.start_ts
+            for result in self.archiveresult_set.all()
+        ]
+
+    @cached_property
+    def oldest_archive_date(self) -> Optional[datetime]:
+        oldest = self.archiveresult_set.all().order_by("-start_ts")[:1]
+        if len(oldest) > 0:
+            return oldest[0].start_ts
 
     @cached_property
     def num_outputs(self):
-        return self.as_link().num_outputs
+        return self.archiveresult_set.filter(status='succeeded').count()
 
     @cached_property
     def url_hash(self):
         return self.as_link().url_hash
 
     @cached_property
-    def base_url(self):
-        return self.as_link().base_url
+    def base_url(self) -> str:
+        from ..util import base_url
+        return base_url(self.url)
 
     @cached_property
-    def link_dir(self):
-        return self.as_link().link_dir
+    def snapshot_dir(self):
+        from ..config import CONFIG
+        return Path(CONFIG['ARCHIVE_DIR']) / self.timestamp
 
     @cached_property
     def archive_path(self):
-        return self.as_link().archive_path
+        from ..config import ARCHIVE_DIR_NAME
+        return '{}/{}'.format(ARCHIVE_DIR_NAME, self.timestamp)
 
     @cached_property
-    def archive_size(self):
-        return self.as_link().archive_size
+    def archive_size(self) -> float:
+        try:
+            return get_dir_size(self.archive_path)[0]
+        except Exception:
+            return 0
 
     @cached_property
     def history(self):
-        from ..index import load_link_details
-        return load_link_details(self.as_link()).history
+        # TODO: use ArchiveResult for this instead of json
+        return self.as_link_with_details().history
 
     @cached_property
     def latest_title(self):
@@ -92,3 +254,100 @@ class Snapshot(models.Model):
             and self.history['title'][-1].output.strip()):
             return self.history['title'][-1].output.strip()
         return None
+
+    @cached_property
+    def domain(self) -> str:
+        from ..util import domain
+        return domain(self.url)
+
+    @cached_property
+    def is_static(self) -> bool:
+        from ..util import is_static_file
+        return is_static_file(self.url)
+
+    @cached_property
+    def details(self) -> Dict:
+        # TODO: Define what details are, and return them accordingly
+        return {"history": {}}
+
+    @property
+    def extension(self) -> str:
+        from ..util import extension
+        return extension(self.url)
+
+    def canonical_outputs(self) -> Dict[str, Optional[str]]:
+        """predict the expected output paths that should be present after archiving"""
+
+        from ..extractors.wget import wget_output_path
+        canonical = {
+            'index_path': 'index.html',
+            'favicon_path': 'favicon.ico',
+            'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain),
+            'wget_path': wget_output_path(self),
+            'warc_path': 'warc',
+            'singlefile_path': 'singlefile.html',
+            'readability_path': 'readability/content.html',
+            'mercury_path': 'mercury/content.html',
+            'pdf_path': 'output.pdf',
+            'screenshot_path': 'screenshot.png',
+            'dom_path': 'output.html',
+            'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url),
+            'git_path': 'git',
+            'media_path': 'media',
+        }
+        if self.is_static:
+            # static binary files like PDF and images are handled slightly differently.
+            # they're just downloaded once and aren't archived separately multiple times, 
+            # so the wget, screenshot, & pdf urls should all point to the same file
+
+            static_path = wget_output_path(self)
+            canonical.update({
+                'title': self.basename,
+                'wget_path': static_path,
+                'pdf_path': static_path,
+                'screenshot_path': static_path,
+                'dom_path': static_path,
+                'singlefile_path': static_path,
+                'readability_path': static_path,
+                'mercury_path': static_path,
+            })
+        return canonical
+
+    def save_tags(self, tags=()):
+        tags_id = []
+        for tag in tags:
+            tags_id.append(Tag.objects.get_or_create(name=tag)[0].id)
+        self.tags.clear()
+        self.tags.add(*tags_id)
+
+
+class ArchiveResultManager(models.Manager):
+    def indexable(self, sorted: bool = True):
+        from ..extractors import ARCHIVE_METHODS_INDEXING_PRECEDENCE
+        INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
+        qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded')
+
+        if sorted:
+            precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
+            qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence')
+        return qs
+
+
+class ArchiveResult(models.Model):
+    snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE)
+    cmd = JSONField()
+    pwd = models.CharField(max_length=256)
+    cmd_version = models.CharField(max_length=32, default=None, null=True, blank=True)
+    output = models.CharField(max_length=512)
+    start_ts = models.DateTimeField()
+    end_ts = models.DateTimeField()
+    status = models.CharField(max_length=16, choices=STATUS_CHOICES)
+    extractor = models.CharField(choices=EXTRACTORS, max_length=32)
+
+    objects = ArchiveResultManager()
+
+    def __str__(self):
+        return self.extractor
+
+    class Meta:
+        ordering = ["-start_ts"]

archivebox/core/settings.py

@@ -2,23 +2,38 @@ __package__ = 'archivebox.core'
 
 import os
 import sys
-from django.utils.crypto import get_random_string
 
+from pathlib import Path
+from django.utils.crypto import get_random_string
 
 from ..config import (                                                          # noqa: F401
     DEBUG,
     SECRET_KEY,
     ALLOWED_HOSTS,
-    PYTHON_DIR,
-    ACTIVE_THEME,
+    PACKAGE_DIR,
+    TEMPLATES_DIR_NAME,
     SQL_INDEX_FILENAME,
     OUTPUT_DIR,
 )
 
-ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
+
+IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3]
+IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ
 IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3]
 
-SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.')
+################################################################################
+### Django Core Settings
+################################################################################
+
+WSGI_APPLICATION = 'core.wsgi.application'
+ROOT_URLCONF = 'core.urls'
+
+LOGIN_URL = '/accounts/login/'
+LOGOUT_REDIRECT_URL = '/'
+PASSWORD_RESET_URL = '/accounts/password_reset/'
+APPEND_SLASH = True
+
+DEBUG = DEBUG or ('--debug' in sys.argv)
 
 INSTALLED_APPS = [
     'django.contrib.auth',
@@ -43,16 +58,31 @@ MIDDLEWARE = [
     'django.contrib.messages.middleware.MessageMiddleware',
 ]
 
-ROOT_URLCONF = 'core.urls'
-APPEND_SLASH = True
+AUTHENTICATION_BACKENDS = [
+    'django.contrib.auth.backends.ModelBackend',
+]
+
+
+################################################################################
+### Staticfile and Template Settings
+################################################################################
+
+STATIC_URL = '/static/'
+
+STATICFILES_DIRS = [
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'static'),
+]
+
+TEMPLATE_DIRS = [
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'core'),
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'admin'),
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME),
+]
+
 TEMPLATES = [
     {
         'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': [
-            os.path.join(PYTHON_DIR, 'themes', ACTIVE_THEME),
-            os.path.join(PYTHON_DIR, 'themes', 'default'),
-            os.path.join(PYTHON_DIR, 'themes'),
-        ],
+        'DIRS': TEMPLATE_DIRS,
         'APP_DIRS': True,
         'OPTIONS': {
             'context_processors': [
@@ -65,18 +95,42 @@ TEMPLATES = [
     },
 ]
 
-WSGI_APPLICATION = 'core.wsgi.application'
+
+################################################################################
+### External Service Settings
+################################################################################
+
+DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME
+DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", str(DATABASE_FILE))
 
 DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME),
+        'NAME': DATABASE_NAME,
     }
 }
 
-AUTHENTICATION_BACKENDS = [
-    'django.contrib.auth.backends.ModelBackend',
-]
+EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
+
+
+################################################################################
+### Security Settings
+################################################################################
+
+SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.')
+
+ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
+
+SECURE_BROWSER_XSS_FILTER = True
+SECURE_CONTENT_TYPE_NOSNIFF = True
+
+CSRF_COOKIE_SECURE = False
+SESSION_COOKIE_SECURE = False
+SESSION_COOKIE_DOMAIN = None
+SESSION_COOKIE_AGE = 1209600  # 2 weeks
+SESSION_EXPIRE_AT_BROWSER_CLOSE = False
+SESSION_SAVE_EVERY_REQUEST = True
+
 AUTH_PASSWORD_VALIDATORS = [
     {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'},
     {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'},
@@ -84,30 +138,23 @@ AUTH_PASSWORD_VALIDATORS = [
     {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'},
 ]
 
-################################################################################
-### Security Settings
-################################################################################
-SECURE_BROWSER_XSS_FILTER = True
-SECURE_CONTENT_TYPE_NOSNIFF = True
-SESSION_COOKIE_SECURE = False
-CSRF_COOKIE_SECURE = False
-SESSION_COOKIE_DOMAIN = None
-SESSION_EXPIRE_AT_BROWSER_CLOSE = False
-SESSION_SAVE_EVERY_REQUEST = True
-SESSION_COOKIE_AGE = 1209600  # 2 weeks
-LOGIN_URL = '/accounts/login/'
-LOGOUT_REDIRECT_URL = '/'
-PASSWORD_RESET_URL = '/accounts/password_reset/'
 
+################################################################################
+### Shell Settings
+################################################################################
 
 SHELL_PLUS = 'ipython'
 SHELL_PLUS_PRINT_SQL = False
 IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner']
 IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell'
 if IS_SHELL:
-    os.environ['PYTHONSTARTUP'] = os.path.join(PYTHON_DIR, 'core', 'welcome_message.py')
+    os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py')
 
 
+################################################################################
+### Internationalization & Localization Settings
+################################################################################
+
 LANGUAGE_CODE = 'en-us'
 TIME_ZONE = 'UTC'
 USE_I18N = False
@@ -116,12 +163,3 @@ USE_TZ = False
 
 DATETIME_FORMAT = 'Y-m-d g:iA'
 SHORT_DATETIME_FORMAT = 'Y-m-d h:iA'
-
-
-EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
-
-STATIC_URL = '/static/'
-STATICFILES_DIRS = [
-    os.path.join(PYTHON_DIR, 'themes', ACTIVE_THEME, 'static'),
-    os.path.join(PYTHON_DIR, 'themes', 'default', 'static'),
-]

archivebox/core/templatetags/core_tags.py

@@ -0,0 +1,47 @@
+from django import template
+from django.urls import reverse
+from django.contrib.admin.templatetags.base import InclusionAdminNode
+from django.templatetags.static import static
+
+
+from typing import Union
+
+from core.models import ArchiveResult
+
+register = template.Library()
+
+@register.simple_tag
+def snapshot_image(snapshot):
+    result = ArchiveResult.objects.filter(snapshot=snapshot, extractor='screenshot', status='succeeded').first()
+    if result:
+        return reverse('Snapshot', args=[f'{str(snapshot.timestamp)}/{result.output}'])
+    
+    return static('archive.png')
+
+@register.filter
+def file_size(num_bytes: Union[int, float]) -> str:
+    for count in ['Bytes','KB','MB','GB']:
+        if num_bytes > -1024.0 and num_bytes < 1024.0:
+            return '%3.1f %s' % (num_bytes, count)
+        num_bytes /= 1024.0
+    return '%3.1f %s' % (num_bytes, 'TB')
+
+def result_list(cl):
+    """
+    Monkey patched result
+    """
+    num_sorted_fields = 0
+    return {
+        'cl': cl,
+        'num_sorted_fields': num_sorted_fields,
+        'results': cl.result_list,
+    }
+
+@register.tag(name='snapshots_grid')
+def result_list_tag(parser, token):
+    return InclusionAdminNode(
+        parser, token,
+        func=result_list,
+        template_name='snapshots_grid.html',
+        takes_context=False,
+    )

archivebox/core/urls.py

@@ -5,20 +5,24 @@ from django.views import static
 from django.conf import settings
 from django.views.generic.base import RedirectView
 
-from core.views import MainIndex, OldIndex, LinkDetails
+from core.views import HomepageView, SnapshotView, PublicIndexView, AddView
 
 
 # print('DEBUG', settings.DEBUG)
 
 urlpatterns = [
+    path('public/', PublicIndexView.as_view(), name='public-index'),
+
     path('robots.txt', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'robots.txt'}),
     path('favicon.ico', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'favicon.ico'}),
 
-    path('docs/', RedirectView.as_view(url='https://github.com/pirate/ArchiveBox/wiki'), name='Docs'),
+    path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'),
 
     path('archive/', RedirectView.as_view(url='/')),
-    path('archive/<path:path>', LinkDetails.as_view(), name='LinkAssets'),
-    path('add/', RedirectView.as_view(url='/admin/core/snapshot/add/')),
+    path('archive/<path:path>', SnapshotView.as_view(), name='Snapshot'),
+
+    path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')),
+    path('add/', AddView.as_view(), name='add'),
     
     path('accounts/login/', RedirectView.as_view(url='/admin/login/')),
     path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')),
@@ -27,8 +31,39 @@ urlpatterns = [
     path('accounts/', include('django.contrib.auth.urls')),
     path('admin/', admin.site.urls),
     
-    path('old.html', OldIndex.as_view(), name='OldHome'),
     path('index.html', RedirectView.as_view(url='/')),
     path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}),
-    path('', MainIndex.as_view(), name='Home'),
+    path('', HomepageView.as_view(), name='Home'),
 ]
+
+    # # Proposed UI URLs spec
+    # path('',                 HomepageView)
+    # path('/add',             AddView)
+    # path('/public',          PublicIndexView)
+    # path('/snapshot/:slug',  SnapshotView)
+    
+    # path('/admin',           admin.site.urls)
+    # path('/accounts',        django.contrib.auth.urls)
+
+    # # Prposed REST API spec
+    # # :slugs can be uuid, short_uuid, or any of the unique index_fields
+    # path('api/v1/'),
+    # path('api/v1/core/'                      [GET])
+    # path('api/v1/core/snapshot/',            [GET, POST, PUT]),
+    # path('api/v1/core/snapshot/:slug',       [GET, PATCH, DELETE]),
+    # path('api/v1/core/archiveresult',        [GET, POST, PUT]),
+    # path('api/v1/core/archiveresult/:slug',  [GET, PATCH, DELETE]),
+    # path('api/v1/core/tag/',                 [GET, POST, PUT]),
+    # path('api/v1/core/tag/:slug',            [GET, PATCH, DELETE]),
+
+    # path('api/v1/cli/',                      [GET])
+    # path('api/v1/cli/{add,list,config,...}', [POST]),  # pass query as kwargs directly to `run_subcommand` and return stdout, stderr, exitcode
+
+    # path('api/v1/extractors/',                    [GET])
+    # path('api/v1/extractors/:extractor/',         [GET]),
+    # path('api/v1/extractors/:extractor/:func',    [GET, POST]),  # pass query as args directly to chosen function
+
+    # future, just an idea:
+    # path('api/v1/scheduler/',                [GET])
+    # path('api/v1/scheduler/task/',           [GET, POST, PUT]),
+    # path('api/v1/scheduler/task/:slug',      [GET, PATCH, DELETE]),

archivebox/core/views.py

@@ -1,59 +1,47 @@
 __package__ = 'archivebox.core'
 
+from io import StringIO
+from contextlib import redirect_stdout
+
 from django.shortcuts import render, redirect
 
 from django.http import HttpResponse
 from django.views import View, static
+from django.views.generic.list import ListView
+from django.views.generic import FormView
+from django.db.models import Q
+from django.contrib.auth.mixins import UserPassesTestMixin
 
 from core.models import Snapshot
+from core.forms import AddLinkForm
 
-from ..index import load_main_index, load_main_index_meta
 from ..config import (
     OUTPUT_DIR,
-    VERSION,
-    FOOTER_INFO,
     PUBLIC_INDEX,
     PUBLIC_SNAPSHOTS,
+    PUBLIC_ADD_VIEW,
+    VERSION,
+    FOOTER_INFO,
 )
-from ..util import base_url
+from main import add
+from ..util import base_url, ansi_to_html
+from ..index.html import snapshot_icons
 
 
-class MainIndex(View):
-    template = 'main_index.html'
-
+class HomepageView(View):
     def get(self, request):
         if request.user.is_authenticated:
             return redirect('/admin/core/snapshot/')
 
         if PUBLIC_INDEX:
-            return redirect('OldHome')
+            return redirect('/public')
         
         return redirect(f'/admin/login/?next={request.path}')
 
-        
 
-class OldIndex(View):
-    template = 'main_index.html'
+class SnapshotView(View):
+    # render static html index from filesystem archive/<timestamp>/index.html
 
-    def get(self, request):
-        if PUBLIC_INDEX or request.user.is_authenticated:
-            all_links = load_main_index(out_dir=OUTPUT_DIR)
-            meta_info = load_main_index_meta(out_dir=OUTPUT_DIR)
-
-            context = {
-                'updated': meta_info['updated'],
-                'num_links': meta_info['num_links'],
-                'links': all_links,
-                'VERSION': VERSION,
-                'FOOTER_INFO': FOOTER_INFO,
-            }
-
-            return render(template_name=self.template, request=request, context=context)
-
-        return redirect(f'/admin/login/?next={request.path}')
-
-
-class LinkDetails(View):
     def get(self, request, path):
         # missing trailing slash -> redirect to index
         if '/' not in path:
@@ -73,7 +61,7 @@ class LinkDetails(View):
         by_ts = {page.timestamp: page for page in all_pages}
         try:
             # print('SERVING STATICFILE', by_ts[slug].link_dir, request.path, path)
-            response = static.serve(request, archivefile, document_root=by_ts[slug].link_dir, show_indexes=True)
+            response = static.serve(request, archivefile, document_root=by_ts[slug].snapshot_dir, show_indexes=True)
             response["Link"] = f'<{by_ts[slug].url}>; rel="canonical"'
             return response
         except KeyError:
@@ -102,3 +90,85 @@ class LinkDetails(View):
             content_type="text/plain",
             status=404,
         )
+
+class PublicIndexView(ListView):
+    template_name = 'public_index.html'
+    model = Snapshot
+    paginate_by = 100
+    ordering = ['title']
+
+    def get_context_data(self, **kwargs):
+        return {
+            **super().get_context_data(**kwargs),
+            'VERSION': VERSION,
+            'FOOTER_INFO': FOOTER_INFO,
+        }
+
+    def get_queryset(self, **kwargs): 
+        qs = super().get_queryset(**kwargs) 
+        query = self.request.GET.get('q')
+        if query:
+            qs = qs.filter(Q(title__icontains=query) | Q(url__icontains=query) | Q(timestamp__icontains=query) | Q(tags__name__icontains=query))
+        for snapshot in qs:
+            snapshot.icons = snapshot_icons(snapshot)
+        return qs
+
+    def get(self, *args, **kwargs):
+        if PUBLIC_INDEX or self.request.user.is_authenticated:
+            response = super().get(*args, **kwargs)
+            return response
+        else:
+            return redirect(f'/admin/login/?next={self.request.path}')
+
+
+class AddView(UserPassesTestMixin, FormView):
+    template_name = "add.html"
+    form_class = AddLinkForm
+
+    def get_initial(self):
+        """Prefill the AddLinkForm with the 'url' GET parameter"""
+        if self.request.method == 'GET':
+            url = self.request.GET.get('url', None)
+            if url:
+                return {'url': url}
+        else:
+            return super().get_initial()
+
+    def test_func(self):
+        return PUBLIC_ADD_VIEW or self.request.user.is_authenticated
+
+    def get_context_data(self, **kwargs):
+        return {
+            **super().get_context_data(**kwargs),
+            'title': "Add URLs",
+            # We can't just call request.build_absolute_uri in the template, because it would include query parameters
+            'absolute_add_path': self.request.build_absolute_uri(self.request.path),
+            'VERSION': VERSION,
+            'FOOTER_INFO': FOOTER_INFO,
+        }
+
+    def form_valid(self, form):
+        url = form.cleaned_data["url"]
+        print(f'[+] Adding URL: {url}')
+        depth = 0 if form.cleaned_data["depth"] == "0" else 1
+        extractors = ','.join(form.cleaned_data["archive_methods"])
+        input_kwargs = {
+            "urls": url,
+            "depth": depth,
+            "update_all": False,
+            "out_dir": OUTPUT_DIR,
+        }
+        if extractors:
+            input_kwargs.update({"extractors": extractors})
+        add_stdout = StringIO()
+        with redirect_stdout(add_stdout):
+            add(**input_kwargs)
+            print(add_stdout.getvalue())
+
+        context = self.get_context_data()
+
+        context.update({
+            "stdout": ansi_to_html(add_stdout.getvalue().strip()),
+            "form": AddLinkForm()
+        })
+        return render(template_name=self.template_name, request=self.request, context=context)

archivebox/extractors/__init__.py

@@ -1,38 +1,42 @@
 __package__ = 'archivebox.extractors'
 
 import os
+from pathlib import Path
 
-from typing import Optional, List, Iterable
-from datetime import datetime
+from typing import Optional, List, Iterable, Union
+
+from django.db.models import QuerySet, Model
 
-from ..index.schema import Link
 from ..index import (
-    load_link_details,
-    write_link_details,
-    patch_main_index,
+    load_snapshot_details,
+    write_snapshot_details,
 )
 from ..util import enforce_types
 from ..logging_util import (
     log_archiving_started,
     log_archiving_paused,
     log_archiving_finished,
-    log_link_archiving_started,
-    log_link_archiving_finished,
+    log_snapshot_archiving_started,
+    log_snapshot_archiving_finished,
     log_archive_method_started,
     log_archive_method_finished,
 )
+from ..search import write_search_index
 
 from .title import should_save_title, save_title
 from .favicon import should_save_favicon, save_favicon
 from .wget import should_save_wget, save_wget
 from .singlefile import should_save_singlefile, save_singlefile
 from .readability import should_save_readability, save_readability
+from .mercury import should_save_mercury, save_mercury
 from .pdf import should_save_pdf, save_pdf
 from .screenshot import should_save_screenshot, save_screenshot
 from .dom import should_save_dom, save_dom
 from .git import should_save_git, save_git
 from .media import should_save_media, save_media
 from .archive_org import should_save_archive_dot_org, save_archive_dot_org
+from .headers import should_save_headers, save_headers
+
 
 def get_default_archive_methods():
     return [
@@ -44,21 +48,26 @@ def get_default_archive_methods():
         ('screenshot', should_save_screenshot, save_screenshot),
         ('dom', should_save_dom, save_dom),
         ('readability', should_save_readability, save_readability), #keep readability below wget and singlefile, as it depends on them
+        ('mercury', should_save_mercury, save_mercury),
         ('git', should_save_git, save_git),
         ('media', should_save_media, save_media),
+        ('headers', should_save_headers, save_headers),
         ('archive_org', should_save_archive_dot_org, save_archive_dot_org),
     ]
 
+ARCHIVE_METHODS_INDEXING_PRECEDENCE = [('readability', 1), ('singlefile', 2), ('dom', 3), ('wget', 4)]
+
 @enforce_types
 def ignore_methods(to_ignore: List[str]):
     ARCHIVE_METHODS = get_default_archive_methods()
     methods = filter(lambda x: x[0] not in to_ignore, ARCHIVE_METHODS)
-    methods = map(lambda x: x[1], methods)
+    methods = map(lambda x: x[0], methods)
     return list(methods)
 
 @enforce_types
-def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[str]=None, skip_index: bool=False) -> Link:
+def archive_snapshot(snapshot: Model, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Model:
     """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+    from core.models import ArchiveResult
 
     ARCHIVE_METHODS = get_default_archive_methods()
     
@@ -68,64 +77,63 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
             if method[0] in methods
         ]
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     try:
-        is_new = not os.path.exists(out_dir)
+        is_new = not Path(out_dir).exists()
         if is_new:
             os.makedirs(out_dir)
+            details = {"history": {}}
+            write_snapshot_details(snapshot, out_dir=out_dir, skip_sql_index=False)
+        else:
+            details = snapshot.details #TODO: This can be retrieved from the sqlite database too.
+                                       # If that makes more sense, it can be easily changed.
 
-        link = load_link_details(link, out_dir=out_dir)
-        write_link_details(link, out_dir=out_dir, skip_sql_index=skip_index)
-        log_link_archiving_started(link, out_dir, is_new)
-        link = link.overwrite(updated=datetime.now())
+        log_snapshot_archiving_started(snapshot, out_dir, is_new)
         stats = {'skipped': 0, 'succeeded': 0, 'failed': 0}
 
         for method_name, should_run, method_function in ARCHIVE_METHODS:
             try:
-                if method_name not in link.history:
-                    link.history[method_name] = []
+                if method_name not in details["history"]:
+                    details["history"][method_name] = []
 
-                if should_run(link, out_dir) or overwrite:
+                if should_run(snapshot, out_dir, overwrite):
                     log_archive_method_started(method_name)
 
-                    result = method_function(link=link, out_dir=out_dir)
-
-                    link.history[method_name].append(result)
+                    result = method_function(snapshot=snapshot, out_dir=out_dir)
 
                     stats[result.status] += 1
                     log_archive_method_finished(result)
+                    write_search_index(snapshot=snapshot, texts=result.index_texts)
+                    ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version,
+                                                 output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status)
+
                 else:
                     # print('{black}      X {}{reset}'.format(method_name, **ANSI))
                     stats['skipped'] += 1
             except Exception as e:
-                raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format(
+                raise Exception('Exception in archive_methods.save_{}(Snapshot(url={}))'.format(
                     method_name,
-                    link.url,
+                    snapshot.url,
                 )) from e
 
         # print('    ', stats)
 
         try:
-            latest_title = link.history['title'][-1].output.strip()
-            if latest_title and len(latest_title) >= len(link.title or ''):
-                link = link.overwrite(title=latest_title)
+            latest_title_archive_result = snapshot.archiveresult_set.filter(extractor="title")
+            if latest_title_archive_result.count() > 0:
+                latest_title = latest_title_archive_result.output.strip()
+                if len(latest_title) >= len(snapshot.title or ''):
+                    snapshot.title = latest_title
         except Exception:
             pass
 
-        write_link_details(link, out_dir=out_dir, skip_sql_index=skip_index)
-        if not skip_index:
-            patch_main_index(link)
+        write_snapshot_details(snapshot, out_dir=out_dir, skip_sql_index=False)
 
-        # # If any changes were made, update the main links index json and html
-        # was_changed = stats['succeeded'] or stats['failed']
-        # if was_changed:
-        #     patch_main_index(link)
-
-        log_link_archiving_finished(link, link.link_dir, is_new, stats)
+        log_snapshot_archiving_finished(snapshot, snapshot.snapshot_dir, is_new, stats)
 
     except KeyboardInterrupt:
         try:
-            write_link_details(link, out_dir=link.link_dir)
+            write_snapshot_details(snapshot, out_dir=snapshot.snapshot_dir)
         except:
             pass
         raise
@@ -134,26 +142,29 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
         print('    ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err))
         raise
 
-    return link
-
+    return snapshot
 
 @enforce_types
-def archive_links(links: List[Link], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[str]=None) -> List[Link]:
-    if not links:
+def archive_snapshots(all_snapshots: Union[QuerySet, List[Model]], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> QuerySet:
+
+    all_snapshots = list(all_snapshots)
+    num_snapshots: int = len(all_snapshots)
+
+    if num_snapshots == 0:
         return []
 
-    log_archiving_started(len(links))
+    log_archiving_started(num_snapshots)
     idx: int = 0
-    link: Link = links[0]
     try:
-        for idx, link in enumerate(links):
-            archive_link(link, overwrite=overwrite, methods=methods, out_dir=link.link_dir)
+        for snapshot in all_snapshots:
+            idx += 1
+            archive_snapshot(snapshot, overwrite=overwrite, methods=methods, out_dir=Path(snapshot.snapshot_dir))
     except KeyboardInterrupt:
-        log_archiving_paused(len(links), idx, link.timestamp)
+        log_archiving_paused(num_snapshots, idx, snapshot.timestamp)
         raise SystemExit(0)
     except BaseException:
         print()
         raise
 
-    log_archiving_finished(len(links))
-    return links
+    log_archiving_finished(num_snapshots)
+    return all_snapshots

archivebox/extractors/archive_org.py

@@ -1,11 +1,13 @@
 __package__ = 'archivebox.extractors'
 
-import os
 
+from pathlib import Path
 from typing import Optional, List, Dict, Tuple
 from collections import defaultdict
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -13,6 +15,7 @@ from ..util import (
 )
 from ..config import (
     TIMEOUT,
+    CURL_ARGS,
     CHECK_SSL_VALIDITY,
     SAVE_ARCHIVE_DOT_ORG,
     CURL_BINARY,
@@ -22,33 +25,34 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
+# output = '{domain}/'
+
 
 @enforce_types
-def should_save_archive_dot_org(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_archive_dot_org(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if is_static_file(snapshot.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'archive.org.txt')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'archive.org.txt').exists():
         # if open(path, 'r').read().strip() != 'None':
         return False
 
     return SAVE_ARCHIVE_DOT_ORG
 
 @enforce_types
-def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_archive_dot_org(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """submit site to archive.org for archiving via their service, save returned archive url"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'archive.org.txt'
     archive_org_url = None
-    submit_url = 'https://web.archive.org/save/{}'.format(link.url)
+    submit_url = 'https://web.archive.org/save/{}'.format(snapshot.url)
     cmd = [
         CURL_BINARY,
-        '--silent',
-        '--location',
+        *CURL_ARGS,
         '--head',
-        '--compressed',
         '--max-time', str(timeout),
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
@@ -57,10 +61,10 @@ def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=T
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
         content_location, errors = parse_archive_dot_org_response(result.stdout)
         if content_location:
-            archive_org_url = 'https://web.archive.org{}'.format(content_location[0])
+            archive_org_url = content_location[0]
         elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]:
             archive_org_url = None
             # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url)))
@@ -80,14 +84,14 @@ def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=T
         # the URL in person, it will attempt to re-archive it, and it'll show the
         # nicer error message explaining why the url was rejected if it fails.
         archive_org_url = archive_org_url or submit_url
-        with open(os.path.join(out_dir, str(output)), 'w', encoding='utf-8') as f:
+        with open(str(out_dir / output), 'w', encoding='utf-8') as f:
             f.write(archive_org_url)
-        chmod_file('archive.org.txt', cwd=out_dir)
+        chmod_file('archive.org.txt', cwd=str(out_dir))
         output = archive_org_url
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/dom.py

@@ -1,10 +1,11 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file, atomic_write
 from ..util import (
     enforce_types,
@@ -19,41 +20,44 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
+# output = 'output.html'
+
 
 @enforce_types
-def should_save_dom(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_dom(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if is_static_file(snapshot.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'output.html')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'output.html').exists():
         return False
 
     return SAVE_DOM
-    
+
 @enforce_types
-def save_dom(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_dom(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """print HTML of site to file using chrome --dump-html"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'output.html'
-    output_path = os.path.join(out_dir, str(output))
+    output_path = out_dir / output
     cmd = [
         *chrome_args(TIMEOUT=timeout),
         '--dump-dom',
-        link.url
+        snapshot.url
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
         atomic_write(output_path, result.stdout)
 
         if result.returncode:
             hints = result.stderr.decode()
             raise ArchiveError('Failed to save DOM', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -62,7 +66,7 @@ def save_dom(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/favicon.py

@@ -1,16 +1,19 @@
 __package__ = 'archivebox.extractors'
 
-import os
+from pathlib import Path
 
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput
 from ..system import chmod_file, run
 from ..util import enforce_types, domain
 from ..config import (
     TIMEOUT,
     SAVE_FAVICON,
     CURL_BINARY,
+    CURL_ARGS,
     CURL_VERSION,
     CHECK_SSL_VALIDITY,
     CURL_USER_AGENT,
@@ -18,46 +21,46 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
+# output = 'favicon.ico'
+
+
 @enforce_types
-def should_save_favicon(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if os.path.exists(os.path.join(out_dir, 'favicon.ico')):
+def should_save_favicon(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[str]=None) -> bool:
+    out_dir = out_dir or snapshot.snapshot_dir
+    if not overwrite and (Path(out_dir) / 'favicon.ico').exists():
         return False
 
     return SAVE_FAVICON
-    
+
 @enforce_types
-def save_favicon(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_favicon(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download site favicon from google's favicon api"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or snapshot.snapshot_dir
     output: ArchiveOutput = 'favicon.ico'
     cmd = [
         CURL_BINARY,
-        '--silent',
+        *CURL_ARGS,
         '--max-time', str(timeout),
-        '--location',
-        '--compressed',
         '--output', str(output),
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
-        'https://www.google.com/s2/favicons?domain={}'.format(domain(link.url)),
+        'https://www.google.com/s2/favicons?domain={}'.format(domain(snapshot.url)),
     ]
-    status = 'pending'
+    status = 'failed'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        run(cmd, cwd=out_dir, timeout=timeout)
-        chmod_file(output, cwd=out_dir)
+        run(cmd, cwd=str(out_dir), timeout=timeout)
+        chmod_file(output, cwd=str(out_dir))
         status = 'succeeded'
     except Exception as err:
-        status = 'failed'
         output = err
     finally:
         timer.end()
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/git.py

@@ -1,10 +1,12 @@
 __package__ = 'archivebox.extractors'
 
-import os
 
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -18,6 +20,7 @@ from ..config import (
     TIMEOUT,
     SAVE_GIT,
     GIT_BINARY,
+    GIT_ARGS,
     GIT_VERSION,
     GIT_DOMAINS,
     CHECK_SSL_VALIDITY
@@ -25,19 +28,25 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
+# output = 'git/'
+# @contents = output.glob('*.*')
+# @exists = self.contents.exists()
+# @size => get_size(self.contents)
+# @num_files => len(self.contents)
 
 @enforce_types
-def should_save_git(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_git(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or snapshot.snapshot_dir
+    if is_static_file(snapshot.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'git')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'git').exists():
         return False
 
     is_clonable_url = (
-        (domain(link.url) in GIT_DOMAINS)
-        or (extension(link.url) == 'git')
+        (domain(snapshot.url) in GIT_DOMAINS)
+        or (extension(snapshot.url) == 'git')
     )
     if not is_clonable_url:
         return False
@@ -46,24 +55,24 @@ def should_save_git(link: Link, out_dir: Optional[str]=None) -> bool:
 
 
 @enforce_types
-def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_git(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download full site using git"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'git'
-    output_path = os.path.join(out_dir, str(output))
-    os.makedirs(output_path, exist_ok=True)
+    output_path = out_dir / output
+    output_path.mkdir(exist_ok=True)
     cmd = [
         GIT_BINARY,
         'clone',
-        '--recursive',
+        *GIT_ARGS,
         *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']),
-        without_query(without_fragment(link.url)),
+        without_query(without_fragment(snapshot.url)),
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=output_path, timeout=timeout + 1)
+        result = run(cmd, cwd=str(output_path), timeout=timeout + 1)
         if result.returncode == 128:
             # ignore failed re-download when the folder already exists
             pass
@@ -71,7 +80,7 @@ def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
             hints = 'Got git response code: {}.'.format(result.returncode)
             raise ArchiveError('Failed to save git clone', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
 
     except Exception as err:
         status = 'failed'
@@ -81,7 +90,7 @@ def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=GIT_VERSION,
         output=output,
         status=status,

archivebox/extractors/headers.py

@@ -0,0 +1,80 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+
+from typing import Optional
+
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput
+from ..system import atomic_write
+from ..util import (
+    enforce_types,
+    get_headers,
+)
+from ..config import (
+    TIMEOUT,
+    CURL_BINARY,
+    CURL_ARGS,
+    CURL_USER_AGENT,
+    CURL_VERSION,
+    CHECK_SSL_VALIDITY,
+    SAVE_HEADERS
+)
+from ..logging_util import TimedProgress
+
+
+# output = 'headers.json'
+
+@enforce_types
+def should_save_headers(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[str]=None) -> bool:
+    out_dir = out_dir or snapshot.snapshot_dir
+
+    if not SAVE_HEADERS:
+        return False
+    
+    if overwrite:
+        return True
+
+    output = Path(out_dir or snapshot.snapshot_dir) / 'headers.json'
+    return not output.exists()
+
+
+@enforce_types
+def save_headers(snapshot: Model, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """Download site headers"""
+
+    out_dir = Path(out_dir or snapshot.snapshot_dir)
+    output_folder = out_dir.absolute()
+    output: ArchiveOutput = 'headers.json'
+
+    status = 'succeeded'
+    timer = TimedProgress(timeout, prefix='      ')
+
+    cmd = [
+        CURL_BINARY,
+        *CURL_ARGS,
+        '--head',
+        '--max-time', str(timeout),
+        *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
+        *([] if CHECK_SSL_VALIDITY else ['--insecure']),
+        snapshot.url,
+    ]
+    try:
+        json_headers = get_headers(snapshot.url, timeout=timeout)
+        output_folder.mkdir(exist_ok=True)
+        atomic_write(str(output_folder / "headers.json"), json_headers)
+    except (Exception, OSError) as err:
+        status = 'failed'
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=CURL_VERSION,
+        output=output,
+        status=status,
+        **timer.stats,
+    )

archivebox/extractors/media.py

@@ -1,10 +1,11 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -13,7 +14,7 @@ from ..util import (
 from ..config import (
     MEDIA_TIMEOUT,
     SAVE_MEDIA,
-    SAVE_PLAYLISTS,
+    YOUTUBEDL_ARGS,
     YOUTUBEDL_BINARY,
     YOUTUBEDL_VERSION,
     CHECK_SSL_VALIDITY
@@ -21,53 +22,40 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
-@enforce_types
-def should_save_media(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+# output = 'media/'
 
-    if is_static_file(link.url):
+@enforce_types
+def should_save_media(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or snapshot.snapshot_dir
+
+    if is_static_file(snapshot.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'media')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'media').exists():
         return False
 
     return SAVE_MEDIA
 
 @enforce_types
-def save_media(link: Link, out_dir: Optional[str]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult:
+def save_media(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult:
     """Download playlists or individual video, audio, and subtitles using youtube-dl"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'media'
-    output_path = os.path.join(out_dir, str(output))
-    os.makedirs(output_path, exist_ok=True)
+    output_path = out_dir / output
+    output_path.mkdir(exist_ok=True)
     cmd = [
         YOUTUBEDL_BINARY,
-        '--write-description',
-        '--write-info-json',
-        '--write-annotations',
-        '--write-thumbnail',
-        '--no-call-home',
-        '--no-check-certificate',
-        '--user-agent',
-        '--all-subs',
-        '--extract-audio',
-        '--keep-video',
-        '--ignore-errors',
-        '--geo-bypass',
-        '--audio-format', 'mp3',
-        '--audio-quality', '320K',
-        '--embed-thumbnail',
-        '--add-metadata',
-        *(['--yes-playlist'] if SAVE_PLAYLISTS else []),
+        *YOUTUBEDL_ARGS,
         *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate']),
-        link.url,
+        snapshot.url,
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=output_path, timeout=timeout + 1)
-        chmod_file(output, cwd=out_dir)
+        result = run(cmd, cwd=str(output_path), timeout=timeout + 1)
+        chmod_file(output, cwd=str(out_dir))
         if result.returncode:
             if (b'ERROR: Unsupported URL' in result.stderr
                 or b'HTTP Error 404' in result.stderr
@@ -90,7 +78,7 @@ def save_media(link: Link, out_dir: Optional[str]=None, timeout: int=MEDIA_TIMEO
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=YOUTUBEDL_VERSION,
         output=output,
         status=status,

archivebox/extractors/mercury.py

@@ -0,0 +1,109 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+
+from subprocess import CompletedProcess
+from typing import Optional, List
+import json
+
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveError
+from ..system import run, atomic_write
+from ..util import (
+    enforce_types,
+    is_static_file,
+
+)
+from ..config import (
+    TIMEOUT,
+    SAVE_MERCURY,
+    DEPENDENCIES,
+    MERCURY_VERSION,
+)
+from ..logging_util import TimedProgress
+
+
+
+@enforce_types
+def ShellError(cmd: List[str], result: CompletedProcess, lines: int=20) -> ArchiveError:
+    # parse out last line of stderr
+    return ArchiveError(
+        f'Got {cmd[0]} response code: {result.returncode}).',
+        " ".join(
+            line.strip()
+            for line in (result.stdout + result.stderr).decode().rsplit('\n', lines)[-lines:]
+            if line.strip()
+        ),
+    )
+
+
+@enforce_types
+def should_save_mercury(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[str]=None) -> bool:
+    out_dir = out_dir or snapshot.snapshot_dir
+    if is_static_file(snapshot.url):
+        return False
+
+    output = Path(out_dir or snapshot.snapshot_dir) / 'mercury'
+    if not overwrite and output.exists():
+        return False
+
+    return SAVE_MERCURY and MERCURY_VERSION
+
+
+@enforce_types
+def save_mercury(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """download reader friendly version using @postlight/mercury-parser"""
+
+    out_dir = Path(out_dir or snapshot.snapshot_dir)
+    output_folder = out_dir.absolute() / "mercury"
+    output = str(output_folder)
+
+    status = 'succeeded'
+    timer = TimedProgress(timeout, prefix='      ')
+    try:
+        # Get plain text version of article
+        cmd = [
+            DEPENDENCIES['MERCURY_BINARY']['path'],
+            snapshot.url,
+            "--format=text"
+        ]
+        result = run(cmd, cwd=out_dir, timeout=timeout)
+        try:
+            article_text = json.loads(result.stdout)
+        except json.JSONDecodeError:
+            raise ShellError(cmd, result)
+        
+        # Get HTML version of article
+        cmd = [
+            DEPENDENCIES['MERCURY_BINARY']['path'],
+            snapshot.url
+        ]
+        result = run(cmd, cwd=out_dir, timeout=timeout)
+        try:
+            article_json = json.loads(result.stdout)
+        except json.JSONDecodeError:
+            raise ShellError(cmd, result)
+
+        output_folder.mkdir(exist_ok=True)
+        atomic_write(str(output_folder / "content.html"), article_json.pop("content"))
+        atomic_write(str(output_folder / "content.txt"), article_text["content"])
+        atomic_write(str(output_folder / "article.json"), article_json)
+
+        # Check for common failure cases
+        if (result.returncode > 0):
+            raise ShellError(cmd, result)
+    except (ArchiveError, Exception, OSError) as err:
+        status = 'failed'
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=MERCURY_VERSION,
+        output=output,
+        status=status,
+        **timer.stats,
+    )

archivebox/extractors/pdf.py

@@ -1,10 +1,11 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -18,40 +19,42 @@ from ..config import (
 )
 from ..logging_util import TimedProgress
 
+# output = 'output.pdf'
 
 @enforce_types
-def should_save_pdf(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_pdf(snapshot: Model, verwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if is_static_file(snapshot.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'output.pdf')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'output.pdf').exists():
         return False
 
     return SAVE_PDF
 
 
 @enforce_types
-def save_pdf(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_pdf(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """print PDF of site to file using chrome --headless"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'output.pdf'
     cmd = [
         *chrome_args(TIMEOUT=timeout),
         '--print-to-pdf',
-        link.url,
+        snapshot.url,
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
 
         if result.returncode:
             hints = (result.stderr or result.stdout).decode()
             raise ArchiveError('Failed to save PDF', hints)
         
-        chmod_file('output.pdf', cwd=out_dir)
+        chmod_file('output.pdf', cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -61,7 +64,7 @@ def save_pdf(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/readability.py

@@ -6,7 +6,9 @@ from tempfile import NamedTemporaryFile
 from typing import Optional
 import json
 
-from ..index.schema import Link, ArchiveResult, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveError
 from ..system import run, atomic_write
 from ..util import (
     enforce_types,
@@ -16,19 +18,21 @@ from ..util import (
 )
 from ..config import (
     TIMEOUT,
+    CURL_BINARY,
     SAVE_READABILITY,
-    READABILITY_BINARY,
+    DEPENDENCIES,
     READABILITY_VERSION,
 )
 from ..logging_util import TimedProgress
 
+
 @enforce_types
-def get_html(link: Link, path: Path) -> str:
+def get_html(snapshot: Model, path: Path) -> str:
     """
     Try to find wget, singlefile and then dom files.
     If none is found, download the url again.
     """
-    canonical = link.canonical_outputs()
+    canonical = snapshot.canonical_outputs()
     abs_path = path.absolute()
     sources = [canonical["singlefile_path"], canonical["wget_path"], canonical["dom_path"]]
     document = None
@@ -40,48 +44,61 @@ def get_html(link: Link, path: Path) -> str:
         except (FileNotFoundError, TypeError):
             continue
     if document is None:
-        return download_url(link.url)
+        return download_url(snapshot.url)
     else:
         return document
 
+
+# output = 'readability/'
+
 @enforce_types
-def should_save_readability(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_readability(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[str]=None) -> bool:
+    out_dir = out_dir or snapshot.link_dir
+    if is_static_file(snapshot.url):
         return False
 
-    output = Path(out_dir or link.link_dir) / 'readability'
-    return SAVE_READABILITY and READABILITY_VERSION and (not output.exists())
+    output = Path(out_dir or snapshot.snapshot_dir) / 'readability'
+    if not overwrite and output.exists():
+        return False
+
+    return SAVE_READABILITY and READABILITY_VERSION
 
 
 @enforce_types
-def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_readability(snapshot: Model, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download reader friendly version using @mozilla/readability"""
 
-    out_dir = Path(out_dir or link.link_dir)
+    out_dir = Path(out_dir or snapshot.snapshot_dir)
     output_folder = out_dir.absolute() / "readability"
     output = str(output_folder)
 
     # Readability Docs: https://github.com/mozilla/readability
 
     status = 'succeeded'
+    # fake command to show the user so they have something to try debugging if get_html fails
+    cmd = [
+        CURL_BINARY,
+        snapshot.url
+    ]
+    readability_content = None
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        document = get_html(link, out_dir)
+        document = get_html(snapshot, out_dir)
         temp_doc = NamedTemporaryFile(delete=False)
         temp_doc.write(document.encode("utf-8"))
         temp_doc.close()
 
         cmd = [
-            READABILITY_BINARY,
+            DEPENDENCIES['READABILITY_BINARY']['path'],
             temp_doc.name
         ]
 
         result = run(cmd, cwd=out_dir, timeout=timeout)
         result_json = json.loads(result.stdout)
         output_folder.mkdir(exist_ok=True)
+        readability_content = result_json.pop("textContent") 
         atomic_write(str(output_folder / "content.html"), result_json.pop("content"))
-        atomic_write(str(output_folder / "content.txt"), result_json.pop("textContent"))
+        atomic_write(str(output_folder / "content.txt"), readability_content)
         atomic_write(str(output_folder / "article.json"), result_json)
 
         # parse out number of files downloaded from last line of stderr:
@@ -99,7 +116,7 @@ def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEO
         # Check for common failure cases
         if (result.returncode > 0):
             raise ArchiveError('Readability was not able to archive the page', hints)
-    except Exception as err:
+    except (Exception, OSError) as err:
         status = 'failed'
         output = err
     finally:
@@ -111,5 +128,6 @@ def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEO
         cmd_version=READABILITY_VERSION,
         output=output,
         status=status,
-        **timer.stats,
+        index_texts= [readability_content] if readability_content else [],
+        **timer.stats,  
     )

archivebox/extractors/screenshot.py

@@ -1,10 +1,11 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -19,39 +20,41 @@ from ..config import (
 from ..logging_util import TimedProgress
 
 
+# output = 'screenshot.png'
 
 @enforce_types
-def should_save_screenshot(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_screenshot(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if is_static_file(snapshot.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'screenshot.png')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'screenshot.png').exists():
         return False
 
     return SAVE_SCREENSHOT
 
 @enforce_types
-def save_screenshot(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_screenshot(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """take screenshot of site using chrome --headless"""
     
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
     output: ArchiveOutput = 'screenshot.png'
     cmd = [
         *chrome_args(TIMEOUT=timeout),
         '--screenshot',
-        link.url,
+        snapshot.url,
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
 
         if result.returncode:
             hints = (result.stderr or result.stdout).decode()
             raise ArchiveError('Failed to save screenshot', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -60,7 +63,7 @@ def save_screenshot(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOU
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/singlefile.py

@@ -5,7 +5,9 @@ from pathlib import Path
 from typing import Optional
 import json
 
-from ..index.schema import Link, ArchiveResult, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
@@ -15,7 +17,7 @@ from ..util import (
 from ..config import (
     TIMEOUT,
     SAVE_SINGLEFILE,
-    SINGLEFILE_BINARY,
+    DEPENDENCIES,
     SINGLEFILE_VERSION,
     CHROME_BINARY,
 )
@@ -23,37 +25,42 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_singlefile(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if is_static_file(link.url):
+def should_save_singlefile(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if is_static_file(snapshot.url):
         return False
 
-    output = Path(out_dir or link.link_dir) / 'singlefile.html'
-    return SAVE_SINGLEFILE and SINGLEFILE_VERSION and (not output.exists())
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'singlefile.html').exists():
+        return False
+
+    return SAVE_SINGLEFILE
 
 
 @enforce_types
-def save_singlefile(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_singlefile(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download full site using single-file"""
 
-    out_dir = out_dir or link.link_dir
-    output = str(Path(out_dir).absolute() / "singlefile.html")
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    output = "singlefile.html"
 
     browser_args = chrome_args(TIMEOUT=0)
 
     # SingleFile CLI Docs: https://github.com/gildas-lormeau/SingleFile/tree/master/cli
+    browser_args = '--browser-args={}'.format(json.dumps(browser_args[1:]))
     cmd = [
-        SINGLEFILE_BINARY,
+        DEPENDENCIES['SINGLEFILE_BINARY']['path'],
         '--browser-executable-path={}'.format(CHROME_BINARY),
-        '--browser-args="{}"'.format(json.dumps(browser_args[1:])),
-        link.url,
+        browser_args,
+        snapshot.url,
         output
+        output,
     ]
 
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
 
         # parse out number of files downloaded from last line of stderr:
         #  "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
@@ -68,18 +75,20 @@ def save_singlefile(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOU
         )
 
         # Check for common failure cases
-        if (result.returncode > 0):
+        if (result.returncode > 0) or not (out_dir / output).is_file():
             raise ArchiveError('SingleFile was not able to archive the page', hints)
-        chmod_file(output)
-    except Exception as err:
+        chmod_file(output, cwd=str(out_dir))
+    except (Exception, OSError) as err:
         status = 'failed'
+        # TODO: Make this prettier. This is necessary to run the command (escape JSON internal quotes).
+        cmd[2] = browser_args.replace('"', "\\\"")
         output = err
     finally:
         timer.end()
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=SINGLEFILE_VERSION,
         output=output,
         status=status,

archivebox/extractors/title.py

@@ -1,12 +1,15 @@
 __package__ = 'archivebox.extractors'
 
 import re
+from html.parser import HTMLParser
+from pathlib import Path
 from typing import Optional
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..util import (
     enforce_types,
-    is_static_file,
     download_url,
     htmldecode,
 )
@@ -15,13 +18,14 @@ from ..config import (
     CHECK_SSL_VALIDITY,
     SAVE_TITLE,
     CURL_BINARY,
+    CURL_ARGS,
     CURL_VERSION,
     CURL_USER_AGENT,
-    setup_django,
 )
 from ..logging_util import TimedProgress
 
 
+
 HTML_TITLE_REGEX = re.compile(
     r'<title.*?>'                      # start matching text after <title> tag
     r'(.[^<>]+)',                      # get everything up to these symbols
@@ -29,46 +33,92 @@ HTML_TITLE_REGEX = re.compile(
 )
 
 
-@enforce_types
-def should_save_title(link: Link, out_dir: Optional[str]=None) -> bool:
-    # if link already has valid title, skip it
-    if link.title and not link.title.lower().startswith('http'):
-        return False
+class TitleParser(HTMLParser):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.title_tag = ""
+        self.title_og = ""
+        self.inside_title_tag = False
 
-    if is_static_file(link.url):
+    @property
+    def title(self):
+        return self.title_tag or self.title_og or None
+
+    def handle_starttag(self, tag, attrs):
+        if tag.lower() == "title" and not self.title_tag:
+            self.inside_title_tag = True
+        elif tag.lower() == "meta" and not self.title_og:
+            attrs = dict(attrs)
+            if attrs.get("property") == "og:title" and attrs.get("content"):
+                self.title_og = attrs.get("content")
+
+    def handle_data(self, data):
+        if self.inside_title_tag and data:
+            self.title_tag += data.strip()
+    
+    def handle_endtag(self, tag):
+        if tag.lower() == "title":
+            self.inside_title_tag = False
+
+
+# output = '{title}'
+
+@enforce_types
+def should_save_title(snapshot: Model, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    # if link already has valid title, skip it
+    if not overwrite and snapshot.title and not snapshot.title.lower().startswith('http'):
         return False
 
     return SAVE_TITLE
 
+def extract_title_with_regex(html):
+    match = re.search(HTML_TITLE_REGEX, html)
+    output = htmldecode(match.group(1).strip()) if match else None
+    return output
+
 @enforce_types
-def save_title(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_title(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """try to guess the page's title from its content"""
 
-    setup_django(out_dir=out_dir)
     from core.models import Snapshot
 
     output: ArchiveOutput = None
     cmd = [
         CURL_BINARY,
-        '--silent',
+        *CURL_ARGS,
         '--max-time', str(timeout),
-        '--location',
-        '--compressed',
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
-        link.url,
+        snapshot.url,
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        html = download_url(link.url, timeout=timeout)
-        match = re.search(HTML_TITLE_REGEX, html)
-        output = htmldecode(match.group(1).strip()) if match else None
-        if output:
-            if not link.title or len(output) >= len(link.title):
-                Snapshot.objects.filter(url=link.url, timestamp=link.timestamp).update(title=output)
+        html = download_url(snapshot.url, timeout=timeout)
+        try:
+            # try using relatively strict html parser first
+            parser = TitleParser()
+            parser.feed(html)
+            output = parser.title
+            if output is None:
+                raise
+        except Exception:
+            # fallback to regex that can handle broken/malformed html
+            output = extract_title_with_regex(html)
+        
+        # if title is better than the one in the db, update db with new title
+        if isinstance(output, str) and output:
+            if not snapshot.title or len(output) >= len(snapshot.title):
+                Snapshot.objects.filter(url=snapshot.url,
+                                        timestamp=snapshot.timestamp)\
+                                .update(title=output)
+                snapshot.title = output
         else:
-            raise ArchiveError('Unable to detect page title')
+            # if no content was returned, dont save a title (because it might be a temporary error)
+            if not html:
+                raise ArchiveError('Unable to detect page title')
+            # output = html[:128]       # use first bit of content as the title
+            output = link.base_url      # use the filename as the title (better UX)
     except Exception as err:
         status = 'failed'
         output = err
@@ -77,7 +127,7 @@ def save_title(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/wget.py

@@ -1,17 +1,17 @@
 __package__ = 'archivebox.extractors'
 
-import os
 import re
+from pathlib import Path
 
 from typing import Optional
 from datetime import datetime
 
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from django.db.models import Model
+
+from ..index.schema import ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
-    is_static_file,
-    without_scheme,
     without_fragment,
     without_query,
     path,
@@ -19,6 +19,7 @@ from ..util import (
     urldecode,
 )
 from ..config import (
+    WGET_ARGS,
     TIMEOUT,
     SAVE_WGET,
     SAVE_WARC,
@@ -35,55 +36,48 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_wget(link: Link, out_dir: Optional[str]=None) -> bool:
-    output_path = wget_output_path(link)
-    out_dir = out_dir or link.link_dir
-    if output_path and os.path.exists(os.path.join(out_dir, output_path)):
+def should_save_wget(snapshot: Model, overwrite: Optional[bool]=False, out_dir: Optional[Path]=None) -> bool:
+    output_path = wget_output_path(snapshot)
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
+    if not overwrite output_path and (out_dir / output_path).exists():
         return False
 
     return SAVE_WGET
 
 
 @enforce_types
-def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_wget(snapshot: Model, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download full site using wget"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or snapshot.snapshot_dir
     if SAVE_WARC:
-        warc_dir = os.path.join(out_dir, 'warc')
-        os.makedirs(warc_dir, exist_ok=True)
-        warc_path = os.path.join('warc', str(int(datetime.now().timestamp())))
+        warc_dir = out_dir / "warc"
+        warc_dir.mkdir(exist_ok=True)
+        warc_path = warc_dir / str(int(datetime.now().timestamp()))
 
     # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
     output: ArchiveOutput = None
     cmd = [
         WGET_BINARY,
         # '--server-response',  # print headers for better error parsing
-        '--no-verbose',
-        '--adjust-extension',
-        '--convert-links',
-        '--force-directories',
-        '--backup-converted',
-        '--span-hosts',
-        '--no-parent',
-        '-e', 'robots=off',
+        *WGET_ARGS,
         '--timeout={}'.format(timeout),
         *(['--restrict-file-names={}'.format(RESTRICT_FILE_NAMES)] if RESTRICT_FILE_NAMES else []),
-        *(['--warc-file={}'.format(warc_path)] if SAVE_WARC else []),
+        *(['--warc-file={}'.format(str(warc_path))] if SAVE_WARC else []),
         *(['--page-requisites'] if SAVE_WGET_REQUISITES else []),
         *(['--user-agent={}'.format(WGET_USER_AGENT)] if WGET_USER_AGENT else []),
-        *(['--load-cookies', COOKIES_FILE] if COOKIES_FILE else []),
+        *(['--load-cookies', str(COOKIES_FILE)] if COOKIES_FILE else []),
         *(['--compression=auto'] if WGET_AUTO_COMPRESSION else []),
         *([] if SAVE_WARC else ['--timestamping']),
         *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate', '--no-hsts']),
-        link.url,
+        snapshot.url,
     ]
 
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
-        output = wget_output_path(link)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
+        output = wget_output_path(snapshot)
 
         # parse out number of files downloaded from last line of stderr:
         #  "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
@@ -111,7 +105,12 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
             if b'ERROR 500: Internal Server Error' in result.stderr:
                 raise ArchiveError('500 Internal Server Error', hints)
             raise ArchiveError('Wget failed or got an error from the server', hints)
-        chmod_file(output, cwd=out_dir)
+        
+        if (out_dir / output).exists():
+            chmod_file(output, cwd=str(out_dir))
+        else:
+            print(f'          {out_dir}/{output}')
+            raise ArchiveError('Failed to find wget output after running', hints)
     except Exception as err:
         status = 'failed'
         output = err
@@ -120,7 +119,7 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=WGET_VERSION,
         output=output,
         status=status,
@@ -129,14 +128,12 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
 
 
 @enforce_types
-def wget_output_path(link: Link) -> Optional[str]:
+def wget_output_path(snapshot: Model) -> Optional[str]:
     """calculate the path to the wgetted .html file, since wget may
     adjust some paths to be different than the base_url path.
 
     See docs on wget --adjust-extension (-E)
     """
-    if is_static_file(link.url):
-        return without_scheme(without_fragment(link.url))
 
     # Wget downloads can save in a number of different ways depending on the url:
     #    https://example.com
@@ -169,27 +166,42 @@ def wget_output_path(link: Link) -> Optional[str]:
     # and there's no way to get the computed output path from wget
     # in order to avoid having to reverse-engineer how they calculate it,
     # we just look in the output folder read the filename wget used from the filesystem
-    full_path = without_fragment(without_query(path(link.url))).strip('/')
-    search_dir = os.path.join(
-        link.link_dir,
-        domain(link.url).replace(":", "+"),
-        urldecode(full_path),
-    )
+    full_path = without_fragment(without_query(path(snapshot.url))).strip('/')
+    search_dir = Path(snapshot.snapshot_dir) / domain(snapshot.url).replace(":", "+") / urldecode(full_path)
     for _ in range(4):
-        if os.path.exists(search_dir):
-            if os.path.isdir(search_dir):
+        if search_dir.exists():
+            if search_dir.is_dir():
                 html_files = [
-                    f for f in os.listdir(search_dir)
-                    if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", f, re.I | re.M)
+                    f for f in search_dir.iterdir()
+                    if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", str(f), re.I | re.M)
                 ]
                 if html_files:
-                    path_from_link_dir = search_dir.split(link.link_dir)[-1].strip('/')
-                    return os.path.join(path_from_link_dir, html_files[0])
+                    return str(html_files[0].relative_to(snapshot.snapshot_dir))
+
+                # sometimes wget'd URLs have no ext and return non-html
+                # e.g. /some/example/rss/all -> some RSS XML content)
+                #      /some/other/url.o4g   -> some binary unrecognized ext)
+                # test this with archivebox add --depth=1 https://getpocket.com/users/nikisweeting/feed/all
+                last_part_of_url = urldecode(full_path.rsplit('/', 1)[-1])
+                for file_present in search_dir.iterdir():
+                    if file_present == last_part_of_url:
+                        return str((search_dir / file_present).relative_to(snapshot.snapshot_dir))
 
         # Move up one directory level
-        search_dir = search_dir.rsplit('/', 1)[0]
+        search_dir = search_dir.parent
 
-        if search_dir == link.link_dir:
+        if search_dir == snapshot.snapshot_dir:
             break
 
+    # check for literally any file present that isnt an empty folder
+    domain_dir = Path(domain(snapshot.url).replace(":", "+"))
+    files_within = list((Path(snapshot.snapshot_dir) / domain_dir).glob('**/*.*'))
+    if files_within:
+        return str((domain_dir / files_within[-1]).relative_to(snapshot.snapshot_dir))
+    
+    # fallback to just the domain dir, dont try to introspect further inside it
+    search_dir = Path(snapshot.snapshot_dir) / domain(snapshot.url).replace(":", "+")
+    if search_dir.is_dir():
+        return domain(snapshot.url).replace(":", "+")
+
     return None

archivebox/index/__init__.py

@@ -1,17 +1,16 @@
 __package__ = 'archivebox.index'
 
-import re
 import os
 import shutil
-import json as pyjson
+from pathlib import Path
 
 from itertools import chain
 from typing import List, Tuple, Dict, Optional, Iterable
 from collections import OrderedDict
 from contextlib import contextmanager
 from urllib.parse import urlparse
+from django.db.models import QuerySet, Q, Model
 
-from ..system import atomic_write
 from ..util import (
     scheme,
     enforce_types,
@@ -21,11 +20,9 @@ from ..config import (
     ARCHIVE_DIR_NAME,
     SQL_INDEX_FILENAME,
     JSON_INDEX_FILENAME,
-    HTML_INDEX_FILENAME,
     OUTPUT_DIR,
     TIMEOUT,
     URL_BLACKLIST_PTN,
-    ANSI,
     stderr,
     OUTPUT_PERMISSIONS
 )
@@ -41,28 +38,29 @@ from ..logging_util import (
 
 from .schema import Link, ArchiveResult
 from .html import (
-    write_html_main_index,
-    write_html_link_details,
+    write_html_snapshot_details,
 )
 from .json import (
-    parse_json_main_index,
-    write_json_main_index,
-    parse_json_link_details, 
-    write_json_link_details,
+    pyjson,
+    load_json_snapshot,
+    write_json_snapshot_details,
 )
 from .sql import (
     write_sql_main_index,
-    parse_sql_main_index,
-    write_sql_link_details,
+    write_sql_snapshot_details,
 )
 
+from ..search import search_backend_enabled, query_search_index
+
 ### Link filtering and checking
 
 @enforce_types
-def merge_links(a: Link, b: Link) -> Link:
-    """deterministially merge two links, favoring longer field values over shorter,
+def merge_snapshots(a: Model, b: Model) -> Model:
+    """deterministially merge two snapshots, favoring longer field values over shorter,
     and "cleaner" values over worse ones.
+    TODO: Check if this makes sense with the new setup
     """
+    return a
     assert a.base_url == b.base_url, f'Cannot merge two links with different URLs ({a.base_url} != {b.base_url})'
 
     # longest url wins (because a fuzzy url will always be shorter)
@@ -113,76 +111,66 @@ def merge_links(a: Link, b: Link) -> Link:
             key=lambda result: result.start_ts,
         )))
 
-    return Link(
+    return Snapshot(
         url=url,
         timestamp=timestamp,
         title=title,
         tags=tags,
-        sources=sources,
-        history=history,
+        #sources=sources,
+        #history=history,
     )
 
 
 @enforce_types
-def validate_links(links: Iterable[Link]) -> List[Link]:
+def validate_snapshots(snapshots: List[Model]) -> List[Model]:
     timer = TimedProgress(TIMEOUT * 4)
     try:
-        links = archivable_links(links)  # remove chrome://, about:, mailto: etc.
-        links = sorted_links(links)      # deterministically sort the links based on timstamp, url
-        links = fix_duplicate_links(links)  # merge/dedupe duplicate timestamps & urls
+        snapshots = archivable_snapshots(snapshots)  # remove chrome://, about:, mailto: etc.
+        snapshots = sorted_snapshots(snapshots)      # deterministically sort the links based on timestamp, url
+        snapshots = fix_duplicate_snapshots(snapshots)  # merge/dedupe duplicate timestamps & urls
     finally:
         timer.end()
 
-    return list(links)
-
+    return list(snapshots)
 
 @enforce_types
-def archivable_links(links: Iterable[Link]) -> Iterable[Link]:
+def archivable_snapshots(snapshots: Iterable[Model]) -> Iterable[Model]:
     """remove chrome://, about:// or other schemed links that cant be archived"""
-    for link in links:
+    for snapshot in snapshots:
         try:
-            urlparse(link.url)
+            urlparse(snapshot.url)
         except ValueError:
             continue
-        if scheme(link.url) not in ('http', 'https', 'ftp'):
+        if scheme(snapshot.url) not in ('http', 'https', 'ftp'):
             continue
-        if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(link.url):
+        if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(snapshot.url):
             continue
 
-        yield link
+        yield snapshot
 
 
 @enforce_types
-def fix_duplicate_links(sorted_links: Iterable[Link]) -> Iterable[Link]:
+def fix_duplicate_snapshots(sorted_snapshots: Iterable[Model]) -> Iterable[Model]:
     """
     ensures that all non-duplicate links have monotonically increasing timestamps
+    TODO: Review how to do this with the new snapshots refactor
     """
-    # from core.models import Snapshot
-
+    return sorted_snapshots
     unique_urls: OrderedDict[str, Link] = OrderedDict()
 
-    for link in sorted_links:
-        if link.url in unique_urls:
+    for snapshot in sorted_snapshots:
+        if snapshot.url in unique_urls:
             # merge with any other links that share the same url
             link = merge_links(unique_urls[link.url], link)
         unique_urls[link.url] = link
 
-    # unique_timestamps: OrderedDict[str, Link] = OrderedDict()
-    # for link in unique_urls.values():
-    #     closest_non_duplicate_ts = lowest_uniq_timestamp(unique_timestamps, link.timestamp)
-    #     if closest_non_duplicate_ts != link.timestamp:
-    #         link = link.overwrite(timestamp=closest_non_duplicate_ts)
-    #         Snapshot.objects.filter(url=link.url).update(timestamp=link.timestamp)
-    #     unique_timestamps[link.timestamp] = link
-
-    # return unique_timestamps.values()
     return unique_urls.values()
 
 
 @enforce_types
-def sorted_links(links: Iterable[Link]) -> Iterable[Link]:
-    sort_func = lambda link: (link.timestamp.split('.', 1)[0], link.url)
-    return sorted(links, key=sort_func, reverse=True)
+def sorted_snapshots(snapshots: Iterable[Model]) -> Iterable[Model]:
+    sort_func = lambda snapshot: (snapshot.timestamp.split('.', 1)[0], snapshot.url)
+    return sorted(snapshots, key=sort_func, reverse=True)
 
 
 @enforce_types
@@ -223,7 +211,7 @@ def lowest_uniq_timestamp(used_timestamps: OrderedDict, timestamp: str) -> str:
 
 @contextmanager
 @enforce_types
-def timed_index_update(out_path: str):
+def timed_index_update(out_path: Path):
     log_indexing_started(out_path)
     timer = TimedProgress(TIMEOUT * 2, prefix='      ')
     try:
@@ -231,72 +219,47 @@ def timed_index_update(out_path: str):
     finally:
         timer.end()
 
-    assert os.path.exists(out_path), f'Failed to write index file: {out_path}'
+    assert out_path.exists(), f'Failed to write index file: {out_path}'
     log_indexing_finished(out_path)
 
 
 @enforce_types
-def write_main_index(links: List[Link], out_dir: str=OUTPUT_DIR, finished: bool=False) -> None:
-    """create index.html file for a given list of links"""
+def write_main_index(snapshots: List[Model], out_dir: Path=OUTPUT_DIR) -> None:
+    """Writes links to sqlite3 file for a given list of links"""
 
-    log_indexing_process_started(len(links))
+    log_indexing_process_started(len(snapshots))
 
     try:
-        with timed_index_update(os.path.join(out_dir, SQL_INDEX_FILENAME)):
-            write_sql_main_index(links, out_dir=out_dir)
-            os.chmod(os.path.join(out_dir, SQL_INDEX_FILENAME), int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
+        with timed_index_update(out_dir / SQL_INDEX_FILENAME):
+            write_sql_main_index(snapshots, out_dir=out_dir)
+            os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
 
-        with timed_index_update(os.path.join(out_dir, JSON_INDEX_FILENAME)):
-            write_json_main_index(links, out_dir=out_dir)
-
-        with timed_index_update(os.path.join(out_dir, HTML_INDEX_FILENAME)):
-            write_html_main_index(links, out_dir=out_dir, finished=finished)
     except (KeyboardInterrupt, SystemExit):
         stderr('[!] Warning: Still writing index to disk...', color='lightyellow')
-        stderr('    Run archivebox init to fix any inconsisntencies from an ungraceful exit.')
-        with timed_index_update(os.path.join(out_dir, SQL_INDEX_FILENAME)):
+        stderr('    Run archivebox init to fix any inconsistencies from an ungraceful exit.')
+        with timed_index_update(out_dir / SQL_INDEX_FILENAME):
             write_sql_main_index(links, out_dir=out_dir)
-            os.chmod(os.path.join(out_dir, SQL_INDEX_FILENAME), int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
+            os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
         raise SystemExit(0)
 
     log_indexing_process_finished()
 
-
 @enforce_types
-def load_main_index(out_dir: str=OUTPUT_DIR, warn: bool=True) -> List[Link]:
-    """parse and load existing index with any new links from import_path merged in"""
-
-    all_links: List[Link] = []
+def load_main_index(out_dir: Path=OUTPUT_DIR, warn: bool=True) -> List[Link]:
+    """
+    Returns all of the snapshots currently in index
+    """
+    from core.models import Snapshot
     try:
-        all_links = list(parse_json_main_index(out_dir))
-        links_from_sql = list(parse_sql_main_index(out_dir))
+        return Snapshot.objects.all()
 
-        json_urls = set(l.url for l in all_links)
-        sql_urls = set(l.url for l in links_from_sql)
-        only_in_sql = sql_urls - json_urls
-        only_in_json = json_urls - sql_urls
-
-        if only_in_json:
-            stderr('{red}[!] Warning: SQL index does not match JSON index!{reset}'.format(**ANSI))
-            if only_in_json:
-                stderr('    > Only in JSON: {}...'.format(', '.join(list(only_in_json)[:5])))
-            if only_in_sql:
-                stderr('    > Only in SQL: {}...'.format(', '.join(list(only_in_sql)[:5])))
-
-            stderr('    To repair the index and re-import any orphaned links run:')
-            stderr('        archivebox init')
-        if only_in_sql:
-            # meh, this harmless, it'll get overwritten on next run anyway
-            pass
     except (KeyboardInterrupt, SystemExit):
         raise SystemExit(0)
 
-    return all_links
-
 @enforce_types
-def load_main_index_meta(out_dir: str=OUTPUT_DIR) -> Optional[dict]:
-    index_path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(index_path):
+def load_main_index_meta(out_dir: Path=OUTPUT_DIR) -> Optional[dict]:
+    index_path = out_dir / JSON_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             meta_dict = pyjson.load(f)
             meta_dict.pop('links')
@@ -306,125 +269,80 @@ def load_main_index_meta(out_dir: str=OUTPUT_DIR) -> Optional[dict]:
 
 
 @enforce_types
-def parse_links_from_source(source_path: str, root_url: Optional[str]=None) -> Tuple[List[Link], List[Link]]:
+def parse_snapshots_from_source(source_path: str, root_url: Optional[str]=None) -> List[Model]:
 
-    from ..parsers import parse_links
+    from ..parsers import parse_snapshots
 
-    new_links: List[Link] = []
+    new_links: List[Model] = []
 
     # parse and validate the import file
-    raw_links, parser_name = parse_links(source_path, root_url=root_url)
-    new_links = validate_links(raw_links)
+    raw_snapshots, parser_name = parse_snapshots(source_path, root_url=root_url)
+    new_snapshots = validate_snapshots(raw_snapshots)
 
     if parser_name:
-        num_parsed = len(raw_links)
+        num_parsed = len(raw_snapshots)
         log_parsing_finished(num_parsed, parser_name)
 
-    return new_links
-
+    return new_snapshots
 
 @enforce_types
-def dedupe_links(existing_links: List[Link],
-                 new_links: List[Link]) -> Tuple[List[Link], List[Link]]:
+def filter_new_urls(snapshots: QuerySet,
+                 new_snapshots: List) -> List:
+    """
+    Returns a list of Snapshots corresponding to the urls that were not present in the index
+    """
+    urls = {snapshot.url: snapshot for snapshot in new_snapshots}
+    filtered_snapshots = snapshots.filter(url__in=urls.keys())
 
-    # merge existing links in out_dir and new links
-    all_links = validate_links(existing_links + new_links)
-    all_link_urls = {link.url for link in existing_links}
-
-    new_links = [
-        link for link in new_links
-        if link.url not in all_link_urls
-    ]
-
-    all_links_deduped = {link.url: link for link in all_links}
-    for i in range(len(new_links)):
-        if new_links[i].url in all_links_deduped.keys():
-            new_links[i] = all_links_deduped[new_links[i].url]
-    log_deduping_finished(len(new_links))
-
-    return all_links, new_links
-
-
-@enforce_types
-def patch_main_index(link: Link, out_dir: str=OUTPUT_DIR) -> None:
-    """hack to in-place update one row's info in the generated index files"""
-
-    # TODO: remove this ASAP, it's ugly, error-prone, and potentially dangerous
-
-    title = link.title or link.latest_outputs(status='succeeded')['title']
-    successful = link.num_outputs
-
-    # Patch JSON main index
-    json_file_links = parse_json_main_index(out_dir)
-    patched_links = []
-    for saved_link in json_file_links:
-        if saved_link.url == link.url:
-            patched_links.append(saved_link.overwrite(
-                title=title,
-                history=link.history,
-                updated=link.updated,
-            ))
-        else:
-            patched_links.append(saved_link)
+    for found_snapshot in filtered_snapshots:
+        urls.pop(found_snapshot.url)
     
-    write_json_main_index(patched_links, out_dir=out_dir)
-
-    # Patch HTML main index
-    html_path = os.path.join(out_dir, 'index.html')
-    with open(html_path, 'r') as f:
-        html = f.read().splitlines()
-
-    for idx, line in enumerate(html):
-        if title and ('<span data-title-for="{}"'.format(link.url) in line):
-            html[idx] = '<span>{}</span>'.format(title)
-        elif successful and ('<span data-number-for="{}"'.format(link.url) in line):
-            html[idx] = '<span>{}</span>'.format(successful)
-            break
-
-    atomic_write(html_path, '\n'.join(html))
+    log_deduping_finished(len(urls.keys()))
 
+    return list(urls.values())
 
 ### Link Details Index
 
 @enforce_types
-def write_link_details(link: Link, out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None:
-    out_dir = out_dir or link.link_dir
+def write_snapshot_details(snapshot: List[Model], out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None:
+    out_dir = out_dir or snapshot.snapshot_dir
 
-    write_json_link_details(link, out_dir=out_dir)
-    write_html_link_details(link, out_dir=out_dir)
+    write_json_snapshot_details(snapshot, out_dir=out_dir)
+    write_html_snapshot_details(snapshot, out_dir=out_dir)
     if not skip_sql_index:
-        write_sql_link_details(link)
+        write_sql_snapshot_details(snapshot)
 
 
 @enforce_types
-def load_link_details(link: Link, out_dir: Optional[str]=None) -> Link:
+def load_snapshot_details(snapshot: Model, out_dir: Optional[str]=None) -> Model:
     """check for an existing link archive in the given directory, 
        and load+merge it into the given link dict
     """
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(snapshot.snapshot_dir)
 
-    existing_link = parse_json_link_details(out_dir)
-    if existing_link:
-        return merge_links(existing_link, link)
+    existing_snapshot = load_json_snapshot(Path(out_dir))
+    if existing_snapshot:
+        return merge_snapshots(existing_snapshot, snapshot)
 
-    return link
+    return snapshot
 
 
 
 LINK_FILTERS = {
-    'exact': lambda link, pattern: (link.url == pattern) or (link.base_url == pattern),
-    'substring': lambda link, pattern: pattern in link.url,
-    'regex': lambda link, pattern: bool(re.match(pattern, link.url)),
-    'domain': lambda link, pattern: link.domain == pattern,
+    'exact': lambda pattern: Q(url=pattern),
+    'substring': lambda pattern: Q(url__icontains=pattern),
+    'regex': lambda pattern: Q(url__iregex=pattern),
+    'domain': lambda pattern: Q(url__istartswith=f"http://{pattern}") | Q(url__istartswith=f"https://{pattern}") | Q(url__istartswith=f"ftp://{pattern}"),
+    'tag': lambda pattern: Q(tags__name=pattern),
 }
 
 @enforce_types
-def link_matches_filter(link: Link, filter_patterns: List[str], filter_type: str='exact') -> bool:
+def q_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet:
+    q_filter = Q()
     for pattern in filter_patterns:
         try:
-            if LINK_FILTERS[filter_type](link, pattern):
-                return True
-        except Exception:
+            q_filter = q_filter | LINK_FILTERS[filter_type](pattern)
+        except KeyError:
             stderr()
             stderr(
                 f'[X] Got invalid pattern for --filter-type={filter_type}:',
@@ -432,163 +350,177 @@ def link_matches_filter(link: Link, filter_patterns: List[str], filter_type: str
             )
             stderr(f'    {pattern}')
             raise SystemExit(2)
+    return snapshots.filter(q_filter)
 
-    return False
+def search_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='search') -> QuerySet:
+    if not search_backend_enabled():
+        stderr()
+        stderr(
+                '[X] The search backend is not enabled, set config.USE_SEARCHING_BACKEND = True',
+                color='red',
+            )
+        raise SystemExit(2)
+    from core.models import Snapshot
+
+    qsearch = Snapshot.objects.none()
+    for pattern in filter_patterns:
+        try:
+            qsearch |= query_search_index(pattern)
+        except:
+            raise SystemExit(2)
+    
+    return snapshots & qsearch
+
+@enforce_types
+def snapshot_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet:
+    if filter_type != 'search':
+        return q_filter(snapshots, filter_patterns, filter_type)
+    else:
+        return search_filter(snapshots, filter_patterns, filter_type)
 
 
-def get_indexed_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_indexed_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """indexed links without checking archive status or data directory validity"""
-    return {
-        link.link_dir: link
-        for link in links
-    }
+    return {snapshot.snapshot_dir: snapshot for snapshot in snapshots}
 
-def get_archived_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_archived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """indexed links that are archived with a valid data directory"""
-    return {
-        link.link_dir: link
-        for link in filter(is_archived, links)
-    }
+    return {snapshot.snapshot_dir: snapshot for snapshot in filter(is_archived, snapshots)}
 
-def get_unarchived_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_unarchived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """indexed links that are unarchived with no data directory or an empty data directory"""
-    return {
-        link.link_dir: link
-        for link in filter(is_unarchived, links)
-    }
+    return {snapshot.snapshot_dir: snapshot for snapshot in filter(is_unarchived, snapshots)}
 
-def get_present_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_present_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that actually exist in the archive/ folder"""
+    from core.models import Snapshot
+
     all_folders = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
-            link = None
+    for entry in (out_dir / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
+            snapshot = None
             try:
-                link = parse_json_link_details(entry.path)
+                snapshot = load_json_snapshot(Path(entry.path))
             except Exception:
                 pass
 
-            all_folders[entry.path] = link
+            all_folders[entry.name] = snapshot
 
     return all_folders
 
-def get_valid_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_valid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs with a valid index matched to the main index and archived content"""
-    return {
-        link.link_dir: link
-        for link in filter(is_valid, links)
-    }
+    return {snapshot.snapshot_dir: snapshot for snapshot in filter(is_valid, snapshots)}
 
-def get_invalid_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_invalid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized"""
-    duplicate = get_duplicate_folders(links, out_dir=OUTPUT_DIR)
-    orphaned = get_orphaned_folders(links, out_dir=OUTPUT_DIR)
-    corrupted = get_corrupted_folders(links, out_dir=OUTPUT_DIR)
-    unrecognized = get_unrecognized_folders(links, out_dir=OUTPUT_DIR)
+    duplicate = get_duplicate_folders(snapshots, out_dir=OUTPUT_DIR)
+    orphaned = get_orphaned_folders(snapshots, out_dir=OUTPUT_DIR)
+    corrupted = get_corrupted_folders(snapshots, out_dir=OUTPUT_DIR)
+    unrecognized = get_unrecognized_folders(snapshots, out_dir=OUTPUT_DIR)
     return {**duplicate, **orphaned, **corrupted, **unrecognized}
 
 
-def get_duplicate_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_duplicate_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that conflict with other directories that have the same link URL or timestamp"""
-    links = list(links)
-    by_url = {link.url: 0 for link in links}
-    by_timestamp = {link.timestamp: 0 for link in links}
-
+    by_url = {}
+    by_timestamp = {}
     duplicate_folders = {}
 
-    indexed_folders = {link.link_dir for link in links}
     data_folders = (
-        entry.path
-        for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME))
-        if entry.is_dir(follow_symlinks=True) and entry.path not in indexed_folders
+        str(entry)
+        for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir()
+            if entry.is_dir() and not snapshots.filter(timestamp=entry.name).exists()
     )
 
-    for path in chain(sorted(indexed_folders), sorted(data_folders)):
-        link = None
+    for path in chain(snapshots.iterator(), data_folders):
+        snapshot = None
+        if type(path) is not str:
+            path = path.snapshot_dir
+
         try:
-            link = parse_json_link_details(path)
+            snapshot = load_json_snapshot(Path(path))
         except Exception:
             pass
 
-        if link:
-            # link folder has same timestamp as different link folder
-            by_timestamp[link.timestamp] = by_timestamp.get(link.timestamp, 0) + 1
-            if by_timestamp[link.timestamp] > 1:
-                duplicate_folders[path] = link
+        if snapshot:
+            # snapshot folder has same timestamp as different link folder
+            by_timestamp[snapshot.timestamp] = by_timestamp.get(snapshot.timestamp, 0) + 1
+            if by_timestamp[snapshot.timestamp] > 1:
+                duplicate_folders[path] = snapshot
 
             # link folder has same url as different link folder
-            by_url[link.url] = by_url.get(link.url, 0) + 1
-            if by_url[link.url] > 1:
-                duplicate_folders[path] = link
-
+            by_url[snapshot.url] = by_url.get(snapshot.url, 0) + 1
+            if by_url[snapshot.url] > 1:
+                duplicate_folders[path] = snapshot
     return duplicate_folders
 
-def get_orphaned_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_orphaned_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that contain a valid index but aren't listed in the main index"""
-    links = list(links)
-    indexed_folders = {link.link_dir: link for link in links}
     orphaned_folders = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
-            link = None
+    for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
+            snapshot = None
             try:
-                link = parse_json_link_details(entry.path)
+                snapshot = load_json_snapshot(entry)
             except Exception:
                 pass
 
-            if link and entry.path not in indexed_folders:
+            if snapshot and not snapshots.filter(timestamp=entry.name).exists():
                 # folder is a valid link data dir with index details, but it's not in the main index
-                orphaned_folders[entry.path] = link
+                orphaned_folders[str(entry)] = snapshot
 
     return orphaned_folders
 
-def get_corrupted_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_corrupted_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that don't contain a valid index and aren't listed in the main index"""
-    return {
-        link.link_dir: link
-        for link in filter(is_corrupt, links)
-    }
+    corrupted = {}
+    for snapshot in snapshots.iterator():
+        if is_corrupt(snapshot):
+            corrupted[snapshot.snapshot_dir] = snapshot
+    return corrupted
 
-def get_unrecognized_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_unrecognized_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     """dirs that don't contain recognizable archive data and aren't listed in the main index"""
-    by_timestamp = {link.timestamp: 0 for link in links}
-    unrecognized_folders: Dict[str, Optional[Link]] = {}
+    unrecognized_folders: Dict[str, Optional[Model]] = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
-            index_exists = os.path.exists(os.path.join(entry.path, 'index.json'))
-            link = None
+    for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
+            index_exists = (entry / "index.json").exists()
+            snapshot = None
             try:
-                link = parse_json_link_details(entry.path)
+                snapshot = load_json_snapshot(entry)
             except KeyError:
                 # Try to fix index
                 if index_exists:
-                    try:
+                    pass
+                    # TODO: Implement the `guess` bit for snapshots
+                    # try:
                         # Last attempt to repair the detail index
-                        link_guessed = parse_json_link_details(entry.path, guess=True)
-                        write_json_link_details(link_guessed, out_dir=entry.path)
-                        link = parse_json_link_details(entry.path)
-                    except Exception:
-                        pass
+                        # link_guessed = parse_json_snapshot_details(str(entry), guess=True)
+                        # write_json_snapshot_details(link_guessed, out_dir=str(entry))
+                        # link = parse_json_link_details(str(entry))
+                    # except Exception:
+                    #     pass
 
-            if index_exists and link is None:
+            if index_exists and snapshot is None:
                 # index exists but it's corrupted or unparseable
-                unrecognized_folders[entry.path] = link
+                unrecognized_folders[str(entry)] = snapshot
             
             elif not index_exists:
                 # link details index doesn't exist and the folder isn't in the main index
-                timestamp = entry.path.rsplit('/', 1)[-1]
-                if timestamp not in by_timestamp:
-                    unrecognized_folders[entry.path] = link
+                timestamp = entry.name
+                if not snapshots.filter(timestamp=timestamp).exists():
+                    unrecognized_folders[str(entry)] = snapshot
 
     return unrecognized_folders
 
 
-def is_valid(link: Link) -> bool:
-    dir_exists = os.path.exists(link.link_dir)
-    index_exists = os.path.exists(os.path.join(link.link_dir, 'index.json'))
+def is_valid(snapshot: Model) -> bool:
+    dir_exists = Path(snapshot.snapshot_dir).exists()
+    index_exists = (Path(snapshot.snapshot_dir) / "index.json").exists()
     if not dir_exists:
         # unarchived links are not included in the valid list
         return False
@@ -596,54 +528,55 @@ def is_valid(link: Link) -> bool:
         return False
     if dir_exists and index_exists:
         try:
-            parsed_link = parse_json_link_details(link.link_dir, guess=True)
-            return link.url == parsed_link.url
+            # TODO: review if the `guess` was necessary here
+            parsed_snapshot = load_json_snapshot(snapshot.snapshot_dir)
+            return snapshot.url == parsed_snapshot.url
         except Exception:
             pass
     return False
 
-def is_corrupt(link: Link) -> bool:
-    if not os.path.exists(link.link_dir):
+def is_corrupt(snapshot: Model) -> bool:
+    if not Path(snapshot.snapshot_dir).exists():
         # unarchived links are not considered corrupt
         return False
 
-    if is_valid(link):
+    if is_valid(snapshot):
         return False
 
     return True
 
-def is_archived(link: Link) -> bool:
-    return is_valid(link) and link.is_archived
+def is_archived(snapshot: Model) -> bool:
+    return is_valid(snapshot) and snapshot.is_archived
     
-def is_unarchived(link: Link) -> bool:
-    if not os.path.exists(link.link_dir):
+def is_unarchived(snapshot: Model) -> bool:
+    if not Path(snapshot.snapshot_dir).exists():
         return True
-    return not link.is_archived
+    return not snapshot.is_archived
 
 
-def fix_invalid_folder_locations(out_dir: str=OUTPUT_DIR) -> Tuple[List[str], List[str]]:
+def fix_invalid_folder_locations(out_dir: Path=OUTPUT_DIR) -> Tuple[List[str], List[str]]:
     fixed = []
     cant_fix = []
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
+    for entry in os.scandir(out_dir / ARCHIVE_DIR_NAME):
         if entry.is_dir(follow_symlinks=True):
-            if os.path.exists(os.path.join(entry.path, 'index.json')):
+            if (Path(entry.path) / 'index.json').exists():
                 try:
-                    link = parse_json_link_details(entry.path)
+                    snapshot = load_json_snapshot(Path(entry.path))
                 except KeyError:
-                    link = None
-                if not link:
+                    snapshot = None
+                if not snapshot:
                     continue
 
-                if not entry.path.endswith(f'/{link.timestamp}'):
-                    dest = os.path.join(out_dir, ARCHIVE_DIR_NAME, link.timestamp)
-                    if os.path.exists(dest):
+                if not entry.path.endswith(f'/{snapshot.timestamp}'):
+                    dest = out_dir / ARCHIVE_DIR_NAME / snapshot.timestamp
+                    if dest.exists():
                         cant_fix.append(entry.path)
                     else:
                         shutil.move(entry.path, dest)
                         fixed.append(dest)
                         timestamp = entry.path.rsplit('/', 1)[-1]
-                        assert link.link_dir == entry.path
-                        assert link.timestamp == timestamp
-                        write_json_link_details(link, out_dir=entry.path)
+                        assert snapshot.snapshot_dir == entry.path
+                        assert snapshot.timestamp == timestamp
+                        write_json_snapshot_details(snapshot, out_dir=entry.path)
 
     return fixed, cant_fix

archivebox/index/csv.py

@@ -2,12 +2,14 @@ __package__ = 'archivebox.index'
 
 from typing import List, Optional, Any
 
+from django.db.models import Model
+
 from ..util import enforce_types
 from .schema import Link
 
 
 @enforce_types
-def links_to_csv(links: List[Link],
+def snapshots_to_csv(snapshots: List[Model],
                  cols: Optional[List[str]]=None,
                  header: bool=True,
                  separator: str=',',
@@ -20,8 +22,8 @@ def links_to_csv(links: List[Link],
         header_str = separator.join(col.ljust(ljust) for col in cols)
 
     row_strs = (
-        link.to_csv(cols=cols, ljust=ljust, separator=separator)
-        for link in links
+        snapshot.as_csv(cols=cols, ljust=ljust, separator=separator)
+        for snapshot in snapshots
     )
 
     return '\n'.join((header_str, *row_strs))

archivebox/index/html.py

@@ -1,13 +1,15 @@
 __package__ = 'archivebox.index'
 
-import os
-
-from string import Template
 from datetime import datetime
 from typing import List, Optional, Iterator, Mapping
+from pathlib import Path
+
+from django.db.models import Model
+from django.utils.html import format_html, mark_safe
+from collections import defaultdict
 
 from .schema import Link
-from ..system import atomic_write, copy_and_overwrite
+from ..system import atomic_write
 from ..logging_util import printable_filesize
 from ..util import (
     enforce_types,
@@ -18,32 +20,27 @@ from ..util import (
 )
 from ..config import (
     OUTPUT_DIR,
-    TEMPLATES_DIR,
     VERSION,
     GIT_SHA,
     FOOTER_INFO,
-    ARCHIVE_DIR_NAME,
     HTML_INDEX_FILENAME,
-    STATIC_DIR_NAME,
-    ROBOTS_TXT_FILENAME,
-    FAVICON_FILENAME,
+    SAVE_ARCHIVE_DOT_ORG,
 )
 
-join = lambda *paths: os.path.join(*paths)
-MAIN_INDEX_TEMPLATE = join(TEMPLATES_DIR, 'main_index.html')
-MAIN_INDEX_ROW_TEMPLATE = join(TEMPLATES_DIR, 'main_index_row.html')
-LINK_DETAILS_TEMPLATE = join(TEMPLATES_DIR, 'link_details.html')
+MAIN_INDEX_TEMPLATE = 'static_index.html'
+MINIMAL_INDEX_TEMPLATE = 'minimal_index.html'
+LINK_DETAILS_TEMPLATE = 'snapshot.html'
 TITLE_LOADING_MSG = 'Not yet archived...'
 
 
 ### Main Links Index
 
 @enforce_types
-def parse_html_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[str]:
+def parse_html_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[str]:
     """parse an archive index html file and return the list of urls"""
 
-    index_path = join(out_dir, HTML_INDEX_FILENAME)
-    if os.path.exists(index_path):
+    index_path = Path(out_dir) / HTML_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             for line in f:
                 if 'class="link-url"' in line:
@@ -51,108 +48,137 @@ def parse_html_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[str]:
     return ()
 
 @enforce_types
-def write_html_main_index(links: List[Link], out_dir: str=OUTPUT_DIR, finished: bool=False) -> None:
-    """write the html link index to a given path"""
-
-    copy_and_overwrite(join(TEMPLATES_DIR, FAVICON_FILENAME), join(out_dir, FAVICON_FILENAME))
-    copy_and_overwrite(join(TEMPLATES_DIR, ROBOTS_TXT_FILENAME), join(out_dir, ROBOTS_TXT_FILENAME))
-    copy_and_overwrite(join(TEMPLATES_DIR, STATIC_DIR_NAME), join(out_dir, STATIC_DIR_NAME))
-    
-    rendered_html = main_index_template(links, finished=finished)
-    atomic_write(join(out_dir, HTML_INDEX_FILENAME), rendered_html)
-
+def generate_index_from_snapshots(snapshots: List[Model], with_headers: bool):
+    if with_headers:
+        output = main_index_template(snapshots)
+    else:
+        output = main_index_template(snapshots, template=MINIMAL_INDEX_TEMPLATE)
+    return output
 
 @enforce_types
-def main_index_template(links: List[Link], finished: bool=True) -> str:
+def main_index_template(snapshots: List[Model], template: str=MAIN_INDEX_TEMPLATE) -> str:
     """render the template for the entire main index"""
 
-    return render_legacy_template(MAIN_INDEX_TEMPLATE, {
+    return render_django_template(template, {
         'version': VERSION,
         'git_sha': GIT_SHA,
-        'num_links': str(len(links)),
-        'status': 'finished' if finished else 'running',
+        'num_snapshots': str(len(snapshots)),
         'date_updated': datetime.now().strftime('%Y-%m-%d'),
         'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'),
-        'rows': '\n'.join(
-            main_index_row_template(link)
-            for link in links
-        ),
-        'footer_info': FOOTER_INFO,
-    })
-
-
-@enforce_types
-def main_index_row_template(link: Link) -> str:
-    """render the template for an individual link row of the main index"""
-
-    from ..extractors.wget import wget_output_path
-
-    return render_legacy_template(MAIN_INDEX_ROW_TEMPLATE, {
-        **link._asdict(extended=True),
-        
-        # before pages are finished archiving, show loading msg instead of title
-        'title': htmlencode(
-            link.title
-            or (link.base_url if link.is_archived else TITLE_LOADING_MSG)
-        ),
-
-        # before pages are finished archiving, show fallback loading favicon
-        'favicon_url': (
-            join(ARCHIVE_DIR_NAME, link.timestamp, 'favicon.ico')
-            # if link['is_archived'] else 'data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs='
-        ),
-
-        # before pages are finished archiving, show the details page instead
-        'wget_url': urlencode(wget_output_path(link) or 'index.html'),
-        
-        # replace commas in tags with spaces, or file extension if it's static
-        'tags': (link.tags or '') + (' {}'.format(link.extension) if link.is_static else ''),
+        'snapshots': snapshots,
+        'FOOTER_INFO': FOOTER_INFO,
     })
 
 
 ### Link Details Index
 
 @enforce_types
-def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None:
-    out_dir = out_dir or link.link_dir
+def write_html_snapshot_details(snapshot: Model, out_dir: Optional[str]=None) -> None:
+    out_dir = out_dir or snapshot.snapshot_dir
 
-    rendered_html = link_details_template(link)
-    atomic_write(join(out_dir, HTML_INDEX_FILENAME), rendered_html)
+    rendered_html = snapshot_details_template(snapshot)
+    atomic_write(str(Path(out_dir) / HTML_INDEX_FILENAME), rendered_html)
 
 
 @enforce_types
-def link_details_template(link: Link) -> str:
+def snapshot_details_template(snapshot: Model) -> str:
 
     from ..extractors.wget import wget_output_path
 
-    link_info = link._asdict(extended=True)
-
-    return render_legacy_template(LINK_DETAILS_TEMPLATE, {
-        **link_info,
-        **link_info['canonical'],
+    return render_django_template(LINK_DETAILS_TEMPLATE, {
+        **snapshot.as_json(),
+        **snapshot.canonical_outputs(),
         'title': htmlencode(
-            link.title
-            or (link.base_url if link.is_archived else TITLE_LOADING_MSG)
+            snapshot.title
+            or (snapshot.base_url if snapshot.is_archived else TITLE_LOADING_MSG)
         ),
-        'url_str': htmlencode(urldecode(link.base_url)),
+        'url_str': htmlencode(urldecode(snapshot.base_url)),
         'archive_url': urlencode(
-            wget_output_path(link)
-            or (link.domain if link.is_archived else '')
+            wget_output_path(snapshot)
+            or (snapshot.domain if snapshot.is_archived else '')
         ) or 'about:blank',
-        'extension': link.extension or 'html',
-        'tags': link.tags or 'untagged',
-        'size': printable_filesize(link.archive_size) if link.archive_size else 'pending',
-        'status': 'archived' if link.is_archived else 'not yet archived',
-        'status_color': 'success' if link.is_archived else 'danger',
-        'oldest_archive_date': ts_to_date(link.oldest_archive_date),
+        'extension': snapshot.extension or 'html',
+        'tags': snapshot.tags_str() or 'untagged',
+        'size': printable_filesize(snapshot.archive_size) if snapshot.archive_size else 'pending',
+        'status': 'archived' if snapshot.is_archived else 'not yet archived',
+        'status_color': 'success' if snapshot.is_archived else 'danger',
+        'oldest_archive_date': ts_to_date(snapshot.oldest_archive_date),
+        'SAVE_ARCHIVE_DOT_ORG': SAVE_ARCHIVE_DOT_ORG,
     })
 
-
 @enforce_types
-def render_legacy_template(template_path: str, context: Mapping[str, str]) -> str:
+def render_django_template(template: str, context: Mapping[str, str]) -> str:
     """render a given html template string with the given template content"""
+    from django.template.loader import render_to_string
 
-    # will be replaced by django templates in the future
-    with open(template_path, 'r', encoding='utf-8') as template:
-        template_str = template.read()
-    return Template(template_str).substitute(**context)
+    return render_to_string(template, context)
+
+
+def snapshot_icons(snapshot) -> str:
+    from core.models import EXTRACTORS
+
+    # start = datetime.now()
+
+    archive_results = snapshot.archiveresult_set.filter(status="succeeded")
+    path = snapshot.archive_path
+    canon = snapshot.canonical_outputs()
+    output = ""
+    output_template = '<a href="/{}/{}" class="exists-{}" title="{}">{}</a> &nbsp;'
+    icons = {
+        "singlefile": "❶",
+        "wget": "🆆",
+        "dom": "🅷",
+        "pdf": "📄",
+        "screenshot": "💻",
+        "media": "📼",
+        "git": "🅶",
+        "archive_org": "🏛",
+        "readability": "🆁",
+        "mercury": "🅼",
+        "warc": "📦"
+    }
+    exclude = ["favicon", "title", "headers", "archive_org"]
+    # Missing specific entry for WARC
+
+    extractor_outputs = defaultdict(lambda: None)
+    for extractor, _ in EXTRACTORS:
+        for result in archive_results:
+            if result.extractor == extractor and result:
+                extractor_outputs[extractor] = result
+
+    for extractor, _ in EXTRACTORS:
+        if extractor not in exclude:
+            existing = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # Check filesystsem to see if anything is actually present (too slow, needs optimization/caching)
+            # if existing:
+            #     existing = (Path(path) / existing)
+            #     if existing.is_file():
+            #         existing = True
+            #     elif existing.is_dir():
+            #         existing = any(existing.glob('*.*'))
+            output += format_html(output_template, path, canon[f"{extractor}_path"], str(bool(existing)),
+                                         extractor, icons.get(extractor, "?"))
+        if extractor == "wget":
+            # warc isn't technically it's own extractor, so we have to add it after wget
+            
+            # get from db (faster but less thurthful)
+            exists = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # get from filesystem (slower but more accurate)
+            # exists = list((Path(path) / canon["warc_path"]).glob("*.warc.gz"))
+            output += format_html(output_template, 'warc/', canon["warc_path"], str(bool(exists)), "warc", icons.get("warc", "?"))
+
+        if extractor == "archive_org":
+            # The check for archive_org is different, so it has to be handled separately
+
+            # get from db (faster)
+            exists = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # get from filesystem (slower)
+            # target_path = Path(path) / "archive.org.txt"
+            # exists = target_path.exists()
+            output += '<a href="{}" class="exists-{}" title="{}">{}</a> '.format(canon["archive_org_path"], str(exists),
+                                                                                        "archive_org", icons.get("archive_org", "?"))
+
+    result = format_html('<span class="files-icons" style="font-size: 1.1em; opacity: 0.8; min-width: 240px; display: inline-block">{}<span>', mark_safe(output))
+    # end = datetime.now()
+    # print(((end - start).total_seconds()*1000) // 1, 'ms')
+    return result

archivebox/index/json.py

@@ -6,9 +6,10 @@ import json as pyjson
 from pathlib import Path
 
 from datetime import datetime
-from typing import List, Optional, Iterator, Any
+from typing import List, Optional, Iterator, Any, Union
+from django.db.models import Model
 
-from .schema import Link, ArchiveResult
+from .schema import Link
 from ..system import atomic_write
 from ..util import enforce_types
 from ..config import (
@@ -32,21 +33,35 @@ MAIN_INDEX_HEADER = {
         'version': VERSION,
         'git_sha': GIT_SHA,
         'website': 'https://ArchiveBox.io',
-        'docs': 'https://github.com/pirate/ArchiveBox/wiki',
-        'source': 'https://github.com/pirate/ArchiveBox',
-        'issues': 'https://github.com/pirate/ArchiveBox/issues',
+        'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki',
+        'source': 'https://github.com/ArchiveBox/ArchiveBox',
+        'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues',
         'dependencies': DEPENDENCIES,
     },
 }
 
-### Main Links Index
+@enforce_types
+def generate_json_index_from_snapshots(snapshots: List[Model], with_headers: bool):
+    snapshots_json = [snapshot.as_json() for snapshot in snapshots]
+    if with_headers:
+        output = {
+            **MAIN_INDEX_HEADER,
+            'num_links': len(snapshots),
+            'updated': datetime.now(),
+            'last_run_cmd': sys.argv,
+            'links': snapshots_json,
+        }
+    else:
+        output = snapshots_json
+    return to_json(output, indent=4, sort_keys=True)
+
 
 @enforce_types
-def parse_json_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
+def parse_json_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]:
     """parse an archive index json file and return the list of links"""
 
-    index_path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(index_path):
+    index_path = Path(out_dir) / JSON_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             links = pyjson.load(f)['links']
             for link_json in links:
@@ -65,68 +80,49 @@ def parse_json_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
                             continue
     return ()
 
-@enforce_types
-def write_json_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    """write the json link index to a given path"""
-
-    assert isinstance(links, List), 'Links must be a list, not a generator.'
-    assert not links or isinstance(links[0].history, dict)
-    assert not links or isinstance(links[0].sources, list)
-
-    if links and links[0].history.get('title'):
-        assert isinstance(links[0].history['title'][0], ArchiveResult)
-
-    if links and links[0].sources:
-        assert isinstance(links[0].sources[0], str)
-
-    main_index_json = {
-        **MAIN_INDEX_HEADER,
-        'num_links': len(links),
-        'updated': datetime.now(),
-        'last_run_cmd': sys.argv,
-        'links': links,
-    }
-    atomic_write(os.path.join(out_dir, JSON_INDEX_FILENAME), main_index_json)
-
-
 ### Link Details Index
 
 @enforce_types
-def write_json_link_details(link: Link, out_dir: Optional[str]=None) -> None:
-    """write a json file with some info about the link"""
+def write_json_snapshot_details(snapshot: Model, out_dir: Optional[str]=None) -> None:
+    """write a json file with some info about the snapshot"""
     
-    out_dir = out_dir or link.link_dir
-    path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    atomic_write(path, link._asdict(extended=True))
+    out_dir = out_dir or snapshot.snapshot_dir
+    path = Path(out_dir) / JSON_INDEX_FILENAME
+    atomic_write(str(path), snapshot.as_json())
 
 
 @enforce_types
-def parse_json_link_details(out_dir: str, guess: Optional[bool]=False) -> Optional[Link]:
-    """load the json link index from a given directory"""
-    existing_index = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(existing_index):
+def load_json_snapshot(out_dir: Path) -> Optional[Model]:
+    """
+    Loads the detail from the local json index
+    """
+    from core.models import Snapshot
+
+    existing_index = Path(out_dir) / JSON_INDEX_FILENAME
+    if existing_index.exists():
         with open(existing_index, 'r', encoding='utf-8') as f:
             try:
-                link_json = pyjson.load(f)
-                return Link.from_json(link_json, guess)
+                output = pyjson.load(f)
+                output = Snapshot.from_json(output)
+                return output
             except pyjson.JSONDecodeError:
                 pass
     return None
 
 
 @enforce_types
-def parse_json_links_details(out_dir: str) -> Iterator[Link]:
-    """read through all the archive data folders and return the parsed links"""
+def parse_json_snapshot_details(out_dir: Union[Path, str]) -> Iterator[dict]:
+    """read through all the archive data folders and return the parsed snapshots"""
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
+    for entry in os.scandir(Path(out_dir) / ARCHIVE_DIR_NAME):
         if entry.is_dir(follow_symlinks=True):
-            if os.path.exists(os.path.join(entry.path, 'index.json')):
+            if (Path(entry.path) / 'index.json').exists():
                 try:
-                    link = parse_json_link_details(entry.path)
+                    snapshot_details = load_json_snapshot(Path(entry.path))
                 except KeyError:
-                    link = None
-                if link:
-                    yield link
+                    snapshot_details = None
+                if snapshot_details:
+                    yield snapshot_details
 
 
 

archivebox/index/schema.py

@@ -1,6 +1,13 @@
+"""
+
+WARNING: THIS FILE IS ALL LEGACY CODE TO BE REMOVED.
+
+DO NOT ADD ANY NEW FEATURES TO THIS FILE, NEW CODE GOES HERE: core/models.py
+
+"""
+
 __package__ = 'archivebox.index'
 
-import os
 from pathlib import Path
 
 from datetime import datetime, timedelta
@@ -32,6 +39,7 @@ class ArchiveResult:
     status: str
     start_ts: datetime
     end_ts: datetime
+    index_texts: Union[List[str], None] = None
     schema: str = 'ArchiveResult'
 
     def __post_init__(self):
@@ -127,7 +135,7 @@ class Link:
 
 
     def __str__(self) -> str:
-        return f'[{self.timestamp}] {self.base_url} "{self.title}"'
+        return f'[{self.timestamp}] {self.url} "{self.title}"'
 
     def __post_init__(self):
         self.typecheck()
@@ -208,6 +216,10 @@ class Link:
             })
         return info
 
+    def as_snapshot(self):
+        from core.models import Snapshot
+        return Snapshot.objects.get(url=self.url)
+
     @classmethod
     def from_json(cls, json_info, guess=False):
         from ..util import parse_date
@@ -250,7 +262,7 @@ class Link:
     @property
     def link_dir(self) -> str:
         from ..config import CONFIG
-        return os.path.join(CONFIG['ARCHIVE_DIR'], self.timestamp)
+        return str(Path(CONFIG['ARCHIVE_DIR']) / self.timestamp)
 
     @property
     def archive_path(self) -> str:
@@ -340,7 +352,7 @@ class Link:
     ### Archive Status Helpers
     @property
     def num_outputs(self) -> int:
-        return len(tuple(filter(None, self.latest_outputs().values())))
+        return self.as_snapshot().num_outputs
 
     @property
     def num_failures(self) -> int:
@@ -369,7 +381,7 @@ class Link:
         )
 
         return any(
-            os.path.exists(os.path.join(ARCHIVE_DIR, self.timestamp, path))
+            (Path(ARCHIVE_DIR) / self.timestamp / path).exists()
             for path in output_paths
         )
 
@@ -396,24 +408,29 @@ class Link:
         return latest
 
 
+
     def canonical_outputs(self) -> Dict[str, Optional[str]]:
         """predict the expected output paths that should be present after archiving"""
 
         from ..extractors.wget import wget_output_path
+        # TODO: banish this awful duplication from the codebase and import these
+        # from their respective extractor files
         canonical = {
             'index_path': 'index.html',
             'favicon_path': 'favicon.ico',
             'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain),
             'wget_path': wget_output_path(self),
-            'warc_path': 'warc',
+            'warc_path': 'warc/',
             'singlefile_path': 'singlefile.html',
             'readability_path': 'readability/content.html',
+            'mercury_path': 'mercury/content.html',
             'pdf_path': 'output.pdf',
             'screenshot_path': 'screenshot.png',
             'dom_path': 'output.html',
             'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url),
-            'git_path': 'git',
-            'media_path': 'media',
+            'git_path': 'git/',
+            'media_path': 'media/',
+            'headers_path': 'headers.json',
         }
         if self.is_static:
             # static binary files like PDF and images are handled slightly differently.
@@ -429,6 +446,7 @@ class Link:
                 'dom_path': static_path,
                 'singlefile_path': static_path,
                 'readability_path': static_path,
+                'mercury_path': static_path,
             })
         return canonical
 

archivebox/index/sql.py

@@ -1,18 +1,21 @@
 __package__ = 'archivebox.index'
 
 from io import StringIO
+from pathlib import Path
 from typing import List, Tuple, Iterator
+from django.db.models import QuerySet, Model
+from django.db import transaction
+from datetime import datetime
 
 from .schema import Link
 from ..util import enforce_types
-from ..config import setup_django, OUTPUT_DIR
+from ..config import OUTPUT_DIR
 
 
 ### Main Links Index
 
 @enforce_types
-def parse_sql_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
-    setup_django(out_dir, check_db=True)
+def parse_sql_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]:
     from core.models import Snapshot
 
     return (
@@ -21,47 +24,60 @@ def parse_sql_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
     )
 
 @enforce_types
-def remove_from_sql_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
-    from core.models import Snapshot
-    from django.db import transaction
-
+def remove_from_sql_main_index(snapshots: QuerySet, out_dir: Path=OUTPUT_DIR) -> None:
     with transaction.atomic():
-        for link in links:
-            Snapshot.objects.filter(url=link.url).delete()
+        snapshots.delete()
 
 @enforce_types
-def write_sql_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
+def write_snapshot_to_index(snapshot: Model):
     from core.models import Snapshot
-    from django.db import transaction
+    try:
+        timestamp = Snapshot.objects.get(url=snapshot.url).timestamp
+    except Snapshot.DoesNotExist:
+        timestamp = snapshot.timestamp
+        if not timestamp:
+            timestamp = str(datetime.now().timestamp())
+        while Snapshot.objects.filter(timestamp=timestamp).exists():
+            print("the timestamp is: ", timestamp)
+            timestamp = str(float(timestamp) + 1.0)
+
+    snapshot.timestamp = timestamp
+    snapshot.save()
+    return snapshot
 
-    with transaction.atomic():
-        for link in links:
-            info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys}
-            try:
-                info['timestamp'] = Snapshot.objects.get(url=link.url).timestamp
-            except Snapshot.DoesNotExist:
-                pass
-            Snapshot.objects.update_or_create(url=link.url, defaults=info)
 
 @enforce_types
-def write_sql_link_details(link: Link, out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
+def write_sql_main_index(snapshots: List[Model], out_dir: Path=OUTPUT_DIR) -> None:
+    with transaction.atomic():
+        for snapshot in snapshots:
+            write_snapshot_to_index(snapshot)
+            
+
+@enforce_types
+def write_sql_snapshot_details(snapshot: Model, out_dir: Path=OUTPUT_DIR) -> None:
     from core.models import Snapshot
-    from django.db import transaction
 
     with transaction.atomic():
-        snap = Snapshot.objects.get(url=link.url)
-        snap.title = link.title
-        snap.tags = link.tags
+        try:
+            snap = Snapshot.objects.get(url=snapshot.url)
+        except Snapshot.DoesNotExist:
+            snap = write_snapshot_to_index(snapshot)
+        snap.title = snapshot.title
+
+        # TODO: If there are actual tags, this will break
+        #tag_set = (
+        #    set(tag.strip() for tag in (snapshot.tags.all() or '').split(','))
+        #)
+        #tag_list = list(tag_set) or []
+
         snap.save()
+        #snap.save_tags(tag_list)
+        return snap
 
 
 
 @enforce_types
-def list_migrations(out_dir: str=OUTPUT_DIR) -> List[Tuple[bool, str]]:
-    setup_django(out_dir, check_db=False)
+def list_migrations(out_dir: Path=OUTPUT_DIR) -> List[Tuple[bool, str]]:
     from django.core.management import call_command
     out = StringIO()
     call_command("showmigrations", list=True, stdout=out)
@@ -77,8 +93,7 @@ def list_migrations(out_dir: str=OUTPUT_DIR) -> List[Tuple[bool, str]]:
     return migrations
 
 @enforce_types
-def apply_migrations(out_dir: str=OUTPUT_DIR) -> List[str]:
-    setup_django(out_dir, check_db=False)
+def apply_migrations(out_dir: Path=OUTPUT_DIR) -> List[str]:
     from django.core.management import call_command
     null, out = StringIO(), StringIO()
     call_command("makemigrations", interactive=False, stdout=null)
@@ -88,7 +103,6 @@ def apply_migrations(out_dir: str=OUTPUT_DIR) -> List[str]:
     return [line.strip() for line in out.readlines() if line.strip()]
 
 @enforce_types
-def get_admins(out_dir: str=OUTPUT_DIR) -> List[str]:
-    setup_django(out_dir, check_db=False)
+def get_admins(out_dir: Path=OUTPUT_DIR) -> List[str]:
     from django.contrib.auth.models import User
     return User.objects.filter(is_superuser=True)

archivebox/logging_util.py

@@ -5,7 +5,9 @@ import os
 import sys
 import time
 import argparse
+from math import log
 from multiprocessing import Process
+from pathlib import Path
 
 from datetime import datetime
 from dataclasses import dataclass
@@ -17,13 +19,13 @@ if TYPE_CHECKING:
 from .util import enforce_types
 from .config import (
     ConfigDict,
+    OUTPUT_DIR,
     PYTHON_ENCODING,
     ANSI,
     IS_TTY,
     TERM_WIDTH,
-    OUTPUT_DIR,
+    SHOW_PROGRESS,
     SOURCES_DIR_NAME,
-    HTML_INDEX_FILENAME,
     stderr,
 )
 
@@ -83,7 +85,6 @@ class TimedProgress:
     """Show a progress bar and measure elapsed time until .end() is called"""
 
     def __init__(self, seconds, prefix=''):
-        from .config import SHOW_PROGRESS
         self.SHOW_PROGRESS = SHOW_PROGRESS
         if self.SHOW_PROGRESS:
             self.p = Process(target=progress_bar, args=(seconds, prefix))
@@ -99,15 +100,23 @@ class TimedProgress:
         
         if self.SHOW_PROGRESS:
             # terminate if we havent already terminated
-            self.p.terminate()
-            self.p.join()
-            self.p.close()
-
-            # clear whole terminal line
             try:
-                sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
-            except (IOError, BrokenPipeError):
-                # ignore when the parent proc has stopped listening to our stdout
+                # kill the progress bar subprocess
+                try:
+                    self.p.close()   # must be closed *before* its terminnated
+                except:
+                    pass
+                self.p.terminate()
+                self.p.join()
+
+
+                # clear whole terminal line
+                try:
+                    sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
+                except (IOError, BrokenPipeError):
+                    # ignore when the parent proc has stopped listening to our stdout
+                    pass
+            except ValueError:
                 pass
 
 
@@ -126,17 +135,18 @@ def progress_bar(seconds: int, prefix: str='') -> None:
                 sys.stdout.write('\r\n')
                 sys.stdout.flush()
             chunks = max_width - len(prefix) - 20
-            progress = s / chunks / seconds * 100
-            bar_width = round(progress/(100/chunks))
+            pct_complete = s / chunks / seconds * 100
+            log_pct = (log(pct_complete or 1, 10) / 2) * 100  # everyone likes faster progress bars ;)
+            bar_width = round(log_pct/(100/chunks))
             last_width = max_width
 
             # ████████████████████           0.9% (1/60sec)
             sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format(
                 prefix,
-                ANSI['green'],
+                ANSI['green' if pct_complete < 80 else 'lightyellow'],
                 (chunk * bar_width).ljust(chunks),
                 ANSI['reset'],
-                round(progress, 1),
+                round(pct_complete, 1),
                 round(s/chunks),
                 seconds,
             ))
@@ -144,7 +154,7 @@ def progress_bar(seconds: int, prefix: str='') -> None:
             time.sleep(1 / chunks)
 
         # ██████████████████████████████████ 100.0% (60/60sec)
-        sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)\n'.format(
+        sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format(
             prefix,
             ANSI['red'],
             chunk * chunks,
@@ -154,6 +164,10 @@ def progress_bar(seconds: int, prefix: str='') -> None:
             seconds,
         ))
         sys.stdout.flush()
+        # uncomment to have it disappear when it hits 100% instead of staying full red:
+        # time.sleep(0.5)
+        # sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
+        # sys.stdout.flush()
     except (KeyboardInterrupt, BrokenPipeError):
         print()
         pass
@@ -240,7 +254,7 @@ def log_archiving_started(num_links: int, resume: Optional[float]=None):
              **ANSI,
         ))
     else:
-        print('{green}[▶] [{}] Collecting content for {} Snapshots in archive...{reset}'.format(
+        print('{green}[▶] [{}] Starting archiving of {} snapshots in index...{reset}'.format(
              start_ts.strftime('%Y-%m-%d %H:%M:%S'),
              num_links,
              **ANSI,
@@ -258,8 +272,8 @@ def log_archiving_paused(num_links: int, idx: int, timestamp: str):
         total=num_links,
     ))
     print()
-    print('    {lightred}Hint:{reset} To view your archive index, open:'.format(**ANSI))
-    print('        {}/{}'.format(OUTPUT_DIR, HTML_INDEX_FILENAME))
+    print('    {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI))
+    print('        archivebox server  # then visit http://127.0.0.1:8000')
     print('    Continue archiving where you left off by running:')
     print('        archivebox update --resume={}'.format(timestamp))
 
@@ -285,13 +299,11 @@ def log_archiving_finished(num_links: int):
     print('    - {} links updated'.format(_LAST_RUN_STATS.succeeded + _LAST_RUN_STATS.failed))
     print('    - {} links had errors'.format(_LAST_RUN_STATS.failed))
     print()
-    print('    {lightred}Hint:{reset} To view your archive index, open:'.format(**ANSI))
-    print('        {}/{}'.format(OUTPUT_DIR, HTML_INDEX_FILENAME))
-    print('    Or run the built-in webserver:')
-    print('        archivebox server')
+    print('    {lightred}Hint:{reset} To manage your archive in a Web UI, run:'.format(**ANSI))
+    print('        archivebox server 0.0.0.0:8000')
 
 
-def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool):
+def log_snapshot_archiving_started(snapshot: Model, snapshot_dir: str, is_new: bool):
     # [*] [2019-03-22 13:46:45] "Log Structured Merge Trees - ben stopford"
     #     http://www.benstopford.com/2015/02/14/log-structured-merge-trees/
     #     > output/archive/1478739709
@@ -300,16 +312,16 @@ def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool):
         symbol_color=ANSI['green' if is_new else 'black'],
         symbol='+' if is_new else '√',
         now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        title=link.title or link.base_url,
+        title=snapshot.title or snapshot.base_url,
         **ANSI,
     ))
-    print('    {blue}{url}{reset}'.format(url=link.url, **ANSI))
+    print('    {blue}{url}{reset}'.format(url=snapshot.url, **ANSI))
     print('    {} {}'.format(
         '>' if is_new else '√',
-        pretty_path(link_dir),
+        pretty_path(snapshot_dir),
     ))
 
-def log_link_archiving_finished(link: "Link", link_dir: str, is_new: bool, stats: dict):
+def log_snapshot_archiving_finished(snapshot: Model, snapshot_dir: str, is_new: bool, stats: dict):
     total = sum(stats.values())
 
     if stats['failed'] > 0 :
@@ -335,6 +347,21 @@ def log_archive_method_finished(result: "ArchiveResult"):
     )
 
     if result.status == 'failed':
+        if result.output.__class__.__name__ == 'TimeoutExpired':
+            duration = (result.end_ts - result.start_ts).seconds
+            hint_header = [
+                '{lightyellow}Extractor timed out after {}s.{reset}'.format(duration, **ANSI),
+            ]
+        else:
+            hint_header = [
+                '{lightyellow}Extractor failed:{reset}'.format(**ANSI),
+                '    {reset}{} {red}{}{reset}'.format(
+                    result.output.__class__.__name__.replace('ArchiveError', ''),
+                    result.output, 
+                    **ANSI,
+                ),
+            ]
+
         # Prettify error output hints string and limit to five lines
         hints = getattr(result.output, 'hints', None) or ()
         if hints:
@@ -344,14 +371,10 @@ def log_archive_method_finished(result: "ArchiveResult"):
                 for line in hints[:5] if line.strip()
             )
 
+
         # Collect and prefix output lines with indentation
         output_lines = [
-            '{lightyellow}Extractor failed:{reset}'.format(**ANSI),
-            '    {reset}{} {red}{}{reset}'.format(
-                result.output.__class__.__name__.replace('ArchiveError', ''),
-                result.output, 
-                **ANSI,
-            ),
+            *hint_header,
             *hints,
             '{}Run to see full output:{}'.format(ANSI['lightred'], ANSI['reset']),
             *(['    cd {};'.format(result.pwd)] if result.pwd else []),
@@ -372,61 +395,60 @@ def log_list_started(filter_patterns: Optional[List[str]], filter_type: str):
     ))
     print('    {}'.format(' '.join(filter_patterns or ())))
 
-def log_list_finished(links):
-    from .index.csv import links_to_csv
+def log_list_finished(snapshots):
+    from .index.csv import snapshots_to_csv
     print()
     print('---------------------------------------------------------------------------------------------------')
-    print(links_to_csv(links, cols=['timestamp', 'is_archived', 'num_outputs', 'url'], header=True, ljust=16, separator=' | '))
+    print(snapshots_to_csv(snapshots, cols=['timestamp', 'is_archived', 'num_outputs', 'url'], header=True, ljust=16, separator=' | '))
     print('---------------------------------------------------------------------------------------------------')
     print()
 
 
-def log_removal_started(links: List["Link"], yes: bool, delete: bool):
-    print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(links), **ANSI))
+def log_removal_started(snapshots: List["Snapshot"], yes: bool, delete: bool):
+    print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(snapshots), **ANSI))
     if delete:
-        file_counts = [link.num_outputs for link in links if os.path.exists(link.link_dir)]
+        file_counts = [snapshot.num_outputs for snapshot in snapshots if Path(snapshot.snapshot_dir).exists()]
         print(
-            f'    {len(links)} Links will be de-listed from the main index, and their archived content folders will be deleted from disk.\n'
+            f'    {len(snapshots)} Snapshots will be de-listed from the main index, and their archived content folders will be deleted from disk.\n'
             f'    ({len(file_counts)} data folders with {sum(file_counts)} archived files will be deleted!)'
         )
     else:
         print(
-            '    Matching links will be de-listed from the main index, but their archived content folders will remain in place on disk.\n'
+            '    Matching snapshots will be de-listed from the main index, but their archived content folders will remain in place on disk.\n'
             '    (Pass --delete if you also want to permanently delete the data folders)'
         )
 
     if not yes:
         print()
-        print('{lightyellow}[?] Do you want to proceed with removing these {} links?{reset}'.format(len(links), **ANSI))
+        print('{lightyellow}[?] Do you want to proceed with removing these {} snapshots?{reset}'.format(len(snapshots), **ANSI))
         try:
             assert input('    y/[n]: ').lower() == 'y'
         except (KeyboardInterrupt, EOFError, AssertionError):
             raise SystemExit(0)
 
-def log_removal_finished(all_links: int, to_keep: int):
-    if all_links == 0:
+def log_removal_finished(all_snapshots: int, to_remove: int):
+    if to_remove == 0:
         print()
-        print('{red}[X] No matching links found.{reset}'.format(**ANSI))
+        print('{red}[X] No matching snapshots found.{reset}'.format(**ANSI))
     else:
-        num_removed = all_links - to_keep
         print()
-        print('{red}[√] Removed {} out of {} links from the archive index.{reset}'.format(
-            num_removed,
-            all_links,
+        print('{red}[√] Removed {} out of {} snapshots from the archive index.{reset}'.format(
+            to_remove,
+            all_snapshots,
             **ANSI,
         ))
-        print('    Index now contains {} links.'.format(to_keep))
+        print('    Index now contains {} snapshots.'.format(all_snapshots - to_remove))
 
 
 def log_shell_welcome_msg():
     from .cli import list_subcommands
 
     print('{green}# ArchiveBox Imports{reset}'.format(**ANSI))
-    print('{green}from archivebox.core.models import Snapshot, User{reset}'.format(**ANSI))
+    print('{green}from core.models import Snapshot, User{reset}'.format(**ANSI))
     print('{green}from archivebox import *\n    {}{reset}'.format("\n    ".join(list_subcommands().keys()), **ANSI))
     print()
     print('[i] Welcome to the ArchiveBox Shell!')
-    print('    https://github.com/pirate/ArchiveBox/wiki/Usage#Shell-Usage')
+    print('    https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage')
     print()
     print('    {lightred}Hint:{reset} Example use:'.format(**ANSI))
     print('        print(Snapshot.objects.filter(is_archived=True).count())')
@@ -438,11 +460,11 @@ def log_shell_welcome_msg():
 ### Helpers
 
 @enforce_types
-def pretty_path(path: str) -> str:
+def pretty_path(path: Union[Path, str]) -> str:
     """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc"""
-    pwd = os.path.abspath('.')
+    pwd = Path('.').resolve()
     # parent = os.path.abspath(os.path.join(pwd, os.path.pardir))
-    return path.replace(pwd + '/', './')
+    return str(path).replace(str(pwd) + '/', './')
 
 
 @enforce_types
@@ -456,17 +478,11 @@ def printable_filesize(num_bytes: Union[int, float]) -> str:
 
 @enforce_types
 def printable_folders(folders: Dict[str, Optional["Link"]],
-                      json: bool=False,
-                      csv: Optional[str]=None) -> str:
-    if json: 
-        from .index.json import to_json
-        return to_json(folders.values(), indent=4, sort_keys=True)
-
-    elif csv:
-        from .index.csv import links_to_csv
-        return links_to_csv(folders.values(), cols=csv.split(','), header=True)
-    
-    return '\n'.join(f'{folder} {link}' for folder, link in folders.items())
+                      with_headers: bool=False) -> str:
+    return '\n'.join(
+        f'{folder} {link and link.url} "{link and link.title}"'
+        for folder, link in folders.items()
+    )
 
 
 
@@ -490,28 +506,33 @@ def printable_folder_status(name: str, folder: Dict) -> str:
         color, symbol, note, num_files = 'lightyellow', '-', 'disabled', '-'
 
     if folder['path']:
-        if os.path.exists(folder['path']):
+        if Path(folder['path']).exists():
             num_files = (
                 f'{len(os.listdir(folder["path"]))} files'
-                if os.path.isdir(folder['path']) else
-                printable_filesize(os.path.getsize(folder['path']))
+                if Path(folder['path']).is_dir() else
+                printable_filesize(Path(folder['path']).stat().st_size)
             )
         else:
             num_files = 'missing'
 
-        if ' ' in folder['path']:
-            folder['path'] = f'"{folder["path"]}"'
+    path = str(folder['path']).replace(str(OUTPUT_DIR), '.') if folder['path'] else ''
+    if path and ' ' in path:
+        path = f'"{path}"'
+
+    # if path is just a plain dot, replace it back with the full path for clarity
+    if path == '.':
+        path = str(OUTPUT_DIR)
 
     return ' '.join((
         ANSI[color],
         symbol,
         ANSI['reset'],
-        name.ljust(22),
-        (folder["path"] or '').ljust(76),
+        name.ljust(21),
         num_files.ljust(14),
         ANSI[color],
-        note,
+        note.ljust(8),
         ANSI['reset'],
+        path.ljust(76),
     ))
 
 
@@ -531,17 +552,18 @@ def printable_dependency_version(name: str, dependency: Dict) -> str:
     else:
         color, symbol, note, version = 'lightyellow', '-', 'disabled', '-'
 
-    if ' ' in dependency["path"]:
-        dependency["path"] = f'"{dependency["path"]}"'
+    path = str(dependency["path"]).replace(str(OUTPUT_DIR), '.') if dependency["path"] else ''
+    if path and ' ' in path:
+        path = f'"{path}"'
 
     return ' '.join((
         ANSI[color],
         symbol,
         ANSI['reset'],
-        name.ljust(22),
-        (dependency["path"] or '').ljust(76),
+        name.ljust(21),
         version.ljust(14),
         ANSI[color],
-        note,
+        note.ljust(8),
         ANSI['reset'],
+        path.ljust(76),
     ))

archivebox/main.py

@@ -3,9 +3,13 @@ __package__ = 'archivebox'
 import os
 import sys
 import shutil
+import platform
+from pathlib import Path
+from datetime import date
 
 from typing import Dict, List, Optional, Iterable, IO, Union
 from crontab import CronTab, CronSlices
+from django.db.models import QuerySet, Model
 
 from .cli import (
     list_subcommands,
@@ -18,17 +22,17 @@ from .cli import (
 from .parsers import (
     save_text_as_source,
     save_file_as_source,
-    parse_links_memory,
+    parse_snapshots_memory,
 )
 from .index.schema import Link
 from .util import enforce_types                         # type: ignore
 from .system import get_dir_size, dedupe_cron_jobs, CRON_COMMENT
 from .index import (
     load_main_index,
-    parse_links_from_source,
-    dedupe_links,
+    parse_snapshots_from_source,
+    filter_new_urls,
     write_main_index,
-    link_matches_filter,
+    snapshot_filter,
     get_indexed_folders,
     get_archived_folders,
     get_unarchived_folders,
@@ -40,19 +44,23 @@ from .index import (
     get_corrupted_folders,
     get_unrecognized_folders,
     fix_invalid_folder_locations,
+    write_snapshot_details,
 )
 from .index.json import (
     parse_json_main_index,
-    parse_json_links_details,
+    parse_json_snapshot_details,
+    generate_json_index_from_snapshots,
 )
 from .index.sql import (
-    parse_sql_main_index,
     get_admins,
     apply_migrations,
     remove_from_sql_main_index,
 )
-from .index.html import parse_html_main_index
-from .extractors import archive_links, archive_link, ignore_methods
+from .index.html import (
+    generate_index_from_snapshots,
+)
+from .index.csv import snapshots_to_csv
+from .extractors import archive_snapshots, archive_snapshot, ignore_methods
 from .config import (
     stderr,
     hint,
@@ -71,7 +79,6 @@ from .config import (
     ARCHIVE_DIR_NAME,
     SOURCES_DIR_NAME,
     LOGS_DIR_NAME,
-    STATIC_DIR_NAME,
     JSON_INDEX_FILENAME,
     HTML_INDEX_FILENAME,
     SQL_INDEX_FILENAME,
@@ -80,7 +87,6 @@ from .config import (
     check_dependencies,
     check_data_folder,
     write_config_file,
-    setup_django,
     VERSION,
     CODE_LOCATIONS,
     EXTERNAL_LOCATIONS,
@@ -107,17 +113,21 @@ from .logging_util import (
     printable_dependency_version,
 )
 
+from .search import flush_search_index, index_snapshots
 
 ALLOWED_IN_OUTPUT_DIR = {
+    'lost+found',
     '.DS_Store',
     '.venv',
     'venv',
     'virtualenv',
     '.virtualenv',
+    'node_modules',
+    'package-lock.json',
+    'static',
     ARCHIVE_DIR_NAME,
     SOURCES_DIR_NAME,
     LOGS_DIR_NAME,
-    STATIC_DIR_NAME,
     SQL_INDEX_FILENAME,
     JSON_INDEX_FILENAME,
     HTML_INDEX_FILENAME,
@@ -126,7 +136,7 @@ ALLOWED_IN_OUTPUT_DIR = {
 }
 
 @enforce_types
-def help(out_dir: str=OUTPUT_DIR) -> None:
+def help(out_dir: Path=OUTPUT_DIR) -> None:
     """Print the ArchiveBox help message and usage"""
 
     all_subcommands = list_subcommands()
@@ -149,7 +159,7 @@ def help(out_dir: str=OUTPUT_DIR) -> None:
     )
 
 
-    if os.path.exists(os.path.join(out_dir, JSON_INDEX_FILENAME)):
+    if (Path(out_dir) / SQL_INDEX_FILENAME).exists():
         print('''{green}ArchiveBox v{}: The self-hosted internet archive.{reset}
 
 {lightred}Active data directory:{reset}
@@ -174,7 +184,7 @@ def help(out_dir: str=OUTPUT_DIR) -> None:
     archivebox update --resume=15109948213.123
 
 {lightred}Documentation:{reset}
-    https://github.com/pirate/ArchiveBox/wiki
+    https://github.com/ArchiveBox/ArchiveBox/wiki
 '''.format(VERSION, out_dir, COMMANDS_HELP_TEXT, **ANSI))
     
     else:
@@ -193,18 +203,20 @@ def help(out_dir: str=OUTPUT_DIR) -> None:
         print('    2. archivebox init')
         print()
         print('For more information, see the documentation here:')
-        print('    https://github.com/pirate/ArchiveBox/wiki')
+        print('    https://github.com/ArchiveBox/ArchiveBox/wiki')
 
 
 @enforce_types
 def version(quiet: bool=False,
-            out_dir: str=OUTPUT_DIR) -> None:
+            out_dir: Path=OUTPUT_DIR) -> None:
     """Print the ArchiveBox version and dependency information"""
 
     if quiet:
         print(VERSION)
     else:
         print('ArchiveBox v{}'.format(VERSION))
+        p = platform.uname()
+        print(sys.implementation.name.title(), p.system, platform.platform(), p.machine, '(in Docker)' if IN_DOCKER else '(not in Docker)')
         print()
 
         print('{white}[i] Dependency versions:{reset}'.format(**ANSI))
@@ -212,7 +224,7 @@ def version(quiet: bool=False,
             print(printable_dependency_version(name, dependency))
         
         print()
-        print('{white}[i] Code locations:{reset}'.format(**ANSI))
+        print('{white}[i] Source-code locations:{reset}'.format(**ANSI))
         for name, folder in CODE_LOCATIONS.items():
             print(printable_folder_status(name, folder))
 
@@ -221,11 +233,14 @@ def version(quiet: bool=False,
         for name, folder in EXTERNAL_LOCATIONS.items():
             print(printable_folder_status(name, folder))
 
+        print()
         if DATA_LOCATIONS['OUTPUT_DIR']['is_valid']:
-            print()
             print('{white}[i] Data locations:{reset}'.format(**ANSI))
             for name, folder in DATA_LOCATIONS.items():
                 print(printable_folder_status(name, folder))
+        else:
+            print()
+            print('{white}[i] Data locations:{reset}'.format(**ANSI))
 
         print()
         check_dependencies()
@@ -235,7 +250,7 @@ def version(quiet: bool=False,
 def run(subcommand: str,
         subcommand_args: Optional[List[str]],
         stdin: Optional[IO]=None,
-        out_dir: str=OUTPUT_DIR) -> None:
+        out_dir: Path=OUTPUT_DIR) -> None:
     """Run a given ArchiveBox subcommand with the given list of args"""
     run_subcommand(
         subcommand=subcommand,
@@ -246,11 +261,17 @@ def run(subcommand: str,
 
 
 @enforce_types
-def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
+def init(force: bool=False, out_dir: Path=OUTPUT_DIR) -> None:
     """Initialize a new ArchiveBox collection in the current directory"""
-    os.makedirs(out_dir, exist_ok=True)
+    from core.models import Snapshot
+    Path(out_dir).mkdir(exist_ok=True)
     is_empty = not len(set(os.listdir(out_dir)) - ALLOWED_IN_OUTPUT_DIR)
-    existing_index = os.path.exists(os.path.join(out_dir, JSON_INDEX_FILENAME))
+
+    if (Path(out_dir) / JSON_INDEX_FILENAME).exists():
+        stderr("[!] This folder contains a JSON index. It is deprecated, and will no longer be kept up to date automatically.", color="lightyellow")
+        stderr("    You can run `archivebox list --json --with-headers > index.json` to manually generate it.", color="lightyellow")
+
+    existing_index = (Path(out_dir) / SQL_INDEX_FILENAME).exists()
 
     if is_empty and not existing_index:
         print('{green}[+] Initializing a new ArchiveBox collection in this folder...{reset}'.format(**ANSI))
@@ -262,11 +283,11 @@ def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
         print('{green}------------------------------------------------------------------{reset}'.format(**ANSI))
     else:
         if force:
-            stderr('[!] This folder appears to already have files in it, but no index.json is present.', color='lightyellow')
+            stderr('[!] This folder appears to already have files in it, but no index.sqlite3 is present.', color='lightyellow')
             stderr('    Because --force was passed, ArchiveBox will initialize anyway (which may overwrite existing files).')
         else:
             stderr(
-                ("{red}[X] This folder appears to already have files in it, but no index.json is present.{reset}\n\n"
+                ("{red}[X] This folder appears to already have files in it, but no index.sqlite3 present.{reset}\n\n"
                 "    You must run init in a completely empty directory, or an existing data folder.\n\n"
                 "    {lightred}Hint:{reset} To import an existing data folder make sure to cd into the folder first, \n"
                 "    then run and run 'archivebox init' to pick up where you left off.\n\n"
@@ -280,32 +301,30 @@ def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
     else:
         print('\n{green}[+] Building archive folder structure...{reset}'.format(**ANSI))
     
-    os.makedirs(SOURCES_DIR, exist_ok=True)
+    Path(SOURCES_DIR).mkdir(exist_ok=True)
     print(f'    √ {SOURCES_DIR}')
     
-    os.makedirs(ARCHIVE_DIR, exist_ok=True)
+    Path(ARCHIVE_DIR).mkdir(exist_ok=True)
     print(f'    √ {ARCHIVE_DIR}')
 
-    os.makedirs(LOGS_DIR, exist_ok=True)
+    Path(LOGS_DIR).mkdir(exist_ok=True)
     print(f'    √ {LOGS_DIR}')
 
     write_config_file({}, out_dir=out_dir)
     print(f'    √ {CONFIG_FILE}')
-    
-    if os.path.exists(os.path.join(out_dir, SQL_INDEX_FILENAME)):
+    if (Path(out_dir) / SQL_INDEX_FILENAME).exists():
         print('\n{green}[*] Verifying main SQL index and running migrations...{reset}'.format(**ANSI))
     else:
         print('\n{green}[+] Building main SQL index and running migrations...{reset}'.format(**ANSI))
     
-    setup_django(out_dir, check_db=False)
-    DATABASE_FILE = os.path.join(out_dir, SQL_INDEX_FILENAME)
+    DATABASE_FILE = Path(out_dir) / SQL_INDEX_FILENAME
     print(f'    √ {DATABASE_FILE}')
     print()
     for migration_line in apply_migrations(out_dir):
         print(f'    {migration_line}')
 
 
-    assert os.path.exists(DATABASE_FILE)
+    assert DATABASE_FILE.exists()
     
     # from django.contrib.auth.models import User
     # if IS_TTY and not User.objects.filter(is_superuser=True).exists():
@@ -315,13 +334,12 @@ def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
     print()
     print('{green}[*] Collecting links from any existing indexes and archive folders...{reset}'.format(**ANSI))
 
-    all_links: Dict[str, Link] = {}
+    all_snapshots = Snapshot.objects.none()
+    pending_snapshots: Dict[str, Snapshot] = {}
+
     if existing_index:
-        all_links = {
-            link.url: link
-            for link in load_main_index(out_dir=out_dir, warn=False)
-        }
-        print('    √ Loaded {} links from existing main index.'.format(len(all_links)))
+        all_snapshots = load_main_index(out_dir=out_dir, warn=False)
+        print('    √ Loaded {} snapshots from existing main index.'.format(all_snapshots.count()))
 
     # Links in data folders that dont match their timestamp
     fixed, cant_fix = fix_invalid_folder_locations(out_dir=out_dir)
@@ -331,56 +349,51 @@ def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
         print('    {lightyellow}! Could not fix {} data directory locations due to conflicts with existing folders.{reset}'.format(len(cant_fix), **ANSI))
 
     # Links in JSON index but not in main index
-    orphaned_json_links = {
-        link.url: link
-        for link in parse_json_main_index(out_dir)
-        if link.url not in all_links
+    orphaned_json_snapshots = {
+        snapshot.url: snapshot
+        for snapshot in parse_json_main_index(out_dir)
+        if not all_snapshots.filter(url=snapshot.url).exists()
     }
-    if orphaned_json_links:
-        all_links.update(orphaned_json_links)
-        print('    {lightyellow}√ Added {} orphaned links from existing JSON index...{reset}'.format(len(orphaned_json_links), **ANSI))
-
-    # Links in SQL index but not in main index
-    orphaned_sql_links = {
-        link.url: link
-        for link in parse_sql_main_index(out_dir)
-        if link.url not in all_links
-    }
-    if orphaned_sql_links:
-        all_links.update(orphaned_sql_links)
-        print('    {lightyellow}√ Added {} orphaned links from existing SQL index...{reset}'.format(len(orphaned_sql_links), **ANSI))
+    if orphaned_json_snapshots:
+        pending_snapshots.update(orphaned_json_snapshots)
+        print('    {lightyellow}√ Added {} orphaned snapshots from deprecated JSON index...{reset}'.format(len(orphaned_json_snapshots), **ANSI))
 
     # Links in data dir indexes but not in main index
-    orphaned_data_dir_links = {
-        link.url: link
-        for link in parse_json_links_details(out_dir)
-        if link.url not in all_links
+    orphaned_data_dir_snapshots = {
+        snapshot.url: snapshot
+        for snapshot in parse_json_snapshot_details(out_dir)
+        if not all_snapshots.filter(url=snapshot.url).exists()
     }
-    if orphaned_data_dir_links:
-        all_links.update(orphaned_data_dir_links)
-        print('    {lightyellow}√ Added {} orphaned links from existing archive directories.{reset}'.format(len(orphaned_data_dir_links), **ANSI))
+    if orphaned_data_dir_snapshots:
+        pending_snapshots.update(orphaned_data_dir_snapshots)
+        print('    {lightyellow}√ Added {} orphaned snapshots from existing archive directories.{reset}'.format(len(orphaned_data_dir_snapshots), **ANSI))
+
+
+    # TODO: Should we remove orphaned folders from the invalid list? With init they are being imported, but the same links that were
+    # listed as just imported are listed as skipped because they are invalid. At the very least I think we should improve this message,
+    # because it makes this command a little more confusing.
 
     # Links in invalid/duplicate data dirs
     invalid_folders = {
-        folder: link
-        for folder, link in get_invalid_folders(all_links.values(), out_dir=out_dir).items()
+        folder: snapshot
+        for folder, snapshot in get_invalid_folders(all_snapshots, out_dir=out_dir).items()
     }
     if invalid_folders:
-        print('    {lightyellow}! Skipped adding {} invalid link data directories.{reset}'.format(len(invalid_folders), **ANSI))
-        print('        X ' + '\n        X '.join(f'{folder} {link}' for folder, link in invalid_folders.items()))
+        print('    {lightyellow}! Skipped adding {} invalid snapshot data directories.{reset}'.format(len(invalid_folders), **ANSI))
+        print('        X ' + '\n        X '.join(f'{folder} {snapshot}' for folder, snapshot in invalid_folders.items()))
         print()
         print('    {lightred}Hint:{reset} For more information about the link data directories that were skipped, run:'.format(**ANSI))
         print('        archivebox status')
         print('        archivebox list --status=invalid')
 
 
-    write_main_index(list(all_links.values()), out_dir=out_dir)
+    write_main_index(list(pending_snapshots.values()), out_dir=out_dir)
 
     print('\n{green}------------------------------------------------------------------{reset}'.format(**ANSI))
     if existing_index:
         print('{green}[√] Done. Verified and updated the existing ArchiveBox collection.{reset}'.format(**ANSI))
     else:
-        print('{green}[√] Done. A new ArchiveBox collection was initialized ({} links).{reset}'.format(len(all_links), **ANSI))
+        print('{green}[√] Done. A new ArchiveBox collection was initialized ({} snapshots).{reset}'.format(len(all_snapshots), **ANSI))
     print()
     print('    {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI))
     print('        archivebox server  # then visit http://127.0.0.1:8000')
@@ -391,9 +404,18 @@ def init(force: bool=False, out_dir: str=OUTPUT_DIR) -> None:
     print('    For more usage and examples, run:')
     print('        archivebox help')
 
+    json_index = Path(out_dir) / JSON_INDEX_FILENAME
+    html_index = Path(out_dir) / HTML_INDEX_FILENAME
+    index_name = f"{date.today()}_index_old"
+    if json_index.exists():
+        json_index.rename(f"{index_name}.json")
+    if html_index.exists():
+        html_index.rename(f"{index_name}.html")
+
+
 
 @enforce_types
-def status(out_dir: str=OUTPUT_DIR) -> None:
+def status(out_dir: Path=OUTPUT_DIR) -> None:
     """Print out some info and statistics about the archive collection"""
 
     check_data_folder(out_dir=out_dir)
@@ -409,21 +431,11 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
     print(f'    Index size: {size} across {num_files} files')
     print()
 
-    links = list(load_main_index(out_dir=out_dir))
-    num_json_links = len(links)
-    num_sql_links = sum(1 for link in parse_sql_main_index(out_dir=out_dir))
-    num_html_links = sum(1 for url in parse_html_main_index(out_dir=out_dir))
-    num_link_details = sum(1 for link in parse_json_links_details(out_dir=out_dir))
-    print(f'    > JSON Main Index: {num_json_links} links'.ljust(36),  f'(found in {JSON_INDEX_FILENAME})')
-    print(f'    > SQL Main Index: {num_sql_links} links'.ljust(36), f'(found in {SQL_INDEX_FILENAME})')
-    print(f'    > HTML Main Index: {num_html_links} links'.ljust(36), f'(found in {HTML_INDEX_FILENAME})')
-    print(f'    > JSON Link Details: {num_link_details} links'.ljust(36), f'(found in {ARCHIVE_DIR_NAME}/*/index.json)')
-
-    if num_html_links != len(links) or num_sql_links != len(links):
-        print()
-        print('    {lightred}Hint:{reset} You can fix index count differences automatically by running:'.format(**ANSI))
-        print('        archivebox init')
-    
+    snapshots = load_main_index(out_dir=out_dir)
+    num_sql_snapshots = snapshots.count()
+    num_snapshot_details = sum(1 for snapshot in parse_json_snapshot_details(out_dir=out_dir))
+    print(f'    > SQL Main Index: {num_sql_snapshots} snapshots'.ljust(36), f'(found in {SQL_INDEX_FILENAME})')
+    print(f'    > JSON Link Details: {num_snapshot_details} snapshots'.ljust(36), f'(found in {ARCHIVE_DIR_NAME}/*/index.json)')
     print()
     print('{green}[*] Scanning archive data directories...{reset}'.format(**ANSI))
     print(ANSI['lightyellow'], f'   {ARCHIVE_DIR}/*', ANSI['reset'])
@@ -431,23 +443,23 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
     size = printable_filesize(num_bytes)
     print(f'    Size: {size} across {num_files} files in {num_dirs} directories')
     print(ANSI['black'])
-    num_indexed = len(get_indexed_folders(links, out_dir=out_dir))
-    num_archived = len(get_archived_folders(links, out_dir=out_dir))
-    num_unarchived = len(get_unarchived_folders(links, out_dir=out_dir))
+    num_indexed = len(get_indexed_folders(snapshots, out_dir=out_dir))
+    num_archived = len(get_archived_folders(snapshots, out_dir=out_dir))
+    num_unarchived = len(get_unarchived_folders(snapshots, out_dir=out_dir))
     print(f'    > indexed: {num_indexed}'.ljust(36), f'({get_indexed_folders.__doc__})')
     print(f'      > archived: {num_archived}'.ljust(36), f'({get_archived_folders.__doc__})')
     print(f'      > unarchived: {num_unarchived}'.ljust(36), f'({get_unarchived_folders.__doc__})')
     
-    num_present = len(get_present_folders(links, out_dir=out_dir))
-    num_valid = len(get_valid_folders(links, out_dir=out_dir))
+    num_present = len(get_present_folders(snapshots, out_dir=out_dir))
+    num_valid = len(get_valid_folders(snapshots, out_dir=out_dir))
     print()
     print(f'    > present: {num_present}'.ljust(36), f'({get_present_folders.__doc__})')
     print(f'      > valid: {num_valid}'.ljust(36), f'({get_valid_folders.__doc__})')
     
-    duplicate = get_duplicate_folders(links, out_dir=out_dir)
-    orphaned = get_orphaned_folders(links, out_dir=out_dir)
-    corrupted = get_corrupted_folders(links, out_dir=out_dir)
-    unrecognized = get_unrecognized_folders(links, out_dir=out_dir)
+    duplicate = get_duplicate_folders(snapshots, out_dir=out_dir)
+    orphaned = get_orphaned_folders(snapshots, out_dir=out_dir)
+    corrupted = get_corrupted_folders(snapshots, out_dir=out_dir)
+    unrecognized = get_unrecognized_folders(snapshots, out_dir=out_dir)
     num_invalid = len({**duplicate, **orphaned, **corrupted, **unrecognized})
     print(f'      > invalid: {num_invalid}'.ljust(36), f'({get_invalid_folders.__doc__})')
     print(f'        > duplicate: {len(duplicate)}'.ljust(36), f'({get_duplicate_folders.__doc__})')
@@ -458,7 +470,7 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
     print(ANSI['reset'])
 
     if num_indexed:
-        print('    {lightred}Hint:{reset} You can list link data directories by status like so:'.format(**ANSI))
+        print('    {lightred}Hint:{reset} You can list snapshot data directories by status like so:'.format(**ANSI))
         print('        archivebox list --status=<status>  (e.g. indexed, corrupted, archived, etc.)')
 
     if orphaned:
@@ -478,7 +490,8 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
     if last_login:
         print(f'    Last UI login: {last_login.username} @ {str(last_login.last_login)[:16]}')
     last_updated = Snapshot.objects.order_by('updated').last()
-    print(f'    Last changes: {str(last_updated.updated)[:16]}')
+    if last_updated:
+        print(f'    Last changes: {str(last_updated.updated)[:16]}')
 
     if not users:
         print()
@@ -486,7 +499,7 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
         print('        archivebox manage createsuperuser')
 
     print()
-    for snapshot in Snapshot.objects.order_by('-updated')[:10]:
+    for snapshot in snapshots.order_by('-updated')[:10]:
         if not snapshot.updated:
             continue
         print(
@@ -502,21 +515,24 @@ def status(out_dir: str=OUTPUT_DIR) -> None:
 
 
 @enforce_types
-def oneshot(url: str, out_dir: str=OUTPUT_DIR):
+def oneshot(url: str, extractors: str="", out_dir: Path=OUTPUT_DIR):
     """
     Create a single URL archive folder with an index.json and index.html, and all the archive method outputs.
     You can run this to archive single pages without needing to create a whole collection with archivebox init.
     """
-    oneshot_link, _ = parse_links_memory([url])
-    if len(oneshot_link) > 1:
+    oneshot_snapshots, _ = parse_snapshots_memory([url])
+    if len(oneshot_snapshots) > 1:
         stderr(
                 '[X] You should pass a single url to the oneshot command',
                 color='red'
             )
         raise SystemExit(2)
-    methods = ignore_methods(['title'])
-    archive_link(oneshot_link[0], out_dir=out_dir, methods=methods, skip_index=True)
-    return oneshot_link
+
+    methods = extractors.split(",") if extractors else ignore_methods(['title'])
+    snapshot = oneshot_snapshots[0]
+    snapshot.save() # Oneshot uses an in-memory database, so this is safe
+    archive_snapshot(snapshot, out_dir=out_dir, methods=methods)
+    return snapshot
 
 @enforce_types
 def add(urls: Union[str, List[str]],
@@ -525,20 +541,23 @@ def add(urls: Union[str, List[str]],
         index_only: bool=False,
         overwrite: bool=False,
         init: bool=False,
-        out_dir: str=OUTPUT_DIR) -> List[Link]:
+        extractors: str="",
+        out_dir: Path=OUTPUT_DIR) -> List[Link]:
     """Add a new URL or list of URLs to your archive"""
+    from core.models import Snapshot
 
     assert depth in (0, 1), 'Depth must be 0 or 1 (depth >1 is not supported yet)'
 
+    extractors = extractors.split(",") if extractors else []
+
     if init:
         run_subcommand('init', stdin=None, pwd=out_dir)
 
     # Load list of links from the existing index
     check_data_folder(out_dir=out_dir)
     check_dependencies()
-    all_links: List[Link] = []
-    new_links: List[Link] = []
-    all_links = load_main_index(out_dir=out_dir)
+    new_snapshots: List[Snapshot] = []
+    all_snapshots = load_main_index(out_dir=out_dir)
 
     log_importing_started(urls=urls, depth=depth, index_only=index_only)
     if isinstance(urls, str):
@@ -548,54 +567,57 @@ def add(urls: Union[str, List[str]],
         # save verbatim args to sources
         write_ahead_log = save_text_as_source('\n'.join(urls), filename='{ts}-import.txt', out_dir=out_dir)
     
-    new_links += parse_links_from_source(write_ahead_log, root_url=None)
+    new_snapshots += parse_snapshots_from_source(write_ahead_log, root_url=None)
 
     # If we're going one level deeper, download each link and look for more links
-    new_links_depth = []
-    if new_links and depth == 1:
-        log_crawl_started(new_links)
-        for new_link in new_links:
-            downloaded_file = save_file_as_source(new_link.url, filename=f'{new_link.timestamp}-crawl-{new_link.domain}.txt', out_dir=out_dir)
-            new_links_depth += parse_links_from_source(downloaded_file, root_url=new_link.url)
+    new_snapshots_depth = []
+    if new_snapshots and depth == 1:
+        log_crawl_started(new_snapshots)
+        for new_snapshot in new_snapshots:
+            # TODO: Check if we need to add domain to the Snapshot model
+            downloaded_file = save_file_as_source(new_snapshot.url, filename=f'{new_snapshot.timestamp}-crawl-{new_snapshot.domain}.txt', out_dir=out_dir)
+            new_snapshots_depth += parse_snapshots_from_source(downloaded_file, root_url=new_snapshot.url)
 
-    imported_links = list({link.url: link for link in (new_links + new_links_depth)}.values())
-    all_links, new_links = dedupe_links(all_links, imported_links)
-    write_main_index(links=all_links, out_dir=out_dir, finished=not new_links)
+    imported_snapshots = [Snapshot(url=snapshot.url) for snapshot in new_snapshots + new_snapshots_depth]
+    new_snapshots = filter_new_urls(all_snapshots, imported_snapshots)
+
+    write_main_index(snapshots=new_snapshots, out_dir=out_dir)
+    all_snapshots = load_main_index(out_dir=out_dir)
 
     if index_only:
-        return all_links
+        return all_snapshots
 
     # Run the archive methods for each link
+    archive_kwargs = {
+        "out_dir": out_dir,
+    }
+    if extractors:
+        archive_kwargs["methods"] = extractors
     if update_all:
-        archive_links(all_links, overwrite=overwrite, out_dir=out_dir)
+        archive_snapshots(all_snapshots, overwrite=overwrite, **archive_kwargs)
     elif overwrite:
-        archive_links(imported_links, overwrite=True, out_dir=out_dir)
-    elif new_links:
-        archive_links(new_links, overwrite=False, out_dir=out_dir)
-    else:
-        # nothing was updated, don't bother re-saving the index
-        return all_links
+        imported_snapshots = Snapshot.objects.filter(url__in=[imported_snapshot.url for imported_snapshot in imported_snapshots])
+        archive_snapshots(imported_snapshots, overwrite=True, **archive_kwargs)
+    elif new_snapshots:
+        archive_snapshots(new_snapshots, overwrite=False, **archive_kwargs)
 
-    # Step 4: Re-write links index with updated titles, icons, and resources
-    all_links = load_main_index(out_dir=out_dir)
-    write_main_index(links=list(all_links), out_dir=out_dir, finished=True)
-    return all_links
+    return all_snapshots
 
 @enforce_types
 def remove(filter_str: Optional[str]=None,
            filter_patterns: Optional[List[str]]=None,
            filter_type: str='exact',
-           links: Optional[List[Link]]=None,
+           snapshots: Optional[QuerySet]=None,
            after: Optional[float]=None,
            before: Optional[float]=None,
            yes: bool=False,
            delete: bool=False,
-           out_dir: str=OUTPUT_DIR) -> List[Link]:
+           out_dir: Path=OUTPUT_DIR) -> List[Link]:
     """Remove the specified URLs from the archive"""
     
     check_data_folder(out_dir=out_dir)
 
-    if links is None:
+    if snapshots is None:
         if filter_str and filter_patterns:
             stderr(
                 '[X] You should pass either a pattern as an argument, '
@@ -611,60 +633,53 @@ def remove(filter_str: Optional[str]=None,
             )
             stderr()
             hint(('To remove all urls you can run:',
-                  'archivebox remove --filter-type=regex ".*"'))
+                'archivebox remove --filter-type=regex ".*"'))
             stderr()
             raise SystemExit(2)
         elif filter_str:
             filter_patterns = [ptn.strip() for ptn in filter_str.split('\n')]
 
-        log_list_started(filter_patterns, filter_type)
-        timer = TimedProgress(360, prefix='      ')
-        try:
-            links = list(list_links(
-                filter_patterns=filter_patterns,
-                filter_type=filter_type,
-                after=after,
-                before=before,
-            ))
-        finally:
-            timer.end()
+    list_kwargs = {
+        "filter_patterns": filter_patterns,
+        "filter_type": filter_type,
+        "after": after,
+        "before": before,
+    }
+    if snapshots:
+        list_kwargs["snapshots"] = snapshots
+
+    log_list_started(filter_patterns, filter_type)
+    timer = TimedProgress(360, prefix='      ')
+    try:
+        snapshots = list_snapshots(**list_kwargs)
+    finally:
+        timer.end()
 
 
-    if not len(links):
+    if not snapshots.exists():
         log_removal_finished(0, 0)
         raise SystemExit(1)
 
 
-    log_list_finished(links)
-    log_removal_started(links, yes=yes, delete=delete)
+    log_list_finished(snapshots)
+    log_removal_started(snapshots, yes=yes, delete=delete)
 
     timer = TimedProgress(360, prefix='      ')
     try:
-        to_keep = []
-        to_delete = []
-        all_links = load_main_index(out_dir=out_dir)
-        for link in all_links:
-            should_remove = (
-                (after is not None and float(link.timestamp) < after)
-                or (before is not None and float(link.timestamp) > before)
-                or link_matches_filter(link, filter_patterns or [], filter_type)
-                or link in links
-            )
-            if should_remove:
-                to_delete.append(link)
-
-                if delete:
-                    shutil.rmtree(link.link_dir, ignore_errors=True)
-            else:
-                to_keep.append(link)
+        for snapshot in snapshots:
+            if delete:
+                shutil.rmtree(snapshot.snapshot_dir, ignore_errors=True)
     finally:
         timer.end()
 
-    remove_from_sql_main_index(links=to_delete, out_dir=out_dir)
-    write_main_index(links=to_keep, out_dir=out_dir, finished=True)
-    log_removal_finished(len(all_links), len(to_keep))
+    to_remove = snapshots.count()
+    all_snapshots = load_main_index(out_dir=out_dir).count()
+
+    flush_search_index(snapshots=snapshots)
+    remove_from_sql_main_index(snapshots=snapshots, out_dir=out_dir)
+    log_removal_finished(all_snapshots, to_remove)
     
-    return to_keep
+    return all_snapshots
 
 @enforce_types
 def update(resume: Optional[float]=None,
@@ -677,56 +692,61 @@ def update(resume: Optional[float]=None,
            status: Optional[str]=None,
            after: Optional[str]=None,
            before: Optional[str]=None,
-           out_dir: str=OUTPUT_DIR) -> List[Link]:
+           extractors: str="",
+           out_dir: Path=OUTPUT_DIR) -> List[Link]:
     """Import any new links from subscriptions and retry any previously failed/skipped links"""
+    from core.models import Snapshot
 
     check_data_folder(out_dir=out_dir)
     check_dependencies()
+    new_snapshots: List[Snapshot] = [] # TODO: Remove input argument: only_new
 
-    # Step 1: Load list of links from the existing index
-    #         merge in and dedupe new links from import_path
-    all_links: List[Link] = []
-    new_links: List[Link] = []
-    all_links = load_main_index(out_dir=out_dir)
+    extractors = extractors.split(",") if extractors else []
 
-    # Step 2: Write updated index with deduped old and new links back to disk
-    write_main_index(links=list(all_links), out_dir=out_dir)
-
-    # Step 3: Filter for selected_links
-    matching_links = list_links(
+    # Step 1: Filter for selected_links
+    matching_snapshots = list_snapshots(
         filter_patterns=filter_patterns,
         filter_type=filter_type,
         before=before,
         after=after,
     )
+
     matching_folders = list_folders(
-        links=list(matching_links),
+        snapshots=matching_snapshots,
         status=status,
         out_dir=out_dir,
     )
-    all_links = [link for link in matching_folders.values() if link]
+    all_snapshots = [snapshot for snapshot in matching_folders.values()]
 
     if index_only:
-        return all_links
+        for snapshot in all_snapshots:
+            write_snapshot_details(snapshot, out_dir=out_dir, skip_sql_index=True)
+        index_snapshots(all_snapshots, out_dir=out_dir)
+        return all_snapshots
         
-    # Step 3: Run the archive methods for each link
-    to_archive = new_links if only_new else all_links
+    # Step 2: Run the archive methods for each link
+    to_archive = new_snapshots if only_new else all_snapshots
     if resume:
         to_archive = [
-            link for link in to_archive
-            if link.timestamp >= str(resume)
+            snapshot for snapshot in to_archive
+            if snapshot.timestamp >= str(resume)
         ]
         if not to_archive:
             stderr('')
             stderr(f'[√] Nothing found to resume after {resume}', color='green')
-            return all_links
+            return all_snapshots
 
-    archive_links(to_archive, overwrite=overwrite, out_dir=out_dir)
+    archive_kwargs = {
+        "out_dir": out_dir,
+    }
+    if extractors:
+        archive_kwargs["methods"] = extractors
+
+    archive_snapshots(to_archive, overwrite=overwrite, **archive_kwargs)
 
     # Step 4: Re-write links index with updated titles, icons, and resources
-    all_links = load_main_index(out_dir=out_dir)
-    write_main_index(links=list(all_links), out_dir=out_dir, finished=True)
-    return all_links
+    all_snapshots = load_main_index(out_dir=out_dir)
+    return all_snapshots
 
 @enforce_types
 def list_all(filter_patterns_str: Optional[str]=None,
@@ -738,7 +758,9 @@ def list_all(filter_patterns_str: Optional[str]=None,
              sort: Optional[str]=None,
              csv: Optional[str]=None,
              json: bool=False,
-             out_dir: str=OUTPUT_DIR) -> Iterable[Link]:
+             html: bool=False,
+             with_headers: bool=False,
+             out_dir: Path=OUTPUT_DIR) -> Iterable[Link]:
     """List, filter, and export information about archive entries"""
     
     check_data_folder(out_dir=out_dir)
@@ -753,8 +775,7 @@ def list_all(filter_patterns_str: Optional[str]=None,
     elif filter_patterns_str:
         filter_patterns = filter_patterns_str.split('\n')
 
-
-    links = list_links(
+    snapshots = list_snapshots(
         filter_patterns=filter_patterns,
         filter_type=filter_type,
         before=before,
@@ -762,72 +783,73 @@ def list_all(filter_patterns_str: Optional[str]=None,
     )
 
     if sort:
-        links = sorted(links, key=lambda link: getattr(link, sort))
+        snapshots = snapshots.order_by(sort)
 
     folders = list_folders(
-        links=list(links),
+        snapshots=snapshots,
         status=status,
         out_dir=out_dir,
     )
-    
-    print(printable_folders(folders, json=json, csv=csv))
+
+    if json: 
+        output = generate_json_index_from_snapshots(folders.values(), with_headers)
+    elif html:
+        output = generate_index_from_snapshots(folders.values(), with_headers)
+    elif csv:
+        output = snapshots_to_csv(folders.values(), cols=csv.split(','), header=with_headers)
+    else:
+        output = printable_folders(folders, with_headers=with_headers)
+    print(output)
     return folders
 
 
 @enforce_types
-def list_links(filter_patterns: Optional[List[str]]=None,
+def list_snapshots(snapshots: Optional[QuerySet]=None,
+               filter_patterns: Optional[List[str]]=None,
                filter_type: str='exact',
                after: Optional[float]=None,
                before: Optional[float]=None,
-               out_dir: str=OUTPUT_DIR) -> Iterable[Link]:
+               out_dir: Path=OUTPUT_DIR) -> Iterable[Link]:
     
     check_data_folder(out_dir=out_dir)
 
-    all_links = load_main_index(out_dir=out_dir)
+    if snapshots:
+        all_snapshots = snapshots
+    else:
+        all_snapshots = load_main_index(out_dir=out_dir)
 
-    for link in all_links:
-        if after is not None and float(link.timestamp) < after:
-            continue
-        if before is not None and float(link.timestamp) > before:
-            continue
-        
-        if filter_patterns:
-            if link_matches_filter(link, filter_patterns, filter_type):
-                yield link
-        else:
-            yield link
+    if after is not None:
+        all_snapshots = all_snapshots.filter(timestamp__lt=after)
+    if before is not None:
+        all_snapshots = all_snapshots.filter(timestamp__gt=before)
+    if filter_patterns:
+        all_snapshots = snapshot_filter(all_snapshots, filter_patterns, filter_type)
+    return all_snapshots
 
 @enforce_types
-def list_folders(links: List[Link],
+def list_folders(snapshots: List[Model],
                  status: str,
-                 out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+                 out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Model]]:
     
     check_data_folder(out_dir=out_dir)
 
-    if status == 'indexed':
-        return get_indexed_folders(links, out_dir=out_dir)
-    elif status == 'archived':
-        return get_archived_folders(links, out_dir=out_dir)
-    elif status == 'unarchived':
-        return get_unarchived_folders(links, out_dir=out_dir)
+    STATUS_FUNCTIONS = {
+        "indexed": get_indexed_folders,
+        "archived": get_archived_folders,
+        "unarchived": get_unarchived_folders,
+        "present": get_present_folders,
+        "valid": get_valid_folders,
+        "invalid": get_invalid_folders,
+        "duplicate": get_duplicate_folders,
+        "orphaned": get_orphaned_folders,
+        "corrupted": get_corrupted_folders,
+        "unrecognized": get_unrecognized_folders,
+    }
 
-    elif status == 'present':
-        return get_present_folders(links, out_dir=out_dir)
-    elif status == 'valid':
-        return get_valid_folders(links, out_dir=out_dir)
-    elif status == 'invalid':
-        return get_invalid_folders(links, out_dir=out_dir)
-
-    elif status == 'duplicate':
-        return get_duplicate_folders(links, out_dir=out_dir)
-    elif status == 'orphaned':
-        return get_orphaned_folders(links, out_dir=out_dir)
-    elif status == 'corrupted':
-        return get_corrupted_folders(links, out_dir=out_dir)
-    elif status == 'unrecognized':
-        return get_unrecognized_folders(links, out_dir=out_dir)
-
-    raise ValueError('Status not recognized.')
+    try:
+        return STATUS_FUNCTIONS[status](snapshots, out_dir=out_dir)
+    except KeyError:
+        raise ValueError('Status not recognized.')
 
 
 @enforce_types
@@ -836,7 +858,7 @@ def config(config_options_str: Optional[str]=None,
            get: bool=False,
            set: bool=False,
            reset: bool=False,
-           out_dir: str=OUTPUT_DIR) -> None:
+           out_dir: Path=OUTPUT_DIR) -> None:
     """Get and set your ArchiveBox project configuration values"""
 
     check_data_folder(out_dir=out_dir)
@@ -882,7 +904,7 @@ def config(config_options_str: Optional[str]=None,
                 stderr(f'    {line}')
                 raise SystemExit(2)
 
-            raw_key, val = line.split('=')
+            raw_key, val = line.split('=', 1)
             raw_key = raw_key.upper().strip()
             key = get_real_name(raw_key)
             if key != raw_key:
@@ -936,12 +958,12 @@ def schedule(add: bool=False,
              every: Optional[str]=None,
              depth: int=0,
              import_path: Optional[str]=None,
-             out_dir: str=OUTPUT_DIR):
+             out_dir: Path=OUTPUT_DIR):
     """Set ArchiveBox to regularly import URLs at specific times using cron"""
     
     check_data_folder(out_dir=out_dir)
 
-    os.makedirs(os.path.join(out_dir, LOGS_DIR_NAME), exist_ok=True)
+    (Path(out_dir) / LOGS_DIR_NAME).mkdir(exist_ok=True)
 
     cron = CronTab(user=True)
     cron = dedupe_cron_jobs(cron)
@@ -955,7 +977,7 @@ def schedule(add: bool=False,
 
     if every or add:
         every = every or 'day'
-        quoted = lambda s: f'"{s}"' if s and ' ' in s else s
+        quoted = lambda s: f'"{s}"' if s and ' ' in str(s) else str(s)
         cmd = [
             'cd',
             quoted(out_dir),
@@ -963,7 +985,7 @@ def schedule(add: bool=False,
             quoted(ARCHIVEBOX_BINARY),
             *(['add', f'--depth={depth}', f'"{import_path}"'] if import_path else ['update']),
             '>',
-            quoted(os.path.join(LOGS_DIR, 'archivebox.log')),
+            quoted(Path(LOGS_DIR) / 'archivebox.log'),
             '2>&1',
 
         ]
@@ -1047,7 +1069,8 @@ def server(runserver_args: Optional[List[str]]=None,
            reload: bool=False,
            debug: bool=False,
            init: bool=False,
-           out_dir: str=OUTPUT_DIR) -> None:
+           createsuperuser: bool=False,
+           out_dir: Path=OUTPUT_DIR) -> None:
     """Run the ArchiveBox HTTP server"""
 
     runserver_args = runserver_args or []
@@ -1055,13 +1078,15 @@ def server(runserver_args: Optional[List[str]]=None,
     if init:
         run_subcommand('init', stdin=None, pwd=out_dir)
 
+    if createsuperuser:
+        run_subcommand('manage', subcommand_args=['createsuperuser'], pwd=out_dir)
+
     # setup config for django runserver
     from . import config
     config.SHOW_PROGRESS = False
     config.DEBUG = config.DEBUG or debug
 
     check_data_folder(out_dir=out_dir)
-    setup_django(out_dir)
 
     from django.core.management import call_command
     from django.contrib.auth.models import User
@@ -1094,11 +1119,10 @@ def server(runserver_args: Optional[List[str]]=None,
 
 
 @enforce_types
-def manage(args: Optional[List[str]]=None, out_dir: str=OUTPUT_DIR) -> None:
+def manage(args: Optional[List[str]]=None, out_dir: Path=OUTPUT_DIR) -> None:
     """Run an ArchiveBox Django management command"""
 
     check_data_folder(out_dir=out_dir)
-    setup_django(out_dir)
     from django.core.management import execute_from_command_line
 
     if (args and "createsuperuser" in args) and (IN_DOCKER and not IS_TTY):
@@ -1110,12 +1134,11 @@ def manage(args: Optional[List[str]]=None, out_dir: str=OUTPUT_DIR) -> None:
 
 
 @enforce_types
-def shell(out_dir: str=OUTPUT_DIR) -> None:
+def shell(out_dir: Path=OUTPUT_DIR) -> None:
     """Enter an interactive ArchiveBox Django shell"""
 
     check_data_folder(out_dir=out_dir)
 
-    setup_django(OUTPUT_DIR)
     from django.core.management import call_command
     call_command("shell_plus")
 

archivebox/manage.py

@@ -8,10 +8,9 @@ if __name__ == '__main__':
     # (e.g. makemigrations), you can comment out this check temporarily
 
     if not ('makemigrations' in sys.argv or 'migrate' in sys.argv):
-        print("[X] Don't run ./manage.py directly, use the archivebox CLI instead e.g.:")
-        print('    archivebox manage createsuperuser')
+        print("[X] Don't run ./manage.py directly (unless you are a developer running makemigrations):")
         print()
-        print('    Hint: Use these archivebox commands instead of the ./manage.py equivalents:')
+        print('    Hint: Use these archivebox CLI commands instead of the ./manage.py equivalents:')
         print('        archivebox init          (migrates the databse to latest version)')
         print('        archivebox server        (runs the Django web server)')
         print('        archivebox shell         (opens an iPython Django shell with all models imported)')

archivebox/package.json

@@ -0,0 +1 @@
+../package.json

archivebox/parsers/__init__.py

@@ -8,11 +8,13 @@ For examples of supported import formats see tests/.
 __package__ = 'archivebox.parsers'
 
 import re
-import os
 from io import StringIO
 
 from typing import IO, Tuple, List, Optional
 from datetime import datetime
+from pathlib import Path 
+
+from django.db.models import Model
 
 from ..system import atomic_write
 from ..config import (
@@ -32,7 +34,9 @@ from ..index.schema import Link
 from ..logging_util import TimedProgress, log_source_saved
 
 from .pocket_html import parse_pocket_html_export
+from .pocket_api import parse_pocket_api_export
 from .pinboard_rss import parse_pinboard_rss_export
+from .wallabag_atom import parse_wallabag_atom_export
 from .shaarli_rss import parse_shaarli_rss_export
 from .medium_rss import parse_medium_rss_export
 from .netscape_html import parse_netscape_html_export
@@ -43,6 +47,8 @@ from .generic_txt import parse_generic_txt_export
 
 PARSERS = (
     # Specialized parsers
+    ('Pocket API', parse_pocket_api_export),
+    ('Wallabag ATOM', parse_wallabag_atom_export),
     ('Pocket HTML', parse_pocket_html_export),
     ('Pinboard RSS', parse_pinboard_rss_export),
     ('Shaarli RSS', parse_shaarli_rss_export),
@@ -60,7 +66,7 @@ PARSERS = (
 
 
 @enforce_types
-def parse_links_memory(urls: List[str], root_url: Optional[str]=None):
+def parse_snapshots_memory(urls: List[str], root_url: Optional[str]=None):
     """
     parse a list of URLS without touching the filesystem
     """
@@ -71,16 +77,16 @@ def parse_links_memory(urls: List[str], root_url: Optional[str]=None):
     file = StringIO()
     file.writelines(urls)
     file.name = "io_string"
-    links, parser = run_parser_functions(file, timer, root_url=root_url)
+    snapshots, parser = run_parser_functions(file, timer, root_url=root_url)
     timer.end()
 
     if parser is None:
         return [], 'Failed to parse'
-    return links, parser
+    return snapshots, parser
     
 
 @enforce_types
-def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Link], str]:
+def parse_snapshots(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Model], str]:
     """parse a list of URLs with their metadata from an 
        RSS feed, bookmarks export, or text file
     """
@@ -89,27 +95,27 @@ def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Li
 
     timer = TimedProgress(TIMEOUT * 4)
     with open(source_file, 'r', encoding='utf-8') as file:
-        links, parser = run_parser_functions(file, timer, root_url=root_url)
+        snapshots, parser = run_parser_functions(file, timer, root_url=root_url)
 
     timer.end()
     if parser is None:
         return [], 'Failed to parse'
-    return links, parser
+    return snapshots, parser
 
 
-def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Link], Optional[str]]:
-    most_links: List[Link] = []
+def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Model], Optional[str]]:
+    most_snapshots: List[Model] = []
     best_parser_name = None
 
     for parser_name, parser_func in PARSERS:
         try:
-            parsed_links = list(parser_func(to_parse, root_url=root_url))
-            if not parsed_links:
+            parsed_snapshots = list(parser_func(to_parse, root_url=root_url))
+            if not parsed_snapshots:
                 raise Exception('no links found')
 
             # print(f'[√] Parser {parser_name} succeeded: {len(parsed_links)} links parsed')
-            if len(parsed_links) > len(most_links):
-                most_links = parsed_links
+            if len(parsed_snapshots) > len(most_snapshots):
+                most_snapshots = parsed_snapshots
                 best_parser_name = parser_name
                 
         except Exception as err:                                                # noqa
@@ -121,23 +127,23 @@ def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None)
             # raise
             pass
     timer.end()
-    return most_links, best_parser_name
+    return most_snapshots, best_parser_name
 
 
 @enforce_types
-def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: str=OUTPUT_DIR) -> str:
+def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: Path=OUTPUT_DIR) -> str:
     ts = str(datetime.now().timestamp()).split('.', 1)[0]
-    source_path = os.path.join(out_dir, SOURCES_DIR_NAME, filename.format(ts=ts))
+    source_path = str(out_dir / SOURCES_DIR_NAME / filename.format(ts=ts))
     atomic_write(source_path, raw_text)
     log_source_saved(source_file=source_path)
     return source_path
 
 
 @enforce_types
-def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: str=OUTPUT_DIR) -> str:
+def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: Path=OUTPUT_DIR) -> str:
     """download a given url's content into output/sources/domain-<timestamp>.txt"""
     ts = str(datetime.now().timestamp()).split('.', 1)[0]
-    source_path = os.path.join(OUTPUT_DIR, SOURCES_DIR_NAME, filename.format(basename=basename(path), ts=ts))
+    source_path = str(OUTPUT_DIR / SOURCES_DIR_NAME / filename.format(basename=basename(path), ts=ts))
 
     if any(path.startswith(s) for s in ('http://', 'https://', 'ftp://')):
         # Source is a URL that needs to be downloaded

archivebox/parsers/generic_html.py

@@ -6,7 +6,8 @@ import re
 from typing import IO, Iterable, Optional
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -29,8 +30,9 @@ class HrefParser(HTMLParser):
 
 
 @enforce_types
-def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Link]:
+def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Model]:
     """Parse Generic HTML for href tags and use only the url (support for title coming later)"""
+    from core.models import Snapshot
 
     html_file.seek(0)
     for line in html_file:
@@ -44,10 +46,10 @@ def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None,
                 url = urljoin(root_url, url)
             
             for archivable_url in re.findall(URL_REGEX, url):
-                yield Link(
+                yield Snapshot(
                     url=htmldecode(archivable_url),
                     timestamp=str(datetime.now().timestamp()),
                     title=None,
-                    tags=None,
-                    sources=[html_file.name],
+                    #tags=None,
+                    #sources=[html_file.name],
                 )

archivebox/parsers/generic_json.py

@@ -5,7 +5,8 @@ import json
 from typing import IO, Iterable
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -13,8 +14,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)"""
+    from core.models import Snapshot
 
     json_file.seek(0)
     links = json.load(json_file)
@@ -56,10 +58,10 @@ def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]:
             elif link.get('name'):
                 title = link['name'].strip()
 
-            yield Link(
+            yield Snapshot(
                 url=htmldecode(url),
                 timestamp=ts_str,
                 title=htmldecode(title) or None,
-                tags=htmldecode(link.get('tags')) or '',
-                sources=[json_file.name],
+                #tags=htmldecode(link.get('tags')) or '',
+                #sources=[json_file.name],
             )

archivebox/parsers/generic_rss.py

@@ -4,7 +4,8 @@ __package__ = 'archivebox.parsers'
 from typing import IO, Iterable
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -12,8 +13,9 @@ from ..util import (
 )
 
 @enforce_types
-def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse RSS XML-format files into links"""
+    from core.models import Snapshot
 
     rss_file.seek(0)
     items = rss_file.read().split('<item>')
@@ -40,10 +42,10 @@ def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
         time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z")
         title = str_between(get_row('title'), '<![CDATA[', ']]').strip()
 
-        yield Link(
+        yield Snapshot(
             url=htmldecode(url),
             timestamp=str(time.timestamp()),
             title=htmldecode(title) or None,
-            tags=None,
-            sources=[rss_file.name],
+            #tags=None,
+            #sources=[rss_file.name],
         )

archivebox/parsers/generic_txt.py

@@ -7,7 +7,8 @@ from typing import IO, Iterable
 from datetime import datetime
 from pathlib import Path
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -16,8 +17,10 @@ from ..util import (
 
 
 @enforce_types
-def parse_generic_txt_export(text_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_generic_txt_export(text_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse raw links from each line in a text file"""
+    # TODO: Check if we should add sources list to the database
+    from core.models import Snapshot
 
     text_file.seek(0)
     for line in text_file.readlines():
@@ -27,12 +30,12 @@ def parse_generic_txt_export(text_file: IO[str], **_kwargs) -> Iterable[Link]:
         # if the line is a local file path that resolves, then we can archive it
         try:
             if Path(line).exists():
-                yield Link(
+                yield Snapshot(
                     url=line,
                     timestamp=str(datetime.now().timestamp()),
                     title=None,
-                    tags=None,
-                    sources=[text_file.name],
+                    #tags=None,
+                    #sources=[text_file.name],
                 )
         except (OSError, PermissionError):
             # nvm, not a valid path...
@@ -40,22 +43,22 @@ def parse_generic_txt_export(text_file: IO[str], **_kwargs) -> Iterable[Link]:
 
         # otherwise look for anything that looks like a URL in the line
         for url in re.findall(URL_REGEX, line):
-            yield Link(
+            yield Snapshot(
                 url=htmldecode(url),
                 timestamp=str(datetime.now().timestamp()),
                 title=None,
-                tags=None,
-                sources=[text_file.name],
+                #tags=None,
+                #sources=[text_file.name],
             )
 
             # look inside the URL for any sub-urls, e.g. for archive.org links
             # https://web.archive.org/web/20200531203453/https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/
             # -> https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/
-            for url in re.findall(URL_REGEX, line[1:]):
-                yield Link(
-                    url=htmldecode(url),
+            for sub_url in re.findall(URL_REGEX, line[1:]):
+                yield Snapshot(
+                    url=htmldecode(sub_url),
                     timestamp=str(datetime.now().timestamp()),
                     title=None,
-                    tags=None,
-                    sources=[text_file.name],
+                    #tags=None,
+                    #sources=[text_file.name],
                 )

archivebox/parsers/medium_rss.py

@@ -4,9 +4,10 @@ __package__ = 'archivebox.parsers'
 from typing import IO, Iterable
 from datetime import datetime
 
+from django.db.models import Model
+
 from xml.etree import ElementTree
 
-from ..index.schema import Link
 from ..util import (
     htmldecode,
     enforce_types,
@@ -14,8 +15,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse Medium RSS feed files into links"""
+    from core.models import Snapshot
 
     rss_file.seek(0)
     root = ElementTree.parse(rss_file).getroot()
@@ -26,10 +28,10 @@ def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
         ts_str = item.find("pubDate").text                              # type: ignore
         time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z")    # type: ignore
         
-        yield Link(
+        yield Snapshot(
             url=htmldecode(url),
             timestamp=str(time.timestamp()),
             title=htmldecode(title) or None,
-            tags=None,
-            sources=[rss_file.name],
+            #tags=None,
+            #sources=[rss_file.name],
         )

archivebox/parsers/netscape_html.py

@@ -6,7 +6,8 @@ import re
 from typing import IO, Iterable
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -14,8 +15,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse netscape-format bookmarks export files (produced by all browsers)"""
+    from core.models import Snapshot
 
     html_file.seek(0)
     pattern = re.compile("<a href=\"(.+?)\" add_date=\"(\\d+)\"[^>]*>(.+)</a>", re.UNICODE | re.IGNORECASE)
@@ -29,11 +31,11 @@ def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
             time = datetime.fromtimestamp(float(match.group(2)))
             title = match.group(3).strip()
 
-            yield Link(
+            yield Snapshot(
                 url=htmldecode(url),
                 timestamp=str(time.timestamp()),
                 title=htmldecode(title) or None,
-                tags=None,
-                sources=[html_file.name],
+                #tags=None,
+                #sources=[html_file.name],
             )
 

archivebox/parsers/pinboard_rss.py

@@ -4,9 +4,10 @@ __package__ = 'archivebox.parsers'
 from typing import IO, Iterable
 from datetime import datetime
 
+from django.db.models import Model
+
 from xml.etree import ElementTree
 
-from ..index.schema import Link
 from ..util import (
     htmldecode,
     enforce_types,
@@ -14,8 +15,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse Pinboard RSS feed files into links"""
+    from core.models import Snapshot
 
     rss_file.seek(0)
     root = ElementTree.parse(rss_file).getroot()
@@ -38,10 +40,10 @@ def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
         else:
             time = datetime.now()
 
-        yield Link(
+        yield Snapshot(
             url=htmldecode(url),
             timestamp=str(time.timestamp()),
             title=htmldecode(title) or None,
-            tags=htmldecode(tags) or None,
-            sources=[rss_file.name],
+            #tags=htmldecode(tags) or None,
+            #sources=[rss_file.name],
         )

archivebox/parsers/pocket_api.py

@@ -0,0 +1,117 @@
+__package__ = 'archivebox.parsers'
+
+
+import re
+
+from typing import IO, Iterable, Optional
+from configparser import ConfigParser
+
+from pathlib import Path
+
+from django.db.models import Model
+
+from ..vendor.pocket import Pocket
+
+from ..util import enforce_types
+from ..system import atomic_write
+from ..config import (
+    SOURCES_DIR,
+    POCKET_CONSUMER_KEY,
+    POCKET_ACCESS_TOKENS,
+)
+
+
+COUNT_PER_PAGE = 500
+API_DB_PATH = Path(SOURCES_DIR) / 'pocket_api.db'
+
+# search for broken protocols that sometimes come from the Pocket API
+_BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))')
+
+
+def get_pocket_articles(api: Pocket, since=None, page=0):
+    body, headers = api.get(
+        state='archive',
+        sort='oldest',
+        since=since,
+        count=COUNT_PER_PAGE,
+        offset=page * COUNT_PER_PAGE,
+    )
+
+    articles = body['list'].values() if isinstance(body['list'], dict) else body['list']
+    returned_count = len(articles)
+
+    yield from articles
+
+    if returned_count == COUNT_PER_PAGE:
+        yield from get_pocket_articles(api, since=since, page=page + 1)
+    else:
+        api.last_since = body['since']
+
+
+def snapshot_from_article(article: dict, sources: list):
+    from core.models import Snapshot
+
+    url: str = article['resolved_url'] or article['given_url']
+    broken_protocol = _BROKEN_PROTOCOL_RE.match(url)
+    if broken_protocol:
+        url = url.replace(f'{broken_protocol.group(1)}:/', f'{broken_protocol.group(1)}://')
+    title = article['resolved_title'] or article['given_title'] or url
+
+    return Snapshot(
+        url=url,
+        timestamp=article['time_read'],
+        title=title,
+        #tags=article.get('tags'),
+        #sources=sources
+    )
+
+
+def write_since(username: str, since: str):
+    if not API_DB_PATH.exists():
+        atomic_write(API_DB_PATH, '')
+
+    since_file = ConfigParser()
+    since_file.optionxform = str
+    since_file.read(API_DB_PATH)
+
+    since_file[username] = {
+        'since': since
+    }
+
+    with open(API_DB_PATH, 'w+') as new:
+        since_file.write(new)
+
+
+def read_since(username: str) -> Optional[str]:
+    if not API_DB_PATH.exists():
+        atomic_write(API_DB_PATH, '')
+
+    config_file = ConfigParser()
+    config_file.optionxform = str
+    config_file.read(API_DB_PATH)
+
+    return config_file.get(username, 'since', fallback=None)
+
+
+@enforce_types
+def should_parse_as_pocket_api(text: str) -> bool:
+    return text.startswith('pocket://')
+
+
+@enforce_types
+def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Model]:
+    """Parse bookmarks from the Pocket API"""
+
+    input_buffer.seek(0)
+    pattern = re.compile(r"^pocket:\/\/(\w+)")
+    for line in input_buffer:
+        if should_parse_as_pocket_api(line):
+            
+            username = pattern.search(line).group(1)
+            api = Pocket(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKENS[username])
+            api.last_since = None
+    
+            for article in get_pocket_articles(api, since=read_since(username)):
+                yield snapshot_from_article(article, sources=[line])
+    
+            write_since(username, api.last_since)

archivebox/parsers/pocket_html.py

@@ -6,7 +6,8 @@ import re
 from typing import IO, Iterable
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -14,8 +15,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""
+    from core.models import Snapshot
 
     html_file.seek(0)
     pattern = re.compile("^\\s*<li><a href=\"(.+)\" time_added=\"(\\d+)\" tags=\"(.*)\">(.+)</a></li>", re.UNICODE)
@@ -29,10 +31,10 @@ def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
             tags = match.group(3)
             title = match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '')
             
-            yield Link(
+            yield Snapshot(
                 url=htmldecode(url),
                 timestamp=str(time.timestamp()),
                 title=htmldecode(title) or None,
-                tags=tags or '',
-                sources=[html_file.name],
+                #tags=tags or '',
+                #sources=[html_file.name],
             )

archivebox/parsers/shaarli_rss.py

@@ -4,7 +4,8 @@ __package__ = 'archivebox.parsers'
 from typing import IO, Iterable
 from datetime import datetime
 
-from ..index.schema import Link
+from django.db.models import Model
+
 from ..util import (
     htmldecode,
     enforce_types,
@@ -13,8 +14,9 @@ from ..util import (
 
 
 @enforce_types
-def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
+def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Model]:
     """Parse Shaarli-specific RSS XML-format files into links"""
+    from core.models import Snapshot
 
     rss_file.seek(0)
     entries = rss_file.read().split('<entry>')[1:]
@@ -41,10 +43,10 @@ def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
         ts_str = str_between(get_row('published'), '<published>', '</published>')
         time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
 
-        yield Link(
+        yield Snapshot(
             url=htmldecode(url),
             timestamp=str(time.timestamp()),
             title=htmldecode(title) or None,
-            tags=None,
-            sources=[rss_file.name],
+            #tags=None,
+            #sources=[rss_file.name],
         )

archivebox/parsers/wallabag_atom.py

@@ -0,0 +1,59 @@
+__package__ = 'archivebox.parsers'
+
+
+from typing import IO, Iterable
+from datetime import datetime
+
+from django.db.models import Model
+
+from ..util import (
+    htmldecode,
+    enforce_types,
+    str_between,
+)
+
+
+@enforce_types
+def parse_wallabag_atom_export(rss_file: IO[str], **_kwargs) -> Iterable[Model]:
+    """Parse Wallabag Atom files into links"""
+    from core.models import Snapshot
+
+    rss_file.seek(0)
+    entries = rss_file.read().split('<entry>')[1:]
+    for entry in entries:
+        # example entry:
+        # <entry>
+        #       <title><![CDATA[Orient Ray vs Mako: Is There Much Difference? - iknowwatches.com]]></title>
+        #       <link rel="alternate" type="text/html"
+        #              href="http://wallabag.drycat.fr/view/14041"/>
+        #       <link rel="via">https://iknowwatches.com/orient-ray-vs-mako/</link>
+        #       <id>wallabag:wallabag.drycat.fr:milosh:entry:14041</id>
+        #       <updated>2020-10-18T09:14:02+02:00</updated>
+        #       <published>2020-10-18T09:13:56+02:00</published>
+        #                   <category term="montres" label="montres" />
+        #                       <content type="html" xml:lang="en">
+        # </entry>
+
+        trailing_removed = entry.split('</entry>', 1)[0]
+        leading_removed = trailing_removed.strip()
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0]
+
+        title = str_between(get_row('title'), '<title><![CDATA[', ']]></title>').strip()
+        url = str_between(get_row('link rel="via"'), '<link rel="via">', '</link>')
+        ts_str = str_between(get_row('published'), '<published>', '</published>')
+        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
+        try:
+            tags = str_between(get_row('category'), 'label="', '" />')
+        except Exception:
+            tags = None
+
+        yield Snapshot(
+            url=htmldecode(url),
+            timestamp=str(time.timestamp()),
+            title=htmldecode(title) or None,
+            #tags=tags or '',
+            #sources=[rss_file.name],
+        )

archivebox/search/__init__.py

@@ -0,0 +1,106 @@
+from typing import List, Union
+from pathlib import Path
+from importlib import import_module
+
+from django.db.models import QuerySet, Model
+
+from archivebox.index.schema import Link
+from archivebox.util import enforce_types
+from archivebox.config import stderr, OUTPUT_DIR, USE_INDEXING_BACKEND, USE_SEARCHING_BACKEND, SEARCH_BACKEND_ENGINE
+
+from .utils import get_indexable_content, log_index_started
+
+def indexing_enabled():
+    return USE_INDEXING_BACKEND
+
+def search_backend_enabled():
+    return USE_SEARCHING_BACKEND
+
+def get_backend():
+    return f'search.backends.{SEARCH_BACKEND_ENGINE}'
+
+def import_backend():
+    backend_string = get_backend()
+    try:
+        backend = import_module(backend_string)
+    except Exception as err:
+        raise Exception("Could not load '%s' as a backend: %s" % (backend_string, err))
+    return backend
+
+@enforce_types
+def write_search_index(snapshot: Model, texts: Union[List[str], None]=None, out_dir: Path=OUTPUT_DIR, skip_text_index: bool=False) -> None:
+    if not indexing_enabled():
+        return
+
+    if not skip_text_index and texts:
+        from core.models import Snapshot
+
+        backend = import_backend()
+        try:
+            backend.index(snapshot_id=str(snapshot.id), texts=texts)
+        except Exception as err:
+            stderr()
+            stderr(
+                f'[X] The search backend threw an exception={err}:',
+                color='red',
+            )
+
+@enforce_types
+def query_search_index(query: str, out_dir: Path=OUTPUT_DIR) -> QuerySet:
+    from core.models import Snapshot
+
+    if search_backend_enabled():
+        backend = import_backend()
+        try:
+            snapshot_ids = backend.search(query)
+        except Exception as err:
+            stderr()
+            stderr(
+                    f'[X] The search backend threw an exception={err}:',
+                color='red',
+                )
+            raise
+        else:
+            # TODO preserve ordering from backend
+            qsearch = Snapshot.objects.filter(pk__in=snapshot_ids)
+            return qsearch
+    
+    return Snapshot.objects.none()
+
+@enforce_types
+def flush_search_index(snapshots: QuerySet):
+    if not indexing_enabled() or not snapshots:
+        return
+    backend = import_backend()
+    snapshot_ids=(str(pk) for pk in snapshots.values_list('pk',flat=True))
+    try:
+        backend.flush(snapshot_ids)
+    except Exception as err:
+        stderr()
+        stderr(
+            f'[X] The search backend threw an exception={err}:',
+        color='red',
+        )
+
+@enforce_types
+def index_snapshots(snapshots: Union[List[Model],None], out_dir: Path=OUTPUT_DIR):
+    if not snapshots:
+        return
+
+    from core.models import Snapshot, ArchiveResult
+
+    for snapshot in snapshots:
+        snap = Snapshot.objects.filter(url=snapshot.url).first()
+        if snap: 
+            results = ArchiveResult.objects.indexable().filter(snapshot=snap)
+            log_index_started(snapshot.url)
+            try:
+                texts = get_indexable_content(results)
+            except Exception as err:
+                stderr()
+                stderr(
+                    f'[X] An Exception ocurred reading the indexable content={err}:',
+                    color='red',
+                    ) 
+            else:
+                write_search_index(snapshot, texts, out_dir=out_dir)

archivebox/search/backends/ripgrep.py

@@ -0,0 +1,45 @@
+import re
+from subprocess import run, PIPE
+from typing import List, Generator
+
+from archivebox.config import ARCHIVE_DIR, RIPGREP_VERSION
+from archivebox.util import enforce_types
+
+RG_IGNORE_EXTENSIONS = ('css','js','orig','svg')
+
+RG_ADD_TYPE = '--type-add'
+RG_IGNORE_ARGUMENTS = f"ignore:*.{{{','.join(RG_IGNORE_EXTENSIONS)}}}"
+RG_DEFAULT_ARGUMENTS = "-ilTignore" # Case insensitive(i), matching files results(l)
+RG_REGEX_ARGUMENT = '-e'
+
+TIMESTAMP_REGEX = r'\/([\d]+\.[\d]+)\/'
+
+ts_regex =  re.compile(TIMESTAMP_REGEX)
+
+@enforce_types
+def index(snapshot_id: str, texts: List[str]):
+    return
+
+@enforce_types
+def flush(snapshot_ids: Generator[str, None, None]):
+    return
+
+@enforce_types
+def search(text: str) -> List[str]:
+    if not RIPGREP_VERSION:
+        raise Exception("ripgrep binary not found, install ripgrep to use this search backend")
+
+    from core.models import Snapshot
+
+    rg_cmd = ['rg', RG_ADD_TYPE, RG_IGNORE_ARGUMENTS, RG_DEFAULT_ARGUMENTS, RG_REGEX_ARGUMENT, text, str(ARCHIVE_DIR)]
+    rg = run(rg_cmd, stdout=PIPE, stderr=PIPE, timeout=60)
+    file_paths = [p.decode() for p in rg.stdout.splitlines()]
+    timestamps = set()
+    for path in file_paths:
+        ts = ts_regex.findall(path)
+        if ts:
+            timestamps.add(ts[0])
+    
+    snap_ids = [str(id) for id in Snapshot.objects.filter(timestamp__in=timestamps).values_list('pk', flat=True)]
+
+    return snap_ids

archivebox/search/backends/sonic.py

@@ -0,0 +1,28 @@
+from typing import List, Generator
+
+from sonic import IngestClient, SearchClient
+
+from archivebox.util import enforce_types
+from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION
+
+MAX_SONIC_TEXT_LENGTH = 2000
+
+@enforce_types
+def index(snapshot_id: str, texts: List[str]):
+    with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl:
+        for text in texts:
+            chunks = [text[i:i+MAX_SONIC_TEXT_LENGTH] for i in range(0, len(text), MAX_SONIC_TEXT_LENGTH)]
+            for chunk in chunks:
+                ingestcl.push(SONIC_COLLECTION, SONIC_BUCKET, snapshot_id, str(chunk))
+
+@enforce_types
+def search(text: str) -> List[str]:
+    with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl:
+        snap_ids = querycl.query(SONIC_COLLECTION, SONIC_BUCKET, text)
+    return snap_ids
+
+@enforce_types
+def flush(snapshot_ids: Generator[str, None, None]):
+    with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl:
+        for id in snapshot_ids:
+            ingestcl.flush_object(SONIC_COLLECTION, SONIC_BUCKET, str(id))

archivebox/search/utils.py

@@ -0,0 +1,45 @@
+from django.db.models import QuerySet
+
+from archivebox.util import enforce_types
+from archivebox.config import ANSI
+
+def log_index_started(url):
+    print('{green}[*] Indexing url: {} in the search index {reset}'.format(url, **ANSI))
+    print( )
+
+def get_file_result_content(res, extra_path, use_pwd=False):
+    if use_pwd: 
+        fpath = f'{res.pwd}/{res.output}'
+    else:
+        fpath = f'{res.output}'
+    
+    if extra_path:
+        fpath = f'{fpath}/{extra_path}'
+
+    with open(fpath, 'r') as file:
+        data = file.read()
+    if data:
+        return [data]
+    return []
+
+
+# This should be abstracted by a plugin interface for extractors
+@enforce_types
+def get_indexable_content(results: QuerySet):
+    if not results:
+        return []
+    # Only use the first method available
+    res, method = results.first(), results.first().extractor
+    if method not in ('readability', 'singlefile', 'dom', 'wget'):
+        return []
+    # This should come from a plugin interface
+
+    # TODO: banish this duplication and get these from the extractor file
+    if method == 'readability':
+        return get_file_result_content(res, 'content.txt')
+    elif method == 'singlefile':
+        return get_file_result_content(res,'',use_pwd=True)
+    elif method == 'dom':
+        return get_file_result_content(res,'',use_pwd=True)
+    elif method == 'wget':
+        return get_file_result_content(res,'',use_pwd=True)

archivebox/system.py

@@ -39,11 +39,17 @@ def atomic_write(path: Union[Path, str], contents: Union[dict, str, bytes], over
     mode = 'wb+' if isinstance(contents, bytes) else 'w'
 
     # print('\n> Atomic Write:', mode, path, len(contents), f'overwrite={overwrite}')
-    with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f:
-        if isinstance(contents, dict):
-            dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
-        elif isinstance(contents, (bytes, str)):
-            f.write(contents)
+    try:
+        with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f:
+            if isinstance(contents, dict):
+                dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
+            elif isinstance(contents, (bytes, str)):
+                f.write(contents)
+    except OSError as e:
+        print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})")
+        print("    For data integrity, ArchiveBox requires a filesystem that supports atomic writes.")
+        print("    Filesystems and network drives that don't implement FSYNC are incompatible and require workarounds.")
+        raise SystemExit(1)
     os.chmod(path, int(OUTPUT_PERMISSIONS, base=8))
 
 @enforce_types
@@ -62,9 +68,9 @@ def chmod_file(path: str, cwd: str='.', permissions: str=OUTPUT_PERMISSIONS) ->
 
 
 @enforce_types
-def copy_and_overwrite(from_path: str, to_path: str):
+def copy_and_overwrite(from_path: Union[str, Path], to_path: Union[str, Path]):
     """copy a given file or directory to a given path, overwriting the destination"""
-    if os.path.isdir(from_path):
+    if Path(from_path).is_dir():
         shutil.rmtree(to_path, ignore_errors=True)
         shutil.copytree(from_path, to_path)
     else:
@@ -74,7 +80,7 @@ def copy_and_overwrite(from_path: str, to_path: str):
 
 
 @enforce_types
-def get_dir_size(path: str, recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]:
+def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]:
     """get the total disk size of a given directory, optionally summing up 
        recursively and limiting to a given filter list
     """
@@ -115,3 +121,43 @@ def dedupe_cron_jobs(cron: CronTab) -> CronTab:
         job.enable()
 
     return cron
+
+
+class suppress_output(object):
+    '''
+    A context manager for doing a "deep suppression" of stdout and stderr in 
+    Python, i.e. will suppress all print, even if the print originates in a 
+    compiled C/Fortran sub-function.
+       This will not suppress raised exceptions, since exceptions are printed
+    to stderr just before a script exits, and after the context manager has
+    exited (at least, I think that is why it lets exceptions through).      
+
+    with suppress_stdout_stderr():
+        rogue_function()
+    '''
+    def __init__(self, stdout=True, stderr=True):
+        # Open a pair of null files
+        # Save the actual stdout (1) and stderr (2) file descriptors.
+        self.stdout, self.stderr = stdout, stderr
+        if stdout:
+            self.null_stdout = os.open(os.devnull, os.O_RDWR)
+            self.real_stdout = os.dup(1)
+        if stderr:
+            self.null_stderr = os.open(os.devnull, os.O_RDWR)
+            self.real_stderr = os.dup(2)
+
+    def __enter__(self):
+        # Assign the null pointers to stdout and stderr.
+        if self.stdout:
+            os.dup2(self.null_stdout, 1)
+        if self.stderr:
+            os.dup2(self.null_stderr, 2)
+
+    def __exit__(self, *_):
+        # Re-assign the real stdout/stderr back to (1) and (2)
+        if self.stdout:
+            os.dup2(self.real_stdout, 1)
+            os.close(self.null_stdout)
+        if self.stderr:
+            os.dup2(self.real_stderr, 2)
+            os.close(self.null_stderr)

archivebox/templates/admin/base.html

@@ -88,8 +88,8 @@
         <div id="user-tools">
             <a href="{% url 'admin:Add' %}">Add ➕</a> /
             <a href="{% url 'Home' %}">Snapshots</a> /
+            <a href="/admin/core/tag/">Tags</a> /
             <a href="/admin/auth/user/">Users</a> /
-            <a href="{% url 'OldHome' %}">Old UI</a> /
             <a href="{% url 'Docs' %}">Docs</a>
              &nbsp; &nbsp;
             {% block welcome-msg %}
@@ -107,6 +107,9 @@
                 <a href="{% url 'admin:password_change' %}">{% trans 'Change password' %}</a> /
                 {% endif %}
                 <a href="{% url 'admin:logout' %}">{% trans 'Log out' %}</a>
+                |
+                <a> <span id="snapshotListView" style="cursor: pointer">☰</span> </a>       
+                <a> <span id="snapshotGridView"style="letter-spacing: -.4em; cursor: pointer;">⣿⣿</span></a> 
             {% endblock %}
         </div>
         {% endif %}
@@ -179,8 +182,63 @@
                 });
             }
         };
+        
+        function redirectWithQuery(uri){
+            uri_query = uri + document.location.search;
+            window.location = uri_query;
+
+        };
+
+        function selectSnapshotListView(){
+            localStorage.setItem('currentSnapshotView', 'List');
+            redirectWithQuery("{% url 'admin:core_snapshot_changelist' %}");
+        };
+
+        function selectSnapshotGridView(){
+            localStorage.setItem('currentSnapshotView', 'Grid');
+            redirectWithQuery("{% url 'admin:grid' %}");
+        };
+        
+        function setPreferredSnapshotView(view){
+            urlPath = window.location.pathname;
+
+            if((view==="Grid") && urlPath == "{% url 'admin:core_snapshot_changelist' %}"){  
+                selectSnapshotGridView();
+            }
+            
+            {% comment %} 
+                else if((view==="List") && urlPath == "{% url 'admin:grid' %}"){
+                selectSnapshotListView();
+            
+            } 
+            {% endcomment %}
+        };
+
+        function setupSnapshotViews() {
+            const preferredSnapshotView = localStorage.getItem('currentSnapshotView');
+            setPreferredSnapshotView(preferredSnapshotView);
+
+            $( document ).ready(function() {
+                
+                $("#snapshotListView").click(function() {
+                    selectSnapshotListView();
+                });  
+                $("#snapshotGridView").click(function() {
+                    selectSnapshotGridView();
+                });  
+
+                $('input:checkbox').change(function(){
+                    if($(this).is(':checked')) 
+                        $(this).parent().parent().parent().parent().addClass('selected-card'); 
+                    else 
+                        $(this).parent().parent().parent().parent().removeClass('selected-card')
+                });
+
+            });
+        };
         $(function () {
             fix_actions();
+            setupSnapshotViews();
         });
     })(django.jQuery);
 </script>

archivebox/templates/admin/private_index.html

@@ -0,0 +1,150 @@
+{% extends "base.html" %}
+{% load static %}
+
+{% block body %}
+    <div id="toolbar">
+        <form id="changelist-search" action="{% url 'public-index' %}" method="get">
+            <div>
+                <label for="searchbar"><img src="/static/admin/img/search.svg" alt="Search"></label>
+                <input type="text" size="40" name="q" value="" id="searchbar" autofocus placeholder="Title, URL, tags, timestamp, or content...".>
+                <input type="submit" value="Search" style="height: 36px; padding-top: 6px; margin: 8px"/>
+                <input type="button"
+                       value="♺"
+                       title="Refresh..."
+                       onclick="location.href='{% url 'public-index' %}'"
+                       style="background-color: rgba(121, 174, 200, 0.8); height: 30px; font-size: 0.8em; margin-top: 12px; padding-top: 6px; float:right">
+                </input>
+            </div>
+        </form>
+    </div>
+    <table id="table-bookmarks">
+        <thead>
+            <tr>
+                <th style="width: 100px;">Bookmarked</th>
+                <th style="width: 26vw;">Snapshot ({{object_list|length}})</th>
+                <th style="width: 140px">Files</th>
+                <th style="width: 16vw;whitespace:nowrap;overflow-x:hidden;">Original URL</th>
+            </tr>
+        </thead>
+            <tbody>
+                {% for link in object_list %}
+                    {% include 'main_index_row.html' with link=link  %}
+                {% endfor %}
+            </tbody>
+        </table>
+        <center>
+            <span class="step-links">
+                {% if page_obj.has_previous %}
+                    <a href="{% url 'public-index' %}?page=1">&laquo; first</a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.previous_page_number }}">previous</a>
+                {% endif %}
+        
+                <span class="current">
+                    Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}.
+                </span>
+        
+                {% if page_obj.has_next %}
+                    <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+                {% endif %}
+            </span>
+    
+            {% if page_obj.has_next %}
+                <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+            {% endif %}
+        </span>
+        <br>
+    </center>
+{% endblock %}
+{% extends "admin/base_site.html" %}
+{% load i18n admin_urls static admin_list %}
+{% load core_tags %}
+
+{% block extrastyle %}
+  {{ block.super }}
+  <link rel="stylesheet" type="text/css" href="{% static "admin/css/changelists.css" %}">
+  {% if cl.formset %}
+    <link rel="stylesheet" type="text/css" href="{% static "admin/css/forms.css" %}">
+  {% endif %}
+  {% if cl.formset or action_form %}
+    <script src="{% url 'admin:jsi18n' %}"></script>
+  {% endif %}
+  {{ media.css }}
+  {% if not actions_on_top and not actions_on_bottom %}
+    <style>
+      #changelist table thead th:first-child {width: inherit}
+    </style>
+  {% endif %}
+{% endblock %}
+
+{% block extrahead %}
+{{ block.super }}
+{{ media.js }}
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %}
+
+{% if not is_popup %}
+{% block breadcrumbs %}
+<div class="breadcrumbs">
+<a href="{% url 'admin:index' %}">{% translate 'Home' %}</a>
+&rsaquo; <a href="{% url 'admin:app_list' app_label=cl.opts.app_label %}">{{ cl.opts.app_config.verbose_name }}</a>
+&rsaquo; {{ cl.opts.verbose_name_plural|capfirst }}
+</div>
+{% endblock %}
+{% endif %}
+
+{% block coltype %}{% endblock %}
+
+{% block content %}
+  <div id="content-main">
+    {% block object-tools %}
+        <ul class="object-tools">
+          {% block object-tools-items %}
+            {% change_list_object_tools %}
+          {% endblock %}
+        </ul>
+    {% endblock %}
+    {% if cl.formset and cl.formset.errors %}
+        <p class="errornote">
+        {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %}
+        </p>
+        {{ cl.formset.non_form_errors }}
+    {% endif %}
+    <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">
+      <div class="changelist-form-container">
+        {% block search %}{% search_form cl %}{% endblock %}
+        {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %}
+
+        <form id="changelist-form" method="post"{% if cl.formset and cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %} novalidate>{% csrf_token %}
+        {% if cl.formset %}
+          <div>{{ cl.formset.management_form }}</div>
+        {% endif %}
+
+        {% block result_list %}
+          {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+          {% comment %} 
+             Table grid
+            {% result_list cl %} 
+          {% endcomment %}
+          {% snapshots_grid cl %}
+          {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+        {% endblock %}
+        {% block pagination %}{% pagination cl %}{% endblock %}
+        </form>
+      </div>
+      {% block filters %}
+        {% if cl.has_filters %}
+          <div id="changelist-filter">
+            <h2>{% translate 'Filter' %}</h2>
+            {% if cl.has_active_filters %}<h3 id="changelist-filter-clear">
+              <a href="{{ cl.clear_all_filters_qs }}">&#10006; {% translate "Clear all filters" %}</a>
+            </h3>{% endif %}
+            {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
+          </div>
+        {% endif %}
+      {% endblock %}
+    </div>
+  </div>
+{% endblock %}

archivebox/templates/admin/private_index_grid.html

@@ -0,0 +1,91 @@
+{% extends "admin/base_site.html" %}
+{% load i18n admin_urls static admin_list %}
+{% load core_tags %}
+
+{% block extrastyle %}
+  {{ block.super }}
+  <link rel="stylesheet" type="text/css" href="{% static "admin/css/changelists.css" %}">
+  {% if cl.formset %}
+    <link rel="stylesheet" type="text/css" href="{% static "admin/css/forms.css" %}">
+  {% endif %}
+  {% if cl.formset or action_form %}
+    <script src="{% url 'admin:jsi18n' %}"></script>
+  {% endif %}
+  {{ media.css }}
+  {% if not actions_on_top and not actions_on_bottom %}
+    <style>
+      #changelist table thead th:first-child {width: inherit}
+    </style>
+  {% endif %}
+{% endblock %}
+
+{% block extrahead %}
+{{ block.super }}
+{{ media.js }}
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %}
+
+{% if not is_popup %}
+{% block breadcrumbs %}
+<div class="breadcrumbs">
+<a href="{% url 'admin:index' %}">{% translate 'Home' %}</a>
+&rsaquo; <a href="{% url 'admin:app_list' app_label=cl.opts.app_label %}">{{ cl.opts.app_config.verbose_name }}</a>
+&rsaquo; {{ cl.opts.verbose_name_plural|capfirst }}
+</div>
+{% endblock %}
+{% endif %}
+
+{% block coltype %}{% endblock %}
+
+{% block content %}
+  <div id="content-main">
+    {% block object-tools %}
+        <ul class="object-tools">
+          {% block object-tools-items %}
+            {% change_list_object_tools %}
+          {% endblock %}
+        </ul>
+    {% endblock %}
+    {% if cl.formset and cl.formset.errors %}
+        <p class="errornote">
+        {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %}
+        </p>
+        {{ cl.formset.non_form_errors }}
+    {% endif %}
+    <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">
+      <div class="changelist-form-container">
+        {% block search %}{% search_form cl %}{% endblock %}
+        {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %}
+
+        <form id="changelist-form" method="post"{% if cl.formset and cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %} novalidate>{% csrf_token %}
+        {% if cl.formset %}
+          <div>{{ cl.formset.management_form }}</div>
+        {% endif %}
+
+        {% block result_list %}
+          {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+          {% comment %} 
+             Table grid
+            {% result_list cl %} 
+          {% endcomment %}
+          {% snapshots_grid cl %}
+          {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+        {% endblock %}
+        {% block pagination %}{% pagination cl %}{% endblock %}
+        </form>
+      </div>
+      {% block filters %}
+        {% if cl.has_filters %}
+          <div id="changelist-filter">
+            <h2>{% translate 'Filter' %}</h2>
+            {% if cl.has_active_filters %}<h3 id="changelist-filter-clear">
+              <a href="{{ cl.clear_all_filters_qs }}">&#10006; {% translate "Clear all filters" %}</a>
+            </h3>{% endif %}
+            {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
+          </div>
+        {% endif %}
+      {% endblock %}
+    </div>
+  </div>
+{% endblock %}