0.5.4 release

mark_safe is dangerous, as the URL's filename could have malicious HTML fragments in it.

Co-authored-by: Nick Sweeting <git@sweeting.me>

.dockerignore

@@ -3,14 +3,19 @@
 *.pyc
 __pycache__/
 .mypy_cache/
+.pytest_cache/
+.github/
 
 venv/
 .venv/
 .docker-venv/
 
-*.egg-info/
 build/
 dist/
+pip_dist/
+!pip_dist/archivebox.egg-info/requires.txt
+brew_dist/
+assets/
 
 data/
 output/

.github/CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contribution Process
 
-1. Confirm your desired features fit into our bigger project goals roadmap: https://github.com/pirate/ArchiveBox#roadmap
+1. Confirm your desired features fit into our bigger project goals [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap).
 2. Open an issue with your planned implementation to discuss
 3. Check in with me before starting development to make sure your work wont conflict with or duplicate existing work
 4. Setup your dev environment, make some changes, and test using the test input files
@@ -9,32 +9,30 @@
 
 **Useful links:**
 
-- https://github.com/pirate/ArchiveBox/issues
-- https://github.com/pirate/ArchiveBox/pulls
-- https://github.com/pirate/ArchiveBox/wiki/Roadmap
-- https://github.com/pirate/ArchiveBox/wiki/Install#manual-setup
+- https://github.com/ArchiveBox/ArchiveBox/issues
+- https://github.com/ArchiveBox/ArchiveBox/pulls
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
 
 ### Development Setup
 
 ```bash
-git clone https://github.com/pirate/ArchiveBox
+git clone https://github.com/ArchiveBox/ArchiveBox
 cd ArchiveBox
-# Optionally create a virtualenv
-pip install -r requirements.txt
-pip install -e .
+# Ideally do this in a virtualenv
+pip install -e '.[dev]'  # or use: pipenv install --dev
 ```
 
 ### Running Tests
 
 ```bash
-./bin/archive tests/*
-# look for errors in stdout/stderr
-# then confirm output html looks right
-
-# if on >v0.4 run the django test suite:
-archivebox manage test
+./bin/lint.sh
+./bin/test.sh
+./bin/build.sh
 ```
 
+For more common tasks see the `Development` section at the bottom of the README.
+
 ### Getting Help
 
-Open issues on Github or contact me https://sweeting.me/#contact.
+Open issues on Github or message me https://sweeting.me/#contact.

.github/FUNDING.yml

@@ -1,3 +1,3 @@
 github: pirate
 patreon: theSquashSH
-custom: ["https://paypal.me/NicholasSweeting", "https://www.blockchain.com/eth/address/0x5D4c34D4a121Fe08d1dDB7969F07550f2dB9f471", "https://www.blockchain.com/btc/address/1HuxXriPE2Bbnag3jJrqa3bkNHrs297dYH"]
+custom: ["https://twitter.com/ArchiveBoxApp", "https://paypal.me/NicholasSweeting", "https://www.blockchain.com/eth/address/0x5D4c34D4a121Fe08d1dDB7969F07550f2dB9f471", "https://www.blockchain.com/btc/address/1HuxXriPE2Bbnag3jJrqa3bkNHrs297dYH"]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,7 +10,8 @@ assignees: ''
 <!--
 Please fill out the following information, 
 feel free to delete sections if they're not applicable 
-or if long issue templates annoy you :)
+or if long issue templates annoy you.
+(the only required section is the version information)
 -->
 
 #### Describe the bug
@@ -35,9 +36,11 @@ If applicable, post any relevant screenshots or copy/pasted terminal output from
 If you're reporting a parsing / importing error, **you must paste a copy of your redacted import file here**.
 -->
 
-#### Software versions
+#### ArchiveBox version
 
- - OS:                        ([e.g. macOS 10.14] the operating system you're running ArchiveBox on)
- - ArchiveBox version:        (`git rev-parse HEAD | head -c7` [e.g. d798117] commit ID of the version you're running)
- - Python version:            (`python3 --version` [e.g. 3.7.0])
- - Chrome version:            (`chromium-browser --version` [e.g. 73.1.2.3] if relevant to bug)
+<!-- Run the `archivebox version` command locally then copy paste the result here: -->
+```logs
+replace this line with the *full*, unshortened output of running `archivebox version`
+```
+<!-- Tickets without full version info will closed until it is provided,
+we need the full output here to help you solve your issue -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -45,6 +45,6 @@ workarounds, or other software you've considered using to fix the problem.
 
 ---
 
- - [ ] I'm willing to contribute dev time / money to fix this issue
+ - [ ] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue
  - [ ] I like ArchiveBox so far / would recommend it to a friend
  - [ ] I've had a lot of difficulty getting ArchiveBox set up

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,10 +1,12 @@
-**IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes, I will close them with great prejudice.  The PEP8 checks I don't follow are intentional. PRs for minor bugfixes, typos, etc are fine.**
+<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes. -->
 
 # Summary
 
-e.g. This PR fixes ABC or adds the ability to do XYZ...
+<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->
 
-**Related issues: #XYZ** (delete this line if there are no related issues)
+# Related issues
+
+<!-- e.g. #123 or Roadmap goal # https://github.com/pirate/ArchiveBox/wiki/Roadmap -->
 
 # Changes these areas
 
@@ -13,9 +15,4 @@ e.g. This PR fixes ABC or adds the ability to do XYZ...
 - [ ] Command line interface
 - [ ] Configuration options
 - [ ] Internal architecture
-- [ ] Archived data layout on disk
-
-# Roadmap Goals
-
-This PR helps us move towards xyz roadmap goal, as outlined here: https://github.com/pirate/ArchiveBox#roadmap
-(delete this section if it's just a bugfix / simple PR)
+- [ ] Snapshot data layout on disk

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,32 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ dev ]
+  pull_request:
+    branches: [ dev ]
+  schedule:
+    - cron: '43 1 * * 2'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        queries: security-extended
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

.github/workflows/debian.yml

@@ -0,0 +1,77 @@
+name: Build Debian package
+
+on:
+  workflow_dispatch:
+  push:
+
+env:
+  DEB_BUILD_OPTIONS: nocheck
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Install packaging dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            python3 python3-dev python3-pip python3-venv python3-all \
+            dh-python debhelper devscripts dput software-properties-common \
+            python3-distutils python3-setuptools python3-wheel python3-stdeb
+
+      - name: Build Debian/Apt sdist_dsc
+        run: |
+          rm -Rf deb_dist/*
+          python3 setup.py --command-packages=stdeb.command sdist_dsc
+
+      - name: Build Debian/Apt bdist_deb
+        run: |
+          python3 setup.py --command-packages=stdeb.command bdist_deb
+
+      - name: Install archivebox from deb
+        run: |
+          cd deb_dist/
+          sudo apt-get install ./archivebox*.deb
+
+      - name: Check ArchiveBox version
+        run: |
+          # must create dir needed for snaps to run as non-root on github actions
+          sudo mkdir -p /run/user/1001 && sudo chmod -R 777 /run/user/1001
+          mkdir "${{ github.workspace }}/data" && cd "${{ github.workspace }}/data"
+          archivebox init
+          archivebox config --set SAVE_READABILITY=False
+          archivebox config --set SAVE_MERCURY=False
+          archivebox config --set SAVE_SINGLEFILE=False
+          archivebox --version
+
+      - name: Add some links to test
+        run: |
+          cd "${{ github.workspace }}/data"
+          archivebox add 'https://example.com'
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd deb_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Debian package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/debian-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: deb_dist
+
+      # - name: Push build to Launchpad PPA
+      #   run: |
+      #     debsign -k "$PGP_KEY_ID" "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"
+      #     dput archivebox "deb_dist/archivebox_${VERSION}-${DEBIAN_VERSION}_source.changes"

.github/workflows/docker.yml

@@ -0,0 +1,81 @@
+name: Build Docker image
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - '**'
+    tags:
+      - 'v*'
+  # pull_request:
+
+env:
+  DOCKER_IMAGE: archivebox-ci
+
+
+jobs:
+  buildx:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+      
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v1
+        with:
+          version: latest
+          install: true
+      
+      - name: Builder instance name
+        run: echo ${{ steps.buildx.outputs.name }}
+      
+      - name: Available platforms
+        run: echo ${{ steps.buildx.outputs.platforms }}
+      
+      - name: Cache Docker layers
+        uses: actions/cache@v2
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Docker Login
+        uses: docker/login-action@v1
+        if: github.event_name != 'pull_request'
+        with:
+           username: ${{ secrets.DOCKER_USERNAME }}
+           password: ${{ secrets.DOCKER_PASSWORD }}
+      
+      - name: Collect Docker tags
+        id: docker_meta
+        uses: crazy-max/ghaction-docker-meta@v1
+        with:
+          images: archivebox/archivebox,nikisweeting/archivebox
+          tag-sha: true
+          tag-semver: |
+            {{version}}
+            {{major}}.{{minor}}
+      
+      - name: Build and push
+        id: docker_build
+        uses: docker/build-push-action@v2
+        with:
+          context: ./
+          file: ./Dockerfile
+          builder: ${{ steps.buildx.outputs.name }}
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.docker_meta.outputs.tags }}
+          cache-from: type=local,src=/tmp/.buildx-cache
+          cache-to: type=local,dest=/tmp/.buildx-cache
+          platforms: linux/amd64,linux/386,linux/arm64,linux/arm/v7
+
+      - name: Image digest
+        run: echo ${{ steps.docker_build.outputs.digest }}

.github/workflows/homebrew.yml

@@ -0,0 +1,50 @@
+name: Build Homebrew package
+
+on:
+  workflow_dispatch:
+  push:
+
+
+jobs:
+  build:
+    runs-on: macos-latest
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      # TODO: modify archivebox.rb to update src url, hashes, and dependencies
+
+      - name: Build Homebrew Bottle
+        run: |
+          pip3 install --upgrade pip setuptools wheel
+          cd brew_dist/
+          brew install --build-bottle ./archivebox.rb
+          # brew bottle archivebox
+
+      - name: Add some links to test
+        run: |
+          mkdir data && cd data
+          archivebox init
+          archivebox add 'https://example.com'
+          archivebox version
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd brew_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Homebrew package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/homebrew-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: brew_dist
+
+      # TODO: push bottle homebrew core PR with latest changes

.github/workflows/lint.yml

@@ -0,0 +1,34 @@
+name: Run linters
+
+on:
+  workflow_dispatch:
+  push:
+
+env:
+  MAX_LINE_LENGTH: 110
+
+jobs:
+  lint:
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+
+      - name: Install flake8
+        run: |
+          pip install flake8
+
+      - name: Lint with flake8
+        run: |
+          # one pass for show-stopper syntax errors or undefined names
+          flake8 archivebox --count --show-source --statistics
+          # one pass for small stylistic things
+          flake8 archivebox --count --max-line-length="$MAX_LINE_LENGTH" --statistics

.github/workflows/pip.yml

@@ -0,0 +1,61 @@
+name: Build Pip package
+
+on:
+  workflow_dispatch:
+  push:
+
+
+jobs:
+  build:
+    runs-on: ubuntu-20.04
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+
+      - name: Build Python Package
+        run: |
+          pip3 install --upgrade pip setuptools wheel
+          rm -Rf pip_dist/*.whl
+          python3 setup.py \
+            sdist --dist-dir=./pip_dist \
+            bdist_wheel --dist-dir=./pip_dist \
+            egg_info --egg-base=./pip_dist
+          pip install pip_dist/archivebox-*.whl
+
+      - name: Add some links to test
+        run: |
+          mkdir data && cd data
+          archivebox init
+          archivebox add 'https://example.com'
+          archivebox version
+          archivebox status
+
+      # - name: Commit built package
+      #   run: |
+      #     cd pip_dist/
+      #     git config --local user.email "action@github.com"
+      #     git config --local user.name "GitHub Action"
+      #     git commit -m "Pip package autobuild" -a
+      
+      # - name: Push build to Github
+      #   uses: ad-m/github-push-action@master
+      #   with:
+      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+      #     repository: ArchiveBox/pip-archivebox
+      #     branch: ${{ github.ref }}
+      #     directory: pip_dist
+
+      # - name: Push build to PyPI
+      #   run: |
+      #     cd pip_dist/
+      #     python3 -m twine upload --repository testpypi pip_dist/*.{whl,tar.gz}
+      #     python3 -m twine upload --repository pypi pip_dist/*.{whl,tar.gz}

.github/workflows/test.yml

@@ -1,101 +1,117 @@
-name: Test workflow
+name: Run tests
 on: [push]
 
 env:
-  MAX_LINE_LENGTH: 110
+  DOCKER_IMAGE: archivebox-ci
+  PYTHONIOENCODING: utf-8
+  PYTHONLEGACYWINDOWSSTDIO: utf-8
+  USE_COLOR: False
 
 jobs:
-  lint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Set up Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-          architecture: x64
-
-      - name: Install flake8
-        run: |
-          pip install flake8
-
-      - name: Lint with flake8
-        run: |
-          # one pass for show-stopper syntax errors or undefined names
-          flake8 archivebox --count --show-source --statistics
-          # one pass for small stylistic things
-          flake8 archivebox --count --max-line-length="$MAX_LINE_LENGTH" --statistics
-
-      # - name: Lint with mypy
-      #   run: |
-      #     pip install mypy
-      #     mypy archivebox || true
-
-  test:
+  python_tests:
     runs-on: ${{ matrix.os }}
 
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
-        python: [3.7, 3.8]
+        os: [ubuntu-20.04, macos-latest, windows-latest]
+        python: [3.7]
 
     steps:
       - uses: actions/checkout@v2
         with:
+          submodules: true
           fetch-depth: 1
 
+      ### Setup Python & JS Languages
       - name: Set up Python ${{ matrix.python }}
         uses: actions/setup-python@v1
         with:
           python-version: ${{ matrix.python }}
           architecture: x64
 
-      - name: Cache virtualenv
-        uses: actions/cache@v2
-        id: cache-venv
+      - name: Set up Node JS 14.7.0
+        uses: actions/setup-node@v1
         with:
-          path: .venv
+          node-version: 14.7.0
+
+      ### Install Python & JS Dependencies
+      - name: Get pip cache dir
+        id: pip-cache
+        run: |
+          echo "::set-output name=dir::$(pip cache dir)"
+
+      - name: Cache pip
+        uses: actions/cache@v2
+        id: cache-pip
+        with:
+          path: ${{ steps.pip-cache.outputs.dir }}
           key: ${{ runner.os }}-${{ matrix.python }}-venv-${{ hashFiles('setup.py') }}
           restore-keys: |
             ${{ runner.os }}-${{ matrix.python }}-venv-
 
-      - name: Create virtualenv
-        if: steps.cache-venv.outputs.cache-hit != 'true'
+      - name: Install pip dependencies
         run: |
-          python3 -m venv .venv
-          source .venv/bin/activate
-          python3 -m pip install --upgrade pip setuptools
-
-      - name: Install dependencies
-        run: |
-          source .venv/bin/activate
+          python -m pip install --upgrade pip setuptools wheel pytest bottle
+          ./bin/build_pip.sh
           python -m pip install .
-          python -m pip install pytest bottle
+
+      - name: Get npm cache dir
+        id: npm-cache
+        run: |
+          echo "::set-output name=dir::$GITHUB_WORKSPACE/node_modules"
+
+      - name: Cache npm
+        uses: actions/cache@v2
+        id: cache-npm
+        with:
+          path: ${{ steps.npm-cache.outputs.dir }}
+          key: ${{ runner.os }}-node_modules-${{ hashFiles('package-lock.json') }}
+          restore-keys: |
+            ${{ runner.os }}-node_modules
+
+      - name: Install npm requirements
+        run: |
+          npm install
+          echo "SINGLEFILE_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/single-file" >> $GITHUB_ENV
+          echo "READABILITY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/readability-extractor" >> $GITHUB_ENV
+          echo "MERCURY_BINARY=$GITHUB_WORKSPACE/node_modules/.bin/mercury-parser" >> $GITHUB_ENV
+
+      ### Run the tests
+      - name: Directory listing for debugging
+        run: |
+          pwd
+          ls
+
+      - name: Archivebox version
+        run: |
+          archivebox version
 
       - name: Test built package with pytest
+        # TODO: remove this exception for windows once we get tests passing on that platform
+        if: ${{ !contains(matrix.os, 'windows') }}
         run: |
-          source .venv/bin/activate
-          python -m pytest -s
+          python -m pytest -s --ignore=archivebox/vendor
 
-  docker-test:
+  docker_tests:
     runs-on: ubuntu-latest
 
     steps:
       - uses: actions/checkout@v2
         with:
+          submodules: true
           fetch-depth: 1
 
-      - uses: satackey/action-docker-layer-caching@v0.0.4
+      # TODO: as of 2020-11 this helper layer broke, upgrade and re-enable this once it's usable again
+      # - uses: satackey/action-docker-layer-caching@v0.0.8
 
       - name: Build image
         run: |
-          docker build . -t archivebox
+          docker build . -t "$DOCKER_IMAGE"
 
       - name: Init data dir
         run: |
-          mkdir data
-          docker run -v "$PWD"/data:/data archivebox init
+          mkdir "${{ github.workspace }}/data"
+          docker run -v "${{ github.workspace }}/data":/data "$DOCKER_IMAGE" init
 
       - name: Run test server
         run: |
@@ -104,42 +120,27 @@ jobs:
 
       - name: Add link
         run: |
-          docker run -v "$PWD"/data:/data --network host archivebox add http://www.test-nginx-1.local
+          docker run -v "$PWD"/data:/data --network host "$DOCKER_IMAGE" add http://www.test-nginx-1.local
 
       - name: Add stdin link
         run: |
-          echo "http://www.test-nginx-2.local" | docker run -i -v "$PWD"/data:/data archivebox add
+          echo "http://www.test-nginx-2.local" | docker run -i --network host -v "$PWD"/data:/data "$DOCKER_IMAGE" add
 
       - name: List links
         run: |
-          docker run -v "$PWD"/data:/data archivebox list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
-          docker run -v "$PWD"/data:/data archivebox list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
-
+          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
+          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
 
       - name: Start docker-compose stack
         run: |
           docker-compose run archivebox init
           docker-compose up -d
-          sleep 4
-          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'django.jQuery'
+          sleep 5
+          curl --silent --location 'http://127.0.0.1:8000' | grep 'ArchiveBox'
+          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'window.django'
 
       - name: Check added urls show up in index
         run: |
           docker-compose run archivebox add 'http://example.com/#test_docker' --index-only
           curl --silent --location 'http://127.0.0.1:8000' | grep 'http://example.com/#test_docker'
-
-      - name: Curl index with PUBLIC_INDEX=False
-        run: |
-          docker-compose run archivebox config --set PUBLIC_INDEX=False
-          docker-compose up -d || true
-          sleep 8
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'Log in'
-          docker-compose down
-
-      - name: Curl index with PUBLIC_INDEX=True
-        run: |
-          docker-compose run archivebox config --set PUBLIC_INDEX=True
-          docker-compose up -d || true
-          sleep 8
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'Add Links'
-          docker-compose down
+          docker-compose down || true

.gitignore

@@ -3,14 +3,24 @@
 *.pyc
 __pycache__/
 .mypy_cache/
+.eggs/
+tests/out/
 
+# Python and Node dependencies
 venv/
 .venv/
 .docker-venv/
+node_modules/
 
-*.egg-info/
+# Packaging artifacts
+archivebox.egg-info
+archivebox-*.tar.gz
 build/
 dist/
 
+# Data folders
 data/
+data1/
+data2/
+data3/
 output/

.gitmodules

@@ -1,3 +1,25 @@
 [submodule "docs"]
-	path = docs
-	url = https://github.com/pirate/ArchiveBox.wiki.git
+    path = docs
+    url = https://github.com/ArchiveBox/ArchiveBox.wiki.git
+
+[submodule "deb_dist"]
+    path = deb_dist
+    url = https://github.com/ArchiveBox/debian-archivebox.git
+[submodule "brew_dist"]
+	path = brew_dist
+	url = https://github.com/ArchiveBox/homebrew-archivebox.git
+[submodule "pip_dist"]
+	path = pip_dist
+	url = https://github.com/ArchiveBox/pip-archivebox.git
+[submodule "docker"]
+	path = docker
+	url = https://github.com/ArchiveBox/docker-archivebox.git
+[submodule "archivebox/vendor/base32-crockford"]
+	path = archivebox/vendor/base32-crockford
+	url = https://github.com/jbittel/base32-crockford
+[submodule "archivebox/vendor/pocket"]
+	path = archivebox/vendor/pocket
+	url = https://github.com/tapanpandita/pocket
+[submodule "archivebox/vendor/django-taggit"]
+	path = archivebox/vendor/django-taggit
+	url = https://github.com/jazzband/django-taggit

.npmignore

@@ -0,0 +1,19 @@
+tests/
+archivebox/
+archivebox.egg-info/
+build/
+dist/
+docs/
+etc/
+.github
+.gitmodules
+.dockerignore
+.flake8
+CNAME
+_config.yml
+docker-compose.yaml
+docker-compose.yml
+Dockerfile
+MANIFEST.in
+Pipfile
+setup.py

Dockerfile

@@ -1,68 +1,117 @@
-# This is the Dockerfile for ArchiveBox, it includes the following major pieces:
-#     git, curl, wget, python3, youtube-dl, google-chrome-stable, ArchiveBox
+# This is the Dockerfile for ArchiveBox, it bundles the following dependencies:
+#     python3, ArchiveBox, curl, wget, git, chromium, youtube-dl, single-file
 # Usage:
-#     docker build . -t archivebox
+#     docker build . -t archivebox --no-cache
 #     docker run -v "$PWD/data":/data archivebox init
 #     docker run -v "$PWD/data":/data archivebox add 'https://example.com'
-# Documentation:
-#     https://github.com/pirate/ArchiveBox/wiki/Docker#docker
+#     docker run -v "$PWD/data":/data -it archivebox manage createsuperuser
+#     docker run -v "$PWD/data":/data -p 8000:8000 archivebox server
 
-FROM python:3.8-slim-buster
+FROM python:3.9-slim-buster
 
 LABEL name="archivebox" \
-      maintainer="Nick Sweeting <archivebox-git@sweeting.me>" \
-      description="All-in-one personal internet archiving container"
+    maintainer="Nick Sweeting <archivebox-docker@sweeting.me>" \
+    description="All-in-one personal internet archiving container" \
+    homepage="https://github.com/ArchiveBox/ArchiveBox" \
+    documentation="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker"
 
+# System-level base config
 ENV TZ=UTC \
     LANGUAGE=en_US:en \
     LC_ALL=C.UTF-8 \
     LANG=C.UTF-8 \
     PYTHONIOENCODING=UTF-8 \
     PYTHONUNBUFFERED=1 \
-    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 \
-    CODE_PATH=/app \
-    VENV_PATH=/venv \
-    DATA_PATH=/data
+    DEBIAN_FRONTEND=noninteractive \
+    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
 
-# First install CLI utils and base deps, then Chrome + Fons
-RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \
-    && apt-get update -qq \
+# Application-level base config
+ENV CODE_DIR=/app \
+    VENV_PATH=/venv \
+    DATA_DIR=/data \
+    NODE_DIR=/node \
+    ARCHIVEBOX_USER="archivebox"
+
+# Create non-privileged user for archivebox and chrome
+RUN groupadd --system $ARCHIVEBOX_USER \
+    && useradd --system --create-home --gid $ARCHIVEBOX_USER --groups audio,video $ARCHIVEBOX_USER
+
+# Install system dependencies
+RUN apt-get update -qq \
     && apt-get install -qq -y --no-install-recommends \
-       apt-transport-https ca-certificates apt-utils gnupg gosu gnupg2 libgconf-2-4 zlib1g-dev \
-       dumb-init jq git wget curl youtube-dl ffmpeg \
-    && curl -sSL "https://dl.google.com/linux/linux_signing_key.pub" | apt-key add - \
-    && echo "deb https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list \
-    && apt-get update -qq \
-    && apt-get install -qq -y --no-install-recommends \
-       google-chrome-stable \
-       fontconfig \
-       fonts-ipafont-gothic \
-       fonts-wqy-zenhei \
-       fonts-thai-tlwg \
-       fonts-kacst \
-       fonts-symbola \
-       fonts-noto \
-       fonts-freefont-ttf \
+        apt-transport-https ca-certificates gnupg2 zlib1g-dev \
+        dumb-init gosu cron unzip curl \
     && rm -rf /var/lib/apt/lists/*
 
-# Run everything from here on out as non-privileged user
-RUN groupadd --system archivebox \
-    && useradd --system --create-home --gid archivebox --groups audio,video archivebox
+# Install apt dependencies
+RUN apt-get update -qq \
+    && apt-get install -qq -y --no-install-recommends \
+        wget curl chromium git ffmpeg youtube-dl ripgrep \
+        fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
+    && rm -rf /var/lib/apt/lists/*
 
-ADD . "$CODE_PATH"
-WORKDIR "$CODE_PATH"
+# Install apt development dependencies
+# RUN apt-get install -qq \
+#     && apt-get install -qq -y --no-install-recommends \
+#         python3 python3-dev python3-pip python3-venv python3-all \
+#         dh-python debhelper devscripts dput software-properties-common \
+#         python3-distutils python3-setuptools python3-wheel python3-stdeb
+
+# Install Node environment
+RUN curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - \
+    && echo 'deb https://deb.nodesource.com/node_15.x buster main' >> /etc/apt/sources.list \
+    && apt-get update -qq \
+    && apt-get install -qq -y --no-install-recommends \
+        nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Node dependencies
+WORKDIR "$NODE_DIR"
+ENV PATH="${PATH}:$NODE_DIR/node_modules/.bin" \
+    npm_config_loglevel=error
+ADD ./package.json ./package.json
+ADD ./package-lock.json ./package-lock.json
+RUN npm ci
+
+# Install Python dependencies
+WORKDIR "$CODE_DIR"
 ENV PATH="${PATH}:$VENV_PATH/bin"
 RUN python -m venv --clear --symlinks "$VENV_PATH" \
-    && pip install --upgrade pip setuptools \
-    && pip install -e .
+    && pip install --upgrade --quiet pip setuptools
+ADD ./pip_dist/archivebox.egg-info/requires.txt "$CODE_DIR/pip_dist/archivebox.egg-info/requires.txt"
+RUN apt-get update -qq \
+    && apt-get install -qq -y --no-install-recommends \
+        build-essential python-dev python3-dev \
+    && grep -B 1000 -E '^$' "$CODE_DIR/pip_dist/archivebox.egg-info/requires.txt" | pip install --quiet -r /dev/stdin \
+    && pip install --quiet "sonic-client==0.0.5" \
+    && apt-get purge -y build-essential python-dev python3-dev \
+    && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
 
-VOLUME "$DATA_PATH"
-WORKDIR "$DATA_PATH"
+# Install ArchiveBox Python package and its dependencies
+WORKDIR "$CODE_DIR"
+ADD . "$CODE_DIR"
+RUN pip install -e .
+
+# Setup ArchiveBox runtime config
+WORKDIR "$DATA_DIR"
+ENV IN_DOCKER=True \
+    CHROME_SANDBOX=False \
+    CHROME_BINARY="chromium" \
+    USE_SINGLEFILE=True \
+    SINGLEFILE_BINARY="$NODE_DIR/node_modules/.bin/single-file" \
+    USE_READABILITY=True \
+    READABILITY_BINARY="$NODE_DIR/node_modules/.bin/readability-extractor" \
+    USE_MERCURY=True \
+    MERCURY_BINARY="$NODE_DIR/node_modules/.bin/mercury-parser"
+
+# Print version for nice docker finish summary
+# RUN archivebox version
+RUN /app/bin/docker_entrypoint.sh archivebox version
+
+# Open up the interfaces to the outside world
+VOLUME "$DATA_DIR"
 EXPOSE 8000
-ENV CHROME_BINARY=google-chrome \
-    CHROME_SANDBOX=False
 
-RUN env ALLOW_ROOT=True archivebox version
-
-ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh", "archivebox"]
-CMD ["server", "0.0.0.0:8000"]
+ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
+CMD ["archivebox", "server", "0.0.0.0:8000"]

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 Nick Sweeting
+Copyright (c) 2020 Nick Sweeting
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

MANIFEST.in

@@ -1,4 +1,4 @@
-include LICENSE
-include README.md
-include archivebox/VERSION
-recursive-include archivebox/themes *
+graft archivebox
+global-exclude .DS_Store
+global-exclude __pycache__
+global-exclude *.pyc

README.md

@@ -1,14 +1,14 @@
 <div align="center">
-<img src="https://i.imgur.com/4nkFjdv.png" height="80px">
-<h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>
+<em><img src="https://i.imgur.com/5B48E3N.png" height="90px"></em>
+<h1>ArchiveBox<br/><sub>Open-source self-hosted web archiving.</sub></h1>
 
-▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
-<a href="https://archivebox.zervice.io/">Demo</a> | 
-<a href="https://github.com/pirate/ArchiveBox">Github</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
-<a href="#background--motivation">Info & Motivation</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community">Community</a> | 
-<a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>
+▶️ <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> |
+<a href="https://archivebox.zervice.io/">Demo</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox">Github</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Documentation</a> |
+<a href="#background--motivation">Info & Motivation</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community</a> |
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap">Roadmap</a>
 
 <pre>
 "Your own personal internet archive" (网站存档 / 爬虫)
@@ -16,172 +16,452 @@
 
 <!--<a href="http://webchat.freenode.net?channels=ArchiveBox&uio=d4"><img src="https://img.shields.io/badge/Community_chat-IRC-%2328A745.svg"/></a>-->
 
-<a href="https://github.com/pirate/ArchiveBox/blob/master/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a>
-<a href="https://github.com/pirate/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a>
-<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a>
-<a href="https://test.pypi.org/project/archivebox/"><img src="https://img.shields.io/badge/Python-%3E%3D3.5-yellow.svg?logo=python&logoColor=yellow"/></a>
-<a href="https://github.com/pirate/ArchiveBox/wiki/Install#dependencies"><img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/></a>
-<a href="https://hub.docker.com/r/nikisweeting/archivebox"><img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a>
+<a href="https://test.pypi.org/project/archivebox/"><img src="https://img.shields.io/badge/Python-%3E%3D3.7-yellow.svg?logo=python&logoColor=yellow"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies"><img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/></a>
+<a href="https://hub.docker.com/r/archivebox/archivebox"><img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/></a><br/>
+<a href="https://github.com/ArchiveBox/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/ArchiveBox/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=active"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/context:python"><img alt="Language grade: Python" src="https://img.shields.io/lgtm/grade/python/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/context:javascript"><img alt="Language grade: JavaScript" src="https://img.shields.io/lgtm/grade/javascript/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/alerts/"><img alt="Total alerts" src="https://img.shields.io/lgtm/alerts/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+
 
 <hr/>
-<br/>
-<i>💥 Attention: Big API changes are coming with the current release (including <code>pip install archivebox</code>)!
-<br/><br/>
-See the <a href="https://github.com/pirate/ArchiveBox/pull/207#issuecomment-494107553">v0.4 release PR</a> for more information.</b>
-<br/>
-<hr/>
-
 </div>
 
-**ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).**
+ArchiveBox is a powerful self-hosted internet archiving solution written in Python. You feed it URLs of pages you want to archive, and it saves them to disk in a variety of formats depending on setup and content within.
 
-You can use it to preserve access to websites you care about by storing them locally offline. ArchiveBox imports lists of URLs, renders the pages in a headless, authenticated, user-scriptable browser, and then archives the content in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the originals disappear off the internet. It automatically extracts assets and media from pages and saves them in easily-accessible folders, with out-of-the-box support for extracting git repositories, audio, video, subtitles, images, PDFs, and more.
-
-#### How does it work?
+**🔢&nbsp; Run ArchiveBox via [Docker Compose (recommended)](#Quickstart), Docker, Apt, Brew, or Pip ([see below](#Quickstart)).**
 
 ```bash
-mkdir data && cd data
-archivebox init
-archivebox add 'https://example.com'
-archivebox add 'https://getpocket.com/users/USERNAME/feed/all' --depth=1
-archivebox server
+apt/brew/pip3 install archivebox
+
+archivebox init                       # run this in an empty folder
+archivebox add 'https://example.com'  # start adding URLs to archive
+curl https://example.com/rss.xml | archivebox add  # or add via stdin
+archivebox schedule --every=day https://example.com/rss.xml
 ```
 
-After installing archivebox, just pass some new links to the `archivebox add` command to start your collection.
+For each URL added, ArchiveBox saves several types of HTML snapshot (wget, Chrome headless, singlefile), a PDF, a screenshot, a WARC archive, any git repositories, images, audio, video, subtitles, article text, [and more...](#output-formats).
 
-ArchiveBox is written in Python 3.7 and uses wget, Chrome headless, youtube-dl, pywb, and other common UNIX tools to save each page you add in multiple redundant formats. It doesn't require a constantly running server or backend, just open the generated `output/index.html` in a browser to view the archive. It can import and export links as JSON (among other formats), so it's easy to script or hook up to other APIs. If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
+```bash
+archivebox server --createsuperuser 0.0.0.0:8000   # use the interactive web UI
+archivebox list 'https://example.com'  # use the CLI commands (--help for more)
+ls ./archive/*/index.json              # or browse directly via the filesystem
+```
+
+You can then manage your snapshots via the [filesystem](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout), [CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [SQLite DB](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py) (`./index.sqlite3`), [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (alpha), or [desktop app](https://github.com/ArchiveBox/electron-archivebox) (alpha).
+
+At the end of the day, the goal is to sleep soundly knowing that the part of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
 
 <div align="center">
+<br/><br/>
+<img src="https://i.imgur.com/PAzXZE8.png" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://i.imgur.com/asPNk8n.png" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://i.imgur.com/PAzXZE8.png" height="70px" alt="bookshelf graphic">
+<br/><br/>
+</div>
 
-<img src="https://i.imgur.com/3tBL7PU.png" width="22%" alt="CLI Screenshot" align="top">
-<img src="https://i.imgur.com/viklZNG.png" width="22%" alt="Desktop index screenshot" align="top">
-<img src="https://i.imgur.com/RefWsXB.jpg" width="22%" alt="Desktop details page Screenshot"/>
-<img src="https://i.imgur.com/M6HhzVx.png" width="22%" alt="Desktop details page Screenshot"/><br/>
-<sup><a href="https://archive.sweeting.me/">Demo</a> | <a href="https://github.com/pirate/ArchiveBox/wiki/Usage">Usage</a> | <a href="#screenshots">Screenshots</a></sup>
+#### ⚡️&nbsp; CLI Usage
+
+```bash
+# archivebox [subcommand] [--args]
+archivebox --version
+archivebox help
+```
+
+- `archivebox init/version/status/config/manage` to administer your collection
+- `archivebox add/remove/update/list` to manage Snapshots in the archive
+- `archivebox schedule` to pull in fresh URLs in regularly from [boorkmarks/history/Pocket/Pinboard/RSS/etc.](#input-formats)
+- `archivebox oneshot` archive single URLs without starting a whole collection
+- `archivebox shell/manage dbshell` open a REPL to use the [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha), or SQL API
+
+<div align="center">
+<br/>
+<sup><a href="https://archivebox.zervice.io/">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></sup>
 <br/>
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
-</div><br/>
+<br/><br/>
+<img src="https://i.imgur.com/njxgSbl.png" width="22%" alt="cli init screenshot" align="top">
+<img src="https://i.imgur.com/p6wK6KM.png" width="22%" alt="server snapshot admin screenshot" align="top">
+<img src="https://i.imgur.com/RefWsXB.jpg" width="28.6%" alt="server snapshot details page screenshot" align="top"/>
+<br/>
+<br/>
+<img src="https://i.imgur.com/T2UAGUD.png" width="49%" alt="grass"/><img src="https://i.imgur.com/T2UAGUD.png" width="49%" alt="grass"/>
+</div>
 
-## Quickstart
 
-ArchiveBox is written in `python3.7` and has [3 main binary dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies): `wget`, `chromium`, and `youtube-dl`.
-To get started, you can [install them manually](https://github.com/pirate/ArchiveBox/wiki/Install) using your system's package manager, use the [automated helper script](https://github.com/pirate/ArchiveBox/wiki/Quickstart), or use the official [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker) container. All three dependencies are optional if [disabled](https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles) in settings.
+### Quickstart
 
-```bash
-# Docker
-mkdir data && cd data
-docker run -v $PWD:/data nikisweeting:archivebox init
-docker run -v $PWD:/data nikisweeting:archivebox add 'https://example.com'
-docker run -v $PWD:/data -p 8000 nikisweeting:archivebox server
-open https://127.0.0.1:8000
-```
+**🖥&nbsp; Supported OSs:** Linux/BSD, macOS, Windows  &nbsp; &nbsp; **🎮&nbsp; CPU Architectures:** x86, amd64, arm7, arm8 (raspi >=3)
+**📦&nbsp; Distributions:** `docker`/`apt`/`brew`/`pip3`/`npm` (in order of completeness)
 
-```bash
-# Docker Compose
-# Download https://github.com/pirate/ArchiveBox/tree/master/docker-compose.yml
+*(click to expand your preferred **► `distribution`** below for full setup instructions)*
+
+<details>
+<summary><b>Get ArchiveBox with <code>docker-compose</code> on any platform (recommended, everything included out-of-the-box)</b></summary>
+
+<i>First make sure you have Docker installed: https://docs.docker.com/get-docker/</i>
+
+<pre lang="bash"><code>
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+curl -O 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/master/docker-compose.yml'
 docker-compose run archivebox init
+docker-compose run archivebox --version
+
+# start the webserver and open the UI (optional)
+docker-compose run archivebox manage createsuperuser
+docker-compose up -d
+open 'http://127.0.0.1:8000'
+
+# you can also add links and manage your archive via the CLI:
 docker-compose run archivebox add 'https://example.com'
-docker-compose up
-```
+docker-compose run archivebox status
+docker-compose run archivebox help  # to see more options
+</code></pre>
 
-```bash
-# Bare Metal
-# Use apt on Ubuntu/Debian, brew on mac, or pkg on BSD
-apt install python3 python3-pip git curl wget youtube-dl chromium-browser
+This is the recommended way to run ArchiveBox because it includes <i>all</i> the extractors like:<br/>
+chrome, wget, youtube-dl, git, etc., full-text search w/ sonic, and many other great features.
 
-pip install archivebox      # install archivebox
+</details>
 
-mkdir data && cd data       # (doesn't have to be called data)
+<details>
+<summary><b>Get ArchiveBox with <code>docker</code> on any platform</b></summary>
+
+<i>First make sure you have Docker installed: https://docs.docker.com/get-docker/</i>
+
+<pre lang="bash"><code>
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+docker run -v $PWD:/data -it archivebox/archivebox init
+docker run -v $PWD:/data -it archivebox/archivebox --version
+
+# start the webserver and open the UI (optional)
+docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
+
+# you can also add links and manage your archive via the CLI:
+docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
+docker run -v $PWD:/data -it archivebox/archivebox status
+docker run -v $PWD:/data -it archivebox/archivebox help  # to see more options
+</code></pre>
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>apt</code> on Ubuntu >=20.04</b></summary>
+
+<i>First make sure you're on Ubuntu >= 20.04, or scroll down for older/non-Ubuntu instructions.</i>
+
+<pre lang="bash"><code>
+# add the repo to your sources and install the archivebox package using apt
+sudo apt install software-properties-common
+sudo add-apt-repository -u ppa:archivebox/archivebox
+sudo apt install archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
 archivebox init
-archivebox add 'https://example.com'  # add URLs via args or stdin
+archivebox --version
 
-# or import an RSS/JSON/XML/TXT feed/list of links
-archivebox add https://getpocket.com/users/USERNAME/feed/all --depth=1
-```
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
 
-Once you've added your first links, open `data/index.html` in a browser to view the static archive.
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
 
-You can also start it as a server with a full web UI to manage your links:
-```bash
-archivebox manage createsuperuser
-archivebox server
-```
+For other Debian-based systems or older Ubuntu systems you can add these sources to `/etc/apt/sources.list`:
 
-You can visit `https://127.0.0.1:8000` in your browser to access it.
+<pre lang="bash"><code>
+deb http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main
+deb-src http://ppa.launchpad.net/archivebox/archivebox/ubuntu focal main
+</code></pre>
 
-[DEMO: archivebox.zervice.io/](https://archivebox.zervice.io)  
-For more information, see the [full Quickstart guide](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.
+Then run `apt update; apt install archivebox; archivebox --version`.
+
+(you may need to install some other dependencies manually however)
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>brew</code> on macOS >=10.13</b></summary>
+
+<i>First make sure you have Homebrew installed: https://brew.sh/#install</i>
+
+<pre lang="bash"><code>
+# install the archivebox package using homebrew
+brew install archivebox/archivebox/archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
+archivebox init
+archivebox --version
+
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
+
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
+
+</details>
+
+<details>
+<summary><b>Get ArchiveBox with <code>pip</code> on any platform</b></summary>
+
+<i>First make sure you have Python >= 3.7 installed: https://realpython.com/installing-python/</i>
+
+<pre lang="bash"><code>
+# install the archivebox package using pip3
+pip3 install archivebox
+
+# create a new empty directory and initalize your collection (can be anywhere)
+mkdir ~/archivebox && cd ~/archivebox
+npm install --prefix . 'git+https://github.com/ArchiveBox/ArchiveBox.git'
+archivebox init
+archivebox --version
+# Install any missing extras like wget/git/chrome/etc. manually as needed
+
+# start the webserver and open the web UI (optional)
+archivebox server --createsuperuser 0.0.0.0:8000
+open http://127.0.0.1:8000
+
+# you can also add URLs and manage the archive via the CLI and filesystem:
+archivebox add 'https://example.com'
+archivebox status
+archivebox list --html --with-headers > index.html
+archivebox list --json --with-headers > index.json
+archivebox help  # to see more options
+</code></pre>
+
+</details>
+
+No matter which install method you choose, they all roughly follow this 3-step process and all provide the same CLI, Web UI, and on-disk data format.
+
+<small>
+
+1. Install ArchiveBox: `apt/brew/pip3 install archivebox`
+2. Start a collection: `archivebox init`
+3. Start archiving: `archivebox add 'https://example.com'`
+
+</small>
+
+<br/>
+<div align="center">
+<img src="https://i.imgur.com/6AmOGJT.png" width="49%" alt="grass"/><img src="https://i.imgur.com/6AmOGJT.png" width="49%" alt="grass"/>
+</div>
+<br/>
+
+<div align="center">
+<img src="https://i.imgur.com/lUuicew.png" width="22.4%" align="top">
+<img src="https://i.imgur.com/p6wK6KM.png" width="35.9%" align="top">
+<img src="https://i.imgur.com/pzq4uXq.png" width="29.7%" align="top">
+<br/><br/>
+<sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
+<br/><br/>
+<a href="https://archivebox.zervice.io">DEMO: <code>https://archivebox.zervice.io</code></a><br/>
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a>
+<br/>
+</div>
+
+## Key Features
+
+- [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally
+- [**Powerful, intuitive command line interface**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) with [modular optional dependencies](#dependencies) 
+- [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (youtube-dl), articles (readability), code (git), etc.](#output-formats)
+- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
+- [**Uses standard, durable, long-term formats**](#saves-lots-of-useful-stuff-for-each-imported-link) like HTML, JSON, PDF, PNG, and WARC
+- [**Usable as a oneshot CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/latest/modules.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox) (ALPHA)
+- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
+- Planned: support for archiving [content requiring a login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (working, but ill-advised until some pending fixes are released)
+- Planned: support for running [JS scripts during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51), e.g. adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hiding](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expander](https://github.com/ArchiveBox/ArchiveBox/issues/345), etc.
+
+<br/>
 
 ---
 
 <div align="center">
-<img src="https://i.imgur.com/PVO88AZ.png" width="80%"/>
+<img src="https://i.imgur.com/OUmgdlH.png" width="96%" alt="lego">
 </div>
 
-# Overview
+## Input formats
 
-Because modern websites are complicated and often rely on dynamic content,
-ArchiveBox archives the sites in **several different formats** beyond what public
-archiving services like Archive.org and Archive.is are capable of saving. Using multiple
-methods and the market-dominant browser to execute JS ensures we can save even the most
-complex, finicky websites in at least a few high-quality, long-term data formats.
-
-ArchiveBox imports a list of URLs from stdin, remote URL, or file, then adds the pages to a local archive folder using wget to create a browsable HTML clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...
-
-Running `./archive` adds only new, unique links into `output/` on each run. Because it will ignore duplicates and only archive each link the first time you add it, you can schedule it to [run on a timer](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) and re-import all your feeds multiple times a day. It will run quickly even if the feeds are large, because it's only archiving the newest links since the last run. For each link, it runs through all the archive methods. Methods that fail will save `None` and be automatically retried on the next run, methods that succeed save their output into the data folder and are never retried/overwritten by subsequent runs. Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/pirate/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs).
-
-All the archived links are stored by date bookmarked in `output/archive/<timestamp>`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM.
-
-#### Can import links from many formats:
+ArchiveBox supports many input formats for URLs, including Pocket & Pinboard exports, Browser bookmarks, Browser history, plain text, HTML, markdown, and more!
 
 ```bash
 echo 'http://example.com' | archivebox add
-archivebox add ~/Downloads/firefox_bookmarks_export.html --depth=1
-archivebox add https://example.com/some/rss/feed.xml --depth=1
+archivebox add 'https://example.com/some/page'
+archivebox add < ~/Downloads/firefox_bookmarks_export.html
+archivebox add < any_text_with_urls_in_it.txt
+archivebox add --depth=1 'https://example.com/some/downloads.html'
+archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
 ```
 
-- <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more)
-- <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format
-- <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, OneTab, and more
 
-See the [Usage: CLI](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
+- <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> TXT, RSS, XML, JSON, CSV, SQL, HTML, Markdown, or [any other text-based format...](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file)
+- <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> [Browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (see instructions for: [Chrome](https://support.google.com/chrome/answer/96816?hl=en), [Firefox](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer), [Safari](http://i.imgur.com/AtcvUZA.png), [IE](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows), [Opera](http://help.opera.com/Windows/12.10/en/importexport.html), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive))
+- <img src="https://getpocket.com/favicon.ico" height="22px"/> [Pocket](https://getpocket.com/export), [Pinboard](https://pinboard.in/export/), [Instapaper](https://www.instapaper.com/user/export), [Shaarli](https://shaarli.readthedocs.io/en/master/Usage/#importexport), [Delicious](https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/), [Reddit Saved](https://github.com/csu/export-saved-reddit), [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html), [Unmark.it](http://help.unmark.it/import-export), [OneTab](https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
 
-#### Saves lots of useful stuff for each imported link:
+See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
+
+It also includes a built-in scheduled import feature with `archivebox schedule` and browser bookmarklet, so you can pull in URLs from RSS feeds, websites, or the filesystem regularly/on-demand.
+
+## Output formats
+
+All of ArchiveBox's state (including the index, snapshot data, and config file) is stored in a single folder called the "ArchiveBox data folder". All `archivebox` CLI commands must be run from inside this folder, and you first create it by running `archivebox init`.
+
+The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard sqlite3 database (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the `archive/` subfolder. Each snapshot subfolder includes a static JSON and HTML index describing its contents, and the snapshot extrator outputs are plain files within the folder (e.g. `media/example.mp4`, `git/somerepo.git`, `static/someimage.png`, etc.)
 
 ```bash
- ls output/archive/<timestamp>/
+ ls ./archive/<timestamp>/
 ```
 
 - **Index:** `index.html` & `index.json` HTML and JSON index files containing metadata and details
-- **Title:** `title` title of the site
-- **Favicon:** `favicon.ico` favicon of the site
-- **WGET Clone:** `example.com/page-name.html` wget clone of the site, with .html appended if not present
-- **WARC:** `warc/<timestamp>.gz` gzipped WARC of all the resources fetched while archiving
-- **PDF:** `output.pdf` Printed PDF of site using headless chrome
-- **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome
-- **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome
-- **URL to Archive.org:** `archive.org.txt` A link to the saved site on archive.org
+- **Title**, **Favicon**, **Headers** Response headers, site favicon, and parsed site title
+- **Wget Clone:** `example.com/page-name.html` wget clone of the site with  `warc/<timestamp>.gz`
+- Chrome Headless
+  - **SingleFile:** `singlefile.html` HTML snapshot rendered with headless Chrome using SingleFile
+  - **PDF:** `output.pdf` Printed PDF of site using headless chrome
+  - **Screenshot:** `screenshot.png` 1440x900 screenshot of site using headless chrome
+  - **DOM Dump:** `output.html` DOM Dump of the HTML after rendering using headless chrome
+  - **Readability:** `article.html/json` Article text extraction using Readability
+- **Archive.org Permalink:** `archive.org.txt` A link to the saved site on archive.org
 - **Audio & Video:** `media/` all audio/video files + playlists, including subtitles & metadata with youtube-dl
 - **Source Code:** `git/` clone of any repository found on github, bitbucket, or gitlab links
-- _More coming soon! See the [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)..._
+- _More coming soon! See the [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)..._
 
-It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/pirate/ArchiveBox/wiki/Configuration) via environment variables or config file.
+It does everything out-of-the-box by default, but you can disable or tweak [individual archive methods](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) via environment variables or config file.
 
-If you're importing URLs with secret tokens in them (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of these methods to avoid leaking private URLs to 3rd party APIs during the archiving process. See the [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details.
+<div align="center">
+<img src="https://i.imgur.com/ucyimDX.png" width="96%" alt="lego graphic">
+</div>
 
-## Key Features
+<br/>
 
-- [**Free & open source**](https://github.com/pirate/ArchiveBox/blob/master/LICENSE), doesn't require signing up for anything, stores all data locally
-- [**Few dependencies**](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) and [simple command line interface](https://github.com/pirate/ArchiveBox/wiki/Usage#CLI-Usage)
-- [**Comprehensive documentation**](https://github.com/pirate/ArchiveBox/wiki), [active development](https://github.com/pirate/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-- **Doesn't require a constantly-running server**, proxy, or native app
-- Easy to set up **[scheduled importing](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) from multiple sources**
-- Uses common, **durable, [long-term formats](#saves-lots-of-useful-stuff-for-each-imported-link)** like HTML, JSON, PDF, PNG, and WARC
-- ~~**Suitable for paywalled / [authenticated content](https://github.com/pirate/ArchiveBox/wiki/Configuration#chrome_user_data_dir)** (can use your cookies)~~ (do not do this until v0.5 is released with some security fixes)
-- Can [**run scripts during archiving**](https://github.com/pirate/ArchiveBox/issues/51) to [scroll pages](https://github.com/pirate/ArchiveBox/issues/80), [close modals](https://github.com/pirate/ArchiveBox/issues/175), expand comment threads, etc.
-- Can also [**mirror content to 3rd-party archiving services**](https://github.com/pirate/ArchiveBox/wiki/Configuration#submit_archive_dot_org) automatically for redundancy
+---
 
-## Background & Motivation
+<br/>
+
+## Dependencies
+
+You don't need to install all the dependencies, ArchiveBox will automatically enable the relevant modules based on whatever you have available, but it's recommended to use the official [Docker image](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) with everything preinstalled.
+
+If you so choose, you can also install ArchiveBox and its dependencies directly on any Linux or macOS systems using the [system package manager](https://github.com/ArchiveBox/ArchiveBox/wiki/Install) or by running the [automated setup script](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart).
+
+ArchiveBox is written in Python 3 so it requires `python3` and `pip3` available on your system. It also uses a set of optional, but highly recommended external dependencies for archiving sites: `wget` (for plain HTML, static files, and WARC saving), `chromium` (for screenshots, PDFs, JS execution, and more), `youtube-dl` (for audio and video), `git` (for cloning git repos), and `nodejs` (for readability and singlefile), and more.
+
+<br/>
+
+---
+
+<div align="center">
+<img src="https://docs.monadical.com/uploads/upload_b6900afc422ae699bfefa2dcda3306f3.png" width="100%" alt="security graphic"/>
+</div>
+
+## Caveats
+
+If you're importing URLs containing secret slugs or pages with private content (e.g Google Docs, CodiMD notepads, etc), you may want to disable some of the extractor modules to avoid leaking private URLs to 3rd party APIs during the archiving process.
+
+```bash
+# don't do this:
+archivebox add 'https://docs.google.com/document/d/12345somelongsecrethere'
+archivebox add 'https://example.com/any/url/you/want/to/keep/secret/'
+
+# without first disabling share the URL with 3rd party APIs:
+archivebox config --set SAVE_ARCHIVE_DOT_ORG=False   # disable saving all URLs in Archive.org
+archivebox config --set SAVE_FAVICON=False      # optional: only the domain is leaked, not full URL
+archivebox config --set CHROME_BINARY=chromium  # optional: switch to chromium to avoid Chrome phoning home to Google
+```
+
+Be aware that malicious archived JS can also read the contents of other pages in your archive due to snapshot CSRF and XSS protections being imperfect. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page for more details.
+
+```bash
+# visiting an archived page with malicious JS:
+https://127.0.0.1:8000/archive/1602401954/example.com/index.html
+
+# example.com/index.js can now make a request to read everything:
+https://127.0.0.1:8000/index.html
+https://127.0.0.1:8000/archive/*
+# then example.com/index.js can send it off to some evil server
+```
+
+Support for saving multiple snapshots of each site over time will be [added soon](https://github.com/ArchiveBox/ArchiveBox/issues/179) (along with the ability to view diffs of the changes between runs). For now ArchiveBox is designed to only archive each URL with each extractor type once. A workaround to take multiple snapshots of the same URL is to make them slightly different by adding a hash:
+
+```bash
+archivebox add 'https://example.com#2020-10-24'
+...
+archivebox add 'https://example.com#2020-10-25'
+```
+
+<br/>
+
+---
+
+<br/>
+
+## Screenshots
+
+<div align="center" width="80%">
+<img src="https://i.imgur.com/PVO88AZ.png" width="80%"/>
+<table>
+<tbody>
+<tr>
+<td>
+<img src="https://i.imgur.com/npareKG.png" alt="brew install archivebox"><br/>
+<img src="https://i.imgur.com/5vSBO2R.png" alt="archivebox version">
+</td>
+<td>
+<img src="https://i.imgur.com/JXXxFzB.png" alt="archivebox init"><br/>
+</td>
+<td>
+<img src="https://i.imgur.com/wNYtV3v.jpg" alt="archivebox add">
+</td>
+<td>
+<img src="https://i.imgur.com/uZcIOn9.png" alt="archivebox data dir">
+</td>
+</tr>
+<tr>
+<td>
+<img src="https://i.imgur.com/H08eaia.png" alt="archivebox server">
+</td>
+<td>
+<img src="https://i.imgur.com/zM4z1aU.png" alt="archivebox server add">
+</td>
+<td>
+<img src="https://i.imgur.com/p6wK6KM.png" alt="archivebox server list">
+</td>
+<td>
+<img src="https://i.imgur.com/pzq4uXq.png" alt="archivebox server detail">
+</td>
+</tr>
+</tbody>
+</table>
+</div>
+<br/>
+
+---
+
+<br/>
+
+<div align="center">
+<img src="https://i.imgur.com/ZSUm9mr.png" width="100%" alt="paisley graphic">
+</div>
+
+# Background & Motivation
 
 Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.
 
@@ -197,15 +477,20 @@ archive internet content enables to you save the stuff you care most about befor
 The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful.
 I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.
 
+Because modern websites are complicated and often rely on dynamic content,
+ArchiveBox archives the sites in **several different formats** beyond what public archiving services like Archive.org and Archive.is are capable of saving. Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.
+
+All the archived links are stored by date bookmarked in `./archive/<timestamp>`, and everything is indexed nicely with JSON & HTML files. The intent is for all the content to be viewable with common software in 50 - 100 years without needing to run ArchiveBox in a VM.
+
 ## Comparison to Other Projects
 
-▶ **Check out our [community page](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.**
+▶ **Check out our [community page](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for an index of web archiving initiatives and projects.**
 
-<img src="https://i.imgur.com/4nkFjdv.png" width="10%" align="left"/> The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
+<img src="https://i.imgur.com/4nkFjdv.png" width="10%" align="left" alt="comparison"/> The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
 
 #### User Interface & Intended Purpose
 
-ArchiveBox differentiates itself from [similar projects](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
+ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
 
 #### Private Local Archives vs Centralized Public Archives
 
@@ -215,73 +500,206 @@ Unlike crawler software that starts from a seed URL and works outwards, or publi
 
 Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files.
 
-## Learn more
+<div align="center">
+<br/>
+<img src="https://i.imgur.com/q0Oe36M.png" width="100%" alt="dependencies graphic">
+</div>
 
-<!--▶ **Join out our [community chat](http://webchat.freenode.net?channels=ArchiveBox&uio=d4) hosted on IRC freenode.net:`#ArchiveBox`!**-->
+## Learn more
 
 Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web!
 
 <img src="https://i.imgur.com/0ZOmOvN.png" width="14%" align="right"/>
 
-- [Community Wiki](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
-  - [The Master Lists](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists)  
+- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+  - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#the-master-lists)  
     _Community-maintained indexes of archiving tools and institutions._
-  - [Web Archiving Software](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects)  
+  - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects)  
     _Open source tools and projects in the internet archiving space._
-  - [Reading List](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Reading-List)  
+  - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#reading-list)  
     _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
-  - [Communities](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Communities)  
+  - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#communities)  
     _A collection of the most active internet archiving communities and initiatives._
-- Check out the ArchiveBox [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
+- Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
 - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
-- Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+- Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+
+<br/>
 
 ---
 
+<div align="center">
+<img src="https://i.imgur.com/SMkGW0L.png" width="100%" alt="documentation graphic">
+</div>
+
 # Documentation
 
 <img src="https://read-the-docs-guidelines.readthedocs-hosted.com/_images/logo-dark.png" width="13%" align="right"/>
 
-We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) for documentation.
+We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation.
 
-You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.
-
-You can build the docs by running:
-
-```python
-cd ArchiveBox
-pipenv install --dev
-sphinx-apidoc -o docs archivebox
-cd docs/
-make html
-# then open docs/_build/html/index.html
-```
+You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder.
 
 ## Getting Started
 
-- [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
-- [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
-- [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)
+- [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart)
+- [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install)
+- [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker)
 
 ## Reference
 
-- [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
-- [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
-- [Supported Sources](https://github.com/pirate/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
-- [Supported Outputs](https://github.com/pirate/ArchiveBox/wiki#can-save-these-things-for-each-site)
-- [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
-- [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
-- [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Install-Chromium)
-- [Security Overview](https://github.com/pirate/ArchiveBox/wiki/Security-Overview)
-- [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)
+- [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage)
+- [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)
+- [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
+- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site)
+- [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving)
+- [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive)
+- [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install)
+- [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview)
+- [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
+- [Python API](https://docs.archivebox.io/en/latest/modules.html) (alpha)
+- [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (alpha)
 
 ## More Info
 
-- [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
-- [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
-- [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
-- [Background & Motivation](https://github.com/pirate/ArchiveBox#background--motivation)
-- [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)
+- [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues)
+- [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)
+- [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
+- [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+- [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation)
+- [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+
+<br/>
+
+---
+
+<div align="center">
+<img src="https://i.imgur.com/EGWjbD4.png" width="100%" alt="development">
+</div>
+
+# ArchiveBox Development
+
+All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap.
+
+Low hanging fruit / easy first tickets:<br/>
+<a href="https://lgtm.com/projects/g/ArchiveBox/ArchiveBox/alerts/"><img alt="Total alerts" src="https://img.shields.io/lgtm/alerts/g/ArchiveBox/ArchiveBox.svg?logo=lgtm&logoWidth=18"/></a>
+
+### Setup the dev environment
+
+#### 1. Clone the main code repo (making sure to pull the submodules as well)
+
+```bash
+git clone --recurse-submodules https://github.com/ArchiveBox/ArchiveBox
+cd ArchiveBox
+git checkout dev  # or the branch you want to test
+git submodule update --init --recursive
+git pull --recurse-submodules
+```
+
+#### 2. Option A: Install the Python, JS, and system dependencies directly on your machine
+
+```bash
+# Install ArchiveBox + python dependencies
+python3 -m venv .venv && source .venv/bin/activate && pip install -e '.[dev]'
+# or: pipenv install --dev && pipenv shell
+
+# Install node dependencies
+npm install
+
+# Check to see if anything is missing
+archivebox --version
+# install any missing dependencies manually, or use the helper script:
+./bin/setup.sh
+```
+
+#### 2. Option B: Build the docker container and use that for development instead
+
+```bash
+# Optional: develop via docker by mounting the code dir into the container
+# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
+# inside the container will reload and pick up your changes
+docker build . -t archivebox
+docker run -it --rm archivebox version
+docker run -it --rm -p 8000:8000 \
+    -v $PWD/data:/data \
+    -v $PWD/archivebox:/app/archivebox \
+    archivebox server 0.0.0.0:8000 --debug --reload
+```
+
+### Common development tasks
+
+See the `./bin/` folder and read the source of the bash scripts within.
+You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`.
+
+#### Run in DEBUG mode
+
+```bash
+archivebox config --set DEBUG=True
+# or
+archivebox server --debug ...
+```
+
+#### Build and run a Github branch
+
+```bash
+docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
+docker run -it -v $PWD:/data archivebox:dev ...
+```
+
+#### Run the linters
+
+```bash
+./bin/lint.sh
+```
+(uses `flake8` and `mypy`)
+
+#### Run the integration tests
+
+```bash
+./bin/test.sh
+```
+(uses `pytest -s`)
+
+#### Make migrations or enter a django shell
+
+Make sure to run this whenever you change things in `models.py`.
+```bash
+cd archivebox/
+./manage.py makemigrations
+
+cd path/to/test/data/
+archivebox shell
+archivebox manage dbshell
+```
+(uses `pytest -s`)
+
+#### Build the docs, pip package, and docker image
+
+(Normally CI takes care of this, but these scripts can be run to do it manually)
+```bash
+./bin/build.sh
+
+# or individually:
+./bin/build_docs.sh
+./bin/build_pip.sh
+./bin/build_deb.sh
+./bin/build_brew.sh
+./bin/build_docker.sh
+```
+
+#### Roll a release
+
+(Normally CI takes care of this, but these scripts can be run to do it manually)
+```bash
+./bin/release.sh
+
+# or individually:
+./bin/release_docs.sh
+./bin/release_pip.sh
+./bin/release_deb.sh
+./bin/release_brew.sh
+./bin/release_docker.sh
+```
 
 ---
 
@@ -289,16 +707,8 @@ make html
 <br/><br/>
 <img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" height="40px"/>
 <br/>
-<sub><i>This project is maintained mostly in <a href="https://nicksweeting.com/blog#About">my spare time</a> with the help from generous contributors.</i></sub>
+<sub><i>This project is maintained mostly in <a href="https://nicksweeting.com/blog#About">my spare time</a> with the help from generous contributors and Monadical.com.</i></sub>
 <br/><br/>
-Contributor Spotlight:<br/><br/>
- 
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/0"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/0"></a>
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/1"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/1"></a>
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/2"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/2"></a>
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/3"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/3"></a>
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/4"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/4"></a>
-<a href="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/links/5"><img src="https://sourcerer.io/fame/pirate/pirate/ArchiveBox/images/5"></a>
 
 <br/>
 <a href="https://github.com/sponsors/pirate">Sponsor us on Github</a>
@@ -307,8 +717,8 @@ Contributor Spotlight:<br/><br/>
 <a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
 <br/>
 
-<a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
-<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
+<a href="https://twitter.com/ArchiveBoxApp"><img src="https://img.shields.io/badge/Tweet-%40ArchiveBoxApp-blue.svg?style=flat"/></a>
+<a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
 
 <br/><br/>
 

archivebox/LICENSE

@@ -0,0 +1 @@
+../LICENSE

archivebox/README.md

@@ -0,0 +1 @@
+../README.md

archivebox/VERSION

@@ -1 +0,0 @@
-0.4.9

archivebox/cli/__init__.py

@@ -5,18 +5,21 @@ import os
 import sys
 import argparse
 
-from typing import Optional, Dict, List, IO
+from typing import Optional, Dict, List, IO, Union
+from pathlib import Path
 
 from ..config import OUTPUT_DIR
 
 from importlib import import_module
 
-CLI_DIR = os.path.dirname(os.path.abspath(__file__))
+CLI_DIR = Path(__file__).resolve().parent
 
 # these common commands will appear sorted before any others for ease-of-use
 meta_cmds = ('help', 'version')
 main_cmds = ('init', 'info', 'config')
-archive_cmds = ('add', 'remove', 'update', 'list')
+archive_cmds = ('add', 'remove', 'update', 'list', 'status')
+
+fake_db = ("oneshot",)
 
 display_first = (*meta_cmds, *main_cmds, *archive_cmds)
 
@@ -55,9 +58,13 @@ def list_subcommands() -> Dict[str, str]:
 def run_subcommand(subcommand: str,
                    subcommand_args: List[str]=None,
                    stdin: Optional[IO]=None,
-                   pwd: Optional[str]=None) -> None:
+                   pwd: Union[Path, str, None]=None) -> None:
     """Run a given ArchiveBox subcommand with the given list of args"""
 
+    if subcommand not in meta_cmds:
+        from ..config import setup_django
+        setup_django(in_memory_db=subcommand in fake_db, check_db=subcommand in archive_cmds)
+
     module = import_module('.archivebox_{}'.format(subcommand), __package__)
     module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
 
@@ -104,11 +111,11 @@ def main(args: Optional[List[str]]=NotProvided, stdin: Optional[IO]=NotProvided,
     )
     command = parser.parse_args(args or ())
 
-    if command.help or command.subcommand is None:
-        command.subcommand = 'help'
-    elif command.version:
+    if command.version:
         command.subcommand = 'version'
-    
+    elif command.help or command.subcommand is None:
+        command.subcommand = 'help'
+
     if command.subcommand not in ('help', 'version', 'status'):
         from ..logging_util import log_cli_command
 
@@ -133,3 +140,5 @@ __all__ = (
     'run_subcommand',
     *SUBCOMMANDS.keys(),
 )
+
+

archivebox/cli/archivebox_add.py

@@ -55,6 +55,24 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         type=int,
         help="Recursively archive all linked pages up to this many hops away"
     )
+    parser.add_argument(
+        "--overwrite",
+        default=False,
+        action="store_true",
+        help="Re-archive URLs from scratch, overwriting any existing files"
+    )
+    parser.add_argument(
+        "--init", #'-i',
+        action='store_true',
+        help="Init/upgrade the curent data directory before adding",
+    )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
     command = parser.parse_args(args or ())
     urls = command.urls
     stdin_urls = accept_stdin(stdin)
@@ -69,6 +87,9 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         depth=command.depth,
         update_all=command.update_all,
         index_only=command.index_only,
+        overwrite=command.overwrite,
+        init=command.init,
+        extractors=command.extract,
         out_dir=pwd or OUTPUT_DIR,
     )
 

archivebox/cli/archivebox_list.py

@@ -23,7 +23,7 @@ from ..index import (
     get_corrupted_folders,
     get_unrecognized_folders,
 )
-from ..logging_util import SmartFormatter, accept_stdin
+from ..logging_util import SmartFormatter, accept_stdin, stderr
 
 
 @docstring(list_all.__doc__)
@@ -46,6 +46,16 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         action='store_true',
         help="Print the output in JSON format with all columns included.",
     )
+    group.add_argument(
+        '--html',
+        action='store_true',
+        help="Print the output in HTML format"
+    )
+    parser.add_argument(
+        '--with-headers',
+        action='store_true',
+        help='Include the headers in the output document' 
+    )
     parser.add_argument(
         '--sort', #'-s',
         type=str,
@@ -88,7 +98,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )
@@ -102,6 +112,13 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     command = parser.parse_args(args or ())
     filter_patterns_str = accept_stdin(stdin)
 
+    if command.with_headers and not (command.json or command.html or command.csv):
+        stderr(
+            '[X] --with-headers can only be used with --json, --html or --csv options.\n',
+            color='red',
+        )
+        raise SystemExit(2)
+
     matching_folders = list_all(
         filter_patterns_str=filter_patterns_str,
         filter_patterns=command.filter_patterns,
@@ -112,6 +129,8 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         sort=command.sort,
         csv=command.csv,
         json=command.json,
+        html=command.html,
+        with_headers=command.with_headers,
         out_dir=pwd or OUTPUT_DIR,
     )
     raise SystemExit(not matching_folders)

archivebox/cli/archivebox_oneshot.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+
+__package__ = 'archivebox.cli'
+__command__ = 'archivebox oneshot'
+
+import sys
+import argparse
+
+from pathlib import Path
+from typing import List, Optional, IO
+
+from ..main import oneshot
+from ..util import docstring
+from ..config import OUTPUT_DIR
+from ..logging_util import SmartFormatter, accept_stdin, stderr
+
+
+@docstring(oneshot.__doc__)
+def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
+    parser = argparse.ArgumentParser(
+        prog=__command__,
+        description=oneshot.__doc__,
+        add_help=True,
+        formatter_class=SmartFormatter,
+    )
+    parser.add_argument(
+        'url',
+        type=str,
+        default=None,
+        help=(
+            'URLs or paths to archive e.g.:\n'
+            '    https://getpocket.com/users/USERNAME/feed/all\n'
+            '    https://example.com/some/rss/feed.xml\n'
+            '    https://example.com\n'
+            '    ~/Downloads/firefox_bookmarks_export.html\n'
+            '    ~/Desktop/sites_list.csv\n'
+        )
+    )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
+    parser.add_argument(
+        '--out-dir',
+        type=str,
+        default=OUTPUT_DIR,
+        help= "Path to save the single archive folder to, e.g. ./example.com_archive"
+    )
+    command = parser.parse_args(args or ())
+    url = command.url
+    stdin_url = accept_stdin(stdin)
+    if (stdin_url and url) or (not stdin and not url):
+        stderr(
+            '[X] You must pass a URL/path to add via stdin or CLI arguments.\n',
+            color='red',
+        )
+        raise SystemExit(2)
+    
+    oneshot(
+        url=stdin_url or url,
+        out_dir=Path(command.out_dir).resolve(),
+        extractors=command.extract,
+    )
+
+
+if __name__ == '__main__':
+    main(args=sys.argv[1:], stdin=sys.stdin)

archivebox/cli/archivebox_remove.py

@@ -50,7 +50,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex','tag'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )

archivebox/cli/archivebox_schedule.py

@@ -36,8 +36,15 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--every', # '-e',
         type=str,
-        default='day',
-        help='Run ArchiveBox once every [timeperiod] (hour/day/week/month/year or cron format e.g. "0 0 * * *")',
+        default=None,
+        help='Run ArchiveBox once every [timeperiod] (hour/day/month/year or cron format e.g. "0 0 * * *")',
+    )
+    parser.add_argument(
+        '--depth', # '-d',
+        type=int,
+        choices=[0, 1],
+        default=0,
+        help='Depth to archive to [0] or 1, see "add" command help for more info.',
     )
     group.add_argument(
         '--clear', # '-c'
@@ -80,6 +87,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         run_all=command.run_all,
         quiet=command.quiet,
         every=command.every,
+        depth=command.depth,
         import_path=command.import_path,
         out_dir=pwd or OUTPUT_DIR,
     )

archivebox/cli/archivebox_server.py

@@ -10,10 +10,9 @@ from typing import Optional, List, IO
 
 from ..main import server
 from ..util import docstring
-from ..config import OUTPUT_DIR
+from ..config import OUTPUT_DIR, BIND_ADDR
 from ..logging_util import SmartFormatter, reject_stdin
 
-
 @docstring(server.__doc__)
 def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional[str]=None) -> None:
     parser = argparse.ArgumentParser(
@@ -26,7 +25,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         'runserver_args',
         nargs='*',
         type=str,
-        default=None,
+        default=[BIND_ADDR],
         help='Arguments to pass to Django runserver'
     )
     parser.add_argument(
@@ -44,6 +43,11 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         action='store_true',
         help='Run archivebox init before starting the server',
     )
+    parser.add_argument(
+        '--createsuperuser',
+        action='store_true',
+        help='Run archivebox manage createsuperuser before starting the server',
+    )
     command = parser.parse_args(args or ())
     reject_stdin(__command__, stdin)
     
@@ -52,6 +56,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         reload=command.reload,
         debug=command.debug,
         init=command.init,
+        createsuperuser=command.createsuperuser,
         out_dir=pwd or OUTPUT_DIR,
     )
 

archivebox/cli/archivebox_update.py

@@ -91,7 +91,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
     parser.add_argument(
         '--filter-type',
         type=str,
-        choices=('exact', 'substring', 'domain', 'regex'),
+        choices=('exact', 'substring', 'domain', 'regex', 'tag', 'search'),
         default='exact',
         help='Type of pattern matching to use when filtering URLs',
     )
@@ -102,6 +102,13 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         default=None,
         help='Update only URLs matching these filter patterns.'
     )
+    parser.add_argument(
+        "--extract",
+        type=str,
+        help="Pass a list of the extractors to be used. If the method name is not correct, it will be ignored. \
+              This does not take precedence over the configuration",
+        default=""
+    )
     command = parser.parse_args(args or ())
     filter_patterns_str = accept_stdin(stdin)
 
@@ -117,6 +124,7 @@ def main(args: Optional[List[str]]=None, stdin: Optional[IO]=None, pwd: Optional
         after=command.after,
         before=command.before,
         out_dir=pwd or OUTPUT_DIR,
+        extractors=command.extract,
     )
     
 

archivebox/cli/tests.py

@@ -7,6 +7,7 @@ import os
 import sys
 import shutil
 import unittest
+from pathlib import Path
 
 from contextlib import contextmanager
 
@@ -109,13 +110,13 @@ class TestInit(unittest.TestCase):
         with output_hidden():
             archivebox_init.main([])
 
-        assert os.path.exists(os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME))
-        assert os.path.exists(os.path.join(OUTPUT_DIR, JSON_INDEX_FILENAME))
-        assert os.path.exists(os.path.join(OUTPUT_DIR, HTML_INDEX_FILENAME))
+        assert (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
+        assert (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
+        assert (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
         assert len(load_main_index(out_dir=OUTPUT_DIR)) == 0
 
     def test_conflicting_init(self):
-        with open(os.path.join(OUTPUT_DIR, 'test_conflict.txt'), 'w+') as f:
+        with open(Path(OUTPUT_DIR) / 'test_conflict.txt', 'w+') as f:
             f.write('test')
 
         try:
@@ -125,9 +126,9 @@ class TestInit(unittest.TestCase):
         except SystemExit:
             pass
 
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME))
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, JSON_INDEX_FILENAME))
-        assert not os.path.exists(os.path.join(OUTPUT_DIR, HTML_INDEX_FILENAME))
+        assert not (Path(OUTPUT_DIR) / SQL_INDEX_FILENAME).exists()
+        assert not (Path(OUTPUT_DIR) / JSON_INDEX_FILENAME).exists()
+        assert not (Path(OUTPUT_DIR) / HTML_INDEX_FILENAME).exists()
         try:
             load_main_index(out_dir=OUTPUT_DIR)
             assert False, 'load_main_index should raise an exception when no index is present'
@@ -159,7 +160,7 @@ class TestAdd(unittest.TestCase):
         assert len(all_links) == 30
 
     def test_add_arg_file(self):
-        test_file = os.path.join(OUTPUT_DIR, 'test.txt')
+        test_file = Path(OUTPUT_DIR) / 'test.txt'
         with open(test_file, 'w+') as f:
             f.write(test_urls)
 

archivebox/config.py

@@ -1,21 +1,44 @@
-__package__ = 'archivebox.config'
+"""
+ArchiveBox config definitons (including defaults and dynamic config options).
+
+Config Usage Example:
+
+    archivebox config --set MEDIA_TIMEOUT=600
+    env MEDIA_TIMEOUT=600 USE_COLOR=False ... archivebox [subcommand] ...
+
+Config Precedence Order:
+
+  1. cli args                 (--update-all / --index-only / etc.)
+  2. shell environment vars   (env USE_COLOR=False archivebox add '...')
+  3. config file              (echo "SAVE_FAVICON=False" >> ArchiveBox.conf)
+  4. defaults                 (defined below in Python)
+
+Documentation:
+
+  https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration
+
+"""
+
+__package__ = 'archivebox'
 
 import os
 import io
 import re
 import sys
-import django
+import json
 import getpass
+import platform
 import shutil
+import django
 
 from hashlib import md5
 from pathlib import Path
-from typing import Optional, Type, Tuple, Dict
+from typing import Optional, Type, Tuple, Dict, Union, List
 from subprocess import run, PIPE, DEVNULL
 from configparser import ConfigParser
 from collections import defaultdict
 
-from .stubs import (
+from .config_stubs import (
     SimpleConfigValueDict,
     ConfigValue,
     ConfigDict,
@@ -23,28 +46,14 @@ from .stubs import (
     ConfigDefaultDict,
 )
 
-# precedence order for config:
-# 1. cli args
-# 2. shell environment vars
-# 3. config file
-# 4. defaults
+############################### Config Schema ##################################
 
-# env USE_COLO=false archivebox add '...'
-# env SHOW_PROGRESS=1 archivebox add '...'
-
-# ******************************************************************************
-# Documentation: https://github.com/pirate/ArchiveBox/wiki/Configuration
-# Use the 'env' command to pass config options to ArchiveBox.  e.g.:
-#     env USE_COLOR=True CHROME_BINARY=chromium archivebox add < example.html
-# ******************************************************************************
-
-################################# User Config ##################################
-
-CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
+CONFIG_SCHEMA: Dict[str, ConfigDefaultDict] = {
     'SHELL_CONFIG': {
         'IS_TTY':                   {'type': bool,  'default': lambda _: sys.stdout.isatty()},
         'USE_COLOR':                {'type': bool,  'default': lambda c: c['IS_TTY']},
-        'SHOW_PROGRESS':            {'type': bool,  'default': lambda c: c['IS_TTY']},
+        'SHOW_PROGRESS':            {'type': bool,  'default': lambda c: (c['IS_TTY'] and platform.system() != 'Darwin')},  # progress bars are buggy on mac, disable for now
+        'IN_DOCKER':                {'type': bool,  'default': False},
         # TODO: 'SHOW_HINTS':       {'type:  bool,  'default': True},
     },
 
@@ -56,17 +65,18 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'MEDIA_TIMEOUT':            {'type': int,   'default': 3600},
         'OUTPUT_PERMISSIONS':       {'type': str,   'default': '755'},
         'RESTRICT_FILE_NAMES':      {'type': str,   'default': 'windows'},
-        'URL_BLACKLIST':            {'type': str,   'default': None},
+        'URL_BLACKLIST':            {'type': str,   'default': r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$'},  # to avoid downloading code assets as their own pages
     },
 
     'SERVER_CONFIG': {
         'SECRET_KEY':               {'type': str,   'default': None},
+        'BIND_ADDR':                {'type': str,   'default': lambda c: ['127.0.0.1:8000', '0.0.0.0:8000'][c['IN_DOCKER']]},
         'ALLOWED_HOSTS':            {'type': str,   'default': '*'},
         'DEBUG':                    {'type': bool,  'default': False},
         'PUBLIC_INDEX':             {'type': bool,  'default': True},
         'PUBLIC_SNAPSHOTS':         {'type': bool,  'default': True},
+        'PUBLIC_ADD_VIEW':          {'type': bool,  'default': False},
         'FOOTER_INFO':              {'type': str,   'default': 'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.'},
-        'ACTIVE_THEME':             {'type': str,   'default': 'default'},
     },
 
     'ARCHIVE_METHOD_TOGGLES': {
@@ -74,13 +84,16 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'SAVE_FAVICON':             {'type': bool,  'default': True, 'aliases': ('FETCH_FAVICON',)},
         'SAVE_WGET':                {'type': bool,  'default': True, 'aliases': ('FETCH_WGET',)},
         'SAVE_WGET_REQUISITES':     {'type': bool,  'default': True, 'aliases': ('FETCH_WGET_REQUISITES',)},
+        'SAVE_SINGLEFILE':          {'type': bool,  'default': True, 'aliases': ('FETCH_SINGLEFILE',)},
+        'SAVE_READABILITY':         {'type': bool,  'default': True, 'aliases': ('FETCH_READABILITY',)},
+        'SAVE_MERCURY':             {'type': bool,  'default': True, 'aliases': ('FETCH_MERCURY',)},
         'SAVE_PDF':                 {'type': bool,  'default': True, 'aliases': ('FETCH_PDF',)},
         'SAVE_SCREENSHOT':          {'type': bool,  'default': True, 'aliases': ('FETCH_SCREENSHOT',)},
         'SAVE_DOM':                 {'type': bool,  'default': True, 'aliases': ('FETCH_DOM',)},
+        'SAVE_HEADERS':             {'type': bool,  'default': True, 'aliases': ('FETCH_HEADERS',)},
         'SAVE_WARC':                {'type': bool,  'default': True, 'aliases': ('FETCH_WARC',)},
         'SAVE_GIT':                 {'type': bool,  'default': True, 'aliases': ('FETCH_GIT',)},
         'SAVE_MEDIA':               {'type': bool,  'default': True, 'aliases': ('FETCH_MEDIA',)},
-        'SAVE_PLAYLISTS':           {'type': bool,  'default': True, 'aliases': ('FETCH_PLAYLISTS',)},
         'SAVE_ARCHIVE_DOT_ORG':     {'type': bool,  'default': True, 'aliases': ('SUBMIT_ARCHIVE_DOT_ORG',)},
     },
 
@@ -89,46 +102,119 @@ CONFIG_DEFAULTS: Dict[str, ConfigDefaultDict] = {
         'GIT_DOMAINS':              {'type': str,   'default': 'github.com,bitbucket.org,gitlab.com'},
         'CHECK_SSL_VALIDITY':       {'type': bool,  'default': True},
 
-        'CURL_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/pirate/ArchiveBox/) curl/{CURL_VERSION}'},
-        'WGET_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/pirate/ArchiveBox/) wget/{WGET_VERSION}'},
+        'CURL_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION}'},
+        'WGET_USER_AGENT':          {'type': str,   'default': 'ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION}'},
         'CHROME_USER_AGENT':        {'type': str,   'default': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'},
 
         'COOKIES_FILE':             {'type': str,   'default': None},
         'CHROME_USER_DATA_DIR':     {'type': str,   'default': None},
 
         'CHROME_HEADLESS':          {'type': bool,  'default': True},
-        'CHROME_SANDBOX':           {'type': bool,  'default': True},
+        'CHROME_SANDBOX':           {'type': bool,  'default': lambda c: not c['IN_DOCKER']},
+        'YOUTUBEDL_ARGS':           {'type': list,  'default': ['--write-description',
+                                                                '--write-info-json',
+                                                                '--write-annotations',
+                                                                '--write-thumbnail',
+                                                                '--no-call-home',
+                                                                '--all-subs',
+                                                                '--yes-playlist',
+                                                                '--continue',
+                                                                '--ignore-errors',
+                                                                '--geo-bypass',
+                                                                '--add-metadata',
+                                                                '--max-filesize=750m',
+                                                                ]},
+                                                                    
 
+        'WGET_ARGS':                {'type': list,  'default': ['--no-verbose',
+                                                                '--adjust-extension',
+                                                                '--convert-links',
+                                                                '--force-directories',
+                                                                '--backup-converted',
+                                                                '--span-hosts',
+                                                                '--no-parent',
+                                                                '-e', 'robots=off',
+                                                                ]},
+        'CURL_ARGS':                {'type': list,  'default': ['--silent',
+                                                                '--location',
+                                                                '--compressed'
+                                                               ]},
+        'GIT_ARGS':                 {'type': list,  'default': ['--recursive']},
+    },
+
+    'SEARCH_BACKEND_CONFIG' : {
+        'USE_INDEXING_BACKEND':     {'type': bool,  'default': True},
+        'USE_SEARCHING_BACKEND':    {'type': bool,  'default': True},
+        'SEARCH_BACKEND_ENGINE':    {'type': str,   'default': 'ripgrep'},
+        'SEARCH_BACKEND_HOST_NAME': {'type': str,   'default': 'localhost'},
+        'SEARCH_BACKEND_PORT':      {'type': int,   'default': 1491},
+        'SEARCH_BACKEND_PASSWORD':  {'type': str,   'default': 'SecretPassword'},
+        # SONIC
+        'SONIC_COLLECTION':         {'type': str,   'default': 'archivebox'},
+        'SONIC_BUCKET':             {'type': str,   'default': 'snapshots'},
     },
 
     'DEPENDENCY_CONFIG': {
         'USE_CURL':                 {'type': bool,  'default': True},
         'USE_WGET':                 {'type': bool,  'default': True},
+        'USE_SINGLEFILE':           {'type': bool,  'default': True},
+        'USE_READABILITY':          {'type': bool,  'default': True},
+        'USE_MERCURY':              {'type': bool,  'default': True},
         'USE_GIT':                  {'type': bool,  'default': True},
         'USE_CHROME':               {'type': bool,  'default': True},
+        'USE_NODE':                 {'type': bool,  'default': True},
         'USE_YOUTUBEDL':            {'type': bool,  'default': True},
-
+        'USE_RIPGREP':              {'type': bool,  'default': True},
+        
         'CURL_BINARY':              {'type': str,   'default': 'curl'},
         'GIT_BINARY':               {'type': str,   'default': 'git'},
         'WGET_BINARY':              {'type': str,   'default': 'wget'},
+        'SINGLEFILE_BINARY':        {'type': str,   'default': 'single-file'},
+        'READABILITY_BINARY':       {'type': str,   'default': 'readability-extractor'},
+        'MERCURY_BINARY':           {'type': str,   'default': 'mercury-parser'},
         'YOUTUBEDL_BINARY':         {'type': str,   'default': 'youtube-dl'},
+        'NODE_BINARY':              {'type': str,   'default': 'node'},
+        'RIPGREP_BINARY':           {'type': str,   'default': 'rg'},
         'CHROME_BINARY':            {'type': str,   'default': None},
+
+        'POCKET_CONSUMER_KEY':      {'type': str,   'default': None},
+        'POCKET_ACCESS_TOKENS':     {'type': dict,  'default': {}},
     },
 }
 
+
+########################## Backwards-Compatibility #############################
+
+
+# for backwards compatibility with old config files, check old/deprecated names for each key
 CONFIG_ALIASES = {
     alias: key
-    for section in CONFIG_DEFAULTS.values()
+    for section in CONFIG_SCHEMA.values()
         for key, default in section.items()
             for alias in default.get('aliases', ())
 }
-USER_CONFIG = {key for section in CONFIG_DEFAULTS.values() for key in section.keys()}
+USER_CONFIG = {key for section in CONFIG_SCHEMA.values() for key in section.keys()}
+
 def get_real_name(key: str) -> str:
+    """get the current canonical name for a given deprecated config key"""
     return CONFIG_ALIASES.get(key.upper().strip(), key.upper().strip())
 
-############################## Derived Config ##############################
 
-# Constants
+
+################################ Constants #####################################
+
+PACKAGE_DIR_NAME = 'archivebox'
+TEMPLATES_DIR_NAME = 'templates'
+
+ARCHIVE_DIR_NAME = 'archive'
+SOURCES_DIR_NAME = 'sources'
+LOGS_DIR_NAME = 'logs'
+SQL_INDEX_FILENAME = 'index.sqlite3'
+JSON_INDEX_FILENAME = 'index.json'
+HTML_INDEX_FILENAME = 'index.html'
+ROBOTS_TXT_FILENAME = 'robots.txt'
+FAVICON_FILENAME = 'favicon.ico'
+CONFIG_FILENAME = 'ArchiveBox.conf'
 
 DEFAULT_CLI_COLORS = {
     'reset': '\033[00;00m',
@@ -177,56 +263,30 @@ STATICFILE_EXTENSIONS = {
     # html, htm, shtml, xhtml, xml, aspx, php, cgi
 }
 
-VERSION_FILENAME = 'VERSION'
-PYTHON_DIR_NAME = 'archivebox'
-TEMPLATES_DIR_NAME = 'themes'
-
-ARCHIVE_DIR_NAME = 'archive'
-SOURCES_DIR_NAME = 'sources'
-LOGS_DIR_NAME = 'logs'
-STATIC_DIR_NAME = 'static'
-SQL_INDEX_FILENAME = 'index.sqlite3'
-JSON_INDEX_FILENAME = 'index.json'
-HTML_INDEX_FILENAME = 'index.html'
-ROBOTS_TXT_FILENAME = 'robots.txt'
-FAVICON_FILENAME = 'favicon.ico'
-CONFIG_FILENAME = 'ArchiveBox.conf'
-
-CONFIG_HEADER = (
-"""# This is the config file for your ArchiveBox collection.
-#
-# You can add options here manually in INI format, or automatically by running:
-#    archivebox config --set KEY=VALUE
-# 
-# If you modify this file manually, make sure to update your archive after by running:
-#    archivebox init
-#
-# A list of all possible config with documentation and examples can be found here:
-#    https://github.com/pirate/ArchiveBox/wiki/Configuration
-
-""")
 
 
-DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
+############################## Derived Config ##################################
+
+
+DYNAMIC_CONFIG_SCHEMA: ConfigDefaultDict = {
     'TERM_WIDTH':               {'default': lambda c: lambda: shutil.get_terminal_size((100, 10)).columns},
     'USER':                     {'default': lambda c: getpass.getuser() or os.getlogin()},
     'ANSI':                     {'default': lambda c: DEFAULT_CLI_COLORS if c['USE_COLOR'] else {k: '' for k in DEFAULT_CLI_COLORS.keys()}},
 
-    'REPO_DIR':                 {'default': lambda c: os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..'))},
-    'PYTHON_DIR':               {'default': lambda c: os.path.join(c['REPO_DIR'], PYTHON_DIR_NAME)},
-    'TEMPLATES_DIR':            {'default': lambda c: os.path.join(c['PYTHON_DIR'], TEMPLATES_DIR_NAME, 'legacy')},
+    'PACKAGE_DIR':              {'default': lambda c: Path(__file__).resolve().parent},
+    'TEMPLATES_DIR':            {'default': lambda c: c['PACKAGE_DIR'] / TEMPLATES_DIR_NAME},
 
-    'OUTPUT_DIR':               {'default': lambda c: os.path.abspath(os.path.expanduser(c['OUTPUT_DIR'])) if c['OUTPUT_DIR'] else os.path.abspath(os.curdir)},
-    'ARCHIVE_DIR':              {'default': lambda c: os.path.join(c['OUTPUT_DIR'], ARCHIVE_DIR_NAME)},
-    'SOURCES_DIR':              {'default': lambda c: os.path.join(c['OUTPUT_DIR'], SOURCES_DIR_NAME)},
-    'LOGS_DIR':                 {'default': lambda c: os.path.join(c['OUTPUT_DIR'], LOGS_DIR_NAME)},
-    'CONFIG_FILE':              {'default': lambda c: os.path.abspath(os.path.expanduser(c['CONFIG_FILE'])) if c['CONFIG_FILE'] else os.path.join(c['OUTPUT_DIR'], CONFIG_FILENAME)},
-    'COOKIES_FILE':             {'default': lambda c: c['COOKIES_FILE'] and os.path.abspath(os.path.expanduser(c['COOKIES_FILE']))},
-    'CHROME_USER_DATA_DIR':     {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (os.path.abspath(os.path.expanduser(c['CHROME_USER_DATA_DIR'])) or None)},
-    'URL_BLACKLIST_PTN':        {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'], re.IGNORECASE)},
+    'OUTPUT_DIR':               {'default': lambda c: Path(c['OUTPUT_DIR']).resolve() if c['OUTPUT_DIR'] else Path(os.curdir).resolve()},
+    'ARCHIVE_DIR':              {'default': lambda c: c['OUTPUT_DIR'] / ARCHIVE_DIR_NAME},
+    'SOURCES_DIR':              {'default': lambda c: c['OUTPUT_DIR'] / SOURCES_DIR_NAME},
+    'LOGS_DIR':                 {'default': lambda c: c['OUTPUT_DIR'] / LOGS_DIR_NAME},
+    'CONFIG_FILE':              {'default': lambda c: Path(c['CONFIG_FILE']).resolve() if c['CONFIG_FILE'] else c['OUTPUT_DIR'] / CONFIG_FILENAME},
+    'COOKIES_FILE':             {'default': lambda c: c['COOKIES_FILE'] and Path(c['COOKIES_FILE']).resolve()},
+    'CHROME_USER_DATA_DIR':     {'default': lambda c: find_chrome_data_dir() if c['CHROME_USER_DATA_DIR'] is None else (Path(c['CHROME_USER_DATA_DIR']).resolve() if c['CHROME_USER_DATA_DIR'] else None)},   # None means unset, so we autodetect it with find_chrome_Data_dir(), but emptystring '' means user manually set it to '', and we should store it as None
+    'URL_BLACKLIST_PTN':        {'default': lambda c: c['URL_BLACKLIST'] and re.compile(c['URL_BLACKLIST'] or '', re.IGNORECASE | re.UNICODE | re.MULTILINE)},
 
     'ARCHIVEBOX_BINARY':        {'default': lambda c: sys.argv[0]},
-    'VERSION':                  {'default': lambda c: open(os.path.join(c['PYTHON_DIR'], VERSION_FILENAME), 'r').read().strip()},
+    'VERSION':                  {'default': lambda c: json.loads((Path(c['PACKAGE_DIR']) / 'package.json').read_text().strip())['version']},
     'GIT_SHA':                  {'default': lambda c: c['VERSION'].split('+')[-1] or 'unknown'},
 
     'PYTHON_BINARY':            {'default': lambda c: sys.executable},
@@ -236,9 +296,10 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'DJANGO_BINARY':            {'default': lambda c: django.__file__.replace('__init__.py', 'bin/django-admin.py')},
     'DJANGO_VERSION':           {'default': lambda c: '{}.{}.{} {} ({})'.format(*django.VERSION)},
 
-    'USE_CURL':                 {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['FETCH_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])},
+    'USE_CURL':                 {'default': lambda c: c['USE_CURL'] and (c['SAVE_FAVICON'] or c['SAVE_TITLE'] or c['SAVE_ARCHIVE_DOT_ORG'])},
     'CURL_VERSION':             {'default': lambda c: bin_version(c['CURL_BINARY']) if c['USE_CURL'] else None},
     'CURL_USER_AGENT':          {'default': lambda c: c['CURL_USER_AGENT'].format(**c)},
+    'CURL_ARGS':                {'default': lambda c: c['CURL_ARGS'] or []},
     'SAVE_FAVICON':             {'default': lambda c: c['USE_CURL'] and c['SAVE_FAVICON']},
     'SAVE_ARCHIVE_DOT_ORG':     {'default': lambda c: c['USE_CURL'] and c['SAVE_ARCHIVE_DOT_ORG']},
 
@@ -248,6 +309,18 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'WGET_USER_AGENT':          {'default': lambda c: c['WGET_USER_AGENT'].format(**c)},
     'SAVE_WGET':                {'default': lambda c: c['USE_WGET'] and c['SAVE_WGET']},
     'SAVE_WARC':                {'default': lambda c: c['USE_WGET'] and c['SAVE_WARC']},
+    'WGET_ARGS':                {'default': lambda c: c['WGET_ARGS'] or []},
+
+    'RIPGREP_VERSION':          {'default': lambda c: bin_version(c['RIPGREP_BINARY']) if c['USE_RIPGREP'] else None},
+
+    'USE_SINGLEFILE':           {'default': lambda c: c['USE_SINGLEFILE'] and c['SAVE_SINGLEFILE']},
+    'SINGLEFILE_VERSION':       {'default': lambda c: bin_version(c['SINGLEFILE_BINARY']) if c['USE_SINGLEFILE'] else None},
+
+    'USE_READABILITY':          {'default': lambda c: c['USE_READABILITY'] and c['SAVE_READABILITY']},
+    'READABILITY_VERSION':      {'default': lambda c: bin_version(c['READABILITY_BINARY']) if c['USE_READABILITY'] else None},
+
+    'USE_MERCURY':              {'default': lambda c: c['USE_MERCURY'] and c['SAVE_MERCURY']},
+    'MERCURY_VERSION':          {'default': lambda c: '1.0.0' if shutil.which(str(bin_path(c['MERCURY_BINARY']))) else None},  # mercury is unversioned
 
     'USE_GIT':                  {'default': lambda c: c['USE_GIT'] and c['SAVE_GIT']},
     'GIT_VERSION':              {'default': lambda c: bin_version(c['GIT_BINARY']) if c['USE_GIT'] else None},
@@ -256,14 +329,21 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
     'USE_YOUTUBEDL':            {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']},
     'YOUTUBEDL_VERSION':        {'default': lambda c: bin_version(c['YOUTUBEDL_BINARY']) if c['USE_YOUTUBEDL'] else None},
     'SAVE_MEDIA':               {'default': lambda c: c['USE_YOUTUBEDL'] and c['SAVE_MEDIA']},
-    'SAVE_PLAYLISTS':           {'default': lambda c: c['SAVE_PLAYLISTS'] and c['SAVE_MEDIA']},
+    'YOUTUBEDL_ARGS':           {'default': lambda c: c['YOUTUBEDL_ARGS'] or []},
 
-    'USE_CHROME':               {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'])},
+    'USE_CHROME':               {'default': lambda c: c['USE_CHROME'] and (c['SAVE_PDF'] or c['SAVE_SCREENSHOT'] or c['SAVE_DOM'] or c['SAVE_SINGLEFILE'])},
     'CHROME_BINARY':            {'default': lambda c: c['CHROME_BINARY'] if c['CHROME_BINARY'] else find_chrome_binary()},
     'CHROME_VERSION':           {'default': lambda c: bin_version(c['CHROME_BINARY']) if c['USE_CHROME'] else None},
+    
     'SAVE_PDF':                 {'default': lambda c: c['USE_CHROME'] and c['SAVE_PDF']},
     'SAVE_SCREENSHOT':          {'default': lambda c: c['USE_CHROME'] and c['SAVE_SCREENSHOT']},
     'SAVE_DOM':                 {'default': lambda c: c['USE_CHROME'] and c['SAVE_DOM']},
+    'SAVE_SINGLEFILE':          {'default': lambda c: c['USE_CHROME'] and c['SAVE_SINGLEFILE'] and c['USE_NODE']},
+    'SAVE_READABILITY':         {'default': lambda c: c['USE_READABILITY'] and c['USE_NODE']},
+    'SAVE_MERCURY':             {'default': lambda c: c['USE_MERCURY'] and c['USE_NODE']},
+    
+    'USE_NODE':                 {'default': lambda c: c['USE_NODE'] and (c['SAVE_READABILITY'] or c['SAVE_SINGLEFILE'] or c['SAVE_MERCURY'])},
+    'NODE_VERSION':             {'default': lambda c: bin_version(c['NODE_BINARY']) if c['USE_NODE'] else None},
 
     'DEPENDENCIES':             {'default': lambda c: get_dependency_info(c)},
     'CODE_LOCATIONS':           {'default': lambda c: get_code_locations(c)},
@@ -276,6 +356,7 @@ DERIVED_CONFIG_DEFAULTS: ConfigDefaultDict = {
 
 ################################### Helpers ####################################
 
+
 def load_config_val(key: str,
                     default: ConfigDefaultValue=None,
                     type: Optional[Type]=None,
@@ -322,15 +403,18 @@ def load_config_val(key: str,
             raise ValueError(f'Invalid configuration option {key}={val} (expected an integer)')
         return int(val)
 
-    raise Exception('Config values can only be str, bool, or int')
+    elif type is list or type is dict:
+        return json.loads(val)
+
+    raise Exception('Config values can only be str, bool, int or json')
 
 
 def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]:
     """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf"""
 
-    out_dir = out_dir or os.path.abspath(os.getenv('OUTPUT_DIR', '.'))
-    config_path = os.path.join(out_dir, CONFIG_FILENAME)
-    if os.path.exists(config_path):
+    out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve()
+    config_path = Path(out_dir) / CONFIG_FILENAME
+    if config_path.exists():
         config_file = ConfigParser()
         config_file.optionxform = str 
         config_file.read(config_path)
@@ -349,12 +433,26 @@ def load_config_file(out_dir: str=None) -> Optional[Dict[str, str]]:
 def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
     """load the ini-formatted config file from OUTPUT_DIR/Archivebox.conf"""
 
-    from ..system import atomic_write
+    from .system import atomic_write
 
-    out_dir = out_dir or os.path.abspath(os.getenv('OUTPUT_DIR', '.'))
-    config_path = os.path.join(out_dir, CONFIG_FILENAME)
+    CONFIG_HEADER = (
+    """# This is the config file for your ArchiveBox collection.
+    #
+    # You can add options here manually in INI format, or automatically by running:
+    #    archivebox config --set KEY=VALUE
+    # 
+    # If you modify this file manually, make sure to update your archive after by running:
+    #    archivebox init
+    #
+    # A list of all possible config with documentation and examples can be found here:
+    #    https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration
+
+    """)
+
+    out_dir = out_dir or Path(os.getenv('OUTPUT_DIR', '.')).resolve()
+    config_path = Path(out_dir) /  CONFIG_FILENAME
     
-    if not os.path.exists(config_path):
+    if not config_path.exists():
         atomic_write(config_path, CONFIG_HEADER)
 
     config_file = ConfigParser()
@@ -364,7 +462,7 @@ def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
     with open(config_path, 'r') as old:
         atomic_write(f'{config_path}.bak', old.read())
 
-    find_section = lambda key: [name for name, opts in CONFIG_DEFAULTS.items() if key in opts][0]
+    find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0]
 
     # Set up sections in empty config file
     for key, val in config.items():
@@ -404,7 +502,7 @@ def write_config_file(config: Dict[str, str], out_dir: str=None) -> ConfigDict:
         with open(f'{config_path}.bak', 'r') as old:
             atomic_write(config_path, old.read())
 
-    if os.path.exists(f'{config_path}.bak'):
+    if Path(f'{config_path}.bak').exists():
         os.remove(f'{config_path}.bak')
 
     return {}
@@ -442,7 +540,7 @@ def load_config(defaults: ConfigDefaultDict,
             stderr('    Check your config for mistakes and try again (your archive data is unaffected).')
             stderr()
             stderr('    For config documentation and examples see:')
-            stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration')
+            stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration')
             stderr()
             raise
             raise SystemExit(2)
@@ -454,8 +552,8 @@ def load_config(defaults: ConfigDefaultDict,
 #     with open(os.path.join(config['OUTPUT_DIR'], CONFIG_FILENAME), 'w+') as f:
 
 
-
-def stderr(*args, color: Optional[str]=None, config: Optional[ConfigDict]=None) -> None:
+# Logging Helpers
+def stdout(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None:
     ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI
 
     if color:
@@ -463,34 +561,60 @@ def stderr(*args, color: Optional[str]=None, config: Optional[ConfigDict]=None)
     else:
         strs = [' '.join(str(a) for a in args), '\n']
 
-    sys.stderr.write(''.join(strs))
+    sys.stdout.write(prefix + ''.join(strs))
 
+def stderr(*args, color: Optional[str]=None, prefix: str='', config: Optional[ConfigDict]=None) -> None:
+    ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI
+
+    if color:
+        strs = [ansi[color], ' '.join(str(a) for a in args), ansi['reset'], '\n']
+    else:
+        strs = [' '.join(str(a) for a in args), '\n']
+
+    sys.stderr.write(prefix + ''.join(strs))
+
+def hint(text: Union[Tuple[str, ...], List[str], str], prefix='    ', config: Optional[ConfigDict]=None) -> None:
+    ansi = DEFAULT_CLI_COLORS if (config or {}).get('USE_COLOR') else ANSI
+
+    if isinstance(text, str):
+        stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text, **ansi))
+    else:
+        stderr('{}{lightred}Hint:{reset} {}'.format(prefix, text[0], **ansi))
+        for line in text[1:]:
+            stderr('{}      {}'.format(prefix, line))
+
+
+# Dependency Metadata Helpers
 def bin_version(binary: Optional[str]) -> Optional[str]:
     """check the presence and return valid version line of a specified binary"""
 
     abspath = bin_path(binary)
-    if not abspath:
+    if not binary or not abspath:
         return None
 
     try:
         version_str = run([abspath, "--version"], stdout=PIPE).stdout.strip().decode()
         # take first 3 columns of first line of version info
         return ' '.join(version_str.split('\n')[0].strip().split()[:3])
-    except Exception:
+    except OSError:
+        pass
         # stderr(f'[X] Unable to find working version of dependency: {binary}', color='red')
         # stderr('    Make sure it\'s installed, then confirm it\'s working by running:')
         # stderr(f'        {binary} --version')
         # stderr()
         # stderr('    If you don\'t want to install it, you can disable it via config. See here for more info:')
-        # stderr('        https://github.com/pirate/ArchiveBox/wiki/Install')
-        # stderr()
-        return None
+        # stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Install')
+    return None
 
 def bin_path(binary: Optional[str]) -> Optional[str]:
     if binary is None:
         return None
 
-    return shutil.which(os.path.expanduser(binary)) or binary
+    node_modules_bin = Path('.') / 'node_modules' / '.bin' / binary
+    if node_modules_bin.exists():
+        return str(node_modules_bin.resolve())
+
+    return shutil.which(str(Path(binary).expanduser())) or shutil.which(str(binary)) or binary
 
 def bin_hash(binary: Optional[str]) -> Optional[str]:
     if binary is None:
@@ -529,8 +653,6 @@ def find_chrome_binary() -> Optional[str]:
         if full_path_exists:
             return name
     
-    stderr('[X] Unable to find a working version of Chrome/Chromium, is it installed and in your $PATH?', color='red')
-    stderr()
     return None
 
 def find_chrome_data_dir() -> Optional[str]:
@@ -553,105 +675,105 @@ def find_chrome_data_dir() -> Optional[str]:
         '~/.config/google-chrome-dev',
     )
     for path in default_profile_paths:
-        full_path = os.path.expanduser(path)
-        if os.path.exists(full_path):
+        full_path = Path(path).resolve()
+        if full_path.exists():
             return full_path
     return None
 
 def wget_supports_compression(config):
-    cmd = [
-        config['WGET_BINARY'],
-        "--compression=auto",
-        "--help",
-    ]
-    return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode
+    try:
+        cmd = [
+            config['WGET_BINARY'],
+            "--compression=auto",
+            "--help",
+        ]
+        return not run(cmd, stdout=DEVNULL, stderr=DEVNULL).returncode
+    except (FileNotFoundError, OSError):
+        return False
 
 def get_code_locations(config: ConfigDict) -> SimpleConfigValueDict:
     return {
-        'REPO_DIR': {
-            'path': os.path.abspath(config['REPO_DIR']),
+        'PACKAGE_DIR': {
+            'path': (config['PACKAGE_DIR']).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['REPO_DIR'], 'archivebox')),
-        },
-        'PYTHON_DIR': {
-            'path': os.path.abspath(config['PYTHON_DIR']),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['PYTHON_DIR'], '__main__.py')),
+            'is_valid': (config['PACKAGE_DIR'] / '__main__.py').exists(),
         },
         'TEMPLATES_DIR': {
-            'path': os.path.abspath(config['TEMPLATES_DIR']),
+            'path': (config['TEMPLATES_DIR']).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['TEMPLATES_DIR'], 'static')),
+            'is_valid': (config['TEMPLATES_DIR'] / 'static').exists(),
         },
+        # 'NODE_MODULES_DIR': {
+        #     'path': ,
+        #     'enabled': ,
+        #     'is_valid': (...).exists(),
+        # },
     }
 
 def get_external_locations(config: ConfigDict) -> ConfigValue:
-    abspath = lambda path: None if path is None else os.path.abspath(path)
+    abspath = lambda path: None if path is None else Path(path).resolve()
     return {
         'CHROME_USER_DATA_DIR': {
             'path': abspath(config['CHROME_USER_DATA_DIR']),
             'enabled': config['USE_CHROME'] and config['CHROME_USER_DATA_DIR'],
-            'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else os.path.exists(os.path.join(config['CHROME_USER_DATA_DIR'], 'Default')),
+            'is_valid': False if config['CHROME_USER_DATA_DIR'] is None else (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists(),
         },
         'COOKIES_FILE': {
             'path': abspath(config['COOKIES_FILE']),
             'enabled': config['USE_WGET'] and config['COOKIES_FILE'],
-            'is_valid': False if config['COOKIES_FILE'] is None else os.path.exists(config['COOKIES_FILE']),
+            'is_valid': False if config['COOKIES_FILE'] is None else Path(config['COOKIES_FILE']).exists(),
         },
     }
 
 def get_data_locations(config: ConfigDict) -> ConfigValue:
     return {
         'OUTPUT_DIR': {
-            'path': os.path.abspath(config['OUTPUT_DIR']),
+            'path': config['OUTPUT_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
+            'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(),
         },
         'SOURCES_DIR': {
-            'path': os.path.abspath(config['SOURCES_DIR']),
+            'path': config['SOURCES_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['SOURCES_DIR']),
+            'is_valid': config['SOURCES_DIR'].exists(),
         },
         'LOGS_DIR': {
-            'path': os.path.abspath(config['LOGS_DIR']),
+            'path': config['LOGS_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['LOGS_DIR']),
+            'is_valid': config['LOGS_DIR'].exists(),
         },
         'ARCHIVE_DIR': {
-            'path': os.path.abspath(config['ARCHIVE_DIR']),
+            'path': config['ARCHIVE_DIR'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['ARCHIVE_DIR']),
+            'is_valid': config['ARCHIVE_DIR'].exists(),
         },
         'CONFIG_FILE': {
-            'path': os.path.abspath(config['CONFIG_FILE']),
+            'path': config['CONFIG_FILE'].resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(config['CONFIG_FILE']),
+            'is_valid': config['CONFIG_FILE'].exists(),
         },
         'SQL_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], SQL_INDEX_FILENAME)),
+            'path': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).resolve(),
             'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], SQL_INDEX_FILENAME)),
-        },
-        'JSON_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], JSON_INDEX_FILENAME)),
-        },
-        'HTML_INDEX': {
-            'path': os.path.abspath(os.path.join(config['OUTPUT_DIR'], HTML_INDEX_FILENAME)),
-            'enabled': True,
-            'is_valid': os.path.exists(os.path.join(config['OUTPUT_DIR'], HTML_INDEX_FILENAME)),
+            'is_valid': (config['OUTPUT_DIR'] / SQL_INDEX_FILENAME).exists(),
         },
     }
 
 def get_dependency_info(config: ConfigDict) -> ConfigValue:
     return {
+        'ARCHIVEBOX_BINARY': {
+            'path': bin_path(config['ARCHIVEBOX_BINARY']),
+            'version': config['VERSION'],
+            'hash': bin_hash(config['ARCHIVEBOX_BINARY']),
+            'enabled': True,
+            'is_valid': True,
+        },
         'PYTHON_BINARY': {
             'path': bin_path(config['PYTHON_BINARY']),
             'version': config['PYTHON_VERSION'],
             'hash': bin_hash(config['PYTHON_BINARY']),
             'enabled': True,
-            'is_valid': bool(config['DJANGO_VERSION']),
+            'is_valid': bool(config['PYTHON_VERSION']),
         },
         'DJANGO_BINARY': {
             'path': bin_path(config['DJANGO_BINARY']),
@@ -663,7 +785,7 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
         'CURL_BINARY': {
             'path': bin_path(config['CURL_BINARY']),
             'version': config['CURL_VERSION'],
-            'hash': bin_hash(config['PYTHON_BINARY']),
+            'hash': bin_hash(config['CURL_BINARY']),
             'enabled': config['USE_CURL'],
             'is_valid': bool(config['CURL_VERSION']),
         },
@@ -674,6 +796,34 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
             'enabled': config['USE_WGET'],
             'is_valid': bool(config['WGET_VERSION']),
         },
+        'NODE_BINARY': {
+            'path': bin_path(config['NODE_BINARY']),
+            'version': config['NODE_VERSION'],
+            'hash': bin_hash(config['NODE_BINARY']),
+            'enabled': config['USE_NODE'],
+            'is_valid': bool(config['NODE_VERSION']),
+        },
+        'SINGLEFILE_BINARY': {
+            'path': bin_path(config['SINGLEFILE_BINARY']),
+            'version': config['SINGLEFILE_VERSION'],
+            'hash': bin_hash(config['SINGLEFILE_BINARY']),
+            'enabled': config['USE_SINGLEFILE'],
+            'is_valid': bool(config['SINGLEFILE_VERSION']),
+        },
+        'READABILITY_BINARY': {
+            'path': bin_path(config['READABILITY_BINARY']),
+            'version': config['READABILITY_VERSION'],
+            'hash': bin_hash(config['READABILITY_BINARY']),
+            'enabled': config['USE_READABILITY'],
+            'is_valid': bool(config['READABILITY_VERSION']),
+        },
+        'MERCURY_BINARY': {
+            'path': bin_path(config['MERCURY_BINARY']),
+            'version': config['MERCURY_VERSION'],
+            'hash': bin_hash(config['MERCURY_BINARY']),
+            'enabled': config['USE_MERCURY'],
+            'is_valid': bool(config['MERCURY_VERSION']),
+        },
         'GIT_BINARY': {
             'path': bin_path(config['GIT_BINARY']),
             'version': config['GIT_VERSION'],
@@ -695,6 +845,21 @@ def get_dependency_info(config: ConfigDict) -> ConfigValue:
             'enabled': config['USE_CHROME'],
             'is_valid': bool(config['CHROME_VERSION']),
         },
+        'RIPGREP_BINARY': {
+            'path': bin_path(config['RIPGREP_BINARY']),
+            'version': config['RIPGREP_VERSION'],
+            'hash': bin_hash(config['RIPGREP_BINARY']),
+            'enabled': config['USE_RIPGREP'],
+            'is_valid': bool(config['RIPGREP_VERSION']),
+        },
+        # TODO: add an entry for the sonic search backend?
+        # 'SONIC_BINARY': {
+        #     'path': bin_path(config['SONIC_BINARY']),
+        #     'version': config['SONIC_VERSION'],
+        #     'hash': bin_hash(config['SONIC_BINARY']),
+        #     'enabled': config['USE_SONIC'],
+        #     'is_valid': bool(config['SONIC_VERSION']),
+        # },
     }
 
 def get_chrome_info(config: ConfigDict) -> ConfigValue:
@@ -710,34 +875,69 @@ def get_chrome_info(config: ConfigDict) -> ConfigValue:
     }
 
 
-################################## Load Config #################################
+# ******************************************************************************
+# ******************************************************************************
+# ******************************** Load Config *********************************
+# ******* (compile the defaults, configs, and metadata all into CONFIG) ********
+# ******************************************************************************
+# ******************************************************************************
 
 
 def load_all_config():
     CONFIG: ConfigDict = {}
-    for section_name, section_config in CONFIG_DEFAULTS.items():
+    for section_name, section_config in CONFIG_SCHEMA.items():
         CONFIG = load_config(section_config, CONFIG)
 
-    return load_config(DERIVED_CONFIG_DEFAULTS, CONFIG)
+    return load_config(DYNAMIC_CONFIG_SCHEMA, CONFIG)
 
+# add all final config values in CONFIG to globals in this file
 CONFIG = load_all_config()
 globals().update(CONFIG)
+# this lets us do:  from .config import DEBUG, MEDIA_TIMEOUT, ...
 
 
-############################## Importable Checkers #############################
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+# ******************************************************************************
+
+
+
+########################### System Environment Setup ###########################
+
+
+# Set timezone to UTC and umask to OUTPUT_PERMISSIONS
+os.environ["TZ"] = 'UTC'
+os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8))  # noqa: F821
+
+# add ./node_modules/.bin to $PATH so we can use node scripts in extractors
+NODE_BIN_PATH = str((Path(CONFIG["OUTPUT_DIR"]).absolute() / 'node_modules' / '.bin'))
+sys.path.append(NODE_BIN_PATH)
+
+# disable stderr "you really shouldnt disable ssl" warnings with library config
+if not CONFIG['CHECK_SSL_VALIDITY']:
+    import urllib3
+    import requests
+    requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)
+    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+
+########################### Config Validity Checkers ###########################
+
 
 def check_system_config(config: ConfigDict=CONFIG) -> None:
     ### Check system environment
     if config['USER'] == 'root':
         stderr('[!] ArchiveBox should never be run as root!', color='red')
         stderr('    For more information, see the security overview documentation:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Security-Overview#do-not-run-as-root')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root')
         raise SystemExit(2)
 
     ### Check Python environment
     if sys.version_info[:3] < (3, 6, 0):
         stderr(f'[X] Python version is not new enough: {config["PYTHON_VERSION"]} (>3.6 is required)', color='red')
-        stderr('    See https://github.com/pirate/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.')
+        stderr('    See https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python for help upgrading your Python installation.')
         raise SystemExit(2)
 
     if config['PYTHON_ENCODING'] not in ('UTF-8', 'UTF8'):
@@ -752,69 +952,73 @@ def check_system_config(config: ConfigDict=CONFIG) -> None:
     # stderr('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY))
     # stderr('[i] Using Chrome data dir: {}'.format(os.path.abspath(CHROME_USER_DATA_DIR)))
     if config['CHROME_USER_DATA_DIR'] is not None:
-        if not os.path.exists(os.path.join(config['CHROME_USER_DATA_DIR'], 'Default')):
+        if not (Path(config['CHROME_USER_DATA_DIR']) / 'Default').exists():
             stderr('[X] Could not find profile "Default" in CHROME_USER_DATA_DIR.', color='red')
             stderr(f'    {config["CHROME_USER_DATA_DIR"]}')
             stderr('    Make sure you set it to a Chrome user data directory containing a Default profile folder.')
             stderr('    For more info see:')
-            stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR')
-            if 'Default' in config['CHROME_USER_DATA_DIR']:
+            stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#CHROME_USER_DATA_DIR')
+            if '/Default' in str(config['CHROME_USER_DATA_DIR']):
                 stderr()
                 stderr('    Try removing /Default from the end e.g.:')
                 stderr('        CHROME_USER_DATA_DIR="{}"'.format(config['CHROME_USER_DATA_DIR'].split('/Default')[0]))
             raise SystemExit(2)
 
+
 def check_dependencies(config: ConfigDict=CONFIG, show_help: bool=True) -> None:
-    invalid = [
-        '{}: {} ({})'.format(name, info['path'] or 'unable to find binary', info['version'] or 'unable to detect version')
-        for name, info in config['DEPENDENCIES'].items()
+    invalid_dependencies = [
+        (name, info) for name, info in config['DEPENDENCIES'].items()
         if info['enabled'] and not info['is_valid']
     ]
-
-    if invalid:
-        stderr('[X] Missing some required dependencies.', color='red')
-        stderr()
-        stderr('    {}'.format('\n    '.join(invalid)))
-        if show_help:
-            stderr()
-            stderr('    To get more info on dependency status run:')
-            stderr('        archivebox --version')
-        raise SystemExit(2)
+    if invalid_dependencies and show_help:
+        stderr(f'[!] Warning: Missing {len(invalid_dependencies)} recommended dependencies', color='lightyellow')
+        for dependency, info in invalid_dependencies:
+            stderr(
+                '    ! {}: {} ({})'.format(
+                    dependency,
+                    info['path'] or 'unable to find binary',
+                    info['version'] or 'unable to detect version',
+                )
+            )
+            if dependency in ('SINGLEFILE_BINARY', 'READABILITY_BINARY', 'MERCURY_BINARY'):
+                hint(('npm install --prefix . "git+https://github.com/ArchiveBox/ArchiveBox.git"',
+                    f'or archivebox config --set SAVE_{dependency.rsplit("_", 1)[0]}=False to silence this warning',
+                    ''), prefix='      ')
+        stderr('')
 
     if config['TIMEOUT'] < 5:
-        stderr()
         stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red')
         stderr('    You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.')
         stderr('    (Setting it to somewhere between 30 and 3000 seconds is recommended)')
         stderr()
         stderr('    If you want to make ArchiveBox run faster, disable specific archive methods instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr()
 
     elif config['USE_CHROME'] and config['TIMEOUT'] < 15:
-        stderr()
         stderr(f'[!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={config["TIMEOUT"]} seconds)', color='red')
         stderr('    Chrome will fail to archive all sites if set to less than ~15 seconds.')
         stderr('    (Setting it to somewhere between 30 and 300 seconds is recommended)')
         stderr()
         stderr('    If you want to make ArchiveBox run faster, disable specific archive methods instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles')
+        stderr()
 
     if config['USE_YOUTUBEDL'] and config['MEDIA_TIMEOUT'] < 20:
-        stderr()
         stderr(f'[!] Warning: MEDIA_TIMEOUT is set too low! (currently set to MEDIA_TIMEOUT={config["MEDIA_TIMEOUT"]} seconds)', color='red')
         stderr('    Youtube-dl will fail to archive all media if set to less than ~20 seconds.')
         stderr('    (Setting it somewhere over 60 seconds is recommended)')
         stderr()
         stderr('    If you want to disable media archiving entirely, set SAVE_MEDIA=False instead:')
-        stderr('        https://github.com/pirate/ArchiveBox/wiki/Configuration#save_media')
-
+        stderr('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_media')
+        stderr()
         
-def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) -> None:
+def check_data_folder(out_dir: Union[str, Path, None]=None, config: ConfigDict=CONFIG) -> None:
     output_dir = out_dir or config['OUTPUT_DIR']
-    assert isinstance(output_dir, str)
+    assert isinstance(output_dir, (str, Path))
 
-    json_index_exists = os.path.exists(os.path.join(output_dir, JSON_INDEX_FILENAME))
-    if not json_index_exists:
+    sql_index_exists = (Path(output_dir) / SQL_INDEX_FILENAME).exists()
+    if not sql_index_exists:
         stderr('[X] No archivebox index found in the current directory.', color='red')
         stderr(f'    {output_dir}', color='lightyellow')
         stderr()
@@ -826,8 +1030,7 @@ def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) ->
         stderr('        archivebox init')
         raise SystemExit(2)
 
-    sql_index_exists = os.path.exists(os.path.join(output_dir, SQL_INDEX_FILENAME))
-    from ..index.sql import list_migrations
+    from .index.sql import list_migrations
 
     pending_migrations = [name for status, name in list_migrations() if not status]
 
@@ -844,31 +1047,38 @@ def check_data_folder(out_dir: Optional[str]=None, config: ConfigDict=CONFIG) ->
         stderr('        archivebox init')
         raise SystemExit(3)
 
-    sources_dir = os.path.join(output_dir, SOURCES_DIR_NAME)
-    if not os.path.exists(sources_dir):
-        os.makedirs(sources_dir)
+    sources_dir = Path(output_dir) / SOURCES_DIR_NAME
+    if not sources_dir.exists():
+        sources_dir.mkdir()
 
 
 
-def setup_django(out_dir: str=None, check_db=False, config: ConfigDict=CONFIG) -> None:
+def setup_django(out_dir: Path=None, check_db=False, config: ConfigDict=CONFIG, in_memory_db=False) -> None:
     check_system_config()
     
-    output_dir = out_dir or config['OUTPUT_DIR']
+    output_dir = out_dir or Path(config['OUTPUT_DIR'])
 
-    assert isinstance(output_dir, str) and isinstance(config['PYTHON_DIR'], str)
+    assert isinstance(output_dir, Path) and isinstance(config['PACKAGE_DIR'], Path)
 
     try:
         import django
-        sys.path.append(config['PYTHON_DIR'])
-        os.environ.setdefault('OUTPUT_DIR', output_dir)
+        sys.path.append(str(config['PACKAGE_DIR']))
+        os.environ.setdefault('OUTPUT_DIR', str(output_dir))
+        assert (config['PACKAGE_DIR'] / 'core' / 'settings.py').exists(), 'settings.py was not found at archivebox/core/settings.py'
         os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings')
-        django.setup()
+
+        if in_memory_db:
+            # Put the db in memory and run migrations in case any command requires it
+            from django.core.management import call_command
+            os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:")
+            django.setup()
+            call_command("migrate", interactive=False, verbosity=0)
+        else:
+            django.setup()
 
         if check_db:
-            sql_index_path = os.path.join(output_dir, SQL_INDEX_FILENAME)
-            assert os.path.exists(sql_index_path), (
+            sql_index_path = Path(output_dir) / SQL_INDEX_FILENAME
+            assert sql_index_path.exists(), (
                 f'No database file {SQL_INDEX_FILENAME} found in OUTPUT_DIR: {config["OUTPUT_DIR"]}')
     except KeyboardInterrupt:
         raise SystemExit(2)
-
-os.umask(0o777 - int(OUTPUT_PERMISSIONS, base=8))  # noqa: F821

archivebox/config_stubs.py

@@ -1,7 +1,9 @@
-from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any
+from pathlib import Path
+from typing import Optional, Dict, Union, Tuple, Callable, Pattern, Type, Any, List
 from mypy_extensions import TypedDict
 
 
+
 SimpleConfigValue = Union[str, bool, int, None, Pattern, Dict[str, Any]]
 SimpleConfigValueDict = Dict[str, SimpleConfigValue]
 SimpleConfigValueGetter = Callable[[], SimpleConfigValue]
@@ -29,34 +31,39 @@ class ConfigDict(BaseConfig, total=False):
     IS_TTY: bool
     USE_COLOR: bool
     SHOW_PROGRESS: bool
+    IN_DOCKER: bool
 
-    OUTPUT_DIR: str
-    CONFIG_FILE: str
+    PACKAGE_DIR: Path
+    OUTPUT_DIR: Path
+    CONFIG_FILE: Path
     ONLY_NEW: bool
     TIMEOUT: int
     MEDIA_TIMEOUT: int
     OUTPUT_PERMISSIONS: str
-    URL_BLACKLIST: Optional[str]
+    RESTRICT_FILE_NAMES: str
+    URL_BLACKLIST: str
 
-    SECRET_KEY: str
+    SECRET_KEY: Optional[str]
+    BIND_ADDR: str
     ALLOWED_HOSTS: str
     DEBUG: bool
     PUBLIC_INDEX: bool
     PUBLIC_SNAPSHOTS: bool
     FOOTER_INFO: str
-    ACTIVE_THEME: str
 
     SAVE_TITLE: bool
     SAVE_FAVICON: bool
     SAVE_WGET: bool
     SAVE_WGET_REQUISITES: bool
+    SAVE_SINGLEFILE: bool
+    SAVE_READABILITY: bool
+    SAVE_MERCURY: bool
     SAVE_PDF: bool
     SAVE_SCREENSHOT: bool
     SAVE_DOM: bool
     SAVE_WARC: bool
     SAVE_GIT: bool
     SAVE_MEDIA: bool
-    SAVE_PLAYLISTS: bool
     SAVE_ARCHIVE_DOT_ORG: bool
 
     RESOLUTION: str
@@ -65,58 +72,32 @@ class ConfigDict(BaseConfig, total=False):
     CURL_USER_AGENT: str
     WGET_USER_AGENT: str
     CHROME_USER_AGENT: str
-    COOKIES_FILE: Optional[str]
-    CHROME_USER_DATA_DIR: Optional[str]
+    COOKIES_FILE: Union[str, Path, None]
+    CHROME_USER_DATA_DIR: Union[str, Path, None]
     CHROME_HEADLESS: bool
     CHROME_SANDBOX: bool
 
     USE_CURL: bool
     USE_WGET: bool
+    USE_SINGLEFILE: bool
+    USE_READABILITY: bool
+    USE_MERCURY: bool
     USE_GIT: bool
     USE_CHROME: bool
     USE_YOUTUBEDL: bool
-
-    CURL_BINARY: Optional[str]
-    GIT_BINARY: Optional[str]
-    WGET_BINARY: Optional[str]
-    YOUTUBEDL_BINARY: Optional[str]
+    CURL_BINARY: str
+    GIT_BINARY: str
+    WGET_BINARY: str
+    SINGLEFILE_BINARY: str
+    READABILITY_BINARY: str
+    MERCURY_BINARY: str
+    YOUTUBEDL_BINARY: str
     CHROME_BINARY: Optional[str]
 
-    TERM_WIDTH: Callable[[], int]
-    USER: str
-    ANSI: Dict[str, str]
-    REPO_DIR: str
-    PYTHON_DIR: str
-    TEMPLATES_DIR: str
-    ARCHIVE_DIR: str
-    SOURCES_DIR: str
-    LOGS_DIR: str
-
-    URL_BLACKLIST_PTN: Optional[Pattern]
-    WGET_AUTO_COMPRESSION: bool
-
-    ARCHIVEBOX_BINARY: str
-    VERSION: str
-    GIT_SHA: str
-
-    PYTHON_BINARY: str
-    PYTHON_ENCODING: str
-    PYTHON_VERSION: str
-
-    DJANGO_BINARY: str
-    DJANGO_VERSION: str
-
-    CURL_VERSION: str
-    WGET_VERSION: str
-    YOUTUBEDL_VERSION: str
-    GIT_VERSION: str
-    CHROME_VERSION: str
-
-    DEPENDENCIES: Dict[str, SimpleConfigValueDict]
-    CODE_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    CONFIG_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    DATA_LOCATIONS: Dict[str, SimpleConfigValueDict]
-    CHROME_OPTIONS: Dict[str, SimpleConfigValue]
+    YOUTUBEDL_ARGS: List[str]
+    WGET_ARGS: List[str]
+    CURL_ARGS: List[str]
+    GIT_ARGS: List[str]
 
 
 ConfigDefaultValueGetter = Callable[[ConfigDict], ConfigValue]

archivebox/core/admin.py

@@ -2,7 +2,6 @@ __package__ = 'archivebox.core'
 
 from io import StringIO
 from contextlib import redirect_stdout
-from pathlib import Path
 
 from django.contrib import admin
 from django.urls import path
@@ -10,16 +9,31 @@ from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.shortcuts import render, redirect
 from django.contrib.auth import get_user_model
+from django import forms
 
-from core.models import Snapshot
-from core.forms import AddLinkForm
+from ..util import htmldecode, urldecode, ansi_to_html
 
-from util import htmldecode, urldecode, ansi_to_html
+from core.models import Snapshot, Tag
+from core.forms import AddLinkForm, TagField
+
+from core.mixins import SearchResultsAdminMixin
+
+from index.html import snapshot_icons
 from logging_util import printable_filesize
 from main import add, remove
 from config import OUTPUT_DIR
 from extractors import archive_links
 
+# Admin URLs
+# /admin/
+# /admin/login/
+# /admin/core/
+# /admin/core/snapshot/
+# /admin/core/snapshot/:uuid/
+# /admin/core/tag/
+# /admin/core/tag/:uuid/
+
+
 # TODO: https://stackoverflow.com/questions/40760880/add-custom-button-to-django-admin-panel
 
 def update_snapshots(modeladmin, request, queryset):
@@ -33,7 +47,7 @@ def update_titles(modeladmin, request, queryset):
     archive_links([
         snapshot.as_link()
         for snapshot in queryset
-    ], overwrite=True, methods=('title',), out_dir=OUTPUT_DIR)
+    ], overwrite=True, methods=('title','favicon'), out_dir=OUTPUT_DIR)
 update_titles.short_description = "Pull title"
 
 def overwrite_snapshots(modeladmin, request, queryset):
@@ -50,21 +64,62 @@ def verify_snapshots(modeladmin, request, queryset):
 verify_snapshots.short_description = "Check"
 
 def delete_snapshots(modeladmin, request, queryset):
-    remove(links=[snapshot.as_link() for snapshot in queryset], yes=True, delete=True, out_dir=OUTPUT_DIR)
+    remove(snapshots=queryset, yes=True, delete=True, out_dir=OUTPUT_DIR)
 
 delete_snapshots.short_description = "Delete"
 
 
-class SnapshotAdmin(admin.ModelAdmin):
-    list_display = ('added', 'title_str', 'url_str', 'files', 'size', 'updated')
-    sort_fields = ('title_str', 'url_str', 'added', 'updated')
+class SnapshotAdminForm(forms.ModelForm):
+    tags = TagField(required=False)
+
+    class Meta:
+        model = Snapshot
+        fields = "__all__"
+
+    def save(self, commit=True):
+        # Based on: https://stackoverflow.com/a/49933068/3509554
+
+        # Get the unsave instance
+        instance = forms.ModelForm.save(self, False)
+        tags = self.cleaned_data.pop("tags")
+
+        #update save_m2m
+        def new_save_m2m():
+            instance.save_tags(tags)
+
+        # Do we need to save all changes now?
+        self.save_m2m = new_save_m2m
+        if commit:
+            instance.save()
+
+        return instance
+
+
+class SnapshotAdmin(SearchResultsAdminMixin, admin.ModelAdmin):
+    list_display = ('added', 'title_str', 'url_str', 'files', 'size')
+    sort_fields = ('title_str', 'url_str', 'added')
     readonly_fields = ('id', 'url', 'timestamp', 'num_outputs', 'is_archived', 'url_hash', 'added', 'updated')
-    search_fields = ('url', 'timestamp', 'title', 'tags')
-    fields = ('title', 'tags', *readonly_fields)
+    search_fields = ['url__icontains', 'timestamp', 'title', 'tags__name']
+    fields = (*readonly_fields, 'title', 'tags')
     list_filter = ('added', 'updated', 'tags')
     ordering = ['-added']
     actions = [delete_snapshots, overwrite_snapshots, update_snapshots, update_titles, verify_snapshots]
     actions_template = 'admin/actions_as_select.html'
+    form = SnapshotAdminForm
+    list_per_page = 40
+
+    def get_urls(self):
+        urls = super().get_urls()
+        custom_urls = [
+            path('grid/', self.admin_site.admin_view(self.grid_view),name='grid')
+        ]
+        return custom_urls + urls
+
+    def get_queryset(self, request):
+        return super().get_queryset(request).prefetch_related('tags')
+
+    def tag_list(self, obj):
+        return ', '.join(obj.tags.values_list('name', flat=True))
 
     def id_str(self, obj):
         return format_html(
@@ -75,65 +130,73 @@ class SnapshotAdmin(admin.ModelAdmin):
     def title_str(self, obj):
         canon = obj.as_link().canonical_outputs()
         tags = ''.join(
-            format_html('<span>{}</span>', tag.strip())
-            for tag in obj.tags.split(',')
-        ) if obj.tags else ''
+            format_html('<a href="/admin/core/snapshot/?tags__id__exact={}"><span class="tag">{}</span></a> ', tag.id, tag)
+            for tag in obj.tags.all()
+            if str(tag).strip()
+        )
         return format_html(
             '<a href="/{}">'
                 '<img src="/{}/{}" class="favicon" onerror="this.remove()">'
             '</a>'
-            '<a href="/{}/{}">'
+            '<a href="/{}/index.html">'
                 '<b class="status-{}">{}</b>'
             '</a>',
             obj.archive_path,
             obj.archive_path, canon['favicon_path'],
-            obj.archive_path, canon['wget_path'] or '',
+            obj.archive_path,
             'fetched' if obj.latest_title or obj.title else 'pending',
             urldecode(htmldecode(obj.latest_title or obj.title or ''))[:128] or 'Pending...'
-        ) + mark_safe(f'<span class="tags">{tags}</span>')
+        ) + mark_safe(f' <span class="tags">{tags}</span>')
 
     def files(self, obj):
-        link = obj.as_link()
-        canon = link.canonical_outputs()
-        out_dir = Path(link.link_dir)
-
-        link_tuple = lambda link, method: (link.archive_path, canon[method], canon[method] and (out_dir / canon[method]).exists())
-
-        return format_html(
-            '<span class="files-icons" style="font-size: 1.2em; opacity: 0.8">'
-                '<a href="/{}/{}/" class="exists-{}" title="Wget clone">🌐 </a> '
-                '<a href="/{}/{}" class="exists-{}" title="PDF">📄</a> '
-                '<a href="/{}/{}" class="exists-{}" title="Screenshot">🖥 </a> '
-                '<a href="/{}/{}" class="exists-{}" title="HTML dump">🅷 </a> '
-                '<a href="/{}/{}/" class="exists-{}" title="WARC">🆆 </a> '
-                '<a href="/{}/{}/" class="exists-{}" title="Media files">📼 </a> '
-                '<a href="/{}/{}/" class="exists-{}" title="Git repos">📦 </a> '
-                '<a href="{}" class="exists-{}" title="Archive.org snapshot">🏛 </a> '
-            '</span>',
-            *link_tuple(link, 'wget_path'),
-            *link_tuple(link, 'pdf_path'),
-            *link_tuple(link, 'screenshot_path'),
-            *link_tuple(link, 'dom_path'),
-            *link_tuple(link, 'warc_path')[:2], any((out_dir / canon['warc_path']).glob('*.warc.gz')),
-            *link_tuple(link, 'media_path')[:2], any((out_dir / canon['media_path']).glob('*')),
-            *link_tuple(link, 'git_path')[:2], any((out_dir / canon['git_path']).glob('*')),
-            canon['archive_org_path'], (out_dir / 'archive.org.txt').exists(),
-        )
+        return snapshot_icons(obj)
 
     def size(self, obj):
+        archive_size = obj.archive_size
+        if archive_size:
+            size_txt = printable_filesize(archive_size)
+            if archive_size > 52428800:
+                size_txt = mark_safe(f'<b>{size_txt}</b>')
+        else:
+            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
         return format_html(
             '<a href="/{}" title="View all files">{}</a>',
             obj.archive_path,
-            printable_filesize(obj.archive_size) if obj.archive_size else 'pending',
+            size_txt,
         )
 
     def url_str(self, obj):
         return format_html(
-            '<a href="{}">{}</a>',
+            '<a href="{}"><code>{}</code></a>',
             obj.url,
             obj.url.split('://www.', 1)[-1].split('://', 1)[-1][:64],
         )
 
+    def grid_view(self, request):
+
+        # cl = self.get_changelist_instance(request)
+
+        # Save before monkey patching to restore for changelist list view
+        saved_change_list_template = self.change_list_template
+        saved_list_per_page = self.list_per_page
+        saved_list_max_show_all = self.list_max_show_all
+
+        # Monkey patch here plus core_tags.py
+        self.change_list_template = 'private_index_grid.html'
+        self.list_per_page = 20
+        self.list_max_show_all = self.list_per_page
+
+        # Call monkey patched view
+        rendered_response = self.changelist_view(request)
+
+        # Restore values
+        self.change_list_template =  saved_change_list_template
+        self.list_per_page = saved_list_per_page
+        self.list_max_show_all = saved_list_max_show_all
+
+        return rendered_response
+        
+
     id_str.short_description = 'ID'
     title_str.short_description = 'Title'
     url_str.short_description = 'Original URL'
@@ -142,6 +205,12 @@ class SnapshotAdmin(admin.ModelAdmin):
     title_str.admin_order_field = 'title'
     url_str.admin_order_field = 'url'
 
+class TagAdmin(admin.ModelAdmin):
+    list_display = ('slug', 'name', 'id')
+    sort_fields = ('id', 'name', 'slug')
+    readonly_fields = ('id',)
+    search_fields = ('id', 'name', 'slug')
+    fields = (*readonly_fields, 'name', 'slug')
 
 
 class ArchiveBoxAdmin(admin.AdminSite):
@@ -191,10 +260,10 @@ class ArchiveBoxAdmin(admin.AdminSite):
             else:
                 context["form"] = form
 
-        return render(template_name='add_links.html', request=request, context=context)
-
+        return render(template_name='add.html', request=request, context=context)
 
 admin.site = ArchiveBoxAdmin()
 admin.site.register(get_user_model())
 admin.site.register(Snapshot, SnapshotAdmin)
+admin.site.register(Tag, TagAdmin)
 admin.site.disable_action('delete_selected')

archivebox/core/forms.py

@@ -3,12 +3,87 @@ __package__ = 'archivebox.core'
 from django import forms
 
 from ..util import URL_REGEX
+from ..vendor.taggit_utils import edit_string_for_tags, parse_tags
 
 CHOICES = (
     ('0', 'depth = 0 (archive just these URLs)'),
     ('1', 'depth = 1 (archive these URLs and all URLs one hop away)'),
 )
 
+from ..extractors import get_default_archive_methods
+
+ARCHIVE_METHODS = [
+    (name, name)
+    for name, _, _ in get_default_archive_methods()
+]
+
+
 class AddLinkForm(forms.Form):
     url = forms.RegexField(label="URLs (one per line)", regex=URL_REGEX, min_length='6', strip=True, widget=forms.Textarea, required=True)
     depth = forms.ChoiceField(label="Archive depth", choices=CHOICES, widget=forms.RadioSelect, initial='0')
+    archive_methods = forms.MultipleChoiceField(
+        label="Archive methods (select at least 1, otherwise all will be used by default)",
+        required=False,
+        widget=forms.SelectMultiple,
+        choices=ARCHIVE_METHODS,
+    )
+    # TODO: hook these up to the view and put them 
+    # in a collapsible UI section labeled "Advanced"
+    #
+    # exclude_patterns = forms.CharField(
+    #     label="Exclude patterns",
+    #     min_length='1',
+    #     required=False,
+    #     initial=URL_BLACKLIST,
+    # )
+    # timeout = forms.IntegerField(
+    #     initial=TIMEOUT,
+    # )
+    # overwrite = forms.BooleanField(
+    #     label="Overwrite any existing Snapshots",
+    #     initial=False,
+    # )
+    # index_only = forms.BooleanField(
+    #     label="Add URLs to index without Snapshotting",
+    #     initial=False,
+    # )
+
+class TagWidgetMixin:
+    def format_value(self, value):
+        if value is not None and not isinstance(value, str):
+            value = edit_string_for_tags(value)
+        return super().format_value(value)
+
+class TagWidget(TagWidgetMixin, forms.TextInput):
+    pass
+
+class TagField(forms.CharField):
+    widget = TagWidget
+
+    def clean(self, value):
+        value = super().clean(value)
+        try:
+            return parse_tags(value)
+        except ValueError:
+            raise forms.ValidationError(
+                "Please provide a comma-separated list of tags."
+            )
+
+    def has_changed(self, initial_value, data_value):
+        # Always return False if the field is disabled since self.bound_data
+        # always uses the initial value in this case.
+        if self.disabled:
+            return False
+
+        try:
+            data_value = self.clean(data_value)
+        except forms.ValidationError:
+            pass
+
+        if initial_value is None:
+            initial_value = []
+
+        initial_value = [tag.name for tag in initial_value]
+        initial_value.sort()
+
+        return initial_value != data_value

archivebox/core/migrations/0006_auto_20201012_1520.py

@@ -0,0 +1,70 @@
+# Generated by Django 3.0.8 on 2020-10-12 15:20
+
+from django.db import migrations, models
+from django.utils.text import slugify
+
+def forwards_func(apps, schema_editor):
+    SnapshotModel = apps.get_model("core", "Snapshot")
+    TagModel = apps.get_model("core", "Tag")
+
+    db_alias = schema_editor.connection.alias
+    snapshots = SnapshotModel.objects.all()
+    for snapshot in snapshots:
+        tags = snapshot.tags
+        tag_set = (
+            set(tag.strip() for tag in (snapshot.tags_old or '').split(','))
+        )
+        tag_set.discard("")
+
+        for tag in tag_set:
+            to_add, _ = TagModel.objects.get_or_create(name=tag, slug=slugify(tag))
+            snapshot.tags.add(to_add)
+
+
+def reverse_func(apps, schema_editor):
+    SnapshotModel = apps.get_model("core", "Snapshot")
+    TagModel = apps.get_model("core", "Tag")
+
+    db_alias = schema_editor.connection.alias
+    snapshots = SnapshotModel.objects.all()
+    for snapshot in snapshots:
+        tags = snapshot.tags.values_list("name", flat=True)
+        snapshot.tags_old = ",".join([tag for tag in tags])
+        snapshot.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0005_auto_20200728_0326'),
+    ]
+
+    operations = [
+        migrations.RenameField(
+            model_name='snapshot',
+            old_name='tags',
+            new_name='tags_old',
+        ),
+        migrations.CreateModel(
+            name='Tag',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('name', models.CharField(max_length=100, unique=True, verbose_name='name')),
+                ('slug', models.SlugField(max_length=100, unique=True, verbose_name='slug')),
+            ],
+            options={
+                'verbose_name': 'Tag',
+                'verbose_name_plural': 'Tags',
+            },
+        ),
+        migrations.AddField(
+            model_name='snapshot',
+            name='tags',
+            field=models.ManyToManyField(to='core.Tag'),
+        ),
+        migrations.RunPython(forwards_func, reverse_func),
+        migrations.RemoveField(
+            model_name='snapshot',
+            name='tags_old',
+        ),
+    ]

archivebox/core/migrations/0007_archiveresult.py

@@ -0,0 +1,97 @@
+# Generated by Django 3.0.8 on 2020-11-04 12:25
+
+import json
+from pathlib import Path
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+from config import CONFIG
+from index.json import to_json
+
+try:
+    JSONField = models.JSONField
+except AttributeError:
+    import jsonfield
+    JSONField = jsonfield.JSONField
+
+
+def forwards_func(apps, schema_editor):
+    from core.models import EXTRACTORS
+
+    Snapshot = apps.get_model("core", "Snapshot")
+    ArchiveResult = apps.get_model("core", "ArchiveResult")
+
+    snapshots = Snapshot.objects.all()
+    for snapshot in snapshots:
+        out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
+
+        try:
+            with open(out_dir / "index.json", "r") as f:
+                fs_index = json.load(f)
+        except Exception as e:
+            continue
+
+        history = fs_index["history"]
+
+        for extractor in history:
+            for result in history[extractor]:
+                ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"] or 'unknown', 
+                start_ts=result["start_ts"], end_ts=result["end_ts"], status=result["status"], pwd=result["pwd"], output=result["output"])
+
+
+def verify_json_index_integrity(snapshot):
+    results = snapshot.archiveresult_set.all()
+    out_dir = Path(CONFIG['ARCHIVE_DIR']) / snapshot.timestamp
+    with open(out_dir / "index.json", "r") as f:
+        index = json.load(f)
+
+    history = index["history"]
+    index_results = [result for extractor in history for result in history[extractor]]
+    flattened_results = [result["start_ts"] for result in index_results]
+    
+    missing_results = [result for result in results if result.start_ts.isoformat() not in flattened_results]
+
+    for missing in missing_results:
+        index["history"][missing.extractor].append({"cmd": missing.cmd, "cmd_version": missing.cmd_version, "end_ts": missing.end_ts.isoformat(),
+                                                    "start_ts": missing.start_ts.isoformat(), "pwd": missing.pwd, "output": missing.output,
+                                                    "schema": "ArchiveResult", "status": missing.status})
+
+    json_index = to_json(index)
+    with open(out_dir / "index.json", "w") as f:
+        f.write(json_index)
+
+
+def reverse_func(apps, schema_editor):
+    Snapshot = apps.get_model("core", "Snapshot")
+    ArchiveResult = apps.get_model("core", "ArchiveResult")
+    for snapshot in Snapshot.objects.all():
+        verify_json_index_integrity(snapshot)
+
+    ArchiveResult.objects.all().delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0006_auto_20201012_1520'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ArchiveResult',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('cmd', JSONField()),
+                ('pwd', models.CharField(max_length=256)),
+                ('cmd_version', models.CharField(max_length=32)),
+                ('status', models.CharField(choices=[('succeeded', 'succeeded'), ('failed', 'failed'), ('skipped', 'skipped')], max_length=16)),
+                ('output', models.CharField(max_length=512)),
+                ('start_ts', models.DateTimeField()),
+                ('end_ts', models.DateTimeField()),
+                ('extractor', models.CharField(choices=[('title', 'title'), ('favicon', 'favicon'), ('wget', 'wget'), ('singlefile', 'singlefile'), ('pdf', 'pdf'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('readability', 'readability'), ('mercury', 'mercury'), ('git', 'git'), ('media', 'media'), ('headers', 'headers'), ('archive_org', 'archive_org')], max_length=32)),
+                ('snapshot', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Snapshot')),
+            ],
+        ),
+        migrations.RunPython(forwards_func, reverse_func),
+    ]

archivebox/core/migrations/0008_auto_20210105_1421.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.3 on 2021-01-05 14:21
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0007_archiveresult'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='cmd_version',
+            field=models.CharField(blank=True, default=None, max_length=32, null=True),
+        ),
+    ]

archivebox/core/mixins.py

@@ -0,0 +1,23 @@
+from django.contrib import messages
+
+from archivebox.search import query_search_index
+
+class SearchResultsAdminMixin(object):
+    def get_search_results(self, request, queryset, search_term):
+        ''' Enhances the search queryset with results from the search backend.
+        '''
+        qs, use_distinct = \
+            super(SearchResultsAdminMixin, self).get_search_results(
+                request, queryset, search_term)
+
+        search_term = search_term.strip()
+        if not search_term:
+            return qs, use_distinct
+        try:
+            qsearch = query_search_index(search_term)
+        except Exception as err:
+            messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}')
+        else:
+            qs = queryset & qsearch
+        finally:
+            return qs, use_distinct

archivebox/core/models.py

@@ -2,11 +2,69 @@ __package__ = 'archivebox.core'
 
 import uuid
 
-from django.db import models
+from django.db import models, transaction
 from django.utils.functional import cached_property
+from django.utils.text import slugify
+from django.db.models import Case, When, Value, IntegerField
 
 from ..util import parse_date
 from ..index.schema import Link
+from ..extractors import get_default_archive_methods, ARCHIVE_METHODS_INDEXING_PRECEDENCE
+
+EXTRACTORS = [(extractor[0], extractor[0]) for extractor in get_default_archive_methods()]
+STATUS_CHOICES = [
+    ("succeeded", "succeeded"),
+    ("failed", "failed"),
+    ("skipped", "skipped")
+]
+
+try:
+    JSONField = models.JSONField
+except AttributeError:
+    import jsonfield
+    JSONField = jsonfield.JSONField
+
+
+class Tag(models.Model):
+    """
+    Based on django-taggit model
+    """
+    name = models.CharField(verbose_name="name", unique=True, blank=False, max_length=100)
+    slug = models.SlugField(verbose_name="slug", unique=True, max_length=100)
+
+    class Meta:
+        verbose_name = "Tag"
+        verbose_name_plural = "Tags"
+
+    def __str__(self):
+        return self.name
+
+    def slugify(self, tag, i=None):
+        slug = slugify(tag)
+        if i is not None:
+            slug += "_%d" % i
+        return slug
+
+    def save(self, *args, **kwargs):
+        if self._state.adding and not self.slug:
+            self.slug = self.slugify(self.name)
+
+            with transaction.atomic():
+                slugs = set(
+                    type(self)
+                    ._default_manager.filter(slug__startswith=self.slug)
+                    .values_list("slug", flat=True)
+                )
+
+                i = None
+                while True:
+                    slug = self.slugify(self.name, i)
+                    if slug not in slugs:
+                        self.slug = slug
+                        return super().save(*args, **kwargs)
+                    i = 1 if i is None else i+1
+        else:
+            return super().save(*args, **kwargs)
 
 
 class Snapshot(models.Model):
@@ -16,11 +74,10 @@ class Snapshot(models.Model):
     timestamp = models.CharField(max_length=32, unique=True, db_index=True)
 
     title = models.CharField(max_length=128, null=True, blank=True, db_index=True)
-    tags = models.CharField(max_length=256, null=True, blank=True, db_index=True)
 
     added = models.DateTimeField(auto_now_add=True, db_index=True)
     updated = models.DateTimeField(null=True, blank=True, db_index=True)
-    # bookmarked = models.DateTimeField()
+    tags = models.ManyToManyField(Tag)
 
     keys = ('url', 'timestamp', 'title', 'tags', 'updated')
 
@@ -41,12 +98,20 @@ class Snapshot(models.Model):
         args = args or self.keys
         return {
             key: getattr(self, key)
+            if key != 'tags' else self.tags_str()
             for key in args
         }
 
     def as_link(self) -> Link:
         return Link.from_json(self.as_json())
 
+    def as_link_with_details(self) -> Link:
+        from ..index import load_link_details
+        return load_link_details(self.as_link())
+
+    def tags_str(self) -> str:
+        return ','.join(self.tags.order_by('name').values_list('name', flat=True))
+
     @cached_property
     def bookmarked(self):
         return parse_date(self.timestamp)
@@ -57,7 +122,7 @@ class Snapshot(models.Model):
 
     @cached_property
     def num_outputs(self):
-        return self.as_link().num_outputs
+        return self.archiveresult_set.filter(status='succeeded').count()
 
     @cached_property
     def url_hash(self):
@@ -81,8 +146,8 @@ class Snapshot(models.Model):
 
     @cached_property
     def history(self):
-        from ..index import load_link_details
-        return load_link_details(self.as_link()).history
+        # TODO: use ArchiveResult for this instead of json
+        return self.as_link_with_details().history
 
     @cached_property
     def latest_title(self):
@@ -92,3 +157,38 @@ class Snapshot(models.Model):
             and self.history['title'][-1].output.strip()):
             return self.history['title'][-1].output.strip()
         return None
+
+    def save_tags(self, tags=()):
+        tags_id = []
+        for tag in tags:
+            tags_id.append(Tag.objects.get_or_create(name=tag)[0].id)
+        self.tags.clear()
+        self.tags.add(*tags_id)
+
+
+class ArchiveResultManager(models.Manager):
+    def indexable(self, sorted: bool = True):
+        INDEXABLE_METHODS = [ r[0] for r in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
+        qs = self.get_queryset().filter(extractor__in=INDEXABLE_METHODS,status='succeeded')
+
+        if sorted:
+            precedence = [ When(extractor=method, then=Value(precedence)) for method, precedence in ARCHIVE_METHODS_INDEXING_PRECEDENCE ]
+            qs = qs.annotate(indexing_precedence=Case(*precedence, default=Value(1000),output_field=IntegerField())).order_by('indexing_precedence')
+        return qs
+
+
+class ArchiveResult(models.Model):
+    snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE)
+    cmd = JSONField()
+    pwd = models.CharField(max_length=256)
+    cmd_version = models.CharField(max_length=32, default=None, null=True, blank=True)
+    output = models.CharField(max_length=512)
+    start_ts = models.DateTimeField()
+    end_ts = models.DateTimeField()
+    status = models.CharField(max_length=16, choices=STATUS_CHOICES)
+    extractor = models.CharField(choices=EXTRACTORS, max_length=32)
+
+    objects = ArchiveResultManager()
+
+    def __str__(self):
+        return self.extractor

archivebox/core/settings.py

@@ -2,23 +2,38 @@ __package__ = 'archivebox.core'
 
 import os
 import sys
-from django.utils.crypto import get_random_string
 
+from pathlib import Path
+from django.utils.crypto import get_random_string
 
 from ..config import (                                                          # noqa: F401
     DEBUG,
     SECRET_KEY,
     ALLOWED_HOSTS,
-    PYTHON_DIR,
-    ACTIVE_THEME,
+    PACKAGE_DIR,
+    TEMPLATES_DIR_NAME,
     SQL_INDEX_FILENAME,
     OUTPUT_DIR,
 )
 
-ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
+
+IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3]
+IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ
 IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3]
 
-SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.')
+################################################################################
+### Django Core Settings
+################################################################################
+
+WSGI_APPLICATION = 'core.wsgi.application'
+ROOT_URLCONF = 'core.urls'
+
+LOGIN_URL = '/accounts/login/'
+LOGOUT_REDIRECT_URL = '/'
+PASSWORD_RESET_URL = '/accounts/password_reset/'
+APPEND_SLASH = True
+
+DEBUG = DEBUG or ('--debug' in sys.argv)
 
 INSTALLED_APPS = [
     'django.contrib.auth',
@@ -43,16 +58,31 @@ MIDDLEWARE = [
     'django.contrib.messages.middleware.MessageMiddleware',
 ]
 
-ROOT_URLCONF = 'core.urls'
-APPEND_SLASH = True
+AUTHENTICATION_BACKENDS = [
+    'django.contrib.auth.backends.ModelBackend',
+]
+
+
+################################################################################
+### Staticfile and Template Settings
+################################################################################
+
+STATIC_URL = '/static/'
+
+STATICFILES_DIRS = [
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'static'),
+]
+
+TEMPLATE_DIRS = [
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'core'),
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME / 'admin'),
+    str(Path(PACKAGE_DIR) / TEMPLATES_DIR_NAME),
+]
+
 TEMPLATES = [
     {
         'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': [
-            os.path.join(PYTHON_DIR, 'themes', ACTIVE_THEME),
-            os.path.join(PYTHON_DIR, 'themes', 'default'),
-            os.path.join(PYTHON_DIR, 'themes'),
-        ],
+        'DIRS': TEMPLATE_DIRS,
         'APP_DIRS': True,
         'OPTIONS': {
             'context_processors': [
@@ -65,18 +95,42 @@ TEMPLATES = [
     },
 ]
 
-WSGI_APPLICATION = 'core.wsgi.application'
+
+################################################################################
+### External Service Settings
+################################################################################
+
+DATABASE_FILE = Path(OUTPUT_DIR) / SQL_INDEX_FILENAME
+DATABASE_NAME = os.environ.get("ARCHIVEBOX_DATABASE_NAME", str(DATABASE_FILE))
 
 DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': os.path.join(OUTPUT_DIR, SQL_INDEX_FILENAME),
+        'NAME': DATABASE_NAME,
     }
 }
 
-AUTHENTICATION_BACKENDS = [
-    'django.contrib.auth.backends.ModelBackend',
-]
+EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
+
+
+################################################################################
+### Security Settings
+################################################################################
+
+SECRET_KEY = SECRET_KEY or get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789-_+!.')
+
+ALLOWED_HOSTS = ALLOWED_HOSTS.split(',')
+
+SECURE_BROWSER_XSS_FILTER = True
+SECURE_CONTENT_TYPE_NOSNIFF = True
+
+CSRF_COOKIE_SECURE = False
+SESSION_COOKIE_SECURE = False
+SESSION_COOKIE_DOMAIN = None
+SESSION_COOKIE_AGE = 1209600  # 2 weeks
+SESSION_EXPIRE_AT_BROWSER_CLOSE = False
+SESSION_SAVE_EVERY_REQUEST = True
+
 AUTH_PASSWORD_VALIDATORS = [
     {'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator'},
     {'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator'},
@@ -84,30 +138,23 @@ AUTH_PASSWORD_VALIDATORS = [
     {'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator'},
 ]
 
-################################################################################
-### Security Settings
-################################################################################
-SECURE_BROWSER_XSS_FILTER = True
-SECURE_CONTENT_TYPE_NOSNIFF = True
-SESSION_COOKIE_SECURE = False
-CSRF_COOKIE_SECURE = False
-SESSION_COOKIE_DOMAIN = None
-SESSION_EXPIRE_AT_BROWSER_CLOSE = False
-SESSION_SAVE_EVERY_REQUEST = True
-SESSION_COOKIE_AGE = 1209600  # 2 weeks
-LOGIN_URL = '/accounts/login/'
-LOGOUT_REDIRECT_URL = '/'
-PASSWORD_RESET_URL = '/accounts/password_reset/'
 
+################################################################################
+### Shell Settings
+################################################################################
 
 SHELL_PLUS = 'ipython'
 SHELL_PLUS_PRINT_SQL = False
 IPYTHON_ARGUMENTS = ['--no-confirm-exit', '--no-banner']
 IPYTHON_KERNEL_DISPLAY_NAME = 'ArchiveBox Django Shell'
 if IS_SHELL:
-    os.environ['PYTHONSTARTUP'] = os.path.join(PYTHON_DIR, 'core', 'welcome_message.py')
+    os.environ['PYTHONSTARTUP'] = str(Path(PACKAGE_DIR) / 'core' / 'welcome_message.py')
 
 
+################################################################################
+### Internationalization & Localization Settings
+################################################################################
+
 LANGUAGE_CODE = 'en-us'
 TIME_ZONE = 'UTC'
 USE_I18N = False
@@ -116,12 +163,3 @@ USE_TZ = False
 
 DATETIME_FORMAT = 'Y-m-d g:iA'
 SHORT_DATETIME_FORMAT = 'Y-m-d h:iA'
-
-
-EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
-
-STATIC_URL = '/static/'
-STATICFILES_DIRS = [
-    os.path.join(PYTHON_DIR, 'themes', ACTIVE_THEME, 'static'),
-    os.path.join(PYTHON_DIR, 'themes', 'default', 'static'),
-]

archivebox/core/templatetags/core_tags.py

@@ -0,0 +1,47 @@
+from django import template
+from django.urls import reverse
+from django.contrib.admin.templatetags.base import InclusionAdminNode
+from django.templatetags.static import static
+
+
+from typing import Union
+
+from core.models import ArchiveResult
+
+register = template.Library()
+
+@register.simple_tag
+def snapshot_image(snapshot):
+    result = ArchiveResult.objects.filter(snapshot=snapshot, extractor='screenshot', status='succeeded').first()
+    if result:
+        return reverse('Snapshot', args=[f'{str(snapshot.timestamp)}/{result.output}'])
+    
+    return static('archive.png')
+
+@register.filter
+def file_size(num_bytes: Union[int, float]) -> str:
+    for count in ['Bytes','KB','MB','GB']:
+        if num_bytes > -1024.0 and num_bytes < 1024.0:
+            return '%3.1f %s' % (num_bytes, count)
+        num_bytes /= 1024.0
+    return '%3.1f %s' % (num_bytes, 'TB')
+
+def result_list(cl):
+    """
+    Monkey patched result
+    """
+    num_sorted_fields = 0
+    return {
+        'cl': cl,
+        'num_sorted_fields': num_sorted_fields,
+        'results': cl.result_list,
+    }
+
+@register.tag(name='snapshots_grid')
+def result_list_tag(parser, token):
+    return InclusionAdminNode(
+        parser, token,
+        func=result_list,
+        template_name='snapshots_grid.html',
+        takes_context=False,
+    )

archivebox/core/urls.py

@@ -5,20 +5,24 @@ from django.views import static
 from django.conf import settings
 from django.views.generic.base import RedirectView
 
-from core.views import MainIndex, OldIndex, LinkDetails
+from core.views import HomepageView, SnapshotView, PublicIndexView, AddView
 
 
 # print('DEBUG', settings.DEBUG)
 
 urlpatterns = [
+    path('public/', PublicIndexView.as_view(), name='public-index'),
+
     path('robots.txt', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'robots.txt'}),
     path('favicon.ico', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'favicon.ico'}),
 
-    path('docs/', RedirectView.as_view(url='https://github.com/pirate/ArchiveBox/wiki'), name='Docs'),
+    path('docs/', RedirectView.as_view(url='https://github.com/ArchiveBox/ArchiveBox/wiki'), name='Docs'),
 
     path('archive/', RedirectView.as_view(url='/')),
-    path('archive/<path:path>', LinkDetails.as_view(), name='LinkAssets'),
-    path('add/', RedirectView.as_view(url='/admin/core/snapshot/add/')),
+    path('archive/<path:path>', SnapshotView.as_view(), name='Snapshot'),
+
+    path('admin/core/snapshot/add/', RedirectView.as_view(url='/add/')),
+    path('add/', AddView.as_view(), name='add'),
     
     path('accounts/login/', RedirectView.as_view(url='/admin/login/')),
     path('accounts/logout/', RedirectView.as_view(url='/admin/logout/')),
@@ -27,8 +31,39 @@ urlpatterns = [
     path('accounts/', include('django.contrib.auth.urls')),
     path('admin/', admin.site.urls),
     
-    path('old.html', OldIndex.as_view(), name='OldHome'),
     path('index.html', RedirectView.as_view(url='/')),
     path('index.json', static.serve, {'document_root': settings.OUTPUT_DIR, 'path': 'index.json'}),
-    path('', MainIndex.as_view(), name='Home'),
+    path('', HomepageView.as_view(), name='Home'),
 ]
+
+    # # Proposed UI URLs spec
+    # path('',                 HomepageView)
+    # path('/add',             AddView)
+    # path('/public',          PublicIndexView)
+    # path('/snapshot/:slug',  SnapshotView)
+    
+    # path('/admin',           admin.site.urls)
+    # path('/accounts',        django.contrib.auth.urls)
+
+    # # Prposed REST API spec
+    # # :slugs can be uuid, short_uuid, or any of the unique index_fields
+    # path('api/v1/'),
+    # path('api/v1/core/'                      [GET])
+    # path('api/v1/core/snapshot/',            [GET, POST, PUT]),
+    # path('api/v1/core/snapshot/:slug',       [GET, PATCH, DELETE]),
+    # path('api/v1/core/archiveresult',        [GET, POST, PUT]),
+    # path('api/v1/core/archiveresult/:slug',  [GET, PATCH, DELETE]),
+    # path('api/v1/core/tag/',                 [GET, POST, PUT]),
+    # path('api/v1/core/tag/:slug',            [GET, PATCH, DELETE]),
+
+    # path('api/v1/cli/',                      [GET])
+    # path('api/v1/cli/{add,list,config,...}', [POST]),  # pass query as kwargs directly to `run_subcommand` and return stdout, stderr, exitcode
+
+    # path('api/v1/extractors/',                    [GET])
+    # path('api/v1/extractors/:extractor/',         [GET]),
+    # path('api/v1/extractors/:extractor/:func',    [GET, POST]),  # pass query as args directly to chosen function
+
+    # future, just an idea:
+    # path('api/v1/scheduler/',                [GET])
+    # path('api/v1/scheduler/task/',           [GET, POST, PUT]),
+    # path('api/v1/scheduler/task/:slug',      [GET, PATCH, DELETE]),

archivebox/core/views.py

@@ -1,59 +1,47 @@
 __package__ = 'archivebox.core'
 
+from io import StringIO
+from contextlib import redirect_stdout
+
 from django.shortcuts import render, redirect
 
 from django.http import HttpResponse
 from django.views import View, static
+from django.views.generic.list import ListView
+from django.views.generic import FormView
+from django.db.models import Q
+from django.contrib.auth.mixins import UserPassesTestMixin
 
 from core.models import Snapshot
+from core.forms import AddLinkForm
 
-from ..index import load_main_index, load_main_index_meta
 from ..config import (
     OUTPUT_DIR,
-    VERSION,
-    FOOTER_INFO,
     PUBLIC_INDEX,
     PUBLIC_SNAPSHOTS,
+    PUBLIC_ADD_VIEW,
+    VERSION,
+    FOOTER_INFO,
 )
-from ..util import base_url
+from main import add
+from ..util import base_url, ansi_to_html
+from ..index.html import snapshot_icons
 
 
-class MainIndex(View):
-    template = 'main_index.html'
-
+class HomepageView(View):
     def get(self, request):
         if request.user.is_authenticated:
             return redirect('/admin/core/snapshot/')
 
         if PUBLIC_INDEX:
-            return redirect('OldHome')
+            return redirect('/public')
         
         return redirect(f'/admin/login/?next={request.path}')
 
-        
 
-class OldIndex(View):
-    template = 'main_index.html'
+class SnapshotView(View):
+    # render static html index from filesystem archive/<timestamp>/index.html
 
-    def get(self, request):
-        if PUBLIC_INDEX or request.user.is_authenticated:
-            all_links = load_main_index(out_dir=OUTPUT_DIR)
-            meta_info = load_main_index_meta(out_dir=OUTPUT_DIR)
-
-            context = {
-                'updated': meta_info['updated'],
-                'num_links': meta_info['num_links'],
-                'links': all_links,
-                'VERSION': VERSION,
-                'FOOTER_INFO': FOOTER_INFO,
-            }
-
-            return render(template_name=self.template, request=request, context=context)
-
-        return redirect(f'/admin/login/?next={request.path}')
-
-
-class LinkDetails(View):
     def get(self, request, path):
         # missing trailing slash -> redirect to index
         if '/' not in path:
@@ -102,3 +90,85 @@ class LinkDetails(View):
             content_type="text/plain",
             status=404,
         )
+
+class PublicIndexView(ListView):
+    template_name = 'public_index.html'
+    model = Snapshot
+    paginate_by = 100
+    ordering = ['title']
+
+    def get_context_data(self, **kwargs):
+        return {
+            **super().get_context_data(**kwargs),
+            'VERSION': VERSION,
+            'FOOTER_INFO': FOOTER_INFO,
+        }
+
+    def get_queryset(self, **kwargs): 
+        qs = super().get_queryset(**kwargs) 
+        query = self.request.GET.get('q')
+        if query:
+            qs = qs.filter(Q(title__icontains=query) | Q(url__icontains=query) | Q(timestamp__icontains=query) | Q(tags__name__icontains=query))
+        for snapshot in qs:
+            snapshot.icons = snapshot_icons(snapshot)
+        return qs
+
+    def get(self, *args, **kwargs):
+        if PUBLIC_INDEX or self.request.user.is_authenticated:
+            response = super().get(*args, **kwargs)
+            return response
+        else:
+            return redirect(f'/admin/login/?next={self.request.path}')
+
+
+class AddView(UserPassesTestMixin, FormView):
+    template_name = "add.html"
+    form_class = AddLinkForm
+
+    def get_initial(self):
+        """Prefill the AddLinkForm with the 'url' GET parameter"""
+        if self.request.method == 'GET':
+            url = self.request.GET.get('url', None)
+            if url:
+                return {'url': url}
+        else:
+            return super().get_initial()
+
+    def test_func(self):
+        return PUBLIC_ADD_VIEW or self.request.user.is_authenticated
+
+    def get_context_data(self, **kwargs):
+        return {
+            **super().get_context_data(**kwargs),
+            'title': "Add URLs",
+            # We can't just call request.build_absolute_uri in the template, because it would include query parameters
+            'absolute_add_path': self.request.build_absolute_uri(self.request.path),
+            'VERSION': VERSION,
+            'FOOTER_INFO': FOOTER_INFO,
+        }
+
+    def form_valid(self, form):
+        url = form.cleaned_data["url"]
+        print(f'[+] Adding URL: {url}')
+        depth = 0 if form.cleaned_data["depth"] == "0" else 1
+        extractors = ','.join(form.cleaned_data["archive_methods"])
+        input_kwargs = {
+            "urls": url,
+            "depth": depth,
+            "update_all": False,
+            "out_dir": OUTPUT_DIR,
+        }
+        if extractors:
+            input_kwargs.update({"extractors": extractors})
+        add_stdout = StringIO()
+        with redirect_stdout(add_stdout):
+            add(**input_kwargs)
+            print(add_stdout.getvalue())
+
+        context = self.get_context_data()
+
+        context.update({
+            "stdout": ansi_to_html(add_stdout.getvalue().strip()),
+            "form": AddLinkForm()
+        })
+        return render(template_name=self.template_name, request=self.request, context=context)

archivebox/extractors/__init__.py

@@ -1,15 +1,17 @@
 __package__ = 'archivebox.extractors'
 
 import os
+from pathlib import Path
 
-from typing import Optional, List, Iterable
+from typing import Optional, List, Iterable, Union
 from datetime import datetime
+from django.db.models import QuerySet
 
 from ..index.schema import Link
+from ..index.sql import write_link_to_sql_index
 from ..index import (
     load_link_details,
     write_link_details,
-    patch_main_index,
 )
 from ..util import enforce_types
 from ..logging_util import (
@@ -21,47 +23,76 @@ from ..logging_util import (
     log_archive_method_started,
     log_archive_method_finished,
 )
+from ..search import write_search_index
 
 from .title import should_save_title, save_title
 from .favicon import should_save_favicon, save_favicon
 from .wget import should_save_wget, save_wget
+from .singlefile import should_save_singlefile, save_singlefile
+from .readability import should_save_readability, save_readability
+from .mercury import should_save_mercury, save_mercury
 from .pdf import should_save_pdf, save_pdf
 from .screenshot import should_save_screenshot, save_screenshot
 from .dom import should_save_dom, save_dom
 from .git import should_save_git, save_git
 from .media import should_save_media, save_media
 from .archive_org import should_save_archive_dot_org, save_archive_dot_org
+from .headers import should_save_headers, save_headers
 
 
-@enforce_types
-def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[str]=None) -> Link:
-    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
-
-    ARCHIVE_METHODS = [
+def get_default_archive_methods():
+    return [
         ('title', should_save_title, save_title),
         ('favicon', should_save_favicon, save_favicon),
         ('wget', should_save_wget, save_wget),
+        ('singlefile', should_save_singlefile, save_singlefile),
         ('pdf', should_save_pdf, save_pdf),
         ('screenshot', should_save_screenshot, save_screenshot),
         ('dom', should_save_dom, save_dom),
+        ('readability', should_save_readability, save_readability), #keep readability below wget and singlefile, as it depends on them
+        ('mercury', should_save_mercury, save_mercury),
         ('git', should_save_git, save_git),
         ('media', should_save_media, save_media),
+        ('headers', should_save_headers, save_headers),
         ('archive_org', should_save_archive_dot_org, save_archive_dot_org),
     ]
-    if methods is not None:
+
+ARCHIVE_METHODS_INDEXING_PRECEDENCE = [('readability', 1), ('singlefile', 2), ('dom', 3), ('wget', 4)]
+
+@enforce_types
+def ignore_methods(to_ignore: List[str]):
+    ARCHIVE_METHODS = get_default_archive_methods()
+    methods = filter(lambda x: x[0] not in to_ignore, ARCHIVE_METHODS)
+    methods = map(lambda x: x[0], methods)
+    return list(methods)
+
+@enforce_types
+def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> Link:
+    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+
+    # TODO: Remove when the input is changed to be a snapshot. Suboptimal approach.
+    from core.models import Snapshot, ArchiveResult
+    try:
+        snapshot = Snapshot.objects.get(url=link.url) # TODO: This will be unnecessary once everything is a snapshot
+    except Snapshot.DoesNotExist:
+        snapshot = write_link_to_sql_index(link)
+
+    ARCHIVE_METHODS = get_default_archive_methods()
+    
+    if methods:
         ARCHIVE_METHODS = [
             method for method in ARCHIVE_METHODS
-            if method[1] in methods
+            if method[0] in methods
         ]
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     try:
-        is_new = not os.path.exists(out_dir)
+        is_new = not Path(out_dir).exists()
         if is_new:
             os.makedirs(out_dir)
 
         link = load_link_details(link, out_dir=out_dir)
-        write_link_details(link, out_dir=link.link_dir)
+        write_link_details(link, out_dir=out_dir, skip_sql_index=False)
         log_link_archiving_started(link, out_dir, is_new)
         link = link.overwrite(updated=datetime.now())
         stats = {'skipped': 0, 'succeeded': 0, 'failed': 0}
@@ -71,7 +102,7 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
                 if method_name not in link.history:
                     link.history[method_name] = []
 
-                if should_run(link, out_dir) or overwrite:
+                if should_run(link, out_dir, overwrite):
                     log_archive_method_started(method_name)
 
                     result = method_function(link=link, out_dir=out_dir)
@@ -80,7 +111,12 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
 
                     stats[result.status] += 1
                     log_archive_method_finished(result)
+                    write_search_index(link=link, texts=result.index_texts)
+                    ArchiveResult.objects.create(snapshot=snapshot, extractor=method_name, cmd=result.cmd, cmd_version=result.cmd_version,
+                                                 output=result.output, pwd=result.pwd, start_ts=result.start_ts, end_ts=result.end_ts, status=result.status)
+
                 else:
+                    # print('{black}      X {}{reset}'.format(method_name, **ANSI))
                     stats['skipped'] += 1
             except Exception as e:
                 raise Exception('Exception in archive_methods.save_{}(Link(url={}))'.format(
@@ -97,13 +133,7 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
         except Exception:
             pass
 
-        write_link_details(link, out_dir=link.link_dir)
-        patch_main_index(link)
-
-        # # If any changes were made, update the main links index json and html
-        # was_changed = stats['succeeded'] or stats['failed']
-        # if was_changed:
-        #     patch_main_index(link)
+        write_link_details(link, out_dir=out_dir, skip_sql_index=False)
 
         log_link_archiving_finished(link, link.link_dir, is_new, stats)
 
@@ -120,24 +150,33 @@ def archive_link(link: Link, overwrite: bool=False, methods: Optional[Iterable[s
 
     return link
 
-
 @enforce_types
-def archive_links(links: List[Link], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[str]=None) -> List[Link]:
-    if not links:
+def archive_links(all_links: Union[Iterable[Link], QuerySet], overwrite: bool=False, methods: Optional[Iterable[str]]=None, out_dir: Optional[Path]=None) -> List[Link]:
+
+    if type(all_links) is QuerySet:
+        num_links: int = all_links.count()
+        get_link = lambda x: x.as_link()
+        all_links = all_links.iterator()
+    else:
+        num_links: int = len(all_links)
+        get_link = lambda x: x
+
+    if num_links == 0:
         return []
 
-    log_archiving_started(len(links))
+    log_archiving_started(num_links)
     idx: int = 0
-    link: Link = links[0]
     try:
-        for idx, link in enumerate(links):
-            archive_link(link, overwrite=overwrite, methods=methods, out_dir=link.link_dir)
+        for link in all_links:
+            idx += 1
+            to_archive = get_link(link)
+            archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir))
     except KeyboardInterrupt:
-        log_archiving_paused(len(links), idx, link.timestamp)
+        log_archiving_paused(num_links, idx, link.timestamp)
         raise SystemExit(0)
     except BaseException:
         print()
         raise
 
-    log_archiving_finished(len(links))
-    return links
+    log_archiving_finished(num_links)
+    return all_links

archivebox/extractors/archive_org.py

@@ -1,7 +1,7 @@
 __package__ = 'archivebox.extractors'
 
-import os
 
+from pathlib import Path
 from typing import Optional, List, Dict, Tuple
 from collections import defaultdict
 
@@ -13,6 +13,7 @@ from ..util import (
 )
 from ..config import (
     TIMEOUT,
+    CURL_ARGS,
     CHECK_SSL_VALIDITY,
     SAVE_ARCHIVE_DOT_ORG,
     CURL_BINARY,
@@ -24,31 +25,29 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_archive_dot_org(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+def should_save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'archive.org.txt')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'archive.org.txt').exists():
         # if open(path, 'r').read().strip() != 'None':
         return False
 
     return SAVE_ARCHIVE_DOT_ORG
 
 @enforce_types
-def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_archive_dot_org(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """submit site to archive.org for archiving via their service, save returned archive url"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'archive.org.txt'
     archive_org_url = None
     submit_url = 'https://web.archive.org/save/{}'.format(link.url)
     cmd = [
         CURL_BINARY,
-        '--silent',
-        '--location',
+        *CURL_ARGS,
         '--head',
-        '--compressed',
         '--max-time', str(timeout),
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
@@ -57,10 +56,10 @@ def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=T
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
         content_location, errors = parse_archive_dot_org_response(result.stdout)
         if content_location:
-            archive_org_url = 'https://web.archive.org{}'.format(content_location[0])
+            archive_org_url = content_location[0]
         elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]:
             archive_org_url = None
             # raise ArchiveError('Archive.org denied by {}/robots.txt'.format(domain(link.url)))
@@ -80,14 +79,14 @@ def save_archive_dot_org(link: Link, out_dir: Optional[str]=None, timeout: int=T
         # the URL in person, it will attempt to re-archive it, and it'll show the
         # nicer error message explaining why the url was rejected if it fails.
         archive_org_url = archive_org_url or submit_url
-        with open(os.path.join(out_dir, str(output)), 'w', encoding='utf-8') as f:
+        with open(str(out_dir / output), 'w', encoding='utf-8') as f:
             f.write(archive_org_url)
-        chmod_file('archive.org.txt', cwd=out_dir)
+        chmod_file('archive.org.txt', cwd=str(out_dir))
         output = archive_org_url
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/dom.py

@@ -1,7 +1,6 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
@@ -21,23 +20,23 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_dom(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+def should_save_dom(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'output.html')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'output.html').exists():
         return False
 
     return SAVE_DOM
-    
+
 @enforce_types
-def save_dom(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_dom(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """print HTML of site to file using chrome --dump-html"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'output.html'
-    output_path = os.path.join(out_dir, str(output))
+    output_path = out_dir / output
     cmd = [
         *chrome_args(TIMEOUT=timeout),
         '--dump-dom',
@@ -46,14 +45,14 @@ def save_dom(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
         atomic_write(output_path, result.stdout)
 
         if result.returncode:
             hints = result.stderr.decode()
             raise ArchiveError('Failed to save DOM', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -62,7 +61,7 @@ def save_dom(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/favicon.py

@@ -1,6 +1,6 @@
 __package__ = 'archivebox.extractors'
 
-import os
+from pathlib import Path
 
 from typing import Optional
 
@@ -11,6 +11,7 @@ from ..config import (
     TIMEOUT,
     SAVE_FAVICON,
     CURL_BINARY,
+    CURL_ARGS,
     CURL_VERSION,
     CHECK_SSL_VALIDITY,
     CURL_USER_AGENT,
@@ -19,45 +20,42 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_favicon(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-    if os.path.exists(os.path.join(out_dir, 'favicon.ico')):
+def should_save_favicon(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'favicon.ico').exists():
         return False
 
     return SAVE_FAVICON
-    
+
 @enforce_types
-def save_favicon(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_favicon(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download site favicon from google's favicon api"""
 
     out_dir = out_dir or link.link_dir
     output: ArchiveOutput = 'favicon.ico'
     cmd = [
         CURL_BINARY,
-        '--silent',
+        *CURL_ARGS,
         '--max-time', str(timeout),
-        '--location',
-        '--compressed',
         '--output', str(output),
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
         'https://www.google.com/s2/favicons?domain={}'.format(domain(link.url)),
     ]
-    status = 'pending'
+    status = 'failed'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        run(cmd, cwd=out_dir, timeout=timeout)
-        chmod_file(output, cwd=out_dir)
+        run(cmd, cwd=str(out_dir), timeout=timeout)
+        chmod_file(output, cwd=str(out_dir))
         status = 'succeeded'
     except Exception as err:
-        status = 'failed'
         output = err
     finally:
         timer.end()
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/git.py

@@ -1,7 +1,7 @@
 __package__ = 'archivebox.extractors'
 
-import os
 
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
@@ -18,6 +18,7 @@ from ..config import (
     TIMEOUT,
     SAVE_GIT,
     GIT_BINARY,
+    GIT_ARGS,
     GIT_VERSION,
     GIT_DOMAINS,
     CHECK_SSL_VALIDITY
@@ -27,12 +28,12 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_git(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+def should_save_git(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'git')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'git').exists():
         return False
 
     is_clonable_url = (
@@ -46,24 +47,24 @@ def should_save_git(link: Link, out_dir: Optional[str]=None) -> bool:
 
 
 @enforce_types
-def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_git(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download full site using git"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'git'
-    output_path = os.path.join(out_dir, str(output))
-    os.makedirs(output_path, exist_ok=True)
+    output_path = out_dir / output
+    output_path.mkdir(exist_ok=True)
     cmd = [
         GIT_BINARY,
         'clone',
-        '--recursive',
+        *GIT_ARGS,
         *([] if CHECK_SSL_VALIDITY else ['-c', 'http.sslVerify=false']),
         without_query(without_fragment(link.url)),
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=output_path, timeout=timeout + 1)
+        result = run(cmd, cwd=str(output_path), timeout=timeout + 1)
         if result.returncode == 128:
             # ignore failed re-download when the folder already exists
             pass
@@ -71,7 +72,7 @@ def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
             hints = 'Got git response code: {}.'.format(result.returncode)
             raise ArchiveError('Failed to save git clone', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
 
     except Exception as err:
         status = 'failed'
@@ -81,7 +82,7 @@ def save_git(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=GIT_VERSION,
         output=output,
         status=status,

archivebox/extractors/headers.py

@@ -0,0 +1,70 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+
+from typing import Optional
+
+from ..index.schema import Link, ArchiveResult, ArchiveOutput
+from ..system import atomic_write
+from ..util import (
+    enforce_types,
+    get_headers,
+)
+from ..config import (
+    TIMEOUT,
+    CURL_BINARY,
+    CURL_ARGS,
+    CURL_USER_AGENT,
+    CURL_VERSION,
+    CHECK_SSL_VALIDITY,
+    SAVE_HEADERS
+)
+from ..logging_util import TimedProgress
+
+@enforce_types
+def should_save_headers(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'headers.json').exists():
+        return False
+
+    return SAVE_HEADERS
+
+
+@enforce_types
+def save_headers(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """Download site headers"""
+
+    out_dir = Path(out_dir or link.link_dir)
+    output_folder = out_dir.absolute()
+    output: ArchiveOutput = 'headers.json'
+
+    status = 'succeeded'
+    timer = TimedProgress(timeout, prefix='      ')
+
+    cmd = [
+        CURL_BINARY,
+        *CURL_ARGS,
+        '--head',
+        '--max-time', str(timeout),
+        *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
+        *([] if CHECK_SSL_VALIDITY else ['--insecure']),
+        link.url,
+    ]
+    try:
+        json_headers = get_headers(link.url, timeout=timeout)
+        output_folder.mkdir(exist_ok=True)
+        atomic_write(str(output_folder / "headers.json"), json_headers)
+    except (Exception, OSError) as err:
+        status = 'failed'
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=CURL_VERSION,
+        output=output,
+        status=status,
+        **timer.stats,
+    )

archivebox/extractors/media.py

@@ -1,7 +1,6 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
@@ -13,7 +12,7 @@ from ..util import (
 from ..config import (
     MEDIA_TIMEOUT,
     SAVE_MEDIA,
-    SAVE_PLAYLISTS,
+    YOUTUBEDL_ARGS,
     YOUTUBEDL_BINARY,
     YOUTUBEDL_VERSION,
     CHECK_SSL_VALIDITY
@@ -22,52 +21,35 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_media(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
-
+def should_save_media(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
 
-    if os.path.exists(os.path.join(out_dir, 'media')):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'media').exists():
         return False
 
     return SAVE_MEDIA
 
 @enforce_types
-def save_media(link: Link, out_dir: Optional[str]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult:
+def save_media(link: Link, out_dir: Optional[Path]=None, timeout: int=MEDIA_TIMEOUT) -> ArchiveResult:
     """Download playlists or individual video, audio, and subtitles using youtube-dl"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'media'
-    output_path = os.path.join(out_dir, str(output))
-    os.makedirs(output_path, exist_ok=True)
+    output_path = out_dir / output
+    output_path.mkdir(exist_ok=True)
     cmd = [
         YOUTUBEDL_BINARY,
-        '--write-description',
-        '--write-info-json',
-        '--write-annotations',
-        '--write-thumbnail',
-        '--no-call-home',
-        '--no-check-certificate',
-        '--user-agent',
-        '--all-subs',
-        '--extract-audio',
-        '--keep-video',
-        '--ignore-errors',
-        '--geo-bypass',
-        '--audio-format', 'mp3',
-        '--audio-quality', '320K',
-        '--embed-thumbnail',
-        '--add-metadata',
-        *(['--yes-playlist'] if SAVE_PLAYLISTS else []),
+        *YOUTUBEDL_ARGS,
         *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate']),
         link.url,
     ]
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=output_path, timeout=timeout + 1)
-        chmod_file(output, cwd=out_dir)
+        result = run(cmd, cwd=str(output_path), timeout=timeout + 1)
+        chmod_file(output, cwd=str(out_dir))
         if result.returncode:
             if (b'ERROR: Unsupported URL' in result.stderr
                 or b'HTTP Error 404' in result.stderr
@@ -90,7 +72,7 @@ def save_media(link: Link, out_dir: Optional[str]=None, timeout: int=MEDIA_TIMEO
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=YOUTUBEDL_VERSION,
         output=output,
         status=status,

archivebox/extractors/mercury.py

@@ -0,0 +1,106 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+
+from subprocess import CompletedProcess
+from typing import Optional, List
+import json
+
+from ..index.schema import Link, ArchiveResult, ArchiveError
+from ..system import run, atomic_write
+from ..util import (
+    enforce_types,
+    is_static_file,
+
+)
+from ..config import (
+    TIMEOUT,
+    SAVE_MERCURY,
+    DEPENDENCIES,
+    MERCURY_VERSION,
+)
+from ..logging_util import TimedProgress
+
+
+
+@enforce_types
+def ShellError(cmd: List[str], result: CompletedProcess, lines: int=20) -> ArchiveError:
+    # parse out last line of stderr
+    return ArchiveError(
+        f'Got {cmd[0]} response code: {result.returncode}).',
+        " ".join(
+            line.strip()
+            for line in (result.stdout + result.stderr).decode().rsplit('\n', lines)[-lines:]
+            if line.strip()
+        ),
+    )
+
+
+@enforce_types
+def should_save_mercury(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    if is_static_file(link.url):
+        return False
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'mercury').exists():
+        return False
+
+    return SAVE_MERCURY
+
+
+@enforce_types
+def save_mercury(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """download reader friendly version using @postlight/mercury-parser"""
+
+    out_dir = Path(out_dir or link.link_dir)
+    output_folder = out_dir.absolute() / "mercury"
+    output = str(output_folder)
+
+    status = 'succeeded'
+    timer = TimedProgress(timeout, prefix='      ')
+    try:
+        # Get plain text version of article
+        cmd = [
+            DEPENDENCIES['MERCURY_BINARY']['path'],
+            link.url,
+            "--format=text"
+        ]
+        result = run(cmd, cwd=out_dir, timeout=timeout)
+        try:
+            article_text = json.loads(result.stdout)
+        except json.JSONDecodeError:
+            raise ShellError(cmd, result)
+        
+        # Get HTML version of article
+        cmd = [
+            DEPENDENCIES['MERCURY_BINARY']['path'],
+            link.url
+        ]
+        result = run(cmd, cwd=out_dir, timeout=timeout)
+        try:
+            article_json = json.loads(result.stdout)
+        except json.JSONDecodeError:
+            raise ShellError(cmd, result)
+
+        output_folder.mkdir(exist_ok=True)
+        atomic_write(str(output_folder / "content.html"), article_json.pop("content"))
+        atomic_write(str(output_folder / "content.txt"), article_text["content"])
+        atomic_write(str(output_folder / "article.json"), article_json)
+
+        # Check for common failure cases
+        if (result.returncode > 0):
+            raise ShellError(cmd, result)
+    except (ArchiveError, Exception, OSError) as err:
+        status = 'failed'
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=MERCURY_VERSION,
+        output=output,
+        status=status,
+        **timer.stats,
+    )

archivebox/extractors/pdf.py

@@ -1,7 +1,6 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
@@ -20,22 +19,22 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_pdf(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+def should_save_pdf(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'output.pdf')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'output.pdf').exists():
         return False
 
     return SAVE_PDF
 
 
 @enforce_types
-def save_pdf(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_pdf(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """print PDF of site to file using chrome --headless"""
 
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'output.pdf'
     cmd = [
         *chrome_args(TIMEOUT=timeout),
@@ -45,13 +44,13 @@ def save_pdf(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
 
         if result.returncode:
             hints = (result.stderr or result.stdout).decode()
             raise ArchiveError('Failed to save PDF', hints)
         
-        chmod_file('output.pdf', cwd=out_dir)
+        chmod_file('output.pdf', cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -61,7 +60,7 @@ def save_pdf(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> A
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/readability.py

@@ -0,0 +1,126 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+from tempfile import NamedTemporaryFile
+
+from typing import Optional
+import json
+
+from ..index.schema import Link, ArchiveResult, ArchiveError
+from ..system import run, atomic_write
+from ..util import (
+    enforce_types,
+    download_url,
+    is_static_file,
+    
+)
+from ..config import (
+    TIMEOUT,
+    CURL_BINARY,
+    SAVE_READABILITY,
+    DEPENDENCIES,
+    READABILITY_VERSION,
+)
+from ..logging_util import TimedProgress
+
+@enforce_types
+def get_html(link: Link, path: Path) -> str:
+    """
+    Try to find wget, singlefile and then dom files.
+    If none is found, download the url again.
+    """
+    canonical = link.canonical_outputs()
+    abs_path = path.absolute()
+    sources = [canonical["singlefile_path"], canonical["wget_path"], canonical["dom_path"]]
+    document = None
+    for source in sources:
+        try:
+            with open(abs_path / source, "r") as f:
+                document = f.read()
+                break
+        except (FileNotFoundError, TypeError):
+            continue
+    if document is None:
+        return download_url(link.url)
+    else:
+        return document
+
+@enforce_types
+def should_save_readability(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    if is_static_file(link.url):
+        return False
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'readability').exists():
+        return False
+
+    return SAVE_READABILITY
+
+
+@enforce_types
+def save_readability(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """download reader friendly version using @mozilla/readability"""
+
+    out_dir = Path(out_dir or link.link_dir)
+    output_folder = out_dir.absolute() / "readability"
+    output = str(output_folder)
+
+    # Readability Docs: https://github.com/mozilla/readability
+
+    status = 'succeeded'
+    # fake command to show the user so they have something to try debugging if get_html fails
+    cmd = [
+        CURL_BINARY,
+        link.url
+    ]
+    readability_content = None
+    timer = TimedProgress(timeout, prefix='      ')
+    try:
+        document = get_html(link, out_dir)
+        temp_doc = NamedTemporaryFile(delete=False)
+        temp_doc.write(document.encode("utf-8"))
+        temp_doc.close()
+
+        cmd = [
+            DEPENDENCIES['READABILITY_BINARY']['path'],
+            temp_doc.name
+        ]
+
+        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result_json = json.loads(result.stdout)
+        output_folder.mkdir(exist_ok=True)
+        readability_content = result_json.pop("textContent") 
+        atomic_write(str(output_folder / "content.html"), result_json.pop("content"))
+        atomic_write(str(output_folder / "content.txt"), readability_content)
+        atomic_write(str(output_folder / "article.json"), result_json)
+
+        # parse out number of files downloaded from last line of stderr:
+        #  "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
+        output_tail = [
+            line.strip()
+            for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:]
+            if line.strip()
+        ]
+        hints = (
+            'Got readability response code: {}.'.format(result.returncode),
+            *output_tail,
+        )
+
+        # Check for common failure cases
+        if (result.returncode > 0):
+            raise ArchiveError('Readability was not able to archive the page', hints)
+    except (Exception, OSError) as err:
+        status = 'failed'
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=READABILITY_VERSION,
+        output=output,
+        status=status,
+        index_texts= [readability_content] if readability_content else [],
+        **timer.stats,  
+    )

archivebox/extractors/screenshot.py

@@ -1,7 +1,6 @@
 __package__ = 'archivebox.extractors'
 
-import os
-
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
@@ -21,21 +20,21 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_screenshot(link: Link, out_dir: Optional[str]=None) -> bool:
-    out_dir = out_dir or link.link_dir
+def should_save_screenshot(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     if is_static_file(link.url):
         return False
-    
-    if os.path.exists(os.path.join(out_dir, 'screenshot.png')):
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'screenshot.png').exists():
         return False
 
     return SAVE_SCREENSHOT
 
 @enforce_types
-def save_screenshot(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_screenshot(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """take screenshot of site using chrome --headless"""
     
-    out_dir = out_dir or link.link_dir
+    out_dir = out_dir or Path(link.link_dir)
     output: ArchiveOutput = 'screenshot.png'
     cmd = [
         *chrome_args(TIMEOUT=timeout),
@@ -45,13 +44,13 @@ def save_screenshot(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOU
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
 
         if result.returncode:
             hints = (result.stderr or result.stdout).decode()
             raise ArchiveError('Failed to save screenshot', hints)
 
-        chmod_file(output, cwd=out_dir)
+        chmod_file(output, cwd=str(out_dir))
     except Exception as err:
         status = 'failed'
         output = err
@@ -60,7 +59,7 @@ def save_screenshot(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOU
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CHROME_VERSION,
         output=output,
         status=status,

archivebox/extractors/singlefile.py

@@ -0,0 +1,92 @@
+__package__ = 'archivebox.extractors'
+
+from pathlib import Path
+
+from typing import Optional
+import json
+
+from ..index.schema import Link, ArchiveResult, ArchiveError
+from ..system import run, chmod_file
+from ..util import (
+    enforce_types,
+    is_static_file,
+    chrome_args,
+)
+from ..config import (
+    TIMEOUT,
+    SAVE_SINGLEFILE,
+    DEPENDENCIES,
+    SINGLEFILE_VERSION,
+    CHROME_BINARY,
+)
+from ..logging_util import TimedProgress
+
+
+@enforce_types
+def should_save_singlefile(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
+    if is_static_file(link.url):
+        return False
+
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and (out_dir / 'singlefile.html').exists():
+        return False
+
+    return SAVE_SINGLEFILE
+
+
+@enforce_types
+def save_singlefile(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+    """download full site using single-file"""
+
+    out_dir = out_dir or Path(link.link_dir)
+    output = "singlefile.html"
+
+    browser_args = chrome_args(TIMEOUT=0)
+
+    # SingleFile CLI Docs: https://github.com/gildas-lormeau/SingleFile/tree/master/cli
+    browser_args = '--browser-args={}'.format(json.dumps(browser_args[1:]))
+    cmd = [
+        DEPENDENCIES['SINGLEFILE_BINARY']['path'],
+        '--browser-executable-path={}'.format(CHROME_BINARY),
+        browser_args,
+        link.url,
+        output,
+    ]
+
+    status = 'succeeded'
+    timer = TimedProgress(timeout, prefix='      ')
+    try:
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
+
+        # parse out number of files downloaded from last line of stderr:
+        #  "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
+        output_tail = [
+            line.strip()
+            for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:]
+            if line.strip()
+        ]
+        hints = (
+            'Got single-file response code: {}.'.format(result.returncode),
+            *output_tail,
+        )
+
+        # Check for common failure cases
+        if (result.returncode > 0) or not (out_dir / output).is_file():
+            raise ArchiveError('SingleFile was not able to archive the page', hints)
+        chmod_file(output, cwd=str(out_dir))
+    except (Exception, OSError) as err:
+        status = 'failed'
+        # TODO: Make this prettier. This is necessary to run the command (escape JSON internal quotes).
+        cmd[2] = browser_args.replace('"', "\\\"")
+        output = err
+    finally:
+        timer.end()
+
+    return ArchiveResult(
+        cmd=cmd,
+        pwd=str(out_dir),
+        cmd_version=SINGLEFILE_VERSION,
+        output=output,
+        status=status,
+        **timer.stats,
+    )

archivebox/extractors/title.py

@@ -1,12 +1,13 @@
 __package__ = 'archivebox.extractors'
 
 import re
+from html.parser import HTMLParser
+from pathlib import Path
 from typing import Optional
 
 from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
 from ..util import (
     enforce_types,
-    is_static_file,
     download_url,
     htmldecode,
 )
@@ -15,13 +16,14 @@ from ..config import (
     CHECK_SSL_VALIDITY,
     SAVE_TITLE,
     CURL_BINARY,
+    CURL_ARGS,
     CURL_VERSION,
     CURL_USER_AGENT,
-    setup_django,
 )
 from ..logging_util import TimedProgress
 
 
+
 HTML_TITLE_REGEX = re.compile(
     r'<title.*?>'                      # start matching text after <title> tag
     r'(.[^<>]+)',                      # get everything up to these symbols
@@ -29,31 +31,58 @@ HTML_TITLE_REGEX = re.compile(
 )
 
 
-@enforce_types
-def should_save_title(link: Link, out_dir: Optional[str]=None) -> bool:
-    # if link already has valid title, skip it
-    if link.title and not link.title.lower().startswith('http'):
-        return False
+class TitleParser(HTMLParser):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.title_tag = ""
+        self.title_og = ""
+        self.inside_title_tag = False
 
-    if is_static_file(link.url):
+    @property
+    def title(self):
+        return self.title_tag or self.title_og or None
+
+    def handle_starttag(self, tag, attrs):
+        if tag.lower() == "title" and not self.title_tag:
+            self.inside_title_tag = True
+        elif tag.lower() == "meta" and not self.title_og:
+            attrs = dict(attrs)
+            if attrs.get("property") == "og:title" and attrs.get("content"):
+                self.title_og = attrs.get("content")
+
+    def handle_data(self, data):
+        if self.inside_title_tag and data:
+            self.title_tag += data.strip()
+    
+    def handle_endtag(self, tag):
+        if tag.lower() == "title":
+            self.inside_title_tag = False
+
+
+@enforce_types
+def should_save_title(link: Link, out_dir: Optional[str]=None, overwrite: Optional[bool]=False) -> bool:
+    # if link already has valid title, skip it
+    if not overwrite and link.title and not link.title.lower().startswith('http'):
         return False
 
     return SAVE_TITLE
 
+def extract_title_with_regex(html):
+    match = re.search(HTML_TITLE_REGEX, html)
+    output = htmldecode(match.group(1).strip()) if match else None
+    return output
+
 @enforce_types
-def save_title(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_title(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """try to guess the page's title from its content"""
 
-    setup_django(out_dir=out_dir)
     from core.models import Snapshot
 
     output: ArchiveOutput = None
     cmd = [
         CURL_BINARY,
-        '--silent',
+        *CURL_ARGS,
         '--max-time', str(timeout),
-        '--location',
-        '--compressed',
         *(['--user-agent', '{}'.format(CURL_USER_AGENT)] if CURL_USER_AGENT else []),
         *([] if CHECK_SSL_VALIDITY else ['--insecure']),
         link.url,
@@ -62,13 +91,29 @@ def save_title(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
     timer = TimedProgress(timeout, prefix='      ')
     try:
         html = download_url(link.url, timeout=timeout)
-        match = re.search(HTML_TITLE_REGEX, html)
-        output = htmldecode(match.group(1).strip()) if match else None
-        if output:
+        try:
+            # try using relatively strict html parser first
+            parser = TitleParser()
+            parser.feed(html)
+            output = parser.title
+            if output is None:
+                raise
+        except Exception:
+            # fallback to regex that can handle broken/malformed html
+            output = extract_title_with_regex(html)
+        
+        # if title is better than the one in the db, update db with new title
+        if isinstance(output, str) and output:
             if not link.title or len(output) >= len(link.title):
-                Snapshot.objects.filter(url=link.url, timestamp=link.timestamp).update(title=output)
+                Snapshot.objects.filter(url=link.url,
+                                        timestamp=link.timestamp)\
+                                .update(title=output)
         else:
-            raise ArchiveError('Unable to detect page title')
+            # if no content was returned, dont save a title (because it might be a temporary error)
+            if not html:
+                raise ArchiveError('Unable to detect page title')
+            # output = html[:128]       # use first bit of content as the title
+            output = link.base_url      # use the filename as the title (better UX)
     except Exception as err:
         status = 'failed'
         output = err
@@ -77,7 +122,7 @@ def save_title(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=CURL_VERSION,
         output=output,
         status=status,

archivebox/extractors/wget.py

@@ -1,7 +1,7 @@
 __package__ = 'archivebox.extractors'
 
-import os
 import re
+from pathlib import Path
 
 from typing import Optional
 from datetime import datetime
@@ -10,8 +10,6 @@ from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
 from ..system import run, chmod_file
 from ..util import (
     enforce_types,
-    is_static_file,
-    without_scheme,
     without_fragment,
     without_query,
     path,
@@ -19,6 +17,7 @@ from ..util import (
     urldecode,
 )
 from ..config import (
+    WGET_ARGS,
     TIMEOUT,
     SAVE_WGET,
     SAVE_WARC,
@@ -35,44 +34,37 @@ from ..logging_util import TimedProgress
 
 
 @enforce_types
-def should_save_wget(link: Link, out_dir: Optional[str]=None) -> bool:
+def should_save_wget(link: Link, out_dir: Optional[Path]=None, overwrite: Optional[bool]=False) -> bool:
     output_path = wget_output_path(link)
-    out_dir = out_dir or link.link_dir
-    if output_path and os.path.exists(os.path.join(out_dir, output_path)):
+    out_dir = out_dir or Path(link.link_dir)
+    if not overwrite and output_path and (out_dir / output_path).exists():
         return False
 
     return SAVE_WGET
 
 
 @enforce_types
-def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) -> ArchiveResult:
+def save_wget(link: Link, out_dir: Optional[Path]=None, timeout: int=TIMEOUT) -> ArchiveResult:
     """download full site using wget"""
 
     out_dir = out_dir or link.link_dir
     if SAVE_WARC:
-        warc_dir = os.path.join(out_dir, 'warc')
-        os.makedirs(warc_dir, exist_ok=True)
-        warc_path = os.path.join('warc', str(int(datetime.now().timestamp())))
+        warc_dir = out_dir / "warc"
+        warc_dir.mkdir(exist_ok=True)
+        warc_path = warc_dir / str(int(datetime.now().timestamp()))
 
     # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
     output: ArchiveOutput = None
     cmd = [
         WGET_BINARY,
         # '--server-response',  # print headers for better error parsing
-        '--no-verbose',
-        '--adjust-extension',
-        '--convert-links',
-        '--force-directories',
-        '--backup-converted',
-        '--span-hosts',
-        '--no-parent',
-        '-e', 'robots=off',
+        *WGET_ARGS,
         '--timeout={}'.format(timeout),
         *(['--restrict-file-names={}'.format(RESTRICT_FILE_NAMES)] if RESTRICT_FILE_NAMES else []),
-        *(['--warc-file={}'.format(warc_path)] if SAVE_WARC else []),
+        *(['--warc-file={}'.format(str(warc_path))] if SAVE_WARC else []),
         *(['--page-requisites'] if SAVE_WGET_REQUISITES else []),
         *(['--user-agent={}'.format(WGET_USER_AGENT)] if WGET_USER_AGENT else []),
-        *(['--load-cookies', COOKIES_FILE] if COOKIES_FILE else []),
+        *(['--load-cookies', str(COOKIES_FILE)] if COOKIES_FILE else []),
         *(['--compression=auto'] if WGET_AUTO_COMPRESSION else []),
         *([] if SAVE_WARC else ['--timestamping']),
         *([] if CHECK_SSL_VALIDITY else ['--no-check-certificate', '--no-hsts']),
@@ -82,7 +74,7 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
     status = 'succeeded'
     timer = TimedProgress(timeout, prefix='      ')
     try:
-        result = run(cmd, cwd=out_dir, timeout=timeout)
+        result = run(cmd, cwd=str(out_dir), timeout=timeout)
         output = wget_output_path(link)
 
         # parse out number of files downloaded from last line of stderr:
@@ -111,7 +103,12 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
             if b'ERROR 500: Internal Server Error' in result.stderr:
                 raise ArchiveError('500 Internal Server Error', hints)
             raise ArchiveError('Wget failed or got an error from the server', hints)
-        chmod_file(output, cwd=out_dir)
+        
+        if (out_dir / output).exists():
+            chmod_file(output, cwd=str(out_dir))
+        else:
+            print(f'          {out_dir}/{output}')
+            raise ArchiveError('Failed to find wget output after running', hints)
     except Exception as err:
         status = 'failed'
         output = err
@@ -120,7 +117,7 @@ def save_wget(link: Link, out_dir: Optional[str]=None, timeout: int=TIMEOUT) ->
 
     return ArchiveResult(
         cmd=cmd,
-        pwd=out_dir,
+        pwd=str(out_dir),
         cmd_version=WGET_VERSION,
         output=output,
         status=status,
@@ -135,9 +132,7 @@ def wget_output_path(link: Link) -> Optional[str]:
 
     See docs on wget --adjust-extension (-E)
     """
-    if is_static_file(link.url):
-        return without_scheme(without_fragment(link.url))
-
+    
     # Wget downloads can save in a number of different ways depending on the url:
     #    https://example.com
     #       > example.com/index.html
@@ -170,26 +165,41 @@ def wget_output_path(link: Link) -> Optional[str]:
     # in order to avoid having to reverse-engineer how they calculate it,
     # we just look in the output folder read the filename wget used from the filesystem
     full_path = without_fragment(without_query(path(link.url))).strip('/')
-    search_dir = os.path.join(
-        link.link_dir,
-        domain(link.url).replace(":", "+"),
-        urldecode(full_path),
-    )
+    search_dir = Path(link.link_dir) / domain(link.url).replace(":", "+") / urldecode(full_path)
     for _ in range(4):
-        if os.path.exists(search_dir):
-            if os.path.isdir(search_dir):
+        if search_dir.exists():
+            if search_dir.is_dir():
                 html_files = [
-                    f for f in os.listdir(search_dir)
-                    if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", f, re.I | re.M)
+                    f for f in search_dir.iterdir()
+                    if re.search(".+\\.[Ss]?[Hh][Tt][Mm][Ll]?$", str(f), re.I | re.M)
                 ]
                 if html_files:
-                    path_from_link_dir = search_dir.split(link.link_dir)[-1].strip('/')
-                    return os.path.join(path_from_link_dir, html_files[0])
+                    return str(html_files[0].relative_to(link.link_dir))
+
+                # sometimes wget'd URLs have no ext and return non-html
+                # e.g. /some/example/rss/all -> some RSS XML content)
+                #      /some/other/url.o4g   -> some binary unrecognized ext)
+                # test this with archivebox add --depth=1 https://getpocket.com/users/nikisweeting/feed/all
+                last_part_of_url = urldecode(full_path.rsplit('/', 1)[-1])
+                for file_present in search_dir.iterdir():
+                    if file_present == last_part_of_url:
+                        return str((search_dir / file_present).relative_to(link.link_dir))
 
         # Move up one directory level
-        search_dir = search_dir.rsplit('/', 1)[0]
+        search_dir = search_dir.parent
 
-        if search_dir == link.link_dir:
+        if str(search_dir) == link.link_dir:
             break
 
+    # check for literally any file present that isnt an empty folder
+    domain_dir = Path(domain(link.url).replace(":", "+"))
+    files_within = list((Path(link.link_dir) / domain_dir).glob('**/*.*'))
+    if files_within:
+        return str((domain_dir / files_within[-1]).relative_to(link.link_dir))
+    
+    # fallback to just the domain dir
+    search_dir = Path(link.link_dir) / domain(link.url).replace(":", "+")
+    if search_dir.is_dir():
+        return domain(link.url).replace(":", "+")
+
     return None

archivebox/index/__init__.py

@@ -1,16 +1,16 @@
 __package__ = 'archivebox.index'
 
-import re
 import os
 import shutil
-import json as pyjson
+from pathlib import Path
 
 from itertools import chain
 from typing import List, Tuple, Dict, Optional, Iterable
 from collections import OrderedDict
 from contextlib import contextmanager
+from urllib.parse import urlparse
+from django.db.models import QuerySet, Q
 
-from ..system import atomic_write
 from ..util import (
     scheme,
     enforce_types,
@@ -20,11 +20,9 @@ from ..config import (
     ARCHIVE_DIR_NAME,
     SQL_INDEX_FILENAME,
     JSON_INDEX_FILENAME,
-    HTML_INDEX_FILENAME,
     OUTPUT_DIR,
     TIMEOUT,
     URL_BLACKLIST_PTN,
-    ANSI,
     stderr,
     OUTPUT_PERMISSIONS
 )
@@ -40,21 +38,20 @@ from ..logging_util import (
 
 from .schema import Link, ArchiveResult
 from .html import (
-    write_html_main_index,
     write_html_link_details,
 )
 from .json import (
-    parse_json_main_index,
-    write_json_main_index,
+    pyjson,
     parse_json_link_details, 
     write_json_link_details,
 )
 from .sql import (
     write_sql_main_index,
-    parse_sql_main_index,
     write_sql_link_details,
 )
 
+from ..search import search_backend_enabled, query_search_index
+
 ### Link filtering and checking
 
 @enforce_types
@@ -62,7 +59,7 @@ def merge_links(a: Link, b: Link) -> Link:
     """deterministially merge two links, favoring longer field values over shorter,
     and "cleaner" values over worse ones.
     """
-    assert a.base_url == b.base_url, 'Cannot merge two links with different URLs'
+    assert a.base_url == b.base_url, f'Cannot merge two links with different URLs ({a.base_url} != {b.base_url})'
 
     # longest url wins (because a fuzzy url will always be shorter)
     url = a.url if len(a.url) > len(b.url) else b.url
@@ -127,46 +124,45 @@ def validate_links(links: Iterable[Link]) -> List[Link]:
     timer = TimedProgress(TIMEOUT * 4)
     try:
         links = archivable_links(links)  # remove chrome://, about:, mailto: etc.
-        links = sorted_links(links)      # deterministically sort the links based on timstamp, url
-        links = uniquefied_links(links)  # merge/dedupe duplicate timestamps & urls
+        links = sorted_links(links)      # deterministically sort the links based on timestamp, url
+        links = fix_duplicate_links(links)  # merge/dedupe duplicate timestamps & urls
     finally:
         timer.end()
 
     return list(links)
 
-
 @enforce_types
 def archivable_links(links: Iterable[Link]) -> Iterable[Link]:
     """remove chrome://, about:// or other schemed links that cant be archived"""
     for link in links:
-        scheme_is_valid = scheme(link.url) in ('http', 'https', 'ftp')
-        not_blacklisted = (not URL_BLACKLIST_PTN.match(link.url)) if URL_BLACKLIST_PTN else True
-        if scheme_is_valid and not_blacklisted:
-            yield link
+        try:
+            urlparse(link.url)
+        except ValueError:
+            continue
+        if scheme(link.url) not in ('http', 'https', 'ftp'):
+            continue
+        if URL_BLACKLIST_PTN and URL_BLACKLIST_PTN.search(link.url):
+            continue
+
+        yield link
 
 
 @enforce_types
-def uniquefied_links(sorted_links: Iterable[Link]) -> Iterable[Link]:
+def fix_duplicate_links(sorted_links: Iterable[Link]) -> Iterable[Link]:
     """
     ensures that all non-duplicate links have monotonically increasing timestamps
     """
+    # from core.models import Snapshot
 
     unique_urls: OrderedDict[str, Link] = OrderedDict()
 
     for link in sorted_links:
-        if link.base_url in unique_urls:
+        if link.url in unique_urls:
             # merge with any other links that share the same url
-            link = merge_links(unique_urls[link.base_url], link)
-        unique_urls[link.base_url] = link
+            link = merge_links(unique_urls[link.url], link)
+        unique_urls[link.url] = link
 
-    unique_timestamps: OrderedDict[str, Link] = OrderedDict()
-    for link in unique_urls.values():
-        new_link = link.overwrite(
-            timestamp=lowest_uniq_timestamp(unique_timestamps, link.timestamp),
-        )
-        unique_timestamps[new_link.timestamp] = new_link
-
-    return unique_timestamps.values()
+    return unique_urls.values()
 
 
 @enforce_types
@@ -213,7 +209,7 @@ def lowest_uniq_timestamp(used_timestamps: OrderedDict, timestamp: str) -> str:
 
 @contextmanager
 @enforce_types
-def timed_index_update(out_path: str):
+def timed_index_update(out_path: Path):
     log_indexing_started(out_path)
     timer = TimedProgress(TIMEOUT * 2, prefix='      ')
     try:
@@ -221,49 +217,45 @@ def timed_index_update(out_path: str):
     finally:
         timer.end()
 
-    assert os.path.exists(out_path), f'Failed to write index file: {out_path}'
+    assert out_path.exists(), f'Failed to write index file: {out_path}'
     log_indexing_finished(out_path)
 
 
 @enforce_types
-def write_main_index(links: List[Link], out_dir: str=OUTPUT_DIR, finished: bool=False) -> None:
-    """create index.html file for a given list of links"""
+def write_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None:
+    """Writes links to sqlite3 file for a given list of links"""
 
     log_indexing_process_started(len(links))
 
-    with timed_index_update(os.path.join(out_dir, SQL_INDEX_FILENAME)):
-        write_sql_main_index(links, out_dir=out_dir)
-        os.chmod(os.path.join(out_dir, SQL_INDEX_FILENAME), int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
+    try:
+        with timed_index_update(out_dir / SQL_INDEX_FILENAME):
+            write_sql_main_index(links, out_dir=out_dir)
+            os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
 
-
-    with timed_index_update(os.path.join(out_dir, JSON_INDEX_FILENAME)):
-        write_json_main_index(links, out_dir=out_dir)
-
-    with timed_index_update(os.path.join(out_dir, HTML_INDEX_FILENAME)):
-        write_html_main_index(links, out_dir=out_dir, finished=finished)
+    except (KeyboardInterrupt, SystemExit):
+        stderr('[!] Warning: Still writing index to disk...', color='lightyellow')
+        stderr('    Run archivebox init to fix any inconsistencies from an ungraceful exit.')
+        with timed_index_update(out_dir / SQL_INDEX_FILENAME):
+            write_sql_main_index(links, out_dir=out_dir)
+            os.chmod(out_dir / SQL_INDEX_FILENAME, int(OUTPUT_PERMISSIONS, base=8)) # set here because we don't write it with atomic writes
+        raise SystemExit(0)
 
     log_indexing_process_finished()
 
-
 @enforce_types
-def load_main_index(out_dir: str=OUTPUT_DIR, warn: bool=True) -> List[Link]:
+def load_main_index(out_dir: Path=OUTPUT_DIR, warn: bool=True) -> List[Link]:
     """parse and load existing index with any new links from import_path merged in"""
+    from core.models import Snapshot
+    try:
+        return Snapshot.objects.all()
 
-    all_links: List[Link] = []
-    all_links = list(parse_json_main_index(out_dir))
-    links_from_sql = list(parse_sql_main_index(out_dir))
-
-    if warn and not set(l.url for l in all_links) == set(l.url for l in links_from_sql):
-        stderr('{red}[!] Warning: SQL index does not match JSON index!{reset}'.format(**ANSI))
-        stderr('    To repair the index and re-import any orphaned links run:')
-        stderr('        archivebox init')
-
-    return all_links
+    except (KeyboardInterrupt, SystemExit):
+        raise SystemExit(0)
 
 @enforce_types
-def load_main_index_meta(out_dir: str=OUTPUT_DIR) -> Optional[dict]:
-    index_path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(index_path):
+def load_main_index_meta(out_dir: Path=OUTPUT_DIR) -> Optional[dict]:
+    index_path = out_dir / JSON_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             meta_dict = pyjson.load(f)
             meta_dict.pop('links')
@@ -273,14 +265,14 @@ def load_main_index_meta(out_dir: str=OUTPUT_DIR) -> Optional[dict]:
 
 
 @enforce_types
-def parse_links_from_source(source_path: str) -> Tuple[List[Link], List[Link]]:
+def parse_links_from_source(source_path: str, root_url: Optional[str]=None) -> Tuple[List[Link], List[Link]]:
 
     from ..parsers import parse_links
 
     new_links: List[Link] = []
 
     # parse and validate the import file
-    raw_links, parser_name = parse_links(source_path)
+    raw_links, parser_name = parse_links(source_path, root_url=root_url)
     new_links = validate_links(raw_links)
 
     if parser_name:
@@ -289,77 +281,58 @@ def parse_links_from_source(source_path: str) -> Tuple[List[Link], List[Link]]:
 
     return new_links
 
+@enforce_types
+def fix_duplicate_links_in_index(snapshots: QuerySet, links: Iterable[Link]) -> Iterable[Link]:
+    """
+    Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB.
+    """
+    unique_urls: OrderedDict[str, Link] = OrderedDict()
+
+    for link in links:
+        index_link = snapshots.filter(url=link.url)
+        if index_link:
+            link = merge_links(index_link[0].as_link(), link)
+
+        unique_urls[link.url] = link
+
+    return unique_urls.values()
 
 @enforce_types
-def dedupe_links(existing_links: List[Link],
-                 new_links: List[Link]) -> Tuple[List[Link], List[Link]]:
-
+def dedupe_links(snapshots: QuerySet,
+                 new_links: List[Link]) -> List[Link]:
+    """
+    The validation of links happened at a different stage. This method will
+    focus on actual deduplication and timestamp fixing.
+    """
+    
     # merge existing links in out_dir and new links
-    all_links = validate_links(existing_links + new_links)
-    all_link_urls = {link.url for link in existing_links}
+    dedup_links = fix_duplicate_links_in_index(snapshots, new_links)
 
     new_links = [
         link for link in new_links
-        if link.url not in all_link_urls
+        if not snapshots.filter(url=link.url).exists()
     ]
 
-    all_links_deduped = {link.url: link for link in all_links}
+    dedup_links_dict = {link.url: link for link in dedup_links}
+
+    # Replace links in new_links with the dedup version
     for i in range(len(new_links)):
-        if new_links[i].url in all_links_deduped.keys():
-            new_links[i] = all_links_deduped[new_links[i].url]
+        if new_links[i].url in dedup_links_dict.keys():
+            new_links[i] = dedup_links_dict[new_links[i].url]
     log_deduping_finished(len(new_links))
 
-    return all_links, new_links
-
-
-@enforce_types
-def patch_main_index(link: Link, out_dir: str=OUTPUT_DIR) -> None:
-    """hack to in-place update one row's info in the generated index files"""
-
-    # TODO: remove this ASAP, it's ugly, error-prone, and potentially dangerous
-
-    title = link.title or link.latest_outputs(status='succeeded')['title']
-    successful = link.num_outputs
-
-    # Patch JSON main index
-    json_file_links = parse_json_main_index(out_dir)
-    patched_links = []
-    for saved_link in json_file_links:
-        if saved_link.url == link.url:
-            patched_links.append(saved_link.overwrite(
-                title=title,
-                history=link.history,
-                updated=link.updated,
-            ))
-        else:
-            patched_links.append(saved_link)
-    
-    write_json_main_index(patched_links, out_dir=out_dir)
-
-    # Patch HTML main index
-    html_path = os.path.join(out_dir, 'index.html')
-    with open(html_path, 'r') as f:
-        html = f.read().splitlines()
-
-    for idx, line in enumerate(html):
-        if title and ('<span data-title-for="{}"'.format(link.url) in line):
-            html[idx] = '<span>{}</span>'.format(title)
-        elif successful and ('<span data-number-for="{}"'.format(link.url) in line):
-            html[idx] = '<span>{}</span>'.format(successful)
-            break
-
-    atomic_write(html_path, '\n'.join(html))
-
+    return new_links
 
 ### Link Details Index
 
 @enforce_types
-def write_link_details(link: Link, out_dir: Optional[str]=None) -> None:
+def write_link_details(link: Link, out_dir: Optional[str]=None, skip_sql_index: bool=False) -> None:
     out_dir = out_dir or link.link_dir
 
     write_json_link_details(link, out_dir=out_dir)
     write_html_link_details(link, out_dir=out_dir)
-    write_sql_link_details(link)
+    if not skip_sql_index:
+        write_sql_link_details(link)
 
 
 @enforce_types
@@ -378,19 +351,20 @@ def load_link_details(link: Link, out_dir: Optional[str]=None) -> Link:
 
 
 LINK_FILTERS = {
-    'exact': lambda link, pattern: (link.url == pattern) or (link.base_url == pattern),
-    'substring': lambda link, pattern: pattern in link.url,
-    'regex': lambda link, pattern: bool(re.match(pattern, link.url)),
-    'domain': lambda link, pattern: link.domain == pattern,
+    'exact': lambda pattern: Q(url=pattern),
+    'substring': lambda pattern: Q(url__icontains=pattern),
+    'regex': lambda pattern: Q(url__iregex=pattern),
+    'domain': lambda pattern: Q(url__istartswith=f"http://{pattern}") | Q(url__istartswith=f"https://{pattern}") | Q(url__istartswith=f"ftp://{pattern}"),
+    'tag': lambda pattern: Q(tags__name=pattern),
 }
 
 @enforce_types
-def link_matches_filter(link: Link, filter_patterns: List[str], filter_type: str='exact') -> bool:
+def q_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet:
+    q_filter = Q()
     for pattern in filter_patterns:
         try:
-            if LINK_FILTERS[filter_type](link, pattern):
-                return True
-        except Exception:
+            q_filter = q_filter | LINK_FILTERS[filter_type](pattern)
+        except KeyError:
             stderr()
             stderr(
                 f'[X] Got invalid pattern for --filter-type={filter_type}:',
@@ -398,80 +372,110 @@ def link_matches_filter(link: Link, filter_patterns: List[str], filter_type: str
             )
             stderr(f'    {pattern}')
             raise SystemExit(2)
+    return snapshots.filter(q_filter)
 
-    return False
+def search_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='search') -> QuerySet:
+    if not search_backend_enabled():
+        stderr()
+        stderr(
+                '[X] The search backend is not enabled, set config.USE_SEARCHING_BACKEND = True',
+                color='red',
+            )
+        raise SystemExit(2)
+    from core.models import Snapshot
+
+    qsearch = Snapshot.objects.none()
+    for pattern in filter_patterns:
+        try:
+            qsearch |= query_search_index(pattern)
+        except:
+            raise SystemExit(2)
+    
+    return snapshots & qsearch
+
+@enforce_types
+def snapshot_filter(snapshots: QuerySet, filter_patterns: List[str], filter_type: str='exact') -> QuerySet:
+    if filter_type != 'search':
+        return q_filter(snapshots, filter_patterns, filter_type)
+    else:
+        return search_filter(snapshots, filter_patterns, filter_type)
 
 
-def get_indexed_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_indexed_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """indexed links without checking archive status or data directory validity"""
+    links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()]
     return {
         link.link_dir: link
         for link in links
     }
 
-def get_archived_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_archived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """indexed links that are archived with a valid data directory"""
+    links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()]
     return {
         link.link_dir: link
         for link in filter(is_archived, links)
     }
 
-def get_unarchived_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_unarchived_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """indexed links that are unarchived with no data directory or an empty data directory"""
+    links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()]
     return {
         link.link_dir: link
         for link in filter(is_unarchived, links)
     }
 
-def get_present_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_present_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that actually exist in the archive/ folder"""
+
     all_folders = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
+    for entry in (out_dir / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
             link = None
             try:
                 link = parse_json_link_details(entry.path)
             except Exception:
                 pass
 
-            all_folders[entry.path] = link
+            all_folders[entry.name] = link
 
     return all_folders
 
-def get_valid_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_valid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs with a valid index matched to the main index and archived content"""
+    links = [snapshot.as_link_with_details() for snapshot in snapshots.iterator()]
     return {
         link.link_dir: link
         for link in filter(is_valid, links)
     }
 
-def get_invalid_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_invalid_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized"""
-    duplicate = get_duplicate_folders(links, out_dir=OUTPUT_DIR)
-    orphaned = get_orphaned_folders(links, out_dir=OUTPUT_DIR)
-    corrupted = get_corrupted_folders(links, out_dir=OUTPUT_DIR)
-    unrecognized = get_unrecognized_folders(links, out_dir=OUTPUT_DIR)
+    duplicate = get_duplicate_folders(snapshots, out_dir=OUTPUT_DIR)
+    orphaned = get_orphaned_folders(snapshots, out_dir=OUTPUT_DIR)
+    corrupted = get_corrupted_folders(snapshots, out_dir=OUTPUT_DIR)
+    unrecognized = get_unrecognized_folders(snapshots, out_dir=OUTPUT_DIR)
     return {**duplicate, **orphaned, **corrupted, **unrecognized}
 
 
-def get_duplicate_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_duplicate_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that conflict with other directories that have the same link URL or timestamp"""
-    links = list(links)
-    by_url = {link.url: 0 for link in links}
-    by_timestamp = {link.timestamp: 0 for link in links}
-
+    by_url = {}
+    by_timestamp = {}
     duplicate_folders = {}
 
-    indexed_folders = {link.link_dir for link in links}
     data_folders = (
-        entry.path
-        for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME))
-        if entry.is_dir(follow_symlinks=True) and entry.path not in indexed_folders
+        str(entry)
+        for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir()
+            if entry.is_dir() and not snapshots.filter(timestamp=entry.name).exists()
     )
 
-    for path in chain(sorted(indexed_folders), sorted(data_folders)):
+    for path in chain(snapshots.iterator(), data_folders):
         link = None
+        if type(path) is not str:
+            path = path.as_link().link_dir
+
         try:
             link = parse_json_link_details(path)
         except Exception:
@@ -487,74 +491,72 @@ def get_duplicate_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[
             by_url[link.url] = by_url.get(link.url, 0) + 1
             if by_url[link.url] > 1:
                 duplicate_folders[path] = link
-
     return duplicate_folders
 
-def get_orphaned_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_orphaned_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that contain a valid index but aren't listed in the main index"""
-    links = list(links)
-    indexed_folders = {link.link_dir: link for link in links}
     orphaned_folders = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
+    for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
             link = None
             try:
-                link = parse_json_link_details(entry.path)
+                link = parse_json_link_details(str(entry))
             except Exception:
                 pass
 
-            if link and entry.path not in indexed_folders:
+            if link and not snapshots.filter(timestamp=entry.name).exists():
                 # folder is a valid link data dir with index details, but it's not in the main index
-                orphaned_folders[entry.path] = link
+                orphaned_folders[str(entry)] = link
 
     return orphaned_folders
 
-def get_corrupted_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_corrupted_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that don't contain a valid index and aren't listed in the main index"""
-    return {
-        link.link_dir: link
-        for link in filter(is_corrupt, links)
-    }
+    corrupted = {}
+    for snapshot in snapshots.iterator():
+        link = snapshot.as_link()
+        if is_corrupt(link):
+            corrupted[link.link_dir] = link
+    return corrupted
 
-def get_unrecognized_folders(links, out_dir: str=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
+def get_unrecognized_folders(snapshots, out_dir: Path=OUTPUT_DIR) -> Dict[str, Optional[Link]]:
     """dirs that don't contain recognizable archive data and aren't listed in the main index"""
-    by_timestamp = {link.timestamp: 0 for link in links}
     unrecognized_folders: Dict[str, Optional[Link]] = {}
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
-        if entry.is_dir(follow_symlinks=True):
-            index_exists = os.path.exists(os.path.join(entry.path, 'index.json'))
+    for entry in (Path(out_dir) / ARCHIVE_DIR_NAME).iterdir():
+        if entry.is_dir():
+            index_exists = (entry / "index.json").exists()
             link = None
             try:
-                link = parse_json_link_details(entry.path)
+                link = parse_json_link_details(str(entry))
             except KeyError:
                 # Try to fix index
                 if index_exists:
                     try:
                         # Last attempt to repair the detail index
-                        link_guessed = parse_json_link_details(entry.path, guess=True)
-                        write_json_link_details(link_guessed, out_dir=entry.path)
-                        link = parse_json_link_details(entry.path)
+                        link_guessed = parse_json_link_details(str(entry), guess=True)
+                        write_json_link_details(link_guessed, out_dir=str(entry))
+                        link = parse_json_link_details(str(entry))
                     except Exception:
                         pass
 
             if index_exists and link is None:
                 # index exists but it's corrupted or unparseable
-                unrecognized_folders[entry.path] = link
+                unrecognized_folders[str(entry)] = link
             
             elif not index_exists:
                 # link details index doesn't exist and the folder isn't in the main index
-                timestamp = entry.path.rsplit('/', 1)[-1]
-                if timestamp not in by_timestamp:
-                    unrecognized_folders[entry.path] = link
+                timestamp = entry.name
+                if not snapshots.filter(timestamp=timestamp).exists():
+                    unrecognized_folders[str(entry)] = link
 
     return unrecognized_folders
 
 
 def is_valid(link: Link) -> bool:
-    dir_exists = os.path.exists(link.link_dir)
-    index_exists = os.path.exists(os.path.join(link.link_dir, 'index.json'))
+    dir_exists = Path(link.link_dir).exists()
+    index_exists = (Path(link.link_dir) / "index.json").exists()
     if not dir_exists:
         # unarchived links are not included in the valid list
         return False
@@ -569,7 +571,7 @@ def is_valid(link: Link) -> bool:
     return False
 
 def is_corrupt(link: Link) -> bool:
-    if not os.path.exists(link.link_dir):
+    if not Path(link.link_dir).exists():
         # unarchived links are not considered corrupt
         return False
 
@@ -582,17 +584,17 @@ def is_archived(link: Link) -> bool:
     return is_valid(link) and link.is_archived
     
 def is_unarchived(link: Link) -> bool:
-    if not os.path.exists(link.link_dir):
+    if not Path(link.link_dir).exists():
         return True
     return not link.is_archived
 
 
-def fix_invalid_folder_locations(out_dir: str=OUTPUT_DIR) -> Tuple[List[str], List[str]]:
+def fix_invalid_folder_locations(out_dir: Path=OUTPUT_DIR) -> Tuple[List[str], List[str]]:
     fixed = []
     cant_fix = []
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
+    for entry in os.scandir(out_dir / ARCHIVE_DIR_NAME):
         if entry.is_dir(follow_symlinks=True):
-            if os.path.exists(os.path.join(entry.path, 'index.json')):
+            if (Path(entry.path) / 'index.json').exists():
                 try:
                     link = parse_json_link_details(entry.path)
                 except KeyError:
@@ -601,8 +603,8 @@ def fix_invalid_folder_locations(out_dir: str=OUTPUT_DIR) -> Tuple[List[str], Li
                     continue
 
                 if not entry.path.endswith(f'/{link.timestamp}'):
-                    dest = os.path.join(out_dir, ARCHIVE_DIR_NAME, link.timestamp)
-                    if os.path.exists(dest):
+                    dest = out_dir / ARCHIVE_DIR_NAME / link.timestamp
+                    if dest.exists():
                         cant_fix.append(entry.path)
                     else:
                         shutil.move(entry.path, dest)

archivebox/index/html.py

@@ -1,13 +1,15 @@
 __package__ = 'archivebox.index'
 
-import os
-
-from string import Template
 from datetime import datetime
 from typing import List, Optional, Iterator, Mapping
+from pathlib import Path
+
+from django.utils.html import format_html, mark_safe
+from collections import defaultdict
 
 from .schema import Link
-from ..system import atomic_write, copy_and_overwrite
+from ..system import atomic_write
+from ..logging_util import printable_filesize
 from ..util import (
     enforce_types,
     ts_to_date,
@@ -17,32 +19,27 @@ from ..util import (
 )
 from ..config import (
     OUTPUT_DIR,
-    TEMPLATES_DIR,
     VERSION,
     GIT_SHA,
     FOOTER_INFO,
-    ARCHIVE_DIR_NAME,
     HTML_INDEX_FILENAME,
-    STATIC_DIR_NAME,
-    ROBOTS_TXT_FILENAME,
-    FAVICON_FILENAME,
+    SAVE_ARCHIVE_DOT_ORG,
 )
 
-join = lambda *paths: os.path.join(*paths)
-MAIN_INDEX_TEMPLATE = join(TEMPLATES_DIR, 'main_index.html')
-MAIN_INDEX_ROW_TEMPLATE = join(TEMPLATES_DIR, 'main_index_row.html')
-LINK_DETAILS_TEMPLATE = join(TEMPLATES_DIR, 'link_details.html')
+MAIN_INDEX_TEMPLATE = 'static_index.html'
+MINIMAL_INDEX_TEMPLATE = 'minimal_index.html'
+LINK_DETAILS_TEMPLATE = 'snapshot.html'
 TITLE_LOADING_MSG = 'Not yet archived...'
 
 
 ### Main Links Index
 
 @enforce_types
-def parse_html_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[str]:
+def parse_html_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[str]:
     """parse an archive index html file and return the list of urls"""
 
-    index_path = join(out_dir, HTML_INDEX_FILENAME)
-    if os.path.exists(index_path):
+    index_path = Path(out_dir) / HTML_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             for line in f:
                 if 'class="link-url"' in line:
@@ -50,62 +47,25 @@ def parse_html_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[str]:
     return ()
 
 @enforce_types
-def write_html_main_index(links: List[Link], out_dir: str=OUTPUT_DIR, finished: bool=False) -> None:
-    """write the html link index to a given path"""
-
-    copy_and_overwrite(join(TEMPLATES_DIR, FAVICON_FILENAME), join(out_dir, FAVICON_FILENAME))
-    copy_and_overwrite(join(TEMPLATES_DIR, ROBOTS_TXT_FILENAME), join(out_dir, ROBOTS_TXT_FILENAME))
-    copy_and_overwrite(join(TEMPLATES_DIR, STATIC_DIR_NAME), join(out_dir, STATIC_DIR_NAME))
-    
-    rendered_html = main_index_template(links, finished=finished)
-    atomic_write(join(out_dir, HTML_INDEX_FILENAME), rendered_html)
-
+def generate_index_from_links(links: List[Link], with_headers: bool):
+    if with_headers:
+        output = main_index_template(links)
+    else:
+        output = main_index_template(links, template=MINIMAL_INDEX_TEMPLATE)
+    return output
 
 @enforce_types
-def main_index_template(links: List[Link], finished: bool=True) -> str:
+def main_index_template(links: List[Link], template: str=MAIN_INDEX_TEMPLATE) -> str:
     """render the template for the entire main index"""
 
-    return render_legacy_template(MAIN_INDEX_TEMPLATE, {
+    return render_django_template(template, {
         'version': VERSION,
         'git_sha': GIT_SHA,
         'num_links': str(len(links)),
-        'status': 'finished' if finished else 'running',
         'date_updated': datetime.now().strftime('%Y-%m-%d'),
         'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'),
-        'rows': '\n'.join(
-            main_index_row_template(link)
-            for link in links
-        ),
-        'footer_info': FOOTER_INFO,
-    })
-
-
-@enforce_types
-def main_index_row_template(link: Link) -> str:
-    """render the template for an individual link row of the main index"""
-
-    from ..extractors.wget import wget_output_path
-
-    return render_legacy_template(MAIN_INDEX_ROW_TEMPLATE, {
-        **link._asdict(extended=True),
-        
-        # before pages are finished archiving, show loading msg instead of title
-        'title': htmlencode(
-            link.title
-            or (link.base_url if link.is_archived else TITLE_LOADING_MSG)
-        ),
-
-        # before pages are finished archiving, show fallback loading favicon
-        'favicon_url': (
-            join(ARCHIVE_DIR_NAME, link.timestamp, 'favicon.ico')
-            # if link['is_archived'] else 'data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs='
-        ),
-
-        # before pages are finished archiving, show the details page instead
-        'wget_url': urlencode(wget_output_path(link) or 'index.html'),
-        
-        # replace commas in tags with spaces, or file extension if it's static
-        'tags': (link.tags or '') + (' {}'.format(link.extension) if link.is_static else ''),
+        'links': [link._asdict(extended=True) for link in links],
+        'FOOTER_INFO': FOOTER_INFO,
     })
 
 
@@ -116,7 +76,7 @@ def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None:
     out_dir = out_dir or link.link_dir
 
     rendered_html = link_details_template(link)
-    atomic_write(join(out_dir, HTML_INDEX_FILENAME), rendered_html)
+    atomic_write(str(Path(out_dir) / HTML_INDEX_FILENAME), rendered_html)
 
 
 @enforce_types
@@ -126,7 +86,7 @@ def link_details_template(link: Link) -> str:
 
     link_info = link._asdict(extended=True)
 
-    return render_legacy_template(LINK_DETAILS_TEMPLATE, {
+    return render_django_template(LINK_DETAILS_TEMPLATE, {
         **link_info,
         **link_info['canonical'],
         'title': htmlencode(
@@ -140,17 +100,87 @@ def link_details_template(link: Link) -> str:
         ) or 'about:blank',
         'extension': link.extension or 'html',
         'tags': link.tags or 'untagged',
+        'size': printable_filesize(link.archive_size) if link.archive_size else 'pending',
         'status': 'archived' if link.is_archived else 'not yet archived',
         'status_color': 'success' if link.is_archived else 'danger',
         'oldest_archive_date': ts_to_date(link.oldest_archive_date),
+        'SAVE_ARCHIVE_DOT_ORG': SAVE_ARCHIVE_DOT_ORG,
     })
 
-
 @enforce_types
-def render_legacy_template(template_path: str, context: Mapping[str, str]) -> str:
+def render_django_template(template: str, context: Mapping[str, str]) -> str:
     """render a given html template string with the given template content"""
+    from django.template.loader import render_to_string
 
-    # will be replaced by django templates in the future
-    with open(template_path, 'r', encoding='utf-8') as template:
-        template_str = template.read()
-    return Template(template_str).substitute(**context)
+    return render_to_string(template, context)
+
+
+def snapshot_icons(snapshot) -> str:
+    from core.models import EXTRACTORS
+
+    # start = datetime.now()
+
+    archive_results = snapshot.archiveresult_set.filter(status="succeeded")
+    link = snapshot.as_link()
+    path = link.archive_path
+    canon = link.canonical_outputs()
+    output = ""
+    output_template = '<a href="/{}/{}" class="exists-{}" title="{}">{}</a> &nbsp;'
+    icons = {
+        "singlefile": "❶",
+        "wget": "🆆",
+        "dom": "🅷",
+        "pdf": "📄",
+        "screenshot": "💻",
+        "media": "📼",
+        "git": "🅶",
+        "archive_org": "🏛",
+        "readability": "🆁",
+        "mercury": "🅼",
+        "warc": "📦"
+    }
+    exclude = ["favicon", "title", "headers", "archive_org"]
+    # Missing specific entry for WARC
+
+    extractor_outputs = defaultdict(lambda: None)
+    for extractor, _ in EXTRACTORS:
+        for result in archive_results:
+            if result.extractor == extractor and result:
+                extractor_outputs[extractor] = result
+
+    for extractor, _ in EXTRACTORS:
+        if extractor not in exclude:
+            existing = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # Check filesystsem to see if anything is actually present (too slow, needs optimization/caching)
+            # if existing:
+            #     existing = (Path(path) / existing)
+            #     if existing.is_file():
+            #         existing = True
+            #     elif existing.is_dir():
+            #         existing = any(existing.glob('*.*'))
+            output += format_html(output_template, path, canon[f"{extractor}_path"], str(bool(existing)),
+                                         extractor, icons.get(extractor, "?"))
+        if extractor == "wget":
+            # warc isn't technically it's own extractor, so we have to add it after wget
+            
+            # get from db (faster but less thurthful)
+            exists = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # get from filesystem (slower but more accurate)
+            # exists = list((Path(path) / canon["warc_path"]).glob("*.warc.gz"))
+            output += format_html(output_template, 'warc/', canon["warc_path"], str(bool(exists)), "warc", icons.get("warc", "?"))
+
+        if extractor == "archive_org":
+            # The check for archive_org is different, so it has to be handled separately
+
+            # get from db (faster)
+            exists = extractor_outputs[extractor] and extractor_outputs[extractor].status == 'succeeded' and extractor_outputs[extractor].output
+            # get from filesystem (slower)
+            # target_path = Path(path) / "archive.org.txt"
+            # exists = target_path.exists()
+            output += '<a href="{}" class="exists-{}" title="{}">{}</a> '.format(canon["archive_org_path"], str(exists),
+                                                                                        "archive_org", icons.get("archive_org", "?"))
+
+    result = format_html('<span class="files-icons" style="font-size: 1.1em; opacity: 0.8; min-width: 240px; display: inline-block">{}<span>', mark_safe(output))
+    # end = datetime.now()
+    # print(((end - start).total_seconds()*1000) // 1, 'ms')
+    return result

archivebox/index/json.py

@@ -6,9 +6,9 @@ import json as pyjson
 from pathlib import Path
 
 from datetime import datetime
-from typing import List, Optional, Iterator, Any
+from typing import List, Optional, Iterator, Any, Union
 
-from .schema import Link, ArchiveResult
+from .schema import Link
 from ..system import atomic_write
 from ..util import enforce_types
 from ..config import (
@@ -32,21 +32,34 @@ MAIN_INDEX_HEADER = {
         'version': VERSION,
         'git_sha': GIT_SHA,
         'website': 'https://ArchiveBox.io',
-        'docs': 'https://github.com/pirate/ArchiveBox/wiki',
-        'source': 'https://github.com/pirate/ArchiveBox',
-        'issues': 'https://github.com/pirate/ArchiveBox/issues',
+        'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki',
+        'source': 'https://github.com/ArchiveBox/ArchiveBox',
+        'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues',
         'dependencies': DEPENDENCIES,
     },
 }
 
-### Main Links Index
+@enforce_types
+def generate_json_index_from_links(links: List[Link], with_headers: bool):
+    if with_headers:
+        output = {
+            **MAIN_INDEX_HEADER,
+            'num_links': len(links),
+            'updated': datetime.now(),
+            'last_run_cmd': sys.argv,
+            'links': links,
+        }
+    else:
+        output = links
+    return to_json(output, indent=4, sort_keys=True)
+
 
 @enforce_types
-def parse_json_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
+def parse_json_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]:
     """parse an archive index json file and return the list of links"""
 
-    index_path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(index_path):
+    index_path = Path(out_dir) / JSON_INDEX_FILENAME
+    if index_path.exists():
         with open(index_path, 'r', encoding='utf-8') as f:
             links = pyjson.load(f)['links']
             for link_json in links:
@@ -65,30 +78,6 @@ def parse_json_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
                             continue
     return ()
 
-@enforce_types
-def write_json_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    """write the json link index to a given path"""
-
-    assert isinstance(links, List), 'Links must be a list, not a generator.'
-    assert not links or isinstance(links[0].history, dict)
-    assert not links or isinstance(links[0].sources, list)
-
-    if links and links[0].history.get('title'):
-        assert isinstance(links[0].history['title'][0], ArchiveResult)
-
-    if links and links[0].sources:
-        assert isinstance(links[0].sources[0], str)
-
-    main_index_json = {
-        **MAIN_INDEX_HEADER,
-        'num_links': len(links),
-        'updated': datetime.now(),
-        'last_run_cmd': sys.argv,
-        'links': links,
-    }
-    atomic_write(os.path.join(out_dir, JSON_INDEX_FILENAME), main_index_json)
-
-
 ### Link Details Index
 
 @enforce_types
@@ -96,15 +85,15 @@ def write_json_link_details(link: Link, out_dir: Optional[str]=None) -> None:
     """write a json file with some info about the link"""
     
     out_dir = out_dir or link.link_dir
-    path = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    atomic_write(path, link._asdict(extended=True))
+    path = Path(out_dir) / JSON_INDEX_FILENAME
+    atomic_write(str(path), link._asdict(extended=True))
 
 
 @enforce_types
-def parse_json_link_details(out_dir: str, guess: Optional[bool]=False) -> Optional[Link]:
+def parse_json_link_details(out_dir: Union[Path, str], guess: Optional[bool]=False) -> Optional[Link]:
     """load the json link index from a given directory"""
-    existing_index = os.path.join(out_dir, JSON_INDEX_FILENAME)
-    if os.path.exists(existing_index):
+    existing_index = Path(out_dir) / JSON_INDEX_FILENAME
+    if existing_index.exists():
         with open(existing_index, 'r', encoding='utf-8') as f:
             try:
                 link_json = pyjson.load(f)
@@ -115,12 +104,12 @@ def parse_json_link_details(out_dir: str, guess: Optional[bool]=False) -> Option
 
 
 @enforce_types
-def parse_json_links_details(out_dir: str) -> Iterator[Link]:
+def parse_json_links_details(out_dir: Union[Path, str]) -> Iterator[Link]:
     """read through all the archive data folders and return the parsed links"""
 
-    for entry in os.scandir(os.path.join(out_dir, ARCHIVE_DIR_NAME)):
+    for entry in os.scandir(Path(out_dir) / ARCHIVE_DIR_NAME):
         if entry.is_dir(follow_symlinks=True):
-            if os.path.exists(os.path.join(entry.path, 'index.json')):
+            if (Path(entry.path) / 'index.json').exists():
                 try:
                     link = parse_json_link_details(entry.path)
                 except KeyError:

archivebox/index/schema.py

@@ -1,6 +1,13 @@
+"""
+
+WARNING: THIS FILE IS ALL LEGACY CODE TO BE REMOVED.
+
+DO NOT ADD ANY NEW FEATURES TO THIS FILE, NEW CODE GOES HERE: core/models.py
+
+"""
+
 __package__ = 'archivebox.index'
 
-import os
 from pathlib import Path
 
 from datetime import datetime, timedelta
@@ -32,6 +39,7 @@ class ArchiveResult:
     status: str
     start_ts: datetime
     end_ts: datetime
+    index_texts: Union[List[str], None] = None
     schema: str = 'ArchiveResult'
 
     def __post_init__(self):
@@ -127,7 +135,7 @@ class Link:
 
 
     def __str__(self) -> str:
-        return f'[{self.timestamp}] {self.base_url} "{self.title}"'
+        return f'[{self.timestamp}] {self.url} "{self.title}"'
 
     def __post_init__(self):
         self.typecheck()
@@ -208,6 +216,10 @@ class Link:
             })
         return info
 
+    def as_snapshot(self):
+        from core.models import Snapshot
+        return Snapshot.objects.get(url=self.url)
+
     @classmethod
     def from_json(cls, json_info, guess=False):
         from ..util import parse_date
@@ -250,7 +262,7 @@ class Link:
     @property
     def link_dir(self) -> str:
         from ..config import CONFIG
-        return os.path.join(CONFIG['ARCHIVE_DIR'], self.timestamp)
+        return str(Path(CONFIG['ARCHIVE_DIR']) / self.timestamp)
 
     @property
     def archive_path(self) -> str:
@@ -340,7 +352,7 @@ class Link:
     ### Archive Status Helpers
     @property
     def num_outputs(self) -> int:
-        return len(tuple(filter(None, self.latest_outputs().values())))
+        return self.as_snapshot().num_outputs
 
     @property
     def num_failures(self) -> int:
@@ -365,10 +377,11 @@ class Link:
             'screenshot.png',
             'output.html',
             'media',
+            'singlefile.html'
         )
 
         return any(
-            os.path.exists(os.path.join(ARCHIVE_DIR, self.timestamp, path))
+            (Path(ARCHIVE_DIR) / self.timestamp / path).exists()
             for path in output_paths
         )
 
@@ -376,7 +389,7 @@ class Link:
         """get the latest output that each archive method produced for link"""
         
         ARCHIVE_METHODS = (
-            'title', 'favicon', 'wget', 'warc', 'pdf',
+            'title', 'favicon', 'wget', 'warc', 'singlefile', 'pdf',
             'screenshot', 'dom', 'git', 'media', 'archive_org',
         )
         latest: Dict[str, ArchiveOutput] = {}
@@ -392,7 +405,6 @@ class Link:
                 latest[archive_method] = history[0].output
             else:
                 latest[archive_method] = None
-
         return latest
 
 
@@ -400,18 +412,24 @@ class Link:
         """predict the expected output paths that should be present after archiving"""
 
         from ..extractors.wget import wget_output_path
+        # TODO: banish this awful duplication from the codebase and import these
+        # from their respective extractor files
         canonical = {
             'index_path': 'index.html',
             'favicon_path': 'favicon.ico',
             'google_favicon_path': 'https://www.google.com/s2/favicons?domain={}'.format(self.domain),
             'wget_path': wget_output_path(self),
-            'warc_path': 'warc',
+            'warc_path': 'warc/',
+            'singlefile_path': 'singlefile.html',
+            'readability_path': 'readability/content.html',
+            'mercury_path': 'mercury/content.html',
             'pdf_path': 'output.pdf',
             'screenshot_path': 'screenshot.png',
             'dom_path': 'output.html',
             'archive_org_path': 'https://web.archive.org/web/{}'.format(self.base_url),
-            'git_path': 'git',
-            'media_path': 'media',
+            'git_path': 'git/',
+            'media_path': 'media/',
+            'headers_path': 'headers.json',
         }
         if self.is_static:
             # static binary files like PDF and images are handled slightly differently.
@@ -425,7 +443,9 @@ class Link:
                 'pdf_path': static_path,
                 'screenshot_path': static_path,
                 'dom_path': static_path,
+                'singlefile_path': static_path,
+                'readability_path': static_path,
+                'mercury_path': static_path,
             })
         return canonical
 
-

archivebox/index/sql.py

@@ -1,18 +1,20 @@
 __package__ = 'archivebox.index'
 
 from io import StringIO
+from pathlib import Path
 from typing import List, Tuple, Iterator
+from django.db.models import QuerySet
+from django.db import transaction
 
 from .schema import Link
 from ..util import enforce_types
-from ..config import setup_django, OUTPUT_DIR
+from ..config import OUTPUT_DIR
 
 
 ### Main Links Index
 
 @enforce_types
-def parse_sql_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
-    setup_django(out_dir, check_db=True)
+def parse_sql_main_index(out_dir: Path=OUTPUT_DIR) -> Iterator[Link]:
     from core.models import Snapshot
 
     return (
@@ -21,43 +23,59 @@ def parse_sql_main_index(out_dir: str=OUTPUT_DIR) -> Iterator[Link]:
     )
 
 @enforce_types
-def remove_from_sql_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
-    from core.models import Snapshot
-    from django.db import transaction
-
+def remove_from_sql_main_index(snapshots: QuerySet, out_dir: Path=OUTPUT_DIR) -> None:
     with transaction.atomic():
-        for link in links:
-            Snapshot.objects.filter(url=link.url).delete()
+        snapshots.delete()
 
 @enforce_types
-def write_sql_main_index(links: List[Link], out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
+def write_link_to_sql_index(link: Link):
     from core.models import Snapshot
-    from django.db import transaction
+    info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys}
+    tags = info.pop("tags")
+    if tags is None:
+        tags = []
+
+    try:
+        info["timestamp"] = Snapshot.objects.get(url=link.url).timestamp
+    except Snapshot.DoesNotExist:
+        while Snapshot.objects.filter(timestamp=info["timestamp"]).exists():
+            info["timestamp"] = str(float(info["timestamp"]) + 1.0)
+
+    snapshot, _ = Snapshot.objects.update_or_create(url=link.url, defaults=info)
+    snapshot.save_tags(tags)
+    return snapshot
 
-    with transaction.atomic():
-        for link in links:
-            info = {k: v for k, v in link._asdict().items() if k in Snapshot.keys}
-            Snapshot.objects.update_or_create(url=link.url, defaults=info)
 
 @enforce_types
-def write_sql_link_details(link: Link, out_dir: str=OUTPUT_DIR) -> None:
-    setup_django(out_dir, check_db=True)
+def write_sql_main_index(links: List[Link], out_dir: Path=OUTPUT_DIR) -> None:
+    with transaction.atomic():
+        for link in links:
+            write_link_to_sql_index(link)
+            
+
+@enforce_types
+def write_sql_link_details(link: Link, out_dir: Path=OUTPUT_DIR) -> None:
     from core.models import Snapshot
-    from django.db import transaction
 
     with transaction.atomic():
-        snap = Snapshot.objects.get(url=link.url, timestamp=link.timestamp)
+        try:
+            snap = Snapshot.objects.get(url=link.url)
+        except Snapshot.DoesNotExist:
+            snap = write_link_to_sql_index(link)
         snap.title = link.title
-        snap.tags = link.tags
+
+        tag_set = (
+            set(tag.strip() for tag in (link.tags or '').split(','))
+        )
+        tag_list = list(tag_set) or []
+
         snap.save()
+        snap.save_tags(tag_list)
 
 
 
 @enforce_types
-def list_migrations(out_dir: str=OUTPUT_DIR) -> List[Tuple[bool, str]]:
-    setup_django(out_dir, check_db=False)
+def list_migrations(out_dir: Path=OUTPUT_DIR) -> List[Tuple[bool, str]]:
     from django.core.management import call_command
     out = StringIO()
     call_command("showmigrations", list=True, stdout=out)
@@ -73,8 +91,7 @@ def list_migrations(out_dir: str=OUTPUT_DIR) -> List[Tuple[bool, str]]:
     return migrations
 
 @enforce_types
-def apply_migrations(out_dir: str=OUTPUT_DIR) -> List[str]:
-    setup_django(out_dir, check_db=False)
+def apply_migrations(out_dir: Path=OUTPUT_DIR) -> List[str]:
     from django.core.management import call_command
     null, out = StringIO(), StringIO()
     call_command("makemigrations", interactive=False, stdout=null)
@@ -84,7 +101,6 @@ def apply_migrations(out_dir: str=OUTPUT_DIR) -> List[str]:
     return [line.strip() for line in out.readlines() if line.strip()]
 
 @enforce_types
-def get_admins(out_dir: str=OUTPUT_DIR) -> List[str]:
-    setup_django(out_dir, check_db=False)
+def get_admins(out_dir: Path=OUTPUT_DIR) -> List[str]:
     from django.contrib.auth.models import User
     return User.objects.filter(is_superuser=True)

archivebox/logging_util.py

@@ -5,7 +5,9 @@ import os
 import sys
 import time
 import argparse
+from math import log
 from multiprocessing import Process
+from pathlib import Path
 
 from datetime import datetime
 from dataclasses import dataclass
@@ -17,13 +19,13 @@ if TYPE_CHECKING:
 from .util import enforce_types
 from .config import (
     ConfigDict,
+    OUTPUT_DIR,
     PYTHON_ENCODING,
     ANSI,
     IS_TTY,
     TERM_WIDTH,
-    OUTPUT_DIR,
+    SHOW_PROGRESS,
     SOURCES_DIR_NAME,
-    HTML_INDEX_FILENAME,
     stderr,
 )
 
@@ -83,7 +85,6 @@ class TimedProgress:
     """Show a progress bar and measure elapsed time until .end() is called"""
 
     def __init__(self, seconds, prefix=''):
-        from .config import SHOW_PROGRESS
         self.SHOW_PROGRESS = SHOW_PROGRESS
         if self.SHOW_PROGRESS:
             self.p = Process(target=progress_bar, args=(seconds, prefix))
@@ -99,15 +100,23 @@ class TimedProgress:
         
         if self.SHOW_PROGRESS:
             # terminate if we havent already terminated
-            self.p.terminate()
-            self.p.join()
-            self.p.close()
-
-            # clear whole terminal line
             try:
-                sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
-            except (IOError, BrokenPipeError):
-                # ignore when the parent proc has stopped listening to our stdout
+                # kill the progress bar subprocess
+                try:
+                    self.p.close()   # must be closed *before* its terminnated
+                except:
+                    pass
+                self.p.terminate()
+                self.p.join()
+
+
+                # clear whole terminal line
+                try:
+                    sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
+                except (IOError, BrokenPipeError):
+                    # ignore when the parent proc has stopped listening to our stdout
+                    pass
+            except ValueError:
                 pass
 
 
@@ -126,17 +135,18 @@ def progress_bar(seconds: int, prefix: str='') -> None:
                 sys.stdout.write('\r\n')
                 sys.stdout.flush()
             chunks = max_width - len(prefix) - 20
-            progress = s / chunks / seconds * 100
-            bar_width = round(progress/(100/chunks))
+            pct_complete = s / chunks / seconds * 100
+            log_pct = (log(pct_complete or 1, 10) / 2) * 100  # everyone likes faster progress bars ;)
+            bar_width = round(log_pct/(100/chunks))
             last_width = max_width
 
             # ████████████████████           0.9% (1/60sec)
             sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format(
                 prefix,
-                ANSI['green'],
+                ANSI['green' if pct_complete < 80 else 'lightyellow'],
                 (chunk * bar_width).ljust(chunks),
                 ANSI['reset'],
-                round(progress, 1),
+                round(pct_complete, 1),
                 round(s/chunks),
                 seconds,
             ))
@@ -144,7 +154,7 @@ def progress_bar(seconds: int, prefix: str='') -> None:
             time.sleep(1 / chunks)
 
         # ██████████████████████████████████ 100.0% (60/60sec)
-        sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)\n'.format(
+        sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format(
             prefix,
             ANSI['red'],
             chunk * chunks,
@@ -154,6 +164,10 @@ def progress_bar(seconds: int, prefix: str='') -> None:
             seconds,
         ))
         sys.stdout.flush()
+        # uncomment to have it disappear when it hits 100% instead of staying full red:
+        # time.sleep(0.5)
+        # sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH()), ANSI['reset']))
+        # sys.stdout.flush()
     except (KeyboardInterrupt, BrokenPipeError):
         print()
         pass
@@ -162,12 +176,10 @@ def progress_bar(seconds: int, prefix: str='') -> None:
 def log_cli_command(subcommand: str, subcommand_args: List[str], stdin: Optional[str], pwd: str):
     from .config import VERSION, ANSI
     cmd = ' '.join(('archivebox', subcommand, *subcommand_args))
-    stdin_hint = ' < /dev/stdin' if not stdin.isatty() else ''
-    stderr('{black}[i] [{now}] ArchiveBox v{VERSION}: {cmd}{stdin_hint}{reset}'.format(
+    stderr('{black}[i] [{now}] ArchiveBox v{VERSION}: {cmd}{reset}'.format(
         now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
         VERSION=VERSION,
         cmd=cmd,
-        stdin_hint=stdin_hint,
         **ANSI,
     ))
     stderr('{black}    > {pwd}{reset}'.format(pwd=pwd, **ANSI))
@@ -198,7 +210,8 @@ def log_deduping_finished(num_new_links: int):
 
 
 def log_crawl_started(new_links):
-    print('{lightred}[*] Starting crawl of {} sites 1 hop out from starting point{reset}'.format(len(new_links), **ANSI))
+    print()
+    print('{green}[*] Starting crawl of {} sites 1 hop out from starting point{reset}'.format(len(new_links), **ANSI))
 
 ### Indexing Stage
 
@@ -241,7 +254,7 @@ def log_archiving_started(num_links: int, resume: Optional[float]=None):
              **ANSI,
         ))
     else:
-        print('{green}[▶] [{}] Collecting content for {} Snapshots in archive...{reset}'.format(
+        print('{green}[▶] [{}] Starting archiving of {} snapshots in index...{reset}'.format(
              start_ts.strftime('%Y-%m-%d %H:%M:%S'),
              num_links,
              **ANSI,
@@ -259,8 +272,8 @@ def log_archiving_paused(num_links: int, idx: int, timestamp: str):
         total=num_links,
     ))
     print()
-    print('    {lightred}Hint:{reset} To view your archive index, open:'.format(**ANSI))
-    print('        {}/{}'.format(OUTPUT_DIR, HTML_INDEX_FILENAME))
+    print('    {lightred}Hint:{reset} To view your archive index, run:'.format(**ANSI))
+    print('        archivebox server  # then visit http://127.0.0.1:8000')
     print('    Continue archiving where you left off by running:')
     print('        archivebox update --resume={}'.format(timestamp))
 
@@ -283,13 +296,11 @@ def log_archiving_finished(num_links: int):
         ANSI['reset'],
     ))
     print('    - {} links skipped'.format(_LAST_RUN_STATS.skipped))
-    print('    - {} links updated'.format(_LAST_RUN_STATS.succeeded))
+    print('    - {} links updated'.format(_LAST_RUN_STATS.succeeded + _LAST_RUN_STATS.failed))
     print('    - {} links had errors'.format(_LAST_RUN_STATS.failed))
     print()
-    print('    {lightred}Hint:{reset} To view your archive index, open:'.format(**ANSI))
-    print('        {}/{}'.format(OUTPUT_DIR, HTML_INDEX_FILENAME))
-    print('    Or run the built-in webserver:')
-    print('        archivebox server')
+    print('    {lightred}Hint:{reset} To manage your archive in a Web UI, run:'.format(**ANSI))
+    print('        archivebox server 0.0.0.0:8000')
 
 
 def log_link_archiving_started(link: "Link", link_dir: str, is_new: bool):
@@ -336,6 +347,21 @@ def log_archive_method_finished(result: "ArchiveResult"):
     )
 
     if result.status == 'failed':
+        if result.output.__class__.__name__ == 'TimeoutExpired':
+            duration = (result.end_ts - result.start_ts).seconds
+            hint_header = [
+                '{lightyellow}Extractor timed out after {}s.{reset}'.format(duration, **ANSI),
+            ]
+        else:
+            hint_header = [
+                '{lightyellow}Extractor failed:{reset}'.format(**ANSI),
+                '    {reset}{} {red}{}{reset}'.format(
+                    result.output.__class__.__name__.replace('ArchiveError', ''),
+                    result.output, 
+                    **ANSI,
+                ),
+            ]
+
         # Prettify error output hints string and limit to five lines
         hints = getattr(result.output, 'hints', None) or ()
         if hints:
@@ -345,14 +371,10 @@ def log_archive_method_finished(result: "ArchiveResult"):
                 for line in hints[:5] if line.strip()
             )
 
+
         # Collect and prefix output lines with indentation
         output_lines = [
-            '{lightred}Failed:{reset}'.format(**ANSI),
-            '    {reset}{} {red}{}{reset}'.format(
-                result.output.__class__.__name__.replace('ArchiveError', ''),
-                result.output, 
-                **ANSI,
-            ),
+            *hint_header,
             *hints,
             '{}Run to see full output:{}'.format(ANSI['lightred'], ANSI['reset']),
             *(['    cd {};'.format(result.pwd)] if result.pwd else []),
@@ -385,7 +407,7 @@ def log_list_finished(links):
 def log_removal_started(links: List["Link"], yes: bool, delete: bool):
     print('{lightyellow}[i] Found {} matching URLs to remove.{reset}'.format(len(links), **ANSI))
     if delete:
-        file_counts = [link.num_outputs for link in links if os.path.exists(link.link_dir)]
+        file_counts = [link.num_outputs for link in links if Path(link.link_dir).exists()]
         print(
             f'    {len(links)} Links will be de-listed from the main index, and their archived content folders will be deleted from disk.\n'
             f'    ({len(file_counts)} data folders with {sum(file_counts)} archived files will be deleted!)'
@@ -404,30 +426,29 @@ def log_removal_started(links: List["Link"], yes: bool, delete: bool):
         except (KeyboardInterrupt, EOFError, AssertionError):
             raise SystemExit(0)
 
-def log_removal_finished(all_links: int, to_keep: int):
+def log_removal_finished(all_links: int, to_remove: int):
     if all_links == 0:
         print()
         print('{red}[X] No matching links found.{reset}'.format(**ANSI))
     else:
-        num_removed = all_links - to_keep
         print()
         print('{red}[√] Removed {} out of {} links from the archive index.{reset}'.format(
-            num_removed,
+            to_remove,
             all_links,
             **ANSI,
         ))
-        print('    Index now contains {} links.'.format(to_keep))
+        print('    Index now contains {} links.'.format(all_links - to_remove))
 
 
 def log_shell_welcome_msg():
     from .cli import list_subcommands
 
     print('{green}# ArchiveBox Imports{reset}'.format(**ANSI))
-    print('{green}from archivebox.core.models import Snapshot, User{reset}'.format(**ANSI))
+    print('{green}from core.models import Snapshot, User{reset}'.format(**ANSI))
     print('{green}from archivebox import *\n    {}{reset}'.format("\n    ".join(list_subcommands().keys()), **ANSI))
     print()
     print('[i] Welcome to the ArchiveBox Shell!')
-    print('    https://github.com/pirate/ArchiveBox/wiki/Usage#Shell-Usage')
+    print('    https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Shell-Usage')
     print()
     print('    {lightred}Hint:{reset} Example use:'.format(**ANSI))
     print('        print(Snapshot.objects.filter(is_archived=True).count())')
@@ -439,11 +460,11 @@ def log_shell_welcome_msg():
 ### Helpers
 
 @enforce_types
-def pretty_path(path: str) -> str:
+def pretty_path(path: Union[Path, str]) -> str:
     """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc"""
-    pwd = os.path.abspath('.')
+    pwd = Path('.').resolve()
     # parent = os.path.abspath(os.path.join(pwd, os.path.pardir))
-    return path.replace(pwd + '/', './')
+    return str(path).replace(str(pwd) + '/', './')
 
 
 @enforce_types
@@ -457,17 +478,11 @@ def printable_filesize(num_bytes: Union[int, float]) -> str:
 
 @enforce_types
 def printable_folders(folders: Dict[str, Optional["Link"]],
-                      json: bool=False,
-                      csv: Optional[str]=None) -> str:
-    if json: 
-        from .index.json import to_json
-        return to_json(folders.values(), indent=4, sort_keys=True)
-
-    elif csv:
-        from .index.csv import links_to_csv
-        return links_to_csv(folders.values(), cols=csv.split(','), header=True)
-    
-    return '\n'.join(f'{folder} {link}' for folder, link in folders.items())
+                      with_headers: bool=False) -> str:
+    return '\n'.join(
+        f'{folder} {link and link.url} "{link and link.title}"'
+        for folder, link in folders.items()
+    )
 
 
 
@@ -491,33 +506,39 @@ def printable_folder_status(name: str, folder: Dict) -> str:
         color, symbol, note, num_files = 'lightyellow', '-', 'disabled', '-'
 
     if folder['path']:
-        if os.path.exists(folder['path']):
+        if Path(folder['path']).exists():
             num_files = (
                 f'{len(os.listdir(folder["path"]))} files'
-                if os.path.isdir(folder['path']) else
-                printable_filesize(os.path.getsize(folder['path']))
+                if Path(folder['path']).is_dir() else
+                printable_filesize(Path(folder['path']).stat().st_size)
             )
         else:
             num_files = 'missing'
 
-        if ' ' in folder['path']:
-            folder['path'] = f'"{folder["path"]}"'
+    path = str(folder['path']).replace(str(OUTPUT_DIR), '.') if folder['path'] else ''
+    if path and ' ' in path:
+        path = f'"{path}"'
+
+    # if path is just a plain dot, replace it back with the full path for clarity
+    if path == '.':
+        path = str(OUTPUT_DIR)
 
     return ' '.join((
         ANSI[color],
         symbol,
         ANSI['reset'],
-        name.ljust(22),
-        (folder["path"] or '').ljust(76),
+        name.ljust(21),
         num_files.ljust(14),
         ANSI[color],
-        note,
+        note.ljust(8),
         ANSI['reset'],
+        path.ljust(76),
     ))
 
 
 @enforce_types
 def printable_dependency_version(name: str, dependency: Dict) -> str:
+    version = None
     if dependency['enabled']:
         if dependency['is_valid']:
             color, symbol, note, version = 'green', '√', 'valid', ''
@@ -531,17 +552,18 @@ def printable_dependency_version(name: str, dependency: Dict) -> str:
     else:
         color, symbol, note, version = 'lightyellow', '-', 'disabled', '-'
 
-    if ' ' in dependency["path"]:
-        dependency["path"] = f'"{dependency["path"]}"'
+    path = str(dependency["path"]).replace(str(OUTPUT_DIR), '.') if dependency["path"] else ''
+    if path and ' ' in path:
+        path = f'"{path}"'
 
     return ' '.join((
         ANSI[color],
         symbol,
         ANSI['reset'],
-        name.ljust(22),
-        (dependency["path"] or '').ljust(76),
+        name.ljust(21),
         version.ljust(14),
         ANSI[color],
-        note,
+        note.ljust(8),
         ANSI['reset'],
+        path.ljust(76),
     ))

archivebox/manage.py

@@ -8,10 +8,9 @@ if __name__ == '__main__':
     # (e.g. makemigrations), you can comment out this check temporarily
 
     if not ('makemigrations' in sys.argv or 'migrate' in sys.argv):
-        print("[X] Don't run ./manage.py directly, use the archivebox CLI instead e.g.:")
-        print('    archivebox manage createsuperuser')
+        print("[X] Don't run ./manage.py directly (unless you are a developer running makemigrations):")
         print()
-        print('    Hint: Use these archivebox commands instead of the ./manage.py equivalents:')
+        print('    Hint: Use these archivebox CLI commands instead of the ./manage.py equivalents:')
         print('        archivebox init          (migrates the databse to latest version)')
         print('        archivebox server        (runs the Django web server)')
         print('        archivebox shell         (opens an iPython Django shell with all models imported)')

archivebox/package.json

@@ -0,0 +1 @@
+../package.json

archivebox/parsers/__init__.py

@@ -8,10 +8,11 @@ For examples of supported import formats see tests/.
 __package__ = 'archivebox.parsers'
 
 import re
-import os
+from io import StringIO
 
-from typing import Tuple, List
+from typing import IO, Tuple, List, Optional
 from datetime import datetime
+from pathlib import Path 
 
 from ..system import atomic_write
 from ..config import (
@@ -22,90 +23,133 @@ from ..config import (
 )
 from ..util import (
     basename,
+    htmldecode,
     download_url,
     enforce_types,
     URL_REGEX,
 )
 from ..index.schema import Link
 from ..logging_util import TimedProgress, log_source_saved
+
 from .pocket_html import parse_pocket_html_export
+from .pocket_api import parse_pocket_api_export
 from .pinboard_rss import parse_pinboard_rss_export
+from .wallabag_atom import parse_wallabag_atom_export
 from .shaarli_rss import parse_shaarli_rss_export
 from .medium_rss import parse_medium_rss_export
 from .netscape_html import parse_netscape_html_export
 from .generic_rss import parse_generic_rss_export
 from .generic_json import parse_generic_json_export
+from .generic_html import parse_generic_html_export
 from .generic_txt import parse_generic_txt_export
 
+PARSERS = (
+    # Specialized parsers
+    ('Pocket API', parse_pocket_api_export),
+    ('Wallabag ATOM', parse_wallabag_atom_export),
+    ('Pocket HTML', parse_pocket_html_export),
+    ('Pinboard RSS', parse_pinboard_rss_export),
+    ('Shaarli RSS', parse_shaarli_rss_export),
+    ('Medium RSS', parse_medium_rss_export),
+    
+    # General parsers
+    ('Netscape HTML', parse_netscape_html_export),
+    ('Generic RSS', parse_generic_rss_export),
+    ('Generic JSON', parse_generic_json_export),
+    ('Generic HTML', parse_generic_html_export),
+
+    # Fallback parser
+    ('Plain Text', parse_generic_txt_export),
+)
+
 
 @enforce_types
-def parse_links(source_file: str) -> Tuple[List[Link], str]:
+def parse_links_memory(urls: List[str], root_url: Optional[str]=None):
+    """
+    parse a list of URLS without touching the filesystem
+    """
+    check_url_parsing_invariants()
+
+    timer = TimedProgress(TIMEOUT * 4)
+    #urls = list(map(lambda x: x + "\n", urls))
+    file = StringIO()
+    file.writelines(urls)
+    file.name = "io_string"
+    links, parser = run_parser_functions(file, timer, root_url=root_url)
+    timer.end()
+
+    if parser is None:
+        return [], 'Failed to parse'
+    return links, parser
+    
+
+@enforce_types
+def parse_links(source_file: str, root_url: Optional[str]=None) -> Tuple[List[Link], str]:
     """parse a list of URLs with their metadata from an 
        RSS feed, bookmarks export, or text file
     """
 
     check_url_parsing_invariants()
-    PARSERS = (
-        # Specialized parsers
-        ('Pocket HTML', parse_pocket_html_export),
-        ('Pinboard RSS', parse_pinboard_rss_export),
-        ('Shaarli RSS', parse_shaarli_rss_export),
-        ('Medium RSS', parse_medium_rss_export),
-        
-        # General parsers
-        ('Netscape HTML', parse_netscape_html_export),
-        ('Generic RSS', parse_generic_rss_export),
-        ('Generic JSON', parse_generic_json_export),
 
-        # Fallback parser
-        ('Plain Text', parse_generic_txt_export),
-    )
     timer = TimedProgress(TIMEOUT * 4)
     with open(source_file, 'r', encoding='utf-8') as file:
-        for parser_name, parser_func in PARSERS:
-            try:
-                links = list(parser_func(file))
-                if links:
-                    timer.end()
-                    return links, parser_name
-            except Exception as err:   # noqa
-                pass
-                # Parsers are tried one by one down the list, and the first one
-                # that succeeds is used. To see why a certain parser was not used
-                # due to error or format incompatibility, uncomment this line:
-                # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err))
-                # raise
+        links, parser = run_parser_functions(file, timer, root_url=root_url)
 
     timer.end()
-    return [], 'Failed to parse'
+    if parser is None:
+        return [], 'Failed to parse'
+    return links, parser
+
+
+def run_parser_functions(to_parse: IO[str], timer, root_url: Optional[str]=None) -> Tuple[List[Link], Optional[str]]:
+    most_links: List[Link] = []
+    best_parser_name = None
+
+    for parser_name, parser_func in PARSERS:
+        try:
+            parsed_links = list(parser_func(to_parse, root_url=root_url))
+            if not parsed_links:
+                raise Exception('no links found')
+
+            # print(f'[√] Parser {parser_name} succeeded: {len(parsed_links)} links parsed')
+            if len(parsed_links) > len(most_links):
+                most_links = parsed_links
+                best_parser_name = parser_name
+                
+        except Exception as err:                                                # noqa
+            # Parsers are tried one by one down the list, and the first one
+            # that succeeds is used. To see why a certain parser was not used
+            # due to error or format incompatibility, uncomment this line:
+            
+            # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err))
+            # raise
+            pass
+    timer.end()
+    return most_links, best_parser_name
 
 
 @enforce_types
-def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: str=OUTPUT_DIR) -> str:
+def save_text_as_source(raw_text: str, filename: str='{ts}-stdin.txt', out_dir: Path=OUTPUT_DIR) -> str:
     ts = str(datetime.now().timestamp()).split('.', 1)[0]
-    source_path = os.path.join(OUTPUT_DIR, SOURCES_DIR_NAME, filename.format(ts=ts))
+    source_path = str(out_dir / SOURCES_DIR_NAME / filename.format(ts=ts))
     atomic_write(source_path, raw_text)
     log_source_saved(source_file=source_path)
     return source_path
 
 
 @enforce_types
-def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: str=OUTPUT_DIR) -> str:
+def save_file_as_source(path: str, timeout: int=TIMEOUT, filename: str='{ts}-{basename}.txt', out_dir: Path=OUTPUT_DIR) -> str:
     """download a given url's content into output/sources/domain-<timestamp>.txt"""
     ts = str(datetime.now().timestamp()).split('.', 1)[0]
-    source_path = os.path.join(OUTPUT_DIR, SOURCES_DIR_NAME, filename.format(basename=basename(path), ts=ts))
+    source_path = str(OUTPUT_DIR / SOURCES_DIR_NAME / filename.format(basename=basename(path), ts=ts))
 
     if any(path.startswith(s) for s in ('http://', 'https://', 'ftp://')):
         # Source is a URL that needs to be downloaded
-        print('{}[*] [{}] Downloading {}{}'.format(
-            ANSI['green'],
-            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            path,
-            ANSI['reset'],
-        ))
+        print(f'    > Downloading {path} contents')
         timer = TimedProgress(timeout, prefix='      ')
         try:
             raw_source_text = download_url(path, timeout=timeout)
+            raw_source_text = htmldecode(raw_source_text)
             timer.end()
         except Exception as e:
             timer.end()

archivebox/parsers/generic_html.py

@@ -0,0 +1,53 @@
+__package__ = 'archivebox.parsers'
+
+
+import re
+
+from typing import IO, Iterable, Optional
+from datetime import datetime
+
+from ..index.schema import Link
+from ..util import (
+    htmldecode,
+    enforce_types,
+    URL_REGEX,
+)
+from html.parser import HTMLParser
+from urllib.parse import urljoin
+
+
+class HrefParser(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.urls = []
+
+    def handle_starttag(self, tag, attrs):
+        if tag == "a":
+            for attr, value in attrs:
+                if attr == "href":
+                    self.urls.append(value)
+
+
+@enforce_types
+def parse_generic_html_export(html_file: IO[str], root_url: Optional[str]=None, **_kwargs) -> Iterable[Link]:
+    """Parse Generic HTML for href tags and use only the url (support for title coming later)"""
+
+    html_file.seek(0)
+    for line in html_file:
+        parser = HrefParser()
+        # example line
+        # <li><a href="http://example.com/ time_added="1478739709" tags="tag1,tag2">example title</a></li>
+        parser.feed(line)
+        for url in parser.urls:
+            if root_url:
+                # resolve relative urls /home.html -> https://example.com/home.html
+                url = urljoin(root_url, url)
+            
+            for archivable_url in re.findall(URL_REGEX, url):
+                yield Link(
+                    url=htmldecode(archivable_url),
+                    timestamp=str(datetime.now().timestamp()),
+                    title=None,
+                    tags=None,
+                    sources=[html_file.name],
+                )

archivebox/parsers/generic_json.py

@@ -13,7 +13,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_generic_json_export(json_file: IO[str]) -> Iterable[Link]:
+def parse_generic_json_export(json_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)"""
 
     json_file.seek(0)

archivebox/parsers/generic_rss.py

@@ -12,7 +12,7 @@ from ..util import (
 )
 
 @enforce_types
-def parse_generic_rss_export(rss_file: IO[str]) -> Iterable[Link]:
+def parse_generic_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse RSS XML-format files into links"""
 
     rss_file.seek(0)

archivebox/parsers/generic_txt.py

@@ -16,7 +16,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_generic_txt_export(text_file: IO[str]) -> Iterable[Link]:
+def parse_generic_txt_export(text_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse raw links from each line in a text file"""
 
     text_file.seek(0)
@@ -25,14 +25,18 @@ def parse_generic_txt_export(text_file: IO[str]) -> Iterable[Link]:
             continue
 
         # if the line is a local file path that resolves, then we can archive it
-        if Path(line).exists():
-            yield Link(
-                url=line,
-                timestamp=str(datetime.now().timestamp()),
-                title=None,
-                tags=None,
-                sources=[text_file.name],
-            )
+        try:
+            if Path(line).exists():
+                yield Link(
+                    url=line,
+                    timestamp=str(datetime.now().timestamp()),
+                    title=None,
+                    tags=None,
+                    sources=[text_file.name],
+                )
+        except (OSError, PermissionError):
+            # nvm, not a valid path...
+            pass
 
         # otherwise look for anything that looks like a URL in the line
         for url in re.findall(URL_REGEX, line):
@@ -47,9 +51,9 @@ def parse_generic_txt_export(text_file: IO[str]) -> Iterable[Link]:
             # look inside the URL for any sub-urls, e.g. for archive.org links
             # https://web.archive.org/web/20200531203453/https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/
             # -> https://www.reddit.com/r/socialism/comments/gu24ke/nypd_officers_claim_they_are_protecting_the_rule/fsfq0sw/
-            for url in re.findall(URL_REGEX, line[1:]):
+            for sub_url in re.findall(URL_REGEX, line[1:]):
                 yield Link(
-                    url=htmldecode(url),
+                    url=htmldecode(sub_url),
                     timestamp=str(datetime.now().timestamp()),
                     title=None,
                     tags=None,

archivebox/parsers/medium_rss.py

@@ -14,7 +14,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_medium_rss_export(rss_file: IO[str]) -> Iterable[Link]:
+def parse_medium_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse Medium RSS feed files into links"""
 
     rss_file.seek(0)

archivebox/parsers/netscape_html.py

@@ -14,7 +14,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_netscape_html_export(html_file: IO[str]) -> Iterable[Link]:
+def parse_netscape_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse netscape-format bookmarks export files (produced by all browsers)"""
 
     html_file.seek(0)

archivebox/parsers/pinboard_rss.py

@@ -14,7 +14,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_pinboard_rss_export(rss_file: IO[str]) -> Iterable[Link]:
+def parse_pinboard_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse Pinboard RSS feed files into links"""
 
     rss_file.seek(0)

archivebox/parsers/pocket_api.py

@@ -0,0 +1,113 @@
+__package__ = 'archivebox.parsers'
+
+
+import re
+
+from typing import IO, Iterable, Optional
+from configparser import ConfigParser
+
+from pathlib import Path
+from ..vendor.pocket import Pocket
+
+from ..index.schema import Link
+from ..util import enforce_types
+from ..system import atomic_write
+from ..config import (
+    SOURCES_DIR,
+    POCKET_CONSUMER_KEY,
+    POCKET_ACCESS_TOKENS,
+)
+
+
+COUNT_PER_PAGE = 500
+API_DB_PATH = Path(SOURCES_DIR) / 'pocket_api.db'
+
+# search for broken protocols that sometimes come from the Pocket API
+_BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))')
+
+
+def get_pocket_articles(api: Pocket, since=None, page=0):
+    body, headers = api.get(
+        state='archive',
+        sort='oldest',
+        since=since,
+        count=COUNT_PER_PAGE,
+        offset=page * COUNT_PER_PAGE,
+    )
+
+    articles = body['list'].values() if isinstance(body['list'], dict) else body['list']
+    returned_count = len(articles)
+
+    yield from articles
+
+    if returned_count == COUNT_PER_PAGE:
+        yield from get_pocket_articles(api, since=since, page=page + 1)
+    else:
+        api.last_since = body['since']
+
+
+def link_from_article(article: dict, sources: list):
+    url: str = article['resolved_url'] or article['given_url']
+    broken_protocol = _BROKEN_PROTOCOL_RE.match(url)
+    if broken_protocol:
+        url = url.replace(f'{broken_protocol.group(1)}:/', f'{broken_protocol.group(1)}://')
+    title = article['resolved_title'] or article['given_title'] or url
+
+    return Link(
+        url=url,
+        timestamp=article['time_read'],
+        title=title,
+        tags=article.get('tags'),
+        sources=sources
+    )
+
+
+def write_since(username: str, since: str):
+    if not API_DB_PATH.exists():
+        atomic_write(API_DB_PATH, '')
+
+    since_file = ConfigParser()
+    since_file.optionxform = str
+    since_file.read(API_DB_PATH)
+
+    since_file[username] = {
+        'since': since
+    }
+
+    with open(API_DB_PATH, 'w+') as new:
+        since_file.write(new)
+
+
+def read_since(username: str) -> Optional[str]:
+    if not API_DB_PATH.exists():
+        atomic_write(API_DB_PATH, '')
+
+    config_file = ConfigParser()
+    config_file.optionxform = str
+    config_file.read(API_DB_PATH)
+
+    return config_file.get(username, 'since', fallback=None)
+
+
+@enforce_types
+def should_parse_as_pocket_api(text: str) -> bool:
+    return text.startswith('pocket://')
+
+
+@enforce_types
+def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]:
+    """Parse bookmarks from the Pocket API"""
+
+    input_buffer.seek(0)
+    pattern = re.compile(r"^pocket:\/\/(\w+)")
+    for line in input_buffer:
+        if should_parse_as_pocket_api(line):
+            
+            username = pattern.search(line).group(1)
+            api = Pocket(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKENS[username])
+            api.last_since = None
+    
+            for article in get_pocket_articles(api, since=read_since(username)):
+                yield link_from_article(article, sources=[line])
+    
+            write_since(username, api.last_since)

archivebox/parsers/pocket_html.py

@@ -14,7 +14,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_pocket_html_export(html_file: IO[str]) -> Iterable[Link]:
+def parse_pocket_html_export(html_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""
 
     html_file.seek(0)

archivebox/parsers/shaarli_rss.py

@@ -13,7 +13,7 @@ from ..util import (
 
 
 @enforce_types
-def parse_shaarli_rss_export(rss_file: IO[str]) -> Iterable[Link]:
+def parse_shaarli_rss_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
     """Parse Shaarli-specific RSS XML-format files into links"""
 
     rss_file.seek(0)

archivebox/parsers/wallabag_atom.py

@@ -0,0 +1,57 @@
+__package__ = 'archivebox.parsers'
+
+
+from typing import IO, Iterable
+from datetime import datetime
+
+from ..index.schema import Link
+from ..util import (
+    htmldecode,
+    enforce_types,
+    str_between,
+)
+
+
+@enforce_types
+def parse_wallabag_atom_export(rss_file: IO[str], **_kwargs) -> Iterable[Link]:
+    """Parse Wallabag Atom files into links"""
+
+    rss_file.seek(0)
+    entries = rss_file.read().split('<entry>')[1:]
+    for entry in entries:
+        # example entry:
+        # <entry>
+        #       <title><![CDATA[Orient Ray vs Mako: Is There Much Difference? - iknowwatches.com]]></title>
+        #       <link rel="alternate" type="text/html"
+        #              href="http://wallabag.drycat.fr/view/14041"/>
+        #       <link rel="via">https://iknowwatches.com/orient-ray-vs-mako/</link>
+        #       <id>wallabag:wallabag.drycat.fr:milosh:entry:14041</id>
+        #       <updated>2020-10-18T09:14:02+02:00</updated>
+        #       <published>2020-10-18T09:13:56+02:00</published>
+        #                   <category term="montres" label="montres" />
+        #                       <content type="html" xml:lang="en">
+        # </entry>
+
+        trailing_removed = entry.split('</entry>', 1)[0]
+        leading_removed = trailing_removed.strip()
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0]
+
+        title = str_between(get_row('title'), '<title><![CDATA[', ']]></title>').strip()
+        url = str_between(get_row('link rel="via"'), '<link rel="via">', '</link>')
+        ts_str = str_between(get_row('published'), '<published>', '</published>')
+        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
+        try:
+            tags = str_between(get_row('category'), 'label="', '" />')
+        except Exception:
+            tags = None
+
+        yield Link(
+            url=htmldecode(url),
+            timestamp=str(time.timestamp()),
+            title=htmldecode(title) or None,
+            tags=tags or '',
+            sources=[rss_file.name],
+        )

archivebox/search/__init__.py

@@ -0,0 +1,108 @@
+from typing import List, Union
+from pathlib import Path
+from importlib import import_module
+
+from django.db.models import QuerySet
+
+from archivebox.index.schema import Link
+from archivebox.util import enforce_types
+from archivebox.config import stderr, OUTPUT_DIR, USE_INDEXING_BACKEND, USE_SEARCHING_BACKEND, SEARCH_BACKEND_ENGINE
+
+from .utils import get_indexable_content, log_index_started
+
+def indexing_enabled():
+    return USE_INDEXING_BACKEND
+
+def search_backend_enabled():
+    return USE_SEARCHING_BACKEND
+
+def get_backend():
+    return f'search.backends.{SEARCH_BACKEND_ENGINE}'
+
+def import_backend():
+    backend_string = get_backend()
+    try:
+        backend = import_module(backend_string)
+    except Exception as err:
+        raise Exception("Could not load '%s' as a backend: %s" % (backend_string, err))
+    return backend
+
+@enforce_types
+def write_search_index(link: Link, texts: Union[List[str], None]=None, out_dir: Path=OUTPUT_DIR, skip_text_index: bool=False) -> None:
+    if not indexing_enabled():
+        return
+
+    if not skip_text_index and texts:
+        from core.models import Snapshot
+
+        snap = Snapshot.objects.filter(url=link.url).first()
+        backend = import_backend()
+        if snap:
+            try:
+                backend.index(snapshot_id=str(snap.id), texts=texts)
+            except Exception as err:
+                stderr()
+                stderr(
+                    f'[X] The search backend threw an exception={err}:',
+                color='red',
+                )
+
+@enforce_types
+def query_search_index(query: str, out_dir: Path=OUTPUT_DIR) -> QuerySet:
+    from core.models import Snapshot
+
+    if search_backend_enabled():
+        backend = import_backend()
+        try:
+            snapshot_ids = backend.search(query)
+        except Exception as err:
+            stderr()
+            stderr(
+                    f'[X] The search backend threw an exception={err}:',
+                color='red',
+                )
+            raise
+        else:
+            # TODO preserve ordering from backend
+            qsearch = Snapshot.objects.filter(pk__in=snapshot_ids)
+            return qsearch
+    
+    return Snapshot.objects.none()
+
+@enforce_types
+def flush_search_index(snapshots: QuerySet):
+    if not indexing_enabled() or not snapshots:
+        return
+    backend = import_backend()
+    snapshot_ids=(str(pk) for pk in snapshots.values_list('pk',flat=True))
+    try:
+        backend.flush(snapshot_ids)
+    except Exception as err:
+        stderr()
+        stderr(
+            f'[X] The search backend threw an exception={err}:',
+        color='red',
+        )
+
+@enforce_types
+def index_links(links: Union[List[Link],None], out_dir: Path=OUTPUT_DIR):
+    if not links:
+        return
+
+    from core.models import Snapshot, ArchiveResult
+
+    for link in links:
+        snap = Snapshot.objects.filter(url=link.url).first()
+        if snap: 
+            results = ArchiveResult.objects.indexable().filter(snapshot=snap)
+            log_index_started(link.url)
+            try:
+                texts = get_indexable_content(results)
+            except Exception as err:
+                stderr()
+                stderr(
+                    f'[X] An Exception ocurred reading the indexable content={err}:',
+                    color='red',
+                    ) 
+            else:
+                write_search_index(link, texts, out_dir=out_dir)

archivebox/search/backends/ripgrep.py

@@ -0,0 +1,45 @@
+import re
+from subprocess import run, PIPE
+from typing import List, Generator
+
+from archivebox.config import ARCHIVE_DIR, RIPGREP_VERSION
+from archivebox.util import enforce_types
+
+RG_IGNORE_EXTENSIONS = ('css','js','orig','svg')
+
+RG_ADD_TYPE = '--type-add'
+RG_IGNORE_ARGUMENTS = f"ignore:*.{{{','.join(RG_IGNORE_EXTENSIONS)}}}"
+RG_DEFAULT_ARGUMENTS = "-ilTignore" # Case insensitive(i), matching files results(l)
+RG_REGEX_ARGUMENT = '-e'
+
+TIMESTAMP_REGEX = r'\/([\d]+\.[\d]+)\/'
+
+ts_regex =  re.compile(TIMESTAMP_REGEX)
+
+@enforce_types
+def index(snapshot_id: str, texts: List[str]):
+    return
+
+@enforce_types
+def flush(snapshot_ids: Generator[str, None, None]):
+    return
+
+@enforce_types
+def search(text: str) -> List[str]:
+    if not RIPGREP_VERSION:
+        raise Exception("ripgrep binary not found, install ripgrep to use this search backend")
+
+    from core.models import Snapshot
+
+    rg_cmd = ['rg', RG_ADD_TYPE, RG_IGNORE_ARGUMENTS, RG_DEFAULT_ARGUMENTS, RG_REGEX_ARGUMENT, text, str(ARCHIVE_DIR)]
+    rg = run(rg_cmd, stdout=PIPE, stderr=PIPE, timeout=60)
+    file_paths = [p.decode() for p in rg.stdout.splitlines()]
+    timestamps = set()
+    for path in file_paths:
+        ts = ts_regex.findall(path)
+        if ts:
+            timestamps.add(ts[0])
+    
+    snap_ids = [str(id) for id in Snapshot.objects.filter(timestamp__in=timestamps).values_list('pk', flat=True)]
+
+    return snap_ids

archivebox/search/backends/sonic.py

@@ -0,0 +1,28 @@
+from typing import List, Generator
+
+from sonic import IngestClient, SearchClient
+
+from archivebox.util import enforce_types
+from archivebox.config import SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD, SONIC_BUCKET, SONIC_COLLECTION
+
+MAX_SONIC_TEXT_LENGTH = 2000
+
+@enforce_types
+def index(snapshot_id: str, texts: List[str]):
+    with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl:
+        for text in texts:
+            chunks = [text[i:i+MAX_SONIC_TEXT_LENGTH] for i in range(0, len(text), MAX_SONIC_TEXT_LENGTH)]
+            for chunk in chunks:
+                ingestcl.push(SONIC_COLLECTION, SONIC_BUCKET, snapshot_id, str(chunk))
+
+@enforce_types
+def search(text: str) -> List[str]:
+    with SearchClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as querycl:
+        snap_ids = querycl.query(SONIC_COLLECTION, SONIC_BUCKET, text)
+    return snap_ids
+
+@enforce_types
+def flush(snapshot_ids: Generator[str, None, None]):
+    with IngestClient(SEARCH_BACKEND_HOST_NAME, SEARCH_BACKEND_PORT, SEARCH_BACKEND_PASSWORD) as ingestcl:
+        for id in snapshot_ids:
+            ingestcl.flush_object(SONIC_COLLECTION, SONIC_BUCKET, str(id))

archivebox/search/utils.py

@@ -0,0 +1,45 @@
+from django.db.models import QuerySet
+
+from archivebox.util import enforce_types
+from archivebox.config import ANSI
+
+def log_index_started(url):
+    print('{green}[*] Indexing url: {} in the search index {reset}'.format(url, **ANSI))
+    print( )
+
+def get_file_result_content(res, extra_path, use_pwd=False):
+    if use_pwd: 
+        fpath = f'{res.pwd}/{res.output}'
+    else:
+        fpath = f'{res.output}'
+    
+    if extra_path:
+        fpath = f'{fpath}/{extra_path}'
+
+    with open(fpath, 'r') as file:
+        data = file.read()
+    if data:
+        return [data]
+    return []
+
+
+# This should be abstracted by a plugin interface for extractors
+@enforce_types
+def get_indexable_content(results: QuerySet):
+    if not results:
+        return []
+    # Only use the first method available
+    res, method = results.first(), results.first().extractor
+    if method not in ('readability', 'singlefile', 'dom', 'wget'):
+        return []
+    # This should come from a plugin interface
+
+    # TODO: banish this duplication and get these from the extractor file
+    if method == 'readability':
+        return get_file_result_content(res, 'content.txt')
+    elif method == 'singlefile':
+        return get_file_result_content(res,'',use_pwd=True)
+    elif method == 'dom':
+        return get_file_result_content(res,'',use_pwd=True)
+    elif method == 'wget':
+        return get_file_result_content(res,'',use_pwd=True)

archivebox/system.py

@@ -16,6 +16,7 @@ from .util import enforce_types, ExtendedEncoder
 from .config import OUTPUT_PERMISSIONS
 
 
+
 def run(*args, input=None, capture_output=True, text=False, **kwargs):
     """Patched of subprocess.run to fix blocking io making timeout=innefective"""
 
@@ -38,11 +39,17 @@ def atomic_write(path: Union[Path, str], contents: Union[dict, str, bytes], over
     mode = 'wb+' if isinstance(contents, bytes) else 'w'
 
     # print('\n> Atomic Write:', mode, path, len(contents), f'overwrite={overwrite}')
-    with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f:
-        if isinstance(contents, dict):
-            dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
-        elif isinstance(contents, (bytes, str)):
-            f.write(contents)
+    try:
+        with lib_atomic_write(path, mode=mode, overwrite=overwrite) as f:
+            if isinstance(contents, dict):
+                dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
+            elif isinstance(contents, (bytes, str)):
+                f.write(contents)
+    except OSError as e:
+        print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})")
+        print("    For data integrity, ArchiveBox requires a filesystem that supports atomic writes.")
+        print("    Filesystems and network drives that don't implement FSYNC are incompatible and require workarounds.")
+        raise SystemExit(1)
     os.chmod(path, int(OUTPUT_PERMISSIONS, base=8))
 
 @enforce_types
@@ -61,9 +68,9 @@ def chmod_file(path: str, cwd: str='.', permissions: str=OUTPUT_PERMISSIONS) ->
 
 
 @enforce_types
-def copy_and_overwrite(from_path: str, to_path: str):
+def copy_and_overwrite(from_path: Union[str, Path], to_path: Union[str, Path]):
     """copy a given file or directory to a given path, overwriting the destination"""
-    if os.path.isdir(from_path):
+    if Path(from_path).is_dir():
         shutil.rmtree(to_path, ignore_errors=True)
         shutil.copytree(from_path, to_path)
     else:
@@ -73,7 +80,7 @@ def copy_and_overwrite(from_path: str, to_path: str):
 
 
 @enforce_types
-def get_dir_size(path: str, recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]:
+def get_dir_size(path: Union[str, Path], recursive: bool=True, pattern: Optional[str]=None) -> Tuple[int, int, int]:
     """get the total disk size of a given directory, optionally summing up 
        recursively and limiting to a given filter list
     """
@@ -114,3 +121,43 @@ def dedupe_cron_jobs(cron: CronTab) -> CronTab:
         job.enable()
 
     return cron
+
+
+class suppress_output(object):
+    '''
+    A context manager for doing a "deep suppression" of stdout and stderr in 
+    Python, i.e. will suppress all print, even if the print originates in a 
+    compiled C/Fortran sub-function.
+       This will not suppress raised exceptions, since exceptions are printed
+    to stderr just before a script exits, and after the context manager has
+    exited (at least, I think that is why it lets exceptions through).      
+
+    with suppress_stdout_stderr():
+        rogue_function()
+    '''
+    def __init__(self, stdout=True, stderr=True):
+        # Open a pair of null files
+        # Save the actual stdout (1) and stderr (2) file descriptors.
+        self.stdout, self.stderr = stdout, stderr
+        if stdout:
+            self.null_stdout = os.open(os.devnull, os.O_RDWR)
+            self.real_stdout = os.dup(1)
+        if stderr:
+            self.null_stderr = os.open(os.devnull, os.O_RDWR)
+            self.real_stderr = os.dup(2)
+
+    def __enter__(self):
+        # Assign the null pointers to stdout and stderr.
+        if self.stdout:
+            os.dup2(self.null_stdout, 1)
+        if self.stderr:
+            os.dup2(self.null_stderr, 2)
+
+    def __exit__(self, *_):
+        # Re-assign the real stdout/stderr back to (1) and (2)
+        if self.stdout:
+            os.dup2(self.real_stdout, 1)
+            os.close(self.null_stdout)
+        if self.stderr:
+            os.dup2(self.real_stderr, 2)
+            os.close(self.null_stderr)

archivebox/templates/admin/base.html

@@ -88,8 +88,8 @@
         <div id="user-tools">
             <a href="{% url 'admin:Add' %}">Add ➕</a> /
             <a href="{% url 'Home' %}">Snapshots</a> /
+            <a href="/admin/core/tag/">Tags</a> /
             <a href="/admin/auth/user/">Users</a> /
-            <a href="{% url 'OldHome' %}">Old UI</a> /
             <a href="{% url 'Docs' %}">Docs</a>
              &nbsp; &nbsp;
             {% block welcome-msg %}
@@ -107,6 +107,9 @@
                 <a href="{% url 'admin:password_change' %}">{% trans 'Change password' %}</a> /
                 {% endif %}
                 <a href="{% url 'admin:logout' %}">{% trans 'Log out' %}</a>
+                |
+                <a> <span id="snapshotListView" style="cursor: pointer">☰</span> </a>       
+                <a> <span id="snapshotGridView"style="letter-spacing: -.4em; cursor: pointer;">⣿⣿</span></a> 
             {% endblock %}
         </div>
         {% endif %}
@@ -179,8 +182,63 @@
                 });
             }
         };
+        
+        function redirectWithQuery(uri){
+            uri_query = uri + document.location.search;
+            window.location = uri_query;
+
+        };
+
+        function selectSnapshotListView(){
+            localStorage.setItem('currentSnapshotView', 'List');
+            redirectWithQuery("{% url 'admin:core_snapshot_changelist' %}");
+        };
+
+        function selectSnapshotGridView(){
+            localStorage.setItem('currentSnapshotView', 'Grid');
+            redirectWithQuery("{% url 'admin:grid' %}");
+        };
+        
+        function setPreferredSnapshotView(view){
+            urlPath = window.location.pathname;
+
+            if((view==="Grid") && urlPath == "{% url 'admin:core_snapshot_changelist' %}"){  
+                selectSnapshotGridView();
+            }
+            
+            {% comment %} 
+                else if((view==="List") && urlPath == "{% url 'admin:grid' %}"){
+                selectSnapshotListView();
+            
+            } 
+            {% endcomment %}
+        };
+
+        function setupSnapshotViews() {
+            const preferredSnapshotView = localStorage.getItem('currentSnapshotView');
+            setPreferredSnapshotView(preferredSnapshotView);
+
+            $( document ).ready(function() {
+                
+                $("#snapshotListView").click(function() {
+                    selectSnapshotListView();
+                });  
+                $("#snapshotGridView").click(function() {
+                    selectSnapshotGridView();
+                });  
+
+                $('input:checkbox').change(function(){
+                    if($(this).is(':checked')) 
+                        $(this).parent().parent().parent().parent().addClass('selected-card'); 
+                    else 
+                        $(this).parent().parent().parent().parent().removeClass('selected-card')
+                });
+
+            });
+        };
         $(function () {
             fix_actions();
+            setupSnapshotViews();
         });
     })(django.jQuery);
 </script>

archivebox/templates/admin/private_index.html

@@ -0,0 +1,150 @@
+{% extends "base.html" %}
+{% load static %}
+
+{% block body %}
+    <div id="toolbar">
+        <form id="changelist-search" action="{% url 'public-index' %}" method="get">
+            <div>
+                <label for="searchbar"><img src="/static/admin/img/search.svg" alt="Search"></label>
+                <input type="text" size="40" name="q" value="" id="searchbar" autofocus placeholder="Title, URL, tags, timestamp, or content...".>
+                <input type="submit" value="Search" style="height: 36px; padding-top: 6px; margin: 8px"/>
+                <input type="button"
+                       value="♺"
+                       title="Refresh..."
+                       onclick="location.href='{% url 'public-index' %}'"
+                       style="background-color: rgba(121, 174, 200, 0.8); height: 30px; font-size: 0.8em; margin-top: 12px; padding-top: 6px; float:right">
+                </input>
+            </div>
+        </form>
+    </div>
+    <table id="table-bookmarks">
+        <thead>
+            <tr>
+                <th style="width: 100px;">Bookmarked</th>
+                <th style="width: 26vw;">Snapshot ({{object_list|length}})</th>
+                <th style="width: 140px">Files</th>
+                <th style="width: 16vw;whitespace:nowrap;overflow-x:hidden;">Original URL</th>
+            </tr>
+        </thead>
+            <tbody>
+                {% for link in object_list %}
+                    {% include 'main_index_row.html' with link=link  %}
+                {% endfor %}
+            </tbody>
+        </table>
+        <center>
+            <span class="step-links">
+                {% if page_obj.has_previous %}
+                    <a href="{% url 'public-index' %}?page=1">&laquo; first</a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.previous_page_number }}">previous</a>
+                {% endif %}
+        
+                <span class="current">
+                    Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}.
+                </span>
+        
+                {% if page_obj.has_next %}
+                    <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+                {% endif %}
+            </span>
+    
+            {% if page_obj.has_next %}
+                <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+            {% endif %}
+        </span>
+        <br>
+    </center>
+{% endblock %}
+{% extends "admin/base_site.html" %}
+{% load i18n admin_urls static admin_list %}
+{% load core_tags %}
+
+{% block extrastyle %}
+  {{ block.super }}
+  <link rel="stylesheet" type="text/css" href="{% static "admin/css/changelists.css" %}">
+  {% if cl.formset %}
+    <link rel="stylesheet" type="text/css" href="{% static "admin/css/forms.css" %}">
+  {% endif %}
+  {% if cl.formset or action_form %}
+    <script src="{% url 'admin:jsi18n' %}"></script>
+  {% endif %}
+  {{ media.css }}
+  {% if not actions_on_top and not actions_on_bottom %}
+    <style>
+      #changelist table thead th:first-child {width: inherit}
+    </style>
+  {% endif %}
+{% endblock %}
+
+{% block extrahead %}
+{{ block.super }}
+{{ media.js }}
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %}
+
+{% if not is_popup %}
+{% block breadcrumbs %}
+<div class="breadcrumbs">
+<a href="{% url 'admin:index' %}">{% translate 'Home' %}</a>
+&rsaquo; <a href="{% url 'admin:app_list' app_label=cl.opts.app_label %}">{{ cl.opts.app_config.verbose_name }}</a>
+&rsaquo; {{ cl.opts.verbose_name_plural|capfirst }}
+</div>
+{% endblock %}
+{% endif %}
+
+{% block coltype %}{% endblock %}
+
+{% block content %}
+  <div id="content-main">
+    {% block object-tools %}
+        <ul class="object-tools">
+          {% block object-tools-items %}
+            {% change_list_object_tools %}
+          {% endblock %}
+        </ul>
+    {% endblock %}
+    {% if cl.formset and cl.formset.errors %}
+        <p class="errornote">
+        {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %}
+        </p>
+        {{ cl.formset.non_form_errors }}
+    {% endif %}
+    <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">
+      <div class="changelist-form-container">
+        {% block search %}{% search_form cl %}{% endblock %}
+        {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %}
+
+        <form id="changelist-form" method="post"{% if cl.formset and cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %} novalidate>{% csrf_token %}
+        {% if cl.formset %}
+          <div>{{ cl.formset.management_form }}</div>
+        {% endif %}
+
+        {% block result_list %}
+          {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+          {% comment %} 
+             Table grid
+            {% result_list cl %} 
+          {% endcomment %}
+          {% snapshots_grid cl %}
+          {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+        {% endblock %}
+        {% block pagination %}{% pagination cl %}{% endblock %}
+        </form>
+      </div>
+      {% block filters %}
+        {% if cl.has_filters %}
+          <div id="changelist-filter">
+            <h2>{% translate 'Filter' %}</h2>
+            {% if cl.has_active_filters %}<h3 id="changelist-filter-clear">
+              <a href="{{ cl.clear_all_filters_qs }}">&#10006; {% translate "Clear all filters" %}</a>
+            </h3>{% endif %}
+            {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
+          </div>
+        {% endif %}
+      {% endblock %}
+    </div>
+  </div>
+{% endblock %}

archivebox/templates/admin/private_index_grid.html

@@ -0,0 +1,91 @@
+{% extends "admin/base_site.html" %}
+{% load i18n admin_urls static admin_list %}
+{% load core_tags %}
+
+{% block extrastyle %}
+  {{ block.super }}
+  <link rel="stylesheet" type="text/css" href="{% static "admin/css/changelists.css" %}">
+  {% if cl.formset %}
+    <link rel="stylesheet" type="text/css" href="{% static "admin/css/forms.css" %}">
+  {% endif %}
+  {% if cl.formset or action_form %}
+    <script src="{% url 'admin:jsi18n' %}"></script>
+  {% endif %}
+  {{ media.css }}
+  {% if not actions_on_top and not actions_on_bottom %}
+    <style>
+      #changelist table thead th:first-child {width: inherit}
+    </style>
+  {% endif %}
+{% endblock %}
+
+{% block extrahead %}
+{{ block.super }}
+{{ media.js }}
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-list{% endblock %}
+
+{% if not is_popup %}
+{% block breadcrumbs %}
+<div class="breadcrumbs">
+<a href="{% url 'admin:index' %}">{% translate 'Home' %}</a>
+&rsaquo; <a href="{% url 'admin:app_list' app_label=cl.opts.app_label %}">{{ cl.opts.app_config.verbose_name }}</a>
+&rsaquo; {{ cl.opts.verbose_name_plural|capfirst }}
+</div>
+{% endblock %}
+{% endif %}
+
+{% block coltype %}{% endblock %}
+
+{% block content %}
+  <div id="content-main">
+    {% block object-tools %}
+        <ul class="object-tools">
+          {% block object-tools-items %}
+            {% change_list_object_tools %}
+          {% endblock %}
+        </ul>
+    {% endblock %}
+    {% if cl.formset and cl.formset.errors %}
+        <p class="errornote">
+        {% if cl.formset.total_error_count == 1 %}{% translate "Please correct the error below." %}{% else %}{% translate "Please correct the errors below." %}{% endif %}
+        </p>
+        {{ cl.formset.non_form_errors }}
+    {% endif %}
+    <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">
+      <div class="changelist-form-container">
+        {% block search %}{% search_form cl %}{% endblock %}
+        {% block date_hierarchy %}{% if cl.date_hierarchy %}{% date_hierarchy cl %}{% endif %}{% endblock %}
+
+        <form id="changelist-form" method="post"{% if cl.formset and cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %} novalidate>{% csrf_token %}
+        {% if cl.formset %}
+          <div>{{ cl.formset.management_form }}</div>
+        {% endif %}
+
+        {% block result_list %}
+          {% if action_form and actions_on_top and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+          {% comment %} 
+             Table grid
+            {% result_list cl %} 
+          {% endcomment %}
+          {% snapshots_grid cl %}
+          {% if action_form and actions_on_bottom and cl.show_admin_actions %}{% admin_actions %}{% endif %}
+        {% endblock %}
+        {% block pagination %}{% pagination cl %}{% endblock %}
+        </form>
+      </div>
+      {% block filters %}
+        {% if cl.has_filters %}
+          <div id="changelist-filter">
+            <h2>{% translate 'Filter' %}</h2>
+            {% if cl.has_active_filters %}<h3 id="changelist-filter-clear">
+              <a href="{{ cl.clear_all_filters_qs }}">&#10006; {% translate "Clear all filters" %}</a>
+            </h3>{% endif %}
+            {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
+          </div>
+        {% endif %}
+      {% endblock %}
+    </div>
+  </div>
+{% endblock %}

archivebox/templates/admin/snapshots_grid.html

@@ -0,0 +1,162 @@
+{% load i18n admin_urls static admin_list %}
+{% load core_tags %}
+
+{% block extrastyle %}
+<style>
+* {
+  -webkit-box-sizing: border-box;
+    -moz-box-sizing: border-box;
+    box-sizing: border-box;
+}
+
+a {
+  text-decoration: none;
+  color: orange;
+}
+h2 {
+  color: #000;
+  margin: 2rem 0 .5rem;
+  font-size: 1.25rem;
+  font-weight: 400;
+  {% comment %} text-transform: uppercase; {% endcomment %}
+}
+
+card.img {
+  display: block;
+    border: 0;
+    width: 100%;
+    height: auto;
+}
+
+/***************************           Cards            *******************************/
+
+.cards {
+  display: grid;
+  grid-template-columns: repeat(auto-fill, minmax(250px, 1fr)); /* see notes below */
+  grid-auto-rows: minmax(200px, auto);
+  grid-gap: 1rem;
+}
+
+.card {
+  /*height: 200px;*/
+  /*background: red;*/
+  border: 2px solid #e7e7e7;
+  border-radius: 4px;
+  -webkit-box-shadow: 0 2px 2px rgba(0, 0, 0, 0.15);
+  box-shadow: 0 2px 2px rgba(0, 0, 0, 0.15);
+  display: flex;
+  /* -webkit-box-orient: vertical; */
+  /* -webkit-box-direction: normal; */
+  -ms-flex-direction: column;
+  flex-direction: column;
+  position: relative;
+  color: #5d5e5e;
+} /* li item */
+
+.thumbnail img {
+    height: 100%;
+    box-sizing: border-box;
+    max-width: 100%;
+    max-height: 100%;
+    width: 100%;
+}
+
+.card-content {
+    font-size: .75rem;
+    padding: .5rem;
+    display: flex;
+    -webkit-box-orient: vertical;
+    -webkit-box-direction: normal;
+    -ms-flex-direction: column;
+    flex-direction: column;
+    -webkit-box-flex: 1;
+    -ms-flex: 1;
+    flex: 1;
+    
+}
+
+.card-content h4{
+vertical-align:bottom;
+margin: 1.2em 0 0em 0;
+}
+
+.category {
+  font-size: .75rem;
+  text-transform: uppercase;
+}
+.category {
+    position: absolute;
+    top: 5%;
+    right: 0;
+    color: #fff;
+    background: #e74c3c;
+    padding: 10px 15px;
+    font-size: 10px;
+    font-weight: 600;
+    text-transform: uppercase;
+}
+
+.category__01 {
+  background-color: #50c6db;
+  
+}
+
+.tags{
+  opacity: 0.8;
+}
+
+footer {
+  border-top: 2px solid #e7e7e7;
+  {% comment %} margin: .5rem 0 0; {% endcomment %}
+  {% comment %} min-height: 30px; {% endcomment %}
+  font-size: .5rem;
+}
+.post-meta {
+  padding: .3rem;
+}
+
+.comments {
+  margin-left: .5rem;
+}
+
+.selected-card{
+  border: 5px solid #ffaa31;
+}
+
+</style>
+
+{% endblock %}
+
+{% block content %}
+    <section class="cards">
+    {% for obj in results %}
+      <article class="card">
+          <picture class="thumbnail">
+            <a href="/{{obj.archive_path}}/index.html">
+              <img class="category__01" src="{% snapshot_image obj%}" alt="" /> 
+            </a>
+          </picture>
+          <div class="card-content">
+            {% if obj.tags_str %}
+            <p class="category category__01 tags">{{obj.tags_str}}</p>
+            {% endif %}
+            {% if obj.title  %}
+            <a href="{% url 'admin:core_snapshot_change' obj.id %}">
+              <h4>{{obj.title|truncatechars:55 }}</h4>
+            </a>
+            {% endif %}
+            {% comment %} <p> TEXT If needed.</p> {% endcomment %}
+        </div><!-- .card-content -->
+        <footer>
+            <div class="post-meta">
+                <span style="float:right;"><input type="checkbox" name="_selected_action" value="{{obj.pk}}" class="action-select"></span>
+                <span class="timestamp">&#128337 {{obj.added}}</span>
+                <span class="comments">📖{{obj.num_outputs}}</span>
+                <span>🗄️{{ obj.archive_size | file_size }}</span>
+            </div>
+        </footer> 
+      </article>
+    {% endfor %}
+    </section>
+
+{% endblock %}

archivebox/templates/core/add.html

@@ -1,4 +1,6 @@
-{% extends "admin/index.html" %}
+{% extends "core/base.html" %}
+
+{% load static %}
 {% load i18n %}
 
 {% block breadcrumbs %}
@@ -8,48 +10,11 @@
     </div>
 {% endblock %}
 
-{% block content %}
-    <style>
-        .dashboard #content {
-            width: 100%;
-            margin-right: 0px;
-            margin-left: 0px;
-        }
-        #submit {
-            border: 1px solid rgba(0,0,0,0.2);
-            padding: 10px;
-            border-radius: 4px;
-            background-color: #f5dd5d;
-            color: #333;
-            font-size: 18px;
-            font-weight: 800;
-        }
-        #add-form button[role=submit]:hover {
-            background-color: #e5cd4d;
-        }
-        #add-form label {
-            display: block;
-            font-size: 16px;
-        }
-        #add-form textarea {
-            width: 100%;
-            min-height: 300px;
-        }
-        #delay-warning div {
-            border: 1px solid red;
-            border-radius: 4px;
-            margin: 10px;
-            padding: 10px;
-            font-size: 15px;
-            background-color: #F5DD5D;
-        }
-        #stdout {
-            background-color: #ded;
-            padding: 10px 10px;
-            border-radius: 4px;
-            white-space: normal;
-        }
-    </style>
+{% block extra_head %}
+    <link rel="stylesheet" href="{% static 'add.css' %}" />
+{% endblock %}
+
+{% block body %}
     <div style="max-width: 550px; margin: auto; float: none">
         <br/><br/>
         {% if stdout %}
@@ -63,7 +28,7 @@
                 <a href="/add" id="submit">&nbsp; Add more URLs ➕</a>
             </center>
         {% else %}
-            <form id="add-form" action="?" method="POST" class="p-form">{% csrf_token %}
+            <form id="add-form" method="POST" class="p-form">{% csrf_token %}
                 <h1>Add new URLs to your archive</h1>
                 <br/>
                 {{ form.as_p }}
@@ -84,6 +49,12 @@
                     <small>(it's safe to leave this page, adding will continue in the background)</small>
                 </div>
             </center>
+            {% if absolute_add_path %}
+            <center id="bookmarklet">
+              <p>Bookmark this link to quickly add to your archive:
+                <a href="javascript:void(window.open('{{ absolute_add_path }}?url='+document.location.href));">Add to ArchiveBox</a></p>
+            </center>
+            {% endif %}
             <script>
                 document.getElementById('add-form').addEventListener('submit', function(event) {
                     setTimeout(function() {
@@ -97,4 +68,6 @@
     </div>
 {% endblock %}
 
+{% block footer %}{% endblock %}
+
 {% block sidebar %}{% endblock %}

archivebox/templates/core/base.html

@@ -0,0 +1,84 @@
+{% load admin_urls %}
+{% load static %}
+
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <title>Archived Sites</title>
+    <meta charset="utf-8" name="viewport" content="width=device-width, initial-scale=1">
+
+    <link rel="stylesheet" href="{% static 'admin/css/base.css' %}">
+    <link rel="stylesheet" type="text/css" href="{% static 'admin.css' %}">
+    <link rel="stylesheet" href="{% static 'bootstrap.min.css' %}">
+    <link rel="stylesheet" href="{% static 'jquery.dataTables.min.css' %}" />
+            {% block extra_head %}
+            {% endblock %}
+    <script src="{% static 'jquery.min.js' %}"></script>
+    <script src="{% static 'jquery.dataTables.min.js' %}"></script>
+    <script>
+        document.addEventListener('error', function (e) {
+            e.target.style.opacity = 0;
+        }, true)
+        jQuery(document).ready(function () {
+            jQuery('#table-bookmarks').DataTable({
+                searching: false,
+                paging: false,
+                stateSave: true, // save state (filtered input, number of entries shown, etc) in localStorage
+                dom: '<lf<t>ip>', // how to show the table and its helpers (filter, etc) in the DOM
+                    order: [[0, 'desc']],
+                    iDisplayLength: 100,
+                });
+            });
+        </script>
+        <base href="{% url 'Home' %}">
+    </head>
+    <body>
+        <div id="container">
+            <div id="header">
+                <div id="branding">
+                    <h1 id="site-name">
+                        <a href="{% url 'public-index' %}" class="header-archivebox" title="Last updated: {{updated}}">
+                            <img src="{% static 'archive.png' %}" alt="Logo" style="height: 30px"/>
+                            ArchiveBox
+                        </a>
+                    </h1>
+                </div>
+                <div id="user-tools">
+                    <a href="/add/">➕&nbsp; Add</a> &nbsp;/&nbsp;
+                    <a href="/">Snapshots</a> &nbsp;/&nbsp;
+                    <a href="/admin/">Admin</a> &nbsp;/&nbsp;
+                    <a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
+                    {% if user.is_authenticated %}
+                        &nbsp; &nbsp;
+                        User
+                        <strong>{% firstof user.get_short_name user.get_username %}</strong> &nbsp; &nbsp;
+                        {% if user.has_usable_password %}
+                            <a href="{% url 'admin:password_change' %}">Change password</a> /
+                        {% endif %}
+                        <a href="{% url 'admin:logout' %}">Log out</a>
+                    {% endif %}
+                </div>
+            </div>
+            <div id="content" class="flex">
+                {% block body %}
+                {% endblock %}
+            </div>
+            {% block footer %}
+                <footer>
+                    <br />
+                    <center>
+                        <small>
+                            Archive created using <a href="https://github.com/ArchiveBox/ArchiveBox" title="Github">ArchiveBox</a> version
+                            <a href="https://github.com/ArchiveBox/ArchiveBox/releases" title="Releases">v{{VERSION}}</a>.
+                            <br/><br/>
+                            {{FOOTER_INFO}}
+                        </small>
+                    </center>
+                    <br />
+                </footer>
+            {% endblock %}
+        </div>
+    </body>
+    
+    </html>

archivebox/templates/core/index_row.html

@@ -0,0 +1,37 @@
+{% load static %}
+
+<tr>
+    <td title="{{link.timestamp}}"> {% if link.bookmarked_date  %} {{ link.bookmarked_date }} {% else %} {{ link.added }} {% endif %} </td>
+    <td class="title-col" style="opacity: {% if link.title %}1{% else %}0.3{% endif %}">
+        {% if link.is_archived %}
+            <a href="archive/{{link.timestamp}}/index.html"><img src="archive/{{link.timestamp}}/favicon.ico" onerror="this.style.display='none'" class="link-favicon" decoding="async"></a>
+        {% else %}
+            <a href="archive/{{link.timestamp}}/index.html"><img src="{% static 'spinner.gif' %}" onerror="this.style.display='none'" class="link-favicon" decoding="async" style="height: 15px"></a>
+        {% endif %}
+
+        <a href="archive/{{link.timestamp}}/index.html" title="{{link.title|default:'Not yet archived...'}}">
+            <span data-title-for="{{link.url}}" data-archived="{{link.is_archived}}">{{link.title|default:'Loading...'}}</span>
+            {% if link.tags_str %}
+                <span class="tags" style="float: right; border-radius: 5px; background-color: #bfdfff; padding: 2px 5px; margin-left: 4px; margin-top: 1px;">
+                    {% if link.tags_str != None %}
+                        {{link.tags_str|default:''}}
+                    {% else %}
+                        {{ link.tags|default:'' }}
+                    {% endif %}
+                </span>
+            {% endif %}
+        </a>
+    </td>
+    <td>
+        <span data-number-for="{{link.url}}" title="Fetching any missing files...">
+            {% if link.icons  %}
+                {{link.icons}} <small style="float:right; opacity: 0.5">{{link.num_outputs}}</small>
+            {% else %}
+                <a href="archive/{{link.timestamp}}/index.html">📄 
+                    {{link.num_outputs}} <img src="{% static 'spinner.gif' %}" onerror="this.style.display='none'" class="files-spinner" decoding="async" style="height: 15px"/>
+                </a>
+            {% endif %}
+        </span>
+    </td>
+   <td style="text-align:left"><a href="{{link.url}}">{{link.url}}</a></td>
+</tr>

archivebox/templates/core/minimal_index.html

@@ -0,0 +1,24 @@
+<!DOCTYPE html>
+<html lang="en">
+    <head>
+        <title>Archived Sites</title>
+        <meta charset="utf-8" name="viewport" content="width=device-width, initial-scale=1">
+    </head>
+    <body data-status="{{status}}">
+        <table id="table-bookmarks">
+            <thead>
+                <tr class="thead-tr">
+                    <th style="width: 100px;">Bookmarked</th>
+                    <th style="width: 26vw;">Saved Link ({{num_links}})</th>
+                    <th style="width: 50px">Files</th>
+                    <th style="width: 16vw;whitespace:nowrap;overflow-x:hidden;">Original URL</th>
+                </tr>
+            </thead>
+            <tbody>
+                {% for link in links %}
+                    {% include "index_row.html" with link=link %} 
+                {% endfor %}
+            </tbody>
+        </table>
+    </body>
+</html>

archivebox/templates/core/public_index.html

@@ -0,0 +1,59 @@
+{% extends "base.html" %}
+{% load static %}
+
+{% block body %}
+    <div id="toolbar">
+        <form id="changelist-search" action="{% url 'public-index' %}" method="get">
+            <div>
+                <label for="searchbar"><img src="/static/admin/img/search.svg" alt="Search"></label>
+                <input type="text" size="40" name="q" value="" id="searchbar" autofocus placeholder="Title, URL, tags, timestamp, or content...".>
+                <input type="submit" value="Search" style="height: 36px; padding-top: 6px; margin: 8px"/>
+                <input type="button"
+                       value="♺"
+                       title="Refresh..."
+                       onclick="location.href='{% url 'public-index' %}'"
+                       style="background-color: rgba(121, 174, 200, 0.8); height: 30px; font-size: 0.8em; margin-top: 12px; padding-top: 6px; float:right">
+                </input>
+            </div>
+        </form>
+    </div>
+    <table id="table-bookmarks">
+        <thead>
+            <tr>
+                <th style="width: 100px;">Bookmarked</th>
+                <th style="width: 26vw;">Snapshot ({{object_list|length}})</th>
+                <th style="width: 140px">Files</th>
+                <th style="width: 16vw;whitespace:nowrap;overflow-x:hidden;">Original URL</th>
+            </tr>
+        </thead>
+            <tbody>
+                {% for link in object_list %}
+                    {% include 'index_row.html' with link=link  %}
+                {% endfor %}
+            </tbody>
+        </table>
+        <center>
+            <span class="step-links">
+                {% if page_obj.has_previous %}
+                    <a href="{% url 'public-index' %}?page=1">&laquo; first</a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.previous_page_number }}">previous</a>
+                {% endif %}
+        
+                <span class="current">
+                    Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}.
+                </span>
+        
+                {% if page_obj.has_next %}
+                    <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                    <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+                {% endif %}
+            </span>
+    
+            {% if page_obj.has_next %}
+                <a href="{% url 'public-index' %}?page={{ page_obj.next_page_number }}">next </a>
+                <a href="{% url 'public-index' %}?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
+            {% endif %}
+        </span>
+        <br>
+    </center>
+{% endblock %}

archivebox/templates/core/snapshot.html

@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 <html lang="en">
     <head>
-        <title>$title</title>
+        <title>{{title}}</title>
         <meta charset="utf-8" name="viewport" content="width=device-width, initial-scale=1">
         <style>
             html, body {
@@ -33,7 +33,7 @@
             }
             .nav > div {
                 min-height: 30px;
-                margin: 8px 0px;
+                line-height: 1.3;
             }
             .header-top a {
                 text-decoration: none;
@@ -68,6 +68,11 @@
                 vertical-align: -2px;
                 margin-right: 4px;
             }
+            .header-toggle {
+                line-height: 14px;
+                font-size: 70px;
+                vertical-align: -8px;
+            }
             
             .info-row {
                 margin-top: 2px;
@@ -76,23 +81,30 @@
             .info-row .alert {
                 margin-bottom: 0px;
             }
-            .card {
+            .header-bottom-frames .card {
                 overflow: hidden;
                 box-shadow: 2px 3px 14px 0px rgba(0,0,0,0.02);
+                margin-top: 10px;
+                border: 1px solid rgba(0,0,0,3);
+                border-radius: 14px;
+                background-color: black;
             }
             .card h4 {
                 font-size: 1.4vw;
             }
             .card-body {
-                font-size: 1vw;
-                padding-top: 1.2vw;
-                padding-left: 1vw;
-                padding-right: 1vw;
-                padding-bottom: 1vw;
+                font-size: 15px;
+                padding: 13px 10px;
+                padding-bottom: 6px;
+                /* padding-left: 3px; */
+                /* padding-right: 3px; */
+                /* padding-bottom: 3px; */
                 line-height: 1.1;
                 word-wrap: break-word;
                 max-height: 102px;
                 overflow: hidden;
+                background-color: #1a1a1a;
+                color: #d3d3d3;
             }
             .card-title {
                 margin-bottom: 4px;
@@ -125,7 +137,7 @@
                 border-top: 3px solid #aa1e55;
             }
             .card.selected-card {
-                border: 2px solid orange;
+                border: 1px solid orange;
                 box-shadow: 0px -6px 13px 1px rgba(0,0,0,0.05);
             }
             .iframe-large {
@@ -173,12 +185,13 @@
                 width: 98%;
                 border: 1px solid rgba(0,0,0,0.2);
                 box-shadow: 4px 4px 4px rgba(0,0,0,0.2);
-                margin-top: 5px;
+                margin-top: 0px;
             }
             .header-bottom-info {
                 color: #6f6f6f;
-                padding-top: 8px;
-                padding-bottom: 13px;
+                padding-top: 0px;
+                padding-bottom: 0px;
+                margin: 0px -15px;
             }
 
             .header-bottom-info > div {
@@ -202,12 +215,30 @@
                 margin-top: 5px;
             }
             .header-bottom-frames .card-title {
-                padding-bottom: 0px;
-                font-size: 1.2vw;
+                width: 100%;
+                text-align: center;
+                font-size: 18px;
                 margin-bottom: 5px;
+                display: inline-block;
+                color: #d3d3d3;
+                font-weight: 200;
+                vertical-align: 0px;
+                margin-top: -6px;
             }
             .header-bottom-frames .card-text {
+                width: 100%;
+                text-align: center;
                 font-size: 0.9em;
+                display: inline-block;
+                position: relative;
+                top: -11px;
+            }
+            .card-text code {
+                padding: .2rem .4rem;
+                font-size: 90%;
+                color: #bd4147;
+                background-color: #101010;
+                border-radius: .25rem;
             }
 
             @media(max-width: 1092px) {
@@ -246,15 +277,15 @@
                         </a>
                     </div>
                     <div class="col-lg-8">
-                        <img src="favicon.ico" alt="Favicon">
+                        <img src="favicon.ico" onerror="this.style.opacity=0" alt="Favicon">
                         &nbsp;&nbsp;
-                        $title
+                        {{title}}
                         &nbsp;&nbsp;
                         <a href="#" class="header-toggle">▾</a>
                         <br/>
                         <small>
-                            <a href="$url" class="header-url" title="$url">
-                                $url_str
+                            <a href="{{url}}" class="header-url" title="{{url}}">
+                                {{url_str}}
                             </a>
                         </small>
                     </div>
@@ -265,37 +296,41 @@
                     <div class="col-lg-4">
                         <div title="Date bookmarked or imported" class="info-chunk">
                             <h5>Added</h5>
-                            $bookmarked_date
+                            {{bookmarked_date}}
                         </div>
                         <div title="Date first archived" class="info-chunk">
                             <h5>First Archived</h5>
-                            $oldest_archive_date
+                            {{oldest_archive_date}}
                         </div>
                         <div title="Date last checked" class="info-chunk">
                             <h5>Last Checked</h5>
-                            $updated_date
+                            {{updated_date}}
                         </div>
                     </div>
                     <div class="col-lg-4">
                         <div class="info-chunk">
                             <h5>Type</h5>
-                            <div class="badge badge-default">$extension</div>
+                            <div class="badge badge-default">{{extension}}</div>
                         </div>
                         <div class="info-chunk">
                             <h5>Tags</h5>
-                            <div class="badge badge-warning">$tags</div> 
+                            <div class="badge badge-warning">{{tags}}</div> 
                         </div>
                         <div class="info-chunk">
                             <h5>Status</h5>
-                            <div class="badge badge-$status_color">$status</div>
+                            <div class="badge badge-{{status_color}}">{{status}}</div>
                         </div>
                         <div class="info-chunk">
                             <h5>Saved</h5>
-                            ✅  $num_outputs
+                            ✅  {{num_outputs}}
                         </div>
                         <div class="info-chunk">
                             <h5>Errors</h5>
-                            ❌  $num_failures
+                            ❌  {{num_failures}}
+                        </div>
+                        <div class="info-chunk">
+                            <h5>Size</h5>
+                            {{size}}
                         </div>
                     </div>
                     <div class="col-lg-4">
@@ -313,80 +348,143 @@
                 <div class="row header-bottom-frames">
                     <div class="col-lg-2">
                         <div class="card selected-card">
-                          <iframe class="card-img-top" src="$archive_url" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
-                          <div class="card-body">
-                            <a href="$archive_url" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$archive_url" target="preview"><h4 class="card-title">Local Archive</h4></a>
-                            <p class="card-text">archive/$domain</p>
+                            <iframe class="card-img-top" src="{{singlefile_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{singlefile_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./singlefile.html</code></p>
+                                </a>
+                                <a href="{{singlefile_path}}" target="preview"><h4 class="card-title">Chrome &gt; SingleFile</h4></a>
                           </div>
                         </div>
                     </div>
                     <div class="col-lg-2">
                         <div class="card">
-                          <iframe class="card-img-top" src="$dom_path" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
-                          <div class="card-body">
-                            <a href="$dom_path" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$dom_path" target="preview"><h4 class="card-title">HTML</h4></a>
-                            <p class="card-text">archive/output.html</p>
+                            <iframe class="card-img-top pdf-frame" src="{{pdf_path}}" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{pdf_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./output.pdf</code></p>
+                                </a>
+                                <a href="{{pdf_path}}" target="preview" id="pdf-btn"><h4 class="card-title">Chrome &gt; PDF</h4></a>
                           </div>
                         </div>
                     </div>
                     <div class="col-lg-2">
                         <div class="card">
-                          <iframe class="card-img-top pdf-frame" src="$pdf_path" scrolling="no"></iframe>
-                          <div class="card-body">
-                            <a href="$pdf_path" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$pdf_path" target="preview" id="pdf-btn"><h4 class="card-title">PDF</h4></a>
-                            <p class="card-text">archive/output.pdf</p>
+                            <img class="card-img-top" src="{{screenshot_path}}" onerror="this.style.opacity=0.2"/>
+                            <div class="card-body">
+                                <a href="{{screenshot_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./screenshot.png</code></p>
+                                </a>
+                                <a href="{{screenshot_path}}" target="preview"><h4 class="card-title">Chrome &gt; Screenshot</h4></a>
                           </div>
                         </div>
                     </div>
                     <div class="col-lg-2">
                         <div class="card">
-                          <img class="card-img-top screenshot" src="$screenshot_path"></iframe>
+                          <iframe class="card-img-top" src="{{archive_url}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
                           <div class="card-body">
-                            <a href="$screenshot_path" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$screenshot_path" target="preview"><h4 class="card-title">Screenshot</h4></a>
-                            <p class="card-text">archive/screenshot.png</p>
+                                <a href="{{archive_url}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./{{domain}}</code></p>
+                                </a>
+                                <a href="{{archive_url}}" target="preview"><h4 class="card-title">Wget &gt; HTML</h4></a>
+                            </div>
+                        </div>
+                    </div>
+                    {% if SAVE_ARCHIVE_DOT_ORG %}
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{archive_org_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{archive_org_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>🌐 web.archive.org/web/...</code></p>
+                                </a>
+                                <a href="{{archive_org_path}}" target="preview"><h4 class="card-title">Archive.Org</h4></a>
+                          </div>
+                        </div>
+                    </div>
+                    {% endif %}
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{url}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{url}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>🌐 {{domain}}</code></p>
+                                </a>
+                                <a href="{{url}}" target="preview"><h4 class="card-title">Original</h4></a>
                           </div>
                         </div>
                     </div>
                     <div class="col-lg-2">
                         <div class="card">
-                          <iframe class="card-img-top" src="$archive_org_path" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
-                          <div class="card-body">
-                            <a href="$archive_org_path" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$archive_org_path" target="preview"><h4 class="card-title">Archive.Org</h4></a>
-                            <p class="card-text">web.archive.org/web/...</p>
+                            <iframe class="card-img-top" src="{{headers_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{headers_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./headers.json</code></p>
+                                </a>
+                                <a href="{{headers_path}}" target="preview"><h4 class="card-title">Headers</h4></a>
                           </div>
                         </div>
                     </div>
                     <div class="col-lg-2">
                         <div class="card">
-                          <iframe class="card-img-top" src="$url" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
-                          <div class="card-body">
-                            <a href="$url" style="float:right" title="Open in new tab..." target="_blank" rel="noopener">
-                                <img src="../../static/external.png" class="external"/>
-                            </a>
-                            <a href="$url" target="preview"><h4 class="card-title">Original</h4></a>
-                            <p class="card-text">$domain</p>
+                            <iframe class="card-img-top" src="{{dom_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{dom_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./output.html</code></p>
+                                </a>
+                                <a href="{{dom_path}}" target="preview"><h4 class="card-title">Chrome &gt; HTML</h4></a>
+                          </div>
+                        </div>
+                    </div>
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{readability_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{readability_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./readability/content.html</code></p>
+                                </a>
+                                <a href="{{readability_path}}" target="preview"><h4 class="card-title">Readability</h4></a>
+                          </div>
+                        </div>
+                    </div>
+                    <br/>
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{mercury_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{mercury_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./mercury/content.html</code></p>
+                                </a>
+                                <a href="{{mercury_path}}" target="preview"><h4 class="card-title">Mercury</h4></a>
+                          </div>
+                        </div>
+                    </div>
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{media_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{media_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./media/*.mp4</code></p>
+                                </a>
+                                <a href="{{media_path}}" target="preview"><h4 class="card-title">Media</h4></a>
+                          </div>
+                        </div>
+                    </div>
+                    <div class="col-lg-2">
+                        <div class="card">
+                            <iframe class="card-img-top" src="{{git_path}}" sandbox="allow-same-origin allow-scripts allow-forms" scrolling="no"></iframe>
+                            <div class="card-body">
+                                <a href="{{git_path}}" title="Open in new tab..." target="_blank" rel="noopener">
+                                    <p class="card-text"><code>./git/*.git</code></p>
+                                </a>
+                                <a href="{{git_path}}" target="preview"><h4 class="card-title">Git</h4></a>
                           </div>
                         </div>
                     </div>
                 </div>
             </div>
         </header>
-        <iframe sandbox="allow-same-origin allow-scripts allow-forms" class="full-page-iframe" src="$archive_url" name="preview"></iframe>
+        <iframe sandbox="allow-same-origin allow-scripts allow-forms" class="full-page-iframe" src="{{singlefile_path}}" name="preview"></iframe>
     
         <script
               src="https://code.jquery.com/jquery-3.2.1.slim.min.js"