Update README.md

when checking link invariants, check for regex match as well
2019-02-27 13:39:35 -05:00 · 2019-02-27 13:37:57 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00
64 changed files with 1591 additions and 983 deletions
@@ -1,6 +1,6 @@
-archiver/output
-archiver/tests
+output
 __pycache__
 .DS_Store
 venv
 .venv
+data
@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at archivebox-coc@sweeting.me. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq
@@ -0,0 +1 @@
+Make sure check in with me first or confirm your desired features line up with our roadmap: https://github.com/pirate/ArchiveBox#roadmap
@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A description of what the bug is, what you expected to happen, 
+and any relevant context about issue.
+
+**Steps to reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Screenshots or log output**
+If applicable, use screenshots or copy/pasted terminal output to help explain your problem.
+
+**Software versions (please complete the following information):**
+ - ArchiveBox version: <commit_id>
+ - Python version: python3 --version [e.g. 3.7.0]
+ - OS: [e.g. macOS 10.14]
+ - Chrome version: chromium-browser --version [e.g. v73.1.2.3]
@@ -0,0 +1,32 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+Type:
+
+ - [ ] General Question or Disussion
+ - [ ] Propose a brand new feature
+ - [ ] Request modification of existing behavior or design
+
+**What is the problem that your feature request solves**
+e.g. I need to be able to archive spanish and french subtitle files
+from a particular <example.com> movie site that's going down soon.
+
+**Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes**
+e.g.
+  specifically: a new archive method to look for multilingual subtitle files related to pages.
+  bigger picture: the ability for custom user scripts to be run in a puppeteer context to archive content
+
+**What hacks or alternative solutions have you tried to solve the problem?**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**How badly do you want this new feature?**
+ - [ ] It's an urgent deal-breaker, I cant live without it
+ - [ ] It's important to add it in the near-mid term future
+ - [ ] It would be nice to have eventualy
+ - [ ] I'm willing to contribute to development
@@ -0,0 +1,16 @@
+# Summary
+
+This PR add the ability to do XYZ...
+
+# Changes these areas
+
+- [ ] Config
+- [ ] Bugfixes
+- [ ] Command line interface
+- [ ] Feature behavior
+- [ ] Internal design
+- [ ] Archived data layout on disk
+
+# Roadmap Goals
+
+This PR helps us move towards ____ roadmap goal, as outlined here: https://github.com/pirate/ArchiveBox#roadmap
@@ -4,7 +4,10 @@

 # python
 __pycache__/
-archiver/venv
+venv
+.venv
+archivebox/.venv
+archivebox/venv

 # vim
 .swp*
@@ -12,3 +15,6 @@ archiver/venv
 # output artifacts
 output
 output/
+data
+data/
+archivebox/output
@@ -0,0 +1,3 @@
+[submodule "docs"]
+	path = docs
+	url = https://github.com/pirate/ArchiveBox.wiki.git
@@ -0,0 +1 @@
+archivebox.io
@@ -1,15 +0,0 @@
-# Donate
-
-Right now I'm working on this project in my spare time and accepting the occational PR,   
-if you want me to dedicate more time to it, donate to support development!
-
- - Ethereum: 0x5B0F85FFc44fD759C2d97f0BE4681279966f3832
- - Bitcoin: https://shapeshift.io/ BTC -> to address above (ETH)
- - Paypal: https://www.paypal.me/NicholasSweeting/25
-
-The eventual goal is to support one or two developers full-time on this project via donations.   
-With more engineering power this can become a distributed archive service with a nice UI,   
-like the Way-Back machine but hosted by everyone!
-
-If have any questions or want to sponsor this project long-term, contact me at  
-bookmark-archiver@sweeting.me
@@ -1,48 +1,57 @@
-FROM debian:stretch
-LABEL maintainer="Nick Sweeting <bookmark-archiver@sweeting.me>"
+FROM node:11-slim
+LABEL maintainer="Nick Sweeting <archivebox-git@sweeting.me>"

 RUN apt-get update \
-    && apt-get install -qy git wget gnupg2 libgconf-2-4 python3 python3-pip \
+    && apt-get install -yq --no-install-recommends \
+        git wget curl youtube-dl gnupg2 libgconf-2-4 python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*

 # Install latest chrome package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
-RUN apt-get update && apt-get install -y curl --no-install-recommends \
+RUN apt-get update && apt-get install -y wget --no-install-recommends \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
-    && apt-get install -y chromium fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
+    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* \
-    && rm -rf /src/*.deb \
-    && ln -s /usr/bin/chromium /usr/bin/chromium-browser
+    && rm -rf /src/*.deb

-# It might be a good idea to use dumb-init to help prevent zombie chrome processes.
-# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
-# RUN chmod +x /usr/local/bin/dumb-init
+# It's a good idea to use dumb-init to help prevent zombie chrome processes.
+ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
+RUN chmod +x /usr/local/bin/dumb-init

-RUN git clone https://github.com/pirate/ArchiveBox /home/chromeuser/app \
-    && pip3 install -r /home/chromeuser/app/archiver/requirements.txt
+# Install puppeteer so it's available in the container.
+RUN npm i puppeteer

-# Add user so we area strong, independent chrome that don't need --no-sandbox.
-RUN groupadd -r chromeuser && useradd -r -g chromeuser -G audio,video chromeuser \
-    && mkdir -p /home/chromeuser/app/archiver/output \
-    && chown -R chromeuser:chromeuser /home/chromeuser/app/archiver/output \
-    && chown -R chromeuser:chromeuser /home/chromeuser
+# Add user so we don't need --no-sandbox.
+RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
+    && mkdir -p /home/pptruser/Downloads \
+    && chown -R pptruser:pptruser /home/pptruser \
+    && chown -R pptruser:pptruser /node_modules

-VOLUME /home/chromeuser/app/archiver/output
+# Install the ArchiveBox repository and pip requirements
+RUN git clone https://github.com/pirate/ArchiveBox /home/pptruser/app \
+    && mkdir -p /data \
+    && chown -R pptruser:pptruser /data \
+    && ln -s /data /home/pptruser/app/archivebox/output \
+    && ln -s /home/pptruser/app/bin/* /bin/ \
+    && ln -s /home/pptruser/app/bin/archive /bin/archivebox \
+    && chown -R pptruser:pptruser /home/pptruser/app/archivebox
+    # && pip3 install -r /home/pptruser/app/archivebox/requirements.txt

-ENV LANG=en_US.UTF-8 \
+VOLUME /data
+
+ENV LANG=C.UTF-8 \
    LANGUAGE=en_US:en \
-    LC_ALL=en_US.UTF-8 \
+    LC_ALL=C.UTF-8 \
    PYTHONIOENCODING=UTF-8 \
    CHROME_SANDBOX=False \
-    OUTPUT_DIR=/home/chromeuser/app/archiver/output
+    CHROME_BINARY=google-chrome-unstable \
+    OUTPUT_DIR=/data

 # Run everything from here on out as non-privileged user
-USER chromeuser
-WORKDIR /home/chromeuser/app
+USER pptruser
+WORKDIR /home/pptruser/app

-# ENTRYPOINT ["dumb-init", "--"]
-# CMD ["/home/chromeuser/app/archive"]
-
-ENTRYPOINT ["python3", "-u", "/home/chromeuser/app/archiver/archive.py"]
+ENTRYPOINT ["dumb-init", "--"]
+CMD ["/bin/archive"]
@@ -1,488 +1,177 @@
-# ArchiveBox: Open source local web archiving <img src="https://nicksweeting.com/images/archive.png" height="22px"/>  [![Github Stars](https://img.shields.io/github/stars/pirate/bookmark-archiver.svg)](https://github.com/pirate/ArchiveBox) [![Twitter URL](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/thesquashSH)
+<div align="center">
+<img src="https://i.imgur.com/4nkFjdv.png" height="80px">
+<h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>

- ### (Recently [renamed](https://github.com/pirate/ArchiveBox/issues/108) from `Bookmark Archiver`)
+▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
+<a href="https://archive.sweeting.me">Demo</a> | 
+<a href="https://archivebox.io">Website</a> | 
+<a href="https://github.com/pirate/ArchiveBox">Github</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Troubleshooting">Troubleshooting</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Changelog">Changelog</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>

-    "Your own personal Way-Back Machine"
+<pre>
+"Your own personal internet archive" (网站存档 / 爬虫)
+</pre>

-▶️ [Quickstart](#quickstart) | [Details](#details) | [Configuration](#configuration) | [Manual Setup](#manual-setup) | [Troubleshooting](#troubleshooting) | [Demo](https://archive.sweeting.me) | [Source](https://github.com/pirate/ArchiveBox/tree/master) | [Changelog](#changelog) | [Donate](https://github.com/pirate/ArchiveBox/blob/master/DONATE.md)
+<a href="https://github.com/pirate/ArchiveBox">
+<img src="https://img.shields.io/badge/Open_source-free-green.svg?logo=git&logoColor=green"/>
+<img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/>
+<img src="https://img.shields.io/badge/License-MIT-lightgreen.svg?logo=MakerBot&logoColor=lightgreen"/>
+<img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/>
+<img src="https://img.shields.io/badge/Python-%3E%3D3.5-yellow.svg?logo=python&logoColor=yellow"/>
+<img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/>
+<img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/>
+</a>
+<hr/>
+</div>

---
+**ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).** 

-Save an archived copy of the websites you visit (the actual *content* of each site, not just the list of links).  Can archive entire browsing history, or just links matching a filter or bookmarks list.
+You can use it to preserve access to websites you care about by storing them locally offline.  ArchiveBox works by rendering the pages in a headless browser, then saving all the requests and fully loaded pages in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the original content dissapears off the internet.  It also automatically extracts assets like git repositories, audio, video, subtitles, images, and PDFs into separate files using `youtube-dl`, `pywb`, and `wget`.

-ArchiveBox can import links from:
+ArchiveBox doesn't require a constantly running server or backend, instead you just run the `./archive` command each time you want to import new links and update the static output.  It can import and export JSON (among other formats), so it's easy to script or hook up to other APIs.  If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).

- - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera)
- - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket
- - <img src="https://pinboard.in/favicon.ico" height="22px"/> Pinboard
- - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS or plain text lists
- - Shaarli, Delicious, Instapaper, Reddit Saved Posts, Wallabag, Unmark.it, and more!
+<div align="center"><sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub></div><br/>

-For each site, it outputs (configurable):
-
- Browsable static HTML archive (wget)
- PDF (Chrome headless)
- Screenshot (Chrome headless)
- HTML after 2s of JS running (Chrome headless)
- Favicon
- Submits URL to archive.org
- Index summary pages: index.html & index.json
-
-The archiving is additive, so you can schedule `./archive` to run regularly and pull new links into the index.
-All the saved content is static and indexed with json files, so it lives forever & is easily parseable, it requires no always-running backend.
-
-[DEMO: archive.sweeting.me](https://archive.sweeting.me)
-
-<img src="https://i.imgur.com/q3Oz9wN.png" width="75%" alt="Desktop Screenshot" align="top"><img src="https://i.imgur.com/TG0fGVo.png" width="25%" alt="Mobile Screenshot" align="top"><br/>
-
-## Quickstart
-
-**1. Get your list of URLs:**
-
-Follow the links here to find instructions for exporting a list of URLs from each service.
-
- - [Pocket](https://getpocket.com/export)
- - [Pinboard](https://pinboard.in/export/)
- - [Instapaper](https://www.instapaper.com/user/export)
- - [Reddit Saved Posts](https://github.com/csu/export-saved-reddit)
- - [Shaarli](https://shaarli.readthedocs.io/en/master/guides/backup-restore-import-export/#export-links-as)
- - [Unmark.it](http://help.unmark.it/import-export)
- - [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html)
- - [Chrome Bookmarks](https://support.google.com/chrome/answer/96816?hl=en)
- - [Firefox Bookmarks](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer)
- - [Safari Bookmarks](http://i.imgur.com/AtcvUZA.png)
- - [Opera Bookmarks](http://help.opera.com/Windows/12.10/en/importexport.html)
- - [Internet Explorer Bookmarks](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows)
- - Chrome History: `./bin/export-browser-history --chrome`
- - Firefox History: `./bin/export-browser-history --firefox`
- - Other File or URL: (e.g. RSS feed) pass as second argument in the next step
-
- (If any of these links are broken, please submit an issue and I'll fix it)
-
-**2. Create your archive:**

+To get started, you can install ArchiveBox [automatically](https://github.com/pirate/ArchiveBox/wiki/Quickstart), follow the [manual instructions](https://github.com/pirate/ArchiveBox/wiki/Install), or use [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker).  
+There are only [3 main dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) beyond `python3`: `wget`, `chromium`, and `youtube-dl`, and you can skip installing them if you don't need those archive methods.
 ```bash
-git clone https://github.com/pirate/ArchiveBox
-cd ArchiveBox/
-./setup                                         # install all dependencies
+git clone https://github.com/pirate/ArchiveBox.git
+cd ArchiveBox
+./setup

-# add a list of links from a file
-./archive ~/Downloads/bookmark_export.html      # replace with the path to your export file or URL from step 1
+# Export your bookmarks, then run the archive command to start archiving!
+./archive ~/Downloads/bookmarks.html

-# OR add a list of links from remote URL
-./archive "https://getpocket.com/users/yourusername/feed/all"  # url to an RSS, html, or json links file
-
-# OR add all the links from your browser history
-./bin/export-browser-history --chrome           # works with --firefox as well, can take path to SQLite history db
-./archive output/sources/chrome_history.json
-
-# OR just continue archiving the existing links in the index
-./archive   # at any point if you just want to continue archiving where you left off, without adding any new links
+# Or pass in links to archive via stdin
+echo 'https://example.com' | ./archive
 ```

-**3. Done!**
+Open `output/index.html` in a browser to view your archive.  [DEMO: archive.sweeting.me](https://archive.sweeting.me)  
+For more information, see the [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.

-You can open `output/index.html` to view your archive.  (favicons will appear next to each title once it has finished downloading)
+## Overview

-If you want to host your archive somewhere to share it with other people, see the [Publishing Your Archive](#publishing-your-archive) section below.
+Because modern websites are complicated and often rely on dynamic content, 
+ArchiveBox archives the sites in **several different formats** beyond what public 
+archiving services like Archive.org and Archive.is are capable of saving.

-**4. (Optional) Schedule it to run every day**
+ArchiveBox imports a list of URLs from stdin, remote url, or file, then adds the pages to a local archive folder using wget to create a browsable html clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...

-You can import links from any local file path or feed url by changing the second argument to `archive.py`.
-ArchiveBox will ignore links that are imported multiple times, it will keep the earliest version that it's seen.
-This means you can add multiple cron jobs to pull links from several different feeds or files each day,
-it will keep the index up-to-date without duplicate links.
+Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.

-This example archives a pocket RSS feed and an export file every 24 hours, and saves the output to a logfile.
-```bash
-0 24 * * * yourusername /opt/ArchiveBox/archive https://getpocket.com/users/yourusername/feed/all > /var/log/archivebox_rss.log
-0 24 * * * yourusername /opt/ArchiveBox/archive /home/darth-vader/Desktop/bookmarks.html > /var/log/archivebox_firefox.log
-```
-(Add the above lines to `/etc/crontab`)
+### Can import links from:

-**Next Steps**  
-  
-If you have any trouble, see the [Troubleshooting](#troubleshooting) section at the bottom.  
-If you'd like to customize options, see the [Configuration](#configuration) section.  
+ - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper
+ - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, HTML, Markdown, or plain text lists
+ - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera, and more)
+ - *Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, and any other text with links in it!*

-If you want something easier than running programs in the command-line, take a look at [Pocket Premium](https://getpocket.com/premium) (yay Mozilla!) and [Pinboard Pro](https://pinboard.in/upgrade/) (yay independent developer!).  Both offer easy-to-use bookmark archiving with full-text-search and other features.
+### Can save these things for each site:

-## Details
-
-`archive.py` is a script that takes a [Pocket-format](https://getpocket.com/export), [JSON-format](https://pinboard.in/export/), [Netscape-format](https://msdn.microsoft.com/en-us/library/aa753582(v=vs.85).aspx), or RSS-formatted list of links, and downloads a clone of each linked website to turn into a browsable archive that you can store locally or host online.
-
-The archiver produces an output folder `output/` containing an `index.html`, `index.json`, and archived copies of all the sites,
-organized by timestamp bookmarked.  It's Powered by [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) Chromium and good 'ol `wget`.
-
-For each sites it saves:
-
- - wget of site, e.g. `en.wikipedia.org/wiki/Example.html` with .html appended if not present
+ - `favicon.ico` favicon of the site
+ - `example.com/page-name.html` wget clone of the site, with .html appended if not present
 - `output.pdf` Printed PDF of site using headless chrome
 - `screenshot.png` 1440x900 screenshot of site using headless chrome
 - `output.html` DOM Dump of the HTML after rendering using headless chrome
 - `archive.org.txt` A link to the saved site on archive.org
- - `audio/` and `video/` for sites like youtube, soundcloud, etc. (using youtube-dl) (WIP)
- - `code/` clone of any repository for github, bitbucket, or gitlab links (WIP)
- - `index.json` JSON index containing link info and archive details
- - `index.html` HTML index containing link info and archive details (optional fancy or simple index)
+ - `warc/` for the html + gzipped warc file <timestamp>.gz
+ - `media/` any mp4, mp3, subtitles, and metadata found using youtube-dl
+ - `git/` clone of any repository for github, bitbucket, or gitlab links
+ - `index.html` & `index.json` HTML and JSON index files containing metadata and details

-Wget doesn't work on sites you need to be logged into, but chrome headless does, see the [Configuration](#configuration)* section for `CHROME_USER_DATA_DIR`.
+By default it does everything, but can disable or tweak [individual options](https://github.com/pirate/ArchiveBox/wiki/Configuration) via environment variables or config file.

-**Large Exports & Estimated Runtime:** 
+The archiving is additive, so you can schedule `./archive` to [run regularly](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) and pull new links into the index.
+All the saved content is static and indexed with JSON files, so it lives forever & is easily parseable, it requires no always-running backend.

-I've found it takes about an hour to download 1000 articles, and they'll take up roughly 1GB.  
-Those numbers are from running it single-threaded on my i5 machine with 50mbps down.  YMMV.  
+### Related Projects

-You can run it in parallel by using the `resume` feature, or by manually splitting export.html into multiple files:
-```bash
-./archive export.html 1498800000 &  # second argument is timestamp to resume downloading from
-./archive export.html 1498810000 &
-./archive export.html 1498820000 &
-./archive export.html 1498830000 &
-```
-Users have reported running it with 50k+ bookmarks with success (though it will take more RAM while running).
+There are tons of great web archiving tools out there.  In particular https://webrecorder.io ([pywb](https://github.com/webrecorder/pywb)) and https://getpolarized.io/ are robust, stable pieces of software that have lots of overlap with ArchiveBox.  ArchiveBox differentiates itself by being primarily a one-shot CLI tool that specializing in importing streams of links from RSS, JSON (good for automatically archiving from a stream of browser history or bookmarks on a schedule), as opposed to a desktop application or web service that requires human interaction to add links.

-If you already imported a huge list of bookmarks and want to import only new
-bookmarks, you can use the `ONLY_NEW` environment variable. This is useful if
-you want to import a bookmark dump periodically and want to skip broken links
-which are already in the index.
+To learn more about the motivation for this project and how it fits into the broader community, see our [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.

-## Configuration
+# Documentation

-You can tweak parameters via environment variables, or by editing `config.py` directly:
-```bash
-env CHROME_BINARY=google-chrome-stable RESOLUTION=1440,900 FETCH_PDF=False ./archive ~/Downloads/bookmarks_export.html
-```

-**Shell Options:**
- - colorize console ouput: `USE_COLOR` value: [`True`]/`False`
- - show progress bar: `SHOW_PROGRESS` value: [`True`]/`False`
- - archive permissions: `OUTPUT_PERMISSIONS` values: [`755`]/`644`/`...`
+<div align="center">
+<img src="https://i.imgur.com/PVO88AZ.png"/>
+<br/>
+    <sub><i>(Recently <a href="https://github.com/pirate/ArchiveBox/issues/108">renamed</a> from <code>Bookmark Archiver</code>)</i></sub>
+</div>

-**Dependency Options:**
- - path to Chrome: `CHROME_BINARY` values: [`chromium-browser`]/`/usr/local/bin/google-chrome`/`...`
- - path to wget: `WGET_BINARY` values: [`wget`]/`/usr/local/bin/wget`/`...`
+---

-**Archive Options:**
- - maximum allowed download time per link: `TIMEOUT` values: [`60`]/`30`/`...`
- - import only new links: `ONLY_NEW` values `True`/[`False`]
- - archive methods (values: [`True`]/`False`):
-   - fetch page with wget: `FETCH_WGET`
-   - fetch images/css/js with wget: `FETCH_WGET_REQUISITES` (True is highly recommended)
-   - print page as PDF: `FETCH_PDF`
-   - fetch a screenshot of the page: `FETCH_SCREENSHOT`
-   - fetch a DOM dump of the page: `FETCH_DOM`
-   - fetch a favicon for the page: `FETCH_FAVICON`
-   - submit the page to archive.org: `SUBMIT_ARCHIVE_DOT_ORG` 
- - screenshot: `RESOLUTION` values: [`1440,900`]/`1024,768`/`...`
- - user agent: `WGET_USER_AGENT` values: [`Wget/1.19.1`]/`"Mozilla/5.0 ..."`/`...`
- - chrome profile: `CHROME_USER_DATA_DIR` values: [`~/Library/Application\ Support/Google/Chrome/Default`]/`/tmp/chrome-profile`/`...`
-    To capture sites that require a user to be logged in, you must specify a path to a chrome profile (which loads the cookies needed for the user to be logged in).  If you don't have an existing chrome profile, create one with `chromium-browser --disable-gpu --user-data-dir=/tmp/chrome-profile`, and log into the sites you need.  Then set `CHROME_USER_DATA_DIR=/tmp/chrome-profile` to make ArchiveBox use that profile.
- - output directory: `OUTPUT_DIR` values: [`$REPO_DIR/output`]/`/srv/www/bookmarks`/`...` Optionally output the archives to an alternative directory.
+We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) for documentation.

- (See defaults & more at the top of `config.py`)
+You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.

-To tweak the outputted html index file's look and feel, just edit the HTML files in `archiver/templates/`.
+## Getting Started

-The chrome/chromium dependency is _optional_ and only required for screenshots, PDF, and DOM dump output, it can be safely ignored if those three methods are disabled.
+ - [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
+ - [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
+ - [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)

-## Publishing Your Archive
+## Reference

-The archive produced by `./archive` is suitable for serving on any provider that can host static html (e.g. github pages!).
+ - [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
+ - [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
+ - [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
+ - [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
+ - [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Chromium-Install)
+ - [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)

-You can also serve it from a home server or VPS by uploading the outputted `output` folder to your web directory, e.g. `/var/www/ArchiveBox` and configuring your webserver.
+## More Info

-Here's a sample nginx configuration that works to serve archive folders:
+ - [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
+ - [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
+ - [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
+ - [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)

-```nginx
-location / {
-    alias       /path/to/ArchiveBox/output/;
-    index       index.html;
-    autoindex   on;               # see directory listing upon clicking "The Files" links
-    try_files   $uri $uri/ =404;
-}
-```

-Make sure you're not running any content as CGI or PHP, you only want to serve static files!
+# Background & Motivation

-Urls look like: `https://archive.example.com/archive/1493350273/en.wikipedia.org/wiki/Dining_philosophers_problem.html`
+Vast treasure troves of knowledge are lost every day on the internet to link rot.  As a society, we have an imperative
+to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.

-**Security WARNING & Content Disclaimer**
+Whether it's to resist censorship by saving articles before they get taken down or editied, or
+just to save a collection of early 2010's flash games you love to play, having the tools to 
+archive internet content enables to you save the stuff you care most about before it dissapears.

-Re-hosting other people's content has security implications for any other sites sharing your hosting domain.  Make sure you understand
-the dangers of hosting unknown archived CSS & JS files [on your shared domain](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy).
-Due to the security risk of serving some malicious JS you archived by accident, it's best to put this on a domain or subdomain
-of its own to keep cookies separate and slightly mitigate [CSRF attacks](https://en.wikipedia.org/wiki/Cross-site_request_forgery) and other nastiness.
+The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. 
+I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.

-You may also want to blacklist your archive in `/robots.txt` if you don't want to be publicly assosciated with all the links you archive via search engine results.
+The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.

-Be aware that some sites you archive may not allow you to rehost their content publicly for copyright reasons,
-it's up to you to host responsibly and respond to takedown requests appropriately.
+*Read more:*

-Please modify the `FOOTER_INFO` config variable to add your contact info to the footer of your index.
+- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
+- Discover the web archiving community on the [community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.
+- Find other archving projects on Github using the [awesome-web-archiving](https://github.com/iipc/awesome-web-archiving) list.
+- Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.

-## Info & Motivation
+To learn more about ArchiveBox's past history and future plans, check out the [roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog).

-This is basically an open-source version of [Pocket Premium](https://getpocket.com/premium) (which you should consider paying for!).
-I got tired of sites I saved going offline or changing their URLS, so I started
-archiving a copy of them locally now, similar to The Way-Back Machine provided
-by [archive.org](https://archive.org).  Self hosting your own archive allows you to save
-PDFs & Screenshots of dynamic sites in addition to static html, something archive.org doesn't do.
+# Screenshots

-Now I can rest soundly knowing important articles and resources I like wont dissapear off the internet.
+<img src="https://i.imgur.com/viklZNG.png" width="75%" alt="Desktop index screenshot" align="top"><img src="https://i.imgur.com/mW2dITg.png" width="25%" alt="Mobile details page screenshot" align="top"><br/>
+<img src="https://i.imgur.com/wnpdAVM.jpg" width="100%" alt="Desktop details page Screenshot"/><br/>
+<img src="https://i.imgur.com/3tBL7PU.png" width="100%" alt="CLI Screenshot">

-My published archive as an example: [archive.sweeting.me](https://archive.sweeting.me).
+---

-## Manual Setup
-
-If you don't like running random setup scripts off the internet (:+1:), you can follow these manual setup instructions.
-
-**1. Install dependencies:** `chromium >= 59`,` wget >= 1.16`, `python3 >= 3.5`  (`google-chrome >= v59` works fine as well)
-
-If you already have Google Chrome installed, or wish to use that instead of Chromium, follow the [Google Chrome Instructions](#google-chrome-instructions).
-
-```bash
-# On Mac:
-brew cask install chromium  # If you already have Google Chrome/Chromium in /Applications/, skip this command
-brew install wget python3
-
-echo -e '#!/bin/bash\n/Applications/Chromium.app/Contents/MacOS/Chromium "$@"' > /usr/local/bin/chromium-browser  # see instructions for google-chrome below
-chmod +x /usr/local/bin/chromium-browser
-```
-
-```bash
-# On Ubuntu/Debian:
-apt install chromium-browser python3 wget
-```
-
-```bash
-# Check that everything worked:
-chromium-browser --version && which wget && which python3 && which curl && echo "[√] All dependencies installed."
-```
-
-**2. Get your bookmark export file:**
-
-Follow the instruction links above in the "Quickstart" section to download your bookmarks export file.
-
-**3. Run the archive script:**
-
-1. Clone this repo `git clone https://github.com/pirate/ArchiveBox`
-3. `cd ArchiveBox/`
-4. `./archive ~/Downloads/bookmarks_export.html`
-
-You may optionally specify a second argument to `archive.py export.html 153242424324` to resume the archive update at a specific timestamp.
-
-If you have any trouble, see the [Troubleshooting](#troubleshooting) section at the bottom.
-
-### Google Chrome Instructions:
-
-I recommend Chromium instead of Google Chrome, since it's open source and doesn't send your data to Google.
-Chromium may have some issues rendering some sites though, so you're welcome to try Google-chrome instead.
-It's also easier to use Google Chrome if you already have it installed, rather than downloading Chromium all over.
-
-1. Install & link google-chrome
-```bash
-# On Mac:
-# If you already have Google Chrome in /Applications/, skip this brew command
-brew cask install google-chrome
-brew install wget python3
-
-echo -e '#!/bin/bash\n/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome "$@"' > /usr/local/bin/google-chrome
-chmod +x /usr/local/bin/google-chrome
-```
-
-```bash
-# On Linux:
-wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
-sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
-apt update; apt install google-chrome-beta python3 wget
-```
-
-2. Set the environment variable `CHROME_BINARY` to `google-chrome` before running:
-
-```bash
-env CHROME_BINARY=google-chrome ./archive ~/Downloads/bookmarks_export.html
-```
-If you're having any trouble trying to set up Google Chrome or Chromium, see the Troubleshooting section below.
-
-## Troubleshooting
-
-### Dependencies
-
-**Python:**
-
-On some Linux distributions the python3 package might not be recent enough.
-If this is the case for you, resort to installing a recent enough version manually.
-```bash
-add-apt-repository ppa:fkrull/deadsnakes && apt update && apt install python3.6
-```
-If you still need help, [the official Python docs](https://docs.python.org/3.6/using/unix.html) are a good place to start.
-
-**Chromium/Google Chrome:**
-
-`archive.py` depends on being able to access a `chromium-browser`/`google-chrome` executable.  The executable used
-defaults to `chromium-browser` but can be manually specified with the environment variable `CHROME_BINARY`:
-
-```bash
-env CHROME_BINARY=/usr/local/bin/chromium-browser ./archive ~/Downloads/bookmarks_export.html
-```
-
-1. Test to make sure you have Chrome on your `$PATH` with:
-
-```bash
-which chromium-browser || which google-chrome
-```
-If no executable is displayed, follow the setup instructions to install and link one of them.
-
-2. If a path is displayed, the next step is to check that it's runnable:
-
-```bash
-chromium-browser --version || google-chrome --version
-```
-If no version is displayed, try the setup instructions again, or confirm that you have permission to access chrome.
-
-3. If a version is displayed and it's `<59`, upgrade it:
-
-```bash
-apt upgrade chromium-browser -y
-# OR
-brew cask upgrade chromium-browser
-```
-
-4. If a version is displayed and it's `>=59`, make sure `archive.py` is running the right one:
-
-```bash
-env CHROME_BINARY=/path/from/step/1/chromium-browser ./archive bookmarks_export.html   # replace the path with the one you got from step 1
-```
-
-
-**Wget & Curl:**
-
-If you're missing `wget` or `curl`, simply install them using `apt` or your package manager of choice.
-See the "Manual Setup" instructions for more details.
-
-If wget times out or randomly fails to download some sites that you have confirmed are online,
-upgrade wget to the most recent version with `brew upgrade wget` or `apt upgrade wget`.  There is
-a bug in versions `<=1.19.1_1` that caused wget to fail for perfectly valid sites.
-
-### Archiving
-
-**No links parsed from export file:**
-
-Please open an [issue](https://github.com/pirate/ArchiveBox/issues) with a description of where you got the export, and
-preferrably your export file attached (you can redact the links).  We'll fix the parser to support your format.
-
-**Lots of skipped sites:**
-
-If you ran the archiver once, it wont re-download sites subsequent times, it will only download new links.
-If you haven't already run it, make sure you have a working internet connection and that the parsed URLs look correct.
-You can check the `archive.py` output or `index.html` to see what links it's downloading.
-
-If you're still having issues, try deleting or moving the `output/archive` folder (back it up first!) and running `./archive` again.
-
-**Lots of errors:**
-
-Make sure you have all the dependencies installed and that you're able to visit the links from your browser normally.
-Open an [issue](https://github.com/pirate/ArchiveBox/issues) with a description of the errors if you're still having problems.
-
-**Lots of broken links from the index:**
-
-Not all sites can be effectively archived with each method, that's why it's best to use a combination of `wget`, PDFs, and screenshots.
-If it seems like more than 10-20% of sites in the archive are broken, open an [issue](https://github.com/pirate/ArchiveBox/issues)
-with some of the URLs that failed to be archived and I'll investigate.
-
-**Removing unwanted links from the index:**
-
-If you accidentally added lots of unwanted links into index and they slow down your archiving, you can use the `bin/purge` script to remove them from your index, which removes everything matching python regexes you pass into it. E.g: `bin/purge -r 'amazon\.com' -r 'google\.com'`. It would prompt before removing links from index, but for extra safety you might want to back up `index.json` first (or put in undex version control).
-
-### Hosting the Archive
-
-If you're having issues trying to host the archive via nginx, make sure you already have nginx running with SSL.
-If you don't, google around, there are plenty of tutorials to help get that set up.  Open an [issue](https://github.com/pirate/ArchiveBox/issues)
-if you have problem with a particular nginx config.
-
-
-## Links
-
-**Similar Projects:**
- - [Reminiscence](https://github.com/kanishka-linux/reminiscence/) extremely similar to BA, uses a Django backend + UI and provides auto tagging and summary features with NLTK
- - [Memex by Worldbrain.io](https://github.com/WorldBrain/Memex) a browser extension that saves all your history and does full-text search
- - [Hypothes.is](https://web.hypothes.is/) a web/pdf/ebook annotation tool that also archives content
- - [Perkeep](https://perkeep.org/) "Perkeep lets you permanently keep your stuff, for life."
- - [Fetching.io](http://fetching.io/) A personal search engine/archiver that lets you search through all archived websites that you've bookmarked
- - [Shaarchiver](https://github.com/nodiscc/shaarchiver) very similar project that archives Firefox, Shaarli, or Delicious bookmarks and all linked media, generating a markdown/HTML index
- - [Webrecorder.io](https://webrecorder.io/) Save full browsing sessions and archive all the content
- - [Wallabag](https://wallabag.org) Save articles you read locally or on your phone
-
-**Discussions:**
- - [Hacker News Discussion](https://news.ycombinator.com/item?id=14272133)
- - [Reddit r/selfhosted Discussion](https://www.reddit.com/r/selfhosted/comments/69eoi3/pocket_stream_archive_your_own_personal_wayback/)
- - [Reddit r/datahoarder Discussion #1](https://www.reddit.com/r/DataHoarder/comments/69e6i9/archive_a_browseable_copy_of_your_saved_pocket/)
- - [Reddit r/datahoarder Discussion #2](https://www.reddit.com/r/DataHoarder/comments/6kepv6/bookmarkarchiver_now_supports_archiving_all_major/)
-
-
-**Tools/Other:**
- - https://github.com/ikreymer/webarchiveplayer#auto-load-warcs
- - [Sheetsee-Pocket](http://jlord.us/sheetsee-pocket/) project that provides a pretty auto-updating index of your Pocket links (without archiving them)
- - [Pocket -> IFTTT -> Dropbox](https://christopher.su/2013/saving-pocket-links-file-day-dropbox-ifttt-launchd/) Post by Christopher Su on his Pocket saving IFTTT recipie
-
-
-## Roadmap
-
-[*Official Roadmap*](https://github.com/pirate/ArchiveBox/issues/120).
-
-If you feel like contributing a PR, some of these tasks are pretty easy.  Feel free to open an issue if you need help getting started in any way!
-
-**Major upcoming changes:**
-
- - finalize python packaging to allow installing via pip and importing individual componenets
- - add an optional web GUI for managing sources, adding new links, and viewing the archive
-
-**Minor upcoming changes:**
- - download closed-captions text from youtube videos
- - body text extraction using [fathom](https://hacks.mozilla.org/2017/04/fathom-a-framework-for-understanding-web-pages/)
- - auto-tagging based on important extracted words
- - audio & video archiving with `youtube-dl`
- - full-text indexing with elasticsearch/elasticlunr/ag
- - video closed-caption downloading on Youtube for full-text indexing of video content
- - automatic text summaries of article with nlp summarization library
- - featured image extraction
- - http support (from my https-only domain)
- - try wgetting dead sites from archive.org (https://github.com/hartator/wayback-machine-downloader)
- 
- 
-## Changelog
-
- - [renamed](https://github.com/pirate/ArchiveBox/issues/108) from **Bookmark Archiver** -> **ArchiveBox**
- - v0.1.0 released
- - support for browser history exporting added with `./bin/export-browser-history`
- - support for chrome `--dump-dom` to output full page HTML after JS executes
- - v0.0.3 released
- - support for chrome `--user-data-dir` to archive sites that need logins
- - fancy individual html & json indexes for each link
- - smartly append new links to existing index instead of overwriting 
-  - v0.0.2 released
- - proper HTML templating instead of format strings (thanks to https://github.com/bardisty!)
- - refactored into separate files, wip audio & video archiving
- - v0.0.1 released
- - Index links now work without nginx url rewrites, archive can now be hosted on github pages
- - added setup.sh script & docstrings & help commands
- - made Chromium the default instead of Google Chrome (yay free software)
- - added [env-variable](https://github.com/pirate/ArchiveBox/pull/25) configuration (thanks to https://github.com/hannah98!)
- - renamed from **Pocket Archive Stream** -> **Bookmark Archiver**
- - added [Netscape-format](https://github.com/pirate/ArchiveBox/pull/20) export support (thanks to https://github.com/ilvar!)
- - added [Pinboard-format](https://github.com/pirate/ArchiveBox/pull/7) export support (thanks to https://github.com/sconeyard!)
- - front-page of HN, oops! apparently I have users to support now :grin:?
- - added Pocket-format export support
- - v0.0.0 released: created Pocket Archive Stream 2017/05/05
- 
- 
- ## Donations
- 
- This project can really flourish with some more engineering effort, but unless it can support   
- me financially I'm unlikely to be able to take it to the next level alone.  It's already pretty   
- functional and robust, but it really deserves to be taken to the next level with a few more   
- talented engineers.  If you want to help sponsor this project long-term or just say thanks or suggest changes, contact  
- me at bookmark-archiver@sweeting.me.
-   
- [Grants / Donations](https://github.com/pirate/ArchiveBox/blob/master/DONATE.md)
+<div align="center">
+<br/><br/>
+<img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" height="40px"/>
+<br/>
+<sub><i>This project is maintained mostly in my spare time with the help from generous contributors.</i></sub>
+<br/><br/>
+<a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
+<br/>
+<br/>
+<a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
+<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
+</div>
@@ -0,0 +1 @@
+theme: jekyll-theme-merlot
@@ -7,83 +7,78 @@ import os
 import sys

 from datetime import datetime
-from subprocess import run
+from peekable import Peekable
+

 from parse import parse_links
-from links import (
-    new_links,
-    validate_links
-)
-from archive_methods import archive_links, _RESULTS_TOTALS
+from links import validate_links, links_after_timestamp
+from archive_methods import archive_link, _RESULTS_TOTALS
 from index import (
    write_links_index,
-    write_link_index,
    parse_json_links_index,
-    parse_json_link_index,
 )
 from config import (
+    ARCHIVE_DIR,
    ONLY_NEW,
-    OUTPUT_PERMISSIONS,
    OUTPUT_DIR,
+    REPO_DIR,
    ANSI,
-    TIMEOUT,
    GIT_SHA,
 )
 from util import (
-    download_url,
-    progress,
-    cleanup_archive,
+    check_dependencies,
+    save_remote_source,
+    save_stdin_source,
    pretty_path,
    migrate_data,
+    check_links_structure,
 )

 __AUTHOR__ = 'Nick Sweeting <git@nicksweeting.com>'
 __VERSION__ = GIT_SHA
-__DESCRIPTION__ = 'ArchiveBox: Create a browsable html archive of a list of links.'
-__DOCUMENTATION__ = 'https://github.com/pirate/ArchiveBox'
+__DESCRIPTION__ = 'ArchiveBox Usage:  Create a browsable html archive of a list of links.'
+__DOCUMENTATION__ = 'https://github.com/pirate/ArchiveBox/wiki'
+

 def print_help():
    print(__DESCRIPTION__)
    print("Documentation:     {}\n".format(__DOCUMENTATION__))
    print("Usage:")
    print("    ./bin/archivebox ~/Downloads/bookmarks_export.html\n")
+    print("")
+    print("    ./bin/archivebox https://example.com/feed.rss\n")
+    print("")
+    print("    echo 'https://examplecom' | ./bin/archivebox\n")


-def merge_links(archive_path=OUTPUT_DIR, import_path=None, only_new=False):
+def load_links(archive_path=OUTPUT_DIR, import_path=None):
    """get new links from file and optionally append them to links in existing archive"""
-    all_links = []
-    if import_path:
-        # parse and validate the import file
-        raw_links = parse_links(import_path)
-        all_links = validate_links(raw_links)

-    # merge existing links in archive_path and new links
    existing_links = []
    if archive_path:
        existing_links = parse_json_links_index(archive_path)
-        all_links = validate_links(existing_links + all_links)
-    
+        check_links_structure(existing_links)
+
+    new_links = []
+    if import_path:
+        # parse and validate the import file
+        raw_links, parser_name = parse_links(import_path)
+        new_links = validate_links(raw_links)
+        check_links_structure(new_links)
+
+    # merge existing links in archive_path and new links
+    all_links = validate_links(existing_links + new_links)
+    check_links_structure(all_links)
    num_new_links = len(all_links) - len(existing_links)
-    if num_new_links and not only_new:
-        print('[{green}+{reset}] [{}] Adding {} new links from {} to {}/index.json'.format(
-            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+
+    if import_path and parser_name:
+        print('    > Adding {} new links to index (parsed import as {})'.format(
            num_new_links,
-            pretty_path(import_path),
-            pretty_path(archive_path),
-            **ANSI,
+            parser_name,
        ))
-    # else:
-    #     print('[*] [{}] No new links added to {}/index.json{}'.format(
-    #         datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-    #         archive_path,
-    #         ' from {}'.format(import_path) if import_path else '',
-    #         **ANSI,
-    #     ))

-    if only_new:
-        return new_links(all_links, existing_links)
+    return all_links, new_links

-    return all_links

 def update_archive(archive_path, links, source=None, resume=None, append=True):
    """update or create index.html+json given a path to an export file containing new links"""
@@ -97,14 +92,44 @@ def update_archive(archive_path, links, source=None, resume=None, append=True):
             **ANSI,
        ))
    else:
-        print('{green}[▶] [{}] Updating files for {} links in archive...{reset}'.format(
+        print('{green}[▶] [{}] Updating content for {} pages in archive...{reset}'.format(
             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
             len(links),
             **ANSI,
        ))

+    check_links_structure(links)
+
+    # prefetch the first link off the generator so that if we pause or fail
+    # immediately we can show that we paused on the first link and not just None
+    to_archive = Peekable(links_after_timestamp(links, resume))
+    idx, link = 0, to_archive.peek(0)
+
    # loop over links and archive them
-    archive_links(archive_path, links, source=source, resume=resume)
+    try:
+        check_dependencies()
+        for idx, link in enumerate(to_archive):
+            link_dir = os.path.join(ARCHIVE_DIR, link['timestamp'])
+            archive_link(link_dir, link)
+
+    except (KeyboardInterrupt, SystemExit, Exception) as e:
+        print('\n{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format(
+            **ANSI,
+            now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            idx=idx+1,
+            timestamp=link['timestamp'],
+            total=len(links),
+        ))
+        print('    To view your archive, open: {}/index.html'.format(OUTPUT_DIR.replace(REPO_DIR + '/', '')))
+        print('    Continue where you left off by running:')
+        print('        {} {}'.format(
+            pretty_path(sys.argv[0]),
+            link['timestamp'],
+        ))
+        if not isinstance(e, KeyboardInterrupt):
+            print()
+            raise e
+        raise SystemExit(1)

    # print timing information & summary
    end_ts = datetime.now().timestamp()
@@ -114,7 +139,7 @@ def update_archive(archive_path, links, source=None, resume=None, append=True):
    else:
        duration = '{0:.2f} sec'.format(seconds, 2)

-    print('{}[√] [{}] Update of {} links complete ({}){}'.format(
+    print('{}[√] [{}] Update of {} pages complete ({}){}'.format(
        ANSI['green'],
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
        len(links),
@@ -124,6 +149,7 @@ def update_archive(archive_path, links, source=None, resume=None, append=True):
    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
+    print('    To view your archive, open: {}/index.html'.format(OUTPUT_DIR.replace(REPO_DIR + '/', '')))


 if __name__ == '__main__':
@@ -138,6 +164,20 @@ if __name__ == '__main__':
    source = sys.argv[1] if argc > 1 else None  # path of links file to import
    resume = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
   
+    stdin_raw_text = ''
+
+    if not sys.stdin.isatty():
+        stdin_raw_text = sys.stdin.read()
+
+    if source and stdin_raw_text:
+        print(
+            '[X] You should pass either a path as an argument, '
+            'or pass a list of links via stdin, but not both.\n'
+        )
+        print_help()
+        raise SystemExit(1)
+
+
    if argc == 1:
        source, resume = None, None
    elif argc == 2:
@@ -162,20 +202,22 @@ if __name__ == '__main__':

    # Step 0: Download url to local file (only happens if a URL is specified instead of local path) 
    if source and any(source.startswith(s) for s in ('http://', 'https://', 'ftp://')):
-        source = download_url(source)
+        source = save_remote_source(source)
+    elif stdin_raw_text:
+        source = save_stdin_source(stdin_raw_text)

    # Step 1: Parse the links and dedupe them with existing archive
-    links = merge_links(archive_path=out_dir, import_path=source, only_new=False)
-    new_links = merge_links(archive_path=out_dir, import_path=source, only_new=True)
+    all_links, new_links = load_links(archive_path=out_dir, import_path=source)

    # Step 2: Write new index
-    write_links_index(out_dir=out_dir, links=links)
+    write_links_index(out_dir=out_dir, links=all_links)

-    # Step 3: Verify folder structure is 1:1 with index
-    # cleanup_archive(out_dir, links)
-
-    # Step 4: Run the archive methods for each link
+    # Step 3: Run the archive methods for each link
    if ONLY_NEW:
        update_archive(out_dir, new_links, source=source, resume=resume, append=True)
    else:
-        update_archive(out_dir, links, source=source, resume=resume, append=True)
+        update_archive(out_dir, all_links, source=source, resume=resume, append=True)
+
+    # Step 4: Re-write links index with updated titles, icons, and resources
+    all_links, _ = load_links(archive_path=out_dir)
+    write_links_index(out_dir=out_dir, links=all_links)
@@ -1,40 +1,54 @@
 import os
-import sys

 from functools import wraps
 from collections import defaultdict
 from datetime import datetime
-from subprocess import run, PIPE, DEVNULL

-from peekable import Peekable
-
-from index import wget_output_path, parse_json_link_index, write_link_index
-from links import links_after_timestamp
+from index import (
+    wget_output_path,
+    parse_json_link_index,
+    write_link_index,
+    patch_index_title_hack,
+)
 from config import (
+    OUTPUT_DIR,
+    CURL_BINARY,
+    GIT_BINARY,
+    WGET_BINARY,
+    YOUTUBEDL_BINARY,
    CHROME_BINARY,
+    FETCH_FAVICON,
+    FETCH_TITLE,
    FETCH_WGET,
    FETCH_WGET_REQUISITES,
    FETCH_PDF,
    FETCH_SCREENSHOT,
    FETCH_DOM,
+    FETCH_WARC,
+    FETCH_GIT,
+    FETCH_MEDIA,
    RESOLUTION,
    CHECK_SSL_VALIDITY,
    SUBMIT_ARCHIVE_DOT_ORG,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
-    FETCH_FAVICON,
+    COOKIES_FILE,
    WGET_USER_AGENT,
    CHROME_USER_DATA_DIR,
    CHROME_SANDBOX,
    TIMEOUT,
+    MEDIA_TIMEOUT,
    ANSI,
    ARCHIVE_DIR,
+    GIT_DOMAINS,
+    GIT_SHA,
 )
 from util import (
-    check_dependencies,
+    without_fragment,
+    fetch_page_title,
    progress,
    chmod_file,
    pretty_path,
+    check_link_structure,
+    run, PIPE, DEVNULL,
 )


@@ -44,90 +58,70 @@ _RESULTS_TOTALS = {   # globals are bad, mmkay
    'failed': 0,
 }

-def archive_links(archive_path, links, source=None, resume=None):
-    check_dependencies()
-
-    to_archive = Peekable(links_after_timestamp(links, resume))
-    idx, link = 0, to_archive.peek(0)
-
-    try:
-        for idx, link in enumerate(to_archive):
-            link_dir = os.path.join(ARCHIVE_DIR, link['timestamp'])
-            archive_link(link_dir, link)
-    
-    except (KeyboardInterrupt, SystemExit, Exception) as e:
-        print('{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format(
-            **ANSI,
-            now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            idx=idx+1,
-            timestamp=link['timestamp'],
-            total=len(links),
-        ))
-        print('    Continue where you left off by running:')
-        print('        {} {}'.format(
-            pretty_path(sys.argv[0]),
-            link['timestamp'],
-        ))
-        if not isinstance(e, KeyboardInterrupt):
-            raise e
-        raise SystemExit(1)
-

 def archive_link(link_dir, link, overwrite=True):
    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""

-    update_existing = os.path.exists(link_dir)
-    if update_existing:
-        link = {
-            **parse_json_link_index(link_dir),
-            **link,
-        }
-    else:
-        os.makedirs(link_dir)
-    
-    log_link_archive(link_dir, link, update_existing)
+    check_link_structure(link)

-    if FETCH_WGET:
-        link = fetch_wget(link_dir, link, overwrite=overwrite)
+    try:
+        update_existing = os.path.exists(link_dir)
+        if update_existing:
+            link = {
+                **parse_json_link_index(link_dir),
+                **link,
+            }
+        else:
+            os.makedirs(link_dir)
+        
+        print_link_status_line(link_dir, link, update_existing)

-    if FETCH_PDF:
-        link = fetch_pdf(link_dir, link, overwrite=overwrite)
+        if FETCH_FAVICON:
+            link = fetch_favicon(link_dir, link, overwrite=overwrite)

-    if FETCH_SCREENSHOT:
-        link = fetch_screenshot(link_dir, link, overwrite=overwrite)
+        if FETCH_TITLE:
+            link = fetch_title(link_dir, link, overwrite=overwrite)

-    if FETCH_DOM:
-        link = fetch_dom(link_dir, link, overwrite=overwrite)
+        if FETCH_WGET:
+            link = fetch_wget(link_dir, link, overwrite=overwrite)

-    if SUBMIT_ARCHIVE_DOT_ORG:
-        link = archive_dot_org(link_dir, link, overwrite=overwrite)
+        if FETCH_PDF:
+            link = fetch_pdf(link_dir, link, overwrite=overwrite)

-    # if FETCH_AUDIO:
-    #     link = fetch_audio(link_dir, link, overwrite=overwrite)
+        if FETCH_SCREENSHOT:
+            link = fetch_screenshot(link_dir, link, overwrite=overwrite)

-    # if FETCH_VIDEO:
-    #     link = fetch_video(link_dir, link, overwrite=overwrite)
+        if FETCH_DOM:
+            link = fetch_dom(link_dir, link, overwrite=overwrite)

-    if FETCH_FAVICON:
-        link = fetch_favicon(link_dir, link, overwrite=overwrite)
+        if SUBMIT_ARCHIVE_DOT_ORG:
+            link = archive_dot_org(link_dir, link, overwrite=overwrite)

-    write_link_index(link_dir, link)
-    # print()
+        if FETCH_GIT:
+            link = fetch_git(link_dir, link, overwrite=overwrite)
+
+        if FETCH_MEDIA:
+            link = fetch_media(link_dir, link, overwrite=overwrite)
+
+        write_link_index(link_dir, link)
+
+    except Exception as err:
+        print('    ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err))
    
    return link

-def log_link_archive(link_dir, link, update_existing):
+def print_link_status_line(link_dir, link, update_existing):
    print('[{symbol_color}{symbol}{reset}] [{now}] "{title}"\n    {blue}{url}{reset}'.format(
        symbol='*' if update_existing else '+',
        symbol_color=ANSI['black' if update_existing else 'green'],
        now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        **link,
+        **{**link, 'title': link['title'] or link['url']},
        **ANSI,
    ))

    print('    > {}{}'.format(pretty_path(link_dir), '' if update_existing else ' (new)'))
-    if link['type']:
-        print('      i {}'.format(link['type']))
+    # if link['type']:
+    #     print('      i {}'.format(link['type']))



@@ -185,7 +179,7 @@ def attach_result_to_link(method):


@attach_result_to_link('wget')
-def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
+def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, warc=FETCH_WARC, timeout=TIMEOUT):
    """download full site using wget"""

    domain_dir = os.path.join(link_dir, link['domain'])
@@ -193,39 +187,74 @@ def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT
    if os.path.exists(domain_dir) and existing_file:
        return {'output': existing_file, 'status': 'skipped'}

+    if warc:
+        warc_dir = os.path.join(link_dir, 'warc')
+        os.makedirs(warc_dir, exist_ok=True)
+        warc_path = os.path.join('warc', str(int(datetime.now().timestamp())))
+
+    # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
    CMD = [
-        # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
-        *'wget -N -E -np -x -H -k -K -S --restrict-file-names=unix'.split(' '),
-        *(('-p',) if FETCH_WGET_REQUISITES else ()),
+        WGET_BINARY,
+        # '--server-response',  # print headers for better error parsing
+        '--no-verbose',
+        '--adjust-extension',
+        '--convert-links',
+        '--force-directories',
+        '--backup-converted',
+        '--span-hosts',
+        '--no-parent',
+        '-e', 'robots=off',
+        '--restrict-file-names=unix',
+        '--timeout={}'.format(timeout),
+        *(() if warc else ('--timestamping',)),
+        *(('--warc-file={}'.format(warc_path),) if warc else ()),
+        *(('--page-requisites',) if FETCH_WGET_REQUISITES else ()),
        *(('--user-agent={}'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
-        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate',))),
+        *(('--load-cookies', COOKIES_FILE) if COOKIES_FILE else ()),
+        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate', '--no-hsts'))),
        link['url'],
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # index.html
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # index.html
        end()
        output = wget_output_path(link, look_in=domain_dir)

+        output_tail = ['          ' + line for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] if line.strip()]
+
+        # parse out number of files downloaded from "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
+        files_downloaded = (
+            int(output_tail[-1].strip().split(' ', 2)[1] or 0)
+            if 'Downloaded:' in output_tail[-1]
+            else 0
+        )
+
        # Check for common failure cases
-        if result.returncode > 0:
-            print('        got wget response code {}:'.format(result.returncode))
-            if result.returncode != 8:
-                print('\n'.join('          ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
+        if result.returncode > 0 and files_downloaded < 1:
+            print('        Got wget response code {}:'.format(result.returncode))
+            print('\n'.join(output_tail))
            if b'403: Forbidden' in result.stderr:
                raise Exception('403 Forbidden (try changing WGET_USER_AGENT)')
            if b'404: Not Found' in result.stderr:
                raise Exception('404 Not Found')
            if b'ERROR 500: Internal Server Error' in result.stderr:
                raise Exception('500 Internal Server Error')
-            if result.returncode == 4:
-                raise Exception('Failed wget download')
+            raise Exception('Got an error from the server')
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e

+        # to let the user copy-paste the command and run it safely we have
+        # to quote some of the arguments that could have spaces in them
+        quoted_cmd = ' '.join(CMD)
+        quoted_cmd = quoted_cmd.replace(WGET_USER_AGENT, '"{}"'.format(WGET_USER_AGENT))
+        if COOKIES_FILE:
+            quoted_cmd = quoted_cmd.replace(COOKIES_FILE, '"{}"'.format(COOKIES_FILE))
+
+        print('        {}Some resources were skipped: {}{}'.format(ANSI['lightyellow'], e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(quoted_cmd))
+        output = e
    return {
        'cmd': CMD,
        'output': output,
@@ -245,11 +274,14 @@ def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
    CMD = [
        *chrome_headless(user_data_dir=user_data_dir),
        '--print-to-pdf',
+        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
        link['url']
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.pdf
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.pdf
        end()
        if result.returncode:
            print('     ', (result.stderr or result.stdout).decode())
@@ -258,8 +290,10 @@ def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
        output = 'output.pdf'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -282,12 +316,14 @@ def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_
        '--screenshot',
        '--window-size={}'.format(resolution),
        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
        # '--full-page',   # TODO: make this actually work using ./bin/screenshot fullPage: true
        link['url'],
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # sreenshot.png
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # sreenshot.png
        end()
        if result.returncode:
            print('     ', (result.stderr or result.stdout).decode())
@@ -296,8 +332,10 @@ def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_
        output = 'screenshot.png'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -320,12 +358,13 @@ def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
    CMD = [
        *chrome_headless(user_data_dir=user_data_dir),
        '--dump-dom',
+        '--timeout={}'.format((timeout) * 1000),
        link['url']
    ]
    end = progress(timeout, prefix='      ')
    try:
        with open(output_path, 'w+') as f:
-            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.html
+            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.html
        end()
        if result.returncode:
            print('     ', (result.stderr).decode())
@@ -334,8 +373,10 @@ def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
        output = 'output.html'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -355,10 +396,19 @@ def archive_dot_org(link_dir, link, timeout=TIMEOUT):
    submit_url = 'https://web.archive.org/save/{}'.format(link['url'])

    success = False
-    CMD = ['curl', '-L', '-I', '-X', 'GET', submit_url]
+    CMD = [
+        CURL_BINARY,
+        '--location',
+        '--head',
+        '--user-agent', 'ArchiveBox/{} (+https://github.com/pirate/ArchiveBox/)'.format(GIT_SHA),
+        '--max-time', str(timeout),
+        '--get',
+        *(() if CHECK_SSL_VALIDITY else ('--insecure',)),
+        submit_url,
+    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # archive.org.txt
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # archive.org.txt
        end()

        # Parse archive.org response headers
@@ -387,8 +437,9 @@ def archive_dot_org(link_dir, link, timeout=TIMEOUT):
            raise Exception('Failed to find "content-location" URL header in Archive.org response.')
    except Exception as e:
        end()
-        print('        Visit url to see output:', ' '.join(CMD))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
        output = e

    if success:
@@ -409,11 +460,16 @@ def fetch_favicon(link_dir, link, timeout=TIMEOUT):
    if os.path.exists(os.path.join(link_dir, 'favicon.ico')):
        return {'output': 'favicon.ico', 'status': 'skipped'}

-    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
+    CMD = [
+        CURL_BINARY,
+        '--max-time', str(timeout),
+        *(() if CHECK_SSL_VALIDITY else ('--insecure',)),
+        'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
+    ]
    fout = open('{}/favicon.ico'.format(link_dir), 'w')
    end = progress(timeout, prefix='      ')
    try:
-        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # favicon.ico
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # favicon.ico
        fout.close()
        end()
        chmod_file('favicon.ico', cwd=link_dir)
@@ -421,8 +477,9 @@ def fetch_favicon(link_dir, link, timeout=TIMEOUT):
    except Exception as e:
        fout.close()
        end()
-        print('        Run to see full output:', ' '.join(CMD))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -430,78 +487,151 @@ def fetch_favicon(link_dir, link, timeout=TIMEOUT):
        'output': output,
    }

-# @attach_result_to_link('audio')
-# def fetch_audio(link_dir, link, timeout=TIMEOUT):
-#     """Download audio rip using youtube-dl"""
+@attach_result_to_link('title')
+def fetch_title(link_dir, link, timeout=TIMEOUT):
+    """try to guess the page's title from its content"""

-#     if link['type'] not in ('soundcloud',)\
-#        and 'audio' not in link['tags']:
-#         return
+    # if link already has valid title, skip it
+    if link['title'] and not link['title'].lower().startswith('http'):
+        return {'output': link['title'], 'status': 'skipped'}

-#     path = os.path.join(link_dir, 'audio')
+    end = progress(timeout, prefix='      ')
+    try:
+        title = fetch_page_title(link['url'], timeout=timeout, progress=False)
+        end()
+        output = title
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e

-#     if not os.path.exists(path) or overwrite:
-#         print('    - Downloading audio')
-#         CMD = [
-#             "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
-#             link['url'],
-#         ]
-#         end = progress(timeout, prefix='      ')
-#         try:
-#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # audio/audio.mp3
-#             end()
-#             if result.returncode:
-#                 print('     ', result.stderr.decode())
-#                 raise Exception('Failed to download audio')
-#             chmod_file('audio.mp3', cwd=link_dir)
-#             return 'audio.mp3'
-#         except Exception as e:
-#             end()
-#             print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-#             print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-#             raise
-#     else:
-#         print('    √ Skipping audio download')
+    # titles should show up in the global index immediatley for better UX,
+    # do a hacky immediate replacement to add them in as we're archiving
+    # TODO: figure out how to do this without gnarly string replacement
+    if title:
+        link['title'] = title
+        patch_index_title_hack(link['url'], title)

-# @attach_result_to_link('video')
-# def fetch_video(link_dir, link, timeout=TIMEOUT):
-#     """Download video rip using youtube-dl"""
+    return {
+        'cmd': 'fetch_page_title("{}")'.format(link['url']),
+        'output': output,
+    }

-#     if link['type'] not in ('youtube', 'youku', 'vimeo')\
-#        and 'video' not in link['tags']:
-#         return
+@attach_result_to_link('media')
+def fetch_media(link_dir, link, timeout=MEDIA_TIMEOUT, overwrite=False):
+    """Download playlists or individual video, audio, and subtitles using youtube-dl"""

-#     path = os.path.join(link_dir, 'video')

-#     if not os.path.exists(path) or overwrite:
-#         print('    - Downloading video')
-#         CMD = [
-#             "youtube-dl -x --video-format mp4 --audio-quality 0 -o '%(title)s.%(ext)s'",
-#             link['url'],
-#         ]
-#         end = progress(timeout, prefix='      ')
-#         try:
-#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # video/movie.mp4
-#             end()
-#             if result.returncode:
-#                 print('     ', result.stderr.decode())
-#                 raise Exception('Failed to download video')
-#             chmod_file('video.mp4', cwd=link_dir)
-#             return 'video.mp4'
-#         except Exception as e:
-#             end()
-#             print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-#             print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-#             raise
-#     else:
-#         print('    √ Skipping video download')
+    # import ipdb; ipdb.set_trace()
+    output = os.path.join(link_dir, 'media')
+    already_done = os.path.exists(output)  # and os.listdir(output)
+    if already_done and not overwrite:
+        return {'output': 'media', 'status': 'skipped'}

+    os.makedirs(output, exist_ok=True)
+    CMD = [
+        YOUTUBEDL_BINARY,
+        '--write-description',
+        '--write-info-json',
+        '--write-annotations',
+        '--yes-playlist',
+        '--write-thumbnail',
+        '--no-call-home',
+        '--no-check-certificate',
+        '--user-agent',
+        '--all-subs',
+        '--extract-audio',
+        '--keep-video',
+        '--ignore-errors',
+        '--geo-bypass',
+        '--audio-format', 'mp3',
+        '--audio-quality', '320K',
+        '--embed-thumbnail',
+        '--add-metadata',
+        *(() if CHECK_SSL_VALIDITY else ('--no-check-certificate',)),
+        link['url'],
+    ]
+
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=output, timeout=timeout + 1)  # audio/audio.mp3
+        chmod_file('media', cwd=link_dir)
+        output = 'media'
+        end()
+        if result.returncode:
+            if (b'ERROR: Unsupported URL' in result.stderr
+                or b'HTTP Error 404' in result.stderr
+                or b'HTTP Error 403' in result.stderr
+                or b'URL could be a direct video link' in result.stderr
+                or b'Unable to extract container ID' in result.stderr):
+                # These happen too frequently on non-media pages to warrant printing to console
+                pass
+            else:
+                print('        got youtubedl response code {}:'.format(result.returncode))
+                print(result.stderr)
+                raise Exception('Failed to download media')
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('git')
+def fetch_git(link_dir, link, timeout=TIMEOUT):
+    """download full site using git"""
+
+    if not (link['domain'] in GIT_DOMAINS
+            or link['url'].endswith('.git')
+            or link['type'] == 'git'):
+        return
+
+    if os.path.exists(os.path.join(link_dir, 'git')):
+        return {'output': 'git', 'status': 'skipped'}
+
+    CMD = [
+        GIT_BINARY,
+        *(() if CHECK_SSL_VALIDITY else ('-c', 'http.sslVerify=false')),
+        'clone',
+        '--mirror',
+        '--recursive',
+        without_fragment(link['url']),
+    ]
+    output = 'git'
+
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # git/<reponame>
+        end()
+
+        if result.returncode > 0:
+            print('        got git response code {}:'.format(result.returncode))
+            raise Exception('Failed git download')
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }

 def chrome_headless(binary=CHROME_BINARY, user_data_dir=CHROME_USER_DATA_DIR):
    args = [binary, '--headless']  # '--disable-gpu'
    if not CHROME_SANDBOX:
        args.append('--no-sandbox')
-    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
+    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome')
    if user_data_dir:
        args.append('--user-data-dir={}'.format(user_data_dir))
    elif os.path.exists(default_profile):
@@ -14,33 +14,44 @@ IS_TTY = sys.stdout.isatty()
 USE_COLOR =              os.getenv('USE_COLOR',              str(IS_TTY)        ).lower() == 'true'
 SHOW_PROGRESS =          os.getenv('SHOW_PROGRESS',          str(IS_TTY)        ).lower() == 'true'
 ONLY_NEW =               os.getenv('ONLY_NEW',               'False'            ).lower() == 'true'
+MEDIA_TIMEOUT =          int(os.getenv('MEDIA_TIMEOUT',      '3600'))
+TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
+OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
+FOOTER_INFO =            os.getenv('FOOTER_INFO',            'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.',)
+
 FETCH_WGET =             os.getenv('FETCH_WGET',             'True'             ).lower() == 'true'
 FETCH_WGET_REQUISITES =  os.getenv('FETCH_WGET_REQUISITES',  'True'             ).lower() == 'true'
-FETCH_AUDIO =            os.getenv('FETCH_AUDIO',            'False'            ).lower() == 'true'
-FETCH_VIDEO =            os.getenv('FETCH_VIDEO',            'False'            ).lower() == 'true'
 FETCH_PDF =              os.getenv('FETCH_PDF',              'True'             ).lower() == 'true'
 FETCH_SCREENSHOT =       os.getenv('FETCH_SCREENSHOT',       'True'             ).lower() == 'true'
 FETCH_DOM =              os.getenv('FETCH_DOM',              'True'             ).lower() == 'true'
+FETCH_WARC =             os.getenv('FETCH_WARC',             'True'             ).lower() == 'true'
+FETCH_GIT =              os.getenv('FETCH_GIT',              'True'             ).lower() == 'true'
+FETCH_MEDIA =            os.getenv('FETCH_MEDIA',            'True'             ).lower() == 'true'
 FETCH_FAVICON =          os.getenv('FETCH_FAVICON',          'True'             ).lower() == 'true'
+FETCH_TITLE =            os.getenv('FETCH_TITLE',            'True'             ).lower() == 'true'
 SUBMIT_ARCHIVE_DOT_ORG = os.getenv('SUBMIT_ARCHIVE_DOT_ORG', 'True'             ).lower() == 'true'
-RESOLUTION =             os.getenv('RESOLUTION',             '1440,1200'        )
+
 CHECK_SSL_VALIDITY =     os.getenv('CHECK_SSL_VALIDITY',     'True'             ).lower() == 'true'
-OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
-CHROME_BINARY =          os.getenv('CHROME_BINARY',          'chromium-browser' )  # change to google-chrome browser if using google-chrome
-WGET_BINARY =            os.getenv('WGET_BINARY',            'wget'             )
-WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',        'ArchiveBox')
-CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',    None)
-TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
-FOOTER_INFO =            os.getenv('FOOTER_INFO',            'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.',)
+RESOLUTION =             os.getenv('RESOLUTION',             '1440,2000'        )
+GIT_DOMAINS =            os.getenv('GIT_DOMAINS',            'github.com,bitbucket.org,gitlab.com').split(',')
+WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',        'ArchiveBox/{GIT_SHA} (+https://github.com/pirate/ArchiveBox/) wget/{WGET_VERSION}')
+COOKIES_FILE =           os.getenv('COOKIES_FILE',           None)
+CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',   None)
+
+CURL_BINARY =            os.getenv('CURL_BINARY',            'curl')
+GIT_BINARY =             os.getenv('GIT_BINARY',             'git')
+WGET_BINARY =            os.getenv('WGET_BINARY',            'wget')
+YOUTUBEDL_BINARY =       os.getenv('YOUTUBEDL_BINARY',       'youtube-dl')
+CHROME_BINARY =          os.getenv('CHROME_BINARY',          None)

 ### Paths
 REPO_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))

-OUTPUT_DIR = os.getenv('OUTPUT_DIR', os.path.join(REPO_DIR, 'output'))
+OUTPUT_DIR = os.path.abspath(os.getenv('OUTPUT_DIR', os.path.join(REPO_DIR, 'output')))
 ARCHIVE_DIR = os.path.join(OUTPUT_DIR, 'archive')
 SOURCES_DIR = os.path.join(OUTPUT_DIR, 'sources')

-PYTHON_PATH = os.path.join(REPO_DIR, 'archiver')
+PYTHON_PATH = os.path.join(REPO_DIR, 'archivebox')
 TEMPLATES_DIR = os.path.join(PYTHON_PATH, 'templates')

 # ******************************************************************************
@@ -49,6 +60,30 @@ TEMPLATES_DIR = os.path.join(PYTHON_PATH, 'templates')

 CHROME_SANDBOX =        os.getenv('CHROME_SANDBOX',         'True'             ).lower() == 'true'

+USE_CHROME = FETCH_PDF or FETCH_SCREENSHOT or FETCH_DOM
+USE_WGET = FETCH_WGET or FETCH_WGET_REQUISITES or FETCH_WARC
+
+if not CHROME_BINARY:
+    common_chrome_executable_names = (
+        'chromium-browser',
+        'chromium',
+        'google-chrome',
+        'google-chrome-stable',
+        'google-chrome-unstable',
+        'google-chrome-beta',
+        'google-chrome-canary',
+        'google-chrome-dev',
+    )
+    for name in common_chrome_executable_names:
+        full_path_exists = shutil.which(name)
+        if full_path_exists:
+            CHROME_BINARY = name
+            break
+    else:
+        CHROME_BINARY = 'chromium-browser'
+
+# print('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY))
+
 ### Terminal Configuration
 TERM_WIDTH = shutil.get_terminal_size((100, 10)).columns
 ANSI = {
@@ -67,13 +102,37 @@ if not USE_COLOR:
    ANSI = {k: '' for k in ANSI.keys()}

 ### Confirm Environment Setup
+GIT_SHA = 'unknown'
 try:
-    GIT_SHA = run(["git", "rev-list", "-1", "HEAD", "./"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    GIT_SHA = run([GIT_BINARY, 'rev-list', '-1', 'HEAD', './'], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
 except Exception:
-    GIT_SHA = 'unknown'
-    print('[!] Warning, you need git installed for some archiving features to save correct version numbers!')
+    print('[!] Warning: unable to determine git version, is git installed and in your $PATH?')

-if sys.stdout.encoding.upper() != 'UTF-8':
+CHROME_VERSION = 'unknown'
+try:
+    chrome_vers_str = run([CHROME_BINARY, "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    CHROME_VERSION = [v for v in chrome_vers_str.strip().split(' ') if v.replace('.', '').isdigit()][0]
+except Exception:
+    if USE_CHROME:
+        print('[!] Warning: unable to determine chrome version, is chrome installed and in your $PATH?')
+
+WGET_VERSION = 'unknown'
+try:
+    wget_vers_str = run([WGET_BINARY, "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    WGET_VERSION = wget_vers_str.split('\n')[0].split(' ')[2]
+except Exception:
+    if USE_WGET:
+        print('[!] Warning: unable to determine wget version, is wget installed and in your $PATH?')
+
+WGET_USER_AGENT = WGET_USER_AGENT.format(GIT_SHA=GIT_SHA[:9], WGET_VERSION=WGET_VERSION)
+
+try:
+    COOKIES_FILE = os.path.abspath(COOKIES_FILE) if COOKIES_FILE else None
+except Exception:
+    print('[!] Warning: unable to get full path to COOKIES_FILE, are you sure you specified it correctly?')
+    raise
+
+if sys.stdout.encoding.upper() not in ('UTF-8', 'UTF8'):
    print('[X] Your system is running python3 scripts with a bad locale setting: {} (it should be UTF-8).'.format(sys.stdout.encoding))
    print('    To fix it, add the line "export PYTHONIOENCODING=UTF-8" to your ~/.bashrc file (without quotes)')
    print('')
@@ -82,3 +141,4 @@ if sys.stdout.encoding.upper() != 'UTF-8':
    print('')
    print('    Alternatively, run this script with:')
    print('        env PYTHONIOENCODING=UTF-8 ./archive.py export.html')
+
@@ -6,6 +6,7 @@ from string import Template
 from distutils.dir_util import copy_tree

 from config import (
+    OUTPUT_DIR,
    TEMPLATES_DIR,
    OUTPUT_PERMISSIONS,
    ANSI,
@@ -17,6 +18,8 @@ from util import (
    wget_output_path,
    derived_link_info,
    pretty_path,
+    check_link_structure,
+    check_links_structure,
 )


@@ -25,21 +28,27 @@ from util import (
 def write_links_index(out_dir, links):
    """create index.html file for a given list of links"""

+    check_links_structure(links)
+
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)

-    write_json_links_index(out_dir, links)
-    write_html_links_index(out_dir, links)
-    
-    print('{green}[√] [{}] Updated main index files:{reset}'.format(
+    print('{green}[*] [{}] Updating main index files...{reset}'.format(
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        **ANSI))
+        **ANSI,
+    ))
+    write_json_links_index(out_dir, links)
    print('    > {}/index.json'.format(pretty_path(out_dir)))
+    
+    write_html_links_index(out_dir, links)
    print('    > {}/index.html'.format(pretty_path(out_dir)))
+    

 def write_json_links_index(out_dir, links):
    """write the json link index to a given path"""

+    check_links_structure(links)
+
    path = os.path.join(out_dir, 'index.json')

    index_json = {
@@ -61,13 +70,17 @@ def parse_json_links_index(out_dir):
    index_path = os.path.join(out_dir, 'index.json')
    if os.path.exists(index_path):
        with open(index_path, 'r', encoding='utf-8') as f:
-            return json.load(f)['links']
+            links = json.load(f)['links']
+            check_links_structure(links)
+            return links

    return []

 def write_html_links_index(out_dir, links):
    """write the html link index to a given path"""

+    check_links_structure(links)
+
    path = os.path.join(out_dir, 'index.html')

    copy_tree(os.path.join(TEMPLATES_DIR, 'static'), os.path.join(out_dir, 'static'))
@@ -102,6 +115,25 @@ def write_html_links_index(out_dir, links):
    chmod_file(path)


+def patch_index_title_hack(link_url, new_title):
+    """hack to update just one link's title in the link index json"""
+
+    json_path = os.path.join(OUTPUT_DIR, 'index.json')
+
+    links = parse_json_links_index(OUTPUT_DIR)
+
+    changed = False
+    for link in links:
+        if link['url'] == link_url:
+            link['title'] = new_title
+            changed = True
+            break
+
+    if changed:
+        write_json_links_index(OUTPUT_DIR, links)
+
+
+
 ### Individual link index

 def write_link_index(out_dir, link):
@@ -112,6 +144,7 @@ def write_link_index(out_dir, link):
 def write_json_link_index(out_dir, link):
    """write a json file with some info about the link"""
    
+    check_link_structure(link)
    path = os.path.join(out_dir, 'index.json')

    print('      √ index.json')
@@ -126,10 +159,13 @@ def parse_json_link_index(out_dir):
    existing_index = os.path.join(out_dir, 'index.json')
    if os.path.exists(existing_index):
        with open(existing_index, 'r', encoding='utf-8') as f:
-            return json.load(f)
+            link_json = json.load(f)
+            check_link_structure(link_json)
+            return link_json
    return {}

 def write_html_link_index(out_dir, link):
+    check_link_structure(link)
    with open(os.path.join(TEMPLATES_DIR, 'link_index_fancy.html'), 'r', encoding='utf-8') as f:
        link_html = f.read()

@@ -141,6 +177,7 @@ def write_html_link_index(out_dir, link):
        f.write(Template(link_html).substitute({
            **link,
            **link['latest'],
+            'title': link['title'] or link['url'],
            'type': link['type'] or 'website',
            'tags': link['tags'] or 'untagged',
            'bookmarked': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
@@ -32,34 +32,33 @@ Link {

 """

-import datetime
 from html import unescape
 from collections import OrderedDict

 from util import (
-    domain,
-    base_url,
-    str_between,
-    get_link_type,
    merge_links,
    wget_output_path,
+    check_link_structure,
+    check_links_structure,
 )
-from config import ANSI


 def validate_links(links):
+    check_links_structure(links)
    links = archivable_links(links)  # remove chrome://, about:, mailto: etc.
    links = uniquefied_links(links)  # merge/dedupe duplicate timestamps & urls
    links = sorted_links(links)      # deterministically sort the links based on timstamp, url
-    
+
    if not links:
        print('[X] No links found :(')
        raise SystemExit(1)

    for link in links:
-        link['title'] = unescape(link['title'])
+        check_link_structure(link)
+
+        link['title'] = unescape(link['title']) if link['title'] else None
        link['latest'] = link.get('latest') or {}
-        
+
        latest = link['latest']
        if not link['latest'].get('wget'):
            link['latest']['wget'] = wget_output_path(link)
@@ -76,25 +75,21 @@ def validate_links(links):
        if not latest.get('favicon'):
            latest['favicon'] = None

+        if not link['latest'].get('title'):
+            link['latest']['title'] = link['title']
+
    return list(links)

-def new_links(all_links, existing_links):
-    """
-    Return all links which are in the all_links but not in the existing_links.
-    This is used to determine which links are new and not indexed jet. Set the
-    ONLY_NEW environment variable to activate this filter mechanism.
-    """
-    existing_urls = {link['url'] for link in existing_links}
-    return [link for link in all_links if link['url'] not in existing_urls]

 def archivable_links(links):
    """remove chrome://, about:// or other schemed links that cant be archived"""
    return (
        link
        for link in links
-        if any(link['url'].startswith(s) for s in ('http://', 'https://', 'ftp://'))
+        if any(link['url'].lower().startswith(s) for s in ('http://', 'https://', 'ftp://'))
    )

+
 def uniquefied_links(sorted_links):
    """
    ensures that all non-duplicate links have monotonically increasing timestamps
@@ -120,10 +115,12 @@ def uniquefied_links(sorted_links):

    return unique_timestamps.values()

+
 def sorted_links(links):
    sort_func = lambda link: (link['timestamp'].split('.', 1)[0], link['url'])
    return sorted(links, key=sort_func, reverse=True)

+
 def links_after_timestamp(links, timestamp=None):
    if not timestamp:
        yield from links
@@ -136,6 +133,7 @@ def links_after_timestamp(links, timestamp=None):
        except (ValueError, TypeError):
            print('Resume value and all timestamp values must be valid numbers.')

+
 def lowest_uniq_timestamp(used_timestamps, timestamp):
    """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2"""

@@ -8,60 +8,58 @@ For examples of supported files see examples/.

 Parsed link schema: {
    'url': 'https://example.com/example/?abc=123&xyc=345#lmnop',
-    'domain': 'example.com',
-    'base_url': 'example.com/example/',
    'timestamp': '15442123124234',
-    'tags': 'abc,def',
    'title': 'Example.com Page Title',
+    'tags': 'abc,def',
    'sources': ['ril_export.html', 'downloads/getpocket.com.txt'],
 }
 """

 import re
 import json
-import xml.etree.ElementTree as etree

 from datetime import datetime
+from collections import OrderedDict
+import xml.etree.ElementTree as etree

+from config import ANSI
 from util import (
-    domain,
-    base_url,
    str_between,
    get_link_type,
+    URL_REGEX,
+    check_url_parsing,
 )


-def get_parsers(file):
-    """return all parsers that work on a given file, defaults to all of them"""
-
-    return {
-        'pocket': parse_pocket_export,
-        'pinboard': parse_json_export,
-        'bookmarks': parse_bookmarks_export,
-        'rss': parse_rss_export,
-        'pinboard_rss': parse_pinboard_rss_feed,
-        'medium_rss': parse_medium_rss_feed,
-    }
-
 def parse_links(path):
    """parse a list of links dictionaries from a bookmark export file"""
    
+    check_url_parsing()
+
    links = []
    with open(path, 'r', encoding='utf-8') as file:
-        for parser_func in get_parsers(file).values():
-            # otherwise try all parsers until one works
+        print('{green}[*] [{}] Parsing new links from output/sources/{}...{reset}'.format(
+            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            path.rsplit('/', 1)[-1],
+            **ANSI,
+        ))
+
+        for parser_name, parser_func in PARSERS.items():
            try:
                links += list(parser_func(file))
                if links:
                    break
-            except (ValueError, TypeError, IndexError, AttributeError, etree.ParseError):
-                # parser not supported on this file
+            except Exception as err:
+                # we try each parser one by one, wong parsers will throw exeptions
+                # if unsupported and we accept the first one that passes
+                # uncomment the following line to see why the parser was unsupported for each attempted format
+                # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err))
                pass

-    return links
+    return links, parser_name


-def parse_pocket_export(html_file):
+def parse_pocket_html_export(html_file):
    """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""

    html_file.seek(0)
@@ -75,17 +73,15 @@ def parse_pocket_export(html_file):
            time = datetime.fromtimestamp(float(match.group(2)))
            info = {
                'url': fixed_url,
-                'domain': domain(fixed_url),
-                'base_url': base_url(fixed_url),
                'timestamp': str(time.timestamp()),
                'tags': match.group(3),
-                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or base_url(fixed_url),
+                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or None,
                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

-def parse_json_export(json_file):
+def parse_pinboard_json_export(json_file):
    """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)"""
    json_file.seek(0)
    json_content = json.load(json_file)
@@ -109,19 +105,18 @@ def parse_json_export(json_file):
            if erg.get('description'):
                title = (erg.get('description') or '').replace(' — Readability', '')
            else:
-                title = erg['title']
+                title = erg['title'].strip()
            info = {
                'url': url,
-                'domain': domain(url),
-                'base_url': base_url(url),
                'timestamp': timestamp,
                'tags': erg.get('tags') or '',
-                'title': title,
+                'title': title or None,
                'sources': [json_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

+
 def parse_rss_export(rss_file):
    """Parse RSS XML-format files into links"""

@@ -142,27 +137,65 @@ def parse_rss_export(rss_file):
        rows = leading_removed.split('\n')

        def get_row(key):
-            return [r for r in rows if r.startswith('<{}>'.format(key))][0]
+            return [r for r in rows if r.strip().startswith('<{}>'.format(key))][0]

-        title = str_between(get_row('title'), '<![CDATA[', ']]')
+        title = str_between(get_row('title'), '<![CDATA[', ']]').strip()
        url = str_between(get_row('link'), '<link>', '</link>')
        ts_str = str_between(get_row('pubDate'), '<pubDate>', '</pubDate>')
        time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z")

        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
            'tags': '',
-            'title': title,
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)

        yield info

-def parse_bookmarks_export(html_file):
+
+def parse_shaarli_rss_export(rss_file):
+    """Parse Shaarli-specific RSS XML-format files into links"""
+
+    rss_file.seek(0)
+    entries = rss_file.read().split('<entry>')[1:]
+    for entry in entries:
+        # example entry:
+        # <entry>
+        #   <title>Aktuelle Trojaner-Welle: Emotet lauert in gefÃ¤lschten Rechnungsmails | heise online</title>
+        #   <link href="https://www.heise.de/security/meldung/Aktuelle-Trojaner-Welle-Emotet-lauert-in-gefaelschten-Rechnungsmails-4291268.html" />
+        #   <id>https://demo.shaarli.org/?cEV4vw</id>
+        #   <published>2019-01-30T06:06:01+00:00</published>
+        #   <updated>2019-01-30T06:06:01+00:00</updated>
+        #   <content type="html" xml:lang="en"><![CDATA[<div class="markdown"><p>&#8212; <a href="https://demo.shaarli.org/?cEV4vw">Permalink</a></p></div>]]></content>
+        # </entry>
+
+        trailing_removed = entry.split('</entry>', 1)[0]
+        leading_removed = trailing_removed.strip()
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0]
+
+        title = str_between(get_row('title'), '<title>', '</title>').strip()
+        url = str_between(get_row('link'), '<link href="', '" />')
+        ts_str = str_between(get_row('published'), '<published>', '</published>')
+        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
+
+        info = {
+            'url': url,
+            'timestamp': str(time.timestamp()),
+            'tags': '',
+            'title': title or None,
+            'sources': [rss_file.name],
+        }
+        info['type'] = get_link_type(info)
+
+        yield info
+
+def parse_netscape_html_export(html_file):
    """Parse netscape-format bookmarks export files (produced by all browsers)"""

    html_file.seek(0)
@@ -178,18 +211,16 @@ def parse_bookmarks_export(html_file):

            info = {
                'url': url,
-                'domain': domain(url),
-                'base_url': base_url(url),
                'timestamp': str(time.timestamp()),
                'tags': "",
-                'title': match.group(3),
+                'title': match.group(3).strip() or None,
                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)

            yield info

-def parse_pinboard_rss_feed(rss_file):
+def parse_pinboard_rss_export(rss_file):
    """Parse Pinboard RSS feed files into links"""

    rss_file.seek(0)
@@ -198,7 +229,7 @@ def parse_pinboard_rss_feed(rss_file):
    for item in items:
        url = item.find("{http://purl.org/rss/1.0/}link").text
        tags = item.find("{http://purl.org/dc/elements/1.1/}subject").text
-        title = item.find("{http://purl.org/rss/1.0/}title").text
+        title = item.find("{http://purl.org/rss/1.0/}title").text.strip()
        ts_str = item.find("{http://purl.org/dc/elements/1.1/}date").text
        #       = 🌈🌈🌈🌈
        #        = 🌈🌈🌈🌈
@@ -211,17 +242,15 @@ def parse_pinboard_rss_feed(rss_file):
        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
            'tags': tags,
-            'title': title,
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)
        yield info

-def parse_medium_rss_feed(rss_file):
+def parse_medium_rss_export(rss_file):
    """Parse Medium RSS feed files into links"""

    rss_file.seek(0)
@@ -231,17 +260,49 @@ def parse_medium_rss_feed(rss_file):
        # for child in item:
        #     print(child.tag, child.text)
        url = item.find("link").text
-        title = item.find("title").text
+        title = item.find("title").text.strip()
        ts_str = item.find("pubDate").text
        time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z")
        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
-            'tags': "",
-            'title': title,
+            'tags': '',
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)
        yield info
+
+
+def parse_plain_text_export(text_file):
+    """Parse raw links from each line in a text file"""
+
+    text_file.seek(0)
+    text_content = text_file.readlines()
+    for line in text_content:
+        if line:
+            urls = re.findall(URL_REGEX, line)
+            
+            for url in urls:
+                url = url.strip()
+                info = {
+                    'url': url,
+                    'timestamp': str(datetime.now().timestamp()),
+                    'tags': '',
+                    'title': None,
+                    'sources': [text_file.name],
+                }
+                info['type'] = get_link_type(info)
+                yield info
+
+
+PARSERS = OrderedDict([
+    ('Pocket HTML', parse_pocket_html_export),
+    ('Pinboard JSON', parse_pinboard_json_export),
+    ('Netscape HTML', parse_netscape_html_export),
+    ('RSS', parse_rss_export),
+    ('Pinboard RSS', parse_pinboard_rss_export),
+    ('Shaarli RSS', parse_shaarli_rss_export),
+    ('Medium RSS', parse_medium_rss_export),
+    ('Plain Text', parse_plain_text_export),
+])
@@ -1,3 +1,5 @@
+from sys import maxsize
+from itertools import islice
 from collections import deque

 _marker = object()
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+
+import re
+from argparse import ArgumentParser
+from os.path import exists, join
+from shutil import rmtree
+from typing import List
+
+from archive import parse_json_link_index
+from config import ARCHIVE_DIR, OUTPUT_DIR
+from index import write_html_links_index, write_json_links_index
+
+
+def cleanup_index(regexes: List[str], proceed: bool, delete: bool) -> None:
+    if not exists(join(OUTPUT_DIR, 'index.json')):
+        exit('index.json is missing; nothing to do')
+
+    compiled = [re.compile(r) for r in regexes]
+    links = parse_json_link_index(OUTPUT_DIR)['links']
+    filtered = []
+    remaining = []
+
+    for l in links:
+        url = l['url']
+        for r in compiled:
+            if r.search(url):
+                filtered.append((l, r))
+                break
+        else:
+            remaining.append(l)
+
+    if not filtered:
+        exit('Search did not match any entries.')
+
+    print('Filtered out {}/{} urls:'.format(len(filtered), len(links)))
+
+    for link, regex in filtered:
+        url = link['url']
+        print(' {url} via {regex}'.format(url=url, regex=regex.pattern))
+
+    if not proceed:
+        answer = input('Remove {} entries from index? [y/n] '.format(
+            len(filtered)))
+        proceed = answer.strip().lower() in ('y', 'yes')
+
+    if not proceed:
+        exit('Aborted')
+
+    write_json_links_index(OUTPUT_DIR, remaining)
+    write_html_links_index(OUTPUT_DIR, remaining)
+
+    if delete:
+        for link, _ in filtered:
+            data_dir = join(ARCHIVE_DIR, link['timestamp'])
+            if exists(data_dir):
+                rmtree(data_dir)
+
+
+if __name__ == '__main__':
+    p = ArgumentParser('Index purging tool')
+    p.add_argument(
+        '--regex',
+        '-r',
+        action='append',
+        help='Regular expression matching URLs to purge',
+    )
+    p.add_argument(
+        '--delete',
+        '-d',
+        action='store_true',
+        default=False,
+        help='Delete webpage files from archive',
+    )
+    p.add_argument(
+        '--yes',
+        '-y',
+        action='store_true',
+        default=False,
+        help='Do not prompt for confirmation',
+    )
+
+    args = p.parse_args()
+    if args.regex:
+        cleanup_index(args.regex, proceed=args.yes, delete=args.delete)
+    else:
+        p.print_help()
@@ -15,49 +15,46 @@
            }
            header {
                background-color: #aa1e55;
-                color: white;
+                color: #1a1a1a;
                padding: 10px;
                padding-top: 0px;
                padding-bottom: 15px;
-                height: 100px;
+                /*height: 40px;*/
            }
            header h1 {
-                font-size: 38px;
+                margin: 7px 0px;
+                font-size: 35px;
                font-weight: 300;
-                color: black;
-                padding-top: 14px;
-                line-height: 1.4;
-                width: 100%;
+                color: #1a1a1a;
            }
-            header h1 small {
-                color: white;
-                font-size:0.45em;
-                margin-left: 10px;
-                display: block;
+            header h1 img {
+                height: 44px;
+                vertical-align: bottom;
            }
-            header h1 small a {
-                text-decoration: none;
-                color: orange;
-                opacity: 0.6;
-                font-weight: 300;
-            }
-            header h1 small a:hover {
-                opacity: 1;
-            }
-            header + div {
-                margin-top: 10px;
+            header a {
+                text-decoration: none !important;
+                color: #1a1a1a;
            }
            .header-center {
-                width: 100%;
+                margin: auto;
+                float: none;
                text-align: center;
+                padding-top: 6px;
+            }
+            .header-center small {
+                color: #eaeaea;
+                opacity: 0.7;
            }
            .header-left {
                float: left;
-                width: 50px;
-                height: 60px;
-                text-align: center;
-                padding: 20px;
-                margin-right: -70px;
+            }
+            .header-right {
+                float: right;
+                padding-top: 17px;
+                padding-right: 10px;
+            }
+            header + div {
+                margin-top: 10px;
            }
            #table-bookmarks_length, #table-bookmarks_filter {
                padding: 0px 15px;
@@ -105,23 +102,19 @@
    </head>
    <body>
        <header>
+            <div class="header-right">
+                <a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> &nbsp; | &nbsp; 
+                <a href="https://github.com/pirate/ArchiveBox">Source</a> &nbsp; | &nbsp; 
+                <a href="https://archivebox.io">Website</a>
+            </div>
            <div class="header-left">
                <a href="?" title="Reload...">
-                    <img src="static/archive.png" style="height: 100%;"/>
-                </a>
-                <br/>
-                <a href="https://github.com/pirate/ArchiveBox">
-                    Github
+                    <h1><img src="static/archive.png"/> ArchiveBox: Index</h1>
                </a>
            </div>
            <div class="header-center">
-                <h1>
-                    &nbsp;Archived Sites
-                    <br/>
-                    <small>
-                        <a href="?">Last updated $time_updated</a><br/>
-                    </small>
-                </h1>
+                Archived Sites<br/>
+                <small>Last updated $time_updated</small>
            </div>
        </header>
        <table id="table-bookmarks">
@@ -41,7 +41,7 @@
                margin-right: 13px;
                color: black;
                height: 53px;
-                margin-top: 7px;
+                margin-top: 12px;
                margin-left: 10px;
            }
            .nav-icon img:hover {
@@ -186,7 +186,9 @@
                <div class="col-lg-4 alert well">
                    Download:
                    <a href="index.json" title="JSON summary of archived link.">JSON</a> | 
-                    <a href="." title="Webserver-provided index of files directory.">Files</a>
+                    <a href="." title="Webserver-provided index of files directory.">Files</a> | 
+                    <a href="media/" title="Audio, Video, and Subtitle files.">Media</a> | 
+                    <a href="git/" title="Any git repos at the url">Git Code</a>
                </div>
                <hr/>
                <div class="col-lg-2">
@@ -0,0 +1,12 @@
+<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Pinboard Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL>
+<p>
+
+<DT><A HREF="https://github.com/trailofbits/algo" ADD_DATE="1542616733" PRIVATE="1" TOREAD="1" TAGS="vpn,scripts,toread">Algo VPN scripts</A>
+<DT><A HREF="http://www.ulisp.com/" ADD_DATE="1542374412" PRIVATE="1" TOREAD="1" TAGS="arduino,avr,embedded,lisp,toread">uLisp</A>
+
+</DL>
+</p>
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" xmlns:syn="http://purl.org/rss/1.0/modules/syndication/" xmlns:admin="http://webns.net/mvcb/">
+  <channel rdf:about="http://pinboard.in">
+    <title>Pinboard (private aaronmueller)</title>
+    <link>https://pinboard.in/u:aaronmueller/private/</link>
+    <description></description>
+    <items>
+      <rdf:Seq>
+        <rdf:li rdf:resource="https://mehkee.com/"/>
+        <rdf:li rdf:resource="https://qmk.fm/"/>
+      </rdf:Seq>
+    </items>
+  </channel>
+
+  <item rdf:about="https://mehkee.com/">
+    <title>Mehkee - Mechanical Keyboard Parts &amp; Accessories</title>
+    <dc:date>2018-11-08T21:29:32+00:00</dc:date>
+    <link>https://mehkee.com/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>keyboard gadget diy</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:gadget"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:diy"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+  <item rdf:about="https://qmk.fm/">
+    <title>QMK Firmware - An open source firmware for AVR and ARM based keyboards</title>
+    <dc:date>2018-11-06T22:36:21+00:00</dc:date>
+    <link>https://qmk.fm/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>firmware keyboard</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:firmware"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+</rdf:RDF>
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+    <posts user="aaronmueller">
+<post href="https://github.com/trailofbits/algo" time="2018-11-19T08:38:53Z" description="Algo VPN scripts" extended="" tag="vpn scripts" hash="18d708f67bb26d843b1cac4530bb52aa"  shared="no" toread="yes" />
+<post href="http://www.ulisp.com/" time="2018-11-16T13:20:12Z" description="uLisp" extended="" tag="arduino avr embedded lisp" hash="2a17ae95925a03a5b9bb38cf7f6c6f9b"  shared="no" toread="yes" />
+</posts>
@@ -0,0 +1,2 @@
+[{"href":"https:\/\/github.com\/trailofbits\/algo","description":"Algo VPN scripts","extended":"","meta":"62325ba3b577683aee854d7f191034dc","hash":"18d708f67bb26d843b1cac4530bb52aa","time":"2018-11-19T08:38:53Z","shared":"no","toread":"yes","tags":"vpn scripts"},
+{"href":"http:\/\/www.ulisp.com\/","description":"uLisp","extended":"","meta":"7bd0c0ef31f69d1459e3d37366e742b3","hash":"2a17ae95925a03a5b9bb38cf7f6c6f9b","time":"2018-11-16T13:20:12Z","shared":"no","toread":"yes","tags":"arduino avr embedded lisp"}]
@@ -9,7 +9,7 @@ from typing import List
 import pytest


-ARCHIVER_BIN = join(dirname(__file__), pardir, 'archive')
+ARCHIVER_BIN = join(dirname(__file__), pardir, 'archive.py')


 class Helper:
@@ -3,45 +3,75 @@ import re
 import sys
 import time
 import json
-import requests
+from urllib.request import Request, urlopen
+from urllib.parse import urlparse

-from datetime import datetime
-from subprocess import run, PIPE, DEVNULL
-from multiprocessing import Process
+from decimal import Decimal
 from urllib.parse import quote
+from datetime import datetime
+from subprocess import TimeoutExpired, Popen, PIPE, DEVNULL, CompletedProcess, CalledProcessError
+from multiprocessing import Process

 from config import (
-    IS_TTY,
-    OUTPUT_PERMISSIONS,
-    REPO_DIR,
-    SOURCES_DIR,
-    OUTPUT_DIR,
-    ARCHIVE_DIR,
-    TIMEOUT,
-    TERM_WIDTH,
-    SHOW_PROGRESS,
    ANSI,
+    IS_TTY,
+    TERM_WIDTH,
+    REPO_DIR,
+    OUTPUT_DIR,
+    SOURCES_DIR,
+    ARCHIVE_DIR,
+    OUTPUT_PERMISSIONS,
+    TIMEOUT,
+    SHOW_PROGRESS,
+    CHECK_SSL_VALIDITY,
+    WGET_USER_AGENT,
+    CURL_BINARY,
+    WGET_BINARY,
    CHROME_BINARY,
+    GIT_BINARY,
+    YOUTUBEDL_BINARY,
+    FETCH_TITLE,
+    FETCH_FAVICON,
    FETCH_WGET,
+    FETCH_WARC,
    FETCH_PDF,
    FETCH_SCREENSHOT,
    FETCH_DOM,
-    FETCH_FAVICON,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
+    FETCH_GIT,
+    FETCH_MEDIA,
    SUBMIT_ARCHIVE_DOT_ORG,
 )

-# URL helpers
-without_scheme = lambda url: url.replace('http://', '').replace('https://', '').replace('ftp://', '')
-without_query = lambda url: url.split('?', 1)[0]
-without_hash = lambda url: url.split('#', 1)[0]
-without_path = lambda url: url.split('/', 1)[0]
-domain = lambda url: without_hash(without_query(without_path(without_scheme(url))))
+# URL helpers: https://docs.python.org/3/library/urllib.parse.html#url-parsing
+scheme = lambda url: urlparse(url).scheme
+without_scheme = lambda url: urlparse(url)._replace(scheme='').geturl().strip('//')
+without_query = lambda url: urlparse(url)._replace(query='').geturl().strip('//')
+without_fragment = lambda url: urlparse(url)._replace(fragment='').geturl().strip('//')
+without_path = lambda url: urlparse(url)._replace(path='', fragment='', query='').geturl().strip('//')
+path = lambda url: urlparse(url).path
+basename = lambda url: urlparse(url).path.rsplit('/', 1)[-1]
+domain = lambda url: urlparse(url).netloc
+query = lambda url: urlparse(url).query
+fragment = lambda url: urlparse(url).fragment
+extension = lambda url: basename(url).rsplit('.', 1)[-1].lower() if '.' in basename(url) else ''
 base_url = lambda url: without_scheme(url)  # uniq base url used to dedupe links

 short_ts = lambda ts: ts.split('.')[0]

+URL_REGEX = re.compile(
+    r'http[s]?://'                    # start matching from allowed schemes
+    r'(?:[a-zA-Z]|[0-9]'              # followed by allowed alphanum characters
+    r'|[$-_@.&+]|[!*\(\),]'           #    or allowed symbols
+    r'|(?:%[0-9a-fA-F][0-9a-fA-F]))'  #    or allowed unicode bytes
+    r'[^\]\[\(\)<>\""\'\s]+',         # stop parsing at these symbols
+    re.IGNORECASE,
+)
+HTML_TITLE_REGEX = re.compile(
+    r'<title>'                         # start matching text after <title> tag
+    r'(.[^<>]+)',                      # get everything up to these symbols
+    re.IGNORECASE,
+)
+

 def check_dependencies():
    """Check that all necessary dependencies are installed, and have valid versions"""
@@ -52,6 +82,20 @@ def check_dependencies():
        print('    See https://github.com/pirate/ArchiveBox#troubleshooting for help upgrading your Python installation.')
        raise SystemExit(1)

+    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
+        if run(['which', CURL_BINARY], stdout=DEVNULL).returncode or run([CURL_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: curl{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CURL_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
+            raise SystemExit(1)
+
+    if FETCH_WGET or FETCH_WARC:
+        if run(['which', WGET_BINARY], stdout=DEVNULL).returncode or run([WGET_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: wget{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(WGET_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
+            raise SystemExit(1)
+
    if FETCH_PDF or FETCH_SCREENSHOT or FETCH_DOM:
        if run(['which', CHROME_BINARY], stdout=DEVNULL).returncode:
            print('{}[X] Missing dependency: {}{}'.format(ANSI['red'], CHROME_BINARY, ANSI['reset']))
@@ -76,28 +120,45 @@ def check_dependencies():
            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

-    if FETCH_WGET:
-        if run(['which', 'wget'], stdout=DEVNULL).returncode or run(['wget', '--version'], stdout=DEVNULL).returncode:
-            print('{red}[X] Missing dependency: wget{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('wget'))
+    if FETCH_GIT:
+        if run(['which', GIT_BINARY], stdout=DEVNULL).returncode or run([GIT_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: git{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(GIT_BINARY))
            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

-    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
-        if run(['which', 'curl'], stdout=DEVNULL).returncode or run(['curl', '--version'], stdout=DEVNULL).returncode:
-            print('{red}[X] Missing dependency: curl{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('curl'))
-            print('    See https://github.com/pirate/ArchiveBox for help.')
-            raise SystemExit(1)
-
-    if FETCH_AUDIO or FETCH_VIDEO:
-        if run(['which', 'youtube-dl'], stdout=DEVNULL).returncode or run(['youtube-dl', '--version'], stdout=DEVNULL).returncode:
+    if FETCH_MEDIA:
+        if run(['which', YOUTUBEDL_BINARY], stdout=DEVNULL).returncode or run([YOUTUBEDL_BINARY, '--version'], stdout=DEVNULL).returncode:
            print('{red}[X] Missing dependency: youtube-dl{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('youtube-dl'))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(YOUTUBEDL_BINARY))
            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)


+def check_url_parsing():
+    """Check that plain text regex URL parsing works as expected"""
+    test_urls = '''
+    https://example1.com/what/is/happening.html?what=1#how-about-this=1
+    https://example2.com/what/is/happening/?what=1#how-about-this=1
+    HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f
+    https://example4.com/what/is/happening.html
+    https://example5.com/
+    https://example6.com
+
+    <test>http://example7.com</test>
+    [https://example8.com/what/is/this.php?what=1]
+    [and http://example9.com?what=1&other=3#and-thing=2]
+    <what>https://example10.com#and-thing=2 "</about>
+    abc<this["https://example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def
+    sdflkf[what](https://example12.com/who/what.php?whoami=1#whatami=2)?am=hi
+    example13.bada
+    and example14.badb
+    <or>htt://example15.badc</that>
+    '''
+    # print('\n'.join(re.findall(URL_REGEX, test_urls)))
+    assert len(re.findall(URL_REGEX, test_urls)) == 12
+
+
 def chmod_file(path, cwd='.', permissions=OUTPUT_PERMISSIONS, timeout=30):
    """chmod -R <permissions> <cwd>/<path>"""

@@ -118,11 +179,10 @@ def progress(seconds=TIMEOUT, prefix=''):
    if not SHOW_PROGRESS:
        return lambda: None

-    chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
-    chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
-
-    def progress_bar(seconds=seconds, prefix=prefix):
+    def progress_bar(seconds, prefix):
        """show timer in the form of progress bar, with percentage and seconds remaining"""
+        chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
+        chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
        try:
            for s in range(seconds * chunks):
                progress = s / chunks / seconds * 100
@@ -156,16 +216,18 @@ def progress(seconds=TIMEOUT, prefix=''):
            print()
            pass

-    p = Process(target=progress_bar)
+    p = Process(target=progress_bar, args=(seconds, prefix))
    p.start()

    def end():
        """immediately finish progress and clear the progressbar line"""
-        nonlocal p
-        if p is None: # protect from double termination
-            return

-        p.terminate()
+        # protect from double termination
+        #if p is None or not hasattr(p, 'kill'):
+        #    return
+        nonlocal p
+        if p is not None:
+            p.terminate()
        p = None

        sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH), ANSI['reset']))  # clear whole terminal line
@@ -174,11 +236,39 @@ def progress(seconds=TIMEOUT, prefix=''):
    return end

 def pretty_path(path):
-    """convert paths like .../ArchiveBox/archiver/../output/abc into output/abc"""
+    """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc"""
    return path.replace(REPO_DIR + '/', '')


-def download_url(url):
+def save_stdin_source(raw_text):
+    if not os.path.exists(SOURCES_DIR):
+        os.makedirs(SOURCES_DIR)
+
+    ts = str(datetime.now().timestamp()).split('.', 1)[0]
+
+    source_path = os.path.join(SOURCES_DIR, '{}-{}.txt'.format('stdin', ts))
+
+    with open(source_path, 'w', encoding='utf-8') as f:
+        f.write(raw_text)
+
+    return source_path
+
+
+def fetch_page_content(url, timeout=TIMEOUT):
+    req = Request(url, headers={'User-Agent': WGET_USER_AGENT})
+
+    if CHECK_SSL_VALIDITY:
+        resp = urlopen(req, timeout=timeout)
+    else:
+        import ssl
+        insecure = ssl._create_unverified_context()
+        resp = urlopen(req, timeout=timeout, context=insecure)
+
+    encoding = resp.headers.get_content_charset() or 'utf-8'
+    return resp.read().decode(encoding)
+
+
+def save_remote_source(url, timeout=TIMEOUT):
    """download a given url's content into downloads/domain.txt"""

    if not os.path.exists(SOURCES_DIR):
@@ -188,26 +278,56 @@ def download_url(url):

    source_path = os.path.join(SOURCES_DIR, '{}-{}.txt'.format(domain(url), ts))

-    print('[*] [{}] Downloading {} > {}'.format(
+    print('{}[*] [{}] Downloading {}{}'.format(
+        ANSI['green'],
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
        url,
-        pretty_path(source_path),
+        ANSI['reset'],
    ))
    end = progress(TIMEOUT, prefix='      ')
    try:
-        downloaded_xml = requests.get(url).content.decode()
+        downloaded_xml = fetch_page_content(url, timeout=timeout)
        end()
    except Exception as e:
        end()
-        print('[!] Failed to download {}\n'.format(url))
+        print('{}[!] Failed to download {}{}\n'.format(
+            ANSI['red'],
+            url,
+            ANSI['reset'],
+        ))
        print('    ', e)
        raise SystemExit(1)

    with open(source_path, 'w', encoding='utf-8') as f:
        f.write(downloaded_xml)

+    print('    > {}'.format(pretty_path(source_path)))
+
    return source_path

+
+def fetch_page_title(url, timeout=10, progress=SHOW_PROGRESS):
+    """Attempt to guess a page's title by downloading the html"""
+    if not FETCH_TITLE:
+        return None
+
+    try:
+        if progress:
+            sys.stdout.write('.')
+            sys.stdout.flush()
+
+        html = fetch_page_content(url, timeout=timeout)
+
+        match = re.search(HTML_TITLE_REGEX, html)
+        return match.group(1).strip() if match else None
+    except Exception as err:
+        # print('[!] Failed to fetch title because of {}: {}'.format(
+        #     err.__class__.__name__,
+        #     err,
+        # ))
+        return None
+
+
 def str_between(string, start, end=None):
    """(<abc>12345</def>, <abc>, </def>)  ->  12345"""

@@ -220,19 +340,19 @@ def str_between(string, start, end=None):
 def get_link_type(link):
    """Certain types of links need to be handled specially, this figures out when that's the case"""

-    if link['base_url'].endswith('.pdf'):
+    if extension(link['url']) == 'pdf':
        return 'PDF'
-    elif link['base_url'].rsplit('.', 1) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
+    elif extension(link['url']) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
        return 'image'
-    elif 'wikipedia.org' in link['domain']:
+    elif 'wikipedia.org' in domain(link['url']).lower():
        return 'wiki'
-    elif 'youtube.com' in link['domain']:
+    elif 'youtube.com' in domain(link['url']).lower():
        return 'youtube'
-    elif 'soundcloud.com' in link['domain']:
+    elif 'soundcloud.com' in domain(link['url']).lower():
        return 'soundcloud'
-    elif 'youku.com' in link['domain']:
+    elif 'youku.com' in domain(link['url']).lower():
        return 'youku'
-    elif 'vimeo.com' in link['domain']:
+    elif 'vimeo.com' in domain(link['url']).lower():
        return 'vimeo'
    return None

@@ -240,19 +360,19 @@ def merge_links(a, b):
    """deterministially merge two links, favoring longer field values over shorter,
    and "cleaner" values over worse ones.
    """
-    longer = lambda key: a[key] if len(a[key]) > len(b[key]) else b[key]
+    longer = lambda key: (a[key] if len(a[key]) > len(b[key]) else b[key]) if (a[key] and b[key]) else (a[key] or b[key])
    earlier = lambda key: a[key] if a[key] < b[key] else b[key]
    
    url = longer('url')
    longest_title = longer('title')
-    cleanest_title = a['title'] if '://' not in a['title'] else b['title']
+    cleanest_title = a['title'] if '://' not in (a['title'] or '') else b['title']
    link = {
        'timestamp': earlier('timestamp'),
        'url': url,
        'domain': domain(url),
        'base_url': base_url(url),
        'tags': longer('tags'),
-        'title': longest_title if '://' not in longest_title else cleanest_title,
+        'title': longest_title if '://' not in (longest_title or '') else cleanest_title,
        'sources': list(set(a.get('sources', []) + b.get('sources', []))),
    }
    link['type'] = get_link_type(link)
@@ -263,15 +383,15 @@ def find_link(folder, links):
    url = parse_url(folder)
    if url:
        for link in links:
-            if (link['base_url'] in url) or (url in link['url']):
+            if (base_url(link['url']) in url) or (url in link['url']):
                return link

    timestamp = folder.split('.')[0]
    for link in links:
        if link['timestamp'].startswith(timestamp):
-            if link['domain'] in os.listdir(os.path.join(ARCHIVE_DIR, folder)):
+            if domain(link['url']) in os.listdir(os.path.join(ARCHIVE_DIR, folder)):
                return link      # careful now, this isn't safe for most ppl
-            if link['domain'] in parse_url(folder):
+            if domain(link['url']) in parse_url(folder):
                return link
    return None

@@ -285,7 +405,7 @@ def parse_url(folder):
                link_json = f.read().strip()
                if link_json:
                    link = json.loads(link_json)
-                    return link['base_url']
+                    return base_url(link['url'])
            except ValueError:
                print('File contains invalid JSON: {}!'.format(link_json))

@@ -341,8 +461,8 @@ def fix_folder_path(archive_path, link_folder, link):
    target = os.path.join(archive_path, link['timestamp'])

    url_in_folder = parse_url(source)
-    if not (url_in_folder in link['base_url']
-            or link['base_url'] in url_in_folder):
+    if not (url_in_folder in base_url(link['url'])
+            or base_url(link['url']) in url_in_folder):
        raise ValueError('The link does not match the url for this folder.')

    if not os.path.exists(target):
@@ -430,12 +550,12 @@ def wget_output_path(link, look_in=None):
    urlencode = lambda s: quote(s, encoding='utf-8', errors='replace')

    if link['type'] in ('PDF', 'image'):
-        return urlencode(link['base_url'])
+        return urlencode(base_url(link['url']))

    # Since the wget algorithm to for -E (appending .html) is incredibly complex
    # instead of trying to emulate it here, we just look in the output folder
    # to see what html file wget actually created as the output
-    wget_folder = link['base_url'].rsplit('/', 1)[0].split('/')
+    wget_folder = base_url(link['url']).rsplit('/', 1)[0].split('/')
    look_in = os.path.join(ARCHIVE_DIR, link['timestamp'], *wget_folder)

    if look_in and os.path.exists(look_in):
@@ -455,7 +575,7 @@ def wget_output_path(link, look_in=None):

    # if re.search(".+\\.[Hh][Tt][Mm][Ll]?$", split_url[0], re.I | re.M):
    #     # already ends in .html
-    #     return urlencode(link['base_url'])
+    #     return urlencode(base_url(link['url']))
    # else:
    #     # .html needs to be appended
    #     without_scheme = split_url[0].split('://', 1)[-1].split('?', 1)[0]
@@ -468,33 +588,99 @@ def wget_output_path(link, look_in=None):
    #             return urlencode('#'.join([without_scheme + '/index.html' + query + '.html', *split_url[1:]]))
    #         elif '/' in without_scheme:
    #             return urlencode('#'.join([without_scheme + '.html', *split_url[1:]]))
-    #         return urlencode(link['base_url'] + '/index.html')
+    #         return urlencode(base_url(link['url']) + '/index.html')


 def derived_link_info(link):
    """extend link info with the archive urls and other derived data"""

-    link_info = {
+    url = link['url']
+
+    extended_info = {
        **link,
-        'date': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
-        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
-        'favicon_url': 'archive/{timestamp}/favicon.ico'.format(**link),
-        'files_url': 'archive/{timestamp}/index.html'.format(**link),
+        'title': link['title'] or base_url(url),
+        'date': datetime.fromtimestamp(Decimal(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
+        'base_url': base_url(url),
+        'domain': domain(url),
+        'basename': basename(url),
+        'path': path(url),
+    }
+
+    # Archive Method Output URLs
+    extended_info = {
+        **extended_info,
+        'favicon_url': 'archive/{timestamp}/favicon.ico'.format(**extended_info),
+        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**extended_info),
+        'files_url': 'archive/{timestamp}/index.html'.format(**extended_info),
        'archive_url': 'archive/{}/{}'.format(link['timestamp'], wget_output_path(link) or 'index.html'),
-        'pdf_link': 'archive/{timestamp}/output.pdf'.format(**link),
-        'screenshot_link': 'archive/{timestamp}/screenshot.png'.format(**link),
-        'dom_link': 'archive/{timestamp}/output.html'.format(**link),
-        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**link),
+        'warc_url': 'archive/{timestamp}/warc'.format(**extended_info),
+        'pdf_link': 'archive/{timestamp}/output.pdf'.format(**extended_info),
+        'screenshot_link': 'archive/{timestamp}/screenshot.png'.format(**extended_info),
+        'dom_link': 'archive/{timestamp}/output.html'.format(**extended_info),
+        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**extended_info),
+        'git_url': 'archive/{timestamp}/git'.format(**extended_info),
+        'media_url': 'archive/{timestamp}/media'.format(**extended_info),
+        
    }

    # PDF and images are handled slightly differently
    # wget, screenshot, & pdf urls all point to the same file
    if link['type'] in ('PDF', 'image'):
-        link_info.update({
-            'archive_url': 'archive/{timestamp}/{base_url}'.format(**link),
-            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'dom_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'title': '{title} ({type})'.format(**link),
+        extended_info.update({
+            'archive_url': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'dom_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'title': link['title'] or basename(link['url']),
        })
-    return link_info
+
+    return extended_info
+
+
+def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs):
+    """Patched of subprocess.run to fix blocking io making timeout=innefective"""
+
+    if input is not None:
+        if 'stdin' in kwargs:
+            raise ValueError('stdin and input arguments may not both be used.')
+        kwargs['stdin'] = PIPE
+
+    if capture_output:
+        if ('stdout' in kwargs) or ('stderr' in kwargs):
+            raise ValueError('stdout and stderr arguments may not be used '
+                             'with capture_output.')
+        kwargs['stdout'] = PIPE
+        kwargs['stderr'] = PIPE
+
+    with Popen(*popenargs, **kwargs) as process:
+        try:
+            stdout, stderr = process.communicate(input, timeout=timeout)
+        except TimeoutExpired:
+            process.kill()
+            try:
+                stdout, stderr = process.communicate(input, timeout=2)
+            except:
+                pass
+            raise TimeoutExpired(popenargs[0][0], timeout)
+        except BaseException as err:
+            process.kill()
+            # We don't call process.wait() as .__exit__ does that for us.
+            raise 
+        retcode = process.poll()
+        if check and retcode:
+            raise CalledProcessError(retcode, process.args,
+                                     output=stdout, stderr=stderr)
+    return CompletedProcess(process.args, retcode, stdout, stderr)
+
+
+def check_link_structure(link):
+    assert isinstance(link, dict)
+    assert isinstance(link.get('url'), str)
+    assert len(link['url']) > 2
+    assert len(re.findall(URL_REGEX, link['url'])) == 1
+
+
+def check_links_structure(links):
+    assert isinstance(links, list)
+    if links:
+        check_link_structure(links[0])
@@ -1,54 +0,0 @@
-#!/usr/bin/env python3
-import argparse
-import re
-from typing import List
-
-from archive import parse_json_link_index
-from config import OUTPUT_DIR
-from index import write_json_links_index
-
-
-def cleanup_index(patterns: List[str], yes=False):
-    regexes = [re.compile(p) for p in patterns]
-
-    index = parse_json_link_index(OUTPUT_DIR)
-    links = index['links']
-
-    filtered = []
-    remaining = []
-    for l in links:
-        url = l['url']
-        for r in regexes:
-            if r.search(url):
-                filtered.append((l, r))
-                break
-        else:
-            remaining.append(l)
-
-
-    print("Filtered out {}/{} urls:".format(len(filtered), len(links)))
-    for link, regex in filtered:
-        url = link['url']
-        print(" {url} via {regex}".format(url=url, regex=regex.pattern))
-
-    proceed = False
-    if yes:
-        proceed = True
-    else:
-        res = input("Remove {} entries from index? [y/n] ".format(len(filtered)))
-        proceed = res.strip().lower() in ('y', 'yes')
-
-    if proceed:
-        write_json_links_index(OUTPUT_DIR, remaining)
-    else:
-        exit('aborting')
-
-
-if __name__ == '__main__':
-    p = argparse.ArgumentParser('Index purging tool')
-    p.add_argument('--regex', '-r', action='append', help='Python regex to filter out')
-    p.add_argument('--yes', action='store_true', default=False, help='Do not propmpt for confirmation')
-
-    args = p.parse_args()
-    regexes = args.regex
-    cleanup_index(regexes, yes=args.yes)
@@ -1 +0,0 @@
-requests
@@ -0,0 +1,18 @@
+# Binaries for running ArchiveBox
+
+This folder contains all the executables that ArchiveBox provides.
+
+
+# Adding it to your `$PATH`
+To be able to run ArchiveBox from anywhere on your system, you can add this entire folder to your path, like so:
+
+**Edit `~/.bash_profile`:**
+```bash
+export PATH=/opt/ArchiveBox/bin:$PATH
+```
+
+# Running executables directly
+
+If you don't want to add ArchiveBox to your `$PATH` you can also call these executables directly with their full path, like so:
+
+`/opt/ArchiveBox/bin/ArchiveBox https://example.com/some/feed.rss`
@@ -1 +1 @@
-../archiver/archive.py
+../archivebox/archive.py
@@ -2,12 +2,14 @@

 REPO_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )"; cd .. && pwd )"

+mkdir -p "$REPO_DIR/output/sources"
+
 if [[ "$1" == "--chrome" ]]; then
    # Google Chrome / Chromium
-    default=$(ls ~/Library/Application\ Support/Google/Chrome/Default/History)
    if [[ -e "$2" ]]; then
        cp "$2" "$REPO_DIR/output/sources/chrome_history.db.tmp"
    else
+        default=$(ls ~/Library/Application\ Support/Google/Chrome/Default/History)
        echo "Defaulting to history db: $default"
        echo "Optionally specify the path to a different sqlite history database as the 2nd argument."
        cp "$default" "$REPO_DIR/output/sources/chrome_history.db.tmp"
@@ -20,10 +22,10 @@ fi

 if [[ "$1" == "--firefox" ]]; then
    # Firefox
-    default=$(ls ~/Library/Application\ Support/Firefox/Profiles/*.default/places.sqlite)
    if [[ -e "$2" ]]; then
        cp "$2" "$REPO_DIR/output/sources/firefox_history.db.tmp"
    else
+        default=$(ls ~/Library/Application\ Support/Firefox/Profiles/*.default/places.sqlite)
        echo "Defaulting to history db: $default"
        echo "Optionally specify the path to a different sqlite history database as the 2nd argument."
        cp "$default" "$REPO_DIR/output/sources/firefox_history.db.tmp"
@@ -0,0 +1 @@
+../archivebox/purge.py
@@ -10,6 +10,7 @@ echo "        - Chromium Browser   (see README for Google-Chrome instructions in
 echo "        - python3"
 echo "        - wget"
 echo "        - curl"
+echo "        - youtube-dl"
 echo ""
 echo "    You may follow Manual Setup instructions in README.md instead if you prefer not to run an unknown script."
 echo "    Press enter to continue, or Ctrl+C to cancel..."
@@ -25,7 +26,6 @@ if which apt-get > /dev/null; then
        echo "[i] You already have google-chrome installed, if you would like to download chromium-browser instead (they work pretty much the same), follow the Manual Setup instructions"
        echo "[+] Linking $(which google-chrome) -> /usr/bin/chromium-browser (press enter to continue, or Ctrl+C to cancel...)"
        read
-        sudo ln -s "$(which google-chrome)" /usr/bin/chromium-browser
    elif which chromium-browser; then
        echo "[i] chromium-browser already installed, using existing installation."
        chromium-browser --version
@@ -34,7 +34,7 @@ if which apt-get > /dev/null; then
        apt install chromium-browser -y
    fi
    echo "[+] Installing python3, wget, curl..."
-    apt install -y python3 python3-distutils python3-requests wget curl
+    apt install -y python3 python3-distutils wget curl youtube-dl

 # On Mac:
 elif which brew > /dev/null; then   # 🐍 eye of newt
@@ -57,7 +57,7 @@ elif which brew > /dev/null; then   # 🐍 eye of newt
        chmod +x /usr/local/bin/chromium-browser
    fi
    echo "[+] Installing python3, wget, curl  (ignore 'already installed' warnings)..."
-    brew install python3 wget curl
+    brew install python3 wget curl youtube-dl
 else
    echo "[X] Could not find aptitude or homebrew! ‼️"
    echo ""
@@ -1 +0,0 @@
-../archiver/purge.py
@@ -1 +0,0 @@
-setup-archivebox
@@ -0,0 +1,21 @@
+version: '3'
+
+services:
+    archivebox:
+        build: .
+        stdin_open: true
+        tty: true
+        environment:
+            - USE_COLOR=False
+            - SHOW_PROGRESS=False
+        volumes:
+            - ./data:/data
+        command: bash -c 'echo "https://github.com/pirate/ArchiveBox" | /bin/archive; tail -f /dev/null'
+
+    nginx:
+        image: 'nginx'
+        ports:
+            - '8098:80'
+        volumes:
+            - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf
+            - ./data:/var/www
@@ -0,0 +1,65 @@
+# Example config file for ArchiveBox
+# Copy this file to ~/.ArchiveBox.conf and then edit it
+# Config file uses python syntax but can also be sourced as a .env
+# For documentation, see:
+#    https://github.com/pirate/ArchiveBox/wiki/Configuration
+
+################################################################################
+## General Settings
+################################################################################
+
+#OUTPUT_DIR=output
+#OUTPUT_PERMISSIONS=755
+#ONLY_NEW=False
+#TIMEOUT=60
+#MEDIA_TIMEOUT=3600
+#TEMPLATES_DIR=archivebox/templates
+#FOOTER_INFO="Run by ACME co."
+
+
+################################################################################
+## Archive Method Toggles
+################################################################################
+
+#FETCH_TITLE=True
+#FETCH_FAVICON=True
+#FETCH_WGET=True
+#FETCH_WARC=True
+#FETCH_PDF=True
+#FETCH_SCREENSHOT=True
+#FETCH_DOM=True
+#FETCH_GIT=True
+#FETCH_MEDIA=False
+#SUBMIT_ARCHIVE_DOT_ORG=True
+
+
+################################################################################
+## Archive Method Options
+################################################################################
+
+#CHECK_SSL_VALIDITY=True
+#FETCH_WGET_REQUISITES=True
+#RESOLUTION="1440,900"
+#WGET_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
+#GIT_DOMAINS="github.com,bitbucket.org,gitlab.com"
+#COOKIES_FILE="path/to/cookies.txt"
+#CHROME_USER_DATA_DIR="~/.config/google-chrome/Default"
+
+
+################################################################################
+## Shell Options
+################################################################################
+
+#USE_COLOR=True
+#SHOW_PROGRESS=True
+
+
+################################################################################
+## Dependency Options
+################################################################################
+
+#CURL_BINARY="curl"
+#GIT_BINARY="git"
+#WGET_BINARY="wget"
+#YOUTUBEDL_BINARY="youtube-dl"
+#CHROME_BINARY="chromium-browser"
@@ -0,0 +1,12 @@
+# Example etc files for deploying ArchiveBox
+
+In this folder are some example config files you can use for setting up ArchiveBox on your machine.
+
+E.g. see `etc/nginx` for an example nginx config to serve your archive, or `etc/cron` for an example cron job that crawls a feed every 24 hours.
+
+Please contribute your etc files here! Example contributions
+
+- supervisord config
+- systemd config
+- apache webserver config
+- other init system, webservers, or scheduler configs
@@ -0,0 +1 @@
+0 24 * * * www-data /opt/ArchiveBox/bin/archive "https://getpocket.com/users/example/feed/all" >> /var/log/ArchiveBox.log
@@ -1,10 +0,0 @@
-server {
-    listen                      80 default_server;
-    server_name                 _;
-
-    root                        /data/output/;
-    index                       index.html;
-    autoindex                   on;
-
-    try_files                   $uri $uri/ $uri.html =404;
-}
@@ -0,0 +1,45 @@
+user www-data;
+pid                                 /var/run/nginx.pid;
+
+worker_processes                    auto;
+timer_resolution                    100ms;
+
+worker_rlimit_nofile                40000;
+events {
+    worker_connections              40000;
+    use epoll;
+    multi_accept on;
+}
+
+http {
+    sendfile on;
+    tcp_nopush on;
+    tcp_nodelay on;
+    server_tokens off;
+    send_timeout 20;
+    keepalive_timeout 65;
+    types_hash_max_size 2048;
+    client_max_body_size 25m;
+
+    include                     mime.types;
+    default_type                application/octet-stream;
+
+    access_log                  /dev/stdout;
+    error_log                   stderr;
+
+    reset_timedout_connection on;
+
+    server_names_hash_bucket_size   64;
+
+    server {
+        listen                      80 default_server;
+        server_name                 _;
+
+        root                        /var/www;
+        index                       index.html;
+        autoindex                   on;
+
+        try_files                   $uri $uri/ $uri.html =404;
+    }
+}
+
@@ -1 +1 @@
-bin/setup-archivebox
+bin/archivebox-setup
				`@@ -0,0 +1 @@`
				`Make sure check in with me first or confirm your desired features line up with our roadmap: https://github.com/pirate/ArchiveBox#roadmap`
				`@@ -0,0 +1 @@`
				`0 24 * * * www-data /opt/ArchiveBox/bin/archive "https://getpocket.com/users/example/feed/all" >> /var/log/ArchiveBox.log`