Update README.md

when checking link invariants, check for regex match as well
2019-02-27 13:39:35 -05:00 · 2019-02-27 13:37:57 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00 · 2019-02-27 04:50:29 -05:00
65 changed files with 2667 additions and 1564 deletions
@@ -0,0 +1,6 @@
+output
+__pycache__
+.DS_Store
+venv
+.venv
+data
@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at archivebox-coc@sweeting.me. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq
@@ -0,0 +1 @@
+Make sure check in with me first or confirm your desired features line up with our roadmap: https://github.com/pirate/ArchiveBox#roadmap
@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A description of what the bug is, what you expected to happen, 
+and any relevant context about issue.
+
+**Steps to reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Screenshots or log output**
+If applicable, use screenshots or copy/pasted terminal output to help explain your problem.
+
+**Software versions (please complete the following information):**
+ - ArchiveBox version: <commit_id>
+ - Python version: python3 --version [e.g. 3.7.0]
+ - OS: [e.g. macOS 10.14]
+ - Chrome version: chromium-browser --version [e.g. v73.1.2.3]
@@ -0,0 +1,32 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+Type:
+
+ - [ ] General Question or Disussion
+ - [ ] Propose a brand new feature
+ - [ ] Request modification of existing behavior or design
+
+**What is the problem that your feature request solves**
+e.g. I need to be able to archive spanish and french subtitle files
+from a particular <example.com> movie site that's going down soon.
+
+**Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes**
+e.g.
+  specifically: a new archive method to look for multilingual subtitle files related to pages.
+  bigger picture: the ability for custom user scripts to be run in a puppeteer context to archive content
+
+**What hacks or alternative solutions have you tried to solve the problem?**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**How badly do you want this new feature?**
+ - [ ] It's an urgent deal-breaker, I cant live without it
+ - [ ] It's important to add it in the near-mid term future
+ - [ ] It would be nice to have eventualy
+ - [ ] I'm willing to contribute to development
@@ -0,0 +1,16 @@
+# Summary
+
+This PR add the ability to do XYZ...
+
+# Changes these areas
+
+- [ ] Config
+- [ ] Bugfixes
+- [ ] Command line interface
+- [ ] Feature behavior
+- [ ] Internal design
+- [ ] Archived data layout on disk
+
+# Roadmap Goals
+
+This PR helps us move towards ____ roadmap goal, as outlined here: https://github.com/pirate/ArchiveBox#roadmap
@@ -1,4 +1,20 @@
-output/
+# OS cruft
 .DS_Store
+._*
+
+# python
 __pycache__/
-archiver/venv
+venv
+.venv
+archivebox/.venv
+archivebox/venv
+
+# vim
+.swp*
+
+# output artifacts
+output
+output/
+data
+data/
+archivebox/output
@@ -0,0 +1,3 @@
+[submodule "docs"]
+	path = docs
+	url = https://github.com/pirate/ArchiveBox.wiki.git
@@ -0,0 +1 @@
+archivebox.io
@@ -0,0 +1,57 @@
+FROM node:11-slim
+LABEL maintainer="Nick Sweeting <archivebox-git@sweeting.me>"
+
+RUN apt-get update \
+    && apt-get install -yq --no-install-recommends \
+        git wget curl youtube-dl gnupg2 libgconf-2-4 python3 python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install latest chrome package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
+RUN apt-get update && apt-get install -y wget --no-install-recommends \
+    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
+    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
+    && apt-get update \
+    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
+      --no-install-recommends \
+    && rm -rf /var/lib/apt/lists/* \
+    && rm -rf /src/*.deb
+
+# It's a good idea to use dumb-init to help prevent zombie chrome processes.
+ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
+RUN chmod +x /usr/local/bin/dumb-init
+
+# Install puppeteer so it's available in the container.
+RUN npm i puppeteer
+
+# Add user so we don't need --no-sandbox.
+RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
+    && mkdir -p /home/pptruser/Downloads \
+    && chown -R pptruser:pptruser /home/pptruser \
+    && chown -R pptruser:pptruser /node_modules
+
+# Install the ArchiveBox repository and pip requirements
+RUN git clone https://github.com/pirate/ArchiveBox /home/pptruser/app \
+    && mkdir -p /data \
+    && chown -R pptruser:pptruser /data \
+    && ln -s /data /home/pptruser/app/archivebox/output \
+    && ln -s /home/pptruser/app/bin/* /bin/ \
+    && ln -s /home/pptruser/app/bin/archive /bin/archivebox \
+    && chown -R pptruser:pptruser /home/pptruser/app/archivebox
+    # && pip3 install -r /home/pptruser/app/archivebox/requirements.txt
+
+VOLUME /data
+
+ENV LANG=C.UTF-8 \
+    LANGUAGE=en_US:en \
+    LC_ALL=C.UTF-8 \
+    PYTHONIOENCODING=UTF-8 \
+    CHROME_SANDBOX=False \
+    CHROME_BINARY=google-chrome-unstable \
+    OUTPUT_DIR=/data
+
+# Run everything from here on out as non-privileged user
+USER pptruser
+WORKDIR /home/pptruser/app
+
+ENTRYPOINT ["dumb-init", "--"]
+CMD ["/bin/archive"]
@@ -1,462 +1,177 @@
-# Bookmark Archiver <img src="https://nicksweeting.com/images/archive.png" height="22px"/>  [![Github Stars](https://img.shields.io/github/stars/pirate/bookmark-archiver.svg)](https://github.com/pirate/bookmark-archiver) [![Twitter URL](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/thesquashSH)
+<div align="center">
+<img src="https://i.imgur.com/4nkFjdv.png" height="80px">
+<h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>

-    "Your own personal Way-Back Machine"
+▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
+<a href="https://archive.sweeting.me">Demo</a> | 
+<a href="https://archivebox.io">Website</a> | 
+<a href="https://github.com/pirate/ArchiveBox">Github</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Troubleshooting">Troubleshooting</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Changelog">Changelog</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>

-▶️ [Quickstart](#quickstart) | [Details](#details) | [Configuration](#configuration) | [Manual Setup](#manual-setup) | [Troubleshooting](#troubleshooting) | [Demo](https://archive.sweeting.me) | [Changelog](#changelog) | [Donate](https://github.com/pirate/bookmark-archiver/blob/master/donate.md)
+<pre>
+"Your own personal internet archive" (网站存档 / 爬虫)
+</pre>

---
+<a href="https://github.com/pirate/ArchiveBox">
+<img src="https://img.shields.io/badge/Open_source-free-green.svg?logo=git&logoColor=green"/>
+<img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/>
+<img src="https://img.shields.io/badge/License-MIT-lightgreen.svg?logo=MakerBot&logoColor=lightgreen"/>
+<img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/>
+<img src="https://img.shields.io/badge/Python-%3E%3D3.5-yellow.svg?logo=python&logoColor=yellow"/>
+<img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/>
+<img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/>
+</a>
+<hr/>
+</div>

-Save an archived copy of all websites you bookmark (the actual *content* of each site, not just the list of bookmarks).
+**ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).** 

-Can import links from:
+You can use it to preserve access to websites you care about by storing them locally offline.  ArchiveBox works by rendering the pages in a headless browser, then saving all the requests and fully loaded pages in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the original content dissapears off the internet.  It also automatically extracts assets like git repositories, audio, video, subtitles, images, and PDFs into separate files using `youtube-dl`, `pywb`, and `wget`.

- - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser Bookmarks (Chrome, Firefox, Safari, IE, Opera)
- - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket
- - <img src="https://pinboard.in/favicon.ico" height="22px"/> Pinboard
- - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS or plain text lists
- - Shaarli, Delicious, Instapaper, Reddit Saved Posts, Wallabag, Unmark.it, and more!
- - Continuous archiving of browser history (every site, not just bookmarked) (coming soon!...)
+ArchiveBox doesn't require a constantly running server or backend, instead you just run the `./archive` command each time you want to import new links and update the static output.  It can import and export JSON (among other formats), so it's easy to script or hook up to other APIs.  If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).

-For each site, it outputs:
+<div align="center"><sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub></div><br/>

- Browsable static HTML archive (wget)
- PDF (Chrome headless)
- Screenshot (Chrome headless)
- DOM dump (Chrome headless)
- Favicon
- Submits URL to archive.org
- Index summary pages: index.html & index.json
-
-The archiving is additive, so you can schedule `./archive` to run regularly and pull new links into the index.
-
-[DEMO: archive.sweeting.me](https://archive.sweeting.me)
-
-<img src="screenshot.png" width="75%" alt="Desktop Screenshot" align="top"><img src="screenshot_mobile.png" width="25%" alt="Mobile Screenshot" align="top"><br/>
-
-## Quickstart
-
-**1. Get your list of URLs:**
-
-Follow the links here to find instructions for exporting bookmarks from each service.
-
- - [Pocket](https://getpocket.com/export)
- - [Pinboard](https://pinboard.in/export/)
- - [Instapaper](https://www.instapaper.com/user/export)
- - [Reddit Saved Posts](https://github.com/csu/export-saved-reddit)
- - [Shaarli](http://shaarli.readthedocs.io/en/master/Backup,-restore,-import-and-export/#export-links-as)
- - [Unmark.it](http://help.unmark.it/import-export)
- - [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html)
- - [Chrome Bookmarks](https://support.google.com/chrome/answer/96816?hl=en)
- - [Firefox Bookmarks](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer)
- - [Safari Bookmarks](http://i.imgur.com/AtcvUZA.png)
- - [Opera Bookmarks](http://help.opera.com/Windows/12.10/en/importexport.html)
- - [Internet Explorer Bookmarks](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows)
- - Other File or URL: (e.g. RSS feed) pass as second argument in the next step
-
- (If any of these links are broken, please submit an issue and I'll fix it)
-
-**2. Create your archive:**

+To get started, you can install ArchiveBox [automatically](https://github.com/pirate/ArchiveBox/wiki/Quickstart), follow the [manual instructions](https://github.com/pirate/ArchiveBox/wiki/Install), or use [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker).  
+There are only [3 main dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) beyond `python3`: `wget`, `chromium`, and `youtube-dl`, and you can skip installing them if you don't need those archive methods.
 ```bash
-git clone https://github.com/pirate/bookmark-archiver
-cd bookmark-archiver/
-./setup                                         # install all dependencies
-./archive ~/Downloads/bookmark_export.html      # replace with the path to your export file or URL from step 1
+git clone https://github.com/pirate/ArchiveBox.git
+cd ArchiveBox
+./setup

-# OR
-./archive "https://getpocket.com/users/yourusername/feed/all"  # url to an RSS, html, or json links file
+# Export your bookmarks, then run the archive command to start archiving!
+./archive ~/Downloads/bookmarks.html

-# OR
-./bin/export-browser-history --chrome           # works with --firefox as well
-./archive output/sources/chrome_history.json
-
-# OR
-./archive   # at any point if you just want to continue archiving where you left off, without adding any new links
+# Or pass in links to archive via stdin
+echo 'https://example.com' | ./archive
 ```

-**3. Done!**
+Open `output/index.html` in a browser to view your archive.  [DEMO: archive.sweeting.me](https://archive.sweeting.me)  
+For more information, see the [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.

-You can open `service/index.html` to view your archive.  (favicons will appear next to each title once it has finished downloading)
+## Overview

-If you want to host your archive somewhere to share it with other people, see the [Publishing Your Archive](#publishing-your-archive) section below.
+Because modern websites are complicated and often rely on dynamic content, 
+ArchiveBox archives the sites in **several different formats** beyond what public 
+archiving services like Archive.org and Archive.is are capable of saving.

-**4. (Optional) Schedule it to run every day**
+ArchiveBox imports a list of URLs from stdin, remote url, or file, then adds the pages to a local archive folder using wget to create a browsable html clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...

-You can import links from any local file path or feed url by changing the second argument to `archive.py`.
-Bookmark Archiver will ignore links that are imported multiple times, it will keep the earliest version that it's seen.
-This means you can add multiple cron jobs to pull links from several different feeds or files each day,
-it will keep the index up-to-date without duplicate links.
+Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.

-This example archives a pocket RSS feed and an export file every 24 hours, and saves the output to a logfile.
-```bash
-0 24 * * * yourusername /opt/bookmark-archiver/archive https://getpocket.com/users/yourusername/feed/all > /var/log/bookmark_archiver_rss.log
-0 24 * * * yourusername /opt/bookmark-archiver/archive /home/darth-vader/Desktop/bookmarks.html > /var/log/bookmark_archiver_firefox.log
-```
-(Add the above lines to `/etc/crontab`)
+### Can import links from:

-**Next Steps**  
-  
-If you have any trouble, see the [Troubleshooting](#troubleshooting) section at the bottom.  
-If you'd like to customize options, see the [Configuration](#configuration) section.  
+ - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper
+ - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, HTML, Markdown, or plain text lists
+ - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera, and more)
+ - *Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, and any other text with links in it!*

-If you want something easier than running programs in the command-line, take a look at [Pocket Premium](https://getpocket.com/premium) (yay Mozilla!) and [Pinboard Pro](https://pinboard.in/upgrade/) (yay independent developer!).  Both offer easy-to-use bookmark archiving with full-text-search and other features.
+### Can save these things for each site:

-## Details
-
-`archive.py` is a script that takes a [Pocket-format](https://getpocket.com/export), [JSON-format](https://pinboard.in/export/), [Netscape-format](https://msdn.microsoft.com/en-us/library/aa753582(v=vs.85).aspx), or RSS-formatted list of links, and downloads a clone of each linked website to turn into a browsable archive that you can store locally or host online.
-
-The archiver produces an output folder `output/` containing an `index.html`, `index.json`, and archived copies of all the sites,
-organized by timestamp bookmarked.  It's Powered by [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) Chromium and good 'ol `wget`.
-
-For each sites it saves:
-
- - wget of site, e.g. `en.wikipedia.org/wiki/Example.html` with .html appended if not present
+ - `favicon.ico` favicon of the site
+ - `example.com/page-name.html` wget clone of the site, with .html appended if not present
 - `output.pdf` Printed PDF of site using headless chrome
 - `screenshot.png` 1440x900 screenshot of site using headless chrome
 - `output.html` DOM Dump of the HTML after rendering using headless chrome
 - `archive.org.txt` A link to the saved site on archive.org
- - `audio/` and `video/` for sites like youtube, soundcloud, etc. (using youtube-dl) (WIP)
- - `code/` clone of any repository for github, bitbucket, or gitlab links (WIP)
- - `index.json` JSON index containing link info and archive details
- - `index.html` HTML index containing link info and archive details (optional fancy or simple index)
+ - `warc/` for the html + gzipped warc file <timestamp>.gz
+ - `media/` any mp4, mp3, subtitles, and metadata found using youtube-dl
+ - `git/` clone of any repository for github, bitbucket, or gitlab links
+ - `index.html` & `index.json` HTML and JSON index files containing metadata and details

-Wget doesn't work on sites you need to be logged into, but chrome headless does, see the [Configuration](#configuration)* section for `CHROME_USER_DATA_DIR`.
+By default it does everything, but can disable or tweak [individual options](https://github.com/pirate/ArchiveBox/wiki/Configuration) via environment variables or config file.

-**Large Exports & Estimated Runtime:** 
+The archiving is additive, so you can schedule `./archive` to [run regularly](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving) and pull new links into the index.
+All the saved content is static and indexed with JSON files, so it lives forever & is easily parseable, it requires no always-running backend.

-I've found it takes about an hour to download 1000 articles, and they'll take up roughly 1GB.  
-Those numbers are from running it single-threaded on my i5 machine with 50mbps down.  YMMV.  
+### Related Projects

-You can run it in parallel by using the `resume` feature, or by manually splitting export.html into multiple files:
-```bash
-./archive export.html 1498800000 &  # second argument is timestamp to resume downloading from
-./archive export.html 1498810000 &
-./archive export.html 1498820000 &
-./archive export.html 1498830000 &
-```
-Users have reported running it with 50k+ bookmarks with success (though it will take more RAM while running).
+There are tons of great web archiving tools out there.  In particular https://webrecorder.io ([pywb](https://github.com/webrecorder/pywb)) and https://getpolarized.io/ are robust, stable pieces of software that have lots of overlap with ArchiveBox.  ArchiveBox differentiates itself by being primarily a one-shot CLI tool that specializing in importing streams of links from RSS, JSON (good for automatically archiving from a stream of browser history or bookmarks on a schedule), as opposed to a desktop application or web service that requires human interaction to add links.

-## Configuration
+To learn more about the motivation for this project and how it fits into the broader community, see our [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.

-You can tweak parameters via environment variables, or by editing `config.py` directly:
-```bash
-env CHROME_BINARY=google-chrome-stable RESOLUTION=1440,900 FETCH_PDF=False ./archive ~/Downloads/bookmarks_export.html
-```
+# Documentation

-**Shell Options:**
- - colorize console ouput: `USE_COLOR` value: [`True`]/`False`
- - show progress bar: `SHOW_PROGRESS` value: [`True`]/`False`
- - archive permissions: `OUTPUT_PERMISSIONS` values: [`755`]/`644`/`...`

-**Dependency Options:**
- - path to Chrome: `CHROME_BINARY` values: [`chromium-browser`]/`/usr/local/bin/google-chrome`/`...`
- - path to wget: `WGET_BINARY` values: [`wget`]/`/usr/local/bin/wget`/`...`
+<div align="center">
+<img src="https://i.imgur.com/PVO88AZ.png"/>
+<br/>
+    <sub><i>(Recently <a href="https://github.com/pirate/ArchiveBox/issues/108">renamed</a> from <code>Bookmark Archiver</code>)</i></sub>
+</div>

-**Archive Options:**
- - maximum allowed download time per link: `TIMEOUT` values: [`60`]/`30`/`...`
- - archive methods (values: [`True`]/`False`):
-   - fetch page with wget: `FETCH_WGET`
-   - fetch images/css/js with wget: `FETCH_WGET_REQUISITES` (True is highly recommended)
-   - print page as PDF: `FETCH_PDF`
-   - fetch a screenshot of the page: `FETCH_SCREENSHOT`
-   - fetch a DOM dump of the page: `FETCH_DOM`
-   - fetch a favicon for the page: `FETCH_FAVICON`
-   - submit the page to archive.org: `SUBMIT_ARCHIVE_DOT_ORG` 
- - screenshot: `RESOLUTION` values: [`1440,900`]/`1024,768`/`...`
- - user agent: `WGET_USER_AGENT` values: [`Wget/1.19.1`]/`"Mozilla/5.0 ..."`/`...`
- - chrome profile: `CHROME_USER_DATA_DIR` values: [`~/Library/Application\ Support/Google/Chrome/Default`]/`/tmp/chrome-profile`/`...`
-    To capture sites that require a user to be logged in, you must specify a path to a chrome profile (which loads the cookies needed for the user to be logged in).  If you don't have an existing chrome profile, create one with `chromium-browser --disable-gpu --user-data-dir=/tmp/chrome-profile`, and log into the sites you need.  Then set `CHROME_USER_DATA_DIR=/tmp/chrome-profile` to make Bookmark Archiver use that profile.
+---

- (See defaults & more at the top of `config.py`)
+We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) for documentation.

-To tweak the outputted html index file's look and feel, just copy the files in `templates/` somewhere else and edit away.  Use the two index config variables above to point the script to your new custom template files. 
+You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.

-The chrome/chromium dependency is _optional_ and only required for screenshots and PDF output, can be safely ignored if both of those are disabled.
+## Getting Started

-## Publishing Your Archive
+ - [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
+ - [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
+ - [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)

-The archive produced by `./archive` is suitable for serving on any provider that can host static html (e.g. github pages!).
+## Reference

-You can also serve it from a home server or VPS by uploading the outputted `output` folder to your web directory, e.g. `/var/www/bookmark-archiver` and configuring your webserver.
+ - [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
+ - [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
+ - [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
+ - [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
+ - [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Chromium-Install)
+ - [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)

-Here's a sample nginx configuration that works to serve archive folders:
+## More Info

-```nginx
-location / {
-    alias       /path/to/bookmark-archiver/output/;
-    index       index.html;
-    autoindex   on;               # see directory listing upon clicking "The Files" links
-    try_files   $uri $uri/ =404;
-}
-```
+ - [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
+ - [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog)
+ - [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
+ - [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)

-Make sure you're not running any content as CGI or PHP, you only want to serve static files!

-Urls look like: `https://archive.example.com/archive/1493350273/en.wikipedia.org/wiki/Dining_philosophers_problem.html`
+# Background & Motivation

-**Security WARNING & Content Disclaimer**
+Vast treasure troves of knowledge are lost every day on the internet to link rot.  As a society, we have an imperative
+to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.

-Re-hosting other people's content has security implications for any other sites sharing your hosting domain.  Make sure you understand
-the dangers of hosting unknown archived CSS & JS files [on your shared domain](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy).
-Due to the security risk of serving some malicious JS you archived by accident, it's best to put this on a domain or subdomain
-of its own to keep cookies separate and slightly mitigate [CSRF attacks](https://en.wikipedia.org/wiki/Cross-site_request_forgery) and other nastiness.
+Whether it's to resist censorship by saving articles before they get taken down or editied, or
+just to save a collection of early 2010's flash games you love to play, having the tools to 
+archive internet content enables to you save the stuff you care most about before it dissapears.

-You may also want to blacklist your archive in `/robots.txt` if you don't want to be publicly assosciated with all the links you archive via search engine results.
+The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. 
+I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.

-Be aware that some sites you archive may not allow you to rehost their content publicly for copyright reasons,
-it's up to you to host responsibly and respond to takedown requests appropriately.
+The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.

-Please modify the `FOOTER_INFO` config variable to add your contact info to the footer of your index.
+*Read more:*

-## Info & Motivation
+- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
+- Discover the web archiving community on the [community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.
+- Find other archving projects on Github using the [awesome-web-archiving](https://github.com/iipc/awesome-web-archiving) list.
+- Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.

-This is basically an open-source version of [Pocket Premium](https://getpocket.com/premium) (which you should consider paying for!).
-I got tired of sites I saved going offline or changing their URLS, so I started
-archiving a copy of them locally now, similar to The Way-Back Machine provided
-by [archive.org](https://archive.org).  Self hosting your own archive allows you to save
-PDFs & Screenshots of dynamic sites in addition to static html, something archive.org doesn't do.
+To learn more about ArchiveBox's past history and future plans, check out the [roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog).

-Now I can rest soundly knowing important articles and resources I like wont dissapear off the internet.
+# Screenshots

-My published archive as an example: [sweeting.me/pocket](https://home.sweeting.me/pocket).
+<img src="https://i.imgur.com/viklZNG.png" width="75%" alt="Desktop index screenshot" align="top"><img src="https://i.imgur.com/mW2dITg.png" width="25%" alt="Mobile details page screenshot" align="top"><br/>
+<img src="https://i.imgur.com/wnpdAVM.jpg" width="100%" alt="Desktop details page Screenshot"/><br/>
+<img src="https://i.imgur.com/3tBL7PU.png" width="100%" alt="CLI Screenshot">

-## Manual Setup
+---

-If you don't like running random setup scripts off the internet (:+1:), you can follow these manual setup instructions.
-
-**1. Install dependencies:** `chromium >= 59`,` wget >= 1.16`, `python3 >= 3.5`  (`google-chrome >= v59` works fine as well)
-
-If you already have Google Chrome installed, or wish to use that instead of Chromium, follow the [Google Chrome Instructions](#google-chrome-instructions).
-
-```bash
-# On Mac:
-brew cask install chromium  # If you already have Google Chrome/Chromium in /Applications/, skip this command
-brew install wget python3
-
-echo -e '#!/bin/bash\n/Applications/Chromium.app/Contents/MacOS/Chromium "$@"' > /usr/local/bin/chromium-browser  # see instructions for google-chrome below
-chmod +x /usr/local/bin/chromium-browser
-```
-
-```bash
-# On Ubuntu/Debian:
-apt install chromium-browser python3 wget
-```
-
-```bash
-# Check that everything worked:
-chromium-browser --version && which wget && which python3 && which curl && echo "[√] All dependencies installed."
-```
-
-**2. Get your bookmark export file:**
-
-Follow the instruction links above in the "Quickstart" section to download your bookmarks export file.
-
-**3. Run the archive script:**
-
-1. Clone this repo `git clone https://github.com/pirate/bookmark-archiver`
-3. `cd bookmark-archiver/`
-4. `./archive ~/Downloads/bookmarks_export.html`
-
-You may optionally specify a second argument to `archive.py export.html 153242424324` to resume the archive update at a specific timestamp.
-
-If you have any trouble, see the [Troubleshooting](#troubleshooting) section at the bottom.
-
-### Google Chrome Instructions:
-
-I recommend Chromium instead of Google Chrome, since it's open source and doesn't send your data to Google.
-Chromium may have some issues rendering some sites though, so you're welcome to try Google-chrome instead.
-It's also easier to use Google Chrome if you already have it installed, rather than downloading Chromium all over.
-
-1. Install & link google-chrome
-```bash
-# On Mac:
-# If you already have Google Chrome in /Applications/, skip this brew command
-brew cask install google-chrome
-brew install wget python3
-
-echo -e '#!/bin/bash\n/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome "$@"' > /usr/local/bin/google-chrome
-chmod +x /usr/local/bin/google-chrome
-```
-
-```bash
-# On Linux:
-wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
-sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
-apt update; apt install google-chrome-beta python3 wget
-```
-
-2. Set the environment variable `CHROME_BINARY` to `google-chrome` before running:
-
-```bash
-env CHROME_BINARY=google-chrome ./archive ~/Downloads/bookmarks_export.html
-```
-If you're having any trouble trying to set up Google Chrome or Chromium, see the Troubleshooting section below.
-
-## Troubleshooting
-
-### Dependencies
-
-**Python:**
-
-On some Linux distributions the python3 package might not be recent enough.
-If this is the case for you, resort to installing a recent enough version manually.
-```bash
-add-apt-repository ppa:fkrull/deadsnakes && apt update && apt install python3.6
-```
-If you still need help, [the official Python docs](https://docs.python.org/3.6/using/unix.html) are a good place to start.
-
-**Chromium/Google Chrome:**
-
-`archive.py` depends on being able to access a `chromium-browser`/`google-chrome` executable.  The executable used
-defaults to `chromium-browser` but can be manually specified with the environment variable `CHROME_BINARY`:
-
-```bash
-env CHROME_BINARY=/usr/local/bin/chromium-browser ./archive ~/Downloads/bookmarks_export.html
-```
-
-1. Test to make sure you have Chrome on your `$PATH` with:
-
-```bash
-which chromium-browser || which google-chrome
-```
-If no executable is displayed, follow the setup instructions to install and link one of them.
-
-2. If a path is displayed, the next step is to check that it's runnable:
-
-```bash
-chromium-browser --version || google-chrome --version
-```
-If no version is displayed, try the setup instructions again, or confirm that you have permission to access chrome.
-
-3. If a version is displayed and it's `<59`, upgrade it:
-
-```bash
-apt upgrade chromium-browser -y
-# OR
-brew cask upgrade chromium-browser
-```
-
-4. If a version is displayed and it's `>=59`, make sure `archive.py` is running the right one:
-
-```bash
-env CHROME_BINARY=/path/from/step/1/chromium-browser ./archive bookmarks_export.html   # replace the path with the one you got from step 1
-```
-
-
-**Wget & Curl:**
-
-If you're missing `wget` or `curl`, simply install them using `apt` or your package manager of choice.
-See the "Manual Setup" instructions for more details.
-
-If wget times out or randomly fails to download some sites that you have confirmed are online,
-upgrade wget to the most recent version with `brew upgrade wget` or `apt upgrade wget`.  There is
-a bug in versions `<=1.19.1_1` that caused wget to fail for perfectly valid sites.
-
-### Archiving
-
-**No links parsed from export file:**
-
-Please open an [issue](https://github.com/pirate/bookmark-archiver/issues) with a description of where you got the export, and
-preferrably your export file attached (you can redact the links).  We'll fix the parser to support your format.
-
-**Lots of skipped sites:**
-
-If you ran the archiver once, it wont re-download sites subsequent times, it will only download new links.
-If you haven't already run it, make sure you have a working internet connection and that the parsed URLs look correct.
-You can check the `archive.py` output or `index.html` to see what links it's downloading.
-
-If you're still having issues, try deleting or moving the `service/archive` folder and running `archive.py` again.
-
-**Lots of errors:**
-
-Make sure you have all the dependencies installed and that you're able to visit the links from your browser normally.
-Open an [issue](https://github.com/pirate/bookmark-archiver/issues) with a description of the errors if you're still having problems.
-
-**Lots of broken links from the index:**
-
-Not all sites can be effectively archived with each method, that's why it's best to use a combination of `wget`, PDFs, and screenshots.
-If it seems like more than 10-20% of sites in the archive are broken, open an [issue](https://github.com/pirate/bookmark-archiver/issues)
-with some of the URLs that failed to be archived and I'll investigate.
-
-### Hosting the Archive
-
-If you're having issues trying to host the archive via nginx, make sure you already have nginx running with SSL.
-If you don't, google around, there are plenty of tutorials to help get that set up.  Open an [issue](https://github.com/pirate/bookmark-archiver/issues)
-if you have problem with a particular nginx config.
-
-## Roadmap
-
-If you feel like contributing a PR, some of these tasks are pretty easy.  Feel free to open an issue if you need help getting started in any way!
-
- - download closed-captions text from youtube videos
- - body text extraction using [fathom](https://hacks.mozilla.org/2017/04/fathom-a-framework-for-understanding-web-pages/)
- - auto-tagging based on important extracted words
- - audio & video archiving with `youtube-dl`
- - full-text indexing with elasticsearch/elasticlunr/ag
- - video closed-caption downloading for full-text indexing video content
- - automatic text summaries of article with summarization library
- - feature image extraction
- - http support (from my https-only domain)
- - try wgetting dead sites from archive.org (https://github.com/hartator/wayback-machine-downloader)
- - live updating from pocket/pinboard
-
-It's possible to pull links via the pocket API or public pocket RSS feeds instead of downloading an html export.
-Once I write a script to do that, we can stick this in `cron` and have it auto-update on it's own.
-
-For now you just have to download `ril_export.html` and run `archive.py` each time it updates. The script
-will run fast subsequent times because it only downloads new links that haven't been archived already.
-
-## Links
-
-**Similar Projects:**
- - [Memex by Worldbrain.io](https://github.com/WorldBrain/Memex) a browser extension that saves all your history and does full-text search
- - [Hypothes.is](https://web.hypothes.is/) a web/pdf/ebook annotation tool that also archives content
- - [Perkeep](https://perkeep.org/) "Perkeep lets you permanently keep your stuff, for life."
- - [Fetching.io](http://fetching.io/) A personal search engine/archiver that lets you search through all archived websites that you've bookmarked
- - [Shaarchiver](https://github.com/nodiscc/shaarchiver) very similar project that archives Firefox, Shaarli, or Delicious bookmarks and all linked media, generating a markdown/HTML index
- - [Webrecorder.io](https://webrecorder.io/) Save full browsing sessions and archive all the content
- - [Wallabag](https://wallabag.org) Save articles you read locally or on your phone
-
-**Discussions:**
- - [Hacker News Discussion](https://news.ycombinator.com/item?id=14272133)
- - [Reddit r/selfhosted Discussion](https://www.reddit.com/r/selfhosted/comments/69eoi3/pocket_stream_archive_your_own_personal_wayback/)
- - [Reddit r/datahoarder Discussion #1](https://www.reddit.com/r/DataHoarder/comments/69e6i9/archive_a_browseable_copy_of_your_saved_pocket/)
- - [Reddit r/datahoarder Discussion #2](https://www.reddit.com/r/DataHoarder/comments/6kepv6/bookmarkarchiver_now_supports_archiving_all_major/)
-
-
-**Tools/Other:**
- - https://github.com/ikreymer/webarchiveplayer#auto-load-warcs
- - [Sheetsee-Pocket](http://jlord.us/sheetsee-pocket/) project that provides a pretty auto-updating index of your Pocket links (without archiving them)
- - [Pocket -> IFTTT -> Dropbox](https://christopher.su/2013/saving-pocket-links-file-day-dropbox-ifttt-launchd/) Post by Christopher Su on his Pocket saving IFTTT recipie
-
-## Changelog
-
- - support for chrome `--dump-dom` to output full page HTML after JS executes
- - v0.0.3 released
- - support for chrome `--user-data-dir` to archive sites that need logins
- - fancy individual html & json indexes for each link
- - smartly append new links to existing index instead of overwriting 
-  - v0.0.2 released
- - proper HTML templating instead of format strings (thanks to https://github.com/bardisty!)
- - refactored into separate files, wip audio & video archiving
- - v0.0.1 released
- - Index links now work without nginx url rewrites, archive can now be hosted on github pages
- - added setup.sh script & docstrings & help commands
- - made Chromium the default instead of Google Chrome (yay free software)
- - added [env-variable](https://github.com/pirate/bookmark-archiver/pull/25) configuration (thanks to https://github.com/hannah98!)
- - renamed from **Pocket Archive Stream** -> **Bookmark Archiver**
- - added [Netscape-format](https://github.com/pirate/bookmark-archiver/pull/20) export support (thanks to https://github.com/ilvar!)
- - added [Pinboard-format](https://github.com/pirate/bookmark-archiver/pull/7) export support (thanks to https://github.com/sconeyard!)
- - front-page of HN, oops! apparently I have users to support now :grin:?
- - added Pocket-format export support
- - v0.0.0 released: created Pocket Archive Stream 2017/05/05
- 
- ## Donations
- 
- This project can really flourish with some more engineering effort, but unless it can support   
- me financially I'm unlikely to be able to take it to the next level alone.  It's already pretty   
- functional and robust, but it really deserves to be taken to the next level with a few more   
- talented engineers.  If you or your foundation wants to sponsor this project long-term, contact  
- me at bookmark-archiver@sweeting.me.
-   
- [Grants / Donations](https://github.com/pirate/bookmark-archiver/blob/master/donate.md)
+<div align="center">
+<br/><br/>
+<img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" height="40px"/>
+<br/>
+<sub><i>This project is maintained mostly in my spare time with the help from generous contributors.</i></sub>
+<br/><br/>
+<a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
+<br/>
+<br/>
+<a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
+<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
+</div>
@@ -0,0 +1 @@
+theme: jekyll-theme-merlot
@@ -1 +1 @@
-bin/bookmark-archiver
+bin/archivebox
@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+# ArchiveBox
+# Nick Sweeting 2017 | MIT License
+# https://github.com/pirate/ArchiveBox
+
+import os
+import sys
+
+from datetime import datetime
+from peekable import Peekable
+
+
+from parse import parse_links
+from links import validate_links, links_after_timestamp
+from archive_methods import archive_link, _RESULTS_TOTALS
+from index import (
+    write_links_index,
+    parse_json_links_index,
+)
+from config import (
+    ARCHIVE_DIR,
+    ONLY_NEW,
+    OUTPUT_DIR,
+    REPO_DIR,
+    ANSI,
+    GIT_SHA,
+)
+from util import (
+    check_dependencies,
+    save_remote_source,
+    save_stdin_source,
+    pretty_path,
+    migrate_data,
+    check_links_structure,
+)
+
+__AUTHOR__ = 'Nick Sweeting <git@nicksweeting.com>'
+__VERSION__ = GIT_SHA
+__DESCRIPTION__ = 'ArchiveBox Usage:  Create a browsable html archive of a list of links.'
+__DOCUMENTATION__ = 'https://github.com/pirate/ArchiveBox/wiki'
+
+
+def print_help():
+    print(__DESCRIPTION__)
+    print("Documentation:     {}\n".format(__DOCUMENTATION__))
+    print("Usage:")
+    print("    ./bin/archivebox ~/Downloads/bookmarks_export.html\n")
+    print("")
+    print("    ./bin/archivebox https://example.com/feed.rss\n")
+    print("")
+    print("    echo 'https://examplecom' | ./bin/archivebox\n")
+
+
+def load_links(archive_path=OUTPUT_DIR, import_path=None):
+    """get new links from file and optionally append them to links in existing archive"""
+
+    existing_links = []
+    if archive_path:
+        existing_links = parse_json_links_index(archive_path)
+        check_links_structure(existing_links)
+
+    new_links = []
+    if import_path:
+        # parse and validate the import file
+        raw_links, parser_name = parse_links(import_path)
+        new_links = validate_links(raw_links)
+        check_links_structure(new_links)
+
+    # merge existing links in archive_path and new links
+    all_links = validate_links(existing_links + new_links)
+    check_links_structure(all_links)
+    num_new_links = len(all_links) - len(existing_links)
+
+    if import_path and parser_name:
+        print('    > Adding {} new links to index (parsed import as {})'.format(
+            num_new_links,
+            parser_name,
+        ))
+
+    return all_links, new_links
+
+
+def update_archive(archive_path, links, source=None, resume=None, append=True):
+    """update or create index.html+json given a path to an export file containing new links"""
+
+    start_ts = datetime.now().timestamp()
+
+    if resume:
+        print('{green}[▶] [{}] Resuming archive downloading from {}...{reset}'.format(
+             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+             resume,
+             **ANSI,
+        ))
+    else:
+        print('{green}[▶] [{}] Updating content for {} pages in archive...{reset}'.format(
+             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+             len(links),
+             **ANSI,
+        ))
+
+    check_links_structure(links)
+
+    # prefetch the first link off the generator so that if we pause or fail
+    # immediately we can show that we paused on the first link and not just None
+    to_archive = Peekable(links_after_timestamp(links, resume))
+    idx, link = 0, to_archive.peek(0)
+
+    # loop over links and archive them
+    try:
+        check_dependencies()
+        for idx, link in enumerate(to_archive):
+            link_dir = os.path.join(ARCHIVE_DIR, link['timestamp'])
+            archive_link(link_dir, link)
+
+    except (KeyboardInterrupt, SystemExit, Exception) as e:
+        print('\n{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format(
+            **ANSI,
+            now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            idx=idx+1,
+            timestamp=link['timestamp'],
+            total=len(links),
+        ))
+        print('    To view your archive, open: {}/index.html'.format(OUTPUT_DIR.replace(REPO_DIR + '/', '')))
+        print('    Continue where you left off by running:')
+        print('        {} {}'.format(
+            pretty_path(sys.argv[0]),
+            link['timestamp'],
+        ))
+        if not isinstance(e, KeyboardInterrupt):
+            print()
+            raise e
+        raise SystemExit(1)
+
+    # print timing information & summary
+    end_ts = datetime.now().timestamp()
+    seconds = end_ts - start_ts
+    if seconds > 60:
+        duration = '{0:.2f} min'.format(seconds / 60, 2)
+    else:
+        duration = '{0:.2f} sec'.format(seconds, 2)
+
+    print('{}[√] [{}] Update of {} pages complete ({}){}'.format(
+        ANSI['green'],
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        len(links),
+        duration,
+        ANSI['reset'],
+    ))
+    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
+    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
+    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
+    print('    To view your archive, open: {}/index.html'.format(OUTPUT_DIR.replace(REPO_DIR + '/', '')))
+
+
+if __name__ == '__main__':
+    argc = len(sys.argv)
+
+    if set(sys.argv).intersection(('-h', '--help', 'help')):
+        print_help()
+        raise SystemExit(0)
+
+    migrate_data()
+
+    source = sys.argv[1] if argc > 1 else None  # path of links file to import
+    resume = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
+   
+    stdin_raw_text = ''
+
+    if not sys.stdin.isatty():
+        stdin_raw_text = sys.stdin.read()
+
+    if source and stdin_raw_text:
+        print(
+            '[X] You should pass either a path as an argument, '
+            'or pass a list of links via stdin, but not both.\n'
+        )
+        print_help()
+        raise SystemExit(1)
+
+
+    if argc == 1:
+        source, resume = None, None
+    elif argc == 2:
+        if all(d.isdigit() for d in sys.argv[1].split('.')):
+            # argv[1] is a resume timestamp
+            source, resume = None, sys.argv[1]
+        else:
+            # argv[1] is a path to a file to import
+            source, resume = sys.argv[1].strip(), None
+    elif argc == 3:
+        source, resume = sys.argv[1].strip(), sys.argv[2]
+    else:
+        print_help()
+        raise SystemExit(1)
+
+    # See if archive folder already exists
+    for out_dir in (OUTPUT_DIR, 'bookmarks', 'pocket', 'pinboard', 'html'):
+        if os.path.exists(out_dir):
+            break
+    else:
+        out_dir = OUTPUT_DIR
+
+    # Step 0: Download url to local file (only happens if a URL is specified instead of local path) 
+    if source and any(source.startswith(s) for s in ('http://', 'https://', 'ftp://')):
+        source = save_remote_source(source)
+    elif stdin_raw_text:
+        source = save_stdin_source(stdin_raw_text)
+
+    # Step 1: Parse the links and dedupe them with existing archive
+    all_links, new_links = load_links(archive_path=out_dir, import_path=source)
+
+    # Step 2: Write new index
+    write_links_index(out_dir=out_dir, links=all_links)
+
+    # Step 3: Run the archive methods for each link
+    if ONLY_NEW:
+        update_archive(out_dir, new_links, source=source, resume=resume, append=True)
+    else:
+        update_archive(out_dir, all_links, source=source, resume=resume, append=True)
+
+    # Step 4: Re-write links index with updated titles, icons, and resources
+    all_links, _ = load_links(archive_path=out_dir)
+    write_links_index(out_dir=out_dir, links=all_links)
@@ -0,0 +1,639 @@
+import os
+
+from functools import wraps
+from collections import defaultdict
+from datetime import datetime
+
+from index import (
+    wget_output_path,
+    parse_json_link_index,
+    write_link_index,
+    patch_index_title_hack,
+)
+from config import (
+    OUTPUT_DIR,
+    CURL_BINARY,
+    GIT_BINARY,
+    WGET_BINARY,
+    YOUTUBEDL_BINARY,
+    CHROME_BINARY,
+    FETCH_FAVICON,
+    FETCH_TITLE,
+    FETCH_WGET,
+    FETCH_WGET_REQUISITES,
+    FETCH_PDF,
+    FETCH_SCREENSHOT,
+    FETCH_DOM,
+    FETCH_WARC,
+    FETCH_GIT,
+    FETCH_MEDIA,
+    RESOLUTION,
+    CHECK_SSL_VALIDITY,
+    SUBMIT_ARCHIVE_DOT_ORG,
+    COOKIES_FILE,
+    WGET_USER_AGENT,
+    CHROME_USER_DATA_DIR,
+    CHROME_SANDBOX,
+    TIMEOUT,
+    MEDIA_TIMEOUT,
+    ANSI,
+    ARCHIVE_DIR,
+    GIT_DOMAINS,
+    GIT_SHA,
+)
+from util import (
+    without_fragment,
+    fetch_page_title,
+    progress,
+    chmod_file,
+    pretty_path,
+    check_link_structure,
+    run, PIPE, DEVNULL,
+)
+
+
+_RESULTS_TOTALS = {   # globals are bad, mmkay
+    'skipped': 0,
+    'succeded': 0,
+    'failed': 0,
+}
+
+
+def archive_link(link_dir, link, overwrite=True):
+    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+
+    check_link_structure(link)
+
+    try:
+        update_existing = os.path.exists(link_dir)
+        if update_existing:
+            link = {
+                **parse_json_link_index(link_dir),
+                **link,
+            }
+        else:
+            os.makedirs(link_dir)
+        
+        print_link_status_line(link_dir, link, update_existing)
+
+        if FETCH_FAVICON:
+            link = fetch_favicon(link_dir, link, overwrite=overwrite)
+
+        if FETCH_TITLE:
+            link = fetch_title(link_dir, link, overwrite=overwrite)
+
+        if FETCH_WGET:
+            link = fetch_wget(link_dir, link, overwrite=overwrite)
+
+        if FETCH_PDF:
+            link = fetch_pdf(link_dir, link, overwrite=overwrite)
+
+        if FETCH_SCREENSHOT:
+            link = fetch_screenshot(link_dir, link, overwrite=overwrite)
+
+        if FETCH_DOM:
+            link = fetch_dom(link_dir, link, overwrite=overwrite)
+
+        if SUBMIT_ARCHIVE_DOT_ORG:
+            link = archive_dot_org(link_dir, link, overwrite=overwrite)
+
+        if FETCH_GIT:
+            link = fetch_git(link_dir, link, overwrite=overwrite)
+
+        if FETCH_MEDIA:
+            link = fetch_media(link_dir, link, overwrite=overwrite)
+
+        write_link_index(link_dir, link)
+
+    except Exception as err:
+        print('    ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err))
+    
+    return link
+
+def print_link_status_line(link_dir, link, update_existing):
+    print('[{symbol_color}{symbol}{reset}] [{now}] "{title}"\n    {blue}{url}{reset}'.format(
+        symbol='*' if update_existing else '+',
+        symbol_color=ANSI['black' if update_existing else 'green'],
+        now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        **{**link, 'title': link['title'] or link['url']},
+        **ANSI,
+    ))
+
+    print('    > {}{}'.format(pretty_path(link_dir), '' if update_existing else ' (new)'))
+    # if link['type']:
+    #     print('      i {}'.format(link['type']))
+
+
+
+def attach_result_to_link(method):
+    """
+    Instead of returning a result={output:'...', status:'success'} object,
+    attach that result to the links's history & latest fields, then return
+    the updated link object.
+    """
+    def decorator(fetch_func):
+        @wraps(fetch_func)
+        def timed_fetch_func(link_dir, link, overwrite=False, **kwargs):
+            # initialize methods and history json field on link
+            link['latest'] = link.get('latest') or {}
+            link['latest'][method] = link['latest'].get(method) or None
+            link['history'] = link.get('history') or {}
+            link['history'][method] = link['history'].get(method) or []
+
+            start_ts = datetime.now().timestamp()
+
+            # if a valid method output is already present, dont run the fetch function
+            if link['latest'][method] and not overwrite:
+                print('      √ {}'.format(method))
+                result = None
+            else:
+                print('      > {}'.format(method))
+                result = fetch_func(link_dir, link, **kwargs)
+
+            end_ts = datetime.now().timestamp()
+            duration = str(end_ts * 1000 - start_ts * 1000).split('.')[0]
+
+            # append a history item recording fail/success
+            history_entry = {
+                'timestamp': str(start_ts).split('.')[0],
+            }
+            if result is None:
+                history_entry['status'] = 'skipped'
+            elif isinstance(result.get('output'), Exception):
+                history_entry['status'] = 'failed'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+            else:
+                history_entry['status'] = 'succeded'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+                link['latest'][method] = result['output']
+
+            _RESULTS_TOTALS[history_entry['status']] += 1
+            
+            return link
+        return timed_fetch_func
+    return decorator
+
+
+@attach_result_to_link('wget')
+def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, warc=FETCH_WARC, timeout=TIMEOUT):
+    """download full site using wget"""
+
+    domain_dir = os.path.join(link_dir, link['domain'])
+    existing_file = wget_output_path(link)
+    if os.path.exists(domain_dir) and existing_file:
+        return {'output': existing_file, 'status': 'skipped'}
+
+    if warc:
+        warc_dir = os.path.join(link_dir, 'warc')
+        os.makedirs(warc_dir, exist_ok=True)
+        warc_path = os.path.join('warc', str(int(datetime.now().timestamp())))
+
+    # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
+    CMD = [
+        WGET_BINARY,
+        # '--server-response',  # print headers for better error parsing
+        '--no-verbose',
+        '--adjust-extension',
+        '--convert-links',
+        '--force-directories',
+        '--backup-converted',
+        '--span-hosts',
+        '--no-parent',
+        '-e', 'robots=off',
+        '--restrict-file-names=unix',
+        '--timeout={}'.format(timeout),
+        *(() if warc else ('--timestamping',)),
+        *(('--warc-file={}'.format(warc_path),) if warc else ()),
+        *(('--page-requisites',) if FETCH_WGET_REQUISITES else ()),
+        *(('--user-agent={}'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
+        *(('--load-cookies', COOKIES_FILE) if COOKIES_FILE else ()),
+        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate', '--no-hsts'))),
+        link['url'],
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # index.html
+        end()
+        output = wget_output_path(link, look_in=domain_dir)
+
+        output_tail = ['          ' + line for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] if line.strip()]
+
+        # parse out number of files downloaded from "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
+        files_downloaded = (
+            int(output_tail[-1].strip().split(' ', 2)[1] or 0)
+            if 'Downloaded:' in output_tail[-1]
+            else 0
+        )
+
+        # Check for common failure cases
+        if result.returncode > 0 and files_downloaded < 1:
+            print('        Got wget response code {}:'.format(result.returncode))
+            print('\n'.join(output_tail))
+            if b'403: Forbidden' in result.stderr:
+                raise Exception('403 Forbidden (try changing WGET_USER_AGENT)')
+            if b'404: Not Found' in result.stderr:
+                raise Exception('404 Not Found')
+            if b'ERROR 500: Internal Server Error' in result.stderr:
+                raise Exception('500 Internal Server Error')
+            raise Exception('Got an error from the server')
+    except Exception as e:
+        end()
+
+        # to let the user copy-paste the command and run it safely we have
+        # to quote some of the arguments that could have spaces in them
+        quoted_cmd = ' '.join(CMD)
+        quoted_cmd = quoted_cmd.replace(WGET_USER_AGENT, '"{}"'.format(WGET_USER_AGENT))
+        if COOKIES_FILE:
+            quoted_cmd = quoted_cmd.replace(COOKIES_FILE, '"{}"'.format(COOKIES_FILE))
+
+        print('        {}Some resources were skipped: {}{}'.format(ANSI['lightyellow'], e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(quoted_cmd))
+        output = e
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('pdf')
+def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
+    """print PDF of site to file using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': wget_output_path(link)}
+    
+    if os.path.exists(os.path.join(link_dir, 'output.pdf')):
+        return {'output': 'output.pdf', 'status': 'skipped'}
+
+    CMD = [
+        *chrome_headless(user_data_dir=user_data_dir),
+        '--print-to-pdf',
+        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.pdf
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to print PDF')
+        chmod_file('output.pdf', cwd=link_dir)
+        output = 'output.pdf'
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('screenshot')
+def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR, resolution=RESOLUTION):
+    """take screenshot of site using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': wget_output_path(link)}
+    
+    if os.path.exists(os.path.join(link_dir, 'screenshot.png')):
+        return {'output': 'screenshot.png', 'status': 'skipped'}
+
+    CMD = [
+        *chrome_headless(user_data_dir=user_data_dir),
+        '--screenshot',
+        '--window-size={}'.format(resolution),
+        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
+        # '--full-page',   # TODO: make this actually work using ./bin/screenshot fullPage: true
+        link['url'],
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # sreenshot.png
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to take screenshot')
+        chmod_file('screenshot.png', cwd=link_dir)
+        output = 'screenshot.png'
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+    
+@attach_result_to_link('dom')
+def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
+    """print HTML of site to file using chrome --dump-html"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': wget_output_path(link)}
+    
+    output_path = os.path.join(link_dir, 'output.html')
+
+    if os.path.exists(output_path):
+        return {'output': 'output.html', 'status': 'skipped'}
+
+    CMD = [
+        *chrome_headless(user_data_dir=user_data_dir),
+        '--dump-dom',
+        '--timeout={}'.format((timeout) * 1000),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        with open(output_path, 'w+') as f:
+            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.html
+        end()
+        if result.returncode:
+            print('     ', (result.stderr).decode())
+            raise Exception('Failed to fetch DOM')
+        chmod_file('output.html', cwd=link_dir)
+        output = 'output.html'
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('archive_org')
+def archive_dot_org(link_dir, link, timeout=TIMEOUT):
+    """submit site to archive.org for archiving via their service, save returned archive url"""
+
+    path = os.path.join(link_dir, 'archive.org.txt')
+    if os.path.exists(path):
+        archive_org_url = open(path, 'r').read().strip()
+        return {'output': archive_org_url, 'status': 'skipped'}
+
+    submit_url = 'https://web.archive.org/save/{}'.format(link['url'])
+
+    success = False
+    CMD = [
+        CURL_BINARY,
+        '--location',
+        '--head',
+        '--user-agent', 'ArchiveBox/{} (+https://github.com/pirate/ArchiveBox/)'.format(GIT_SHA),
+        '--max-time', str(timeout),
+        '--get',
+        *(() if CHECK_SSL_VALIDITY else ('--insecure',)),
+        submit_url,
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # archive.org.txt
+        end()
+
+        # Parse archive.org response headers
+        headers = defaultdict(list)
+
+        # lowercase all the header names and store in dict
+        for header in result.stdout.splitlines():
+            if b':' not in header or not header.strip():
+                continue
+            name, val = header.decode().split(':', 1)
+            headers[name.lower().strip()].append(val.strip())
+
+        # Get successful archive url in "content-location" header or any errors
+        content_location = headers['content-location']
+        errors = headers['x-archive-wayback-runtime-error']
+
+        if content_location:
+            saved_url = 'https://web.archive.org{}'.format(content_location[0])
+            success = True
+        elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]:
+            output = submit_url
+            # raise Exception('Archive.org denied by {}/robots.txt'.format(link['domain']))
+        elif errors:
+            raise Exception(', '.join(errors))
+        else:
+            raise Exception('Failed to find "content-location" URL header in Archive.org response.')
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    if success:
+        with open(os.path.join(link_dir, 'archive.org.txt'), 'w', encoding='utf-8') as f:
+            f.write(saved_url)
+        chmod_file('archive.org.txt', cwd=link_dir)
+        output = saved_url
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('favicon')
+def fetch_favicon(link_dir, link, timeout=TIMEOUT):
+    """download site favicon from google's favicon api"""
+
+    if os.path.exists(os.path.join(link_dir, 'favicon.ico')):
+        return {'output': 'favicon.ico', 'status': 'skipped'}
+
+    CMD = [
+        CURL_BINARY,
+        '--max-time', str(timeout),
+        *(() if CHECK_SSL_VALIDITY else ('--insecure',)),
+        'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
+    ]
+    fout = open('{}/favicon.ico'.format(link_dir), 'w')
+    end = progress(timeout, prefix='      ')
+    try:
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # favicon.ico
+        fout.close()
+        end()
+        chmod_file('favicon.ico', cwd=link_dir)
+        output = 'favicon.ico'
+    except Exception as e:
+        fout.close()
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('title')
+def fetch_title(link_dir, link, timeout=TIMEOUT):
+    """try to guess the page's title from its content"""
+
+    # if link already has valid title, skip it
+    if link['title'] and not link['title'].lower().startswith('http'):
+        return {'output': link['title'], 'status': 'skipped'}
+
+    end = progress(timeout, prefix='      ')
+    try:
+        title = fetch_page_title(link['url'], timeout=timeout, progress=False)
+        end()
+        output = title
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    # titles should show up in the global index immediatley for better UX,
+    # do a hacky immediate replacement to add them in as we're archiving
+    # TODO: figure out how to do this without gnarly string replacement
+    if title:
+        link['title'] = title
+        patch_index_title_hack(link['url'], title)
+
+    return {
+        'cmd': 'fetch_page_title("{}")'.format(link['url']),
+        'output': output,
+    }
+
+@attach_result_to_link('media')
+def fetch_media(link_dir, link, timeout=MEDIA_TIMEOUT, overwrite=False):
+    """Download playlists or individual video, audio, and subtitles using youtube-dl"""
+
+
+    # import ipdb; ipdb.set_trace()
+    output = os.path.join(link_dir, 'media')
+    already_done = os.path.exists(output)  # and os.listdir(output)
+    if already_done and not overwrite:
+        return {'output': 'media', 'status': 'skipped'}
+
+    os.makedirs(output, exist_ok=True)
+    CMD = [
+        YOUTUBEDL_BINARY,
+        '--write-description',
+        '--write-info-json',
+        '--write-annotations',
+        '--yes-playlist',
+        '--write-thumbnail',
+        '--no-call-home',
+        '--no-check-certificate',
+        '--user-agent',
+        '--all-subs',
+        '--extract-audio',
+        '--keep-video',
+        '--ignore-errors',
+        '--geo-bypass',
+        '--audio-format', 'mp3',
+        '--audio-quality', '320K',
+        '--embed-thumbnail',
+        '--add-metadata',
+        *(() if CHECK_SSL_VALIDITY else ('--no-check-certificate',)),
+        link['url'],
+    ]
+
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=output, timeout=timeout + 1)  # audio/audio.mp3
+        chmod_file('media', cwd=link_dir)
+        output = 'media'
+        end()
+        if result.returncode:
+            if (b'ERROR: Unsupported URL' in result.stderr
+                or b'HTTP Error 404' in result.stderr
+                or b'HTTP Error 403' in result.stderr
+                or b'URL could be a direct video link' in result.stderr
+                or b'Unable to extract container ID' in result.stderr):
+                # These happen too frequently on non-media pages to warrant printing to console
+                pass
+            else:
+                print('        got youtubedl response code {}:'.format(result.returncode))
+                print(result.stderr)
+                raise Exception('Failed to download media')
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('git')
+def fetch_git(link_dir, link, timeout=TIMEOUT):
+    """download full site using git"""
+
+    if not (link['domain'] in GIT_DOMAINS
+            or link['url'].endswith('.git')
+            or link['type'] == 'git'):
+        return
+
+    if os.path.exists(os.path.join(link_dir, 'git')):
+        return {'output': 'git', 'status': 'skipped'}
+
+    CMD = [
+        GIT_BINARY,
+        *(() if CHECK_SSL_VALIDITY else ('-c', 'http.sslVerify=false')),
+        'clone',
+        '--mirror',
+        '--recursive',
+        without_fragment(link['url']),
+    ]
+    output = 'git'
+
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # git/<reponame>
+        end()
+
+        if result.returncode > 0:
+            print('        got git response code {}:'.format(result.returncode))
+            raise Exception('Failed git download')
+    except Exception as e:
+        end()
+        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+def chrome_headless(binary=CHROME_BINARY, user_data_dir=CHROME_USER_DATA_DIR):
+    args = [binary, '--headless']  # '--disable-gpu'
+    if not CHROME_SANDBOX:
+        args.append('--no-sandbox')
+    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome')
+    if user_data_dir:
+        args.append('--user-data-dir={}'.format(user_data_dir))
+    elif os.path.exists(default_profile):
+        args.append('--user-data-dir={}'.format(default_profile))
+    return args
@@ -0,0 +1,144 @@
+import os
+import sys
+import shutil
+
+from subprocess import run, PIPE
+
+# ******************************************************************************
+# * TO SET YOUR CONFIGURATION, EDIT THE VALUES BELOW, or use the 'env' command *
+# * e.g.                                                                       *
+# * env USE_COLOR=True CHROME_BINARY=google-chrome ./archive.py export.html    *
+# ******************************************************************************
+
+IS_TTY = sys.stdout.isatty()
+USE_COLOR =              os.getenv('USE_COLOR',              str(IS_TTY)        ).lower() == 'true'
+SHOW_PROGRESS =          os.getenv('SHOW_PROGRESS',          str(IS_TTY)        ).lower() == 'true'
+ONLY_NEW =               os.getenv('ONLY_NEW',               'False'            ).lower() == 'true'
+MEDIA_TIMEOUT =          int(os.getenv('MEDIA_TIMEOUT',      '3600'))
+TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
+OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
+FOOTER_INFO =            os.getenv('FOOTER_INFO',            'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.',)
+
+FETCH_WGET =             os.getenv('FETCH_WGET',             'True'             ).lower() == 'true'
+FETCH_WGET_REQUISITES =  os.getenv('FETCH_WGET_REQUISITES',  'True'             ).lower() == 'true'
+FETCH_PDF =              os.getenv('FETCH_PDF',              'True'             ).lower() == 'true'
+FETCH_SCREENSHOT =       os.getenv('FETCH_SCREENSHOT',       'True'             ).lower() == 'true'
+FETCH_DOM =              os.getenv('FETCH_DOM',              'True'             ).lower() == 'true'
+FETCH_WARC =             os.getenv('FETCH_WARC',             'True'             ).lower() == 'true'
+FETCH_GIT =              os.getenv('FETCH_GIT',              'True'             ).lower() == 'true'
+FETCH_MEDIA =            os.getenv('FETCH_MEDIA',            'True'             ).lower() == 'true'
+FETCH_FAVICON =          os.getenv('FETCH_FAVICON',          'True'             ).lower() == 'true'
+FETCH_TITLE =            os.getenv('FETCH_TITLE',            'True'             ).lower() == 'true'
+SUBMIT_ARCHIVE_DOT_ORG = os.getenv('SUBMIT_ARCHIVE_DOT_ORG', 'True'             ).lower() == 'true'
+
+CHECK_SSL_VALIDITY =     os.getenv('CHECK_SSL_VALIDITY',     'True'             ).lower() == 'true'
+RESOLUTION =             os.getenv('RESOLUTION',             '1440,2000'        )
+GIT_DOMAINS =            os.getenv('GIT_DOMAINS',            'github.com,bitbucket.org,gitlab.com').split(',')
+WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',        'ArchiveBox/{GIT_SHA} (+https://github.com/pirate/ArchiveBox/) wget/{WGET_VERSION}')
+COOKIES_FILE =           os.getenv('COOKIES_FILE',           None)
+CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',   None)
+
+CURL_BINARY =            os.getenv('CURL_BINARY',            'curl')
+GIT_BINARY =             os.getenv('GIT_BINARY',             'git')
+WGET_BINARY =            os.getenv('WGET_BINARY',            'wget')
+YOUTUBEDL_BINARY =       os.getenv('YOUTUBEDL_BINARY',       'youtube-dl')
+CHROME_BINARY =          os.getenv('CHROME_BINARY',          None)
+
+### Paths
+REPO_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+
+OUTPUT_DIR = os.path.abspath(os.getenv('OUTPUT_DIR', os.path.join(REPO_DIR, 'output')))
+ARCHIVE_DIR = os.path.join(OUTPUT_DIR, 'archive')
+SOURCES_DIR = os.path.join(OUTPUT_DIR, 'sources')
+
+PYTHON_PATH = os.path.join(REPO_DIR, 'archivebox')
+TEMPLATES_DIR = os.path.join(PYTHON_PATH, 'templates')
+
+# ******************************************************************************
+# ********************** Do not edit below this point **************************
+# ******************************************************************************
+
+CHROME_SANDBOX =        os.getenv('CHROME_SANDBOX',         'True'             ).lower() == 'true'
+
+USE_CHROME = FETCH_PDF or FETCH_SCREENSHOT or FETCH_DOM
+USE_WGET = FETCH_WGET or FETCH_WGET_REQUISITES or FETCH_WARC
+
+if not CHROME_BINARY:
+    common_chrome_executable_names = (
+        'chromium-browser',
+        'chromium',
+        'google-chrome',
+        'google-chrome-stable',
+        'google-chrome-unstable',
+        'google-chrome-beta',
+        'google-chrome-canary',
+        'google-chrome-dev',
+    )
+    for name in common_chrome_executable_names:
+        full_path_exists = shutil.which(name)
+        if full_path_exists:
+            CHROME_BINARY = name
+            break
+    else:
+        CHROME_BINARY = 'chromium-browser'
+
+# print('[i] Using Chrome binary: {}'.format(shutil.which(CHROME_BINARY) or CHROME_BINARY))
+
+### Terminal Configuration
+TERM_WIDTH = shutil.get_terminal_size((100, 10)).columns
+ANSI = {
+    'reset': '\033[00;00m',
+    'lightblue': '\033[01;30m',
+    'lightyellow': '\033[01;33m',
+    'lightred': '\033[01;35m',
+    'red': '\033[01;31m',
+    'green': '\033[01;32m',
+    'blue': '\033[01;34m',
+    'white': '\033[01;37m',
+    'black': '\033[01;30m',
+}
+if not USE_COLOR:
+    # dont show colors if USE_COLOR is False
+    ANSI = {k: '' for k in ANSI.keys()}
+
+### Confirm Environment Setup
+GIT_SHA = 'unknown'
+try:
+    GIT_SHA = run([GIT_BINARY, 'rev-list', '-1', 'HEAD', './'], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+except Exception:
+    print('[!] Warning: unable to determine git version, is git installed and in your $PATH?')
+
+CHROME_VERSION = 'unknown'
+try:
+    chrome_vers_str = run([CHROME_BINARY, "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    CHROME_VERSION = [v for v in chrome_vers_str.strip().split(' ') if v.replace('.', '').isdigit()][0]
+except Exception:
+    if USE_CHROME:
+        print('[!] Warning: unable to determine chrome version, is chrome installed and in your $PATH?')
+
+WGET_VERSION = 'unknown'
+try:
+    wget_vers_str = run([WGET_BINARY, "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    WGET_VERSION = wget_vers_str.split('\n')[0].split(' ')[2]
+except Exception:
+    if USE_WGET:
+        print('[!] Warning: unable to determine wget version, is wget installed and in your $PATH?')
+
+WGET_USER_AGENT = WGET_USER_AGENT.format(GIT_SHA=GIT_SHA[:9], WGET_VERSION=WGET_VERSION)
+
+try:
+    COOKIES_FILE = os.path.abspath(COOKIES_FILE) if COOKIES_FILE else None
+except Exception:
+    print('[!] Warning: unable to get full path to COOKIES_FILE, are you sure you specified it correctly?')
+    raise
+
+if sys.stdout.encoding.upper() not in ('UTF-8', 'UTF8'):
+    print('[X] Your system is running python3 scripts with a bad locale setting: {} (it should be UTF-8).'.format(sys.stdout.encoding))
+    print('    To fix it, add the line "export PYTHONIOENCODING=UTF-8" to your ~/.bashrc file (without quotes)')
+    print('')
+    print('    Confirm that it\'s fixed by opening a new shell and running:')
+    print('        python3 -c "import sys; print(sys.stdout.encoding)"   # should output UTF-8')
+    print('')
+    print('    Alternatively, run this script with:')
+    print('        env PYTHONIOENCODING=UTF-8 ./archive.py export.html')
+
@@ -6,6 +6,7 @@ from string import Template
 from distutils.dir_util import copy_tree

 from config import (
+    OUTPUT_DIR,
    TEMPLATES_DIR,
    OUTPUT_PERMISSIONS,
    ANSI,
@@ -17,6 +18,8 @@ from util import (
    wget_output_path,
    derived_link_info,
    pretty_path,
+    check_link_structure,
+    check_links_structure,
 )


@@ -25,26 +28,32 @@ from util import (
 def write_links_index(out_dir, links):
    """create index.html file for a given list of links"""

+    check_links_structure(links)
+
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)

-    write_json_links_index(out_dir, links)
-    write_html_links_index(out_dir, links)
-    
-    print('{green}[√] [{}] Updated main index files:{reset}'.format(
+    print('{green}[*] [{}] Updating main index files...{reset}'.format(
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        **ANSI))
+        **ANSI,
+    ))
+    write_json_links_index(out_dir, links)
    print('    > {}/index.json'.format(pretty_path(out_dir)))
+    
+    write_html_links_index(out_dir, links)
    print('    > {}/index.html'.format(pretty_path(out_dir)))
+    

 def write_json_links_index(out_dir, links):
    """write the json link index to a given path"""

+    check_links_structure(links)
+
    path = os.path.join(out_dir, 'index.json')

    index_json = {
-        'info': 'Bookmark Archiver Index',
-        'help': 'https://github.com/pirate/bookmark-archiver',
+        'info': 'ArchiveBox Index',
+        'help': 'https://github.com/pirate/ArchiveBox',
        'version': GIT_SHA,
        'num_links': len(links),
        'updated': str(datetime.now().timestamp()),
@@ -61,17 +70,24 @@ def parse_json_links_index(out_dir):
    index_path = os.path.join(out_dir, 'index.json')
    if os.path.exists(index_path):
        with open(index_path, 'r', encoding='utf-8') as f:
-            return json.load(f)['links']
+            links = json.load(f)['links']
+            check_links_structure(links)
+            return links

    return []

 def write_html_links_index(out_dir, links):
    """write the html link index to a given path"""

+    check_links_structure(links)
+
    path = os.path.join(out_dir, 'index.html')

    copy_tree(os.path.join(TEMPLATES_DIR, 'static'), os.path.join(out_dir, 'static'))

+    with open(os.path.join(out_dir, 'robots.txt'), 'w+') as f:
+        f.write('User-agent: *\nDisallow: /')
+
    with open(os.path.join(TEMPLATES_DIR, 'index.html'), 'r', encoding='utf-8') as f:
        index_html = f.read()

@@ -99,6 +115,25 @@ def write_html_links_index(out_dir, links):
    chmod_file(path)


+def patch_index_title_hack(link_url, new_title):
+    """hack to update just one link's title in the link index json"""
+
+    json_path = os.path.join(OUTPUT_DIR, 'index.json')
+
+    links = parse_json_links_index(OUTPUT_DIR)
+
+    changed = False
+    for link in links:
+        if link['url'] == link_url:
+            link['title'] = new_title
+            changed = True
+            break
+
+    if changed:
+        write_json_links_index(OUTPUT_DIR, links)
+
+
+
 ### Individual link index

 def write_link_index(out_dir, link):
@@ -109,6 +144,7 @@ def write_link_index(out_dir, link):
 def write_json_link_index(out_dir, link):
    """write a json file with some info about the link"""
    
+    check_link_structure(link)
    path = os.path.join(out_dir, 'index.json')

    print('      √ index.json')
@@ -123,10 +159,13 @@ def parse_json_link_index(out_dir):
    existing_index = os.path.join(out_dir, 'index.json')
    if os.path.exists(existing_index):
        with open(existing_index, 'r', encoding='utf-8') as f:
-            return json.load(f)
+            link_json = json.load(f)
+            check_link_structure(link_json)
+            return link_json
    return {}

 def write_html_link_index(out_dir, link):
+    check_link_structure(link)
    with open(os.path.join(TEMPLATES_DIR, 'link_index_fancy.html'), 'r', encoding='utf-8') as f:
        link_html = f.read()

@@ -138,6 +177,7 @@ def write_html_link_index(out_dir, link):
        f.write(Template(link_html).substitute({
            **link,
            **link['latest'],
+            'title': link['title'] or link['url'],
            'type': link['type'] or 'website',
            'tags': link['tags'] or 'untagged',
            'bookmarked': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
@@ -1,5 +1,5 @@
 """
-In Bookmark Archiver, a Link represents a single entry that we track in the 
+In ArchiveBox, a Link represents a single entry that we track in the 
 json index.  All links pass through all archiver functions and the latest,
 most up-to-date canonical output for each is stored in "latest".

@@ -32,33 +32,34 @@ Link {

 """

-import datetime
 from html import unescape
+from collections import OrderedDict

 from util import (
-    domain,
-    base_url,
-    str_between,
-    get_link_type,
    merge_links,
    wget_output_path,
+    check_link_structure,
+    check_links_structure,
 )
-from config import ANSI


 def validate_links(links):
+    check_links_structure(links)
    links = archivable_links(links)  # remove chrome://, about:, mailto: etc.
    links = uniquefied_links(links)  # merge/dedupe duplicate timestamps & urls
    links = sorted_links(links)      # deterministically sort the links based on timstamp, url
-    
+
    if not links:
        print('[X] No links found :(')
        raise SystemExit(1)

    for link in links:
-        link['title'] = unescape(link['title'])
+        check_link_structure(link)
+
+        link['title'] = unescape(link['title']) if link['title'] else None
        link['latest'] = link.get('latest') or {}
-        
+
+        latest = link['latest']
        if not link['latest'].get('wget'):
            link['latest']['wget'] = wget_output_path(link)

@@ -71,6 +72,12 @@ def validate_links(links):
        if not link['latest'].get('dom'):
            link['latest']['dom'] = None

+        if not latest.get('favicon'):
+            latest['favicon'] = None
+
+        if not link['latest'].get('title'):
+            link['latest']['title'] = link['title']
+
    return list(links)


@@ -79,15 +86,16 @@ def archivable_links(links):
    return (
        link
        for link in links
-        if any(link['url'].startswith(s) for s in ('http://', 'https://', 'ftp://'))
+        if any(link['url'].lower().startswith(s) for s in ('http://', 'https://', 'ftp://'))
    )

+
 def uniquefied_links(sorted_links):
    """
    ensures that all non-duplicate links have monotonically increasing timestamps
    """

-    unique_urls = {}
+    unique_urls = OrderedDict()

    lower = lambda url: url.lower().strip()
    without_www = lambda url: url.replace('://www.', '://', 1)
@@ -100,17 +108,19 @@ def uniquefied_links(sorted_links):
            link = merge_links(unique_urls[fuzzy_url], link)
        unique_urls[fuzzy_url] = link

-    unique_timestamps = {}
+    unique_timestamps = OrderedDict()
    for link in unique_urls.values():
        link['timestamp'] = lowest_uniq_timestamp(unique_timestamps, link['timestamp'])
        unique_timestamps[link['timestamp']] = link

    return unique_timestamps.values()

+
 def sorted_links(links):
-    sort_func = lambda link: (link['timestamp'], link['url'])
+    sort_func = lambda link: (link['timestamp'].split('.', 1)[0], link['url'])
    return sorted(links, key=sort_func, reverse=True)

+
 def links_after_timestamp(links, timestamp=None):
    if not timestamp:
        yield from links
@@ -123,6 +133,7 @@ def links_after_timestamp(links, timestamp=None):
        except (ValueError, TypeError):
            print('Resume value and all timestamp values must be valid numbers.')

+
 def lowest_uniq_timestamp(used_timestamps, timestamp):
    """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2"""

@@ -1,3 +1,5 @@
+# coding: utf-8
+
 """
 Everything related to parsing links from bookmark services.

@@ -6,60 +8,58 @@ For examples of supported files see examples/.

 Parsed link schema: {
    'url': 'https://example.com/example/?abc=123&xyc=345#lmnop',
-    'domain': 'example.com',
-    'base_url': 'example.com/example/',
    'timestamp': '15442123124234',
-    'tags': 'abc,def',
    'title': 'Example.com Page Title',
+    'tags': 'abc,def',
    'sources': ['ril_export.html', 'downloads/getpocket.com.txt'],
 }
 """

 import re
 import json
-import xml.etree.ElementTree as etree

 from datetime import datetime
+from collections import OrderedDict
+import xml.etree.ElementTree as etree

+from config import ANSI
 from util import (
-    domain,
-    base_url,
    str_between,
    get_link_type,
+    URL_REGEX,
+    check_url_parsing,
 )


-def get_parsers(file):
-    """return all parsers that work on a given file, defaults to all of them"""
-
-    return {
-        'pocket': parse_pocket_export,
-        'pinboard': parse_json_export,
-        'bookmarks': parse_bookmarks_export,
-        'rss': parse_rss_export,
-        'pinboard_rss': parse_pinboard_rss_feed,
-        'medium_rss': parse_medium_rss_feed,
-    }
-
 def parse_links(path):
    """parse a list of links dictionaries from a bookmark export file"""
    
+    check_url_parsing()
+
    links = []
    with open(path, 'r', encoding='utf-8') as file:
-        for parser_func in get_parsers(file).values():
-            # otherwise try all parsers until one works
+        print('{green}[*] [{}] Parsing new links from output/sources/{}...{reset}'.format(
+            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            path.rsplit('/', 1)[-1],
+            **ANSI,
+        ))
+
+        for parser_name, parser_func in PARSERS.items():
            try:
                links += list(parser_func(file))
                if links:
                    break
-            except (ValueError, TypeError, IndexError, AttributeError, etree.ParseError):
-                # parser not supported on this file
+            except Exception as err:
+                # we try each parser one by one, wong parsers will throw exeptions
+                # if unsupported and we accept the first one that passes
+                # uncomment the following line to see why the parser was unsupported for each attempted format
+                # print('[!] Parser {} failed: {} {}'.format(parser_name, err.__class__.__name__, err))
                pass

-    return links
+    return links, parser_name


-def parse_pocket_export(html_file):
+def parse_pocket_html_export(html_file):
    """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""

    html_file.seek(0)
@@ -73,19 +73,16 @@ def parse_pocket_export(html_file):
            time = datetime.fromtimestamp(float(match.group(2)))
            info = {
                'url': fixed_url,
-                'domain': domain(fixed_url),
-                'base_url': base_url(fixed_url),
                'timestamp': str(time.timestamp()),
                'tags': match.group(3),
-                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or base_url(fixed_url),
+                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or None,
                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

-def parse_json_export(json_file):
-    """Parse JSON-format bookmarks export files (produced by pinboard.in/export/)"""
-
+def parse_pinboard_json_export(json_file):
+    """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)"""
    json_file.seek(0)
    json_content = json.load(json_file)
    for line in json_content:
@@ -97,20 +94,29 @@ def parse_json_export(json_file):
                timestamp = str(erg['timestamp']/10000000)  # chrome/ff histories use a very precise timestamp
            elif erg.get('time'):
                timestamp = str(datetime.strptime(erg['time'].split(',', 1)[0], '%Y-%m-%dT%H:%M:%SZ').timestamp())
+            elif erg.get('created_at'):
+                timestamp = str(datetime.strptime(erg['created_at'], '%Y-%m-%dT%H:%M:%S%z').timestamp())
            else:
                timestamp = str(datetime.now().timestamp())
+            if erg.get('href'):
+                url = erg['href']
+            else:
+                url = erg['url']
+            if erg.get('description'):
+                title = (erg.get('description') or '').replace(' — Readability', '')
+            else:
+                title = erg['title'].strip()
            info = {
-                'url': erg['href'],
-                'domain': domain(erg['href']),
-                'base_url': base_url(erg['href']),
+                'url': url,
                'timestamp': timestamp,
                'tags': erg.get('tags') or '',
-                'title': (erg.get('description') or '').replace(' — Readability', ''),
+                'title': title or None,
                'sources': [json_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

+
 def parse_rss_export(rss_file):
    """Parse RSS XML-format files into links"""

@@ -131,27 +137,65 @@ def parse_rss_export(rss_file):
        rows = leading_removed.split('\n')

        def get_row(key):
-            return [r for r in rows if r.startswith('<{}>'.format(key))][0]
+            return [r for r in rows if r.strip().startswith('<{}>'.format(key))][0]

-        title = str_between(get_row('title'), '<![CDATA[', ']]')
+        title = str_between(get_row('title'), '<![CDATA[', ']]').strip()
        url = str_between(get_row('link'), '<link>', '</link>')
        ts_str = str_between(get_row('pubDate'), '<pubDate>', '</pubDate>')
        time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z")

        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
            'tags': '',
-            'title': title,
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)

        yield info

-def parse_bookmarks_export(html_file):
+
+def parse_shaarli_rss_export(rss_file):
+    """Parse Shaarli-specific RSS XML-format files into links"""
+
+    rss_file.seek(0)
+    entries = rss_file.read().split('<entry>')[1:]
+    for entry in entries:
+        # example entry:
+        # <entry>
+        #   <title>Aktuelle Trojaner-Welle: Emotet lauert in gefÃ¤lschten Rechnungsmails | heise online</title>
+        #   <link href="https://www.heise.de/security/meldung/Aktuelle-Trojaner-Welle-Emotet-lauert-in-gefaelschten-Rechnungsmails-4291268.html" />
+        #   <id>https://demo.shaarli.org/?cEV4vw</id>
+        #   <published>2019-01-30T06:06:01+00:00</published>
+        #   <updated>2019-01-30T06:06:01+00:00</updated>
+        #   <content type="html" xml:lang="en"><![CDATA[<div class="markdown"><p>&#8212; <a href="https://demo.shaarli.org/?cEV4vw">Permalink</a></p></div>]]></content>
+        # </entry>
+
+        trailing_removed = entry.split('</entry>', 1)[0]
+        leading_removed = trailing_removed.strip()
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0]
+
+        title = str_between(get_row('title'), '<title>', '</title>').strip()
+        url = str_between(get_row('link'), '<link href="', '" />')
+        ts_str = str_between(get_row('published'), '<published>', '</published>')
+        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
+
+        info = {
+            'url': url,
+            'timestamp': str(time.timestamp()),
+            'tags': '',
+            'title': title or None,
+            'sources': [rss_file.name],
+        }
+        info['type'] = get_link_type(info)
+
+        yield info
+
+def parse_netscape_html_export(html_file):
    """Parse netscape-format bookmarks export files (produced by all browsers)"""

    html_file.seek(0)
@@ -167,18 +211,16 @@ def parse_bookmarks_export(html_file):

            info = {
                'url': url,
-                'domain': domain(url),
-                'base_url': base_url(url),
                'timestamp': str(time.timestamp()),
                'tags': "",
-                'title': match.group(3),
+                'title': match.group(3).strip() or None,
                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)

            yield info

-def parse_pinboard_rss_feed(rss_file):
+def parse_pinboard_rss_export(rss_file):
    """Parse Pinboard RSS feed files into links"""

    rss_file.seek(0)
@@ -187,7 +229,7 @@ def parse_pinboard_rss_feed(rss_file):
    for item in items:
        url = item.find("{http://purl.org/rss/1.0/}link").text
        tags = item.find("{http://purl.org/dc/elements/1.1/}subject").text
-        title = item.find("{http://purl.org/rss/1.0/}title").text
+        title = item.find("{http://purl.org/rss/1.0/}title").text.strip()
        ts_str = item.find("{http://purl.org/dc/elements/1.1/}date").text
        #       = 🌈🌈🌈🌈
        #        = 🌈🌈🌈🌈
@@ -200,17 +242,15 @@ def parse_pinboard_rss_feed(rss_file):
        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
            'tags': tags,
-            'title': title,
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)
        yield info

-def parse_medium_rss_feed(rss_file):
+def parse_medium_rss_export(rss_file):
    """Parse Medium RSS feed files into links"""

    rss_file.seek(0)
@@ -220,17 +260,49 @@ def parse_medium_rss_feed(rss_file):
        # for child in item:
        #     print(child.tag, child.text)
        url = item.find("link").text
-        title = item.find("title").text
+        title = item.find("title").text.strip()
        ts_str = item.find("pubDate").text
        time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %Z")
        info = {
            'url': url,
-            'domain': domain(url),
-            'base_url': base_url(url),
            'timestamp': str(time.timestamp()),
-            'tags': "",
-            'title': title,
+            'tags': '',
+            'title': title or None,
            'sources': [rss_file.name],
        }
        info['type'] = get_link_type(info)
        yield info
+
+
+def parse_plain_text_export(text_file):
+    """Parse raw links from each line in a text file"""
+
+    text_file.seek(0)
+    text_content = text_file.readlines()
+    for line in text_content:
+        if line:
+            urls = re.findall(URL_REGEX, line)
+            
+            for url in urls:
+                url = url.strip()
+                info = {
+                    'url': url,
+                    'timestamp': str(datetime.now().timestamp()),
+                    'tags': '',
+                    'title': None,
+                    'sources': [text_file.name],
+                }
+                info['type'] = get_link_type(info)
+                yield info
+
+
+PARSERS = OrderedDict([
+    ('Pocket HTML', parse_pocket_html_export),
+    ('Pinboard JSON', parse_pinboard_json_export),
+    ('Netscape HTML', parse_netscape_html_export),
+    ('RSS', parse_rss_export),
+    ('Pinboard RSS', parse_pinboard_rss_export),
+    ('Shaarli RSS', parse_shaarli_rss_export),
+    ('Medium RSS', parse_medium_rss_export),
+    ('Plain Text', parse_plain_text_export),
+])
@@ -1,3 +1,5 @@
+from sys import maxsize
+from itertools import islice
 from collections import deque

 _marker = object()
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+
+import re
+from argparse import ArgumentParser
+from os.path import exists, join
+from shutil import rmtree
+from typing import List
+
+from archive import parse_json_link_index
+from config import ARCHIVE_DIR, OUTPUT_DIR
+from index import write_html_links_index, write_json_links_index
+
+
+def cleanup_index(regexes: List[str], proceed: bool, delete: bool) -> None:
+    if not exists(join(OUTPUT_DIR, 'index.json')):
+        exit('index.json is missing; nothing to do')
+
+    compiled = [re.compile(r) for r in regexes]
+    links = parse_json_link_index(OUTPUT_DIR)['links']
+    filtered = []
+    remaining = []
+
+    for l in links:
+        url = l['url']
+        for r in compiled:
+            if r.search(url):
+                filtered.append((l, r))
+                break
+        else:
+            remaining.append(l)
+
+    if not filtered:
+        exit('Search did not match any entries.')
+
+    print('Filtered out {}/{} urls:'.format(len(filtered), len(links)))
+
+    for link, regex in filtered:
+        url = link['url']
+        print(' {url} via {regex}'.format(url=url, regex=regex.pattern))
+
+    if not proceed:
+        answer = input('Remove {} entries from index? [y/n] '.format(
+            len(filtered)))
+        proceed = answer.strip().lower() in ('y', 'yes')
+
+    if not proceed:
+        exit('Aborted')
+
+    write_json_links_index(OUTPUT_DIR, remaining)
+    write_html_links_index(OUTPUT_DIR, remaining)
+
+    if delete:
+        for link, _ in filtered:
+            data_dir = join(ARCHIVE_DIR, link['timestamp'])
+            if exists(data_dir):
+                rmtree(data_dir)
+
+
+if __name__ == '__main__':
+    p = ArgumentParser('Index purging tool')
+    p.add_argument(
+        '--regex',
+        '-r',
+        action='append',
+        help='Regular expression matching URLs to purge',
+    )
+    p.add_argument(
+        '--delete',
+        '-d',
+        action='store_true',
+        default=False,
+        help='Delete webpage files from archive',
+    )
+    p.add_argument(
+        '--yes',
+        '-y',
+        action='store_true',
+        default=False,
+        help='Do not prompt for confirmation',
+    )
+
+    args = p.parse_args()
+    if args.regex:
+        cleanup_index(args.regex, proceed=args.yes, delete=args.delete)
+    else:
+        p.print_help()
@@ -0,0 +1,162 @@
+<html lang="en">
+    <head>
+        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
+        <title>Archived Sites</title>
+        <style>
+            html, body {
+                width: 100%;
+                height: 100%;
+                font-size: 18px;
+                font-weight: 200;
+                text-align: center;
+                margin: 0px;
+                padding: 0px;
+                font-family: "Gill Sans", Helvetica, sans-serif;
+            }
+            header {
+                background-color: #aa1e55;
+                color: #1a1a1a;
+                padding: 10px;
+                padding-top: 0px;
+                padding-bottom: 15px;
+                /*height: 40px;*/
+            }
+            header h1 {
+                margin: 7px 0px;
+                font-size: 35px;
+                font-weight: 300;
+                color: #1a1a1a;
+            }
+            header h1 img {
+                height: 44px;
+                vertical-align: bottom;
+            }
+            header a {
+                text-decoration: none !important;
+                color: #1a1a1a;
+            }
+            .header-center {
+                margin: auto;
+                float: none;
+                text-align: center;
+                padding-top: 6px;
+            }
+            .header-center small {
+                color: #eaeaea;
+                opacity: 0.7;
+            }
+            .header-left {
+                float: left;
+            }
+            .header-right {
+                float: right;
+                padding-top: 17px;
+                padding-right: 10px;
+            }
+            header + div {
+                margin-top: 10px;
+            }
+            #table-bookmarks_length, #table-bookmarks_filter {
+                padding: 0px 15px;
+            }
+            table {
+                padding: 6px;
+                width: 100%;
+            }
+            table thead th {
+                font-weight: 400;
+            }
+            table tr {
+                height: 35px;
+            }
+            tbody tr:nth-child(odd) {
+               background-color: #ffebeb !important;
+            }
+            table tr td {
+                white-space: nowrap;
+                overflow: hidden;
+                /*padding-bottom: 0.4em;*/
+                /*padding-top: 0.4em;*/
+                padding-left: 2px;
+                text-align: center;
+            }
+            table tr td a {
+                text-decoration: none;
+            }
+            table tr td img, table tr td object {
+                display: inline-block;
+                margin: auto;
+                height: 24px;
+                width: 24px;
+                padding: 0px;
+                padding-right: 5px;
+                vertical-align: middle;
+                margin-left: 4px;
+            }
+            #table-bookmarks {
+                width: 100%; 
+                overflow-y: scroll;
+                table-layout: fixed;
+            }
+        </style>
+    </head>
+    <body>
+        <header>
+            <div class="header-right">
+                <a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> &nbsp; | &nbsp; 
+                <a href="https://github.com/pirate/ArchiveBox">Source</a> &nbsp; | &nbsp; 
+                <a href="https://archivebox.io">Website</a>
+            </div>
+            <div class="header-left">
+                <a href="?" title="Reload...">
+                    <h1><img src="static/archive.png"/> ArchiveBox: Index</h1>
+                </a>
+            </div>
+            <div class="header-center">
+                Archived Sites<br/>
+                <small>Last updated $time_updated</small>
+            </div>
+        </header>
+        <table id="table-bookmarks">
+            <thead>
+                <tr>
+                    <th style="width: 80px;">Bookmarked</th>
+                    <th style="width: 26px;">Files</th>
+                    <th style="width: 26vw;">Saved Link ($num_links)</th>
+                    <th style="width: 30px;">PNG</th>
+                    <th style="width: 30px">PDF</th>
+                    <th style="width: 30px">HTML</th>
+                    <th style="width: 30px">A.org</th>
+                    <th style="width: 16vw;whitespace:nowrap;overflow-x:hidden;">Original URL</th>
+                </tr>
+            </thead>
+            <tbody>$rows</tbody>
+        </table>
+        <footer>
+            <br/>
+            <center>
+                <small>
+                    Archive created using <a href="https://github.com/pirate/ArchiveBox" title="Github">ArchiveBox</a>
+                    version <a href="https://github.com/pirate/ArchiveBox/commit/$git_sha" title="Git commit">$short_git_sha</a> &nbsp; | &nbsp; 
+                    Download index as <a href="index.json" title="JSON summary of archived links.">JSON</a>
+                    <br/><br/>
+                    $footer_info
+                </small>
+            </center>
+            <br/>
+        </footer>
+        <link rel="stylesheet" href="static/jquery.dataTables.min.css"/>
+        <script src="static/jquery.min.js"></script>
+        <script src="static/jquery.dataTables.min.js"></script>
+        <script>
+            jQuery(document).ready(function() {
+                jQuery('#table-bookmarks').DataTable({
+                    stateSave: true, // save state (filtered input, number of entries shown, etc) in localStorage
+                    dom: '<lf<t>ip>', // how to show the table and its helpers (filter, etc) in the DOM
+                    order: [[0, 'desc']],
+                    iDisplayLength: 100,
+                });
+            });
+        </script>
+    </body>
+</html>
@@ -1,17 +1,16 @@
 <tr>
    <td title="Bookmarked timestamp: $timestamp">$date</td>
    <td>
-        <a href="?" title="Refresh status...">
+        <a href="$files_url" title="Link Index">
            <img src="$favicon_url" onerror="this.src='static/spinner.gif'" class="link-favicon">
        </a>
    </td>
    <td style="text-align: left"><a href="$archive_url" style="font-size:1.4em;text-decoration:none;color:black;" title="$title">
        $title <small style="background-color: #eee;border-radius:4px; float:right">$tags</small>
    </td>
-    <td><a href="$files_url" title="Files">📂</a></td>
-    <td><a href="$dom_link" title="HTML">📄</a></td>
-    <td><a href="$pdf_link" title="PDF">📜</a></td>
    <td><a href="$screenshot_link" title="Screenshot">🖼</a></td>
+    <td><a href="$pdf_link" title="PDF">📜</a></td>
+    <td><a href="$dom_link" title="DOM">📄</a></td>
    <td><a href="$archive_org_url" title="Archive.org">🏛</a></td>
    <td style="text-align: left"><!--🔗 <img src="$google_favicon_url" height="16px">--> <a href="$url">$url</a></td>
 </tr>
@@ -56,7 +56,7 @@
            <hr/>
            <a href="./../../index.html" class="nav-icon" title="Archived Sites">
                <img src="https://nicksweeting.com/images/archive.png" alt="Archive Icon" height="20px">
-                Bookmark Archiver: Link Index
+                ArchiveBox: Link Index
            </a>
        </footer>
    </body>
@@ -27,7 +27,7 @@
                font-size: calc(16px + 1vw);
            }
            .collapse-icon {
-                float: left;
+                float: right;
                color: black;
                width: 126px;
                font-size: 0.8em;
@@ -36,12 +36,12 @@
                margin-left: -35px;
            }
            .nav-icon img {
-                float: right;
+                float: left;
                display: block;
                margin-right: 13px;
                color: black;
                height: 53px;
-                margin-top: 7px;
+                margin-top: 12px;
                margin-left: 10px;
            }
            .nav-icon img:hover {
@@ -152,12 +152,7 @@
                }
            }
        </style>
-        <script
-          src="https://code.jquery.com/jquery-3.2.1.slim.min.js"
-          integrity="sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
-          crossorigin="anonymous"></script>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/css/bootstrap.min.css" integrity="sha384-rwoIResjU2yc3z8GV/NPeZWAv56rSmLldC3R/AZzGRnGxQQKnKkoFVhFQhNUwEyJ" crossorigin="anonymous">
-        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/js/bootstrap.min.js" integrity="sha384-vBWWzlZJ8ea9aCX4pEW3rVHjgjt7zpkNpZk+02D9phzyeVkE+jo0ieGizqPLForn" crossorigin="anonymous"></script>
    </head>
    <body>
        <header>
@@ -177,7 +172,7 @@
        <div class="site-header container-fluid">
            <div class="row archive-page-header">
                <div class="col-lg-4 alert well">
-                    Added: <small title="Timestamp: $bookmarked_ts">$bookmarked</small>
+                    Bookmarked: <small title="Timestamp: $bookmarked_ts">$bookmarked</small>
                    &nbsp; | &nbsp;
                    Last updated: <small title="Timestamp: $updated_ts">$updated</small>
                </div>
@@ -191,7 +186,9 @@
                <div class="col-lg-4 alert well">
                    Download:
                    <a href="index.json" title="JSON summary of archived link.">JSON</a> | 
-                    <a href="." title="Webserver-provided index of files directory.">Files</a>
+                    <a href="." title="Webserver-provided index of files directory.">Files</a> | 
+                    <a href="media/" title="Audio, Video, and Subtitle files.">Media</a> | 
+                    <a href="git/" title="Any git repos at the url">Git Code</a>
                </div>
                <hr/>
                <div class="col-lg-2">
@@ -269,48 +266,54 @@
            </div>
        </div>
        <iframe sandbox="allow-same-origin allow-scripts allow-forms" class="full-page-iframe" src="$wget" name="preview"></iframe>
+    
+        <script
+              src="https://code.jquery.com/jquery-3.2.1.slim.min.js"
+              integrity="sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
+              crossorigin="anonymous"></script>
+        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/js/bootstrap.min.js" integrity="sha384-vBWWzlZJ8ea9aCX4pEW3rVHjgjt7zpkNpZk+02D9phzyeVkE+jo0ieGizqPLForn" crossorigin="anonymous"></script>
+
+        <script>
+            // show selected file in iframe when preview card is clicked
+            jQuery('.card').on('click', function(e) {
+                jQuery('.selected-card').removeClass('selected-card')
+                jQuery(e.target).closest('.card').addClass('selected-card')
+            })
+            jQuery('.card a[target=preview]').on('click', function(e) {
+                if (e.currentTarget.href.endsWith('.pdf')) {
+                    jQuery('.full-page-iframe')[0].removeAttribute('sandbox')
+                } else {
+                    jQuery('.full-page-iframe')[0].sandbox = "allow-same-origin allow-scripts allow-forms"
+                }
+                return true
+            })
+
+            // un-sandbox iframes showing pdfs (required to display pdf viewer)
+            jQuery('iframe').map(function() {
+                if (this.src.endsWith('.pdf')) {
+                    this.removeAttribute('sandbox')
+                    this.src = this.src
+                }
+            })
+
+            // hide header when collapse icon is clicked
+            jQuery('.collapse-icon').on('click', function() {
+                if (jQuery('.collapse-icon').text().includes('▾')) {
+                    jQuery('.collapse-icon').text('▸')
+                    jQuery('.site-header').hide()
+                    jQuery('.full-page-iframe').addClass('iframe-large')
+                } else {
+                    jQuery('.collapse-icon').text('▾')
+                    jQuery('.site-header').show()
+                    jQuery('.full-page-iframe').removeClass('iframe-large')
+                }
+                return true
+            })
+
+            // hide all preview iframes on small screens
+            if (window.innerWidth < 1091) {
+                jQuery('.card a[target=preview]').attr('target', '_self')
+            }
+        </script>
    </body>
-
-    <script>
-        // show selected file in iframe when preview card is clicked
-        jQuery('.card').on('click', function(e) {
-            jQuery('.selected-card').removeClass('selected-card')
-            jQuery(e.target).closest('.card').addClass('selected-card')
-        })
-        jQuery('.card a[target=preview]').on('click', function(e) {
-            if (e.currentTarget.href.endsWith('.pdf')) {
-                jQuery('.full-page-iframe')[0].removeAttribute('sandbox')
-            } else {
-                jQuery('.full-page-iframe')[0].sandbox = "allow-same-origin allow-scripts allow-forms"
-            }
-            return true
-        })
-
-        // un-sandbox iframes showing pdfs (required to display pdf viewer)
-        jQuery('iframe').map(function() {
-            if (this.src.endsWith('.pdf')) {
-                this.removeAttribute('sandbox')
-                this.src = this.src
-            }
-        })
-
-        // hide header when collapse icon is clicked
-        jQuery('.collapse-icon').on('click', function() {
-            if (jQuery('.collapse-icon').text().includes('▾')) {
-                jQuery('.collapse-icon').text('▸')
-                jQuery('.site-header').hide()
-                jQuery('.full-page-iframe').addClass('iframe-large')
-            } else {
-                jQuery('.collapse-icon').text('▾')
-                jQuery('.site-header').show()
-                jQuery('.full-page-iframe').removeClass('iframe-large')
-            }
-            return true
-        })
-
-        // hide all preview iframes on small screens
-        if (window.innerWidth < 1091) {
-            jQuery('.card a[target=preview]').attr('target', '_self')
-        }
-    </script>
 </html>
@@ -0,0 +1 @@
+table.dataTable{width:100%;margin:0 auto;clear:both;border-collapse:separate;border-spacing:0}table.dataTable thead th,table.dataTable tfoot th{font-weight:bold}table.dataTable thead th,table.dataTable thead td{padding:10px 18px;border-bottom:1px solid #111}table.dataTable thead th:active,table.dataTable thead td:active{outline:none}table.dataTable tfoot th,table.dataTable tfoot td{padding:10px 18px 6px 18px;border-top:1px solid #111}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;*cursor:hand;background-repeat:no-repeat;background-position:center right}table.dataTable thead .sorting{background-image:url("sort_both.png")}table.dataTable thead .sorting_asc{background-image:url("sort_asc.png")}table.dataTable thead .sorting_desc{background-image:url("sort_desc.png")}table.dataTable thead .sorting_asc_disabled{background-image:url("sort_asc_disabled.png")}table.dataTable thead .sorting_desc_disabled{background-image:url("sort_desc_disabled.png")}table.dataTable tbody tr{background-color:#ffffff}table.dataTable tbody tr.selected{background-color:#B0BED9}table.dataTable tbody th,table.dataTable tbody td{padding:8px 10px}table.dataTable.row-border tbody th,table.dataTable.row-border tbody td,table.dataTable.display tbody th,table.dataTable.display tbody td{border-top:1px solid #ddd}table.dataTable.row-border tbody tr:first-child th,table.dataTable.row-border tbody tr:first-child td,table.dataTable.display tbody tr:first-child th,table.dataTable.display tbody tr:first-child td{border-top:none}table.dataTable.cell-border tbody th,table.dataTable.cell-border tbody td{border-top:1px solid #ddd;border-right:1px solid #ddd}table.dataTable.cell-border tbody tr th:first-child,table.dataTable.cell-border tbody tr td:first-child{border-left:1px solid #ddd}table.dataTable.cell-border tbody tr:first-child th,table.dataTable.cell-border tbody tr:first-child td{border-top:none}table.dataTable.stripe tbody tr.odd,table.dataTable.display tbody tr.odd{background-color:#f9f9f9}table.dataTable.stripe tbody tr.odd.selected,table.dataTable.display tbody tr.odd.selected{background-color:#acbad4}table.dataTable.hover tbody tr:hover,table.dataTable.display tbody tr:hover{background-color:#f6f6f6}table.dataTable.hover tbody tr:hover.selected,table.dataTable.display tbody tr:hover.selected{background-color:#aab7d1}table.dataTable.order-column tbody tr>.sorting_1,table.dataTable.order-column tbody tr>.sorting_2,table.dataTable.order-column tbody tr>.sorting_3,table.dataTable.display tbody tr>.sorting_1,table.dataTable.display tbody tr>.sorting_2,table.dataTable.display tbody tr>.sorting_3{background-color:#fafafa}table.dataTable.order-column tbody tr.selected>.sorting_1,table.dataTable.order-column tbody tr.selected>.sorting_2,table.dataTable.order-column tbody tr.selected>.sorting_3,table.dataTable.display tbody tr.selected>.sorting_1,table.dataTable.display tbody tr.selected>.sorting_2,table.dataTable.display tbody tr.selected>.sorting_3{background-color:#acbad5}table.dataTable.display tbody tr.odd>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd>.sorting_1{background-color:#f1f1f1}table.dataTable.display tbody tr.odd>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd>.sorting_2{background-color:#f3f3f3}table.dataTable.display tbody tr.odd>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd>.sorting_3{background-color:whitesmoke}table.dataTable.display tbody tr.odd.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_1{background-color:#a6b4cd}table.dataTable.display tbody tr.odd.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_2{background-color:#a8b5cf}table.dataTable.display tbody tr.odd.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_3{background-color:#a9b7d1}table.dataTable.display tbody tr.even>.sorting_1,table.dataTable.order-column.stripe tbody tr.even>.sorting_1{background-color:#fafafa}table.dataTable.display tbody tr.even>.sorting_2,table.dataTable.order-column.stripe tbody tr.even>.sorting_2{background-color:#fcfcfc}table.dataTable.display tbody tr.even>.sorting_3,table.dataTable.order-column.stripe tbody tr.even>.sorting_3{background-color:#fefefe}table.dataTable.display tbody tr.even.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_1{background-color:#acbad5}table.dataTable.display tbody tr.even.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_2{background-color:#aebcd6}table.dataTable.display tbody tr.even.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_3{background-color:#afbdd8}table.dataTable.display tbody tr:hover>.sorting_1,table.dataTable.order-column.hover tbody tr:hover>.sorting_1{background-color:#eaeaea}Line truncated
@@ -0,0 +1,166 @@
+/*!
+ DataTables 1.10.19
+ ©2008-2018 SpryMedia Ltd - datatables.net/license
+*/
+(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(E){return h(E,window,document)}):"object"===typeof exports?module.exports=function(E,H){E||(E=window);H||(H="undefined"!==typeof window?require("jquery"):require("jquery")(E));return h(H,E,E.document)}:h(jQuery,window,document)})(function(h,E,H,k){function Z(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()),
+d[c]=e,"o"===b[1]&&Z(a[e])});a._hungarianMap=d}function J(a,b,c){a._hungarianMap||Z(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),J(a[d],b[d],c)):b[d]=b[e]})}function Ca(a){var b=n.defaults.oLanguage,c=b.sDecimal;c&&Da(c);if(a){var d=a.sZeroRecords;!a.sEmptyTable&&(d&&"No data available in table"===b.sEmptyTable)&&F(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(d&&"Loading..."===b.sLoadingRecords)&&F(a,
+a,"sZeroRecords","sLoadingRecords");a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&c!==a&&Da(a)}}function fb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%":
+"");"boolean"===typeof a.scrollX&&(a.scrollX=a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b<c;b++)a[b]&&J(n.models.oSearch,a[b])}function gb(a){A(a,"orderable","bSortable");A(a,"orderData","aDataSort");A(a,"orderSequence","asSorting");A(a,"orderDataType","sortDataType");var b=a.aDataSort;"number"===typeof b&&!h.isArray(b)&&(a.aDataSort=[b])}function hb(a){if(!n.__browser){var b={};n.__browser=b;var c=h("<div/>").css({position:"fixed",top:0,left:-1*h(E).scrollLeft(),height:1,width:1,
+overflow:"hidden"}).append(h("<div/>").css({position:"absolute",top:1,left:1,width:100,overflow:"scroll"}).append(h("<div/>").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,n.__browser);a.oScroll.iBarWidth=n.__browser.barWidth}
+function ib(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=n.defaults.column,d=a.aoColumns.length,c=h.extend({},n.models.oColumn,c,{nTh:b?b:H.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},n.models.oSearch,c[d]);ka(a,d,h(b).data())}function ka(a,b,c){var b=a.aoColumns[b],
+d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(gb(c),J(n.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),c.sClass&&e.addClass(c.sClass),h.extend(b,c),F(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),F(b,c,"aDataSort"));var g=b.mData,j=S(g),i=b.mRender?
+S(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return N(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone,
+b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function $(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;c<d;c++)b[c].nTh.style.width=b[c].sWidth}b=a.oScroll;(""!==b.sY||""!==b.sX)&&la(a);r(a,null,"column-sizing",[a])}function aa(a,b){var c=ma(a,"bVisible");return"number"===
+typeof c[b]?c[b]:null}function ba(a,b){var c=ma(a,"bVisible"),c=h.inArray(b,c);return-1!==c?c:null}function V(a){var b=0;h.each(a.aoColumns,function(a,d){d.bVisible&&"none"!==h(d.nTh).css("display")&&b++});return b}function ma(a,b){var c=[];h.map(a.aoColumns,function(a,e){a[b]&&c.push(e)});return c}function Ga(a){var b=a.aoColumns,c=a.aoData,d=n.ext.type.detect,e,f,g,j,i,h,l,q,t;e=0;for(f=b.length;e<f;e++)if(l=b[e],t=[],!l.sType&&l._sManualType)l.sType=l._sManualType;else if(!l.sType){g=0;for(j=d.length;g<
+j;g++){i=0;for(h=c.length;i<h;i++){t[i]===k&&(t[i]=B(a,i,e,"type"));q=d[g](t[i],a);if(!q&&g!==d.length-1)break;if("html"===q)break}if(q){l.sType=q;break}}l.sType||(l.sType="string")}}function jb(a,b,c,d){var e,f,g,j,i,m,l=a.aoColumns;if(b)for(e=b.length-1;0<=e;e--){m=b[e];var q=m.targets!==k?m.targets:m.aTargets;h.isArray(q)||(q=[q]);f=0;for(g=q.length;f<g;f++)if("number"===typeof q[f]&&0<=q[f]){for(;l.length<=q[f];)Ea(a);d(q[f],m)}else if("number"===typeof q[f]&&0>q[f])d(l.length+q[f],m);else if("string"===
+typeof q[f]){j=0;for(i=l.length;j<i;j++)("_all"==q[f]||h(l[j].nTh).hasClass(q[f]))&&d(j,m)}}if(c){e=0;for(a=c.length;e<a;e++)d(e,c[e])}}function O(a,b,c,d){var e=a.aoData.length,f=h.extend(!0,{},n.models.oRow,{src:c?"dom":"data",idx:e});f._aData=b;a.aoData.push(f);for(var g=a.aoColumns,j=0,i=g.length;j<i;j++)g[j].sType=null;a.aiDisplayMaster.push(e);b=a.rowIdFn(b);b!==k&&(a.aIds[b]=f);(c||!a.oFeatures.bDeferRender)&&Ha(a,e,c,d);return e}function na(a,b){var c;b instanceof h||(b=h(b));return b.map(function(b,
+e){c=Ia(a,e);return O(a,c.data,e,c.cells)})}function B(a,b,c,d){var e=a.iDraw,f=a.aoColumns[c],g=a.aoData[b]._aData,j=f.sDefaultContent,i=f.fnGetData(g,d,{settings:a,row:b,col:c});if(i===k)return a.iDrawError!=e&&null===j&&(K(a,0,"Requested unknown parameter "+("function"==typeof f.mData?"{function}":"'"+f.mData+"'")+" for row "+b+", column "+c,4),a.iDrawError=e),j;if((i===g||null===i)&&null!==j&&d!==k)i=j;else if("function"===typeof i)return i.call(g);return null===i&&"display"==d?"":i}function kb(a,
+b,c,d){a.aoColumns[c].fnSetData(a.aoData[b]._aData,d,{settings:a,row:b,col:c})}function Ja(a){return h.map(a.match(/(\\.|[^\.])+/g)||[""],function(a){return a.replace(/\\\./g,".")})}function S(a){if(h.isPlainObject(a)){var b={};h.each(a,function(a,c){c&&(b[a]=S(c))});return function(a,c,f,g){var j=b[c]||b._;return j!==k?j(a,c,f,g):a}}if(null===a)return function(a){return a};if("function"===typeof a)return function(b,c,f,g){return a(b,c,f,g)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||
+-1!==a.indexOf("("))){var c=function(a,b,f){var g,j;if(""!==f){j=Ja(f);for(var i=0,m=j.length;i<m;i++){f=j[i].match(ca);g=j[i].match(W);if(f){j[i]=j[i].replace(ca,"");""!==j[i]&&(a=a[j[i]]);g=[];j.splice(0,i+1);j=j.join(".");if(h.isArray(a)){i=0;for(m=a.length;i<m;i++)g.push(c(a[i],b,j))}a=f[0].substring(1,f[0].length-1);a=""===a?g:g.join(a);break}else if(g){j[i]=j[i].replace(W,"");a=a[j[i]]();continue}if(null===a||a[j[i]]===k)return k;a=a[j[i]]}}return a};return function(b,e){return c(b,e,a)}}return function(b){return b[a]}}
+function N(a){if(h.isPlainObject(a))return N(a._);if(null===a)return function(){};if("function"===typeof a)return function(b,d,e){a(b,"set",d,e)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||-1!==a.indexOf("("))){var b=function(a,d,e){var e=Ja(e),f;f=e[e.length-1];for(var g,j,i=0,m=e.length-1;i<m;i++){g=e[i].match(ca);j=e[i].match(W);if(g){e[i]=e[i].replace(ca,"");a[e[i]]=[];f=e.slice();f.splice(0,i+1);g=f.join(".");if(h.isArray(d)){j=0;for(m=d.length;j<m;j++)f={},b(f,d[j],g),
+a[e[i]].push(f)}else a[e[i]]=d;return}j&&(e[i]=e[i].replace(W,""),a=a[e[i]](d));if(null===a[e[i]]||a[e[i]]===k)a[e[i]]={};a=a[e[i]]}if(f.match(W))a[f.replace(W,"")](d);else a[f.replace(ca,"")]=d};return function(c,d){return b(c,d,a)}}return function(b,d){b[a]=d}}function Ka(a){return D(a.aoData,"_aData")}function oa(a){a.aoData.length=0;a.aiDisplayMaster.length=0;a.aiDisplay.length=0;a.aIds={}}function pa(a,b,c){for(var d=-1,e=0,f=a.length;e<f;e++)a[e]==b?d=e:a[e]>b&&a[e]--; -1!=d&&c===k&&a.splice(d,
+1)}function da(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c<f;c++)g(j[c],c)}}e._aSortData=null;e._aFilterData=null;g=a.aoColumns;if(d!==k)g[d].sType=null;else{c=0;for(f=g.length;c<f;c++)g[c].sType=null;La(a,e)}}function Ia(a,b,c,d){var e=[],f=b.firstChild,g,
+j,i=0,m,l=a.aoColumns,q=a._rowReadObject,d=d!==k?d:q?{}:[],t=function(a,b){if("string"===typeof a){var c=a.indexOf("@");-1!==c&&(c=a.substring(c+1),N(a)(d,b.getAttribute(c)))}},G=function(a){if(c===k||c===i)j=l[i],m=h.trim(a.innerHTML),j&&j._bAttrSrc?(N(j.mData._)(d,m),t(j.mData.sort,a),t(j.mData.type,a),t(j.mData.filter,a)):q?(j._setter||(j._setter=N(j.mData)),j._setter(d,m)):d[i]=m;i++};if(f)for(;f;){g=f.nodeName.toUpperCase();if("TD"==g||"TH"==g)G(f),e.push(f);f=f.nextSibling}else{e=b.anCells;
+f=0;for(g=e.length;f<g;f++)G(e[f])}if(b=b.firstChild?b:b.nTr)(b=b.getAttribute("id"))&&N(a.rowId)(d,b);return{data:d,cells:e}}function Ha(a,b,c,d){var e=a.aoData[b],f=e._aData,g=[],j,i,m,l,q;if(null===e.nTr){j=c||H.createElement("tr");e.nTr=j;e.anCells=g;j._DT_RowIndex=b;La(a,e);l=0;for(q=a.aoColumns.length;l<q;l++){m=a.aoColumns[l];i=c?d[l]:H.createElement(m.sCellType);i._DT_CellIndex={row:b,column:l};g.push(i);if((!c||m.mRender||m.mData!==l)&&(!h.isPlainObject(m.mData)||m.mData._!==l+".display"))i.innerHTML=
+B(a,b,l,"display");m.sClass&&(i.className+=" "+m.sClass);m.bVisible&&!c?j.appendChild(i):!m.bVisible&&c&&i.parentNode.removeChild(i);m.fnCreatedCell&&m.fnCreatedCell.call(a.oInstance,i,B(a,b,l),f,b,l)}r(a,"aoRowCreatedCallback",null,[j,f,b,g])}e.nTr.setAttribute("role","row")}function La(a,b){var c=b.nTr,d=b._aData;if(c){var e=a.rowIdFn(d);e&&(c.id=e);d.DT_RowClass&&(e=d.DT_RowClass.split(" "),b.__rowc=b.__rowc?qa(b.__rowc.concat(e)):e,h(c).removeClass(b.__rowc.join(" ")).addClass(d.DT_RowClass));
+d.DT_RowAttr&&h(c).attr(d.DT_RowAttr);d.DT_RowData&&h(c).data(d.DT_RowData)}}function lb(a){var b,c,d,e,f,g=a.nTHead,j=a.nTFoot,i=0===h("th, td",g).length,m=a.oClasses,l=a.aoColumns;i&&(e=h("<tr/>").appendTo(g));b=0;for(c=l.length;b<c;b++)f=l[b],d=h(f.nTh).addClass(f.sClass),i&&d.appendTo(e),a.oFeatures.bSort&&(d.addClass(f.sSortingClass),!1!==f.bSortable&&(d.attr("tabindex",a.iTabIndex).attr("aria-controls",a.sTableId),Ma(a,f.nTh,b))),f.sTitle!=d[0].innerHTML&&d.html(f.sTitle),Na(a,"header")(a,d,
+f,m);i&&ea(a.aoHeader,g);h(g).find(">tr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(m.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(m.sFooterTH);if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b<c;b++)f=l[b],f.nTf=a[b].cell,f.sClass&&h(f.nTf).addClass(f.sClass)}}function fa(a,b,c){var d,e,f,g=[],j=[],i=a.aoColumns.length,m;if(b){c===k&&(c=!1);d=0;for(e=b.length;d<e;d++){g[d]=b[d].slice();g[d].nTr=b[d].nTr;for(f=i-1;0<=f;f--)!a.aoColumns[f].bVisible&&!c&&g[d].splice(f,1);j.push([])}d=
+0;for(e=g.length;d<e;d++){if(a=g[d].nTr)for(;f=a.firstChild;)a.removeChild(f);f=0;for(b=g[d].length;f<b;f++)if(m=i=1,j[d][f]===k){a.appendChild(g[d][f].cell);for(j[d][f]=1;g[d+i]!==k&&g[d][f].cell==g[d+i][f].cell;)j[d+i][f]=1,i++;for(;g[d][f+m]!==k&&g[d][f].cell==g[d][f+m].cell;){for(c=0;c<i;c++)j[d+c][f+m]=1;m++}h(g[d][f].cell).attr("rowspan",i).attr("colspan",m)}}}}function P(a){var b=r(a,"aoPreDrawCallback","preDraw",[a]);if(-1!==h.inArray(!1,b))C(a,!1);else{var b=[],c=0,d=a.asStripeClasses,e=
+d.length,f=a.oLanguage,g=a.iInitDisplayStart,j="ssp"==y(a),i=a.aiDisplay;a.bDrawing=!0;g!==k&&-1!==g&&(a._iDisplayStart=j?g:g>=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=-1);var g=a._iDisplayStart,m=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!mb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:m;for(j=j?0:g;j<f;j++){var l=i[j],q=a.aoData[l];null===q.nTr&&Ha(a,l);var t=q.nTr;if(0!==e){var G=d[c%e];q._sRowStripe!=G&&(h(t).removeClass(q._sRowStripe).addClass(G),
+q._sRowStripe=G)}r(a,"aoRowCallback",null,[t,q._aData,c,j,l]);b.push(t);c++}}else c=f.sZeroRecords,1==a.iDraw&&"ajax"==y(a)?c=f.sLoadingRecords:f.sEmptyTable&&0===a.fnRecordsTotal()&&(c=f.sEmptyTable),b[0]=h("<tr/>",{"class":e?d[0]:""}).append(h("<td />",{valign:"top",colSpan:V(a),"class":a.oClasses.sRowEmpty}).html(c))[0];r(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,m,i]);r(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,m,i]);d=h(a.nTBody);d.children().detach();
+d.append(h(b));r(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;c.bSort&&nb(a);d?ga(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;P(a);a._drawHold=!1}function ob(a){var b=a.oClasses,c=h(a.nTable),c=h("<div/>").insertBefore(c),d=a.oFeatures,e=h("<div/>",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore=
+a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,m,l,q,k=0;k<f.length;k++){g=null;j=f[k];if("<"==j){i=h("<div/>")[0];m=f[k+1];if("'"==m||'"'==m){l="";for(q=2;f[k+q]!=m;)l+=f[k+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(m=l.split("."),i.id=m[0].substr(1,m[0].length-1),i.className=m[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;k+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=pb(a);else if("f"==j&&
+d.bFilter)g=qb(a);else if("r"==j&&d.bProcessing)g=rb(a);else if("t"==j)g=sb(a);else if("i"==j&&d.bInfo)g=tb(a);else if("p"==j&&d.bPaginate)g=ub(a);else if(0!==n.ext.feature.length){i=n.ext.feature;q=0;for(m=i.length;q<m;q++)if(j==i[q].cFeature){g=i[q].fnInit(a);break}}g&&(i=a.aanFeatures,i[j]||(i[j]=[]),i[j].push(g),e.append(g))}c.replaceWith(e);a.nHolding=null}function ea(a,b){var c=h(b).children("tr"),d,e,f,g,j,i,m,l,q,k;a.splice(0,a.length);f=0;for(i=c.length;f<i;f++)a.push([]);f=0;for(i=c.length;f<
+i;f++){d=c[f];for(e=d.firstChild;e;){if("TD"==e.nodeName.toUpperCase()||"TH"==e.nodeName.toUpperCase()){l=1*e.getAttribute("colspan");q=1*e.getAttribute("rowspan");l=!l||0===l||1===l?1:l;q=!q||0===q||1===q?1:q;g=0;for(j=a[f];j[g];)g++;m=g;k=1===l?!0:!1;for(j=0;j<l;j++)for(g=0;g<q;g++)a[f+g][m+j]={cell:e,unique:k},a[f+g].nTr=d}e=e.nextSibling}}}function ra(a,b,c){var d=[];c||(c=a.aoHeader,b&&(c=[],ea(c,b)));for(var b=0,e=c.length;b<e;b++)for(var f=0,g=c[b].length;f<g;f++)if(c[b][f].unique&&(!d[f]||
+!a.bSortCellsTop))d[f]=c[b][f].cell;return d}function sa(a,b,c){r(a,"aoServerParams","serverParams",[b]);if(b&&h.isArray(b)){var d={},e=/(.*?)\[\]$/;h.each(b,function(a,b){var c=b.name.match(e);c?(c=c[0],d[c]||(d[c]=[]),d[c].push(b.value)):d[b.name]=b.value});b=d}var f,g=a.ajax,j=a.oInstance,i=function(b){r(a,null,"xhr",[a,b,a.jqXHR]);c(b)};if(h.isPlainObject(g)&&g.data){f=g.data;var m="function"===typeof f?f(b,a):f,b="function"===typeof f&&m?m:h.extend(!0,b,m);delete g.data}m={data:b,success:function(b){var c=
+b.error||b.sError;c&&K(a,0,c);a.json=b;i(b)},dataType:"json",cache:!1,type:a.sServerMethod,error:function(b,c){var d=r(a,null,"xhr",[a,null,a.jqXHR]);-1===h.inArray(!0,d)&&("parsererror"==c?K(a,0,"Invalid JSON response",1):4===b.readyState&&K(a,0,"Ajax error",7));C(a,!1)}};a.oAjaxData=b;r(a,null,"preXhr",[a,b]);a.fnServerData?a.fnServerData.call(j,a.sAjaxSource,h.map(b,function(a,b){return{name:b,value:a}}),i,a):a.sAjaxSource||"string"===typeof g?a.jqXHR=h.ajax(h.extend(m,{url:g||a.sAjaxSource})):
+"function"===typeof g?a.jqXHR=g.call(j,b,i,a):(a.jqXHR=h.ajax(h.extend(m,g)),g.data=f)}function mb(a){return a.bAjaxDataGet?(a.iDraw++,C(a,!0),sa(a,vb(a),function(b){wb(a,b)}),!1):!0}function vb(a){var b=a.aoColumns,c=b.length,d=a.oFeatures,e=a.oPreviousSearch,f=a.aoPreSearchCols,g,j=[],i,m,l,k=X(a);g=a._iDisplayStart;i=!1!==d.bPaginate?a._iDisplayLength:-1;var t=function(a,b){j.push({name:a,value:b})};t("sEcho",a.iDraw);t("iColumns",c);t("sColumns",D(b,"sName").join(","));t("iDisplayStart",g);t("iDisplayLength",
+i);var G={draw:a.iDraw,columns:[],order:[],start:g,length:i,search:{value:e.sSearch,regex:e.bRegex}};for(g=0;g<c;g++)m=b[g],l=f[g],i="function"==typeof m.mData?"function":m.mData,G.columns.push({data:i,name:m.sName,searchable:m.bSearchable,orderable:m.bSortable,search:{value:l.sSearch,regex:l.bRegex}}),t("mDataProp_"+g,i),d.bFilter&&(t("sSearch_"+g,l.sSearch),t("bRegex_"+g,l.bRegex),t("bSearchable_"+g,m.bSearchable)),d.bSort&&t("bSortable_"+g,m.bSortable);d.bFilter&&(t("sSearch",e.sSearch),t("bRegex",
+e.bRegex));d.bSort&&(h.each(k,function(a,b){G.order.push({column:b.col,dir:b.dir});t("iSortCol_"+a,b.col);t("sSortDir_"+a,b.dir)}),t("iSortingCols",k.length));b=n.ext.legacy.ajax;return null===b?a.sAjaxSource?j:G:b?j:G}function wb(a,b){var c=ta(a,b),d=b.sEcho!==k?b.sEcho:b.draw,e=b.iTotalRecords!==k?b.iTotalRecords:b.recordsTotal,f=b.iTotalDisplayRecords!==k?b.iTotalDisplayRecords:b.recordsFiltered;if(d){if(1*d<a.iDraw)return;a.iDraw=1*d}oa(a);a._iRecordsTotal=parseInt(e,10);a._iRecordsDisplay=parseInt(f,
+10);d=0;for(e=c.length;d<e;d++)O(a,c[d]);a.aiDisplay=a.aiDisplayMaster.slice();a.bAjaxDataGet=!1;P(a);a._bInitComplete||ua(a,b);a.bAjaxDataGet=!0;C(a,!1)}function ta(a,b){var c=h.isPlainObject(a.ajax)&&a.ajax.dataSrc!==k?a.ajax.dataSrc:a.sAjaxDataProp;return"data"===c?b.aaData||b[c]:""!==c?S(c)(b):b}function qb(a){var b=a.oClasses,c=a.sTableId,d=a.oLanguage,e=a.oPreviousSearch,f=a.aanFeatures,g='<input type="search" class="'+b.sFilterInput+'"/>',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_",
+g):j+g,b=h("<div/>",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("<label/>").append(j)),f=function(){var b=!this.value?"":this.value;b!=e.sSearch&&(ga(a,{sSearch:b,bRegex:e.bRegex,bSmart:e.bSmart,bCaseInsensitive:e.bCaseInsensitive}),a._iDisplayStart=0,P(a))},g=null!==a.searchDelay?a.searchDelay:"ssp"===y(a)?400:0,i=h("input",b).val(e.sSearch).attr("placeholder",d.sSearchPlaceholder).on("keyup.DT search.DT input.DT paste.DT cut.DT",g?Oa(f,g):f).on("keypress.DT",function(a){if(13==a.keyCode)return!1}).attr("aria-controls",
+c);h(a.nTable).on("search.dt.DT",function(b,c){if(a===c)try{i[0]!==H.activeElement&&i.val(e.sSearch)}catch(d){}});return b[0]}function ga(a,b,c){var d=a.oPreviousSearch,e=a.aoPreSearchCols,f=function(a){d.sSearch=a.sSearch;d.bRegex=a.bRegex;d.bSmart=a.bSmart;d.bCaseInsensitive=a.bCaseInsensitive};Ga(a);if("ssp"!=y(a)){xb(a,b.sSearch,c,b.bEscapeRegex!==k?!b.bEscapeRegex:b.bRegex,b.bSmart,b.bCaseInsensitive);f(b);for(b=0;b<e.length;b++)yb(a,e[b].sSearch,b,e[b].bEscapeRegex!==k?!e[b].bEscapeRegex:e[b].bRegex,
+e[b].bSmart,e[b].bCaseInsensitive);zb(a)}else f(b);a.bFiltered=!0;r(a,null,"search",[a])}function zb(a){for(var b=n.ext.search,c=a.aiDisplay,d,e,f=0,g=b.length;f<g;f++){for(var j=[],i=0,m=c.length;i<m;i++)e=c[i],d=a.aoData[e],b[f](a,d._aFilterData,e,d._aData,i)&&j.push(e);c.length=0;h.merge(c,j)}}function yb(a,b,c,d,e,f){if(""!==b){for(var g=[],j=a.aiDisplay,d=Pa(b,d,e,f),e=0;e<j.length;e++)b=a.aoData[j[e]]._aFilterData[c],d.test(b)&&g.push(j[e]);a.aiDisplay=g}}function xb(a,b,c,d,e,f){var d=Pa(b,
+d,e,f),f=a.oPreviousSearch.sSearch,g=a.aiDisplayMaster,j,e=[];0!==n.ext.search.length&&(c=!0);j=Ab(a);if(0>=b.length)a.aiDisplay=g.slice();else{if(j||c||f.length>b.length||0!==b.indexOf(f)||a.bSorted)a.aiDisplay=g.slice();b=a.aiDisplay;for(c=0;c<b.length;c++)d.test(a.aoData[b[c]]._sFilterRow)&&e.push(b[c]);a.aiDisplay=e}}function Pa(a,b,c,d){a=b?a:Qa(a);c&&(a="^(?=.*?"+h.map(a.match(/"[^"]+"|[^ ]+/g)||[""],function(a){if('"'===a.charAt(0))var b=a.match(/^"(.*)"$/),a=b?b[1]:a;return a.replace('"',
+"")}).join(")(?=.*?")+").*$");return RegExp(a,d?"i":"")}function Ab(a){var b=a.aoColumns,c,d,e,f,g,j,i,h,l=n.ext.type.search;c=!1;d=0;for(f=a.aoData.length;d<f;d++)if(h=a.aoData[d],!h._aFilterData){j=[];e=0;for(g=b.length;e<g;e++)c=b[e],c.bSearchable?(i=B(a,d,e,"filter"),l[c.sType]&&(i=l[c.sType](i)),null===i&&(i=""),"string"!==typeof i&&i.toString&&(i=i.toString())):i="",i.indexOf&&-1!==i.indexOf("&")&&(va.innerHTML=i,i=Wb?va.textContent:va.innerText),i.replace&&(i=i.replace(/[\r\n]/g,"")),j.push(i);
+h._aFilterData=j;h._sFilterRow=j.join("  ");c=!0}return c}function Bb(a){return{search:a.sSearch,smart:a.bSmart,regex:a.bRegex,caseInsensitive:a.bCaseInsensitive}}function Cb(a){return{sSearch:a.search,bSmart:a.smart,bRegex:a.regex,bCaseInsensitive:a.caseInsensitive}}function tb(a){var b=a.sTableId,c=a.aanFeatures.i,d=h("<div/>",{"class":a.oClasses.sInfo,id:!c?b+"_info":null});c||(a.aoDrawCallback.push({fn:Db,sName:"information"}),d.attr("role","status").attr("aria-live","polite"),h(a.nTable).attr("aria-describedby",
+b+"_info"));return d[0]}function Db(a){var b=a.aanFeatures.i;if(0!==b.length){var c=a.oLanguage,d=a._iDisplayStart+1,e=a.fnDisplayEnd(),f=a.fnRecordsTotal(),g=a.fnRecordsDisplay(),j=g?c.sInfo:c.sInfoEmpty;g!==f&&(j+=" "+c.sInfoFiltered);j+=c.sInfoPostFix;j=Eb(a,j);c=c.fnInfoCallback;null!==c&&(j=c.call(a.oInstance,a,d,e,f,g,j));h(b).html(j)}}function Eb(a,b){var c=a.fnFormatNumber,d=a._iDisplayStart+1,e=a._iDisplayLength,f=a.fnRecordsDisplay(),g=-1===e;return b.replace(/_START_/g,c.call(a,d)).replace(/_END_/g,
+c.call(a,a.fnDisplayEnd())).replace(/_MAX_/g,c.call(a,a.fnRecordsTotal())).replace(/_TOTAL_/g,c.call(a,f)).replace(/_PAGE_/g,c.call(a,g?1:Math.ceil(d/e))).replace(/_PAGES_/g,c.call(a,g?1:Math.ceil(f/e)))}function ha(a){var b,c,d=a.iInitDisplayStart,e=a.aoColumns,f;c=a.oFeatures;var g=a.bDeferLoading;if(a.bInitialised){ob(a);lb(a);fa(a,a.aoHeader);fa(a,a.aoFooter);C(a,!0);c.bAutoWidth&&Fa(a);b=0;for(c=e.length;b<c;b++)f=e[b],f.sWidth&&(f.nTh.style.width=v(f.sWidth));r(a,null,"preInit",[a]);T(a);e=
+y(a);if("ssp"!=e||g)"ajax"==e?sa(a,[],function(c){var f=ta(a,c);for(b=0;b<f.length;b++)O(a,f[b]);a.iInitDisplayStart=d;T(a);C(a,!1);ua(a,c)},a):(C(a,!1),ua(a))}else setTimeout(function(){ha(a)},200)}function ua(a,b){a._bInitComplete=!0;(b||a.oInit.aaData)&&$(a);r(a,null,"plugin-init",[a,b]);r(a,"aoInitComplete","init",[a,b])}function Ra(a,b){var c=parseInt(b,10);a._iDisplayLength=c;Sa(a);r(a,null,"length",[a,c])}function pb(a){for(var b=a.oClasses,c=a.sTableId,d=a.aLengthMenu,e=h.isArray(d[0]),f=
+e?d[0]:d,d=e?d[1]:d,e=h("<select/>",{name:c+"_length","aria-controls":c,"class":b.sLengthSelect}),g=0,j=f.length;g<j;g++)e[0][g]=new Option("number"===typeof d[g]?a.fnFormatNumber(d[g]):d[g],f[g]);var i=h("<div><label/></div>").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).on("change.DT",function(){Ra(a,h(this).val());P(a)});h(a.nTable).on("length.dt.DT",function(b,c,d){a===
+c&&h("select",i).val(d)});return i[0]}function ub(a){var b=a.sPaginationType,c=n.ext.pager[b],d="function"===typeof c,e=function(a){P(a)},b=h("<div/>").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;l<k;l++)Na(a,"pageButton")(a,f.p[l],l,h,b,i)}else c.fnUpdate(a,
+e)},sName:"pagination"}));return b}function Ta(a,b,c){var d=a._iDisplayStart,e=a._iDisplayLength,f=a.fnRecordsDisplay();0===f||-1===e?d=0:"number"===typeof b?(d=b*e,d>f&&(d=0)):"first"==b?d=0:"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e<f&&(d+=e):"last"==b?d=Math.floor((f-1)/e)*e:K(a,0,"Unknown paging action: "+b,5);b=a._iDisplayStart!==d;a._iDisplayStart=d;b&&(r(a,null,"page",[a]),c&&P(a));return b}function rb(a){return h("<div/>",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]}
+function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");r(a,null,"processing",[a,b])}function sb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),m=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("<div/>",{"class":f.sScrollWrapper}).append(h("<div/>",{"class":f.sScrollHead}).css({overflow:"hidden",
+position:"relative",border:0,width:d?!d?null:v(d):"100%"}).append(h("<div/>",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("<div/>",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:v(d)}).append(b));l&&i.append(h("<div/>",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:v(d):"100%"}).append(h("<div/>",
+{"class":f.sScrollFootInner}).append(m.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:la,sName:"scrolling"});return i[0]}function la(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth,
+f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,m=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),n=t.children("table"),o=h(a.nTHead),p=h(a.nTable),s=p[0],r=s.style,u=a.nTFoot?h(a.nTFoot):null,x=a.oBrowser,U=x.bScrollOversize,Xb=D(a.aoColumns,"nTh"),Q,L,R,w,Ua=[],y=[],z=[],A=[],B,C=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};L=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!==
+L&&a.scrollBarVis!==k)a.scrollBarVis=L,$(a);else{a.scrollBarVis=L;p.children("thead, tfoot").remove();u&&(R=u.clone().prependTo(p),Q=u.find("tr"),R=R.find("tr"));w=o.clone().prependTo(p);o=o.find("tr");L=w.find("tr");w.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(ra(a,w),function(b,c){B=aa(a,b);c.style.width=a.aoColumns[B].sWidth});u&&I(function(a){a.style.width=""},R);f=p.outerWidth();if(""===c){r.width="100%";if(U&&(p.find("tbody").height()>j.offsetHeight||
+"scroll"==l.css("overflow-y")))r.width=v(p.outerWidth()-b);f=p.outerWidth()}else""!==d&&(r.width=v(d),f=p.outerWidth());I(C,L);I(function(a){z.push(a.innerHTML);Ua.push(v(h(a).css("width")))},L);I(function(a,b){if(h.inArray(a,Xb)!==-1)a.style.width=Ua[b]},o);h(L).height(0);u&&(I(C,R),I(function(a){A.push(a.innerHTML);y.push(v(h(a).css("width")))},R),I(function(a,b){a.style.width=y[b]},Q),h(R).height(0));I(function(a,b){a.innerHTML='<div class="dataTables_sizing">'+z[b]+"</div>";a.childNodes[0].style.height=
+"0";a.childNodes[0].style.overflow="hidden";a.style.width=Ua[b]},L);u&&I(function(a,b){a.innerHTML='<div class="dataTables_sizing">'+A[b]+"</div>";a.childNodes[0].style.height="0";a.childNodes[0].style.overflow="hidden";a.style.width=y[b]},R);if(p.outerWidth()<f){Q=j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(U&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=v(Q-b);(""===c||""!==d)&&K(a,1,"Possible column misalignment",6)}else Q="100%";q.width=v(Q);
+g.width=v(Q);u&&(a.nScrollFoot.style.width=v(Q));!e&&U&&(q.height=v(s.offsetHeight+b));c=p.outerWidth();m[0].style.width=v(c);i.width=v(c);d=p.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(x.bScrollbarLeft?"Left":"Right");i[e]=d?b+"px":"0px";u&&(n[0].style.width=v(c),t[0].style.width=v(c),t[0].style[e]=d?b+"px":"0px");p.children("colgroup").insertBefore(p.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function I(a,b,c){for(var d=0,e=0,
+f=b.length,g,j;e<f;){g=b[e].firstChild;for(j=c?c[e].firstChild:null;g;)1===g.nodeType&&(c?a(g,j,d):a(g,d),d++),g=g.nextSibling,j=c?j.nextSibling:null;e++}}function Fa(a){var b=a.nTable,c=a.aoColumns,d=a.oScroll,e=d.sY,f=d.sX,g=d.sXInner,j=c.length,i=ma(a,"bVisible"),m=h("th",a.nTHead),l=b.getAttribute("width"),k=b.parentNode,t=!1,n,o,p=a.oBrowser,d=p.bScrollOversize;(n=b.style.width)&&-1!==n.indexOf("%")&&(l=n);for(n=0;n<i.length;n++)o=c[i[n]],null!==o.sWidth&&(o.sWidth=Fb(o.sWidthOrig,k),t=!0);if(d||
+!t&&!f&&!e&&j==V(a)&&j==m.length)for(n=0;n<j;n++)i=aa(a,n),null!==i&&(c[i].sWidth=v(m.eq(n).width()));else{j=h(b).clone().css("visibility","hidden").removeAttr("id");j.find("tbody tr").remove();var s=h("<tr/>").appendTo(j.find("tbody"));j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");m=ra(a,j.find("thead")[0]);for(n=0;n<i.length;n++)o=c[i[n]],m[n].style.width=null!==o.sWidthOrig&&""!==o.sWidthOrig?v(o.sWidthOrig):
+"",o.sWidthOrig&&f&&h(m[n]).append(h("<div/>").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(n=0;n<i.length;n++)t=i[n],o=c[t],h(Gb(a,t)).clone(!1).append(o.sContentPadding).appendTo(s);h("[name]",j).removeAttr("name");o=h("<div/>").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()<k.clientWidth&&l&&j.width(k.clientWidth)):e?j.width(k.clientWidth):
+l&&j.width(l);for(n=e=0;n<i.length;n++)k=h(m[n]),g=k.outerWidth()-k.width(),k=p.bBounding?Math.ceil(m[n].getBoundingClientRect().width):k.outerWidth(),e+=k,c[i[n]].sWidth=v(k-g);b.style.width=v(e);o.remove()}l&&(b.style.width=v(l));if((l||f)&&!a._reszEvt)b=function(){h(E).on("resize.DT-"+a.sInstance,Oa(function(){$(a)}))},d?setTimeout(b,1E3):b(),a._reszEvt=!0}function Fb(a,b){if(!a)return 0;var c=h("<div/>").css("width",v(a)).appendTo(b||H.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a,
+b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("<td/>").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;f<g;f++)c=B(a,f,b,"display")+"",c=c.replace(Yb,""),c=c.replace(/&nbsp;/g," "),c.length>d&&(d=c.length,e=f);return e}function v(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function X(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var m=[];f=function(a){a.length&&
+!h.isArray(a[0])?m.push(a):h.merge(m,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;a<m.length;a++){i=m[a][0];f=e[i].aDataSort;b=0;for(c=f.length;b<c;b++)g=f[b],j=e[g].sType||"string",m[a]._idx===k&&(m[a]._idx=h.inArray(m[a][1],e[g].asSorting)),d.push({src:i,col:g,dir:m[a][1],index:m[a]._idx,type:j,formatter:n.ext.type.order[j+"-pre"]})}return d}function nb(a){var b,c,d=[],e=n.ext.type.order,f=a.aoData,g=0,j,i=a.aiDisplayMaster,h;Ga(a);h=X(a);b=0;for(c=h.length;b<
+c;b++)j=h[b],j.formatter&&g++,Ib(a,j.col);if("ssp"!=y(a)&&0!==h.length){b=0;for(c=i.length;b<c;b++)d[i[b]]=b;g===h.length?i.sort(function(a,b){var c,e,g,j,i=h.length,k=f[a]._aSortData,n=f[b]._aSortData;for(g=0;g<i;g++)if(j=h[g],c=k[j.col],e=n[j.col],c=c<e?-1:c>e?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return c<e?-1:c>e?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,n=f[a]._aSortData,o=f[b]._aSortData;for(j=0;j<k;j++)if(i=h[j],c=n[i.col],g=o[i.col],i=e[i.type+"-"+i.dir]||e["string-"+i.dir],
+c=i(c,g),0!==c)return c;c=d[a];g=d[b];return c<g?-1:c>g?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=X(a),a=a.oLanguage.oAria,f=0,g=d.length;f<g;f++){c=d[f];var j=c.asSorting;b=c.sTitle.replace(/<.*?>/g,"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0<e.length&&e[0].col==f?(i.setAttribute("aria-sort","asc"==e[0].dir?"ascending":"descending"),c=j[e[0].index+1]||j[0]):c=j[0],b+="asc"===c?a.sSortAscending:a.sSortDescending);i.setAttribute("aria-label",b)}}function Va(a,
+b,c,d){var e=a.aaSorting,f=a.aoColumns[b].asSorting,g=function(a,b){var c=a._idx;c===k&&(c=h.inArray(a[1],f));return c+1<f.length?c+1:b?null:0};"number"===typeof e[0]&&(e=a.aaSorting=[e]);c&&a.oFeatures.bSortMulti?(c=h.inArray(b,D(e,"0")),-1!==c?(b=g(e[c],!0),null===b&&1===e.length&&(b=0),null===b?e.splice(c,1):(e[c][1]=f[b],e[c]._idx=b)):(e.push([b,f[0],0]),e[e.length-1]._idx=0)):e.length&&e[0][0]==b?(b=g(e[0]),e.length=1,e[0][1]=f[b],e[0]._idx=b):(e.length=0,e.push([b,f[0]]),e[0]._idx=0);T(a);"function"==
+typeof d&&d(a)}function Ma(a,b,c,d){var e=a.aoColumns[c];Wa(b,{},function(b){!1!==e.bSortable&&(a.oFeatures.bProcessing?(C(a,!0),setTimeout(function(){Va(a,c,b.shiftKey,d);"ssp"!==y(a)&&C(a,!1)},0)):Va(a,c,b.shiftKey,d))})}function wa(a){var b=a.aLastSort,c=a.oClasses.sSortColumn,d=X(a),e=a.oFeatures,f,g;if(e.bSort&&e.bSortClasses){e=0;for(f=b.length;e<f;e++)g=b[e].src,h(D(a.aoData,"anCells",g)).removeClass(c+(2>e?e+1:3));e=0;for(f=d.length;e<f;e++)g=d[e].src,h(D(a.aoData,"anCells",g)).addClass(c+
+(2>e?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=n.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,ba(a,b)));for(var f,g=n.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j<i;j++)if(c=a.aoData[j],c._aSortData||(c._aSortData=[]),!c._aSortData[b]||d)f=d?e[j]:B(a,j,b,"sort"),c._aSortData[b]=g?g(f):f}function xa(a){if(a.oFeatures.bStateSave&&!a.bDestroying){var b={time:+new Date,start:a._iDisplayStart,length:a._iDisplayLength,order:h.extend(!0,[],a.aaSorting),search:Bb(a.oPreviousSearch),
+columns:h.map(a.aoColumns,function(b,d){return{visible:b.bVisible,search:Bb(a.aoPreSearchCols[d])}})};r(a,"aoStateSaveParams","stateSaveParams",[a,b]);a.oSavedState=b;a.fnStateSaveCallback.call(a.oInstance,a,b)}}function Kb(a,b,c){var d,e,f=a.aoColumns,b=function(b){if(b&&b.time){var g=r(a,"aoStateLoadParams","stateLoadParams",[a,b]);if(-1===h.inArray(!1,g)&&(g=a.iStateDuration,!(0<g&&b.time<+new Date-1E3*g)&&!(b.columns&&f.length!==b.columns.length))){a.oLoadedState=h.extend(!0,{},b);b.start!==k&&
+(a._iDisplayStart=b.start,a.iInitDisplayStart=b.start);b.length!==k&&(a._iDisplayLength=b.length);b.order!==k&&(a.aaSorting=[],h.each(b.order,function(b,c){a.aaSorting.push(c[0]>=f.length?[0,c[1]]:c)}));b.search!==k&&h.extend(a.oPreviousSearch,Cb(b.search));if(b.columns){d=0;for(e=b.columns.length;d<e;d++)g=b.columns[d],g.visible!==k&&(f[d].bVisible=g.visible),g.search!==k&&h.extend(a.aoPreSearchCols[d],Cb(g.search))}r(a,"aoStateLoaded","stateLoaded",[a,b])}}c()};if(a.oFeatures.bStateSave){var g=
+a.fnStateLoadCallback.call(a.oInstance,a,b);g!==k&&b(g)}else c()}function ya(a){var b=n.settings,a=h.inArray(a,D(b,"nTable"));return-1!==a?b[a]:null}function K(a,b,c,d){c="DataTables warning: "+(a?"table id="+a.sTableId+" - ":"")+c;d&&(c+=". For more information about this error, please see http://datatables.net/tn/"+d);if(b)E.console&&console.log&&console.log(c);else if(b=n.ext,b=b.sErrMode||b.errMode,a&&r(a,null,"error",[a,d,c]),"alert"==b)alert(c);else{if("throw"==b)throw Error(c);"function"==
+typeof b&&b(a,d,c)}}function F(a,b,c,d){h.isArray(c)?h.each(c,function(c,d){h.isArray(d)?F(a,b,d[0],d[1]):F(a,b,d)}):(d===k&&(d=c),b[c]!==k&&(a[d]=b[c]))}function Xa(a,b,c){var d,e;for(e in b)b.hasOwnProperty(e)&&(d=b[e],h.isPlainObject(d)?(h.isPlainObject(a[e])||(a[e]={}),h.extend(!0,a[e],d)):a[e]=c&&"data"!==e&&"aaData"!==e&&h.isArray(d)?d.slice():d);return a}function Wa(a,b,c){h(a).on("click.DT",b,function(b){h(a).blur();c(b)}).on("keypress.DT",b,function(a){13===a.which&&(a.preventDefault(),c(a))}).on("selectstart.DT",
+function(){return!1})}function z(a,b,c,d){c&&a[b].push({fn:c,sName:d})}function r(a,b,c,d){var e=[];b&&(e=h.map(a[b].slice().reverse(),function(b){return b.fn.apply(a.oInstance,d)}));null!==c&&(b=h.Event(c+".dt"),h(a.nTable).trigger(b,d),e.push(b.result));return e}function Sa(a){var b=a._iDisplayStart,c=a.fnDisplayEnd(),d=a._iDisplayLength;b>=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=n.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"===
+typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ia(a,b){var c=[],c=Lb.numbers_length,d=Math.floor(c/2);b<=c?c=Y(0,b):a<=d?(c=Y(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=Y(b-(c-2),b):(c=Y(a-d+2,a+d-1),c.push("ellipsis"),c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function Da(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Ya)},"html-num":function(b){return za(b,
+a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Ya)}},function(b,c){x.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(x.type.search[b+a]=x.type.search.html)})}function Mb(a){return function(){var b=[ya(this[n.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return n.ext.internal[a].apply(this,b)}}var n=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new s(ya(this[x.iApiIndex])):new s(this)};
+this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):(""!==d.sX||""!==d.sY)&&la(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a,
+b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data():
+c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]};
+this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return ya(this[x.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust();
+(d===k||d)&&h.draw();return 0};this.fnVersionCheck=x.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=x.internal;for(var e in n.ext.internal)e&&(this[e]=Mb(e));this.each(function(){var e={},g=1<d?Xa(e,a,!0):a,j=0,i,e=this.getAttribute("id"),m=!1,l=n.defaults,q=h(this);if("table"!=this.nodeName.toLowerCase())K(null,0,"Non-table node initialisation ("+this.nodeName+")",2);else{fb(l);gb(l.column);J(l,l,!0);J(l.column,l.column,!0);J(l,h.extend(g,q.data()));var t=n.settings,
+j=0;for(i=t.length;j<i;j++){var o=t[j];if(o.nTable==this||o.nTHead&&o.nTHead.parentNode==this||o.nTFoot&&o.nTFoot.parentNode==this){var s=g.bRetrieve!==k?g.bRetrieve:l.bRetrieve;if(c||s)return o.oInstance;if(g.bDestroy!==k?g.bDestroy:l.bDestroy){o.oInstance.fnDestroy();break}else{K(o,0,"Cannot reinitialise DataTable",3);return}}if(o.sTableId==this.id){t.splice(j,1);break}}if(null===e||""===e)this.id=e="DataTables_Table_"+n.ext._unique++;var p=h.extend(!0,{},n.models.oSettings,{sDestroyWidth:q[0].style.width,
+sInstance:e,sTableId:e});p.nTable=this;p.oApi=b.internal;p.oInit=g;t.push(p);p.oInstance=1===b.length?b:q.dataTable();fb(g);Ca(g.oLanguage);g.aLengthMenu&&!g.iDisplayLength&&(g.iDisplayLength=h.isArray(g.aLengthMenu[0])?g.aLengthMenu[0][0]:g.aLengthMenu[0]);g=Xa(h.extend(!0,{},l),g);F(p.oFeatures,g,"bPaginate bLengthChange bFilter bSort bSortMulti bInfo bProcessing bAutoWidth bSortClasses bServerSide bDeferRender".split(" "));F(p,g,["asStripeClasses","ajax","fnServerData","fnFormatNumber","sServerMethod",
+"aaSorting","aaSortingFixed","aLengthMenu","sPaginationType","sAjaxSource","sAjaxDataProp","iStateDuration","sDom","bSortCellsTop","iTabIndex","fnStateLoadCallback","fnStateSaveCallback","renderer","searchDelay","rowId",["iCookieDuration","iStateDuration"],["oSearch","oPreviousSearch"],["aoSearchCols","aoPreSearchCols"],["iDisplayLength","_iDisplayLength"]]);F(p.oScroll,g,[["sScrollX","sX"],["sScrollXInner","sXInner"],["sScrollY","sY"],["bScrollCollapse","bCollapse"]]);F(p.oLanguage,g,"fnInfoCallback");
+z(p,"aoDrawCallback",g.fnDrawCallback,"user");z(p,"aoServerParams",g.fnServerParams,"user");z(p,"aoStateSaveParams",g.fnStateSaveParams,"user");z(p,"aoStateLoadParams",g.fnStateLoadParams,"user");z(p,"aoStateLoaded",g.fnStateLoaded,"user");z(p,"aoRowCallback",g.fnRowCallback,"user");z(p,"aoRowCreatedCallback",g.fnCreatedRow,"user");z(p,"aoHeaderCallback",g.fnHeaderCallback,"user");z(p,"aoFooterCallback",g.fnFooterCallback,"user");z(p,"aoInitComplete",g.fnInitComplete,"user");z(p,"aoPreDrawCallback",
+g.fnPreDrawCallback,"user");p.rowIdFn=S(g.rowId);hb(p);var u=p.oClasses;h.extend(u,n.ext.classes,g.oClasses);q.addClass(u.sTable);p.iInitDisplayStart===k&&(p.iInitDisplayStart=g.iDisplayStart,p._iDisplayStart=g.iDisplayStart);null!==g.iDeferLoading&&(p.bDeferLoading=!0,e=h.isArray(g.iDeferLoading),p._iRecordsDisplay=e?g.iDeferLoading[0]:g.iDeferLoading,p._iRecordsTotal=e?g.iDeferLoading[1]:g.iDeferLoading);var v=p.oLanguage;h.extend(!0,v,g.oLanguage);v.sUrl&&(h.ajax({dataType:"json",url:v.sUrl,success:function(a){Ca(a);
+J(l.oLanguage,a);h.extend(true,v,a);ha(p)},error:function(){ha(p)}}),m=!0);null===g.asStripeClasses&&(p.asStripeClasses=[u.sStripeOdd,u.sStripeEven]);var e=p.asStripeClasses,x=q.children("tbody").find("tr").eq(0);-1!==h.inArray(!0,h.map(e,function(a){return x.hasClass(a)}))&&(h("tbody tr",this).removeClass(e.join(" ")),p.asDestroyStripes=e.slice());e=[];t=this.getElementsByTagName("thead");0!==t.length&&(ea(p.aoHeader,t[0]),e=ra(p));if(null===g.aoColumns){t=[];j=0;for(i=e.length;j<i;j++)t.push(null)}else t=
+g.aoColumns;j=0;for(i=t.length;j<i;j++)Ea(p,e?e[j]:null);jb(p,g.aoColumnDefs,t,function(a,b){ka(p,a,b)});if(x.length){var w=function(a,b){return a.getAttribute("data-"+b)!==null?b:null};h(x[0]).children("th, td").each(function(a,b){var c=p.aoColumns[a];if(c.mData===a){var d=w(b,"sort")||w(b,"order"),e=w(b,"filter")||w(b,"search");if(d!==null||e!==null){c.mData={_:a+".display",sort:d!==null?a+".@data-"+d:k,type:d!==null?a+".@data-"+d:k,filter:e!==null?a+".@data-"+e:k};ka(p,a)}}})}var U=p.oFeatures,
+e=function(){if(g.aaSorting===k){var a=p.aaSorting;j=0;for(i=a.length;j<i;j++)a[j][1]=p.aoColumns[j].asSorting[0]}wa(p);U.bSort&&z(p,"aoDrawCallback",function(){if(p.bSorted){var a=X(p),b={};h.each(a,function(a,c){b[c.src]=c.dir});r(p,null,"order",[p,a,b]);Jb(p)}});z(p,"aoDrawCallback",function(){(p.bSorted||y(p)==="ssp"||U.bDeferRender)&&wa(p)},"sc");var a=q.children("caption").each(function(){this._captionSide=h(this).css("caption-side")}),b=q.children("thead");b.length===0&&(b=h("<thead/>").appendTo(q));
+p.nTHead=b[0];b=q.children("tbody");b.length===0&&(b=h("<tbody/>").appendTo(q));p.nTBody=b[0];b=q.children("tfoot");if(b.length===0&&a.length>0&&(p.oScroll.sX!==""||p.oScroll.sY!==""))b=h("<tfoot/>").appendTo(q);if(b.length===0||b.children().length===0)q.addClass(u.sNoFooter);else if(b.length>0){p.nTFoot=b[0];ea(p.aoFooter,p.nTFoot)}if(g.aaData)for(j=0;j<g.aaData.length;j++)O(p,g.aaData[j]);else(p.bDeferLoading||y(p)=="dom")&&na(p,h(p.nTBody).children("tr"));p.aiDisplay=p.aiDisplayMaster.slice();
+p.bInitialised=true;m===false&&ha(p)};g.bStateSave?(U.bStateSave=!0,z(p,"aoDrawCallback",xa,"state_save"),Kb(p,g,e)):e()}});b=null;return this},x,s,o,u,Za={},Nb=/[\r\n]/g,Aa=/<.*?>/g,Zb=/^\d{2,4}[\.\/\-]\d{1,2}[\.\/\-]\d{1,2}([T ]{1}\d{1,2}[:\.]\d{2}([\.:]\d{2})?)?$/,$b=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Ya=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfkɃΞ]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Ob=function(a){var b=parseInt(a,10);return!isNaN(b)&&
+isFinite(a)?b:null},Pb=function(a,b){Za[b]||(Za[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,"").replace(Za[b],"."):a},$a=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Pb(a,b));c&&d&&(a=a.replace(Ya,""));return!isNaN(parseFloat(a))&&isFinite(a)},Qb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:$a(a.replace(Aa,""),b,c)?!0:null},D=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;e<f;e++)a[e]&&a[e][b]&&d.push(a[e][b][c]);else for(;e<
+f;e++)a[e]&&d.push(a[e][b]);return d},ja=function(a,b,c,d){var e=[],f=0,g=b.length;if(d!==k)for(;f<g;f++)a[b[f]][c]&&e.push(a[b[f]][c][d]);else for(;f<g;f++)e.push(a[b[f]][c]);return e},Y=function(a,b){var c=[],d;b===k?(b=0,d=a):(d=b,b=a);for(var e=b;e<d;e++)c.push(e);return c},Rb=function(a){for(var b=[],c=0,d=a.length;c<d;c++)a[c]&&b.push(a[c]);return b},qa=function(a){var b;a:{if(!(2>a.length)){b=a.slice().sort();for(var c=b[0],d=1,e=b.length;d<e;d++){if(b[d]===c){b=!1;break a}c=b[d]}}b=!0}if(b)return a.slice();
+b=[];var e=a.length,f,g=0,d=0;a:for(;d<e;d++){c=a[d];for(f=0;f<g;f++)if(b[f]===c)continue a;b.push(c);g++}return b};n.util={throttle:function(a,b){var c=b!==k?b:200,d,e;return function(){var b=this,g=+new Date,j=arguments;d&&g<d+c?(clearTimeout(e),e=setTimeout(function(){d=k;a.apply(b,j)},c)):(d=g,a.apply(b,j))}},escapeRegex:function(a){return a.replace($b,"\\$1")}};var A=function(a,b,c){a[b]!==k&&(a[c]=a[b])},ca=/\[.*?\]$/,W=/\(\)$/,Qa=n.util.escapeRegex,va=h("<div>")[0],Wb=va.textContent!==k,Yb=
+/<.*?>/g,Oa=n.util.throttle,Sb=[],w=Array.prototype,ac=function(a){var b,c,d=n.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};s=function(a,b){if(!(this instanceof
+s))return new s(a,b);var c=[],d=function(a){(a=ac(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;e<f;e++)d(a[e]);else d(a);this.context=qa(c);b&&h.merge(this,b);this.selector={rows:null,cols:null,opts:null};s.extend(this,this,Sb)};n.Api=s;h.extend(s.prototype,{any:function(){return 0!==this.count()},concat:w.concat,context:[],count:function(){return this.flatten().length},each:function(a){for(var b=0,c=this.length;b<c;b++)a.call(this,this[b],b,this);return this},eq:function(a){var b=
+this.context;return b.length>a?new s(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c<d;c++)a.call(this,this[c],c,this)&&b.push(this[c]);return new s(this.context,b)},flatten:function(){var a=[];return new s(this.context,a.concat.apply(a,this.toArray()))},join:w.join,indexOf:w.indexOf||function(a,b){for(var c=b||0,d=this.length;c<d;c++)if(this[c]===a)return c;return-1},iterator:function(a,b,c,d){var e=[],f,g,j,h,m,l=this.context,
+n,o,u=this.selector;"string"===typeof a&&(d=c,c=b,b=a,a=!1);g=0;for(j=l.length;g<j;g++){var r=new s(l[g]);if("table"===b)f=c.call(r,l[g],g),f!==k&&e.push(f);else if("columns"===b||"rows"===b)f=c.call(r,l[g],this[g],g),f!==k&&e.push(f);else if("column"===b||"column-rows"===b||"row"===b||"cell"===b){o=this[g];"column-rows"===b&&(n=Ba(l[g],u.opts));h=0;for(m=o.length;h<m;h++)f=o[h],f="cell"===b?c.call(r,l[g],f.row,f.column,g,h):c.call(r,l[g],f,g,h,n),f!==k&&e.push(f)}}return e.length||d?(a=new s(l,a?
+e.concat.apply([],e):e),b=a.selector,b.rows=u.rows,b.cols=u.cols,b.opts=u.opts,a):this},lastIndexOf:w.lastIndexOf||function(a,b){return this.indexOf.apply(this.toArray.reverse(),arguments)},length:0,map:function(a){var b=[];if(w.map)b=w.map.call(this,a,this);else for(var c=0,d=this.length;c<d;c++)b.push(a.call(this,this[c],c));return new s(this.context,b)},pluck:function(a){return this.map(function(b){return b[a]})},pop:w.pop,push:w.push,reduce:w.reduce||function(a,b){return ib(this,a,b,0,this.length,
+1)},reduceRight:w.reduceRight||function(a,b){return ib(this,a,b,this.length-1,-1,-1)},reverse:w.reverse,selector:null,shift:w.shift,slice:function(){return new s(this.context,this)},sort:w.sort,splice:w.splice,toArray:function(){return w.slice.call(this)},to$:function(){return h(this)},toJQuery:function(){return h(this)},unique:function(){return new s(this.context,qa(this))},unshift:w.unshift});s.extend=function(a,b,c){if(c.length&&b&&(b instanceof s||b.__dt_wrapper)){var d,e,f,g=function(a,b,c){return function(){var d=
+b.apply(a,arguments);s.extend(d,d,c.methodExt);return d}};d=0;for(e=c.length;d<e;d++)f=c[d],b[f.name]="function"===typeof f.val?g(a,f.val,f):h.isPlainObject(f.val)?{}:f.val,b[f.name].__dt_wrapper=!0,s.extend(a,b[f.name],f.propExt)}};s.register=o=function(a,b){if(h.isArray(a))for(var c=0,d=a.length;c<d;c++)s.register(a[c],b);else for(var e=a.split("."),f=Sb,g,j,c=0,d=e.length;c<d;c++){g=(j=-1!==e[c].indexOf("()"))?e[c].replace("()",""):e[c];var i;a:{i=0;for(var m=f.length;i<m;i++)if(f[i].name===g){i=
+f[i];break a}i=null}i||(i={name:g,val:{},methodExt:[],propExt:[]},f.push(i));c===d-1?i.val=b:f=j?i.methodExt:i.propExt}};s.registerPlural=u=function(a,b,c){s.register(a,c);s.register(b,function(){var a=c.apply(this,arguments);return a===this?this:a instanceof s?a.length?h.isArray(a[0])?new s(a.context,a[0]):a[0]:k:a})};o("tables()",function(a){var b;if(a){b=s;var c=this.context;if("number"===typeof a)a=[c[a]];else var d=h.map(c,function(a){return a.nTable}),a=h(d).filter(a).map(function(){var a=h.inArray(this,
+d);return c[a]}).toArray();b=new b(a)}else b=this;return b});o("table()",function(a){var a=this.tables(a),b=a.context;return b.length?new s(b[0]):a});u("tables().nodes()","table().node()",function(){return this.iterator("table",function(a){return a.nTable},1)});u("tables().body()","table().body()",function(){return this.iterator("table",function(a){return a.nTBody},1)});u("tables().header()","table().header()",function(){return this.iterator("table",function(a){return a.nTHead},1)});u("tables().footer()",
+"table().footer()",function(){return this.iterator("table",function(a){return a.nTFoot},1)});u("tables().containers()","table().container()",function(){return this.iterator("table",function(a){return a.nTableWrapper},1)});o("draw()",function(a){return this.iterator("table",function(b){"page"===a?P(b):("string"===typeof a&&(a="full-hold"===a?!1:!0),T(b,!1===a))})});o("page()",function(a){return a===k?this.page.info().page:this.iterator("table",function(b){Ta(b,a)})});o("page.info()",function(){if(0===
+this.context.length)return k;var a=this.context[0],b=a._iDisplayStart,c=a.oFeatures.bPaginate?a._iDisplayLength:-1,d=a.fnRecordsDisplay(),e=-1===c;return{page:e?0:Math.floor(b/c),pages:e?1:Math.ceil(d/c),start:b,end:a.fnDisplayEnd(),length:c,recordsTotal:a.fnRecordsTotal(),recordsDisplay:d,serverSide:"ssp"===y(a)}});o("page.len()",function(a){return a===k?0!==this.context.length?this.context[0]._iDisplayLength:k:this.iterator("table",function(b){Ra(b,a)})});var Tb=function(a,b,c){if(c){var d=new s(a);
+d.one("draw",function(){c(d.ajax.json())})}if("ssp"==y(a))T(a,b);else{C(a,!0);var e=a.jqXHR;e&&4!==e.readyState&&e.abort();sa(a,[],function(c){oa(a);for(var c=ta(a,c),d=0,e=c.length;d<e;d++)O(a,c[d]);T(a,b);C(a,!1)})}};o("ajax.json()",function(){var a=this.context;if(0<a.length)return a[0].json});o("ajax.params()",function(){var a=this.context;if(0<a.length)return a[0].oAjaxData});o("ajax.reload()",function(a,b){return this.iterator("table",function(c){Tb(c,!1===b,a)})});o("ajax.url()",function(a){var b=
+this.context;if(a===k){if(0===b.length)return k;b=b[0];return b.ajax?h.isPlainObject(b.ajax)?b.ajax.url:b.ajax:b.sAjaxSource}return this.iterator("table",function(b){h.isPlainObject(b.ajax)?b.ajax.url=a:b.ajax=a})});o("ajax.url().load()",function(a,b){return this.iterator("table",function(c){Tb(c,!1===b,a)})});var ab=function(a,b,c,d,e){var f=[],g,j,i,m,l,n;i=typeof b;if(!b||"string"===i||"function"===i||b.length===k)b=[b];i=0;for(m=b.length;i<m;i++){j=b[i]&&b[i].split&&!b[i].match(/[\[\(:]/)?b[i].split(","):
+[b[i]];l=0;for(n=j.length;l<n;l++)(g=c("string"===typeof j[l]?h.trim(j[l]):j[l]))&&g.length&&(f=f.concat(g))}a=x.selector[a];if(a.length){i=0;for(m=a.length;i<m;i++)f=a[i](d,e,f)}return qa(f)},bb=function(a){a||(a={});a.filter&&a.search===k&&(a.search=a.filter);return h.extend({search:"none",order:"current",page:"all"},a)},cb=function(a){for(var b=0,c=a.length;b<c;b++)if(0<a[b].length)return a[0]=a[b],a[0].length=1,a.length=1,a.context=[a.context[b]],a;a.length=0;return a},Ba=function(a,b){var c,
+d,e,f=[],g=a.aiDisplay;e=a.aiDisplayMaster;var j=b.search;c=b.order;d=b.page;if("ssp"==y(a))return"removed"===j?[]:Y(0,e.length);if("current"==d){c=a._iDisplayStart;for(d=a.fnDisplayEnd();c<d;c++)f.push(g[c])}else if("current"==c||"applied"==c)if("none"==j)f=e.slice();else if("applied"==j)f=g.slice();else{if("removed"==j){var i={};c=0;for(d=g.length;c<d;c++)i[g[c]]=null;f=h.map(e,function(a){return!i.hasOwnProperty(a)?a:null})}}else if("index"==c||"original"==c){c=0;for(d=a.aoData.length;c<d;c++)"none"==
+j?f.push(c):(e=h.inArray(c,g),(-1===e&&"removed"==j||0<=e&&"applied"==j)&&f.push(c))}return f};o("rows()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=bb(b),c=this.iterator("table",function(c){var e=b,f;return ab("row",a,function(a){var b=Ob(a),i=c.aoData;if(b!==null&&!e)return[b];f||(f=Ba(c,e));if(b!==null&&h.inArray(b,f)!==-1)return[b];if(a===null||a===k||a==="")return f;if(typeof a==="function")return h.map(f,function(b){var c=i[b];return a(b,c._aData,c.nTr)?b:null});if(a.nodeName){var b=
+a._DT_RowIndex,m=a._DT_CellIndex;if(b!==k)return i[b]&&i[b].nTr===a?[b]:[];if(m)return i[m.row]&&i[m.row].nTr===a?[m.row]:[];b=h(a).closest("*[data-dt-row]");return b.length?[b.data("dt-row")]:[]}if(typeof a==="string"&&a.charAt(0)==="#"){b=c.aIds[a.replace(/^#/,"")];if(b!==k)return[b.idx]}b=Rb(ja(c.aoData,f,"nTr"));return h(b).filter(a).map(function(){return this._DT_RowIndex}).toArray()},c,e)},1);c.selector.rows=a;c.selector.opts=b;return c});o("rows().nodes()",function(){return this.iterator("row",
+function(a,b){return a.aoData[b].nTr||k},1)});o("rows().data()",function(){return this.iterator(!0,"rows",function(a,b){return ja(a.aoData,b,"_aData")},1)});u("rows().cache()","row().cache()",function(a){return this.iterator("row",function(b,c){var d=b.aoData[c];return"search"===a?d._aFilterData:d._aSortData},1)});u("rows().invalidate()","row().invalidate()",function(a){return this.iterator("row",function(b,c){da(b,c,a)})});u("rows().indexes()","row().index()",function(){return this.iterator("row",
+function(a,b){return b},1)});u("rows().ids()","row().id()",function(a){for(var b=[],c=this.context,d=0,e=c.length;d<e;d++)for(var f=0,g=this[d].length;f<g;f++){var h=c[d].rowIdFn(c[d].aoData[this[d][f]]._aData);b.push((!0===a?"#":"")+h)}return new s(c,b)});u("rows().remove()","row().remove()",function(){var a=this;this.iterator("row",function(b,c,d){var e=b.aoData,f=e[c],g,h,i,m,l;e.splice(c,1);g=0;for(h=e.length;g<h;g++)if(i=e[g],l=i.anCells,null!==i.nTr&&(i.nTr._DT_RowIndex=g),null!==l){i=0;for(m=
+l.length;i<m;i++)l[i]._DT_CellIndex.row=g}pa(b.aiDisplayMaster,c);pa(b.aiDisplay,c);pa(a[d],c,!1);0<b._iRecordsDisplay&&b._iRecordsDisplay--;Sa(b);c=b.rowIdFn(f._aData);c!==k&&delete b.aIds[c]});this.iterator("table",function(a){for(var c=0,d=a.aoData.length;c<d;c++)a.aoData[c].idx=c});return this});o("rows.add()",function(a){var b=this.iterator("table",function(b){var c,f,g,h=[];f=0;for(g=a.length;f<g;f++)c=a[f],c.nodeName&&"TR"===c.nodeName.toUpperCase()?h.push(na(b,c)[0]):h.push(O(b,c));return h},
+1),c=this.rows(-1);c.pop();h.merge(c,b);return c});o("row()",function(a,b){return cb(this.rows(a,b))});o("row().data()",function(a){var b=this.context;if(a===k)return b.length&&this.length?b[0].aoData[this[0]]._aData:k;var c=b[0].aoData[this[0]];c._aData=a;h.isArray(a)&&c.nTr.id&&N(b[0].rowId)(a,c.nTr.id);da(b[0],this[0],"data");return this});o("row().node()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]].nTr||null:null});o("row.add()",function(a){a instanceof h&&
+a.length&&(a=a[0]);var b=this.iterator("table",function(b){return a.nodeName&&"TR"===a.nodeName.toUpperCase()?na(b,a)[0]:O(b,a)});return this.row(b[0])});var db=function(a,b){var c=a.context;if(c.length&&(c=c[0].aoData[b!==k?b:a[0]])&&c._details)c._details.remove(),c._detailsShow=k,c._details=k},Ub=function(a,b){var c=a.context;if(c.length&&a.length){var d=c[0].aoData[a[0]];if(d._details){(d._detailsShow=b)?d._details.insertAfter(d.nTr):d._details.detach();var e=c[0],f=new s(e),g=e.aoData;f.off("draw.dt.DT_details column-visibility.dt.DT_details destroy.dt.DT_details");
+0<D(g,"_details").length&&(f.on("draw.dt.DT_details",function(a,b){e===b&&f.rows({page:"current"}).eq(0).each(function(a){a=g[a];a._detailsShow&&a._details.insertAfter(a.nTr)})}),f.on("column-visibility.dt.DT_details",function(a,b){if(e===b)for(var c,d=V(b),f=0,h=g.length;f<h;f++)c=g[f],c._details&&c._details.children("td[colspan]").attr("colspan",d)}),f.on("destroy.dt.DT_details",function(a,b){if(e===b)for(var c=0,d=g.length;c<d;c++)g[c]._details&&db(f,c)}))}}};o("row().child()",function(a,b){var c=
+this.context;if(a===k)return c.length&&this.length?c[0].aoData[this[0]]._details:k;if(!0===a)this.child.show();else if(!1===a)db(this);else if(c.length&&this.length){var d=c[0],c=c[0].aoData[this[0]],e=[],f=function(a,b){if(h.isArray(a)||a instanceof h)for(var c=0,k=a.length;c<k;c++)f(a[c],b);else a.nodeName&&"tr"===a.nodeName.toLowerCase()?e.push(a):(c=h("<tr><td/></tr>").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=V(d),e.push(c[0]))};f(a,b);c._details&&c._details.detach();c._details=h(e);
+c._detailsShow&&c._details.insertAfter(c.nTr)}return this});o(["row().child.show()","row().child().show()"],function(){Ub(this,!0);return this});o(["row().child.hide()","row().child().hide()"],function(){Ub(this,!1);return this});o(["row().child.remove()","row().child().remove()"],function(){db(this);return this});o("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var bc=/^([^:]+):(name|visIdx|visible)$/,Vb=function(a,b,
+c,d,e){for(var c=[],d=0,f=e.length;d<f;d++)c.push(B(a,e[d],b));return c};o("columns()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=bb(b),c=this.iterator("table",function(c){var e=a,f=b,g=c.aoColumns,j=D(g,"sName"),i=D(g,"nTh");return ab("column",e,function(a){var b=Ob(a);if(a==="")return Y(g.length);if(b!==null)return[b>=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Vb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(bc):
+"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],10);if(b<0){var n=h.map(g,function(a,b){return a.bVisible?b:null});return[n[n.length+b]]}return[aa(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)},
+1);c.selector.cols=a;c.selector.opts=b;return c});u("columns().header()","column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});u("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});u("columns().data()","column().data()",function(){return this.iterator("column-rows",Vb,1)});u("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData},
+1)});u("columns().cache()","column().cache()",function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ja(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});u("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ja(a.aoData,e,"anCells",b)},1)});u("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData,
+i,m,l;if(a!==k&&g.bVisible!==a){if(a){var n=h.inArray(!0,D(f,"bVisible"),c+1);i=0;for(m=j.length;i<m;i++)l=j[i].nTr,f=j[i].anCells,l&&l.insertBefore(f[c],f[n]||null)}else h(D(b.aoData,"anCells",c)).detach();g.bVisible=a;fa(b,b.aoHeader);fa(b,b.aoFooter);b.aiDisplay.length||h(b.nTBody).find("td[colspan]").attr("colspan",V(b));xa(b)}});a!==k&&(this.iterator("column",function(c,e){r(c,null,"column-visibility",[c,e,a,b])}),(b===k||b)&&this.columns.adjust());return c});u("columns().indexes()","column().index()",
+function(a){return this.iterator("column",function(b,c){return"visible"===a?ba(b,c):c},1)});o("columns.adjust()",function(){return this.iterator("table",function(a){$(a)},1)});o("column.index()",function(a,b){if(0!==this.context.length){var c=this.context[0];if("fromVisible"===a||"toData"===a)return aa(c,b);if("fromData"===a||"toVisible"===a)return ba(c,b)}});o("column()",function(a,b){return cb(this.columns(a,b))});o("cells()",function(a,b,c){h.isPlainObject(a)&&(a.row===k?(c=a,a=null):(c=b,b=null));
+h.isPlainObject(b)&&(c=b,b=null);if(null===b||b===k)return this.iterator("table",function(b){var d=a,e=bb(c),f=b.aoData,g=Ba(b,e),j=Rb(ja(f,g,"anCells")),i=h([].concat.apply([],j)),l,m=b.aoColumns.length,n,o,u,s,r,v;return ab("cell",d,function(a){var c=typeof a==="function";if(a===null||a===k||c){n=[];o=0;for(u=g.length;o<u;o++){l=g[o];for(s=0;s<m;s++){r={row:l,column:s};if(c){v=f[l];a(r,B(b,l,s),v.anCells?v.anCells[s]:null)&&n.push(r)}else n.push(r)}}return n}if(h.isPlainObject(a))return a.column!==
+k&&a.row!==k&&h.inArray(a.row,g)!==-1?[a]:[];c=i.filter(a).map(function(a,b){return{row:b._DT_CellIndex.row,column:b._DT_CellIndex.column}}).toArray();if(c.length||!a.nodeName)return c;v=h(a).closest("*[data-dt-row]");return v.length?[{row:v.data("dt-row"),column:v.data("dt-column")}]:[]},b,e)});var d=this.columns(b),e=this.rows(a),f,g,j,i,m;this.iterator("table",function(a,b){f=[];g=0;for(j=e[b].length;g<j;g++){i=0;for(m=d[b].length;i<m;i++)f.push({row:e[b][g],column:d[b][i]})}},1);var l=this.cells(f,
+c);h.extend(l.selector,{cols:b,rows:a,opts:c});return l});u("cells().nodes()","cell().node()",function(){return this.iterator("cell",function(a,b,c){return(a=a.aoData[b])&&a.anCells?a.anCells[c]:k},1)});o("cells().data()",function(){return this.iterator("cell",function(a,b,c){return B(a,b,c)},1)});u("cells().cache()","cell().cache()",function(a){a="search"===a?"_aFilterData":"_aSortData";return this.iterator("cell",function(b,c,d){return b.aoData[c][a][d]},1)});u("cells().render()","cell().render()",
+function(a){return this.iterator("cell",function(b,c,d){return B(b,c,d,a)},1)});u("cells().indexes()","cell().index()",function(){return this.iterator("cell",function(a,b,c){return{row:b,column:c,columnVisible:ba(a,c)}},1)});u("cells().invalidate()","cell().invalidate()",function(a){return this.iterator("cell",function(b,c,d){da(b,c,a,d)})});o("cell()",function(a,b,c){return cb(this.cells(a,b,c))});o("cell().data()",function(a){var b=this.context,c=this[0];if(a===k)return b.length&&c.length?B(b[0],
+c[0].row,c[0].column):k;kb(b[0],c[0].row,c[0].column,a);da(b[0],c[0].row,"data",c[0].column);return this});o("order()",function(a,b){var c=this.context;if(a===k)return 0!==c.length?c[0].aaSorting:k;"number"===typeof a?a=[[a,b]]:a.length&&!h.isArray(a[0])&&(a=Array.prototype.slice.call(arguments));return this.iterator("table",function(b){b.aaSorting=a.slice()})});o("order.listener()",function(a,b,c){return this.iterator("table",function(d){Ma(d,a,b,c)})});o("order.fixed()",function(a){if(!a){var b=
+this.context,b=b.length?b[0].aaSortingFixed:k;return h.isArray(b)?{pre:b}:b}return this.iterator("table",function(b){b.aaSortingFixed=h.extend(!0,{},a)})});o(["columns().order()","column().order()"],function(a){var b=this;return this.iterator("table",function(c,d){var e=[];h.each(b[d],function(b,c){e.push([c,a])});c.aaSorting=e})});o("search()",function(a,b,c,d){var e=this.context;return a===k?0!==e.length?e[0].oPreviousSearch.sSearch:k:this.iterator("table",function(e){e.oFeatures.bFilter&&ga(e,
+h.extend({},e.oPreviousSearch,{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),1)})});u("columns().search()","column().search()",function(a,b,c,d){return this.iterator("column",function(e,f){var g=e.aoPreSearchCols;if(a===k)return g[f].sSearch;e.oFeatures.bFilter&&(h.extend(g[f],{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),ga(e,e.oPreviousSearch,1))})});o("state()",function(){return this.context.length?this.context[0].oSavedState:
+null});o("state.clear()",function(){return this.iterator("table",function(a){a.fnStateSaveCallback.call(a.oInstance,a,{})})});o("state.loaded()",function(){return this.context.length?this.context[0].oLoadedState:null});o("state.save()",function(){return this.iterator("table",function(a){xa(a)})});n.versionCheck=n.fnVersionCheck=function(a){for(var b=n.version.split("."),a=a.split("."),c,d,e=0,f=a.length;e<f;e++)if(c=parseInt(b[e],10)||0,d=parseInt(a[e],10)||0,c!==d)return c>d;return!0};n.isDataTable=
+n.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;if(a instanceof n.Api)return!0;h.each(n.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:null;if(e.nTable===b||f===b||g===b)c=!0});return c};n.tables=n.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(n.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new s(c):c};n.camelToHungarian=J;o("$()",function(a,b){var c=
+this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){o(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0]=h.map(a[0].split(/\s/),function(a){return!a.match(/\.dt\b/)?a+".dt":a}).join(" ");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});o("clear()",function(){return this.iterator("table",function(a){oa(a)})});o("settings()",function(){return new s(this.context,this.context)});o("init()",function(){var a=
+this.context;return a.length?a[0].oInit:null});o("data()",function(){return this.iterator("table",function(a){return D(a.aoData,"_aData")}).flatten()});o("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),o;b.bDestroying=!0;r(b,"aoDestroyCallback","destroy",[b]);a||(new s(b)).columns().visible(!0);k.off(".DT").find(":not(tbody *)").off(".DT");
+h(E).off(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];wa(b);h(l).removeClass(b.asStripeClasses.join(" "));h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable),
+(o=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%o])}));c=h.inArray(b,n.settings);-1!==c&&n.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){o(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,m){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,m)})})});o("i18n()",function(a,b,c){var d=this.context[0],a=S(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]:
+a._);return a.replace("%d",c)});n.version="1.10.19";n.settings=[];n.models={};n.models.oSearch={bCaseInsensitive:!0,sSearch:"",bRegex:!1,bSmart:!0};n.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};n.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null,
+sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,sTitle:null,sType:null,sWidth:null,sWidthOrig:null};n.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1,
+bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+
+a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"},
+oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({},
+n.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",renderer:null,rowId:"DT_RowId"};Z(n.defaults);n.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null};
+Z(n.defaults.column);n.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[],
+aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button",
+iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal:
+this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};n.ext=x={buttons:{},
+classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:n.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:n.version};h.extend(x,{afnFiltering:x.search,aTypes:x.type.detect,ofnSearch:x.type.search,oSort:x.type.order,afnSortData:x.order,aoFeatures:x.feature,oApi:x.internal,oStdClasses:x.classes,oPagination:x.pager});
+h.extend(n.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled",
+sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"",
+sJUIHeader:"",sJUIFooter:""});var Lb=n.ext.pager;h.extend(Lb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ia(a,b)]},simple_numbers:function(a,b){return["previous",ia(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ia(a,b),"next","last"]},first_last_numbers:function(a,b){return["first",ia(a,b),"last"]},_numbers:ia,numbers_length:7});h.extend(!0,n.ext.renderer,{pageButton:{_:function(a,b,c,d,e,
+f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},m,l,n=0,o=function(b,d){var k,s,u,r,v=function(b){Ta(a,b.data.action,true)};k=0;for(s=d.length;k<s;k++){r=d[k];if(h.isArray(r)){u=h("<"+(r.DT_el||"div")+"/>").appendTo(b);o(u,r)}else{m=null;l="";switch(r){case "ellipsis":b.append('<span class="ellipsis">&#x2026;</span>');break;case "first":m=j.sFirst;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":m=j.sPrevious;l=r+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":m=
+j.sNext;l=r+(e<f-1?"":" "+g.sPageButtonDisabled);break;case "last":m=j.sLast;l=r+(e<f-1?"":" "+g.sPageButtonDisabled);break;default:m=r+1;l=e===r?g.sPageButtonActive:""}if(m!==null){u=h("<a>",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[r],"data-dt-idx":n,tabindex:a.iTabIndex,id:c===0&&typeof r==="string"?a.sTableId+"_"+r:null}).html(m).appendTo(b);Wa(u,{action:r},v);n++}}}},s;try{s=h(b).find(H.activeElement).data("dt-idx")}catch(u){}o(h(b).empty(),d);s!==k&&h(b).find("[data-dt-idx="+
+s+"]").focus()}}});h.extend(n.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&!Zb.test(a))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":null},function(a,b){var c=b.oLanguage.sDecimal;return $a(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Qb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)||
+"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(n.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Nb," "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Pb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(x.type.order,{"date-pre":function(a){a=Date.parse(a);return isNaN(a)?-Infinity:a},"html-pre":function(a){return M(a)?
+"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return a<b?-1:a>b?1:0},"string-desc":function(a,b){return a<b?1:a>b?-1:0}});Da("");h.extend(!0,n.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:
+c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("<div/>").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("<span/>").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]==
+"asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var eb=function(a){return"string"===typeof a?a.replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;"):a};n.render={number:function(a,b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return eb(f);h=h.toFixed(c);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g,
+a)+f+(e||"")}}},text:function(){return{display:eb,filter:eb}}};h.extend(n.ext.internal,{_fnExternApiFunc:Mb,_fnBuildAjax:sa,_fnAjaxUpdate:mb,_fnAjaxParameters:vb,_fnAjaxUpdateDraw:wb,_fnAjaxDataSrc:ta,_fnAddColumn:Ea,_fnColumnOptions:ka,_fnAdjustColumnSizing:$,_fnVisibleToColumnIndex:aa,_fnColumnIndexToVisible:ba,_fnVisbleColumns:V,_fnGetColumns:ma,_fnColumnTypes:Ga,_fnApplyColumnDefs:jb,_fnHungarianMap:Z,_fnCamelToHungarian:J,_fnLanguageCompat:Ca,_fnBrowserDetect:hb,_fnAddData:O,_fnAddTr:na,_fnNodeToDataIndex:function(a,
+b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:kb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:S,_fnSetObjectDataFn:N,_fnGetDataMaster:Ka,_fnClearTable:oa,_fnDeleteIndex:pa,_fnInvalidate:da,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:lb,_fnDrawHead:fa,_fnDraw:P,_fnReDraw:T,_fnAddOptionsHtml:ob,_fnDetectHeader:ea,_fnGetUniqueThs:ra,_fnFeatureHtmlFilter:qb,_fnFilterComplete:ga,_fnFilterCustom:zb,
+_fnFilterColumn:yb,_fnFilter:xb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:Ab,_fnFeatureHtmlInfo:tb,_fnUpdateInfo:Db,_fnInfoMacros:Eb,_fnInitialise:ha,_fnInitComplete:ua,_fnLengthChange:Ra,_fnFeatureHtmlLength:pb,_fnFeatureHtmlPaginate:ub,_fnPageChange:Ta,_fnFeatureHtmlProcessing:rb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:sb,_fnScrollDraw:la,_fnApplyToChildren:I,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:v,
+_fnSortFlatten:X,_fnSort:nb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:wa,_fnSortData:Ib,_fnSaveState:xa,_fnLoadState:Kb,_fnSettingsFromNode:ya,_fnLog:K,_fnMap:F,_fnBindAction:Wa,_fnCallbackReg:z,_fnCallbackFire:r,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnExtend:Xa,_fnCalculateEnd:function(){}});h.fn.dataTable=n;n.$=h;h.fn.dataTableSettings=n.settings;h.fn.dataTableExt=n.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()};
+h.each(n,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable});
@@ -0,0 +1,2 @@
+/*! jQuery v3.3.1 | (c) JS Foundation and other contributors | jquery.org/license */
+!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(e,t){"use strict";var n=[],r=e.document,i=Object.getPrototypeOf,o=n.slice,a=n.concat,s=n.push,u=n.indexOf,l={},c=l.toString,f=l.hasOwnProperty,p=f.toString,d=p.call(Object),h={},g=function e(t){return"function"==typeof t&&"number"!=typeof t.nodeType},y=function e(t){return null!=t&&t===t.window},v={type:!0,src:!0,noModule:!0};function m(e,t,n){var i,o=(t=t||r).createElement("script");if(o.text=e,n)for(i in v)n[i]&&(o[i]=n[i]);t.head.appendChild(o).parentNode.removeChild(o)}function x(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?l[c.call(e)]||"object":typeof e}var b="3.3.1",w=function(e,t){return new w.fn.init(e,t)},T=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g;w.fn=w.prototype={jquery:"3.3.1",constructor:w,length:0,toArray:function(){return o.call(this)},get:function(e){return null==e?o.call(this):e<0?this[e+this.length]:this[e]},pushStack:function(e){var t=w.merge(this.constructor(),e);return t.prevObject=this,t},each:function(e){return w.each(this,e)},map:function(e){return this.pushStack(w.map(this,function(t,n){return e.call(t,n,t)}))},slice:function(){return this.pushStack(o.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(e){var t=this.length,n=+e+(e<0?t:0);return this.pushStack(n>=0&&n<t?[this[n]]:[])},end:function(){return this.prevObject||this.constructor()},push:s,sort:n.sort,splice:n.splice},w.extend=w.fn.extend=function(){var e,t,n,r,i,o,a=arguments[0]||{},s=1,u=arguments.length,l=!1;for("boolean"==typeof a&&(l=a,a=arguments[s]||{},s++),"object"==typeof a||g(a)||(a={}),s===u&&(a=this,s--);s<u;s++)if(null!=(e=arguments[s]))for(t in e)n=a[t],a!==(r=e[t])&&(l&&r&&(w.isPlainObject(r)||(i=Array.isArray(r)))?(i?(i=!1,o=n&&Array.isArray(n)?n:[]):o=n&&w.isPlainObject(n)?n:{},a[t]=w.extend(l,o,r)):void 0!==r&&(a[t]=r));return a},w.extend({expando:"jQuery"+("3.3.1"+Math.random()).replace(/\D/g,""),isReady:!0,error:function(e){throw new Error(e)},noop:function(){},isPlainObject:function(e){var t,n;return!(!e||"[object Object]"!==c.call(e))&&(!(t=i(e))||"function"==typeof(n=f.call(t,"constructor")&&t.constructor)&&p.call(n)===d)},isEmptyObject:function(e){var t;for(t in e)return!1;return!0},globalEval:function(e){m(e)},each:function(e,t){var n,r=0;if(C(e)){for(n=e.length;r<n;r++)if(!1===t.call(e[r],r,e[r]))break}else for(r in e)if(!1===t.call(e[r],r,e[r]))break;return e},trim:function(e){return null==e?"":(e+"").replace(T,"")},makeArray:function(e,t){var n=t||[];return null!=e&&(C(Object(e))?w.merge(n,"string"==typeof e?[e]:e):s.call(n,e)),n},inArray:function(e,t,n){return null==t?-1:u.call(t,e,n)},merge:function(e,t){for(var n=+t.length,r=0,i=e.length;r<n;r++)e[i++]=t[r];return e.length=i,e},grep:function(e,t,n){for(var r,i=[],o=0,a=e.length,s=!n;o<a;o++)(r=!t(e[o],o))!==s&&i.push(e[o]);return i},map:function(e,t,n){var r,i,o=0,s=[];if(C(e))for(r=e.length;o<r;o++)null!=(i=t(e[o],o,n))&&s.push(i);else for(o in e)null!=(i=t(e[o],o,n))&&s.push(i);return a.apply([],s)},guid:1,support:h}),"function"==typeof Symbol&&(w.fn[Symbol.iterator]=n[Symbol.iterator]),w.each("Boolean Number String Function Array Date RegExp Object Error Symbol".split(" "),function(e,t){l["[object "+t+"]"]=t.toLowerCase()});function C(e){var t=!!e&&"length"in e&&e.length,n=x(e);return!g(e)&&!y(e)&&("array"===n||0===t||"number"==typeof t&&t>0&&t-1 in e)}var E=function(e){var t,n,r,i,o,a,s,u,l,c,f,p,d,h,g,y,v,m,x,b="sizzle"+1*new Date,w=e.document,T=0,C=0,E=ae(),k=ae(),S=ae(),D=function(e,t){return e===t&&(f=!0),0},N={}.hasOwnProperty,A=[],j=A.pop,q=A.push,L=A.push,H=A.slice,O=function(e,t){for(var n=0,r=e.length;n<r;n++)if(e[n]===t)return n;return-1},P="checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|ismap|loop|multiple|open|readonly|required|scoped",M="[\\x20\\t\\r\\n\\f]",R="(?:\\\\.|[\\w-]|[^\0-\\xa0])+",I="\\["+M+"*("+R+")(?:"+M+"*([*^$|!~]?=)"+M+"*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|("+R+"))|)"+M+"*\\]",W=":("+R+")(?:\\((('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|((?:\\\\.|[^\\\\()[\\]]|"+I+")*)|.*)\\)|)",$=new RegExp(M+"+","g"),B=new RegExp("^"+M+"+|((?:^|[^\\\\])(?:\\\\.)*)"+M+"+$","g"),F=new RegExp("^"+M+"*,"+M+"*"),_=new RegExp("^"+M+"*([>+~]|"+M+")"+M+"*"),z=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),X=new RegExp(W),U=new RegExp("^"+R+"$"),V={ID:new RegExp("^#("+R+")"),CLASS:new RegExp("^\\.("+R+")"),TAG:new RegExp("^("+R+"|[*])"),ATTR:new RegExp("^"+I),PSEUDO:new RegExp("^"+W),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+P+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)Line truncated
@@ -21,7 +21,7 @@
        <DT><A HREF="https://duckduckgo.com/?q=firefox+export+bookmarks&t=ffhp&ia=web" ADD_DATE="1497562974" LAST_MODIFIED="1497562974" ICON_URI="https://duckduckgo.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAG7UlEQVRYha2XbYwbVxWG798oorJQhYKqSJPxzJ35gWDVComiKrK86/H4Y+2J21/5kIyUgiKk1GqArVpEXQlKSlFNm35IFMVREgQNJE4KAsqCpiGtsrsIJjvO2N6xvbPj9SbZJBvjdsmGzcYvP+xxxt6PbBFHeiTP9T3nfe+ZO+NrQjYZmsJ4aoo3WU+w2fqTO7S5BA839Sd3aPUEm60p3qSmMJ7N1n1gWArnqyfYbD3Bw2F2F9eYVTi1h11cwz2nlvDmLIXz/c/CmsJ46gk2W9/Fo76LR11hrXqCTVmKyKxvVmTqCTZVV1irm5dgs5+5I5bCDNQV1prtCDsrsRRmwE7QdE3hVOd7Z05N4VQ7QdOWwgx0O9dToz2+KXE7zjVqCg87xmWcYrU4p9YUHpsizqmOaTvGZWoKDzvONR5ownSJO5vJjrPZ8ruv4E5+HHfy41j862ksvDGC+p7HHmjEjrfbX1O8SceEuZ4JTWE8M1GuYcfvi8/Edmh2nMeN359Ef9xbbOL6ywdgx/kNmYnt0BwTdpzHTJRrrLkn7DibteM87GEuoymMx+qI23EehQMhFA6EYP/mXTSbzR4jn7x/DFdSsQ1NWB0T9jCXaY+x2R7xsiL6ZmIUVozX2mb47EyMwuHKMzHcna+v6oI7br4+AndOP3aczxJCiBXjtZkYRVkRfV0Dlbg3Z3UGy4ros2IUDuaer2HxxrUNxZ24+vxeuHP7cdevxL05QgghRUVkrBhFtbP66SinupOMt38IAJiYvIpj7+n4p351lfDpP5p4Nv0XfPzLcxsamI5yKiGEVGO8ZsUoiorIkGpESE0PU1QjQsoMMAPTwxRu9DPtDfjzX+mb6kL9YBz9NdyYAWbArUkqUW+uOtx2Ux7mMtVhCjf66ftPwNx8E8Xp62sKH3z5HCbys/jXuWPor+GmPMxliorIVIcpKlFvjlTCrGWG+AYhhExHOLUapXDzj/eyXZHHd7+Fw79Q1zTxpfhreOH1P+G2Po7+Gm6mI+3bUAmzViXMWqQSpSh3BqdCfKMSpXBz/s3DXZFvvHAKj+9+C3PzzVUG5uabmJtv4rY+jkqUYvopDtd+8Ag++fXnsXjOgyvf245KlGKqs9hyhFMrUQpSiVCU5Y6rCEU/Y+mDq579/mgtTaK1+De0bp3AXfNbuP3BQ1jRt/RwM7OtW5MQQsoyp1YiFKQcoTA7BsoRin70b4YxOzu7tvJ/ZnCvGlwl5rA8sRW33vkCrL1sT01CCDFlTi1HKIgZvm+gJPENM0zhprTrK5icnFxTf6UkdsUWz3q6NLIPY+6729FfywxTlKT2LTBlTjXDFGQqxFqlEGu5B/u5+IdzGxpYnti6ppgZppjew/aOOYsNsdZUiLVIKeTNTYUpNFlkzCCXmQpT9PPhkcNrGmgtTWLF2IYVfQv+/cFDuPHati6fnvVgeWIrVvQtuPajL3ZrmUEuo8kiMxWmKIW8OVIICalSiMKQvUkzwAyUQhT9jD/3NBYWFtY0URvZi9nvbEcj+zDunP9czx749KwHV77/SE8tM8AMGLI3WQpRFEJCimiyyJRCFCWp/SouyqzVb0DbH4ZlWesaWMv0WhTl9q0uSbxWCrW7TgghpBDk1KJMkZdFX0HyKkWZop/1NuKNk0dWzV2PguRV8rLoK8oUhWB7LxBCCMnLoq8gUxidLhiSN1eQKdx8dOoEAODm0jWUbt03s1QtdOf8+ekv4/A7T+C3hx5Ff74htX/9DInXCp3F9pwJDInmCkEBeVlMqwrjMSSqFYICHM7/NI3l5WW8Xz2B/aMSRj7ah1z5OEbtM3jp6E68dHQn9o9K2D8qIfOzr8Oda0hUUxXGk5fFdOc6t+pEpPoYz2WJNoygAEMWk51rzQgKMIICxlN7sbCw0DWwf1TCU2cC3c9u0kd3wsm7LFFN9TEeQxaTneuG6lvnmG74mAEn0fC3TegSzRlBAdoeP0zT7DGw+3cbG9AlmlN9jMfwt8WNoADD94CTseEXk5clAZclAZOSmCaEkMKgV7nsZ62xsTFo1z/uCv1k7EU8rx7CG39/FcWTr+DUs4/hx28/gZMjj1qFQa9CCCGTkph26hl+MbmhuLsTeoA28pKASwGqaUPtDXPhwgWf3ZzKniy+ad1c6j2mrTQb1qTEZ5wVakOi71KAanlJgB6gjQeuvD9UH+PRB2kuHxCQDwjQhzhV94vJu7du5FqtltpqtdR79+5p3bdiq6U2tYtp3S8m9SFO7eYNtm/DZxJ3hzYk+vQhTtUDAhyqh/ahemgfZl99DvPHj2D++BHMvPhtGMpXu3P0IU51Ovd/iYuyyGh+IaUN0twlP2tNDglwc8nPWtogzWl+IXVRXv8PbH/8Fyy0G0H5udT6AAAAAElFTkSuQmCC">firefox export bookmarks at DuckDuckGo</A>
        <DT><A HREF="https://duckduckgo.com/?q=archive+firefox+bookmarks&t=ffab&ia=web" ADD_DATE="1497562974" LAST_MODIFIED="1497562974" ICON_URI="https://duckduckgo.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAG7UlEQVRYha2XbYwbVxWG798oorJQhYKqSJPxzJ35gWDVComiKrK86/H4Y+2J21/5kIyUgiKk1GqArVpEXQlKSlFNm35IFMVREgQNJE4KAsqCpiGtsrsIJjvO2N6xvbPj9SbZJBvjdsmGzcYvP+xxxt6PbBFHeiTP9T3nfe+ZO+NrQjYZmsJ4aoo3WU+w2fqTO7S5BA839Sd3aPUEm60p3qSmMJ7N1n1gWArnqyfYbD3Bw2F2F9eYVTi1h11cwz2nlvDmLIXz/c/CmsJ46gk2W9/Fo76LR11hrXqCTVmKyKxvVmTqCTZVV1irm5dgs5+5I5bCDNQV1prtCDsrsRRmwE7QdE3hVOd7Z05N4VQ7QdOWwgx0O9dToz2+KXE7zjVqCg87xmWcYrU4p9YUHpsizqmOaTvGZWoKDzvONR5ownSJO5vJjrPZ8ruv4E5+HHfy41j862ksvDGC+p7HHmjEjrfbX1O8SceEuZ4JTWE8M1GuYcfvi8/Edmh2nMeN359Ef9xbbOL6ywdgx/kNmYnt0BwTdpzHTJRrrLkn7DibteM87GEuoymMx+qI23EehQMhFA6EYP/mXTSbzR4jn7x/DFdSsQ1NWB0T9jCXaY+x2R7xsiL6ZmIUVozX2mb47EyMwuHKMzHcna+v6oI7br4+AndOP3aczxJCiBXjtZkYRVkRfV0Dlbg3Z3UGy4ros2IUDuaer2HxxrUNxZ24+vxeuHP7cdevxL05QgghRUVkrBhFtbP66SinupOMt38IAJiYvIpj7+n4p351lfDpP5p4Nv0XfPzLcxsamI5yKiGEVGO8ZsUoiorIkGpESE0PU1QjQsoMMAPTwxRu9DPtDfjzX+mb6kL9YBz9NdyYAWbArUkqUW+uOtx2Ux7mMtVhCjf66ftPwNx8E8Xp62sKH3z5HCbys/jXuWPor+GmPMxliorIVIcpKlFvjlTCrGWG+AYhhExHOLUapXDzj/eyXZHHd7+Fw79Q1zTxpfhreOH1P+G2Po7+Gm6mI+3bUAmzViXMWqQSpSh3BqdCfKMSpXBz/s3DXZFvvHAKj+9+C3PzzVUG5uabmJtv4rY+jkqUYvopDtd+8Ag++fXnsXjOgyvf245KlGKqs9hyhFMrUQpSiVCU5Y6rCEU/Y+mDq579/mgtTaK1+De0bp3AXfNbuP3BQ1jRt/RwM7OtW5MQQsoyp1YiFKQcoTA7BsoRin70b4YxOzu7tvJ/ZnCvGlwl5rA8sRW33vkCrL1sT01CCDFlTi1HKIgZvm+gJPENM0zhprTrK5icnFxTf6UkdsUWz3q6NLIPY+6729FfywxTlKT2LTBlTjXDFGQqxFqlEGu5B/u5+IdzGxpYnti6ppgZppjew/aOOYsNsdZUiLVIKeTNTYUpNFlkzCCXmQpT9PPhkcNrGmgtTWLF2IYVfQv+/cFDuPHati6fnvVgeWIrVvQtuPajL3ZrmUEuo8kiMxWmKIW8OVIICalSiMKQvUkzwAyUQhT9jD/3NBYWFtY0URvZi9nvbEcj+zDunP9czx749KwHV77/SE8tM8AMGLI3WQpRFEJCimiyyJRCFCWp/SouyqzVb0DbH4ZlWesaWMv0WhTl9q0uSbxWCrW7TgghpBDk1KJMkZdFX0HyKkWZop/1NuKNk0dWzV2PguRV8rLoK8oUhWB7LxBCCMnLoq8gUxidLhiSN1eQKdx8dOoEAODm0jWUbt03s1QtdOf8+ekv4/A7T+C3hx5Ff74htX/9DInXCp3F9pwJDInmCkEBeVlMqwrjMSSqFYICHM7/NI3l5WW8Xz2B/aMSRj7ah1z5OEbtM3jp6E68dHQn9o9K2D8qIfOzr8Oda0hUUxXGk5fFdOc6t+pEpPoYz2WJNoygAEMWk51rzQgKMIICxlN7sbCw0DWwf1TCU2cC3c9u0kd3wsm7LFFN9TEeQxaTneuG6lvnmG74mAEn0fC3TegSzRlBAdoeP0zT7DGw+3cbG9AlmlN9jMfwt8WNoADD94CTseEXk5clAZclAZOSmCaEkMKgV7nsZ62xsTFo1z/uCv1k7EU8rx7CG39/FcWTr+DUs4/hx28/gZMjj1qFQa9CCCGTkph26hl+MbmhuLsTeoA28pKASwGqaUPtDXPhwgWf3ZzKniy+ad1c6j2mrTQb1qTEZ5wVakOi71KAanlJgB6gjQeuvD9UH+PRB2kuHxCQDwjQhzhV94vJu7du5FqtltpqtdR79+5p3bdiq6U2tYtp3S8m9SFO7eYNtm/DZxJ3hzYk+vQhTtUDAhyqh/ahemgfZl99DvPHj2D++BHMvPhtGMpXu3P0IU51Ovd/iYuyyGh+IaUN0twlP2tNDglwc8nPWtogzWl+IXVRXv8PbH/8Fyy0G0H5udT6AAAAAElFTkSuQmCC">archive firefox bookmarks at DuckDuckGo</A>
        <DT><A HREF="https://github.com/nodiscc" ADD_DATE="1497562974" LAST_MODIFIED="1497562974" ICON_URI="https://assets-cdn.github.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADC0lEQVRYhb2Wv2tUQRDHv9Fwefdul535zp7GRrGz9RcoiEIK/wdBjBhFOwsVxMI/wMbSQrSyiWKwE0t/NHbRIoURJdqI2iRBBZVocxeez93L5XI68Jo3M/v5zuzOvgf0b9F7P2Uid8zsZTRb3tpu/9rabv+KZstm9tJE7njvpwDEdazb28qy3E1y2sgfXeBaj5E/SE6XZbl7YLCqBpI3jVzpF5wQskLypqqGdVcdyTeDgutPJN/03Q3n3ISRi8OCV7qx6Jyb6AkPIez9F/CqiBDC3hxcjXxbS5gz8v4gooxc7OTO1d6/DSHoXwKMvF1fRFWPd9xj3vupaPahOnYUeUyRx9WxjGYfOqM4BgCqejwh7nb90O1LVVGW5Z5qnKqGRqOxC8CmRBM3NRqNXfUTX5blnsza+6rVz2SC9vc8NH1YWZb7M1s0040ZN/JnKkhEJjcqQEQmMwJ+AhiHOnc2FdAm3wPgRgUAIMl3KYY6dxYkp1NO7/2pIcABAN77UykGyWnUx6TbnnVfnz1MVUNqm42cQzT7+pfDbGFY8K6Z2UKdE82+ItWaaDY/bAHR7HWKlRRg5NKwBRi5lBSQdQBbhsjfki00isxm7oATw6KLyInkVovMguStjPMFgNEh8EejyIvMGN6CtlrHUs5OwPWN0klez62vrdYxAPDR7EsuyMh7AMYHYG8z8l5u3Wj2BYAHAKjqjVVVqtecc0co8qQi4nubvMsQznS+YpsTwFHfbB5U584ZOWPk9xy8w7mxmqmq26PZtw5shSIXAIwZ+TzRkRkAIwkBIybyoBe0Uv03Vd3+R7aIXK0GNZvNA0VR7Kx/SFT1UK7nzrnD/QgQkaup/FGSzyqVPupq895PqeqVVqt1NNP+rrXXgpN8hh7TNV69MkXkfA9YymSN1r/GWge6KIod0Wx+NSnGpxrCJe/9SVW9DKAYREA0my+KYke/lZDkw8wVLesVQPIhBvi5GWEIp6PZx0EFRPITQziD9NT0bY4iF8zslZGf0fndztiYkZ/N7BVDuAjAbQT8X+w36KQvZccCoxkAAAAASUVORK5CYII=">nodiscc (nodiscc) · GitHub</A>
-        <DT><A HREF="https://github.com/pirate/bookmark-archiver#troubleshooting" ADD_DATE="1497562975" LAST_MODIFIED="1497562975" ICON_URI="https://assets-cdn.github.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADC0lEQVRYhb2Wv2tUQRDHv9Fwefdul535zp7GRrGz9RcoiEIK/wdBjBhFOwsVxMI/wMbSQrSyiWKwE0t/NHbRIoURJdqI2iRBBZVocxeez93L5XI68Jo3M/v5zuzOvgf0b9F7P2Uid8zsZTRb3tpu/9rabv+KZstm9tJE7njvpwDEdazb28qy3E1y2sgfXeBaj5E/SE6XZbl7YLCqBpI3jVzpF5wQskLypqqGdVcdyTeDgutPJN/03Q3n3ISRi8OCV7qx6Jyb6AkPIez9F/CqiBDC3hxcjXxbS5gz8v4gooxc7OTO1d6/DSHoXwKMvF1fRFWPd9xj3vupaPahOnYUeUyRx9WxjGYfOqM4BgCqejwh7nb90O1LVVGW5Z5qnKqGRqOxC8CmRBM3NRqNXfUTX5blnsza+6rVz2SC9vc8NH1YWZb7M1s0040ZN/JnKkhEJjcqQEQmMwJ+AhiHOnc2FdAm3wPgRgUAIMl3KYY6dxYkp1NO7/2pIcABAN77UykGyWnUx6TbnnVfnz1MVUNqm42cQzT7+pfDbGFY8K6Z2UKdE82+ItWaaDY/bAHR7HWKlRRg5NKwBRi5lBSQdQBbhsjfki00isxm7oATw6KLyInkVovMguStjPMFgNEh8EejyIvMGN6CtlrHUs5OwPWN0klez62vrdYxAPDR7EsuyMh7AMYHYG8z8l5u3Wj2BYAHAKjqjVVVqtecc0co8qQi4nubvMsQznS+YpsTwFHfbB5U584ZOWPk9xy8w7mxmqmq26PZtw5shSIXAIwZ+TzRkRkAIwkBIybyoBe0Uv03Vd3+R7aIXK0GNZvNA0VR7Kx/SFT1UK7nzrnD/QgQkaup/FGSzyqVPupq895PqeqVVqt1NNP+rrXXgpN8hh7TNV69MkXkfA9YymSN1r/GWge6KIod0Wx+NSnGpxrCJe/9SVW9DKAYREA0my+KYke/lZDkw8wVLesVQPIhBvi5GWEIp6PZx0EFRPITQziD9NT0bY4iF8zslZGf0fndztiYkZ/N7BVDuAjAbQT8X+w36KQvZccCoxkAAAAASUVORK5CYII=">pirate/bookmark-archiver · Github</A>
+        <DT><A HREF="https://github.com/pirate/ArchiveBox#troubleshooting" ADD_DATE="1497562975" LAST_MODIFIED="1497562975" ICON_URI="https://assets-cdn.github.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADC0lEQVRYhb2Wv2tUQRDHv9Fwefdul535zp7GRrGz9RcoiEIK/wdBjBhFOwsVxMI/wMbSQrSyiWKwE0t/NHbRIoURJdqI2iRBBZVocxeez93L5XI68Jo3M/v5zuzOvgf0b9F7P2Uid8zsZTRb3tpu/9rabv+KZstm9tJE7njvpwDEdazb28qy3E1y2sgfXeBaj5E/SE6XZbl7YLCqBpI3jVzpF5wQskLypqqGdVcdyTeDgutPJN/03Q3n3ISRi8OCV7qx6Jyb6AkPIez9F/CqiBDC3hxcjXxbS5gz8v4gooxc7OTO1d6/DSHoXwKMvF1fRFWPd9xj3vupaPahOnYUeUyRx9WxjGYfOqM4BgCqejwh7nb90O1LVVGW5Z5qnKqGRqOxC8CmRBM3NRqNXfUTX5blnsza+6rVz2SC9vc8NH1YWZb7M1s0040ZN/JnKkhEJjcqQEQmMwJ+AhiHOnc2FdAm3wPgRgUAIMl3KYY6dxYkp1NO7/2pIcABAN77UykGyWnUx6TbnnVfnz1MVUNqm42cQzT7+pfDbGFY8K6Z2UKdE82+ItWaaDY/bAHR7HWKlRRg5NKwBRi5lBSQdQBbhsjfki00isxm7oATw6KLyInkVovMguStjPMFgNEh8EejyIvMGN6CtlrHUs5OwPWN0klez62vrdYxAPDR7EsuyMh7AMYHYG8z8l5u3Wj2BYAHAKjqjVVVqtecc0co8qQi4nubvMsQznS+YpsTwFHfbB5U584ZOWPk9xy8w7mxmqmq26PZtw5shSIXAIwZ+TzRkRkAIwkBIybyoBe0Uv03Vd3+R7aIXK0GNZvNA0VR7Kx/SFT1UK7nzrnD/QgQkaup/FGSzyqVPupq895PqeqVVqt1NNP+rrXXgpN8hh7TNV69MkXkfA9YymSN1r/GWge6KIod0Wx+NSnGpxrCJe/9SVW9DKAYREA0my+KYke/lZDkw8wVLesVQPIhBvi5GWEIp6PZx0EFRPITQziD9NT0bY4iF8zslZGf0fndztiYkZ/N7BVDuAjAbQT8X+w36KQvZccCoxkAAAAASUVORK5CYII=">pirate/ArchiveBox · Github</A>
        <DT><A HREF="http://www.cs.unc.edu/~fabian/papers/foniks-oak11.pdf" ADD_DATE="1497562976" LAST_MODIFIED="1497562976" ICON_URI="https://assets-cdn.github.com/favicon.ico" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADC0lEQVRYhb2Wv2tUQRDHv9Fwefdul535zp7GRrGz9RcoiEIK/wdBjBhFOwsVxMI/wMbSQrSyiWKwE0t/NHbRIoURJdqI2iRBBZVocxeez93L5XI68Jo3M/v5zuzOvgf0b9F7P2Uid8zsZTRb3tpu/9rabv+KZstm9tJE7njvpwDEdazb28qy3E1y2sgfXeBaj5E/SE6XZbl7YLCqBpI3jVzpF5wQskLypqqGdVcdyTeDgutPJN/03Q3n3ISRi8OCV7qx6Jyb6AkPIez9F/CqiBDC3hxcjXxbS5gz8v4gooxc7OTO1d6/DSHoXwKMvF1fRFWPd9xj3vupaPahOnYUeUyRx9WxjGYfOqM4BgCqejwh7nb90O1LVVGW5Z5qnKqGRqOxC8CmRBM3NRqNXfUTX5blnsza+6rVz2SC9vc8NH1YWZb7M1s0040ZN/JnKkhEJjcqQEQmMwJ+AhiHOnc2FdAm3wPgRgUAIMl3KYY6dxYkp1NO7/2pIcABAN77UykGyWnUx6TbnnVfnz1MVUNqm42cQzT7+pfDbGFY8K6Z2UKdE82+ItWaaDY/bAHR7HWKlRRg5NKwBRi5lBSQdQBbhsjfki00isxm7oATw6KLyInkVovMguStjPMFgNEh8EejyIvMGN6CtlrHUs5OwPWN0klez62vrdYxAPDR7EsuyMh7AMYHYG8z8l5u3Wj2BYAHAKjqjVVVqtecc0co8qQi4nubvMsQznS+YpsTwFHfbB5U584ZOWPk9xy8w7mxmqmq26PZtw5shSIXAIwZ+TzRkRkAIwkBIybyoBe0Uv03Vd3+R7aIXK0GNZvNA0VR7Kx/SFT1UK7nzrnD/QgQkaup/FGSzyqVPupq895PqeqVVqt1NNP+rrXXgpN8hh7TNV69MkXkfA9YymSN1r/GWge6KIod0Wx+NSnGpxrCJe/9SVW9DKAYREA0my+KYke/lZDkw8wVLesVQPIhBvi5GWEIp6PZx0EFRPITQziD9NT0bY4iF8zslZGf0fndztiYkZ/N7BVDuAjAbQT8X+w36KQvZccCoxkAAAAASUVORK5CYII=">Phonotactic Reconstruction of Encrypted VoIP Conversations</A>
        <DT><A HREF="https://www.ghacks.net/2009/07/23/firefox-bookmarks-archiver/" ADD_DATE="1497562974" LAST_MODIFIED="1497562974" ICON_URI="https://www.ghacks.net/wp-content/uploads/2005/10/favicon.ico" ICON="data:image/png;base64,AAABAAEAEBAAAAEAGABoAwAAFgAAACgAAAAQAAAAIAAAAAEAGAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAD+/P76/Pzs7Ozg2uXNyeW/ueOXmuKJleOpsOrAvubNzOff3ebv7+77/f39/f7+/v79/vv+/f77+v3s7vjIy++5vO+truh9keNggNuRpemrtua/0Orz+f79/P38/v3+/v78/f78/vna7PpyltlTfNxsjOSdruq4u+6grPFlidxxl+SdtummvN7w+Pz9/v3+//79/fv49/2cqttZgNtFeOJGeeNbhdaqtubGxeuyw/VuleFvm9ybuui2yOb5/fv+/v79/Pzj4ve8vumzuvCZqO1ojeCUsvSJq+pyneGAqOqYuupsm+F7ouCkveXk6/b8/f74+fqhsuSRouios/G5v+vDx+mjtu9Jht43h+k+iuJ5qeG61vdsnNeNsevU3vD7/v7h7fxUgNRDe+JSg96JpObByu+dt+s6ieM3jOkzj+dBkt281PK31vFmm9XC1/T7/f7d6fV8ntxfkuRFhN1Bg+NtnOalyvRElts4kuc4k+c9leWn0ezm7/ebweaNuuD3/P7z9fnY1+vV3PGsyu1bld9Cjd9nouCFw+1QnN48mONWoNnC5/fY5/XP6vqHstr1+v7x+P3AyeTAz+7f6fTS5vdhpeA5kOlssOG54/iv2e/C5/iZ0ux2s9ve8/u91un3/P72+/yCpNpIh96JtOXg8vvZ7/xjp90+meGQxezy+/ze8vpcpto/ouG+4/Tw9vb8/v78/fu91O9eldxMjdp+seLp+P7I6vdNoeJIouDC5vbw/f6Jxek9oOWi0Or4/P3+/v73/v3v+f3E1+qex/FOktubyur2+/yez+05pN93ueXy/fy94/dQo9e+3PH7/f7//v78/v38/f32+PfA2exanuBVmuHg8vza8vxVqeBPpN3a8vva6/Wm0+P0+v38/v7+/v79/f77/P77/v3t+fyrz+lwp9XT7PXl+fxost1VptbQ6fjz+Pz0/vv+/vz9/v7+/v/8/vz8/v38/vz8/fz5/P3x+v7y/Pv5+vnJ5PPQ8Pn2/Pz8/fz+/v38/f7+/v3///4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA">Firefox Bookmarks Archiver - gHacks Tech News</A>
    </DL><p>
@@ -0,0 +1,12 @@
+<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Pinboard Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL>
+<p>
+
+<DT><A HREF="https://github.com/trailofbits/algo" ADD_DATE="1542616733" PRIVATE="1" TOREAD="1" TAGS="vpn,scripts,toread">Algo VPN scripts</A>
+<DT><A HREF="http://www.ulisp.com/" ADD_DATE="1542374412" PRIVATE="1" TOREAD="1" TAGS="arduino,avr,embedded,lisp,toread">uLisp</A>
+
+</DL>
+</p>
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" xmlns:syn="http://purl.org/rss/1.0/modules/syndication/" xmlns:admin="http://webns.net/mvcb/">
+  <channel rdf:about="http://pinboard.in">
+    <title>Pinboard (private aaronmueller)</title>
+    <link>https://pinboard.in/u:aaronmueller/private/</link>
+    <description></description>
+    <items>
+      <rdf:Seq>
+        <rdf:li rdf:resource="https://mehkee.com/"/>
+        <rdf:li rdf:resource="https://qmk.fm/"/>
+      </rdf:Seq>
+    </items>
+  </channel>
+
+  <item rdf:about="https://mehkee.com/">
+    <title>Mehkee - Mechanical Keyboard Parts &amp; Accessories</title>
+    <dc:date>2018-11-08T21:29:32+00:00</dc:date>
+    <link>https://mehkee.com/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>keyboard gadget diy</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:gadget"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:diy"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+  <item rdf:about="https://qmk.fm/">
+    <title>QMK Firmware - An open source firmware for AVR and ARM based keyboards</title>
+    <dc:date>2018-11-06T22:36:21+00:00</dc:date>
+    <link>https://qmk.fm/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>firmware keyboard</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:firmware"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+</rdf:RDF>
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+    <posts user="aaronmueller">
+<post href="https://github.com/trailofbits/algo" time="2018-11-19T08:38:53Z" description="Algo VPN scripts" extended="" tag="vpn scripts" hash="18d708f67bb26d843b1cac4530bb52aa"  shared="no" toread="yes" />
+<post href="http://www.ulisp.com/" time="2018-11-16T13:20:12Z" description="uLisp" extended="" tag="arduino avr embedded lisp" hash="2a17ae95925a03a5b9bb38cf7f6c6f9b"  shared="no" toread="yes" />
+</posts>
@@ -0,0 +1,2 @@
+[{"href":"https:\/\/github.com\/trailofbits\/algo","description":"Algo VPN scripts","extended":"","meta":"62325ba3b577683aee854d7f191034dc","hash":"18d708f67bb26d843b1cac4530bb52aa","time":"2018-11-19T08:38:53Z","shared":"no","toread":"yes","tags":"vpn scripts"},
+{"href":"http:\/\/www.ulisp.com\/","description":"uLisp","extended":"","meta":"7bd0c0ef31f69d1459e3d37366e742b3","hash":"2a17ae95925a03a5b9bb38cf7f6c6f9b","time":"2018-11-16T13:20:12Z","shared":"no","toread":"yes","tags":"arduino avr embedded lisp"}]
@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+import json
+import os
+from os.path import dirname, pardir, join
+from subprocess import check_output, check_call
+from tempfile import TemporaryDirectory
+from typing import List
+
+import pytest
+
+
+ARCHIVER_BIN = join(dirname(__file__), pardir, 'archive.py')
+
+
+class Helper:
+    def __init__(self, output_dir: str):
+        self.output_dir = output_dir
+
+    def run(self, links, env=None, env_defaults=None):
+        if env_defaults is None:
+            env_defaults = {
+                # we don't wanna spam archive.org witin our tests..
+                'SUBMIT_ARCHIVE_DOT_ORG': 'False',
+            }
+        if env is None:
+            env = {}
+
+        env = dict(**env_defaults, **env)
+
+        jj = []
+        for url in links:
+            jj.append({
+                'href': url,
+                'description': url,
+            })
+        input_json = join(self.output_dir, 'input.json')
+        with open(input_json, 'w') as fo:
+            json.dump(jj, fo)
+
+        if env is None:
+            env = {}
+        env['OUTPUT_DIR'] = self.output_dir
+        check_call(
+            [ARCHIVER_BIN, input_json],
+            env={**os.environ.copy(), **env},
+        )
+
+
+class TestArchiver:
+    def setup(self):
+        # self.tdir = TemporaryDirectory(dir='hello')
+        class AAA:
+            name = 'hello'
+        self.tdir = AAA()
+
+    def teardown(self):
+        pass
+        # self.tdir.cleanup()
+
+    @property
+    def output_dir(self):
+        return self.tdir.name
+
+    def test_fetch_favicon_false(self):
+        h = Helper(self.output_dir)
+
+        h.run(links=[
+            'https://google.com',
+        ], env={
+            'FETCH_FAVICON': 'False',
+        })
+        # for now no asserts, good enough if it isn't failing
+
+    def test_3000_links(self):
+        """
+        The pages are deliberatly unreachable. The tool should gracefully process all of them even though individual links are failing.
+        """
+        h = Helper(self.output_dir)
+
+        h.run(links=[
+            f'https://localhost:123/whatever_{i}.html' for i in range(3000)
+        ], env={
+            'FETCH_FAVICON': 'False',
+            'FETCH_SCREENSHOT': 'False',
+            'FETCH_PDF': 'False',
+            'FETCH_DOM': 'False',
+            'CHECK_SSL_VALIDITY': 'False',
+        })
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])
@@ -3,45 +3,75 @@ import re
 import sys
 import time
 import json
-import requests
+from urllib.request import Request, urlopen
+from urllib.parse import urlparse

-from datetime import datetime
-from subprocess import run, PIPE, DEVNULL
-from multiprocessing import Process
+from decimal import Decimal
 from urllib.parse import quote
+from datetime import datetime
+from subprocess import TimeoutExpired, Popen, PIPE, DEVNULL, CompletedProcess, CalledProcessError
+from multiprocessing import Process

 from config import (
-    IS_TTY,
-    OUTPUT_PERMISSIONS,
-    REPO_DIR,
-    SOURCES_DIR,
-    OUTPUT_DIR,
-    ARCHIVE_DIR,
-    TIMEOUT,
-    TERM_WIDTH,
-    SHOW_PROGRESS,
    ANSI,
+    IS_TTY,
+    TERM_WIDTH,
+    REPO_DIR,
+    OUTPUT_DIR,
+    SOURCES_DIR,
+    ARCHIVE_DIR,
+    OUTPUT_PERMISSIONS,
+    TIMEOUT,
+    SHOW_PROGRESS,
+    CHECK_SSL_VALIDITY,
+    WGET_USER_AGENT,
+    CURL_BINARY,
+    WGET_BINARY,
    CHROME_BINARY,
+    GIT_BINARY,
+    YOUTUBEDL_BINARY,
+    FETCH_TITLE,
+    FETCH_FAVICON,
    FETCH_WGET,
+    FETCH_WARC,
    FETCH_PDF,
    FETCH_SCREENSHOT,
    FETCH_DOM,
-    FETCH_FAVICON,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
+    FETCH_GIT,
+    FETCH_MEDIA,
    SUBMIT_ARCHIVE_DOT_ORG,
 )

-# URL helpers
-without_scheme = lambda url: url.replace('http://', '').replace('https://', '').replace('ftp://', '')
-without_query = lambda url: url.split('?', 1)[0]
-without_hash = lambda url: url.split('#', 1)[0]
-without_path = lambda url: url.split('/', 1)[0]
-domain = lambda url: without_hash(without_query(without_path(without_scheme(url))))
+# URL helpers: https://docs.python.org/3/library/urllib.parse.html#url-parsing
+scheme = lambda url: urlparse(url).scheme
+without_scheme = lambda url: urlparse(url)._replace(scheme='').geturl().strip('//')
+without_query = lambda url: urlparse(url)._replace(query='').geturl().strip('//')
+without_fragment = lambda url: urlparse(url)._replace(fragment='').geturl().strip('//')
+without_path = lambda url: urlparse(url)._replace(path='', fragment='', query='').geturl().strip('//')
+path = lambda url: urlparse(url).path
+basename = lambda url: urlparse(url).path.rsplit('/', 1)[-1]
+domain = lambda url: urlparse(url).netloc
+query = lambda url: urlparse(url).query
+fragment = lambda url: urlparse(url).fragment
+extension = lambda url: basename(url).rsplit('.', 1)[-1].lower() if '.' in basename(url) else ''
 base_url = lambda url: without_scheme(url)  # uniq base url used to dedupe links

 short_ts = lambda ts: ts.split('.')[0]

+URL_REGEX = re.compile(
+    r'http[s]?://'                    # start matching from allowed schemes
+    r'(?:[a-zA-Z]|[0-9]'              # followed by allowed alphanum characters
+    r'|[$-_@.&+]|[!*\(\),]'           #    or allowed symbols
+    r'|(?:%[0-9a-fA-F][0-9a-fA-F]))'  #    or allowed unicode bytes
+    r'[^\]\[\(\)<>\""\'\s]+',         # stop parsing at these symbols
+    re.IGNORECASE,
+)
+HTML_TITLE_REGEX = re.compile(
+    r'<title>'                         # start matching text after <title> tag
+    r'(.[^<>]+)',                      # get everything up to these symbols
+    re.IGNORECASE,
+)
+

 def check_dependencies():
    """Check that all necessary dependencies are installed, and have valid versions"""
@@ -49,14 +79,28 @@ def check_dependencies():
    python_vers = float('{}.{}'.format(sys.version_info.major, sys.version_info.minor))
    if python_vers < 3.5:
        print('{}[X] Python version is not new enough: {} (>3.5 is required){}'.format(ANSI['red'], python_vers, ANSI['reset']))
-        print('    See https://github.com/pirate/bookmark-archiver#troubleshooting for help upgrading your Python installation.')
+        print('    See https://github.com/pirate/ArchiveBox#troubleshooting for help upgrading your Python installation.')
        raise SystemExit(1)

+    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
+        if run(['which', CURL_BINARY], stdout=DEVNULL).returncode or run([CURL_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: curl{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CURL_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
+            raise SystemExit(1)
+
+    if FETCH_WGET or FETCH_WARC:
+        if run(['which', WGET_BINARY], stdout=DEVNULL).returncode or run([WGET_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: wget{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(WGET_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
+            raise SystemExit(1)
+
    if FETCH_PDF or FETCH_SCREENSHOT or FETCH_DOM:
        if run(['which', CHROME_BINARY], stdout=DEVNULL).returncode:
            print('{}[X] Missing dependency: {}{}'.format(ANSI['red'], CHROME_BINARY, ANSI['reset']))
            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CHROME_BINARY))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

        # parse chrome --version e.g. Google Chrome 61.0.3114.0 canary / Chromium 59.0.3029.110 built on Ubuntu, running on Ubuntu 16.04
@@ -68,36 +112,53 @@ def check_dependencies():
            if int(version) < 59:
                print(version_lines)
                print('{red}[X] Chrome version must be 59 or greater for headless PDF, screenshot, and DOM saving{reset}'.format(**ANSI))
-                print('    See https://github.com/pirate/bookmark-archiver for help.')
+                print('    See https://github.com/pirate/ArchiveBox for help.')
                raise SystemExit(1)
        except (IndexError, TypeError, OSError):
            print('{red}[X] Failed to parse Chrome version, is it installed properly?{reset}'.format(**ANSI))
            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CHROME_BINARY))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

-    if FETCH_WGET:
-        if run(['which', 'wget'], stdout=DEVNULL).returncode or run(['wget', '--version'], stdout=DEVNULL).returncode:
-            print('{red}[X] Missing dependency: wget{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('wget'))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
+    if FETCH_GIT:
+        if run(['which', GIT_BINARY], stdout=DEVNULL).returncode or run([GIT_BINARY, '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: git{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(GIT_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

-    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
-        if run(['which', 'curl'], stdout=DEVNULL).returncode or run(['curl', '--version'], stdout=DEVNULL).returncode:
-            print('{red}[X] Missing dependency: curl{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('curl'))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
-
-    if FETCH_AUDIO or FETCH_VIDEO:
-        if run(['which', 'youtube-dl'], stdout=DEVNULL).returncode or run(['youtube-dl', '--version'], stdout=DEVNULL).returncode:
+    if FETCH_MEDIA:
+        if run(['which', YOUTUBEDL_BINARY], stdout=DEVNULL).returncode or run([YOUTUBEDL_BINARY, '--version'], stdout=DEVNULL).returncode:
            print('{red}[X] Missing dependency: youtube-dl{reset}'.format(**ANSI))
-            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('youtube-dl'))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(YOUTUBEDL_BINARY))
+            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)


+def check_url_parsing():
+    """Check that plain text regex URL parsing works as expected"""
+    test_urls = '''
+    https://example1.com/what/is/happening.html?what=1#how-about-this=1
+    https://example2.com/what/is/happening/?what=1#how-about-this=1
+    HTtpS://example3.com/what/is/happening/?what=1#how-about-this=1f
+    https://example4.com/what/is/happening.html
+    https://example5.com/
+    https://example6.com
+
+    <test>http://example7.com</test>
+    [https://example8.com/what/is/this.php?what=1]
+    [and http://example9.com?what=1&other=3#and-thing=2]
+    <what>https://example10.com#and-thing=2 "</about>
+    abc<this["https://example11.com/what/is#and-thing=2?whoami=23&where=1"]that>def
+    sdflkf[what](https://example12.com/who/what.php?whoami=1#whatami=2)?am=hi
+    example13.bada
+    and example14.badb
+    <or>htt://example15.badc</that>
+    '''
+    # print('\n'.join(re.findall(URL_REGEX, test_urls)))
+    assert len(re.findall(URL_REGEX, test_urls)) == 12
+
+
 def chmod_file(path, cwd='.', permissions=OUTPUT_PERMISSIONS, timeout=30):
    """chmod -R <permissions> <cwd>/<path>"""

@@ -118,11 +179,10 @@ def progress(seconds=TIMEOUT, prefix=''):
    if not SHOW_PROGRESS:
        return lambda: None

-    chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
-    chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
-
-    def progress_bar(seconds=seconds, prefix=prefix):
+    def progress_bar(seconds, prefix):
        """show timer in the form of progress bar, with percentage and seconds remaining"""
+        chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
+        chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
        try:
            for s in range(seconds * chunks):
                progress = s / chunks / seconds * 100
@@ -156,23 +216,59 @@ def progress(seconds=TIMEOUT, prefix=''):
            print()
            pass

-    p = Process(target=progress_bar)
+    p = Process(target=progress_bar, args=(seconds, prefix))
    p.start()

    def end():
        """immediately finish progress and clear the progressbar line"""
-        p.terminate()
+
+        # protect from double termination
+        #if p is None or not hasattr(p, 'kill'):
+        #    return
+        nonlocal p
+        if p is not None:
+            p.terminate()
+        p = None
+
        sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH), ANSI['reset']))  # clear whole terminal line
        sys.stdout.flush()

    return end

 def pretty_path(path):
-    """convert paths like .../bookmark-archiver/archiver/../output/abc into output/abc"""
+    """convert paths like .../ArchiveBox/archivebox/../output/abc into output/abc"""
    return path.replace(REPO_DIR + '/', '')


-def download_url(url):
+def save_stdin_source(raw_text):
+    if not os.path.exists(SOURCES_DIR):
+        os.makedirs(SOURCES_DIR)
+
+    ts = str(datetime.now().timestamp()).split('.', 1)[0]
+
+    source_path = os.path.join(SOURCES_DIR, '{}-{}.txt'.format('stdin', ts))
+
+    with open(source_path, 'w', encoding='utf-8') as f:
+        f.write(raw_text)
+
+    return source_path
+
+
+def fetch_page_content(url, timeout=TIMEOUT):
+    req = Request(url, headers={'User-Agent': WGET_USER_AGENT})
+
+    if CHECK_SSL_VALIDITY:
+        resp = urlopen(req, timeout=timeout)
+    else:
+        import ssl
+        insecure = ssl._create_unverified_context()
+        resp = urlopen(req, timeout=timeout, context=insecure)
+
+    encoding = resp.headers.get_content_charset() or 'utf-8'
+    return resp.read().decode(encoding)
+
+
+def save_remote_source(url, timeout=TIMEOUT):
    """download a given url's content into downloads/domain.txt"""

    if not os.path.exists(SOURCES_DIR):
@@ -182,26 +278,56 @@ def download_url(url):

    source_path = os.path.join(SOURCES_DIR, '{}-{}.txt'.format(domain(url), ts))

-    print('[*] [{}] Downloading {} > {}'.format(
+    print('{}[*] [{}] Downloading {}{}'.format(
+        ANSI['green'],
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
        url,
-        pretty_path(source_path),
+        ANSI['reset'],
    ))
    end = progress(TIMEOUT, prefix='      ')
    try:
-        downloaded_xml = requests.get(url).content.decode()
+        downloaded_xml = fetch_page_content(url, timeout=timeout)
        end()
    except Exception as e:
        end()
-        print('[!] Failed to download {}\n'.format(url))
+        print('{}[!] Failed to download {}{}\n'.format(
+            ANSI['red'],
+            url,
+            ANSI['reset'],
+        ))
        print('    ', e)
        raise SystemExit(1)

    with open(source_path, 'w', encoding='utf-8') as f:
        f.write(downloaded_xml)

+    print('    > {}'.format(pretty_path(source_path)))
+
    return source_path

+
+def fetch_page_title(url, timeout=10, progress=SHOW_PROGRESS):
+    """Attempt to guess a page's title by downloading the html"""
+    if not FETCH_TITLE:
+        return None
+
+    try:
+        if progress:
+            sys.stdout.write('.')
+            sys.stdout.flush()
+
+        html = fetch_page_content(url, timeout=timeout)
+
+        match = re.search(HTML_TITLE_REGEX, html)
+        return match.group(1).strip() if match else None
+    except Exception as err:
+        # print('[!] Failed to fetch title because of {}: {}'.format(
+        #     err.__class__.__name__,
+        #     err,
+        # ))
+        return None
+
+
 def str_between(string, start, end=None):
    """(<abc>12345</def>, <abc>, </def>)  ->  12345"""

@@ -214,19 +340,19 @@ def str_between(string, start, end=None):
 def get_link_type(link):
    """Certain types of links need to be handled specially, this figures out when that's the case"""

-    if link['base_url'].endswith('.pdf'):
+    if extension(link['url']) == 'pdf':
        return 'PDF'
-    elif link['base_url'].rsplit('.', 1) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
+    elif extension(link['url']) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
        return 'image'
-    elif 'wikipedia.org' in link['domain']:
+    elif 'wikipedia.org' in domain(link['url']).lower():
        return 'wiki'
-    elif 'youtube.com' in link['domain']:
+    elif 'youtube.com' in domain(link['url']).lower():
        return 'youtube'
-    elif 'soundcloud.com' in link['domain']:
+    elif 'soundcloud.com' in domain(link['url']).lower():
        return 'soundcloud'
-    elif 'youku.com' in link['domain']:
+    elif 'youku.com' in domain(link['url']).lower():
        return 'youku'
-    elif 'vimeo.com' in link['domain']:
+    elif 'vimeo.com' in domain(link['url']).lower():
        return 'vimeo'
    return None

@@ -234,19 +360,19 @@ def merge_links(a, b):
    """deterministially merge two links, favoring longer field values over shorter,
    and "cleaner" values over worse ones.
    """
-    longer = lambda key: a[key] if len(a[key]) > len(b[key]) else b[key]
+    longer = lambda key: (a[key] if len(a[key]) > len(b[key]) else b[key]) if (a[key] and b[key]) else (a[key] or b[key])
    earlier = lambda key: a[key] if a[key] < b[key] else b[key]
    
    url = longer('url')
    longest_title = longer('title')
-    cleanest_title = a['title'] if '://' not in a['title'] else b['title']
+    cleanest_title = a['title'] if '://' not in (a['title'] or '') else b['title']
    link = {
        'timestamp': earlier('timestamp'),
        'url': url,
        'domain': domain(url),
        'base_url': base_url(url),
        'tags': longer('tags'),
-        'title': longest_title if '://' not in longest_title else cleanest_title,
+        'title': longest_title if '://' not in (longest_title or '') else cleanest_title,
        'sources': list(set(a.get('sources', []) + b.get('sources', []))),
    }
    link['type'] = get_link_type(link)
@@ -257,15 +383,15 @@ def find_link(folder, links):
    url = parse_url(folder)
    if url:
        for link in links:
-            if (link['base_url'] in url) or (url in link['url']):
+            if (base_url(link['url']) in url) or (url in link['url']):
                return link

    timestamp = folder.split('.')[0]
    for link in links:
        if link['timestamp'].startswith(timestamp):
-            if link['domain'] in os.listdir(os.path.join(ARCHIVE_DIR, folder)):
+            if domain(link['url']) in os.listdir(os.path.join(ARCHIVE_DIR, folder)):
                return link      # careful now, this isn't safe for most ppl
-            if link['domain'] in parse_url(folder):
+            if domain(link['url']) in parse_url(folder):
                return link
    return None

@@ -279,7 +405,7 @@ def parse_url(folder):
                link_json = f.read().strip()
                if link_json:
                    link = json.loads(link_json)
-                    return link['base_url']
+                    return base_url(link['url'])
            except ValueError:
                print('File contains invalid JSON: {}!'.format(link_json))

@@ -313,7 +439,7 @@ def manually_merge_folders(source, target):
    assert answer in ('', 'a', 'b', 'q'), 'Invalid choice.'

    if answer == 'q':
-        print('\nJust run Bookmark Archiver again to pick up where you left off.')
+        print('\nJust run ArchiveBox again to pick up where you left off.')
        raise SystemExit(0)
    elif answer == '':
        return
@@ -335,8 +461,8 @@ def fix_folder_path(archive_path, link_folder, link):
    target = os.path.join(archive_path, link['timestamp'])

    url_in_folder = parse_url(source)
-    if not (url_in_folder in link['base_url']
-            or link['base_url'] in url_in_folder):
+    if not (url_in_folder in base_url(link['url'])
+            or base_url(link['url']) in url_in_folder):
        raise ValueError('The link does not match the url for this folder.')

    if not os.path.exists(target):
@@ -403,7 +529,7 @@ def cleanup_archive(archive_path, links):
        for folder, link in bad_folders:
            fix_folder_path(archive_path, folder, link)
    elif bad_folders:
-        print('[!] Warning! {} folders need to be merged, fix by running bookmark archiver.'.format(len(bad_folders)))
+        print('[!] Warning! {} folders need to be merged, fix by running ArchiveBox.'.format(len(bad_folders)))

    if unmatched:
        print('[!] Warning! {} unrecognized folders in html/archive/'.format(len(unmatched)))
@@ -424,12 +550,12 @@ def wget_output_path(link, look_in=None):
    urlencode = lambda s: quote(s, encoding='utf-8', errors='replace')

    if link['type'] in ('PDF', 'image'):
-        return urlencode(link['base_url'])
+        return urlencode(base_url(link['url']))

    # Since the wget algorithm to for -E (appending .html) is incredibly complex
    # instead of trying to emulate it here, we just look in the output folder
    # to see what html file wget actually created as the output
-    wget_folder = link['base_url'].rsplit('/', 1)[0].split('/')
+    wget_folder = base_url(link['url']).rsplit('/', 1)[0].split('/')
    look_in = os.path.join(ARCHIVE_DIR, link['timestamp'], *wget_folder)

    if look_in and os.path.exists(look_in):
@@ -449,7 +575,7 @@ def wget_output_path(link, look_in=None):

    # if re.search(".+\\.[Hh][Tt][Mm][Ll]?$", split_url[0], re.I | re.M):
    #     # already ends in .html
-    #     return urlencode(link['base_url'])
+    #     return urlencode(base_url(link['url']))
    # else:
    #     # .html needs to be appended
    #     without_scheme = split_url[0].split('://', 1)[-1].split('?', 1)[0]
@@ -462,33 +588,99 @@ def wget_output_path(link, look_in=None):
    #             return urlencode('#'.join([without_scheme + '/index.html' + query + '.html', *split_url[1:]]))
    #         elif '/' in without_scheme:
    #             return urlencode('#'.join([without_scheme + '.html', *split_url[1:]]))
-    #         return urlencode(link['base_url'] + '/index.html')
+    #         return urlencode(base_url(link['url']) + '/index.html')


 def derived_link_info(link):
    """extend link info with the archive urls and other derived data"""

-    link_info = {
+    url = link['url']
+
+    extended_info = {
        **link,
-        'date': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
-        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
-        'favicon_url': 'archive/{timestamp}/favicon.ico'.format(**link),
-        'files_url': 'archive/{timestamp}/index.html'.format(**link),
-        'archive_url': 'archive/{}/{}'.format(link['timestamp'], wget_output_path(link)),
-        'pdf_link': 'archive/{timestamp}/output.pdf'.format(**link),
-        'screenshot_link': 'archive/{timestamp}/screenshot.png'.format(**link),
-        'dom_link': 'archive/{timestamp}/output.html'.format(**link),
-        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**link),
+        'title': link['title'] or base_url(url),
+        'date': datetime.fromtimestamp(Decimal(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
+        'base_url': base_url(url),
+        'domain': domain(url),
+        'basename': basename(url),
+        'path': path(url),
+    }
+
+    # Archive Method Output URLs
+    extended_info = {
+        **extended_info,
+        'favicon_url': 'archive/{timestamp}/favicon.ico'.format(**extended_info),
+        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**extended_info),
+        'files_url': 'archive/{timestamp}/index.html'.format(**extended_info),
+        'archive_url': 'archive/{}/{}'.format(link['timestamp'], wget_output_path(link) or 'index.html'),
+        'warc_url': 'archive/{timestamp}/warc'.format(**extended_info),
+        'pdf_link': 'archive/{timestamp}/output.pdf'.format(**extended_info),
+        'screenshot_link': 'archive/{timestamp}/screenshot.png'.format(**extended_info),
+        'dom_link': 'archive/{timestamp}/output.html'.format(**extended_info),
+        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**extended_info),
+        'git_url': 'archive/{timestamp}/git'.format(**extended_info),
+        'media_url': 'archive/{timestamp}/media'.format(**extended_info),
+        
    }

    # PDF and images are handled slightly differently
    # wget, screenshot, & pdf urls all point to the same file
    if link['type'] in ('PDF', 'image'):
-        link_info.update({
-            'archive_url': 'archive/{timestamp}/{base_url}'.format(**link),
-            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'dom_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'title': '{title} ({type})'.format(**link),
+        extended_info.update({
+            'archive_url': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'dom_link': 'archive/{timestamp}/{base_url}'.format(**extended_info),
+            'title': link['title'] or basename(link['url']),
        })
-    return link_info
+
+    return extended_info
+
+
+def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs):
+    """Patched of subprocess.run to fix blocking io making timeout=innefective"""
+
+    if input is not None:
+        if 'stdin' in kwargs:
+            raise ValueError('stdin and input arguments may not both be used.')
+        kwargs['stdin'] = PIPE
+
+    if capture_output:
+        if ('stdout' in kwargs) or ('stderr' in kwargs):
+            raise ValueError('stdout and stderr arguments may not be used '
+                             'with capture_output.')
+        kwargs['stdout'] = PIPE
+        kwargs['stderr'] = PIPE
+
+    with Popen(*popenargs, **kwargs) as process:
+        try:
+            stdout, stderr = process.communicate(input, timeout=timeout)
+        except TimeoutExpired:
+            process.kill()
+            try:
+                stdout, stderr = process.communicate(input, timeout=2)
+            except:
+                pass
+            raise TimeoutExpired(popenargs[0][0], timeout)
+        except BaseException as err:
+            process.kill()
+            # We don't call process.wait() as .__exit__ does that for us.
+            raise 
+        retcode = process.poll()
+        if check and retcode:
+            raise CalledProcessError(retcode, process.args,
+                                     output=stdout, stderr=stderr)
+    return CompletedProcess(process.args, retcode, stdout, stderr)
+
+
+def check_link_structure(link):
+    assert isinstance(link, dict)
+    assert isinstance(link.get('url'), str)
+    assert len(link['url']) > 2
+    assert len(re.findall(URL_REGEX, link['url'])) == 1
+
+
+def check_links_structure(links):
+    assert isinstance(links, list)
+    if links:
+        check_link_structure(links[0])
@@ -1,170 +0,0 @@
-#!/usr/bin/env python3
-# Bookmark Archiver
-# Nick Sweeting 2017 | MIT License
-# https://github.com/pirate/bookmark-archiver
-
-import os
-import sys
-
-from datetime import datetime
-from subprocess import run
-
-from parse import parse_links
-from links import validate_links
-from archive_methods import archive_links, _RESULTS_TOTALS
-from index import (
-    write_links_index,
-    write_link_index,
-    parse_json_links_index,
-    parse_json_link_index,
-)
-from config import (
-    OUTPUT_PERMISSIONS,
-    OUTPUT_DIR,
-    ANSI,
-    TIMEOUT,
-    GIT_SHA,
-)
-from util import (
-    download_url,
-    progress,
-    cleanup_archive,
-    pretty_path,
-    migrate_data,
-)
-
-__AUTHOR__ = 'Nick Sweeting <git@nicksweeting.com>'
-__VERSION__ = GIT_SHA
-__DESCRIPTION__ = 'Bookmark Archiver: Create a browsable html archive of a list of links.'
-__DOCUMENTATION__ = 'https://github.com/pirate/bookmark-archiver'
-
-def print_help():
-    print(__DESCRIPTION__)
-    print("Documentation:     {}\n".format(__DOCUMENTATION__))
-    print("Usage:")
-    print("    ./bin/bookmark-archiver ~/Downloads/bookmarks_export.html\n")
-
-
-def merge_links(archive_path=OUTPUT_DIR, import_path=None):
-    """get new links from file and optionally append them to links in existing archive"""
-    all_links = []
-    if import_path:
-        # parse and validate the import file
-        raw_links = parse_links(import_path)
-        all_links = validate_links(raw_links)
-
-    # merge existing links in archive_path and new links
-    existing_links = []
-    if archive_path:
-        existing_links = parse_json_links_index(archive_path)
-        all_links = validate_links(existing_links + all_links)
-    
-    num_new_links = len(all_links) - len(existing_links)
-    if num_new_links:
-        print('[{green}+{reset}] [{}] Adding {} new links from {} to {}/index.json'.format(
-            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            num_new_links,
-            pretty_path(import_path),
-            pretty_path(archive_path),
-            **ANSI,
-        ))
-    # else:
-    #     print('[*] [{}] No new links added to {}/index.json{}'.format(
-    #         datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-    #         archive_path,
-    #         ' from {}'.format(import_path) if import_path else '',
-    #         **ANSI,
-    #     ))
-
-    return all_links
-
-def update_archive(archive_path, links, source=None, resume=None, append=True):
-    """update or create index.html+json given a path to an export file containing new links"""
-
-    start_ts = datetime.now().timestamp()
-
-    if resume:
-        print('{green}[▶] [{}] Resuming archive downloading from {}...{reset}'.format(
-             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-             resume,
-             **ANSI,
-        ))
-    else:
-        print('{green}[▶] [{}] Updating files for {} links in archive...{reset}'.format(
-             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-             len(links),
-             **ANSI,
-        ))
-
-    # loop over links and archive them
-    archive_links(archive_path, links, source=source, resume=resume)
-
-    # print timing information & summary
-    end_ts = datetime.now().timestamp()
-    seconds = end_ts - start_ts
-    if seconds > 60:
-        duration = '{0:.2f} min'.format(seconds / 60, 2)
-    else:
-        duration = '{0:.2f} sec'.format(seconds, 2)
-
-    print('{}[√] [{}] Update of {} links complete ({}){}'.format(
-        ANSI['green'],
-        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        len(links),
-        duration,
-        ANSI['reset'],
-    ))
-    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
-    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
-    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
-
-
-if __name__ == '__main__':
-    argc = len(sys.argv)
-
-    if set(sys.argv).intersection(('-h', '--help', 'help')):
-        print_help()
-        raise SystemExit(0)
-
-    migrate_data()
-
-    source = sys.argv[1] if argc > 1 else None  # path of links file to import
-    resume = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
-   
-    if argc == 1:
-        source, resume = None, None
-    elif argc == 2:
-        if all(d.isdigit() for d in sys.argv[1].split('.')):
-            # argv[1] is a resume timestamp
-            source, resume = None, sys.argv[1]
-        else:
-            # argv[1] is a path to a file to import
-            source, resume = sys.argv[1].strip(), None
-    elif argc == 3:
-        source, resume = sys.argv[1].strip(), sys.argv[2]
-    else:
-        print_help()
-        raise SystemExit(1)
-
-    # See if archive folder already exists
-    for out_dir in (OUTPUT_DIR, 'bookmarks', 'pocket', 'pinboard', 'html'):
-        if os.path.exists(out_dir):
-            break
-    else:
-        out_dir = OUTPUT_DIR
-
-    # Step 0: Download url to local file (only happens if a URL is specified instead of local path) 
-    if source and any(source.startswith(s) for s in ('http://', 'https://', 'ftp://')):
-        source = download_url(source)
-
-    # Step 1: Parse the links and dedupe them with existing archive
-    links = merge_links(archive_path=out_dir, import_path=source)
-    
-    # Step 2: Write new index
-    write_links_index(out_dir=out_dir, links=links)
-
-    # Step 3: Verify folder structure is 1:1 with index
-    # cleanup_archive(out_dir, links)
-
-    # Step 4: Run the archive methods for each link
-    update_archive(out_dir, links, source=source, resume=resume, append=True)
@@ -1,503 +0,0 @@
-import os
-import sys
-
-from functools import wraps
-from collections import defaultdict
-from datetime import datetime
-from subprocess import run, PIPE, DEVNULL
-
-from peekable import Peekable
-
-from index import wget_output_path, parse_json_link_index, write_link_index
-from links import links_after_timestamp
-from config import (
-    CHROME_BINARY,
-    FETCH_WGET,
-    FETCH_WGET_REQUISITES,
-    FETCH_PDF,
-    FETCH_SCREENSHOT,
-    FETCH_DOM,
-    RESOLUTION,
-    CHECK_SSL_VALIDITY,
-    SUBMIT_ARCHIVE_DOT_ORG,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
-    FETCH_FAVICON,
-    WGET_USER_AGENT,
-    CHROME_USER_DATA_DIR,
-    TIMEOUT,
-    ANSI,
-    ARCHIVE_DIR,
-)
-from util import (
-    check_dependencies,
-    progress,
-    chmod_file,
-    pretty_path,
-)
-
-
-_RESULTS_TOTALS = {   # globals are bad, mmkay
-    'skipped': 0,
-    'succeded': 0,
-    'failed': 0,
-}
-
-def archive_links(archive_path, links, source=None, resume=None):
-    check_dependencies()
-
-    to_archive = Peekable(links_after_timestamp(links, resume))
-    idx, link = 0, to_archive.peek(0)
-
-    try:
-        for idx, link in enumerate(to_archive):
-            link_dir = os.path.join(ARCHIVE_DIR, link['timestamp'])
-            archive_link(link_dir, link)
-    
-    except (KeyboardInterrupt, SystemExit, Exception) as e:
-        print('{lightyellow}[X] [{now}] Downloading paused on link {timestamp} ({idx}/{total}){reset}'.format(
-            **ANSI,
-            now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            idx=idx+1,
-            timestamp=link['timestamp'],
-            total=len(links),
-        ))
-        print('    Continue where you left off by running:')
-        print('        {} {}'.format(
-            pretty_path(sys.argv[0]),
-            link['timestamp'],
-        ))
-        if not isinstance(e, KeyboardInterrupt):
-            raise e
-        raise SystemExit(1)
-
-
-def archive_link(link_dir, link, overwrite=True):
-    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
-
-    update_existing = os.path.exists(link_dir)
-    if update_existing:
-        link = {
-            **parse_json_link_index(link_dir),
-            **link,
-        }
-    else:
-        os.makedirs(link_dir)
-    
-    log_link_archive(link_dir, link, update_existing)
-
-    if FETCH_WGET:
-        link = fetch_wget(link_dir, link, overwrite=overwrite)
-
-    if FETCH_PDF:
-        link = fetch_pdf(link_dir, link, overwrite=overwrite)
-
-    if FETCH_SCREENSHOT:
-        link = fetch_screenshot(link_dir, link, overwrite=overwrite)
-
-    if FETCH_DOM:
-        link = fetch_dom(link_dir, link, overwrite=overwrite)
-
-    if SUBMIT_ARCHIVE_DOT_ORG:
-        link = archive_dot_org(link_dir, link, overwrite=overwrite)
-
-    # if FETCH_AUDIO:
-    #     link = fetch_audio(link_dir, link, overwrite=overwrite)
-
-    # if FETCH_VIDEO:
-    #     link = fetch_video(link_dir, link, overwrite=overwrite)
-
-    if FETCH_FAVICON:
-        link = fetch_favicon(link_dir, link, overwrite=overwrite)
-
-    write_link_index(link_dir, link)
-    # print()
-    
-    return link
-
-def log_link_archive(link_dir, link, update_existing):
-    print('[{symbol_color}{symbol}{reset}] [{now}] "{title}"\n    {blue}{url}{reset}'.format(
-        symbol='*' if update_existing else '+',
-        symbol_color=ANSI['black' if update_existing else 'green'],
-        now=datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        **link,
-        **ANSI,
-    ))
-
-    print('    > {}{}'.format(pretty_path(link_dir), '' if update_existing else ' (new)'))
-    if link['type']:
-        print('      i {}'.format(link['type']))
-
-
-
-def attach_result_to_link(method):
-    """
-    Instead of returning a result={output:'...', status:'success'} object,
-    attach that result to the links's history & latest fields, then return
-    the updated link object.
-    """
-    def decorator(fetch_func):
-        @wraps(fetch_func)
-        def timed_fetch_func(link_dir, link, overwrite=False, **kwargs):
-            # initialize methods and history json field on link
-            link['latest'] = link.get('latest') or {}
-            link['latest'][method] = link['latest'].get(method) or None
-            link['history'] = link.get('history') or {}
-            link['history'][method] = link['history'].get(method) or []
-
-            start_ts = datetime.now().timestamp()
-
-            # if a valid method output is already present, dont run the fetch function
-            if link['latest'][method] and not overwrite:
-                print('      √ {}'.format(method))
-                result = None
-            else:
-                print('      > {}'.format(method))
-                result = fetch_func(link_dir, link, **kwargs)
-
-            end_ts = datetime.now().timestamp()
-            duration = str(end_ts * 1000 - start_ts * 1000).split('.')[0]
-
-            # append a history item recording fail/success
-            history_entry = {
-                'timestamp': str(start_ts).split('.')[0],
-            }
-            if result is None:
-                history_entry['status'] = 'skipped'
-            elif isinstance(result.get('output'), Exception):
-                history_entry['status'] = 'failed'
-                history_entry['duration'] = duration
-                history_entry.update(result or {})
-                link['history'][method].append(history_entry)
-            else:
-                history_entry['status'] = 'succeded'
-                history_entry['duration'] = duration
-                history_entry.update(result or {})
-                link['history'][method].append(history_entry)
-                link['latest'][method] = result['output']
-
-            _RESULTS_TOTALS[history_entry['status']] += 1
-            
-            return link
-        return timed_fetch_func
-    return decorator
-
-
-@attach_result_to_link('wget')
-def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
-    """download full site using wget"""
-
-    domain_dir = os.path.join(link_dir, link['domain'])
-    existing_file = wget_output_path(link)
-    if os.path.exists(domain_dir) and existing_file:
-        return {'output': existing_file, 'status': 'skipped'}
-
-    CMD = [
-        # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
-        *'wget -N -E -np -x -H -k -K -S --restrict-file-names=unix'.split(' '),
-        *(('-p',) if FETCH_WGET_REQUISITES else ()),
-        *(('--user-agent="{}"'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
-        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate',))),
-        link['url'],
-    ]
-    end = progress(timeout, prefix='      ')
-    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # index.html
-        end()
-        output = wget_output_path(link, look_in=domain_dir)
-
-        # Check for common failure cases
-        if result.returncode > 0 and result.returncode != 8:
-            print('        got wget response code {}:'.format(result.returncode))
-            print('\n'.join('          ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
-            if result.returncode == 4:
-                raise Exception('Failed wget download')
-        if result.returncode > 0 and b'403: Forbidden' in result.stderr:
-            raise Exception('403 Forbidden (try changing WGET_USER_AGENT)')
-        if result.returncode > 0 and b'404: Not Found' in result.stderr:
-            raise Exception('404 Not Found')
-        if result.returncode > 0 and b'ERROR 500: Internal Server Error' in result.stderr:
-            raise Exception('500 Internal Server Error')
-    except Exception as e:
-        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-
-@attach_result_to_link('pdf')
-def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
-    """print PDF of site to file using chrome --headless"""
-
-    if link['type'] in ('PDF', 'image'):
-        return {'output': wget_output_path(link)}
-    
-    if os.path.exists(os.path.join(link_dir, 'output.pdf')):
-        return {'output': 'output.pdf', 'status': 'skipped'}
-
-    CMD = [
-        *chrome_headless(user_data_dir=user_data_dir),
-        '--print-to-pdf',
-        link['url']
-    ]
-    end = progress(timeout, prefix='      ')
-    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.pdf
-        end()
-        if result.returncode:
-            print('     ', (result.stderr or result.stdout).decode())
-            raise Exception('Failed to print PDF')
-        chmod_file('output.pdf', cwd=link_dir)
-        output = 'output.pdf'
-    except Exception as e:
-        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-@attach_result_to_link('screenshot')
-def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR, resolution=RESOLUTION):
-    """take screenshot of site using chrome --headless"""
-
-    if link['type'] in ('PDF', 'image'):
-        return {'output': wget_output_path(link)}
-    
-    if os.path.exists(os.path.join(link_dir, 'screenshot.png')):
-        return {'output': 'screenshot.png', 'status': 'skipped'}
-
-    CMD = [
-        *chrome_headless(user_data_dir=user_data_dir),
-        '--screenshot',
-        '--window-size={}'.format(resolution),
-        link['url']
-    ]
-    end = progress(timeout, prefix='      ')
-    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # sreenshot.png
-        end()
-        if result.returncode:
-            print('     ', (result.stderr or result.stdout).decode())
-            raise Exception('Failed to take screenshot')
-        chmod_file('screenshot.png', cwd=link_dir)
-        output = 'screenshot.png'
-    except Exception as e:
-        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-    
-@attach_result_to_link('dom')
-def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
-    """print HTML of site to file using chrome --dump-html"""
-
-    if link['type'] in ('PDF', 'image'):
-        return {'output': wget_output_path(link)}
-    
-    output_path = os.path.join(link_dir, 'output.html')
-
-    if os.path.exists(output_path):
-        return {'output': 'output.html', 'status': 'skipped'}
-
-    CMD = [
-        *chrome_headless(user_data_dir=user_data_dir),
-        '--dump-dom',
-        link['url']
-    ]
-    end = progress(timeout, prefix='      ')
-    try:
-        with open(output_path, 'w+') as f:
-            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.html
-        end()
-        if result.returncode:
-            print('     ', (result.stderr).decode())
-            raise Exception('Failed to fetch DOM')
-        chmod_file('output.html', cwd=link_dir)
-        output = 'output.html'
-    except Exception as e:
-        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-@attach_result_to_link('archive_org')
-def archive_dot_org(link_dir, link, timeout=TIMEOUT):
-    """submit site to archive.org for archiving via their service, save returned archive url"""
-
-    path = os.path.join(link_dir, 'archive.org.txt')
-    if os.path.exists(path):
-        archive_org_url = open(path, 'r').read().strip()
-        return {'output': archive_org_url, 'status': 'skipped'}
-
-    submit_url = 'https://web.archive.org/save/{}'.format(link['url'].split('?', 1)[0])
-
-    success = False
-    CMD = ['curl', '-I', submit_url]
-    end = progress(timeout, prefix='      ')
-    try:
-        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # archive.org.txt
-        end()
-
-        # Parse archive.org response headers
-        headers = defaultdict(list)
-
-        # lowercase all the header names and store in dict
-        for header in result.stdout.splitlines():
-            if b':' not in header or not header.strip():
-                continue
-            name, val = header.decode().split(':', 1)
-            headers[name.lower().strip()].append(val.strip())
-
-        # Get successful archive url in "content-location" header or any errors
-        content_location = headers['content-location']
-        errors = headers['x-archive-wayback-runtime-error']
-
-        if content_location:
-            saved_url = 'https://web.archive.org{}'.format(content_location[0])
-            success = True
-        elif len(errors) == 1 and 'RobotAccessControlException' in errors[0]:
-            output = submit_url
-            # raise Exception('Archive.org denied by {}/robots.txt'.format(link['domain']))
-        elif errors:
-            raise Exception(', '.join(errors))
-        else:
-            raise Exception('Failed to find "content-location" URL header in Archive.org response.')
-    except Exception as e:
-        end()
-        print('        Visit url to see output:', ' '.join(CMD))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    if success:
-        with open(os.path.join(link_dir, 'archive.org.txt'), 'w', encoding='utf-8') as f:
-            f.write(saved_url)
-        chmod_file('archive.org.txt', cwd=link_dir)
-        output = saved_url
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-@attach_result_to_link('favicon')
-def fetch_favicon(link_dir, link, timeout=TIMEOUT):
-    """download site favicon from google's favicon api"""
-
-    if os.path.exists(os.path.join(link_dir, 'favicon.ico')):
-        return {'output': 'favicon.ico', 'status': 'skipped'}
-
-    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
-    fout = open('{}/favicon.ico'.format(link_dir), 'w')
-    end = progress(timeout, prefix='      ')
-    try:
-        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # favicon.ico
-        fout.close()
-        end()
-        chmod_file('favicon.ico', cwd=link_dir)
-        output = 'favicon.ico'
-    except Exception as e:
-        fout.close()
-        end()
-        print('        Run to see full output:', ' '.join(CMD))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-# @attach_result_to_link('audio')
-# def fetch_audio(link_dir, link, timeout=TIMEOUT):
-#     """Download audio rip using youtube-dl"""
-
-#     if link['type'] not in ('soundcloud',)\
-#        and 'audio' not in link['tags']:
-#         return
-
-#     path = os.path.join(link_dir, 'audio')
-
-#     if not os.path.exists(path) or overwrite:
-#         print('    - Downloading audio')
-#         CMD = [
-#             "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
-#             link['url'],
-#         ]
-#         end = progress(timeout, prefix='      ')
-#         try:
-#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # audio/audio.mp3
-#             end()
-#             if result.returncode:
-#                 print('     ', result.stderr.decode())
-#                 raise Exception('Failed to download audio')
-#             chmod_file('audio.mp3', cwd=link_dir)
-#             return 'audio.mp3'
-#         except Exception as e:
-#             end()
-#             print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-#             print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-#             raise
-#     else:
-#         print('    √ Skipping audio download')
-
-# @attach_result_to_link('video')
-# def fetch_video(link_dir, link, timeout=TIMEOUT):
-#     """Download video rip using youtube-dl"""
-
-#     if link['type'] not in ('youtube', 'youku', 'vimeo')\
-#        and 'video' not in link['tags']:
-#         return
-
-#     path = os.path.join(link_dir, 'video')
-
-#     if not os.path.exists(path) or overwrite:
-#         print('    - Downloading video')
-#         CMD = [
-#             "youtube-dl -x --video-format mp4 --audio-quality 0 -o '%(title)s.%(ext)s'",
-#             link['url'],
-#         ]
-#         end = progress(timeout, prefix='      ')
-#         try:
-#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # video/movie.mp4
-#             end()
-#             if result.returncode:
-#                 print('     ', result.stderr.decode())
-#                 raise Exception('Failed to download video')
-#             chmod_file('video.mp4', cwd=link_dir)
-#             return 'video.mp4'
-#         except Exception as e:
-#             end()
-#             print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-#             print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-#             raise
-#     else:
-#         print('    √ Skipping video download')
-
-
-def chrome_headless(binary=CHROME_BINARY, user_data_dir=CHROME_USER_DATA_DIR):
-    args = [binary, '--headless']  # '--disable-gpu'
-    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
-    if user_data_dir:
-        args.append('--user-data-dir={}'.format(user_data_dir))
-    elif os.path.exists(default_profile):
-        args.append('--user-data-dir={}'.format(default_profile))
-    return args
@@ -1,81 +0,0 @@
-import os
-import sys
-import shutil
-
-from subprocess import run, PIPE
-
-# ******************************************************************************
-# * TO SET YOUR CONFIGURATION, EDIT THE VALUES BELOW, or use the 'env' command *
-# * e.g.                                                                       *
-# * env USE_COLOR=True CHROME_BINARY=google-chrome ./archive.py export.html    *
-# ******************************************************************************
-
-IS_TTY = sys.stdout.isatty()
-USE_COLOR =              os.getenv('USE_COLOR',              str(IS_TTY)        ).lower() == 'true'
-SHOW_PROGRESS =          os.getenv('SHOW_PROGRESS',          str(IS_TTY)        ).lower() == 'true'
-FETCH_WGET =             os.getenv('FETCH_WGET',             'True'             ).lower() == 'true'
-FETCH_WGET_REQUISITES =  os.getenv('FETCH_WGET_REQUISITES',  'True'             ).lower() == 'true'
-FETCH_AUDIO =            os.getenv('FETCH_AUDIO',            'False'            ).lower() == 'true'
-FETCH_VIDEO =            os.getenv('FETCH_VIDEO',            'False'            ).lower() == 'true'
-FETCH_PDF =              os.getenv('FETCH_PDF',              'True'             ).lower() == 'true'
-FETCH_SCREENSHOT =       os.getenv('FETCH_SCREENSHOT',       'True'             ).lower() == 'true'
-FETCH_DOM =              os.getenv('FETCH_DOM',              'True'             ).lower() == 'true'
-FETCH_FAVICON =          os.getenv('FETCH_FAVICON',          'True'             ).lower() == 'true'
-SUBMIT_ARCHIVE_DOT_ORG = os.getenv('SUBMIT_ARCHIVE_DOT_ORG', 'True'             ).lower() == 'true'
-RESOLUTION =             os.getenv('RESOLUTION',             '1440,1200'        )
-CHECK_SSL_VALIDITY =     os.getenv('CHECK_SSL_VALIDITY',     'True'             ).lower() == 'true'
-OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
-CHROME_BINARY =          os.getenv('CHROME_BINARY',          'chromium-browser' )  # change to google-chrome browser if using google-chrome
-WGET_BINARY =            os.getenv('WGET_BINARY',            'wget'             )
-WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',         None)
-CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',    None)
-TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
-FOOTER_INFO =            os.getenv('FOOTER_INFO',            'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.',)
-
-### Paths
-REPO_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
-
-OUTPUT_DIR = os.path.join(REPO_DIR, 'output')
-ARCHIVE_DIR = os.path.join(OUTPUT_DIR, 'archive')
-SOURCES_DIR = os.path.join(OUTPUT_DIR, 'sources')
-
-PYTHON_PATH = os.path.join(REPO_DIR, 'archiver')
-TEMPLATES_DIR = os.path.join(PYTHON_PATH, 'templates')
-
-# ******************************************************************************
-# ********************** Do not edit below this point **************************
-# ******************************************************************************
-
-### Terminal Configuration
-TERM_WIDTH = shutil.get_terminal_size((100, 10)).columns
-ANSI = {
-    'reset': '\033[00;00m',
-    'lightblue': '\033[01;30m',
-    'lightyellow': '\033[01;33m',
-    'lightred': '\033[01;35m',
-    'red': '\033[01;31m',
-    'green': '\033[01;32m',
-    'blue': '\033[01;34m',
-    'white': '\033[01;37m',
-    'black': '\033[01;30m',
-}
-if not USE_COLOR:
-    # dont show colors if USE_COLOR is False
-    ANSI = {k: '' for k in ANSI.keys()}
-
-### Confirm Environment Setup
-try:
-    GIT_SHA = run(["git", "rev-list", "-1", "HEAD", "./"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
-except Exception:
-    GIT_SHA = 'unknown'
-    print('[!] Warning, you need git installed for some archiving features to save correct version numbers!')
-
-if sys.stdout.encoding.upper() != 'UTF-8':
-    print('[X] Your system is running python3 scripts with a bad locale setting: {} (it should be UTF-8).'.format(sys.stdout.encoding))
-    print('    To fix it, add the line "export PYTHONIOENCODING=utf8" to your ~/.bashrc file (without quotes)')
-    print('')
-    print('    Confirm that it\'s fixed by opening a new shell and running:')
-    print('        python3 -c "import sys; print(sys.stdout.encoding)"   # should output UTF-8')
-    print('')
-    print('    Alternatively, run this script with:')
-    print('        env PYTHONIOENCODING=utf8 ./archive.py export.html')
@@ -1,146 +0,0 @@
-<html lang="en">
-    <head>
-        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
-        <title>Archived Sites</title>
-        <style>
-            html, body {
-                width: 100%;
-                height: 100%;
-                font-size: 18px;
-                font-weight: 200;
-                text-align: center;
-                margin: 0px;
-                padding: 0px;
-                font-family: "Gill Sans", Helvetica, sans-serif;
-            }
-            header {
-                background-color: #aa1e55;
-                color: white;
-                padding: 10px;
-                padding-top: 0px;
-                padding-bottom: 15px;
-                height: 100px;
-            }
-            header h1 {
-                font-size: 38px;
-                font-weight: 300;
-                color: black;
-                padding-top: 14px;
-                line-height: 1.4;
-                width: 100%;
-            }
-            header h1 small {
-                color: white;
-                font-size:0.45em;
-                margin-left: 10px;
-                display: block;
-            }
-            header h1 small a {
-                text-decoration: none;
-                color: orange;
-                opacity: 0.6
-                font-weight: 300;
-            }
-            header h1 small a:hover {
-                opacity: 1;
-            }
-            .header-center {
-                width: 100%;
-                text-align: center;
-            }
-            .header-right {
-                float: right;
-                width: 50px;
-                height: 60px;
-                text-align: center;
-                padding: 20px;
-                margin-left: -100px;
-            }
-            table {
-                padding: 6px;
-                width: 100%;
-            }
-            table thead th {
-                font-weight: 400;
-            }
-            table tr {
-                height: 35px;
-            }
-            tbody tr:nth-child(odd) {
-               background-color: #ffebeb;
-            }
-            table tr td {
-                white-space: nowrap;
-                overflow: hidden;
-                /*padding-bottom: 0.4em;*/
-                /*padding-top: 0.4em;*/
-                padding-left: 2px;
-                text-align: center;
-            }
-            table tr td a {
-                text-decoration: none;
-            }
-            table tr td img, table tr td object {
-                display: inline-block;
-                margin: auto;
-                height: 24px;
-                width: 24px;
-                padding: 0px;
-                padding-right: 5px;
-                vertical-align: middle;
-                margin-left: 4px;
-            }
-        </style>
-    </head>
-    <body>
-        <header>
-            <div class="header-right">
-                <a href="?" title="Reload...">
-                    <img src="static/archive.png" style="height: 100%;"/>
-                </a>
-                <br/>
-                <a href="https://pirate.github.io/bookmark-archiver">
-                    Docs
-                </a>
-            </div>
-            <div class="header-center">
-                <h1>
-                    Archived Sites
-                    <br/>
-                    <small>
-                        <a href="?"> $num_links links</a> &nbsp; &nbsp; Last updated $time_updated<br/>
-                    </small>
-                </h1>
-            </div>
-        </header>
-        <table style="width:100%;height: 90%; overflow-y: scroll;table-layout: fixed">
-            <thead>
-                <tr>
-                    <th style="width: 120px;">Bookmarked On</th>
-                    <th style="width: 2vw;">Status</th>
-                    <th style="width: 43vw;">Saved Articles ($num_links)</th>
-                    <th style="width: 50px">Index</th>
-                    <th style="width: 50px">HTML</th>
-                    <th style="width: 50px">PDF</th>
-                    <th style="width: 60px;font-size:0.8em;">Screenshot</th>
-                    <th style="width: 50px">A.org</th>
-                    <th style="width: 100px;whitespace:nowrap;overflow-x:scroll;display:block">Original URL</th>
-                </tr>
-            </thead>
-            <tbody>$rows</tbody>
-        </table>
-        <footer>
-            <br/>
-            <center>
-                <small>
-                    Created using <a href="https://github.com/pirate/bookmark-archiver">Bookmark Archiver</a>
-                    version <a href="https://github.com/pirate/bookmark-archiver/commit/$git_sha">$short_git_sha</a> &nbsp; | &nbsp; 
-                    Download index as <a href="index.json">JSON</a>
-                    <br/><br/>
-                    $footer_info
-                </small>
-            </center>
-            <br/>
-        </footer>
-    </body>
-</html>
@@ -0,0 +1,18 @@
+# Binaries for running ArchiveBox
+
+This folder contains all the executables that ArchiveBox provides.
+
+
+# Adding it to your `$PATH`
+To be able to run ArchiveBox from anywhere on your system, you can add this entire folder to your path, like so:
+
+**Edit `~/.bash_profile`:**
+```bash
+export PATH=/opt/ArchiveBox/bin:$PATH
+```
+
+# Running executables directly
+
+If you don't want to add ArchiveBox to your `$PATH` you can also call these executables directly with their full path, like so:
+
+`/opt/ArchiveBox/bin/ArchiveBox https://example.com/some/feed.rss`
@@ -0,0 +1 @@
+../archivebox/archive.py
@@ -2,12 +2,14 @@

 REPO_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )"; cd .. && pwd )"

+mkdir -p "$REPO_DIR/output/sources"
+
 if [[ "$1" == "--chrome" ]]; then
    # Google Chrome / Chromium
-    default=$(ls ~/Library/Application\ Support/Google/Chrome/Default/History)
    if [[ -e "$2" ]]; then
        cp "$2" "$REPO_DIR/output/sources/chrome_history.db.tmp"
    else
+        default=$(ls ~/Library/Application\ Support/Google/Chrome/Default/History)
        echo "Defaulting to history db: $default"
        echo "Optionally specify the path to a different sqlite history database as the 2nd argument."
        cp "$default" "$REPO_DIR/output/sources/chrome_history.db.tmp"
@@ -20,10 +22,10 @@ fi

 if [[ "$1" == "--firefox" ]]; then
    # Firefox
-    default=$(ls ~/Library/Application\ Support/Firefox/Profiles/*.default/places.sqlite)
    if [[ -e "$2" ]]; then
        cp "$2" "$REPO_DIR/output/sources/firefox_history.db.tmp"
    else
+        default=$(ls ~/Library/Application\ Support/Firefox/Profiles/*.default/places.sqlite)
        echo "Defaulting to history db: $default"
        echo "Optionally specify the path to a different sqlite history database as the 2nd argument."
        cp "$default" "$REPO_DIR/output/sources/firefox_history.db.tmp"
@@ -0,0 +1 @@
+../archivebox/purge.py
@@ -1,15 +1,16 @@
 #!/bin/bash
-# Bookmark Archiver Setup Script
+# ArchiveBox Setup Script
 # Nick Sweeting 2017 | MIT License
-# https://github.com/pirate/bookmark-archiver
+# https://github.com/pirate/ArchiveBox

-echo "[i] Installing bookmark-archiver dependencies. 📦"
+echo "[i] Installing ArchiveBox dependencies. 📦"
 echo ""
 echo "    You may be prompted for a password in order to install the following dependencies:"
 echo "        - Chromium Browser   (see README for Google-Chrome instructions instead)"
 echo "        - python3"
 echo "        - wget"
 echo "        - curl"
+echo "        - youtube-dl"
 echo ""
 echo "    You may follow Manual Setup instructions in README.md instead if you prefer not to run an unknown script."
 echo "    Press enter to continue, or Ctrl+C to cancel..."
@@ -25,7 +26,6 @@ if which apt-get > /dev/null; then
        echo "[i] You already have google-chrome installed, if you would like to download chromium-browser instead (they work pretty much the same), follow the Manual Setup instructions"
        echo "[+] Linking $(which google-chrome) -> /usr/bin/chromium-browser (press enter to continue, or Ctrl+C to cancel...)"
        read
-        sudo ln -s "$(which google-chrome)" /usr/bin/chromium-browser
    elif which chromium-browser; then
        echo "[i] chromium-browser already installed, using existing installation."
        chromium-browser --version
@@ -34,7 +34,7 @@ if which apt-get > /dev/null; then
        apt install chromium-browser -y
    fi
    echo "[+] Installing python3, wget, curl..."
-    apt install python3 wget curl
+    apt install -y python3 python3-distutils wget curl youtube-dl

 # On Mac:
 elif which brew > /dev/null; then   # 🐍 eye of newt
@@ -57,7 +57,7 @@ elif which brew > /dev/null; then   # 🐍 eye of newt
        chmod +x /usr/local/bin/chromium-browser
    fi
    echo "[+] Installing python3, wget, curl  (ignore 'already installed' warnings)..."
-    brew install python3 wget curl
+    brew install python3 wget curl youtube-dl
 else
    echo "[X] Could not find aptitude or homebrew! ‼️"
    echo ""
@@ -84,5 +84,5 @@ echo ""
 echo "[X] Failed to install some dependencies! ‼️"
 echo "    - Try the Manual Setup instructions in the README.md"
 echo "    - Try the Troubleshooting: Dependencies instructions in the README.md"
-echo "    - Open an issue on github to get help: https://github.com/pirate/bookmark-archiver/issues"
+echo "    - Open an issue on github to get help: https://github.com/pirate/ArchiveBox/issues"
 exit 1
@@ -1 +1 @@
-../archiver/archive.py
+archivebox
@@ -0,0 +1,21 @@
+version: '3'
+
+services:
+    archivebox:
+        build: .
+        stdin_open: true
+        tty: true
+        environment:
+            - USE_COLOR=False
+            - SHOW_PROGRESS=False
+        volumes:
+            - ./data:/data
+        command: bash -c 'echo "https://github.com/pirate/ArchiveBox" | /bin/archive; tail -f /dev/null'
+
+    nginx:
+        image: 'nginx'
+        ports:
+            - '8098:80'
+        volumes:
+            - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf
+            - ./data:/var/www
@@ -1,15 +0,0 @@
-# Donate
-
-Right now I'm working on this project in my spare time and accepting the occational PR,   
-if you want me to dedicate more time to it, donate to support development!
-
- - Ethereum: 0x5B0F85FFc44fD759C2d97f0BE4681279966f3832
- - Bitcoin: https://shapeshift.io/ BTC -> to address above (ETH)
- - Paypal: https://www.paypal.me/NicholasSweeting/25
-
-The eventual goal is to support one or two developers full-time on this project via donations.   
-With more engineering power this can become a distributed archive service with a nice UI,   
-like the Way-Back machine but hosted by everyone!
-
-If have any questions or want to sponsor this project long-term, contact me at  
-bookmark-archiver@sweeting.me
@@ -0,0 +1,65 @@
+# Example config file for ArchiveBox
+# Copy this file to ~/.ArchiveBox.conf and then edit it
+# Config file uses python syntax but can also be sourced as a .env
+# For documentation, see:
+#    https://github.com/pirate/ArchiveBox/wiki/Configuration
+
+################################################################################
+## General Settings
+################################################################################
+
+#OUTPUT_DIR=output
+#OUTPUT_PERMISSIONS=755
+#ONLY_NEW=False
+#TIMEOUT=60
+#MEDIA_TIMEOUT=3600
+#TEMPLATES_DIR=archivebox/templates
+#FOOTER_INFO="Run by ACME co."
+
+
+################################################################################
+## Archive Method Toggles
+################################################################################
+
+#FETCH_TITLE=True
+#FETCH_FAVICON=True
+#FETCH_WGET=True
+#FETCH_WARC=True
+#FETCH_PDF=True
+#FETCH_SCREENSHOT=True
+#FETCH_DOM=True
+#FETCH_GIT=True
+#FETCH_MEDIA=False
+#SUBMIT_ARCHIVE_DOT_ORG=True
+
+
+################################################################################
+## Archive Method Options
+################################################################################
+
+#CHECK_SSL_VALIDITY=True
+#FETCH_WGET_REQUISITES=True
+#RESOLUTION="1440,900"
+#WGET_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
+#GIT_DOMAINS="github.com,bitbucket.org,gitlab.com"
+#COOKIES_FILE="path/to/cookies.txt"
+#CHROME_USER_DATA_DIR="~/.config/google-chrome/Default"
+
+
+################################################################################
+## Shell Options
+################################################################################
+
+#USE_COLOR=True
+#SHOW_PROGRESS=True
+
+
+################################################################################
+## Dependency Options
+################################################################################
+
+#CURL_BINARY="curl"
+#GIT_BINARY="git"
+#WGET_BINARY="wget"
+#YOUTUBEDL_BINARY="youtube-dl"
+#CHROME_BINARY="chromium-browser"
@@ -0,0 +1,12 @@
+# Example etc files for deploying ArchiveBox
+
+In this folder are some example config files you can use for setting up ArchiveBox on your machine.
+
+E.g. see `etc/nginx` for an example nginx config to serve your archive, or `etc/cron` for an example cron job that crawls a feed every 24 hours.
+
+Please contribute your etc files here! Example contributions
+
+- supervisord config
+- systemd config
+- apache webserver config
+- other init system, webservers, or scheduler configs
@@ -0,0 +1 @@
+0 24 * * * www-data /opt/ArchiveBox/bin/archive "https://getpocket.com/users/example/feed/all" >> /var/log/ArchiveBox.log
@@ -0,0 +1,45 @@
+user www-data;
+pid                                 /var/run/nginx.pid;
+
+worker_processes                    auto;
+timer_resolution                    100ms;
+
+worker_rlimit_nofile                40000;
+events {
+    worker_connections              40000;
+    use epoll;
+    multi_accept on;
+}
+
+http {
+    sendfile on;
+    tcp_nopush on;
+    tcp_nodelay on;
+    server_tokens off;
+    send_timeout 20;
+    keepalive_timeout 65;
+    types_hash_max_size 2048;
+    client_max_body_size 25m;
+
+    include                     mime.types;
+    default_type                application/octet-stream;
+
+    access_log                  /dev/stdout;
+    error_log                   stderr;
+
+    reset_timedout_connection on;
+
+    server_names_hash_bucket_size   64;
+
+    server {
+        listen                      80 default_server;
+        server_name                 _;
+
+        root                        /var/www;
+        index                       index.html;
+        autoindex                   on;
+
+        try_files                   $uri $uri/ $uri.html =404;
+    }
+}
+
@@ -1 +1 @@
-bin/setup-bookmark-archiver
+bin/archivebox-setup
				`@@ -0,0 +1 @@`
				`Make sure check in with me first or confirm your desired features line up with our roadmap: https://github.com/pirate/ArchiveBox#roadmap`
				`@@ -0,0 +1 @@`
				table.dataTable{width:100%;margin:0 auto;clear:both;border-collapse:separate;border-spacing:0}table.dataTable thead th,table.dataTable tfoot th{font-weight:bold}table.dataTable thead th,table.dataTable thead td{padding:10px 18px;border-bottom:1px solid #111}table.dataTable thead th:active,table.dataTable thead td:active{outline:none}table.dataTable tfoot th,table.dataTable tfoot td{padding:10px 18px 6px 18px;border-top:1px solid #111}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;*cursor:hand;background-repeat:no-repeat;background-position:center right}table.dataTable thead .sorting{background-image:url("sort_both.png")}table.dataTable thead .sorting_asc{background-image:url("sort_asc.png")}table.dataTable thead .sorting_desc{background-image:url("sort_desc.png")}table.dataTable thead .sorting_asc_disabled{background-image:url("sort_asc_disabled.png")}table.dataTable thead .sorting_desc_disabled{background-image:url("sort_desc_disabled.png")}table.dataTable tbody tr{background-color:#ffffff}table.dataTable tbody tr.selected{background-color:#B0BED9}table.dataTable tbody th,table.dataTable tbody td{padding:8px 10px}table.dataTable.row-border tbody th,table.dataTable.row-border tbody td,table.dataTable.display tbody th,table.dataTable.display tbody td{border-top:1px solid #ddd}table.dataTable.row-border tbody tr:first-child th,table.dataTable.row-border tbody tr:first-child td,table.dataTable.display tbody tr:first-child th,table.dataTable.display tbody tr:first-child td{border-top:none}table.dataTable.cell-border tbody th,table.dataTable.cell-border tbody td{border-top:1px solid #ddd;border-right:1px solid #ddd}table.dataTable.cell-border tbody tr th:first-child,table.dataTable.cell-border tbody tr td:first-child{border-left:1px solid #ddd}table.dataTable.cell-border tbody tr:first-child th,table.dataTable.cell-border tbody tr:first-child td{border-top:none}table.dataTable.stripe tbody tr.odd,table.dataTable.display tbody tr.odd{background-color:#f9f9f9}table.dataTable.stripe tbody tr.odd.selected,table.dataTable.display tbody tr.odd.selected{background-color:#acbad4}table.dataTable.hover tbody tr:hover,table.dataTable.display tbody tr:hover{background-color:#f6f6f6}table.dataTable.hover tbody tr:hover.selected,table.dataTable.display tbody tr:hover.selected{background-color:#aab7d1}table.dataTable.order-column tbody tr>.sorting_1,table.dataTable.order-column tbody tr>.sorting_2,table.dataTable.order-column tbody tr>.sorting_3,table.dataTable.display tbody tr>.sorting_1,table.dataTable.display tbody tr>.sorting_2,table.dataTable.display tbody tr>.sorting_3{background-color:#fafafa}table.dataTable.order-column tbody tr.selected>.sorting_1,table.dataTable.order-column tbody tr.selected>.sorting_2,table.dataTable.order-column tbody tr.selected>.sorting_3,table.dataTable.display tbody tr.selected>.sorting_1,table.dataTable.display tbody tr.selected>.sorting_2,table.dataTable.display tbody tr.selected>.sorting_3{background-color:#acbad5}table.dataTable.display tbody tr.odd>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd>.sorting_1{background-color:#f1f1f1}table.dataTable.display tbody tr.odd>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd>.sorting_2{background-color:#f3f3f3}table.dataTable.display tbody tr.odd>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd>.sorting_3{background-color:whitesmoke}table.dataTable.display tbody tr.odd.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_1{background-color:#a6b4cd}table.dataTable.display tbody tr.odd.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_2{background-color:#a8b5cf}table.dataTable.display tbody tr.odd.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.odd.selected>.sorting_3{background-color:#a9b7d1}table.dataTable.display tbody tr.even>.sorting_1,table.dataTable.order-column.stripe tbody tr.even>.sorting_1{background-color:#fafafa}table.dataTable.display tbody tr.even>.sorting_2,table.dataTable.order-column.stripe tbody tr.even>.sorting_2{background-color:#fcfcfc}table.dataTable.display tbody tr.even>.sorting_3,table.dataTable.order-column.stripe tbody tr.even>.sorting_3{background-color:#fefefe}table.dataTable.display tbody tr.even.selected>.sorting_1,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_1{background-color:#acbad5}table.dataTable.display tbody tr.even.selected>.sorting_2,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_2{background-color:#aebcd6}table.dataTable.display tbody tr.even.selected>.sorting_3,table.dataTable.order-column.stripe tbody tr.even.selected>.sorting_3{background-color:#afbdd8}table.dataTable.display tbody tr:hover>.sorting_1,table.dataTable.order-column.hover tbody tr:hover>.sorting_1{background-color:#eaeaea}Line truncated
				`@@ -0,0 +1 @@`
				`0 24 * * * www-data /opt/ArchiveBox/bin/archive "https://getpocket.com/users/example/feed/all" >> /var/log/ArchiveBox.log`