update docs

add user-data-dir support for chrome headless
switch back to relative paths
2017-10-30 04:08:11 -05:00 · 2017-10-30 04:01:59 -05:00 · 2017-10-30 03:36:42 -05:00 · 2017-10-30 03:31:28 -05:00 · 2017-10-30 03:29:53 -05:00 · 2017-10-30 03:28:28 -05:00
15 changed files with 1817 additions and 680 deletions
@@ -1,92 +1,19 @@
 # Pocket archive output folder
 pocket/
+bookmarks/
+pinboard/
+html/
+downloads/

 # Byte-compiled / optimized / DLL files
 __pycache__/
-*.py[cod]
-*$py.class

-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
+# Virtualenv
 env/
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-*.egg-info/
-.installed.cfg
-*.egg
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*,cover
-.hypothesis/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# IPython Notebook
-.ipynb_checkpoints
-
-# pyenv
-.python-version
-
-# celery beat schedule file
-celerybeat-schedule
-
-# dotenv
-.env
-
-# virtualenv
-venv/
 ENV/
-
-# Spyder project settings
-.spyderproject
-
-# Rope project settings
-.ropeproject
+venv/
+VENV/
+.venv/
+.env/
+.python-version
+.env
@@ -1,13 +1,13 @@
-# Bookmark Archiver <img src="https://getpocket.com/favicon.ico" height="22px"/> <img src="https://pinboard.in/favicon.ico" height="22px"/> <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> [![Twitter URL](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/thesquashSH)
+# Bookmark Archiver <img src="https://getpocket.com/favicon.ico" height="22px"/> <img src="https://pinboard.in/favicon.ico" height="22px"/> <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> [![Github Stars](https://img.shields.io/github/stars/pirate/bookmark-archiver.svg)](https://github.com/pirate/bookmark-archiver) [![Twitter URL](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/thesquashSH)

 ▶️ [Quickstart](#quickstart) | [Details](#details) | [Configuration](#configuration) | [Manual Setup](#manual-setup) | [Troubleshooting](#troubleshooting) | [Changelog](#changelog)

-Save an archived copy of all websites you star (the actual *content* of the sites, not just the list of bookmarks).
+Save an archived copy of all websites you bookmark (the actual *content* of the sites, not just the list of bookmarks).

 Outputs browsable static html archives of each site, a PDF, a screenshot, and a link to a copy on archive.org, all indexed in a nice html file.  
 (Your own personal Way-Back Machine) [DEMO: sweeting.me/pocket](https://home.sweeting.me/pocket)

-**Supports: Browser Bookmarks (Chrome, Firefox, Safari, IE, Opera), Pocket, Pinboard, Shaarli, Delicious, Instapaper, Unmark.it, and more!**
+**Supports: Browser Bookmarks (Chrome, Firefox, Safari, IE, Opera), Pocket, Pinboard, Reddit, Wallabag, Shaarli, Delicious, Instapaper, Unmark.it, RSS, and more!**

 ![](screenshot.png)

@@ -20,13 +20,16 @@ Follow the links here to find instructions for exporting bookmarks from each ser
 - [Pocket](https://getpocket.com/export)
 - [Pinboard](https://pinboard.in/export/)
 - [Instapaper](https://www.instapaper.com/user/export)
- - [Shaarli](http://sebsauvage.net/wiki/lib/exe/fetch.php?media=php:php_shaarli:shaarli_cap16_dragbutton.png)
+ - [Reddit Saved Posts](https://github.com/csu/export-saved-reddit)
+ - [Shaarli](http://shaarli.readthedocs.io/en/master/Backup,-restore,-import-and-export/#export-links-as)
 - [Unmark.it](http://help.unmark.it/import-export)
+ - [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html)
 - [Chrome Bookmarks](https://support.google.com/chrome/answer/96816?hl=en)
 - [Firefox Bookmarks](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer)
 - [Safari Bookmarks](http://i.imgur.com/AtcvUZA.png)
 - [Opera Bookmarks](http://help.opera.com/Windows/12.10/en/importexport.html)
 - [Internet Explorer Bookmarks](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows)
+ - RSS: pass the url as the second argument in the next step

 (If any of these links are broken, please submit an issue and I'll fix it)

@@ -35,19 +38,29 @@ Follow the links here to find instructions for exporting bookmarks from each ser
 ```bash
 git clone https://github.com/pirate/bookmark-archiver
 cd bookmark-archiver/
-./setup.sh
-./archive.py ~/Downloads/bookmark_export.html   # replace this path with the path to your bookmarks export file
+./setup.sh #install ALL dependencies
+./archive.py ~/Downloads/bookmark_export.html   # replace with the path to your export file from step 1
+
+# OR
+./archive.py https://getpocket.com/users/yourusername/feed/all  # url to an RSS, html, or json links file
 ```

+**3. Done!**
+
 You can open `service/index.html` to view your archive.  (favicons will appear next to each title once it has finished downloading)

+If you want to host your archive somewhere to share it with other people, see the [Publishing Your Archive](#publishing-your-archive) section below.
+
+If you want to run this as a regular script that pulls new URLs, stick it in `cron` with the second parameter as the URL to your RSS feed.
+
 If you have any trouble, see the [Troubleshooting](#troubleshooting) section at the bottom.  
+If you'd like to customize options, see the [Configuration](#configuration) section.  

 If you want something easier than running programs in the command-line, take a look at [Pocket Premium](https://getpocket.com/premium) (yay Mozilla!) and [Pinboard Pro](https://pinboard.in/upgrade/), which both offer easy-to-use bookmark archiving with full-text-search.

 ## Details

-`archive.py` is a script that takes a [Pocket-format](https://getpocket.com/export), [Pinboard-format](https://pinboard.in/export/), or [Netscape-format](https://msdn.microsoft.com/en-us/library/aa753582(v=vs.85).aspx) bookmark export file, and turns it into a browsable archive that you can store locally or host online.
+`archive.py` is a script that takes a [Pocket-format](https://getpocket.com/export), [Pinboard-format](https://pinboard.in/export/), or [Netscape-format](https://msdn.microsoft.com/en-us/library/aa753582(v=vs.85).aspx) bookmark export file, and downloads a clone of each linked website to turn into a browsable archive that you can store locally or host online.

 The archiver produces a folder like `pocket/` containing an `index.html`, and archived copies of all the sites,
 organized by starred timestamp.  It's Powered by the [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) Chromium and good 'ol `wget`.
@@ -58,8 +71,11 @@ For each sites it saves:
 - `screenshot.png` 1440x900 screenshot of site using headless chrome
 - `output.pdf` Printed PDF of site using headless chrome
 - `archive.org.txt` A link to the saved site on archive.org
- - `link.json` A json file containing link info and archive status
 - `audio/` and `video/` for sites like youtube, soundcloud, etc. (using youtube-dl) (WIP)
+ - `index.json` JSON index containing link info and archive details
+ - `index.html` HTML index containing link info and archive details
+
+Wget doesn't work on sites you need to be logged into, but chrome headless does, see the [Configuration](#configuration)* section for `CHROME_USER_DATA_DIR`.

 **Large Exports & Estimated Runtime:** 

@@ -82,35 +98,51 @@ You can tweak parameters via environment variables, or by editing `config.py` di
 env CHROME_BINARY=google-chrome-stable RESOLUTION=1440,900 FETCH_PDF=False ./archive.py ~/Downloads/bookmarks_export.html
 ```

- - Archive methods: `FETCH_WGET`, `FETCH_PDF`, `FETCH_SCREENSHOT`, `FETCH_FAVICON`, `SUBMIT_ARCHIVE_DOT_ORG` values: [`True`]/`False`
- - Archive images/css/js: `FETCH_WGET_REQUISITES` values: [`True`]/`False`  (True is highly recommended)
- - Screenshot: `RESOLUTION` values: [`1440,900`]/`1024,768`/`...`
- - Outputted Files: `ARCHIVE_PERMISSIONS` values: [`755`]/`644`/`...`
- - Path to Chrome: `CHROME_BINARY` values: [`chromium-browser`]/`/usr/local/bin/chromium-browser`/`...`
- - Path to wget: `WGET_BINARY` values: [`wget`]/`/usr/local/bin/wget`/`...`
- - Download timemout: `TIMEOUT` values: [`60`]/`30`/`...`
+**Shell Options:**
+ - colorize console ouput: `USE_COLOR` value: [`True`]/`False`
+ - show progress bar: `SHOW_PROGRESS` value: [`True`]/`False`
+ - archive permissions: `ARCHIVE_PERMISSIONS` values: [`755`]/`644`/`...`
+
+**Dependency Options:**
+ - path to Chrome: `CHROME_BINARY` values: [`chromium-browser`]/`/usr/local/bin/google-chrome`/`...`
+ - path to wget: `WGET_BINARY` values: [`wget`]/`/usr/local/bin/wget`/`...`
+
+**Archive Options:**
+ - maximum allowed download time per link: `TIMEOUT` values: [`60`]/`30`/`...`
+ - archive methods (values: [`True`]/`False`):
+   - fetch page with wget: `FETCH_WGET`
+   - fetch images/css/js with wget: `FETCH_WGET_REQUISITES` (True is highly recommended)
+   - print page as PDF: `FETCH_PDF`
+   - fetch a screenshot of the page: `FETCH_SCREENSHOT`
+   - fetch a favicon for the page: `FETCH_FAVICON`
+   - submit the page to archive.org: `SUBMIT_ARCHIVE_DOT_ORG` 
+ - screenshot: `RESOLUTION` values: [`1440,900`]/`1024,768`/`...`
+ - user agent: `WGET_USER_AGENT` values: [`Wget/1.19.1`]/`"Mozilla/5.0 ..."`/`...`
+ - chrome profile: `CHROME_USER_DATA_DIR` values: `~/Library/Application\ Support/Google/Chrome/Default`/`/tmp/chrome-profile`/`...`
+    To capture sites that require a user to be logged in, you must specify a path to a chrome profile (which loads the cookies needed for the user to be logged in).  If you don't have an existing chrome profile, create one with `chromium-browser --disable-gpu --user-data-dir=/tmp/chrome-profile`, and log into the sites you need.  Then set `CHROME_USER_DATA_DIR=/tmp/chrome-profile` to make Bookmark Archiver use that profile.
+
+**Index Options:**
 - html index template: `INDEX_TEMPLATE` value:  `templates/index.html`/`...`
 - html index row template: `INDEX_ROW_TEMPLATE` value:  `templates/index_row.html`/`...`
+ - html link index template: `LINK_INDEX_TEMPLATE` value: `templates/link_index_fancy.html`/`templates/link_index.html`/`...`

 (See defaults & more at the top of `config.py`)

 To tweak the outputted html index file's look and feel, just copy the files in `templates/` somewhere else and edit away.  Use the two index config variables above to point the script to your new custom template files. 

-The templates use format strings (not a proper templating engine like jinja2), which is why the CSS is double-bracketed `{{...}}`.
+The chrome/chromium dependency is _optional_ and only required for screenshots and PDF output, can be safely ignored if both of those are disabled.

 ## Publishing Your Archive

-The archive produced by `./archive.py` is suitable for serving on any provider that 
-can host static html (e.g. github pages!).
+The archive produced by `./archive.py` is suitable for serving on any provider that can host static html (e.g. github pages!).

-You can also serve it from a home server or VPS by uploading the archive folder 
-to your web directory, e.g. `/var/www/pocket` and configuring your webserver.
+You can also serve it from a home server or VPS by uploading the outputted `html` folder to your web directory, e.g. `/var/www/bookmark-archiver` and configuring your webserver.

 Here's a sample nginx configuration that works to serve archive folders:

 ```nginx
-location /pocket/ {
-    alias       /var/www/pocket/;
+location / {
+    alias       /var/www/bookmark-archiver/;
    index       index.html;
    autoindex   on;                         # see directory listing upon clicking "The Files" links
    try_files   $uri $uri/ =404;
@@ -119,7 +151,7 @@ location /pocket/ {

 Make sure you're not running any content as CGI or PHP, you only want to serve static files!

-Urls look like: `https://sweeting.me/archive/archive/1493350273/en.wikipedia.org/wiki/Dining_philosophers_problem.html`
+Urls look like: `https://archive.example.com/archive/1493350273/en.wikipedia.org/wiki/Dining_philosophers_problem.html`

 **Security WARNING & Content Disclaimer**

@@ -146,11 +178,12 @@ My published archive as an example: [sweeting.me/pocket](https://home.sweeting.m

 ## Manual Setup

-If you don't like `sudo` running random setup scripts off the internet (which you shouldn't), you can follow these manual setup instructions.
+If you don't like running random setup scripts off the internet (:+1:), you can follow these manual setup instructions.

-**1. Install dependencies:** `chromium >= 59`,` wget >= 1.16`, `python3 >= 3.5`  (google-chrome >= v59 also works well)
+**1. Install dependencies:** `chromium >= 59`,` wget >= 1.16`, `python3 >= 3.5`  (`google-chrome >= v59` works fine as well)

 If you already have Google Chrome installed, or wish to use that instead of Chromium, follow the [Google Chrome Instructions](#google-chrome-instructions).
+
 ```bash
 # On Mac:
 brew cask install chromium  # If you already have Google Chrome/Chromium in /Applications/, skip this command
@@ -232,27 +265,40 @@ If you still need help, [the official Python docs](https://docs.python.org/3.6/u

 `archive.py` depends on being able to access a `chromium-browser`/`google-chrome` executable.  The executable used
 defaults to `chromium-browser` but can be manually specified with the environment variable `CHROME_BINARY`:
+
 ```bash
 env CHROME_BINARY=/usr/local/bin/chromium-browser ./archive.py ~/Downloads/bookmarks_export.html
 ```

 1. Test to make sure you have Chrome on your `$PATH` with:
+
 ```bash
 which chromium-browser || which google-chrome
 ```
 If no executable is displayed, follow the setup instructions to install and link one of them.

 2. If a path is displayed, the next step is to check that it's runnable:
+
 ```bash
 chromium-browser --version || google-chrome --version
 ```
 If no version is displayed, try the setup instructions again, or confirm that you have permission to access chrome.

-3. If a version is displayed and it's `>=59`, make sure `archive.py` is running the right one:
+3. If a version is displayed and it's `<59`, upgrade it:
+
+```bash
+apt upgrade chromium-browser -y
+# OR
+brew cask upgrade chromium-browser
+```
+
+4. If a version is displayed and it's `>=59`, make sure `archive.py` is running the right one:
+
 ```bash
 env CHROME_BINARY=/path/from/step/1/chromium-browser ./archive.py bookmarks_export.html   # replace the path with the one you got from step 1
 ```

+
 **Wget & Curl:**

 If you're missing `wget` or `curl`, simply install them using `apt` or your package manager of choice.
@@ -298,6 +344,7 @@ if you have problem with a particular nginx config.

 If you feel like contributing a PR, some of these tasks are pretty easy.  Feel free to open an issue if you need help getting started in any way!

+ - download closed-captions text from youtube videos
 - body text extraction using [fathom](https://hacks.mozilla.org/2017/04/fathom-a-framework-for-understanding-web-pages/)
 - auto-tagging based on important extracted words
 - audio & video archiving with `youtube-dl`
@@ -325,9 +372,16 @@ will run fast subsequent times because it only downloads new links that haven't
 - https://webrecorder.io/
 - https://github.com/ikreymer/webarchiveplayer#auto-load-warcs
 - [Shaarchiver](https://github.com/nodiscc/shaarchiver) very similar project that archives Firefox, Shaarli, or Delicious bookmarks and all linked media, generating a markdown/HTML index
+ - [Sheetsee-Pocket](http://jlord.us/sheetsee-pocket/) project that provides a pretty auto-updating index of your Pocket links (without archiving them)
+ - [Pocket -> IFTTT -> Dropbox](https://christopher.su/2013/saving-pocket-links-file-day-dropbox-ifttt-launchd/) Post by Christopher Su on his Pocket saving IFTTT recipie

 ## Changelog

+ - v0.0.2 released
+ - support for chrome `--user-data-dir` to archive sites that need logins
+ - fancy individual html & json indexes for each link
+ - smartly append new links to existing index instead of overwriting 
+ - proper HTML templating instead of format strings (thanks to https://github.com/bardisty!)
 - refactored into separate files, wip audio & video archiving
 - v0.0.1 released
 - Index links now work without nginx url rewrites, archive can now be hosted on github pages
@@ -3,77 +3,92 @@
 # Nick Sweeting 2017 | MIT License
 # https://github.com/pirate/bookmark-archiver

-import os
 import sys

 from datetime import datetime
-from subprocess import run

-from parse import parse_export
-from index import dump_index
-from fetch import dump_website
+from parse import parse_links
+from links import validate_links
+from archive_methods import archive_links, _RESULTS_TOTALS
+from index import (
+    write_links_index,
+    write_link_index,
+    parse_json_links_index,
+    parse_json_link_index,
+)
 from config import (
    ARCHIVE_PERMISSIONS,
+    HTML_FOLDER,
+    ARCHIVE_FOLDER,
+    ANSI,
+    TIMEOUT,
+)
+from util import (
+    download_url,
    check_dependencies,
+    progress,
+    cleanup_archive,
 )

-DESCRIPTION = 'Bookmark Archiver: Create a browsable html archive of a list of links.'
+__DESCRIPTION__ = 'Bookmark Archiver: Create a browsable html archive of a list of links.'
 __DOCUMENTATION__ = 'https://github.com/pirate/bookmark-archiver'


-def create_archive(export_file, service=None, resume=None):
-    """update or create index.html and download archive of all links"""
+def update_archive(export_path, links, resume=None, append=True):
+    """update or create index.html+json given a path to an export file containing new links"""

-    with open(export_file, 'r', encoding='utf-8') as f:
-        print('[+] [{}] Starting archive from {} export file.'.format(
-            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            export_file
-        ))
+    start_ts = datetime.now().timestamp()

-        links, service = parse_export(f, service=service)
+    # loop over links and archive them
+    archive_links(ARCHIVE_FOLDER, links, export_path, resume=resume)

-        if resume:
-            try:
-                links = [
-                    link
-                    for link in links
-                    if float(link['timestamp']) >= float(resume)
-                ]
-            except TypeError:
-                print('Resume value and all timestamp values must be valid numbers.')
-
-    if not links or not service:
-        print('[X] No links found in {}, is it a {} export file?'.format(export_file, service))
-        raise SystemExit(1)
-
-    if not os.path.exists(service):
-        os.makedirs(service)
-
-    if not os.path.exists(os.path.join(service, 'archive')):
-        os.makedirs(os.path.join(service, 'archive'))
-
-    dump_index(links, service)
-
-    run(['chmod', '-R', ARCHIVE_PERMISSIONS, service], timeout=30)
-
-    print('[*] [{}] Created archive index with {} links.'.format(
+    # print timing information & summary
+    end_ts = datetime.now().timestamp()
+    seconds = round(end_ts - start_ts, 1)
+    duration = '{} min'.format(seconds / 60) if seconds > 60 else '{} sec'.format(seconds)
+    print('{}[√] [{}] Archive update complete ({}){}'.format(
+        ANSI['green'],
        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-        len(links),
+        duration,
+        ANSI['reset'],
+    ))
+    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
+    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
+    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
+
+
+def update_index(export_path, resume=None, append=True):
+    """handling parsing new links into the json index, returns a set of clean links"""
+
+    # parse an validate the export file
+    new_links = validate_links(parse_links(export_path))
+
+    # load existing links if archive folder is present
+    existing_links = []
+    if append:
+        existing_links = parse_json_links_index(HTML_FOLDER)
+        links = validate_links(existing_links + new_links)
+        
+
+    # merge existing links and new links
+    num_new_links = len(links) - len(existing_links)
+    print('[*] [{}] Adding {} new links from {} to index'.format(
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        num_new_links,
+        export_path,
    ))

-    check_dependencies()
+    # write link index html & json
+    write_links_index(HTML_FOLDER, links)

-    for link in links:
-        dump_website(link, service)
-
-    print('[√] [{}] Archive update complete.'.format(datetime.now().strftime('%Y-%m-%d %H:%M:%S')))
+    return links


 if __name__ == '__main__':
    argc = len(sys.argv)

    if argc < 2 or sys.argv[1] in ('-h', '--help', 'help'):
-        print(DESCRIPTION)
+        print(__DESCRIPTION__)
        print("Documentation:     {}".format(__DOCUMENTATION__))
        print("")
        print("Usage:")
@@ -81,8 +96,14 @@ if __name__ == '__main__':
        print("")
        raise SystemExit(0)

-    export_file = sys.argv[1]                                       # path to export file
-    export_type = sys.argv[2] if argc > 2 else None                 # select export_type for file format select
-    resume_from = sys.argv[3] if argc > 3 else None                 # timestamp to resume dowloading from
+    export_path = sys.argv[1]                        # path to export file
+    resume_from = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from

-    create_archive(export_file, service=export_type, resume=resume_from)
+    if any(export_path.startswith(s) for s in ('http://', 'https://', 'ftp://')):
+        export_path = download_url(export_path)
+
+    links = update_index(export_path, resume=resume_from, append=True)
+
+    # make sure folder structure is sane
+    cleanup_archive(ARCHIVE_FOLDER, links)
+    update_archive(export_path, links, resume=resume_from, append=True)
@@ -0,0 +1,427 @@
+import os
+
+from functools import wraps
+from datetime import datetime
+from subprocess import run, PIPE, DEVNULL
+
+from index import html_appended_url, parse_json_link_index, write_link_index
+from links import links_after_timestamp
+from config import (
+    ARCHIVE_DIR,
+    CHROME_BINARY,
+    FETCH_WGET,
+    FETCH_WGET_REQUISITES,
+    FETCH_PDF,
+    FETCH_SCREENSHOT,
+    RESOLUTION,
+    SUBMIT_ARCHIVE_DOT_ORG,
+    FETCH_AUDIO,
+    FETCH_VIDEO,
+    FETCH_FAVICON,
+    WGET_USER_AGENT,
+    CHROME_USER_DATA_DIR,
+    TIMEOUT,
+    ANSI,
+)
+from util import (
+    check_dependencies,
+    progress,
+    chmod_file,
+)
+
+
+_RESULTS_TOTALS = {   # globals are bad, mmkay
+    'skipped': 0,
+    'succeded': 0,
+    'failed': 0,
+}
+
+def archive_links(out_dir, links, export_path, resume=None):
+    check_dependencies()
+
+    to_archive = links_after_timestamp(links, resume)
+    try:
+        for idx, link in enumerate(to_archive):
+            link_dir = os.path.join(out_dir, link['timestamp'])
+            archive_link(link_dir, link)
+    
+    except (KeyboardInterrupt, SystemExit, Exception) as e:
+        print('{red}[X] Archive update stopped on #{idx} out of {total} links{reset}'.format(
+            **ANSI,
+            idx=idx,
+            total=len(list(to_archive)),
+        ))
+        print('    Continue where you left off by running:')
+        print('       ./archive.py {} {}'.format(
+            export_path,
+            link['timestamp'],
+        ))
+        if not isinstance(e, KeyboardInterrupt):
+            raise e
+        raise SystemExit(1)
+
+
+def archive_link(out_dir, link, overwrite=False):
+    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+
+    link = {**parse_json_link_index(out_dir), **link}
+    log_link_archive(out_dir, link)
+
+    if FETCH_WGET:
+        link = fetch_wget(out_dir, link, overwrite=overwrite)
+
+    if FETCH_PDF:
+        link = fetch_pdf(out_dir, link, overwrite=overwrite)
+
+    if FETCH_SCREENSHOT:
+        link = fetch_screenshot(out_dir, link, overwrite=overwrite)
+
+    if SUBMIT_ARCHIVE_DOT_ORG:
+        link = archive_dot_org(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_AUDIO:
+    #     link = fetch_audio(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_VIDEO:
+    #     link = fetch_video(out_dir, link, overwrite=overwrite)
+
+    if FETCH_FAVICON:
+        link = fetch_favicon(out_dir, link, overwrite=overwrite)
+
+    write_link_index(out_dir, link)
+    
+    return link
+
+def log_link_archive(out_dir, link):
+    update_existing = os.path.exists(out_dir)
+    if not update_existing:
+        os.makedirs(out_dir)
+    
+    print('[{symbol_color}{symbol}{reset}] [{timestamp}] "{title}": {blue}{base_url}{reset}'.format(
+        symbol='*' if update_existing else '+',
+        symbol_color=ANSI['black' if update_existing else 'green'],
+        **link,
+        **ANSI,
+    ))
+    if link['type']:
+        print('    i Type: {}'.format(link['type']))
+
+    print('    {} ({})'.format(out_dir, 'updating' if update_existing else 'creating'))
+
+
+
+def attach_result_to_link(method):
+    """
+    Instead of returning a result={output:'...', status:'success'} object,
+    attach that result to the links's history & latest fields, then return
+    the updated link object.
+    """
+    def decorator(fetch_func):
+        @wraps(fetch_func)
+        def timed_fetch_func(out_dir, link, overwrite=False, **kwargs):
+            # initialize methods and history json field on link
+            link['latest'] = link.get('latest') or {}
+            link['latest'][method] = link['latest'].get(method) or None
+            link['history'] = link.get('history') or {}
+            link['history'][method] = link['history'].get(method) or []
+
+            start_ts = datetime.now().timestamp()
+
+            # if a valid method output is already present, dont run the fetch function
+            if link['latest'][method] and not overwrite:
+                print('    √ Skipping: {}'.format(method))
+                result = None
+            else:
+                print('    - Fetching: {}'.format(method))
+                result = fetch_func(out_dir, link, **kwargs)
+
+            end_ts = datetime.now().timestamp()
+            duration = str(end_ts * 1000 - start_ts * 1000).split('.')[0]
+
+            # append a history item recording fail/success
+            history_entry = {
+                'timestamp': str(start_ts).split('.')[0],
+            }
+            if result is None:
+                history_entry['status'] = 'skipped'
+            elif isinstance(result.get('output'), Exception):
+                history_entry['status'] = 'failed'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+            else:
+                history_entry['status'] = 'succeded'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+                link['latest'][method] = result['output']
+            
+            _RESULTS_TOTALS[history_entry['status']] += 1
+            
+            return link
+        return timed_fetch_func
+    return decorator
+
+
+@attach_result_to_link('wget')
+def fetch_wget(out_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
+    """download full site using wget"""
+
+    if os.path.exists(os.path.join(out_dir, link['domain'])):
+        return {'output': html_appended_url(link), 'status': 'skipped'}
+
+    CMD = [
+        *'wget --timestamping --adjust-extension --no-parent'.split(' '),                # Docs: https://www.gnu.org/software/wget/manual/wget.html
+        *(('--page-requisites', '--convert-links') if FETCH_WGET_REQUISITES else ()),
+        *(('--user-agent="{}"'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
+        link['url'],
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # index.html
+        end()
+        output = html_appended_url(link)
+        if result.returncode > 0:
+            print('       got wget response code {}:'.format(result.returncode))
+            print('\n'.join('         ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
+            # raise Exception('Failed to wget download')
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('pdf')
+def fetch_pdf(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
+    """print PDF of site to file using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'output.pdf')):
+        return {'output': 'output.pdf', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --print-to-pdf'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # output.pdf
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to print PDF')
+        output = 'output.pdf'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('screenshot')
+def fetch_screenshot(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR, resolution=RESOLUTION):
+    """take screenshot of site using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'screenshot.png')):
+        return {'output': 'screenshot.png', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --screenshot'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        '--window-size={}'.format(resolution),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # sreenshot.png
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to take screenshot')
+        chmod_file('screenshot.png', cwd=out_dir)
+        output = 'screenshot.png'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+    
+
+@attach_result_to_link('archive_org')
+def archive_dot_org(out_dir, link, timeout=TIMEOUT):
+    """submit site to archive.org for archiving via their service, save returned archive url"""
+
+    path = os.path.join(out_dir, 'archive.org.txt')
+    if os.path.exists(path):
+        archive_org_url = open(path, 'r').read().strip()
+        return {'output': archive_org_url, 'status': 'skipped'}
+
+    submit_url = 'https://web.archive.org/save/{}'.format(link['url'].split('?', 1)[0])
+
+    success = False
+    CMD = ['curl', '-I', submit_url]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # archive.org.txt
+        end()
+
+        # Parse archive.org response headers
+        headers = result.stdout.splitlines()
+        content_location = [h for h in headers if b'Content-Location: ' in h]
+        errors = [h for h in headers if h and b'X-Archive-Wayback-Runtime-Error: ' in h]
+
+        if content_location:
+            archive_path = content_location[0].split(b'Content-Location: ', 1)[-1].decode('utf-8')
+            saved_url = 'https://web.archive.org{}'.format(archive_path)
+            success = True
+
+        elif len(errors) == 1 and b'RobotAccessControlException' in errors[0]:
+            output = submit_url
+            # raise Exception('Archive.org denied by {}/robots.txt'.format(link['domain']))
+        elif errors:
+            raise Exception(', '.join(e.decode() for e in errors))
+        else:
+            raise Exception('Failed to find "Content-Location" URL header in Archive.org response.')
+    except Exception as e:
+        end()
+        print('       Visit url to see output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    if success:
+        with open(os.path.join(out_dir, 'archive.org.txt'), 'w', encoding='utf-8') as f:
+            f.write(saved_url)
+        chmod_file('archive.org.txt', cwd=out_dir)
+        output = saved_url
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('favicon')
+def fetch_favicon(out_dir, link, timeout=TIMEOUT):
+    """download site favicon from google's favicon api"""
+
+    if os.path.exists(os.path.join(out_dir, 'favicon.ico')):
+        return {'output': 'favicon.ico', 'status': 'skipped'}
+
+    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
+    fout = open('{}/favicon.ico'.format(out_dir), 'w')
+    end = progress(timeout, prefix='      ')
+    try:
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # favicon.ico
+        fout.close()
+        end()
+        chmod_file('favicon.ico', cwd=out_dir)
+        output = 'favicon.ico'
+    except Exception as e:
+        fout.close()
+        end()
+        print('       Run to see full output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+# @attach_result_to_link('audio')
+# def fetch_audio(out_dir, link, timeout=TIMEOUT):
+#     """Download audio rip using youtube-dl"""
+
+#     if link['type'] not in ('soundcloud',)\
+#        and 'audio' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'audio')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading audio')
+#         CMD = [
+#             "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # audio/audio.mp3
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download audio')
+#             chmod_file('audio.mp3', cwd=out_dir)
+#             return 'audio.mp3'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping audio download')
+
+# @attach_result_to_link('video')
+# def fetch_video(out_dir, link, timeout=TIMEOUT):
+#     """Download video rip using youtube-dl"""
+
+#     if link['type'] not in ('youtube', 'youku', 'vimeo')\
+#        and 'video' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'video')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading video')
+#         CMD = [
+#             "youtube-dl -x --video-format mp4 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # video/movie.mp4
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download video')
+#             chmod_file('video.mp4', cwd=out_dir)
+#             return 'video.mp4'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping video download')
+
+
+def chrome_data_dir_args(user_data_dir=CHROME_USER_DATA_DIR):
+    default = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
+    if user_data_dir:
+        return ('--user-data-dir={}'.format(user_data_dir),)
+    elif os.path.exists(default):
+        return ('--user-data-dir={}'.format(default),)
+    return ()
@@ -1,68 +1,78 @@
 import os
 import sys
+import shutil

 from subprocess import run, PIPE

-# os.getenv('VARIABLE', 'DEFAULT') gets the value of environment
-# variable "VARIABLE" and if it is not set, sets it to 'DEFAULT'
-
-# for boolean values, check to see if the string is 'true', and
-# if so, the python variable will be True
+# ******************************************************************************
+# * TO SET YOUR CONFIGURATION, EDIT THE VALUES BELOW, or use the 'env' command *
+# * e.g.                                                                       *
+# * env USE_COLOR=True CHROME_BINARY=google-chrome ./archive.py export.html    *
+# ******************************************************************************

+IS_TTY = sys.stdout.isatty()
+USE_COLOR =              os.getenv('USE_COLOR',              str(IS_TTY)        ).lower() == 'true'
+SHOW_PROGRESS =          os.getenv('SHOW_PROGRESS',          str(IS_TTY)        ).lower() == 'true'
 FETCH_WGET =             os.getenv('FETCH_WGET',             'True'             ).lower() == 'true'
 FETCH_WGET_REQUISITES =  os.getenv('FETCH_WGET_REQUISITES',  'True'             ).lower() == 'true'
-FETCH_AUDIO =            os.getenv('FETCH_AUDIO',            'False'             ).lower() == 'true'
-FETCH_VIDEO =            os.getenv('FETCH_VIDEO',            'False'             ).lower() == 'true'
+FETCH_AUDIO =            os.getenv('FETCH_AUDIO',            'False'            ).lower() == 'true'
+FETCH_VIDEO =            os.getenv('FETCH_VIDEO',            'False'            ).lower() == 'true'
 FETCH_PDF =              os.getenv('FETCH_PDF',              'True'             ).lower() == 'true'
 FETCH_SCREENSHOT =       os.getenv('FETCH_SCREENSHOT',       'True'             ).lower() == 'true'
 FETCH_FAVICON =          os.getenv('FETCH_FAVICON',          'True'             ).lower() == 'true'
 SUBMIT_ARCHIVE_DOT_ORG = os.getenv('SUBMIT_ARCHIVE_DOT_ORG', 'True'             ).lower() == 'true'
 RESOLUTION =             os.getenv('RESOLUTION',             '1440,900'         )
 ARCHIVE_PERMISSIONS =    os.getenv('ARCHIVE_PERMISSIONS',    '755'              )
+ARCHIVE_DIR =            os.getenv('ARCHIVE_DIR',            '')
 CHROME_BINARY =          os.getenv('CHROME_BINARY',          'chromium-browser' )  # change to google-chrome browser if using google-chrome
 WGET_BINARY =            os.getenv('WGET_BINARY',            'wget'             )
+WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',         None)
+CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',    None)
 TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
+LINK_INDEX_TEMPLATE =    os.getenv('LINK_INDEX_TEMPLATE',    'templates/link_index_fancy.html')
 INDEX_TEMPLATE =         os.getenv('INDEX_TEMPLATE',         'templates/index.html')
 INDEX_ROW_TEMPLATE =     os.getenv('INDEX_ROW_TEMPLATE',     'templates/index_row.html')

+### Output Paths
+ROOT_FOLDER = os.path.dirname(os.path.abspath(__file__))
+HTML_FOLDER = os.path.join(ARCHIVE_DIR, 'html')
+ARCHIVE_FOLDER = os.path.join(HTML_FOLDER, 'archive')
+os.chdir(ROOT_FOLDER)

-def check_dependencies():
-    print('[*] Checking Dependencies:')
+# ******************************************************************************
+# ********************** Do not edit below this point **************************
+# ******************************************************************************

-    python_vers = float('{}.{}'.format(sys.version_info.major, sys.version_info.minor))
-    if python_vers < 3.5:
-        print('[X] Python version is not new enough: {} (>3.5 is required)'.format(python_vers))
-        print('    See https://github.com/pirate/bookmark-archiver#troubleshooting for help upgrading your Python installation.')
-        raise SystemExit(1)
+### Terminal Configuration
+TERM_WIDTH = shutil.get_terminal_size((100, 10)).columns
+ANSI = {
+    'reset': '\033[00;00m',
+    'lightblue': '\033[01;30m',
+    'lightyellow': '\033[01;33m',
+    'lightred': '\033[01;35m',
+    'red': '\033[01;31m',
+    'green': '\033[01;32m',
+    'blue': '\033[01;34m',
+    'white': '\033[01;37m',
+    'black': '\033[01;30m',
+}
+if not USE_COLOR:
+    # dont show colors if USE_COLOR is False
+    ANSI = {k: '' for k in ANSI.keys()}

-    if FETCH_PDF or FETCH_SCREENSHOT:
-        if run(['which', CHROME_BINARY]).returncode:
-            print('[X] Missing dependency: {}'.format(CHROME_BINARY))
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
+### Confirm Environment Setup
+try:
+    GIT_SHA = run(["git", "rev-list", "-1", "HEAD", "./"], stdout=PIPE, cwd=ROOT_FOLDER).stdout.strip().decode()
+except Exception:
+    GIT_SHA = None
+    print('[!] Warning, you need git installed for some archiving features to save correct version numbers!')

-        # parse chrome --version e.g. Google Chrome 61.0.3114.0 canary / Chromium 59.0.3029.110 built on Ubuntu, running on Ubuntu 16.04
-        result = run([CHROME_BINARY, '--version'], stdout=PIPE)
-        version = result.stdout.decode('utf-8').replace('Google Chrome ', '').replace('Chromium ', '').split(' ', 1)[0].split('.', 1)[0]  # TODO: regex might be better
-        if int(version) < 59:
-            print('[X] Chrome version must be 59 or greater for headless PDF and screenshot saving')
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
-
-    if FETCH_WGET:
-        if run(['which', 'wget']).returncode:
-            print('[X] Missing dependency: wget')
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
-
-    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
-        if run(['which', 'curl']).returncode:
-            print('[X] Missing dependency: curl')
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
-
-    if FETCH_AUDIO or FETCH_VIDEO:
-        if run(['which', 'youtube-dl']).returncode:
-            print('[X] Missing dependency: youtube-dl')
-            print('    See https://github.com/pirate/bookmark-archiver for help.')
-            raise SystemExit(1)
+if sys.stdout.encoding.upper() != 'UTF-8':
+    print('[X] Your system is running python3 scripts with a bad locale setting: {} (it should be UTF-8).'.format(sys.stdout.encoding))
+    print('    To fix it, add the line "export PYTHONIOENCODING=utf8" to your ~/.bashrc file (without quotes)')
+    print('')
+    print('    Confirm that it\'s fixed by opening a new shell and running:')
+    print('        python3 -c "import sys; print(sys.stdout.encoding)"   # should output UTF-8')
+    print('')
+    print('    Alternatively, run this script with:')
+    print('        env PYTHONIOENCODING=utf8 ./archive.py export.html')
@@ -1,270 +0,0 @@
-import os
-import json
-
-from datetime import datetime
-from subprocess import run, PIPE, DEVNULL
-
-from parse import derived_link_info
-from config import (
-    ARCHIVE_PERMISSIONS,
-    FETCH_WGET,
-    FETCH_WGET_REQUISITES,
-    FETCH_PDF,
-    FETCH_SCREENSHOT,
-    RESOLUTION,
-    SUBMIT_ARCHIVE_DOT_ORG,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
-    FETCH_FAVICON,
-)
-
-
-def chmod_file(path, cwd='.', permissions='755', timeout=30):
-    if not os.path.exists(os.path.join(cwd, path)):
-        raise Exception('Failed to chmod: {} does not exist (did the previous step fail?)'.format(path))
-
-    chmod_result = run(['chmod', '-R', permissions, path], cwd=cwd, stdout=DEVNULL, stderr=PIPE, timeout=timeout)
-    if chmod_result.returncode == 1:
-        print('     ', chmod_result.stderr.decode())
-        raise Exception('Failed to chmod {}/{}'.format(cwd, path))
-
-def fetch_wget(out_dir, link, overwrite=False, requisites=True, timeout=60):
-    """download full site using wget"""
-
-    domain = link['base_url'].split('/', 1)[0]
-    if not os.path.exists(os.path.join(out_dir, domain)) or overwrite:
-        print('    - Downloading Full Site')
-        CMD = [
-            *'wget --timestamping --adjust-extension --no-parent'.split(' '),                # Docs: https://www.gnu.org/software/wget/manual/wget.html
-            *(('--page-requisites', '--convert-links') if requisites else ()),
-            link['url'],
-        ]
-        try:
-            result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout)  # dom.html
-            if result.returncode > 0:
-                print('     ', result.stderr.decode().split('\n')[-1])
-                raise Exception('Failed to wget download')
-            chmod_file(domain, cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping site download')
-
-def fetch_pdf(out_dir, link, overwrite=False, timeout=60, chrome_binary='chromium-browser'):
-    """print PDF of site to file using chrome --headless"""
-
-    path = os.path.join(out_dir, 'output.pdf')
-
-    if (not os.path.exists(path) or overwrite) and link['type'] not in ('PDF', 'image'):
-        print('    - Printing PDF')
-        CMD = [
-            chrome_binary,
-            *'--headless --disable-gpu --print-to-pdf'.split(' '),
-            link['url']
-        ]
-        try:
-            result = run(CMD, stdout=DEVNULL, stderr=PIPE, cwd=out_dir, timeout=timeout)  # output.pdf
-            if result.returncode:
-                print('     ', result.stderr.decode())
-                raise Exception('Failed to print PDF')
-            chmod_file('output.pdf', cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping PDF print')
-
-def fetch_screenshot(out_dir, link, overwrite=False, timeout=60, chrome_binary='chromium-browser', resolution='1440,900'):
-    """take screenshot of site using chrome --headless"""
-
-    path = os.path.join(out_dir, 'screenshot.png')
-
-    if (not os.path.exists(path) or overwrite) and link['type'] not in ('PDF', 'image'):
-        print('    - Snapping Screenshot')
-        CMD = [
-            chrome_binary,
-            *'--headless --disable-gpu --screenshot'.split(' '),
-            '--window-size={}'.format(resolution),
-            link['url']
-        ]
-        try:
-            result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout)  # sreenshot.png
-            if result.returncode:
-                print('     ', result.stderr.decode())
-                raise Exception('Failed to take screenshot')
-            chmod_file('screenshot.png', cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping screenshot')
-
-def archive_dot_org(out_dir, link, overwrite=False, timeout=60):
-    """submit site to archive.org for archiving via their service, save returned archive url"""
-
-    path = os.path.join(out_dir, 'archive.org.txt')
-
-    if not os.path.exists(path) or overwrite:
-        print('    - Submitting to archive.org')
-        submit_url = 'https://web.archive.org/save/{}'.format(link['url'].split('?', 1)[0])
-
-        success = False
-        CMD = ['curl', '-I', submit_url]
-        try:
-            result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=out_dir, timeout=timeout)  # archive.org
-            headers = result.stdout.splitlines()
-            content_location = [h for h in headers if b'Content-Location: ' in h]
-            if content_location:
-                archive_path = content_location[0].split(b'Content-Location: ', 1)[-1].decode('utf-8')
-                saved_url = 'https://web.archive.org{}'.format(archive_path)
-                success = True
-            else:
-                raise Exception('Failed to find "Content-Location" URL header in Archive.org response.')
-        except Exception as e:
-            print('       Visit url to see output:', ' '.join(CMD))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-
-        if success:
-            with open('{}/archive.org.txt'.format(out_dir), 'w', encoding='utf-8') as f:
-                f.write(saved_url)
-            chmod_file('archive.org.txt', cwd=out_dir)
-
-    else:
-        print('    √ Skipping archive.org')
-
-def fetch_favicon(out_dir, link, overwrite=False, timeout=60):
-    """download site favicon from google's favicon api"""
-
-    path = os.path.join(out_dir, 'favicon.ico')
-
-    if not os.path.exists(path) or overwrite:
-        print('    - Fetching Favicon')
-        CMD = 'curl https://www.google.com/s2/favicons?domain={domain}'.format(**link).split(' ')
-        fout = open('{}/favicon.ico'.format(out_dir), 'w')
-        try:
-            run([*CMD], stdout=fout, stderr=DEVNULL, cwd=out_dir, timeout=timeout)  # favicon.ico
-            chmod_file('favicon.ico', cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', ' '.join(CMD))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-        fout.close()
-    else:
-        print('    √ Skipping favicon')
-
-def fetch_audio(out_dir, link, overwrite=False, timeout=60):
-    """Download audio rip using youtube-dl"""
-
-    if link['type'] not in ('soundcloud',):
-        return
-
-    path = os.path.join(out_dir, 'audio')
-
-    if not os.path.exists(path) or overwrite:
-        print('    - Downloading audio')
-        CMD = [
-            "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
-            link['url'],
-        ]
-        try:
-            result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout)  # sreenshot.png
-            if result.returncode:
-                print('     ', result.stderr.decode())
-                raise Exception('Failed to download audio')
-            chmod_file('audio', cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping audio download')
-
-def fetch_video(out_dir, link, overwrite=False, timeout=60):
-    """Download video rip using youtube-dl"""
-
-    if link['type'] not in ('youtube', 'youku', 'vimeo'):
-        return
-
-    path = os.path.join(out_dir, 'video')
-
-    if not os.path.exists(path) or overwrite:
-        print('    - Downloading video')
-        CMD = [
-            "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
-            link['url'],
-        ]
-        try:
-            result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout)  # sreenshot.png
-            if result.returncode:
-                print('     ', result.stderr.decode())
-                raise Exception('Failed to download video')
-            chmod_file('video', cwd=out_dir)
-        except Exception as e:
-            print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping video download')
-
-def dump_link_info(out_dir, link, update=True):
-    """write a json file with some info about the link"""
-
-    info_file_path = os.path.join(out_dir, 'link.json')
-
-    if (not os.path.exists(info_file_path) or update):
-        print('    - Creating link info file')
-        try:
-            link_json = derived_link_info(link)
-            link_json['archived_timstamp'] = str(datetime.now().timestamp()).split('.')[0]
-
-            with open(info_file_path, 'w', encoding='utf-8') as link_file:
-                link_file.write(json.dumps(
-                    link_json,
-                    indent=4,
-                    default=str) + '\n')
-
-            chmod_file('link.json', cwd=out_dir)
-        except Exception as e:
-            print('       Failed: {} {}'.format(e.__class__.__name__, e))
-    else:
-        print('    √ Skipping link info file')
-
-
-def dump_website(link, service, overwrite=False, permissions=ARCHIVE_PERMISSIONS):
-    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
-
-    print('[+] [{timestamp} ({time})] "{title}": {base_url}'.format(**link))
-
-    out_dir = os.path.join(service, 'archive', link['timestamp'])
-    if not os.path.exists(out_dir):
-        os.makedirs(out_dir)
-
-    run(['chmod', permissions, out_dir], timeout=5)
-
-    if link['type']:
-        print('    i Type: {}'.format(link['type']))
-
-    if not (link['url'].startswith('http') or link['url'].startswith('ftp')):
-        print('    X Skipping: invalid link.')
-        return
-
-    if FETCH_WGET:
-        fetch_wget(out_dir, link, overwrite=overwrite, requisites=FETCH_WGET_REQUISITES)
-
-    if FETCH_PDF:
-        fetch_pdf(out_dir, link, overwrite=overwrite)
-
-    if FETCH_SCREENSHOT:
-        fetch_screenshot(out_dir, link, overwrite=overwrite, resolution=RESOLUTION)
-
-    if SUBMIT_ARCHIVE_DOT_ORG:
-        archive_dot_org(out_dir, link, overwrite=overwrite)
-
-    if FETCH_AUDIO:
-        fetch_audio(out_dir, link, overwrite=overwrite)
-
-    if FETCH_VIDEO:
-        fetch_video(out_dir, link, overwrite=overwrite)
-
-    if FETCH_FAVICON:
-        fetch_favicon(out_dir, link, overwrite=overwrite)
-
-    dump_link_info(out_dir, link)
@@ -1,30 +1,145 @@
 import os
+import json
+
 from datetime import datetime
+from string import Template

-from config import INDEX_TEMPLATE, INDEX_ROW_TEMPLATE
-from parse import derived_link_info
+from config import (
+    INDEX_TEMPLATE,
+    INDEX_ROW_TEMPLATE,
+    LINK_INDEX_TEMPLATE,
+    ARCHIVE_PERMISSIONS,
+    ARCHIVE_DIR,
+    ANSI,
+    GIT_SHA,
+)
+from util import (
+    chmod_file,
+    html_appended_url,
+    derived_link_info,
+)


-def dump_index(links, service):
-    """create index.html file for a given list of links and service"""
+### Homepage index for all the links
+
+def write_links_index(out_dir, links):
+    """create index.html file for a given list of links"""
+
+    if not os.path.exists(out_dir):
+        os.makedirs(out_dir)
+
+    print('[i] [{}] Updating {}{}{} links in archive index...'.format(
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        ANSI['green'],
+        len(links),
+        ANSI['reset'],
+    ))
+    
+    write_json_links_index(out_dir, links)
+    write_html_links_index(out_dir, links)
+
+def write_json_links_index(out_dir, links):
+    """write the json link index to a given path"""
+
+    path = os.path.join(out_dir, 'index.json')
+
+    index_json = {
+        'info': 'Bookmark Archiver Index',
+        'help': 'https://github.com/pirate/bookmark-archiver',
+        'version': GIT_SHA,
+        'num_links': len(links),
+        'updated': str(datetime.now().timestamp()),
+        'links': links,
+    }
+
+    with open(path, 'w', encoding='utf-8') as f:
+        json.dump(index_json, f, indent=4, default=str)
+
+    chmod_file(path)
+
+def parse_json_links_index(out_dir):
+    """load the index in a given directory and merge it with the given link"""
+    index_path = os.path.join(out_dir, 'index.json')
+    if os.path.exists(index_path):
+        with open(index_path, 'r', encoding='utf-8') as f:
+            return json.load(f)['links']
+
+    return []
+
+def write_html_links_index(out_dir, links):
+    """write the html link index to a given path"""
+
+    path = os.path.join(out_dir, 'index.html')

    with open(INDEX_TEMPLATE, 'r', encoding='utf-8') as f:
        index_html = f.read()

-    # TODO: refactor this out into index_template.html
    with open(INDEX_ROW_TEMPLATE, 'r', encoding='utf-8') as f:
-        link_html = f.read()
+        link_row_html = f.read()

-    article_rows = '\n'.join(
-        link_html.format(**derived_link_info(link)) for link in links
+    link_rows = '\n'.join(
+        Template(link_row_html).substitute(**derived_link_info(link))
+        for link in links
    )

    template_vars = {
        'num_links': len(links),
        'date_updated': datetime.now().strftime('%Y-%m-%d'),
        'time_updated': datetime.now().strftime('%Y-%m-%d %H:%M'),
-        'rows': article_rows,
+        'rows': link_rows,
    }

-    with open(os.path.join(service, 'index.html'), 'w', encoding='utf-8') as f:
-        f.write(index_html.format(**template_vars))
+    with open(path, 'w', encoding='utf-8') as f:
+        f.write(Template(index_html).substitute(**template_vars))
+
+    chmod_file(path)
+
+
+### Individual link index
+
+def write_link_index(out_dir, link):
+    link['updated'] = str(datetime.now().timestamp())
+    write_json_link_index(out_dir, link)
+    write_html_link_index(out_dir, link)
+
+def write_json_link_index(out_dir, link):
+    """write a json file with some info about the link"""
+    
+    path = os.path.join(out_dir, 'index.json')
+
+    print('    √ Updating: index.json')
+
+    with open(path, 'w', encoding='utf-8') as f:
+        json.dump(link, f, indent=4, default=str)
+
+    chmod_file(path)
+
+def parse_json_link_index(out_dir):
+    """load the json link index from a given directory"""
+    existing_index = os.path.join(out_dir, 'index.json')
+    if os.path.exists(existing_index):
+        with open(existing_index, 'r', encoding='utf-8') as f:
+            return json.load(f)
+    return {}
+
+def write_html_link_index(out_dir, link):
+    with open(LINK_INDEX_TEMPLATE, 'r', encoding='utf-8') as f:
+        link_html = f.read()
+
+    path = os.path.join(out_dir, 'index.html')
+
+    print('    √ Updating: index.html')
+
+    with open(path, 'w', encoding='utf-8') as f:
+        f.write(Template(link_html).substitute({
+            **link,
+            **link['latest'],
+            'type': link['type'] or 'website',
+            'tags': link['tags'] or 'untagged',
+            'bookmarked': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
+            'updated': datetime.fromtimestamp(float(link['updated'])).strftime('%Y-%m-%d %H:%M'),
+            'archive_org': link['latest']['archive_org'] or 'https://web.archive.org/save/{}'.format(link['url']),
+            'wget': link['latest']['wget'] or link['domain'],
+        }))
+
+    chmod_file(path)
@@ -0,0 +1,121 @@
+"""
+In Bookmark Archiver, a Link represents a single entry that we track in the 
+json index.  All links pass through all archiver functions and the latest,
+most up-to-date canonical output for each is stored in "latest".
+
+
+Link {
+    timestamp: str,     (how we uniquely id links)        _   _  _ _  ___
+    url: str,                                            | \ / \ |\| ' |
+    base_url: str,                                       |_/ \_/ | |   |
+    domain: str,                                          _   _ _ _ _  _
+    tags: str,                                           |_) /| |\| | / `
+    type: str,                                           |  /"| | | | \_,
+    title: str,                                              ,-'"`-.
+    sources: [str],                                     /// /  @ @  \ \\\\
+    latest: {                                           \ :=| ,._,. |=:  /
+        ...,                                            || ,\ \_../ /. ||
+        pdf: 'output.pdf',                              ||','`-._))'`.`||
+        wget: 'example.com/1234/index.html'             `-'     (/    `-'
+    },
+    history: {
+        ...
+        pdf: [
+            {timestamp: 15444234325, status: 'skipped', result='output.pdf'},
+            ...
+        ],
+        wget: [
+            {timestamp: 11534435345, status: 'succeded', result='donuts.com/eat/them.html'}
+        ]
+    },
+}
+
+"""
+
+from util import (
+    domain,
+    base_url,
+    str_between,
+    get_link_type,
+    merge_links,
+)
+
+
+def validate_links(links):
+    links = archivable_links(links)  # remove chrome://, about:, mailto: etc.
+    links = uniquefied_links(links)  # merge/dedupe duplicate timestamps & urls
+    links = sorted_links(links)      # deterministically sort the links based on timstamp, url
+    
+    if not links:
+        print('[X] No links found :(')
+        raise SystemExit(1)
+
+    return list(links)
+
+
+def archivable_links(links):
+    """remove chrome://, about:// or other schemed links that cant be archived"""
+    return (
+        link
+        for link in links
+        if any(link['url'].startswith(s) for s in ('http://', 'https://', 'ftp://'))
+    )
+
+def uniquefied_links(sorted_links):
+    """
+    ensures that all non-duplicate links have monotonically increasing timestamps
+    """
+
+    unique_urls = {}
+
+    lower = lambda url: url.lower().strip()
+    without_www = lambda url: url.replace('://www.', '://', 1)
+    without_trailing_slash = lambda url: url[:-1] if url[-1] == '/' else url.replace('/?', '?')
+
+    for link in sorted_links:
+        fuzzy_url = without_www(without_trailing_slash(lower(link['url'])))
+        if fuzzy_url in unique_urls:
+            # merge with any other links that share the same url
+            link = merge_links(unique_urls[fuzzy_url], link)
+        unique_urls[fuzzy_url] = link
+
+    unique_timestamps = {}
+    for link in unique_urls.values():
+        link['timestamp'] = lowest_uniq_timestamp(unique_timestamps, link['timestamp'])
+        unique_timestamps[link['timestamp']] = link
+
+    return unique_timestamps.values()
+
+def sorted_links(links):
+    sort_func = lambda link: (link['timestamp'], link['url'])
+    return sorted(links, key=sort_func, reverse=True)
+
+def links_after_timestamp(links, timestamp=None):
+    if not timestamp:
+        yield from links
+        return
+
+    print('[.] [{}] Resuming...'.format(timestamp))
+    for link in links:
+        try:
+            if float(link['timestamp']) <= float(timestamp):
+                yield link
+        except (ValueError, TypeError):
+            print('Resume value and all timestamp values must be valid numbers.')
+
+def lowest_uniq_timestamp(used_timestamps, timestamp):
+    """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2"""
+
+    timestamp = timestamp.split('.')[0]
+    nonce = 0
+
+    # first try 152323423 before 152323423.0
+    if timestamp not in used_timestamps:
+        return timestamp
+
+    new_timestamp = '{}.{}'.format(timestamp, nonce)
+    while new_timestamp in used_timestamps:
+        nonce += 1
+        new_timestamp = '{}.{}'.format(timestamp, nonce)
+
+    return new_timestamp
@@ -1,75 +1,81 @@
+"""
+Everything related to parsing links from bookmark services.
+
+For a list of supported services, see the README.md.
+For examples of supported files see examples/.
+
+Parsed link schema: {
+    'url': 'https://example.com/example/?abc=123&xyc=345#lmnop',
+    'domain': 'example.com',
+    'base_url': 'example.com/example/',
+    'timestamp': '15442123124234',
+    'tags': 'abc,def',
+    'title': 'Example.com Page Title',
+    'sources': ['ril_export.html', 'downloads/getpocket.com.txt'],
+}
+"""
+
 import re
-import time
 import json

 from datetime import datetime

+from util import (
+    domain,
+    base_url,
+    str_between,
+    get_link_type,
+)

-def parse_export(file, service=None):
+
+def get_parsers(file):
+    """return all parsers that work on a given file, defaults to all of them"""
+
+    return {
+        'pocket': parse_pocket_export,
+        'pinboard': parse_json_export,
+        'bookmarks': parse_bookmarks_export,
+        'rss': parse_rss_export,
+    }
+
+def parse_links(path):
    """parse a list of links dictionaries from a bookmark export file"""
+    
+    links = []
+    with open(path, 'r', encoding='utf-8') as file:
+        for parser_func in get_parsers(file).values():
+            # otherwise try all parsers until one works
+            try:
+                links += list(parser_func(file))
+                if links:
+                    break
+            except (ValueError, TypeError, IndexError):
+                # parser not supported on this file
+                pass

-    # if specific service was passed via command line
-    if service == "pocket":
-        links = parse_pocket_export(file)
-    elif service == "pinboard":
-        links = parse_json_export(file)
-    elif service == "bookmarks":
-        links = parse_bookmarks_export(file)
-    else:
-        # otherwise try all parsers until one works
-        try:
-            links = list(parse_json_export(file))
-            service = 'pinboard'
-        except Exception:
-            links = list(parse_pocket_export(file))
-            if links:
-                service = 'pocket'
-            else:
-                links = list(parse_bookmarks_export(file))
-                service = 'bookmarks'
+    return links

-    links = valid_links(links)              # remove chrome://, about:, mailto: etc.
-    links = uniquefied_links(links)         # fix duplicate timestamps, returns sorted list
-    return links, service
-
-
-def get_link_type(link):
-    """Certain types of links need to be handled specially, this figures out when that's the case"""
-
-    if link['base_url'].endswith('.pdf'):
-        return 'PDF'
-    elif link['base_url'].rsplit('.', 1) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
-        return 'image'
-    elif 'wikipedia.org' in link['domain']:
-        return 'wiki'
-    elif 'youtube.com' in link['domain']:
-        return 'youtube'
-    elif 'soundcloud.com' in link['domain']:
-        return 'soundcloud'
-    elif 'youku.com' in link['domain']:
-        return 'youku'
-    elif 'vimeo.com' in link['domain']:
-        return 'vimeo'
-    return None

 def parse_pocket_export(html_file):
    """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""

    html_file.seek(0)
-    pattern = re.compile("^\\s*<li><a href=\"(.+)\" time_added=\"(\\d+)\" tags=\"(.*)\">(.+)</a></li>", re.UNICODE)   # see sample input in ./example_ril_export.html
+    pattern = re.compile("^\\s*<li><a href=\"(.+)\" time_added=\"(\\d+)\" tags=\"(.*)\">(.+)</a></li>", re.UNICODE)
    for line in html_file:
+        # example line
+        # <li><a href="http://example.com/ time_added="1478739709" tags="tag1,tag2">example title</a></li>
        match = pattern.search(line)
        if match:
            fixed_url = match.group(1).replace('http://www.readability.com/read?url=', '')           # remove old readability prefixes to get original url
-            without_scheme = fixed_url.replace('http://', '').replace('https://', '')
+            time = datetime.fromtimestamp(float(match.group(2)))
            info = {
                'url': fixed_url,
-                'domain': without_scheme.split('/')[0],    # without pathname
-                'base_url': without_scheme.split('?')[0],  # without query args
-                'time': datetime.fromtimestamp(int(match.group(2))).strftime('%Y-%m-%d %H:%M'),
-                'timestamp': match.group(2),
+                'domain': domain(fixed_url),
+                'base_url': base_url(fixed_url),
+                'timestamp': str(time.timestamp()),
                'tags': match.group(3),
-                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or without_scheme,
+                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or base_url(fixed_url),
+                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)
            yield info
@@ -80,137 +86,86 @@ def parse_json_export(json_file):
    json_file.seek(0)
    json_content = json.load(json_file)
    for line in json_content:
+        # example line
+        # {"href":"http:\/\/www.reddit.com\/r\/example","description":"title here","extended":"","meta":"18a973f09c9cc0608c116967b64e0419","hash":"910293f019c2f4bb1a749fb937ba58e3","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"reddit android"}]
        if line:
            erg = line
+            time = datetime.strptime(erg['time'].split(',', 1)[0], '%Y-%m-%dT%H:%M:%SZ')
            info = {
                'url': erg['href'],
-                'domain': erg['href'].replace('http://', '').replace('https://', '').split('/')[0],
-                'base_url': erg['href'].replace('https://', '').replace('http://', '').split('?')[0],
-                'time': datetime.fromtimestamp(int(time.mktime(time.strptime(erg['time'].split(',')[0], '%Y-%m-%dT%H:%M:%SZ')))),
-                'timestamp': str(int(time.mktime(time.strptime(erg['time'].split(',')[0], '%Y-%m-%dT%H:%M:%SZ')))),
+                'domain': domain(erg['href']),
+                'base_url': base_url(erg['href']),
+                'timestamp': str(time.timestamp()),
                'tags': erg['tags'],
                'title': erg['description'].replace(' — Readability', ''),
+                'sources': [json_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

+def parse_rss_export(rss_file):
+    """Parse RSS XML-format files into links"""
+
+    rss_file.seek(0)
+    items = rss_file.read().split('</item>\n<item>')
+    for item in items:
+        # example item:
+        # <item>
+        # <title><![CDATA[How JavaScript works: inside the V8 engine]]></title>
+        # <category>Unread</category>
+        # <link>https://blog.sessionstack.com/how-javascript-works-inside</link>
+        # <guid>https://blog.sessionstack.com/how-javascript-works-inside</guid>
+        # <pubDate>Mon, 21 Aug 2017 14:21:58 -0500</pubDate>
+        # </item>
+
+        trailing_removed = item.split('</item>', 1)[0]
+        leading_removed = trailing_removed.split('<item>', 1)[-1]
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r for r in rows if r.startswith('<{}>'.format(key))][0]
+
+        title = str_between(get_row('title'), '<![CDATA[', ']]')
+        url = str_between(get_row('link'), '<link>', '</link>')
+        ts_str = str_between(get_row('pubDate'), '<pubDate>', '</pubDate>')
+        time = datetime.strptime(ts_str, "%a, %d %b %Y %H:%M:%S %z")
+
+        info = {
+            'url': url,
+            'domain': domain(url),
+            'base_url': base_url(url),
+            'timestamp': str(time.timestamp()),
+            'tags': '',
+            'title': title,
+            'sources': [rss_file.name],
+        }
+        info['type'] = get_link_type(info)
+
+        yield info
+
 def parse_bookmarks_export(html_file):
    """Parse netscape-format bookmarks export files (produced by all browsers)"""

    html_file.seek(0)
    pattern = re.compile("<a href=\"(.+?)\" add_date=\"(\\d+)\"[^>]*>(.+)</a>", re.UNICODE | re.IGNORECASE)
    for line in html_file:
+        # example line
+        # <DT><A HREF="https://example.com/?q=1+2" ADD_DATE="1497562974" LAST_MODIFIED="1497562974" ICON_URI="https://example.com/favicon.ico" ICON="data:image/png;base64,...">example bookmark title</A>
+        
        match = pattern.search(line)
        if match:
            url = match.group(1)
-            secs = match.group(2)
-            dt = datetime.fromtimestamp(int(secs))
+            time = datetime.fromtimestamp(float(match.group(2)))

            info = {
                'url': url,
-                'domain': url.replace('http://', '').replace('https://', '').split('/')[0],
-                'base_url': url.replace('https://', '').replace('http://', '').split('?')[0],
-                'time': dt,
-                'timestamp': secs,
+                'domain': domain(url),
+                'base_url': base_url(url),
+                'timestamp': str(time.timestamp()),
                'tags': "",
                'title': match.group(3),
+                'sources': [html_file.name],
            }
-
            info['type'] = get_link_type(info)
+
            yield info
-
-
-def next_uniq_timestamp(used_timestamps, timestamp):
-    """resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2"""
-
-    if timestamp not in used_timestamps:
-        return timestamp
-
-    if '.' in timestamp:
-        timestamp, nonce = timestamp.split('.')
-        nonce = int(nonce)
-    else:
-        nonce = 1
-
-    new_timestamp = '{}.{}'.format(timestamp, nonce)
-
-    while new_timestamp in used_timestamps:
-        nonce += 1
-        new_timestamp = '{}.{}'.format(timestamp, nonce)
-
-    return new_timestamp
-
-def uniquefied_links(links):
-    """uniqueify link timestamps by de-duping using url, returns links sorted most recent -> oldest
-
-    needed because firefox will produce exports where many links share the same timestamp, this func
-    ensures that all non-duplicate links have monotonically increasing timestamps
-    """
-
-    links = list(reversed(sorted(links, key=lambda l: (l['timestamp'], l['url']))))
-    seen_timestamps = {}
-
-    for link in links:
-        t = link['timestamp']
-        if t in seen_timestamps:
-            if link['url'] == seen_timestamps[t]['url']:
-                # don't create new unique timestamp if link is the same
-                continue
-            else:
-                # resolve duplicate timstamp by appending a decimal
-                link['timestamp'] = next_uniq_timestamp(seen_timestamps, link['timestamp'])
-        seen_timestamps[link['timestamp']] = link
-
-    return links
-
-def valid_links(links):
-    """remove chrome://, about:// or other schemed links that cant be archived"""
-    return (link for link in links if link['url'].startswith('http') or link['url'].startswith('ftp'))
-
-
-def html_appended_url(link):
-    """calculate the path to the wgetted .html file, since wget may
-    adjust some paths to be different than the base_url path.
-
-    See docs on wget --adjust-extension."""
-
-    split_url = link['url'].split('#', 1)
-
-    if re.search(".+\\.[Hh][Tt][Mm][Ll]?$", split_url[0], re.I | re.M):
-        # already ends in .html
-        return link['base_url']
-    else:
-        # .html needs to be appended
-        without_scheme = split_url[0].split('://', 1)[-1]
-        if without_scheme.endswith('/'):
-            return '#'.join([without_scheme + 'index.html', *split_url[1:]])
-        return '#'.join([without_scheme + '.html', *split_url[1:]])
-
-
-def derived_link_info(link):
-    """extend link info with the archive urls and other derived data"""
-
-    link_info = {
-        **link,
-        'files_url': 'archive/{timestamp}/'.format(**link),
-        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**link),
-        'favicon_url': 'archive/{timestamp}/favicon.ico'.format(**link),
-        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
-    }
-
-    # PDF and images are handled slightly differently
-    # wget, screenshot, & pdf urls all point to the same file
-    if link['type'] in ('PDF', 'image'):
-        link_info.update({
-            'archive_url': 'archive/{timestamp}/{base_url}'.format(**link),
-            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**link),
-            'title': '{title} ({type})'.format(**link),
-        })
-    else:
-        link_info.update({
-            'archive_url': 'archive/{}/{}'.format(link['timestamp'], html_appended_url(link)),
-            'pdf_link': 'archive/{timestamp}/output.pdf'.format(**link),
-            'screenshot_link': 'archive/{timestamp}/screenshot.png'.format(**link)
-        })
-    return link_info
@@ -28,6 +28,7 @@ if which apt-get > /dev/null; then
        sudo ln -s "$(which google-chrome)" /usr/bin/chromium-browser
    elif which chromium-browser; then
        echo "[i] chromium-browser already installed, using existing installation."
+        apt upgrade chromium-browser -y
    else
        echo "[+] Installing chromium-browser..."
        apt install chromium-browser -y
@@ -43,6 +44,7 @@ elif which brew > /dev/null; then   # 🐍 eye of newt
        chmod +x /usr/local/bin/chromium-browser

    elif which chromium-browser; then
+        brew cask upgrade chromium-browser
        echo "[+] Linking /usr/local/bin/chromium-browser -> /Applications/Chromium.app"
        echo -e '#!/bin/bash\n/Applications/Chromium.app/Contents/MacOS/Chromium "$@"' > /usr/local/bin/chromium-browser
        chmod +x /usr/local/bin/chromium-browser
@@ -3,7 +3,7 @@
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
        <title>Archived Sites</title>
        <style>
-            html, body {{
+            html, body {
                width: 100%;
                height: 100%;
                font-size: 20px;
@@ -12,71 +12,71 @@
                margin: 0px;
                padding: 0px;
                font-family: "Gill Sans", Helvetica, sans-serif;
-            }}
-            header {{
+            }
+            header {
                background-color: #aa1e55;
                color: white;
                padding: 10px;
-            }}
-            header h1 {{
+            }
+            header h1 {
                font-weight: 300;
                color: black;
                margin-top: 10px;
                margin-bottom: 12px;
-            }}
-            header h1 small {{
+            }
+            header h1 small {
                color: white;
                font-size:0.5em;
-            }}
-            header h1 small a {{
+            }
+            header h1 small a {
                text-decoration: none;
                color: orange;
                opacity: 0.6
                font-weight: 300;
-            }}
-            header h1 small a:hover {{
+            }
+            header h1 small a:hover {
                opacity: 1;
-            }}
-            table {{
+            }
+            table {
                padding: 6px;
                width: 100%;
-            }}
-            table thead th {{
+            }
+            table thead th {
                font-weight: 400;
-            }}
-            tbody tr:nth-child(odd) {{
+            }
+            tbody tr:nth-child(odd) {
               background-color: #ffebeb;
-            }}
-            table tr td {{
+            }
+            table tr td {
                white-space: nowrap;
                overflow: hidden;
                padding-bottom: 0.4em;
                padding-top: 0.4em;
                padding-left: 2px;
-            }}
-            table tr td img {{
+            }
+            table tr td img {
                height: 24px;
                padding: 0px;
                padding-right: 5px;
                text-indent: -10000px;
-            }}
+            }
        </style>
    </head>
    <body>
        <header>
-            <h1 title="Last modified {time_updated}">
+            <h1 title="Last modified $time_updated">
                <img src="https://nicksweeting.com/images/archive.png" height="36px">
                Archived Sites <img src="https://getpocket.com/favicon.ico" height="36px"> <br/>
                <small>
-                    Archived with: <a href="https://github.com/pirate/bookmark-archiver">Bookmark Archiver</a> on {date_updated}
+                    <a href="https://github.com/pirate/bookmark-archiver">Bookmark Archiver</a>
                </small>
            </h1>
        </header>
        <table style="width:100%;height: 90%; overflow-y: scroll;table-layout: fixed">
            <thead>
                <tr>
-                    <th style="width: 120px;"><img src="https://getpocket.com/favicon.ico" height="12px"> Starred</th>
-                    <th style="width: 45vw;">Saved Articles ({num_links})</th>
+                    <th style="width: 120px;"><img src="https://getpocket.com/favicon.ico" height="12px"> Bookmarked</th>
+                    <th style="width: 45vw;">Saved Articles ($num_links)</th>
                    <th style="width: 50px">Files</th>
                    <th style="width: 50px">PDF</th>
                    <th style="width: 60px;font-size:0.8em;">Screenshot</th>
@@ -84,7 +84,7 @@
                    <th style="width: 100px;whitespace:nowrap;overflow-x:scroll;display:block">Original URL</th>
                </tr>
            </thead>
-            <tbody>{rows}</tbody>
+            <tbody>$rows</tbody>
        </table>
    </body>
 </html>
@@ -1,12 +1,12 @@
 <tr>
-    <td>{time}</td>
-    <td><a href="{archive_url}" style="font-size:1.4em;text-decoration:none;color:black;" title="{title}">
-        <img src="{favicon_url}">
-        {title} <small style="background-color: #eee;border-radius:4px; float:right">{tags}</small>
+    <td title="Bookmarked timestamp: $timestamp">$date</td>
+    <td><a href="$archive_url" style="font-size:1.4em;text-decoration:none;color:black;" title="$title">
+        <img src="$favicon_url">
+        $title <small style="background-color: #eee;border-radius:4px; float:right">$tags</small>
    </td>
-    <td style="text-align:center"><a href="{files_url}" title="Files">📂</a></td>
-    <td style="text-align:center"><a href="{pdf_link}" title="PDF">📄</a></td>
-    <td style="text-align:center"><a href="{screenshot_link}" title="Screenshot">🖼</a></td>
-    <td style="text-align:center"><a href="{archive_org_url}" title="Archive.org">🏛</a></td>
-    <td>🔗 <img src="{google_favicon_url}" height="16px"> <a href="{url}">{url}</a></td>
-</tr
+    <td style="text-align:center"><a href="$files_url" title="Files">📂</a></td>
+    <td style="text-align:center"><a href="$pdf_link" title="PDF">📄</a></td>
+    <td style="text-align:center"><a href="$screenshot_link" title="Screenshot">🖼</a></td>
+    <td style="text-align:center"><a href="$archive_org_url" title="Archive.org">🏛</a></td>
+    <td>🔗 <img src="$google_favicon_url" height="16px"> <a href="$url">$url</a></td>
+</tr>
@@ -0,0 +1,59 @@
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>$title</title>
+        
+    </head>
+    <body>
+        <header>
+            <h1>
+                <img src="$favicon" height="20px"> $title<br/>
+                <a href="$url" class="title-url">
+                    <small>$base_url</small>
+                </a>
+            </h1>
+        </header>
+        <hr/>
+        <div>
+            Tags: $tags<br/>
+            Type: $type<br/>
+            <br/>
+            Bookmarked:<br/>
+                $bookmarked<br/>
+            Archived:<br/>
+                $updated<br/>
+        </div>
+        <hr/>
+        <ul>
+            <li>
+                <a href="$url"><b>Original</b></a><br/>
+                $base_url<br/>&nbsp;
+            </li>
+            <li>
+                <a href="$wget"><b>Local Archive</b></a><br/>
+                archive/$timestamp/$domain<br/>&nbsp;
+            </li>
+            <li>
+                <a href="$pdf" id="pdf-btn"><b>PDF</b></a><br/>
+                archive/$timestamp/output.pdf<br/>&nbsp;
+            </li>
+            <li>
+                <a href="$screenshot"><b>Screenshot</b></a><br/>
+                archive/$timestamp/screenshot.png<br/>&nbsp;
+            </li>
+            <li>
+                <a href="$archive_org"><b>Archive.Org</b></a><br/>
+                web.archive.org/web/$base_url<br/>&nbsp;
+            </li>
+        </ul>
+        <footer>
+            <hr/>
+            <a href="index.json">JSON</a> | <a href=".">Files</a>
+            <hr/>
+            <a href="./../../index.html" class="nav-icon" title="Archived Sites">
+                <img src="https://nicksweeting.com/images/archive.png" alt="Archive Icon" height="20px">
+                Bookmark Archiver: Link Index
+            </a>
+        </footer>
+    </body>
+</html>
@@ -0,0 +1,268 @@
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>$title</title>
+        <style>
+            html, body {
+                width: 100%;
+                height: 100%;
+            }
+            body {
+                background-color: #ddd;
+            }
+            header {
+                width: 100%;
+                background-color: #aa1e55;
+                margin: 0px;
+                text-align: center;
+                color: white;
+            }
+            header h1 {
+                padding-top: 5px;
+                padding-bottom: 5px;
+                margin: 0px;
+                font-weight: 200;
+                font-family: "Gill Sans", Helvetica, sans-serif;
+            }
+            .collapse-icon {
+                float: right;
+                color: black;
+                width: 126px;
+                font-size: 0.7em;
+                margin-top: 20px;
+                margin-right: -30px;
+                margin-left: -150px;
+            }
+            .nav-icon img {
+                display: inline-block;
+                margin-right: -200px;
+                float: left;
+                color: black;
+                height: 53px;
+                margin-top: 7px;
+                margin-left: 10px;
+            }
+            .nav-icon img:hover {
+                opacity: 0.5;
+            }
+            .title-url {
+                color: black;
+            }
+            .archive-page-header {
+                margin-top: 5px;
+                margin-bottom: 5px;
+            }
+            h1 small {
+                opacity: 0.4;
+                font-size: 0.6em;
+            }
+            h1 small:hover {
+                opacity: 0.8;
+            }
+            .card {
+                box-shadow: 2px 3px 14px 0px rgba(0,0,0,0.02);
+            }
+            .card h4 {
+                font-size: 1.4vw;
+            }
+            .card-body {
+                font-size: 1vw;
+                padding-top: 2vw;
+                padding-left: 1vw;
+                padding-right: 1vw;
+                padding-bottom: 5vw;
+                word-wrap: break-word;
+                max-height: 102px;
+                overflow: hidden;
+            }
+            .card-img-top {
+                border: 0px;
+                padding: 0px;
+                margin: 0px;
+                overflow: hidden;
+                opacity: 0.8;
+                border-top: 1px solid gray;
+                border-radius: 3px;
+                border-bottom: 1px solid #ddd;
+                height: 425px;
+                width: 400%;
+                margin-bottom: -330px;
+
+                transform: scale(0.25); 
+                transform-origin: 0 0;
+            }
+            .full-page-iframe {
+                border-top: 1px solid #ddd;
+                width: 100%;
+                height: 69vh;
+                margin: 0px;
+                border: 0px;
+                border-top: 3px solid #aa1e55;
+            }
+            .card.selected-card {
+                border: 2px solid orange;
+                box-shadow: 0px -6px 13px 1px rgba(0,0,0,0.05);
+            }
+            .iframe-large {
+                height: 93%;
+                margin-top: -10px;
+            }
+
+            @media(max-width: 1092px) {
+                iframe {
+                    display: none;
+                }
+            }
+                
+
+            @media(max-width: 728px) {
+                .card h4 {
+                    font-size: 5vw;
+                }
+                .card-body {
+                    font-size: 4vw;
+                }
+                .card {
+                    margin-bottom: 5px;
+                }
+                header > h1 > a.collapse-icon, header > h1 > a.nav-icon {
+                    display: none;
+                }
+            }
+        </style>
+        <script
+          src="https://code.jquery.com/jquery-3.2.1.slim.min.js"
+          integrity="sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
+          crossorigin="anonymous"></script>
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/css/bootstrap.min.css" integrity="sha384-rwoIResjU2yc3z8GV/NPeZWAv56rSmLldC3R/AZzGRnGxQQKnKkoFVhFQhNUwEyJ" crossorigin="anonymous">
+        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/js/bootstrap.min.js" integrity="sha384-vBWWzlZJ8ea9aCX4pEW3rVHjgjt7zpkNpZk+02D9phzyeVkE+jo0ieGizqPLForn" crossorigin="anonymous"></script>
+    </head>
+    <body>
+        <header>
+            <h1 class="page-title">
+                <a href="#" class="collapse-icon" title="Collapse Navbar">
+                    [-]
+                </a>
+                <a href="./../../index.html" class="nav-icon" title="Archived Sites">
+                    <img src="https://nicksweeting.com/images/archive.png" alt="Archive Icon">
+                </a>
+                $title<br/>
+                <a href="$url" class="title-url">
+                    <small><img src="$favicon" height="20px"> $base_url</small>
+                </a>
+            </h1>
+        </header>
+        <div class="site-header container-fluid">
+            <div class="row archive-page-header">
+                <div class="col-lg-2">
+                    <div class="alert alert-warning">
+                        Tags:
+                        <span class="badge badge-success">$tags</span> 
+                        <span class="badge badge-default">$type</span>
+                        <div style="height:5px"></div>
+                        Bookmarked:<br/>
+                            <small>$bookmarked<br/></small>
+                        Archived:<br/>
+                            <small>$updated</small>
+                        <hr/>
+                        <a href="index.json">JSON</a> | <a href=".">Files</a>
+                    </div>
+                </div>
+                <div class="col-lg-2">
+                    <div class="card selected-card">
+                      <iframe class="card-img-top" src="$wget" sandbox="allow-same-origin allow-scripts allow-forms"></iframe>
+                      <div class="card-body">
+                        <a href="$wget" style="float:right"><small>➡️</small></a>
+                        <a href="$wget" target="preview"><h4 class="card-title">Local Archive</h4></a>
+                        <p class="card-text">archive/$domain</p>
+                      </div>
+                    </div>
+                </div>
+                <div class="col-lg-2">
+                    <div class="card">
+                      <iframe class="card-img-top" src="$pdf"></iframe>
+                      <div class="card-body">
+                        <a href="$pdf" style="float:right"><small>➡️</small></a>
+                        <a href="$pdf" target="preview" id="pdf-btn"><h4 class="card-title">PDF</h4></a>
+                        <p class="card-text">archive/output.pdf</p>
+                      </div>
+                    </div>
+                </div>
+                <div class="col-lg-2">
+                    <div class="card">
+                      <iframe class="card-img-top" src="$screenshot" sandbox="allow-same-origin allow-scripts allow-forms"></iframe>
+                      <div class="card-body">
+                        <a href="$screenshot" style="float:right"><small>➡️</small></a>
+                        <a href="$screenshot" target="preview"><h4 class="card-title">Screenshot</h4></a>
+                        <p class="card-text">archive/screenshot.png</p>
+                      </div>
+                    </div>
+                </div>
+                <div class="col-lg-2">
+                    <div class="card">
+                      <iframe class="card-img-top" src="$url" sandbox="allow-same-origin allow-scripts allow-forms"></iframe>
+                      <div class="card-body">
+                        <a href="$url" style="float:right"><small>➡️</small></a>
+                        <a href="$url" target="preview"><h4 class="card-title">Original</h4></a>
+                        <p class="card-text">$domain</p>
+                      </div>
+                    </div>
+                </div>
+                <div class="col-lg-2">
+                    <div class="card">
+                      <iframe class="card-img-top" src="$archive_org" sandbox="allow-same-origin allow-scripts allow-forms"></iframe>
+                      <div class="card-body">
+                        <a href="$archive_org" style="float:right"><small>➡️</small></a>
+                        <a href="$archive_org" target="preview"><h4 class="card-title">Archive.Org</h4></a>
+                        <p class="card-text">web.archive.org/web/...</p>
+                      </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+        <iframe sandbox="allow-same-origin allow-scripts allow-forms" class="full-page-iframe" src="$wget" name="preview"></iframe>
+    </body>
+
+    <script>
+        // show selected file in iframe when preview card is clicked
+        jQuery('.card').on('click', function(e) {
+            jQuery('.selected-card').removeClass('selected-card')
+            jQuery(e.target).closest('.card').addClass('selected-card')
+        })
+        jQuery('.card a[target=preview]').on('click', function(e) {
+            if (e.currentTarget.href.endsWith('.pdf')) {
+                jQuery('.full-page-iframe')[0].removeAttribute('sandbox')
+            } else {
+                jQuery('.full-page-iframe')[0].sandbox = "allow-same-origin allow-scripts allow-forms"
+            }
+            return true
+        })
+
+        // un-sandbox iframes showing pdfs (required to display pdf viewer)
+        jQuery('iframe').map(function() {
+            if (this.src.endsWith('.pdf')) {
+                this.removeAttribute('sandbox')
+                this.src = this.src
+            }
+        })
+
+        // hide header when collapse icon is clicked
+        jQuery('.collapse-icon').on('click', function() {
+            if (jQuery('.collapse-icon').text().includes('[-]')) {
+                jQuery('.collapse-icon').text('[+]')
+                jQuery('.site-header').hide()
+                jQuery('.full-page-iframe').addClass('iframe-large')
+            } else {
+                jQuery('.collapse-icon').text('[-]')
+                jQuery('.site-header').show()
+                jQuery('.full-page-iframe').removeClass('iframe-large')
+            }
+            return true
+        })
+
+        // hide all preview iframes on small screens
+        if (window.innerWidth < 1091) {
+            jQuery('.card a[target=preview]').attr('target', '_self')
+        }
+    </script>
+</html>
@@ -0,0 +1,448 @@
+import os
+import re
+import sys
+import time
+import json
+import requests
+
+from datetime import datetime
+from subprocess import run, PIPE, DEVNULL
+from multiprocessing import Process
+
+from config import (
+    IS_TTY,
+    ARCHIVE_PERMISSIONS,
+    ARCHIVE_DIR,
+    TIMEOUT,
+    TERM_WIDTH,
+    SHOW_PROGRESS,
+    ANSI,
+    CHROME_BINARY,
+    FETCH_WGET,
+    FETCH_PDF,
+    FETCH_SCREENSHOT,
+    FETCH_FAVICON,
+    FETCH_AUDIO,
+    FETCH_VIDEO,
+    SUBMIT_ARCHIVE_DOT_ORG,
+)
+
+# URL helpers
+without_scheme = lambda url: url.replace('http://', '').replace('https://', '').replace('ftp://', '')
+without_query = lambda url: url.split('?', 1)[0]
+without_hash = lambda url: url.split('#', 1)[0]
+without_path = lambda url: url.split('/', 1)[0]
+domain = lambda url: without_hash(without_query(without_path(without_scheme(url))))
+base_url = lambda url: without_query(without_scheme(url))
+
+short_ts = lambda ts: ts.split('.')[0]
+
+
+def check_dependencies():
+    """Check that all necessary dependencies are installed, and have valid versions"""
+
+    print('[*] Checking Dependencies:')
+
+    python_vers = float('{}.{}'.format(sys.version_info.major, sys.version_info.minor))
+    if python_vers < 3.5:
+        print('{}[X] Python version is not new enough: {} (>3.5 is required){}'.format(ANSI['red'], python_vers, ANSI['reset']))
+        print('    See https://github.com/pirate/bookmark-archiver#troubleshooting for help upgrading your Python installation.')
+        raise SystemExit(1)
+
+    if FETCH_PDF or FETCH_SCREENSHOT:
+        if run(['which', CHROME_BINARY]).returncode:
+            print('{}[X] Missing dependency: {}{}'.format(ANSI['red'], CHROME_BINARY, ANSI['reset']))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CHROME_BINARY))
+            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            raise SystemExit(1)
+
+        # parse chrome --version e.g. Google Chrome 61.0.3114.0 canary / Chromium 59.0.3029.110 built on Ubuntu, running on Ubuntu 16.04
+        try:
+            result = run([CHROME_BINARY, '--version'], stdout=PIPE)
+            version = result.stdout.decode('utf-8').replace('Google Chrome ', '').replace('Chromium ', '').split(' ', 1)[0].split('.', 1)[0]  # TODO: regex might be better
+            if int(version) < 59:
+                print('{red}[X] Chrome version must be 59 or greater for headless PDF and screenshot saving{reset}'.format(**ANSI))
+                print('    See https://github.com/pirate/bookmark-archiver for help.')
+                raise SystemExit(1)
+        except (TypeError, OSError):
+            print('{red}[X] Failed to parse Chrome version, is it installed properly?{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format(CHROME_BINARY))
+            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            raise SystemExit(1)
+
+    if FETCH_WGET:
+        if run(['which', 'wget']).returncode or run(['wget', '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: wget{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('wget'))
+            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            raise SystemExit(1)
+
+    if FETCH_FAVICON or SUBMIT_ARCHIVE_DOT_ORG:
+        if run(['which', 'curl']).returncode or run(['curl', '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: curl{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('curl'))
+            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            raise SystemExit(1)
+
+    if FETCH_AUDIO or FETCH_VIDEO:
+        if run(['which', 'youtube-dl']).returncode or run(['youtube-dl', '--version'], stdout=DEVNULL).returncode:
+            print('{red}[X] Missing dependency: youtube-dl{reset}'.format(**ANSI))
+            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('youtube-dl'))
+            print('    See https://github.com/pirate/bookmark-archiver for help.')
+            raise SystemExit(1)
+
+
+def chmod_file(path, cwd='.', permissions=ARCHIVE_PERMISSIONS, timeout=30):
+    """chmod -R <permissions> <cwd>/<path>"""
+
+    if not os.path.exists(os.path.join(cwd, path)):
+        raise Exception('Failed to chmod: {} does not exist (did the previous step fail?)'.format(path))
+
+    chmod_result = run(['chmod', '-R', permissions, path], cwd=cwd, stdout=DEVNULL, stderr=PIPE, timeout=timeout)
+    if chmod_result.returncode == 1:
+        print('     ', chmod_result.stderr.decode())
+        raise Exception('Failed to chmod {}/{}'.format(cwd, path))
+
+
+def progress(seconds=TIMEOUT, prefix=''):
+    """Show a (subprocess-controlled) progress bar with a <seconds> timeout,
+       returns end() function to instantly finish the progress
+    """
+
+    if not SHOW_PROGRESS:
+        return lambda: None
+
+    chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
+    chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
+
+    def progress_bar(seconds=seconds, prefix=prefix):
+        """show timer in the form of progress bar, with percentage and seconds remaining"""
+        try:
+            for s in range(seconds * chunks):
+                progress = s / chunks / seconds * 100
+                bar_width = round(progress/(100/chunks))
+
+                # ████████████████████           0.9% (1/60sec)
+                sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)'.format(
+                    prefix,
+                    ANSI['green'],
+                    (chunk * bar_width).ljust(chunks),
+                    ANSI['reset'],
+                    round(progress, 1),
+                    round(s/chunks),
+                    seconds,
+                ))
+                sys.stdout.flush()
+                time.sleep(1 / chunks)
+
+            # ██████████████████████████████████ 100.0% (60/60sec)
+            sys.stdout.write('\r{0}{1}{2}{3} {4}% ({5}/{6}sec)\n'.format(
+                prefix,
+                ANSI['red'],
+                chunk * chunks,
+                ANSI['reset'],
+                100.0,
+                seconds,
+                seconds,
+            ))
+            sys.stdout.flush()
+        except KeyboardInterrupt:
+            print()
+            pass
+
+    p = Process(target=progress_bar)
+    p.start()
+
+    def end():
+        """immediately finish progress and clear the progressbar line"""
+        p.terminate()
+        sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH), ANSI['reset']))  # clear whole terminal line
+        sys.stdout.flush()
+
+    return end
+
+
+def download_url(url):
+    """download a given url's content into downloads/domain.txt"""
+
+    download_dir = os.path.join(ARCHIVE_DIR, 'downloads')
+
+    if not os.path.exists(download_dir):
+        os.makedirs(download_dir)
+
+    url_domain = url.split('/', 3)[2]
+    output_path = os.path.join(download_dir, '{}.txt'.format(url_domain))
+    
+    print('[*] [{}] Downloading {} > {}'.format(
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        url,
+        output_path,
+    ))
+    end = progress(TIMEOUT, prefix='      ')
+    try:
+        downloaded_xml = requests.get(url).content.decode()
+        end()
+    except Exception as e:
+        end()
+        print('[!] Failed to download {}\n'.format(url))
+        print('    ', e)
+        raise SystemExit(1)
+
+    with open(output_path, 'w', encoding='utf-8') as f:
+        f.write(downloaded_xml)
+
+    return output_path
+
+def str_between(string, start, end=None):
+    """(<abc>12345</def>, <abc>, </def>)  ->  12345"""
+
+    content = string.split(start, 1)[-1]
+    if end is not None:
+        content = content.rsplit(end, 1)[0]
+
+    return content
+
+def get_link_type(link):
+    """Certain types of links need to be handled specially, this figures out when that's the case"""
+
+    if link['base_url'].endswith('.pdf'):
+        return 'PDF'
+    elif link['base_url'].rsplit('.', 1) in ('pdf', 'png', 'jpg', 'jpeg', 'svg', 'bmp', 'gif', 'tiff', 'webp'):
+        return 'image'
+    elif 'wikipedia.org' in link['domain']:
+        return 'wiki'
+    elif 'youtube.com' in link['domain']:
+        return 'youtube'
+    elif 'soundcloud.com' in link['domain']:
+        return 'soundcloud'
+    elif 'youku.com' in link['domain']:
+        return 'youku'
+    elif 'vimeo.com' in link['domain']:
+        return 'vimeo'
+    return None
+
+def merge_links(a, b):
+    """deterministially merge two links, favoring longer field values over shorter,
+    and "cleaner" values over worse ones.
+    """
+    longer = lambda key: a[key] if len(a[key]) > len(b[key]) else b[key]
+    earlier = lambda key: a[key] if a[key] < b[key] else b[key]
+    
+    url = longer('url')
+    longest_title = longer('title')
+    cleanest_title = a['title'] if '://' not in a['title'] else b['title']
+    link = {
+        'timestamp': earlier('timestamp'),
+        'url': url,
+        'domain': domain(url),
+        'base_url': base_url(url),
+        'tags': longer('tags'),
+        'title': longest_title if '://' not in longest_title else cleanest_title,
+        'sources': list(set(a.get('sources', []) + b.get('sources', []))),
+    }
+    link['type'] = get_link_type(link)
+    return link
+
+def find_link(folder, links):
+    """for a given archive folder, find the corresponding link object in links"""
+    url = parse_url(folder)
+    if url:
+        for link in links:
+            if (link['base_url'] in url) or (url in link['url']):
+                return link
+
+    timestamp = folder.split('.')[0]
+    for link in links:
+        if link['timestamp'].startswith(timestamp):
+            if link['domain'] in os.listdir('./html/archive/' + folder):
+                return link      # careful now, this isn't safe for most ppl
+            if link['domain'] in parse_url(folder):
+                return link
+    return None
+
+
+def parse_url(folder):
+    """for a given archive folder, figure out what url it's for"""
+    link_json = os.path.join('./html/archive/' + folder, 'index.json')
+    if os.path.exists(link_json):
+        with open(link_json, 'r') as f:
+            try:
+                link_json = f.read().strip()
+                if link_json:
+                    link = json.loads(link_json)
+                    return link['base_url']
+            except ValueError:
+                print('File contains invalid JSON: {}!'.format(link_json))
+
+    archive_org_txt = os.path.join('./html/archive/' + folder, 'archive.org.txt')
+    if os.path.exists(archive_org_txt):
+        with open(archive_org_txt, 'r') as f:
+            original_link = f.read().strip().split('/http', 1)[-1]
+            with_scheme = 'http{}'.format(original_link)
+            return with_scheme
+
+    return ''
+
+def manually_merge_folders(source, target):
+    """prompt for user input to resolve a conflict between two archive folders"""
+
+    if not IS_TTY:
+        return
+
+    fname = lambda path: path.split('/')[-1]
+
+    print('    {} and {} have conflicting files, which do you want to keep?'.format(fname(source), fname(target)))
+    print('      - [enter]: do nothing (keep both)')
+    print('      - a:       keep everything from {}'.format(source))
+    print('      - b:       keep everything from {}'.format(target))
+    print('      - q:       quit and resolve the conflict manually')
+    try:
+        answer = input('> ').strip().lower()
+    except KeyboardInterrupt:
+        answer = 'q'
+
+    assert answer in ('', 'a', 'b', 'q'), 'Invalid choice.'
+
+    if answer == 'q':
+        print('\nJust run Bookmark Archiver again to pick up where you left off.')
+        raise SystemExit(0)
+    elif answer == '':
+        return
+
+    files_in_source = set(os.listdir(source))
+    files_in_target = set(os.listdir(target))
+    for file in files_in_source.intersection(files_in_target):
+        if file in files_in_target:
+            to_delete = target if answer == 'a' else source
+            run(['rm', '-Rf', os.path.join(to_delete, file)])
+        run(['mv', os.path.join(source, file), os.path.join(target, file)])
+
+    if not set(os.listdir(source)):
+        run(['rm', '-Rf', source])
+
+def merge_folders(path, folder, link):
+    """given a folder, merge it to the canonical 'correct' path for the given link object"""
+    source, target = os.path.join(path, folder), os.path.join(path, link['timestamp'])
+
+    base_url = parse_url(source)
+    if not (base_url in link['base_url']
+            or link['base_url'] in base_url):
+        raise ValueError('The link does not match the url for this folder.')
+
+    if not os.path.exists(target):
+        # target doesn't exist so nothing needs merging, simply move A to B
+        if run(['mv', source, target]).returncode:
+            print('Failed to move {} to {}!'.format(source, target))
+            return False
+    else:
+        # target folder exists, check for conflicting files and attempt manual merge
+        files_in_source = set(os.listdir(source))
+        files_in_target = set(os.listdir(target))
+
+        if not files_in_source.intersection(files_in_target):
+            # no conflicts, move everything from A to B
+            for file in files_in_source:
+                run(['mv', os.path.join(source, file), os.path.join(target, file)])
+
+    if os.path.exists(source):
+        files_in_source = set(os.listdir(source))
+        if files_in_source:
+            manually_merge_folders(source, target)
+        else:
+            run(['rm', '-R', source])
+
+
+def cleanup_archive(path, links):
+    """move any incorrectly named folders to their canonical locations"""
+    
+    # for each folder that exists, see if we can match it up with a known good link
+    # if we can, then merge the two folders, if not, move it to lost & found
+
+    # for each timestamp, find similar timestamped folders
+    # check each folder for a "domain.com" folder or 
+
+    unmatched = []
+    bad_folders = []
+
+    if not os.path.exists(path):
+        return
+
+    for folder in os.listdir(path):
+        if not os.listdir(os.path.join(path, folder)):
+            # delete empty folders
+            run(['rm', '-R', os.path.join(path, folder)])
+        else:
+            link = find_link(folder, links)
+            if link is None:
+                unmatched.append(folder)
+                continue
+            
+            if folder != link['timestamp']:
+                bad_folders.append((folder, link))
+    
+    if bad_folders:
+        print('[!] Fixing {} improperly named folders in archive...'.format(len(bad_folders)))
+        for folder, link in bad_folders:
+            merge_folders(path, folder, link)
+
+    if unmatched:
+        print('[!] Warning! {} unrecognized folders in html/archive/'.format(len(unmatched)))
+        print('    '+ '\n    '.join(unmatched))
+
+
+def html_appended_url(link):
+    """calculate the path to the wgetted .html file, since wget may
+    adjust some paths to be different than the base_url path.
+
+    See docs on wget --adjust-extension.
+    """
+
+    if link['type'] in ('PDF', 'image'):
+        return link['base_url']
+
+    split_url = link['url'].split('#', 1)
+    query = ('%3F' + link['url'].split('?', 1)[-1]) if '?' in link['url'] else ''
+
+    if re.search(".+\\.[Hh][Tt][Mm][Ll]?$", split_url[0], re.I | re.M):
+        # already ends in .html
+        return link['base_url']
+    else:
+        # .html needs to be appended
+        without_scheme = split_url[0].split('://', 1)[-1].split('?', 1)[0]
+        if without_scheme.endswith('/'):
+            if query:
+                return '#'.join([without_scheme + 'index.html' + query + '.html', *split_url[1:]])
+            return '#'.join([without_scheme + 'index.html', *split_url[1:]])
+        else:
+            if query:
+                return '#'.join([without_scheme + '/index.html' + query + '.html', *split_url[1:]])
+            elif '/' in without_scheme:
+                return '#'.join([without_scheme + '.html', *split_url[1:]])
+            return link['base_url'] + '/index.html'
+
+
+def derived_link_info(link):
+    """extend link info with the archive urls and other derived data"""
+
+    link_info = {
+        **link,
+        'date': datetime.fromtimestamp(float(link['timestamp'])).strftime('%Y-%m-%d %H:%M'),
+        'google_favicon_url': 'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
+        'favicon_url': './archive/{timestamp}/favicon.ico'.format(**link),
+        'files_url': './archive/{timestamp}/index.html'.format(**link),
+        'archive_url': './archive/{}/{}'.format(link['timestamp'], html_appended_url(link)),
+        'pdf_link': './archive/{timestamp}/output.pdf'.format(**link),
+        'screenshot_link': './archive/{timestamp}/screenshot.png'.format(**link),
+        'archive_org_url': 'https://web.archive.org/web/{base_url}'.format(**link),
+    }
+
+    # PDF and images are handled slightly differently
+    # wget, screenshot, & pdf urls all point to the same file
+    if link['type'] in ('PDF', 'image'):
+        link_info.update({
+            'archive_url': 'archive/{timestamp}/{base_url}'.format(**link),
+            'pdf_link': 'archive/{timestamp}/{base_url}'.format(**link),
+            'screenshot_link': 'archive/{timestamp}/{base_url}'.format(**link),
+            'title': '{title} ({type})'.format(**link),
+        })
+    return link_info
Author	SHA1	Message	Date
Nick Sweeting	2b4384c311	update docs	2017-10-30 04:08:11 -05:00
Nick Sweeting	5758cc2a78	add user-data-dir support for chrome headless	2017-10-30 04:01:59 -05:00
Nick Sweeting	81ab050cd2	switch back to relative paths	2017-10-30 03:36:42 -05:00
Nick Sweeting	cb0d6526d7	fix infinite appending paths	2017-10-30 03:31:28 -05:00
Nick Sweeting	c8068caff0	debugging2	2017-10-30 03:29:53 -05:00
Nick Sweeting	f477224ae3	debugging	2017-10-30 03:28:28 -05:00
Nick Sweeting	597a5f6997	use absolute paths instead of relative	2017-10-30 03:19:52 -05:00
Nick Sweeting	734c99cfdb	fix parser bailing out with IndexError	2017-10-30 03:03:31 -05:00
Nick Sweeting	965130815d	check if folder exists before trying to cleanup	2017-10-30 02:55:39 -05:00
Nick Sweeting	318b9ae1db	finished manual link merging logic to fix folder conflicts	2017-10-30 02:50:37 -05:00
Nick Sweeting	db26ab9aa9	add basic html and fancy link index options	2017-10-30 02:49:59 -05:00
Nick Sweeting	a95912679e	refactoring and fancy new link index	2017-10-23 04:58:41 -05:00
Nick Sweeting	1249493fcd	better config system	2017-10-23 04:57:34 -05:00
Nick Sweeting	77ab8ebda6	cleanup archive.py	2017-10-23 04:56:21 -05:00
Nick Sweeting	3f3531ed4c	fix timestamp uniquification	2017-10-18 19:33:31 -05:00
Nick Sweeting	7e4ca8a6ac	link docstring	2017-10-18 19:08:33 -05:00
Nick Sweeting	d74394fa45	encoding bugfixes	2017-10-18 17:47:19 -05:00
Nick Sweeting	eb47155a12	major refactor + ability to handle http downloads	2017-10-18 17:39:09 -05:00
Nick Sweeting	f31da350c5	Update README.md	2017-09-07 17:06:37 -05:00
Nick Sweeting	e0a8ca91b4	add link to reddit exporter	2017-09-07 16:56:35 -05:00
Nick Sweeting	d558f58bb8	add Wallabag instructions	2017-09-04 17:11:28 -05:00
Nick Sweeting	01149fa6ac	Merge pull request #45 from klauern/patch-1 ensure encoding check is case-insensitive	2017-09-04 16:40:56 -05:00
Nick Klauer	6bc2be4c70	change case goodness me...:(	2017-09-04 16:38:43 -05:00
Nick Klauer	e6e3bfead1	ensure encoding check is case-insensitive encoding will fail if it isn't 'UTF-8', resulting in encodings of 'utf-8' to fail, too... This normalizes it.	2017-09-04 09:47:52 -05:00
Nick Sweeting	4799998884	Update README.md	2017-08-27 20:10:21 -05:00
Nick Sweeting	9d3e3c5c65	add Sheetsee-pocket link	2017-08-27 20:00:49 -05:00
Nick Sweeting	a0c39bc725	Merge pull request #43 from m-rossi/custom_archive_dir Add custom archive directory as configuration option.	2017-08-22 01:31:21 -04:00
Marco Rossi	17d07239f0	Add custom archive directory as configuration option.	2017-08-20 17:37:49 +02:00
Nick Sweeting	0a75dee583	Merge pull request #42 from nodiscc/patch-2 clarify TIMEOUT option purpose + archive options	2017-08-03 11:43:18 -05:00
Nick Sweeting	2d4a9c08df	Merge pull request #37 from nodiscc/patch-1 README: clarify chrome/ium dependency,	2017-08-03 11:42:22 -05:00
nodiscc	cb6fcf4a2f	README: clarify archive method options	2017-08-03 16:29:56 +02:00
nodiscc	e499b23543	clarify TIMEOUT option purpose This option does not set a timeout for establishing connections (non-responding hosts): instead it is the maximum allowed time for a page download. If the page is large, setting a low timeout value may cause the transfer to abort, even if the transfer speed is good enough.	2017-08-03 16:24:41 +02:00
nodiscc	5dcfa3e42c	README: clarify chrome/ium dependency, Note that the chromium dependency is only required for screenshots and PDF output to work. Users only wanting to copy the website using `wget` are not required to install it. Fix Shaarli export link	2017-08-03 14:58:49 +02:00
Nick Sweeting	1e0bb2c854	Update README.md	2017-07-25 23:37:30 -05:00
Nick Sweeting	2c40249fba	Update README.md	2017-07-25 14:15:46 -05:00
Nick Sweeting	50a06f5e52	correct mistake about chrome headless sessions	2017-07-22 02:34:17 -05:00
Nick Sweeting	ea1d3541b2	remove jeckl config	2017-07-11 19:43:45 -05:00
Nick Sweeting	700eb63a74	Set theme jekyll-theme-merlot	2017-07-11 19:41:42 -05:00
Nick Sweeting	d8fa4508e8	add google keywords	2017-07-11 19:40:50 -05:00
Nick Sweeting	58de8cb964	Update README.md	2017-07-08 18:42:10 -05:00
Nick Sweeting	e37401d74b	Update fetch.py	2017-07-08 12:51:06 -05:00
Nick Sweeting	3d261c4734	Merge pull request #36 from tscs37/patch-1 Quick Fix for Wrong Argument Splitting on wget	2017-07-08 12:49:17 -05:00
Tim Schuster	ba8307c27e	Quick Fix for Wrong Argument Splitting on wget When setting the user agent, the resulting CLI arguments for wget where `['wget', '--timestamping', '--adjust-extension', '--no-parent', '--page-requisites', '--convert-links', '-', '-', 'u', 's', 'e', 'r', '-', 'a', 'g', 'e', 'n', 't', '=', '"', 'L', 'y', 'n', 'x', '"', 'https://example.org']`, with this fix it turns into `['wget', '--timestamping', '--adjust-extension', '--no-parent', '--page-requisites', '--convert-links', '--user-agent="Lynx"', '', 'https://example.org']`	2017-07-08 15:19:43 +02:00
Nick Sweeting	4bc3698ee3	Set theme jekyll-theme-minimal	2017-07-07 14:10:59 -05:00
Nick Sweeting	31924c091e	add bookmarks output to gitignore	2017-07-06 17:33:57 -05:00
Nick Sweeting	a7d1213459	add note about logged-in sites	2017-07-06 16:58:53 -05:00
Nick Sweeting	7196486766	add docs for changing WGET_USER_AGENT option	2017-07-06 16:54:07 -05:00
Nick Sweeting	acf59faa06	add custom WGET_USER_AGENT override option	2017-07-06 16:51:41 -05:00
Nick Sweeting	83306391ed	enforce utf-8 stdout encoding	2017-07-06 16:44:04 -05:00
Nick Sweeting	648223fc6c	wonky exception	2017-07-05 17:30:00 -05:00
Nick Sweeting	5122fa0738	better format wget output	2017-07-05 17:28:09 -05:00
Nick Sweeting	02f711b8cb	fix CHROME_BINARY and TIMEOUT configs not being used	2017-07-05 17:26:36 -05:00
Nick Sweeting	0d4ebe9418	show full bookmarked time in tooltip	2017-07-05 17:16:10 -05:00
Nick Sweeting	2265f2aaf0	properly handle querystrings for wget .html appended links	2017-07-05 17:15:56 -05:00
Nick Sweeting	6bb91fbb45	dont hide real exceptions	2017-07-05 17:15:39 -05:00
Nick Sweeting	4db30779a3	remove term Star from vocab	2017-07-05 17:15:17 -05:00
Nick Sweeting	b894e0ff92	properly handle Archive.org denied by robots.txt	2017-07-05 16:57:19 -05:00
Nick Sweeting	9e4b97340d	update readme with new options	2017-07-05 16:44:25 -05:00
Nick Sweeting	a167d2a1f4	colorize output and add progress bar	2017-07-05 16:44:25 -05:00
Nick Sweeting	172c4ad5b8	Update README.md	2017-07-05 13:18:22 -05:00
Nick Sweeting	396aec61a7	Merge pull request #33 from bardisty/add-template-strings Use template strings for substitution in HTML output	2017-07-05 13:17:25 -05:00
Brian Hardisty	26b5e4aa3c	Use `$identifier` over `${identifier}` `${identifier}` has the potential to conflict with JavaScript template literals.	2017-07-05 03:22:11 -07:00
Brian Hardisty	28445d26e8	Remove obsolete note on double-bracketed CSS in templates	2017-07-05 03:07:21 -07:00
Brian Hardisty	31ec3203c5	Use template strings for substitution in HTML output `str.format()` can only use substitutions identified by braces (`{` and `}`). This has the potential to conflict with other code in the HTML template, such as CSS or JavaScript. Template strings can use substitutions identified by `$` or `${}`, e.g.: `$identifier` or `${identifier}`. These substitutions won't conflict with CSS or JavaScript, allowing users to write HTML templates that don't require double braces anywhere there's a substitution conflict. This is especially useful when one is using a build tool to generate the final CSS/JavaScript/HTML. https://docs.python.org/3/library/string.html#template-strings	2017-07-05 02:59:09 -07:00
Nick Sweeting	bca2dceec0	chrome upgrade instructions	2017-07-04 06:46:15 -05:00
Nick Sweeting	010c3d9771	upgrade chrome in setup.sh	2017-07-04 06:40:19 -05:00