Update README.md

fix double parsing by accident
show dots as progress bar for titles fetching
2019-02-06 15:40:53 -08:00 · 2019-02-04 22:08:08 -08:00 · 2019-02-04 22:07:59 -08:00 · 2019-02-04 22:07:48 -08:00 · 2019-02-04 21:18:42 -08:00 · 2019-02-04 21:18:17 -08:00
25 changed files with 819 additions and 364 deletions
@@ -3,3 +3,4 @@ __pycache__
 .DS_Store
 venv
 .venv
+data
@@ -8,31 +8,21 @@ assignees: ''
 ---

 **Describe the bug**
-A clear and concise description of what the bug is.
+A description of what the bug is, what you expected to happen, 
+and any relevant context about issue.

-**To Reproduce**
+**Steps to reproduce**
 Steps to reproduce the behavior:
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
 4. See error

-**Expected behavior**
-A clear and concise description of what you expected to happen.
+**Screenshots or log output**
+If applicable, use screenshots or copy/pasted terminal output to help explain your problem.

-**Screenshots**
-If applicable, add screenshots to help explain your problem.
-
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
-
-**Additional context**
-Add any other context about the problem here.
+**Software versions (please complete the following information):**
+ - ArchiveBox version: <commit_id>
+ - Python version: python3 --version [e.g. 3.7.0]
+ - OS: [e.g. macOS 10.14]
+ - Chrome version: chromium-browser --version [e.g. v73.1.2.3]
@@ -7,14 +7,26 @@ assignees: ''

 ---

-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+Type:

-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+ - [ ] General Question or Disussion
+ - [ ] Propose a brand new feature
+ - [ ] Request modification of existing behavior or design

-**Describe alternatives you've considered**
+**What is the problem that your feature request solves**
+e.g. I need to be able to archive spanish and french subtitle files
+from a particular <example.com> movie site that's going down soon.
+
+**Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes**
+e.g.
+  specifically: a new archive method to look for multilingual subtitle files related to pages.
+  bigger picture: the ability for custom user scripts to be run in a puppeteer context to archive content
+
+**What hacks or alternative solutions have you tried to solve the problem?**
 A clear and concise description of any alternative solutions or features you've considered.

-**Additional context**
-Add any other context or screenshots about the feature request here.
+**How badly do you want this new feature?**
+ - [ ] It's an urgent deal-breaker, I cant live without it
+ - [ ] It's important to add it in the near-mid term future
+ - [ ] It would be nice to have eventualy
+ - [ ] I'm willing to contribute to development
@@ -1,48 +1,56 @@
-FROM debian:stretch
+FROM node:11-slim
 LABEL maintainer="Nick Sweeting <archivebox-git@sweeting.me>"

 RUN apt-get update \
-    && apt-get install -qy git wget gnupg2 libgconf-2-4 python3 python3-pip \
+    && apt-get install -yq --no-install-recommends \
+        git wget curl youtube-dl gnupg2 libgconf-2-4 python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*

 # Install latest chrome package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
-RUN apt-get update && apt-get install -y curl --no-install-recommends \
+RUN apt-get update && apt-get install -y wget --no-install-recommends \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
-    && apt-get install -y chromium fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
+    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* \
-    && rm -rf /src/*.deb \
-    && ln -s /usr/bin/chromium /usr/bin/chromium-browser
+    && rm -rf /src/*.deb

-# It might be a good idea to use dumb-init to help prevent zombie chrome processes.
-# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
-# RUN chmod +x /usr/local/bin/dumb-init
+# It's a good idea to use dumb-init to help prevent zombie chrome processes.
+ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
+RUN chmod +x /usr/local/bin/dumb-init

-RUN git clone https://github.com/pirate/ArchiveBox /home/chromeuser/app \
-    && pip3 install -r /home/chromeuser/app/archivebox/requirements.txt
+# Install puppeteer so it's available in the container.
+RUN npm i puppeteer

-# Add user so we area strong, independent chrome that don't need --no-sandbox.
-RUN groupadd -r chromeuser && useradd -r -g chromeuser -G audio,video chromeuser \
-    && mkdir -p /home/chromeuser/app/archivebox/output \
-    && chown -R chromeuser:chromeuser /home/chromeuser/app/archivebox/output \
-    && chown -R chromeuser:chromeuser /home/chromeuser
+# Add user so we don't need --no-sandbox.
+RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
+    && mkdir -p /home/pptruser/Downloads \
+    && chown -R pptruser:pptruser /home/pptruser \
+    && chown -R pptruser:pptruser /node_modules

-VOLUME /home/chromeuser/app/archivebox/output
+# Install the ArchiveBox repository and pip requirements
+RUN git clone https://github.com/pirate/ArchiveBox /home/pptruser/app \
+    && mkdir -p /data \
+    && chown -R pptruser:pptruser /data \
+    && ln -s /data /home/pptruser/app/archivebox/output \
+    && ln -s /home/pptruser/app/bin/archivebox /bin/archive \
+    && chown -R pptruser:pptruser /home/pptruser/app/archivebox
+    # && pip3 install -r /home/pptruser/app/archivebox/requirements.txt

-ENV LANG=en_US.UTF-8 \
+VOLUME /data
+
+ENV LANG=C.UTF-8 \
    LANGUAGE=en_US:en \
-    LC_ALL=en_US.UTF-8 \
+    LC_ALL=C.UTF-8 \
    PYTHONIOENCODING=UTF-8 \
    CHROME_SANDBOX=False \
-    OUTPUT_DIR=/home/chromeuser/app/archivebox/output
+    CHROME_BINARY=google-chrome-unstable \
+    OUTPUT_DIR=/data

 # Run everything from here on out as non-privileged user
-USER chromeuser
-WORKDIR /home/chromeuser/app
+USER pptruser
+WORKDIR /home/pptruser/app

-# ENTRYPOINT ["dumb-init", "--"]
-# CMD ["/home/chromeuser/app/archive"]
-
-ENTRYPOINT ["python3", "-u", "/home/chromeuser/app/archivebox/archive.py"]
+ENTRYPOINT ["dumb-init", "--"]
+CMD ["/bin/archive"]
@@ -1,73 +1,127 @@
-![Logo](https://i.imgur.com/PVO88AZ.png)
+<div align="center">
+<img src="https://i.imgur.com/4nkFjdv.png" height="80px">
+<h1>ArchiveBox<br/><sub>The open-source self-hosted web archive.</sub></h1>

-# ArchiveBox <br/> The open source self-hosted web archive <img src="https://nicksweeting.com/images/archive.png" height="22px"/>  [![Github Stars](https://img.shields.io/github/stars/pirate/bookmark-archiver.svg)](https://github.com/pirate/ArchiveBox) [![Twitter URL](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/thesquashSH)
+▶️ <a href="https://github.com/pirate/ArchiveBox/wiki/Quickstart">Quickstart</a> | 
+<a href="https://archive.sweeting.me">Demo</a> | 
+<a href="https://archivebox.io">Website</a> | 
+<a href="https://github.com/pirate/ArchiveBox">Github</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Troubleshooting">Troubleshooting</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Changelog">Changelog</a> | 
+<a href="https://github.com/pirate/ArchiveBox/wiki/Roadmap">Roadmap</a>
+
+<pre>
+"Your own personal internet archive" (网站存档 / 爬虫)
+</pre>
+
+<a href="https://github.com/pirate/ArchiveBox">
+<img src="https://img.shields.io/badge/Open_source-free-green.svg?logo=git&logoColor=green"/>
+<img src="https://img.shields.io/github/last-commit/pirate/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/>
+<img src="https://img.shields.io/badge/License-MIT-lightgreen.svg?logo=MakerBot&logoColor=lightgreen"/>
+<img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/>
+<img src="https://img.shields.io/badge/Python-%3E%3D3.5-yellow.svg?logo=python&logoColor=yellow"/>
+<img src="https://img.shields.io/badge/Chromium-%3E%3D59-orange.svg?logo=Google+Chrome&logoColor=orange"/>
+<img src="https://img.shields.io/badge/Docker-all%20platforms-lightblue.svg?logo=docker&logoColor=lightblue"/>
+</a>
+<hr/>
+</div>
+
+**ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).** 
+
+You can use it to preserve access to websites you care about by storing them locally offline.  ArchiveBox works by rendering the pages in a headless browser, then saving all the requests and fully loaded pages in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the original content dissapears off the internet.  It also automatically extracts assets like git repositories, audio, video, subtitles, images, and PDFs into separate files using `youtube-dl`, `pywb`, and `wget`.
+
+ArchiveBox doesn't require a constantly running server or backend, instead you just run the `./archive` command each time you want to import new links and update the static output.  It can import and export JSON (among other formats), so it's easy to script or hook up to other APIs.  If you run it on a schedule and import from browser history or bookmarks regularly, you can sleep soundly knowing that the slice of the internet you care about will be automatically preserved in multiple, durable long-term formats that will be accessible for decades (or longer).
+
+<div align="center"><sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub></div><br/>


-### (Recently [renamed](https://github.com/pirate/ArchiveBox/issues/108) from `Bookmark Archiver`)
-
-
-    "Your own personal Way-Back Machine"
-
-💻 [Demo](https://archive.sweeting.me) | [Website](https://archivebox.io/) | [Source](https://github.com/pirate/ArchiveBox/tree/master) | [Changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog) | [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap)
-
-▶️ [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart) | [Details](https://github.com/pirate/ArchiveBox/wiki) | [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) | [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)
-
---
-
-Save an archived copy of the websites you visit (the actual *content* of each site, not just the list of links).  Can archive entire browsing history, or just links matching a filter or bookmarks list.
-
-ArchiveBox can import links from:
-
- - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera)
- - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket
- - <img src="https://pinboard.in/favicon.ico" height="22px"/> Pinboard
- - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS or plain text lists
- - Shaarli, Delicious, Instapaper, Reddit Saved Posts, Wallabag, Unmark.it, and more!
-
-For each site, it outputs (configurable):
-
- Browsable static HTML archive (wget)
- PDF (Chrome headless)
- Screenshot (Chrome headless)
- HTML after 2s of JS running (Chrome headless)
- Favicon
- Submits URL to archive.org
- Index summary pages: index.html & index.json
-
-The archiving is additive, so you can schedule `./archive` to run regularly and pull new links into the index.
-All the saved content is static and indexed with json files, so it lives forever & is easily parseable, it requires no always-running backend.
-
-[DEMO: archive.sweeting.me](https://archive.sweeting.me)
-
-[![](https://img.shields.io/badge/Donate-Patreon-%23DD5D76.svg)](https://www.patreon.com/theSquashSH)
-
+To get started, you can install ArchiveBox [automatically](https://github.com/pirate/ArchiveBox/wiki/Quickstart), follow the [manual instructions](https://github.com/pirate/ArchiveBox/wiki/Install), or use [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker).  
+There are only [3 main dependencies](https://github.com/pirate/ArchiveBox/wiki/Install#dependencies) beyond `python3`: `wget`, `chromium`, and `youtube-dl`, and you can skip installing them if you don't need those archive methods.
 ```bash
 git clone https://github.com/pirate/ArchiveBox.git
 cd ArchiveBox
 ./setup

 # Export your bookmarks, then run the archive command to start archiving!
-./archive ~/Downloads/firefox_bookmarks.html
+./archive ~/Downloads/bookmarks.html
+
+# Or pass in links to archive via stdin
+echo 'https://example.com' | ./archive
 ```

+Open `output/index.html` in a browser to view your archive.  [DEMO: archive.sweeting.me](https://archive.sweeting.me)  
+For more information, see the [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart), [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage), and [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration) docs.
+
+## Overview
+
+Because modern websites are complicated and often rely on dynamic content, 
+ArchiveBox archives the sites in **several different formats** beyond what public 
+archiving services like Archive.org and Archive.is are capable of saving.
+
+ArchiveBox imports a list of URLs from stdin, remote url, or file, then adds the pages to a local archive folder using wget to create a browsable html clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...
+
+Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.
+
+### Can import links from:
+
+ - <img src="https://getpocket.com/favicon.ico" height="22px"/> Pocket, Pinboard, Instapaper
+ - <img src="https://nicksweeting.com/images/rss.svg" height="22px"/> RSS, XML, JSON, or plain text lists
+ - <img src="https://nicksweeting.com/images/bookmarks.png" height="22px"/> Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera, and more)
+ - *Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, and any other text with links in it!*
+
+### Can save these things for each site:
+
+ - `favicon.ico` favicon of the site
+ - `example.com/page-name.html` wget clone of the site, with .html appended if not present
+ - `output.pdf` Printed PDF of site using headless chrome
+ - `screenshot.png` 1440x900 screenshot of site using headless chrome
+ - `output.html` DOM Dump of the HTML after rendering using headless chrome
+ - `archive.org.txt` A link to the saved site on archive.org
+ - `warc/` for the html + gzipped warc file <timestamp>.gz
+ - `media/` any mp4, mp3, subtitles, and metadata found using youtube-dl
+ - `git/` clone of any repository for github, bitbucket, or gitlab links
+ - `index.html` & `index.json` HTML and JSON index files containing metadata and details
+
+By default it does everything, but can disable or tweak individual options via environment variables or config file.
+
+The archiving is additive, so you can schedule `./archive` to run regularly and pull new links into the index.
+All the saved content is static and indexed with JSON files, so it lives forever & is easily parseable, it requires no always-running backend.
+
+### Related Projects
+
+There are tons of great web archiving tools out there.  In particular https://Webrecorder.io (pywb) and https://getpolarized.io/ are robust, stable pieces of software that have lots of overlap with ArchiveBox.  ArchiveBox differentiates itself by being primarily a one-shot CLI tool that specializing in importing streams of links from RSS, JSON (good for automatically archiving from a stream of browser history or bookmarks on a schedule), as opposed to a desktop application or web service that requires human interaction to add links.
+
+To learn more about the motivation for this project and how it fits into the broader community, see our [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.

 # Documentation

+
+<div align="center">
+<img src="https://i.imgur.com/PVO88AZ.png"/>
+<br/>
+    <sub><i>(Recently <a href="https://github.com/pirate/ArchiveBox/issues/108">renamed</a> from <code>Bookmark Archiver</code>)</i></sub>
+</div>
+
+---
+
 We use the [Github wiki system](https://github.com/pirate/ArchiveBox/wiki) for documentation.

 You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/pirate/ArchiveBox/wiki/Home) folder.

 ## Getting Started

- - [Details & Motivation](https://github.com/pirate/ArchiveBox/wiki)
 - [Quickstart](https://github.com/pirate/ArchiveBox/wiki/Quickstart)
 - [Install](https://github.com/pirate/ArchiveBox/wiki/Install)
+ - [Docker](https://github.com/pirate/ArchiveBox/wiki/Docker)

 ## Reference

+ - [Usage](https://github.com/pirate/ArchiveBox/wiki/Usage)
 - [Configuration](https://github.com/pirate/ArchiveBox/wiki/Configuration)
- - [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Chromium-Install)
+ - [Scheduled Archiving](https://github.com/pirate/ArchiveBox/wiki/Scheduled-Archiving)
 - [Publishing Your Archive](https://github.com/pirate/ArchiveBox/wiki/Publishing-Your-Archive)
+ - [Chromium Install](https://github.com/pirate/ArchiveBox/wiki/Chromium-Install)
 - [Troubleshooting](https://github.com/pirate/ArchiveBox/wiki/Troubleshooting)

 ## More Info
@@ -77,11 +131,47 @@ You can also access the docs locally by looking in the [`ArchiveBox/docs/`](http
 - [Donations](https://github.com/pirate/ArchiveBox/wiki/Donations)
 - [Web Archiving Community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community)

+
+# Background & Motivation
+
+Vast treasure troves of knowledge are lost every day on the internet to link rot.  As a society, we have an imperative
+to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.
+
+Whether it's to resist censorship by saving articles before they get taken down or editied, or
+just to save a collection of early 2010's flash games you love to play, having the tools to 
+archive internet content enables to you save the stuff you care most about before it dissapears.
+
+The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. 
+I don't think everything should be preserved in an automated fashion, making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about.
+
+The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
+
+*Read more:*
+
+- Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
+- Discover the web archiving community on the [community](https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community) wiki page.
+- Find other archving projects on Github using the [awesome-web-archiving](https://github.com/iipc/awesome-web-archiving) list.
+- Or reach out to me for questions and comments via [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+
+To learn more about ArchiveBox's past history and future plans, check out the [roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap) and [changelog](https://github.com/pirate/ArchiveBox/wiki/Changelog).
+
 # Screenshots

-<img src="https://i.imgur.com/q3Oz9wN.png" width="75%" alt="Desktop Screenshot" align="top"><img src="https://i.imgur.com/TG0fGVo.png" width="25%" alt="Mobile Screenshot" align="top"><br/>
+<img src="https://i.imgur.com/viklZNG.png" width="75%" alt="Desktop index screenshot" align="top"><img src="https://i.imgur.com/mW2dITg.png" width="25%" alt="Mobile details page screenshot" align="top"><br/>
+<img src="https://i.imgur.com/wnpdAVM.jpg" width="100%" alt="Desktop details page Screenshot"/><br/>
 <img src="https://i.imgur.com/3tBL7PU.png" width="100%" alt="CLI Screenshot">

 ---

-<img src="https://i.imgur.com/JffzvjI.png" alt="Logo" style="width: 50%; border-radius: 14px; text-align: center"/>
+<div align="center">
+<br/><br/>
+<img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" height="40px"/>
+<br/>
+<sub><i>This project is maintained mostly in my spare time with the help from generous contributors.</i></sub>
+<br/><br/>
+<a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Donate_to_support_development-via_Patreon-%23DD5D76.svg?style=flat"/></a>
+<br/>
+<br/>
+<a href="https://twitter.com/thesquashSH"><img src="https://img.shields.io/badge/Tweet-%40theSquashSH-blue.svg?style=flat"/></a>
+<a href="https://github.com/pirate/ArchiveBox"><img src="https://img.shields.io/github/stars/pirate/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>
+</div>
@@ -31,6 +31,7 @@ from config import (
 )
 from util import (
    download_url,
+    save_source,
    progress,
    cleanup_archive,
    pretty_path,
@@ -39,14 +40,18 @@ from util import (

 __AUTHOR__ = 'Nick Sweeting <git@nicksweeting.com>'
 __VERSION__ = GIT_SHA
-__DESCRIPTION__ = 'ArchiveBox: Create a browsable html archive of a list of links.'
-__DOCUMENTATION__ = 'https://github.com/pirate/ArchiveBox'
+__DESCRIPTION__ = 'ArchiveBox Usage:  Create a browsable html archive of a list of links.'
+__DOCUMENTATION__ = 'https://github.com/pirate/ArchiveBox/wiki'

 def print_help():
    print(__DESCRIPTION__)
    print("Documentation:     {}\n".format(__DOCUMENTATION__))
    print("Usage:")
    print("    ./bin/archivebox ~/Downloads/bookmarks_export.html\n")
+    print("")
+    print("    ./bin/archivebox https://example.com/feed.rss\n")
+    print("")
+    print("    echo 'https://examplecom' | ./bin/archivebox\n")


 def merge_links(archive_path=OUTPUT_DIR, import_path=None, only_new=False):
@@ -54,7 +59,7 @@ def merge_links(archive_path=OUTPUT_DIR, import_path=None, only_new=False):
    all_links = []
    if import_path:
        # parse and validate the import file
-        raw_links = parse_links(import_path)
+        raw_links, parser_name = parse_links(import_path)
        all_links = validate_links(raw_links)

    # merge existing links in archive_path and new links
@@ -62,14 +67,14 @@ def merge_links(archive_path=OUTPUT_DIR, import_path=None, only_new=False):
    if archive_path:
        existing_links = parse_json_links_index(archive_path)
        all_links = validate_links(existing_links + all_links)
-    
+
    num_new_links = len(all_links) - len(existing_links)
    if num_new_links and not only_new:
-        print('[{green}+{reset}] [{}] Adding {} new links from {} to {}/index.json'.format(
+        print('{green}[+] [{}] Adding {} new links to index from {} ({} format){reset}'.format(
            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
            num_new_links,
            pretty_path(import_path),
-            pretty_path(archive_path),
+            parser_name,
            **ANSI,
        ))
    # else:
@@ -97,7 +102,7 @@ def update_archive(archive_path, links, source=None, resume=None, append=True):
             **ANSI,
        ))
    else:
-        print('{green}[▶] [{}] Updating files for {} links in archive...{reset}'.format(
+        print('{green}[▶] [{}] Downloading content for {} pages in archive...{reset}'.format(
             datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
             len(links),
             **ANSI,
@@ -138,6 +143,20 @@ if __name__ == '__main__':
    source = sys.argv[1] if argc > 1 else None  # path of links file to import
    resume = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
   
+    stdin_raw_text = []
+
+    if not sys.stdin.isatty():
+        stdin_raw_text = sys.stdin.read()
+
+    if source and stdin_raw_text:
+        print(
+            '[X] You should pass either a path as an argument, '
+            'or pass a list of links via stdin, but not both.\n'
+        )
+        print_help()
+        raise SystemExit(1)
+
+
    if argc == 1:
        source, resume = None, None
    elif argc == 2:
@@ -163,10 +182,11 @@ if __name__ == '__main__':
    # Step 0: Download url to local file (only happens if a URL is specified instead of local path) 
    if source and any(source.startswith(s) for s in ('http://', 'https://', 'ftp://')):
        source = download_url(source)
+    elif stdin_raw_text:
+        source = save_source(stdin_raw_text)

    # Step 1: Parse the links and dedupe them with existing archive
-    links = merge_links(archive_path=out_dir, import_path=source, only_new=False)
-    new_links = merge_links(archive_path=out_dir, import_path=source, only_new=True)
+    links = merge_links(archive_path=out_dir, import_path=source, only_new=ONLY_NEW)

    # Step 2: Write new index
    write_links_index(out_dir=out_dir, links=links)
@@ -175,7 +195,4 @@ if __name__ == '__main__':
    # cleanup_archive(out_dir, links)

    # Step 4: Run the archive methods for each link
-    if ONLY_NEW:
-        update_archive(out_dir, new_links, source=source, resume=resume, append=True)
-    else:
-        update_archive(out_dir, links, source=source, resume=resume, append=True)
+    update_archive(out_dir, links, source=source, resume=resume, append=True)
@@ -1,10 +1,10 @@
 import os
+import re
 import sys

 from functools import wraps
 from collections import defaultdict
 from datetime import datetime
-from subprocess import run, PIPE, DEVNULL

 from peekable import Peekable

@@ -23,8 +23,6 @@ from config import (
    RESOLUTION,
    CHECK_SSL_VALIDITY,
    SUBMIT_ARCHIVE_DOT_ORG,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
    FETCH_FAVICON,
    WGET_USER_AGENT,
    CHROME_USER_DATA_DIR,
@@ -34,12 +32,14 @@ from config import (
    ANSI,
    ARCHIVE_DIR,
    GIT_DOMAINS,
+    GIT_SHA,
 )
 from util import (
    check_dependencies,
    progress,
    chmod_file,
    pretty_path,
+    run, PIPE, DEVNULL
 )


@@ -81,53 +81,46 @@ def archive_links(archive_path, links, source=None, resume=None):
 def archive_link(link_dir, link, overwrite=True):
    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""

-    update_existing = os.path.exists(link_dir)
-    if update_existing:
-        link = {
-            **parse_json_link_index(link_dir),
-            **link,
-        }
-    else:
-        os.makedirs(link_dir)
-    
-    log_link_archive(link_dir, link, update_existing)
+    try:
+        update_existing = os.path.exists(link_dir)
+        if update_existing:
+            link = {
+                **parse_json_link_index(link_dir),
+                **link,
+            }
+        else:
+            os.makedirs(link_dir)
+        
+        log_link_archive(link_dir, link, update_existing)

-    if FETCH_FAVICON:
-        link = fetch_favicon(link_dir, link, overwrite=overwrite)
+        if FETCH_FAVICON:
+            link = fetch_favicon(link_dir, link, overwrite=overwrite)

-    if FETCH_WGET:
-        link = fetch_wget(link_dir, link, overwrite=overwrite)
+        if FETCH_WGET:
+            link = fetch_wget(link_dir, link, overwrite=overwrite)

-    if FETCH_PDF:
-        link = fetch_pdf(link_dir, link, overwrite=overwrite)
+        if FETCH_PDF:
+            link = fetch_pdf(link_dir, link, overwrite=overwrite)

-    if FETCH_SCREENSHOT:
-        link = fetch_screenshot(link_dir, link, overwrite=overwrite)
+        if FETCH_SCREENSHOT:
+            link = fetch_screenshot(link_dir, link, overwrite=overwrite)

-    if FETCH_DOM:
-        link = fetch_dom(link_dir, link, overwrite=overwrite)
+        if FETCH_DOM:
+            link = fetch_dom(link_dir, link, overwrite=overwrite)

-    if FETCH_WARC:
-        link = fetch_warc(link_dir, link, overwrite=overwrite)
+        if SUBMIT_ARCHIVE_DOT_ORG:
+            link = archive_dot_org(link_dir, link, overwrite=overwrite)

-    if SUBMIT_ARCHIVE_DOT_ORG:
-        link = archive_dot_org(link_dir, link, overwrite=overwrite)
+        if FETCH_GIT:
+            link = fetch_git(link_dir, link, overwrite=overwrite)

-    # if FETCH_AUDIO:
-    #     link = fetch_audio(link_dir, link, overwrite=overwrite)
+        if FETCH_MEDIA:
+            link = fetch_media(link_dir, link, overwrite=overwrite)

-    # if FETCH_VIDEO:
-    #     link = fetch_video(link_dir, link, overwrite=overwrite)
+        write_link_index(link_dir, link)

-    if FETCH_GIT:
-        link = fetch_git(link_dir, link, overwrite=overwrite)
-
-    if FETCH_MEDIA:
-        link = fetch_media(link_dir, link, overwrite=overwrite)
-
-
-    write_link_index(link_dir, link)
-    # print()
+    except Exception as err:
+        print('    ! Failed to archive link: {}: {}'.format(err.__class__.__name__, err))
    
    return link

@@ -200,7 +193,7 @@ def attach_result_to_link(method):


@attach_result_to_link('wget')
-def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
+def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, warc=FETCH_WARC, timeout=TIMEOUT):
    """download full site using wget"""

    domain_dir = os.path.join(link_dir, link['domain'])
@@ -208,37 +201,63 @@ def fetch_wget(link_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT
    if os.path.exists(domain_dir) and existing_file:
        return {'output': existing_file, 'status': 'skipped'}

+    if warc:
+        warc_dir = os.path.join(link_dir, 'warc')
+        os.makedirs(warc_dir, exist_ok=True)
+        warc_path = os.path.join('warc', str(int(datetime.now().timestamp())))
+
+    # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
    CMD = [
-        # WGET CLI Docs: https://www.gnu.org/software/wget/manual/wget.html
-        *'wget -N -E -np -x -H -k -K -S --restrict-file-names=unix'.split(' '),
-        *(('-p',) if FETCH_WGET_REQUISITES else ()),
+        'wget',
+        # '--server-response',  # print headers for better error parsing
+        '--no-verbose',
+        '--adjust-extension',
+        '--convert-links',
+        '--force-directories',
+        '--backup-converted',
+        '--span-hosts',
+        '--no-parent',
+        '--restrict-file-names=unix',
+        '--timeout={}'.format(timeout),
+        *(() if warc else ('--timestamping',)),
+        *(('--warc-file={}'.format(warc_path),) if warc else ()),
+        *(('--page-requisites',) if FETCH_WGET_REQUISITES else ()),
        *(('--user-agent={}'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
-        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate',))),
+        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate', '--no-hsts'))),
        link['url'],
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # index.html
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # index.html
        end()
        output = wget_output_path(link, look_in=domain_dir)

+        output_tail = ['          ' + line for line in (result.stdout + result.stderr).decode().rsplit('\n', 3)[-3:] if line.strip()]
+
+        # parse out number of files downloaded from "Downloaded: 76 files, 4.0M in 1.6s (2.52 MB/s)"
+        files_downloaded = (
+            int(output_tail[-1].strip().split(' ', 2)[1] or 0)
+            if 'Downloaded:' in output_tail[-1]
+            else 0
+        )
+
        # Check for common failure cases
-        if result.returncode > 0:
-            print('        got wget response code {}:'.format(result.returncode))
-            if result.returncode != 8:
-                print('\n'.join('          ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
+        if result.returncode > 0 and files_downloaded < 1:
+            print('        Got wget response code {}:'.format(result.returncode))
+            print('\n'.join(output_tail))
            if b'403: Forbidden' in result.stderr:
                raise Exception('403 Forbidden (try changing WGET_USER_AGENT)')
            if b'404: Not Found' in result.stderr:
                raise Exception('404 Not Found')
            if b'ERROR 500: Internal Server Error' in result.stderr:
                raise Exception('500 Internal Server Error')
-            if result.returncode == 4:
-                raise Exception('Failed wget download')
+            raise Exception('Got an error from the server')
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        {}Some resources were skipped: {}{}'.format(ANSI['lightyellow'], e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD).replace(WGET_USER_AGENT, '"{}"'.format(WGET_USER_AGENT))))
        output = e

    return {
@@ -260,11 +279,14 @@ def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
    CMD = [
        *chrome_headless(user_data_dir=user_data_dir),
        '--print-to-pdf',
+        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
        link['url']
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.pdf
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.pdf
        end()
        if result.returncode:
            print('     ', (result.stderr or result.stdout).decode())
@@ -273,8 +295,10 @@ def fetch_pdf(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
        output = 'output.pdf'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -297,12 +321,14 @@ def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_
        '--screenshot',
        '--window-size={}'.format(resolution),
        '--hide-scrollbars',
+        '--timeout={}'.format((timeout) * 1000),
+        *(() if CHECK_SSL_VALIDITY else ('--disable-web-security', '--ignore-certificate-errors')),
        # '--full-page',   # TODO: make this actually work using ./bin/screenshot fullPage: true
        link['url'],
    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # sreenshot.png
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=link_dir, timeout=timeout)  # sreenshot.png
        end()
        if result.returncode:
            print('     ', (result.stderr or result.stdout).decode())
@@ -311,8 +337,10 @@ def fetch_screenshot(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_
        output = 'screenshot.png'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -335,12 +363,13 @@ def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
    CMD = [
        *chrome_headless(user_data_dir=user_data_dir),
        '--dump-dom',
+        '--timeout={}'.format((timeout) * 1000),
        link['url']
    ]
    end = progress(timeout, prefix='      ')
    try:
        with open(output_path, 'w+') as f:
-            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout + 1)  # output.html
+            result = run(CMD, stdout=f, stderr=PIPE, cwd=link_dir, timeout=timeout)  # output.html
        end()
        if result.returncode:
            print('     ', (result.stderr).decode())
@@ -349,8 +378,10 @@ def fetch_dom(link_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DI
        output = 'output.html'
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -370,10 +401,19 @@ def archive_dot_org(link_dir, link, timeout=TIMEOUT):
    submit_url = 'https://web.archive.org/save/{}'.format(link['url'])

    success = False
-    CMD = ['curl', '-L', '-I', '-X', 'GET', submit_url]
+    CMD = [
+        'curl',
+        '--location',
+        '--head',
+        '--user-agent', 'ArchiveBox/{} (+https://github.com/pirate/ArchiveBox/)'.format(GIT_SHA),
+        '--max-time', str(timeout),
+        '--get',
+        *(() if CHECK_SSL_VALIDITY else ('--insecure',)),
+        submit_url,
+    ]
    end = progress(timeout, prefix='      ')
    try:
-        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # archive.org.txt
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # archive.org.txt
        end()

        # Parse archive.org response headers
@@ -402,8 +442,9 @@ def archive_dot_org(link_dir, link, timeout=TIMEOUT):
            raise Exception('Failed to find "content-location" URL header in Archive.org response.')
    except Exception as e:
        end()
-        print('        Visit url to see output:', ' '.join(CMD))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
        output = e

    if success:
@@ -424,11 +465,15 @@ def fetch_favicon(link_dir, link, timeout=TIMEOUT):
    if os.path.exists(os.path.join(link_dir, 'favicon.ico')):
        return {'output': 'favicon.ico', 'status': 'skipped'}

-    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
+    CMD = [
+        'curl',
+        '--max-time', str(timeout),
+        'https://www.google.com/s2/favicons?domain={domain}'.format(**link),
+    ]
    fout = open('{}/favicon.ico'.format(link_dir), 'w')
    end = progress(timeout, prefix='      ')
    try:
-        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout + 1)  # favicon.ico
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=link_dir, timeout=timeout)  # favicon.ico
        fout.close()
        end()
        chmod_file('favicon.ico', cwd=link_dir)
@@ -436,8 +481,9 @@ def fetch_favicon(link_dir, link, timeout=TIMEOUT):
    except Exception as e:
        fout.close()
        end()
-        print('        Run to see full output:', ' '.join(CMD))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -452,7 +498,7 @@ def fetch_media(link_dir, link, timeout=MEDIA_TIMEOUT, overwrite=False):

    # import ipdb; ipdb.set_trace()
    output = os.path.join(link_dir, 'media')
-    already_done = os.path.exists(output) and os.listdir(output)
+    already_done = os.path.exists(output)  # and os.listdir(output)
    if already_done and not overwrite:
        return {'output': 'media', 'status': 'skipped'}

@@ -480,9 +526,16 @@ def fetch_media(link_dir, link, timeout=MEDIA_TIMEOUT, overwrite=False):
    end = progress(timeout, prefix='      ')
    try:
        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=output, timeout=timeout + 1)  # audio/audio.mp3
+        chmod_file('media', cwd=link_dir)
+        output = 'media'
        end()
        if result.returncode:
-            if b'ERROR: Unsupported URL' in result.stderr:
+            if (b'ERROR: Unsupported URL' in result.stderr
+                or b'HTTP Error 404' in result.stderr
+                or b'HTTP Error 403' in result.stderr
+                or b'URL could be a direct video link' in result.stderr
+                or b'Unable to extract container ID' in result.stderr):
+                # These happen too frequently on non-media pages to warrant printing to console
                pass
            else:
                print('        got youtubedl response code {}:'.format(result.returncode))
@@ -490,8 +543,10 @@ def fetch_media(link_dir, link, timeout=MEDIA_TIMEOUT, overwrite=False):
                raise Exception('Failed to download media')
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -512,7 +567,7 @@ def fetch_git(link_dir, link, timeout=TIMEOUT):
    if os.path.exists(os.path.join(link_dir, 'git')):
        return {'output': 'git', 'status': 'skipped'}

-    CMD = ['git', 'clone', '--recursive', link['url'], 'git']
+    CMD = ['git', 'clone', '--mirror', '--recursive', link['url'].split('#')[0], 'git']
    output = 'git'

    end = progress(timeout, prefix='      ')
@@ -525,8 +580,10 @@ def fetch_git(link_dir, link, timeout=TIMEOUT):
            raise Exception('Failed git download')
    except Exception as e:
        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        print('        Run to see full output:')
+        print('            cd {};'.format(link_dir))
+        print('            {}'.format(' '.join(CMD)))
        output = e

    return {
@@ -534,59 +591,11 @@ def fetch_git(link_dir, link, timeout=TIMEOUT):
        'output': output,
    }

-
-@attach_result_to_link('warc')
-def fetch_warc(link_dir, link, timeout=TIMEOUT):
-    """download full site using wget's warc saving feature"""
-
-    output = os.path.join(link_dir, 'warc')
-    if os.path.exists(output) and os.listdir(output):
-        return {'output': 'warc', 'status': 'skipped'}
-
-    os.makedirs(output, exist_ok=True)
-    CMD = [
-        'wget',
-        '--warc-file={}'.format(int(datetime.now().timestamp())),
-        *(('--user-agent={}'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
-        *((() if CHECK_SSL_VALIDITY else ('--no-check-certificate',))),
-        link['url'],
-    ]
-
-    end = progress(timeout, prefix='      ')
-    try:
-        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=output, timeout=timeout + 1)  # warc/at-00000.warc.gz
-        end()
-
-        # Check for common failure cases
-        if result.returncode > 0:
-            print('        got wget response code {}:'.format(result.returncode))
-            if result.returncode != 8:
-                print('\n'.join('          ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
-            if b'403: Forbidden' in result.stderr:
-                raise Exception('403 Forbidden (try changing WGET_USER_AGENT)')
-            if b'404: Not Found' in result.stderr:
-                raise Exception('404 Not Found')
-            if b'ERROR 500: Internal Server Error' in result.stderr:
-                raise Exception('500 Internal Server Error')
-            if result.returncode == 4:
-                raise Exception('Failed warc download')
-    except Exception as e:
-        end()
-        print('        Run to see full output:', 'cd {}; {}'.format(link_dir, ' '.join(CMD)))
-        print('        {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
-        output = e
-
-    return {
-        'cmd': CMD,
-        'output': output,
-    }
-
-
 def chrome_headless(binary=CHROME_BINARY, user_data_dir=CHROME_USER_DATA_DIR):
    args = [binary, '--headless']  # '--disable-gpu'
    if not CHROME_SANDBOX:
        args.append('--no-sandbox')
-    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
+    default_profile = os.path.expanduser('~/Library/Application Support/Google/Chrome')
    if user_data_dir:
        args.append('--user-data-dir={}'.format(user_data_dir))
    elif os.path.exists(default_profile):
@@ -11,37 +11,38 @@ from subprocess import run, PIPE
 # ******************************************************************************

 IS_TTY = sys.stdout.isatty()
+ONLY_NEW =               os.getenv('ONLY_NEW',               'False'            ).lower() == 'true'
 USE_COLOR =              os.getenv('USE_COLOR',              str(IS_TTY)        ).lower() == 'true'
 SHOW_PROGRESS =          os.getenv('SHOW_PROGRESS',          str(IS_TTY)        ).lower() == 'true'
-ONLY_NEW =               os.getenv('ONLY_NEW',               'False'            ).lower() == 'true'
+OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
+MEDIA_TIMEOUT =          int(os.getenv('MEDIA_TIMEOUT',      '3600'))
+TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
+
 FETCH_WGET =             os.getenv('FETCH_WGET',             'True'             ).lower() == 'true'
 FETCH_WGET_REQUISITES =  os.getenv('FETCH_WGET_REQUISITES',  'True'             ).lower() == 'true'
-FETCH_AUDIO =            os.getenv('FETCH_AUDIO',            'False'            ).lower() == 'true'
-FETCH_VIDEO =            os.getenv('FETCH_VIDEO',            'False'            ).lower() == 'true'
 FETCH_PDF =              os.getenv('FETCH_PDF',              'True'             ).lower() == 'true'
 FETCH_SCREENSHOT =       os.getenv('FETCH_SCREENSHOT',       'True'             ).lower() == 'true'
 FETCH_DOM =              os.getenv('FETCH_DOM',              'True'             ).lower() == 'true'
 FETCH_WARC =             os.getenv('FETCH_WARC',             'True'             ).lower() == 'true'
 FETCH_GIT =              os.getenv('FETCH_GIT',              'True'             ).lower() == 'true'
-FETCH_MEDIA =            os.getenv('FETCH_MEDIA',            'False'            ).lower() == 'true'
+FETCH_MEDIA =            os.getenv('FETCH_MEDIA',            'True'             ).lower() == 'true'
 FETCH_FAVICON =          os.getenv('FETCH_FAVICON',          'True'             ).lower() == 'true'
 SUBMIT_ARCHIVE_DOT_ORG = os.getenv('SUBMIT_ARCHIVE_DOT_ORG', 'True'             ).lower() == 'true'
-RESOLUTION =             os.getenv('RESOLUTION',             '1440,1200'        )
+
 CHECK_SSL_VALIDITY =     os.getenv('CHECK_SSL_VALIDITY',     'True'             ).lower() == 'true'
-OUTPUT_PERMISSIONS =     os.getenv('OUTPUT_PERMISSIONS',     '755'              )
+RESOLUTION =             os.getenv('RESOLUTION',             '1440,2000'        )
+GIT_DOMAINS =            os.getenv('GIT_DOMAINS',            'github.com,bitbucket.org,gitlab.com').split(',')
+WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',        'ArchiveBox/{GIT_SHA} (+https://github.com/pirate/ArchiveBox/) wget/{WGET_VERSION}')
+CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',    None)
+
 CHROME_BINARY =          os.getenv('CHROME_BINARY',          None)  # change to google-chrome browser if using google-chrome
 WGET_BINARY =            os.getenv('WGET_BINARY',            'wget'             )
-WGET_USER_AGENT =        os.getenv('WGET_USER_AGENT',        'ArchiveBox')
-CHROME_USER_DATA_DIR =   os.getenv('CHROME_USER_DATA_DIR',    None)
-TIMEOUT =                int(os.getenv('TIMEOUT',            '60'))
-MEDIA_TIMEOUT =          int(os.getenv('MEDIA_TIMEOUT',      '3600'))
 FOOTER_INFO =            os.getenv('FOOTER_INFO',            'Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.',)
-GIT_DOMAINS =            os.getenv('GIT_DOMAINS',            'github.com,bitbucket.org,gitlab.com').split(',')

 ### Paths
 REPO_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))

-OUTPUT_DIR = os.getenv('OUTPUT_DIR', os.path.join(REPO_DIR, 'output'))
+OUTPUT_DIR = os.path.abspath(os.getenv('OUTPUT_DIR', os.path.join(REPO_DIR, 'output')))
 ARCHIVE_DIR = os.path.join(OUTPUT_DIR, 'archive')
 SOURCES_DIR = os.path.join(OUTPUT_DIR, 'sources')

@@ -54,11 +55,16 @@ TEMPLATES_DIR = os.path.join(PYTHON_PATH, 'templates')

 CHROME_SANDBOX =        os.getenv('CHROME_SANDBOX',         'True'             ).lower() == 'true'

+USE_CHROME = FETCH_PDF or FETCH_SCREENSHOT or FETCH_DOM
+USE_WGET = FETCH_WGET or FETCH_WGET_REQUISITES or FETCH_WARC
+
 if not CHROME_BINARY:
    common_chrome_executable_names = (
        'chromium-browser',
        'chromium',
        'google-chrome',
+        'google-chrome-stable',
+        'google-chrome-unstable',
        'google-chrome-beta',
        'google-chrome-canary',
        'google-chrome-dev',
@@ -91,12 +97,29 @@ if not USE_COLOR:
    ANSI = {k: '' for k in ANSI.keys()}

 ### Confirm Environment Setup
+GIT_SHA = 'unknown'
 try:
    GIT_SHA = run(["git", "rev-list", "-1", "HEAD", "./"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
 except Exception:
-    GIT_SHA = 'unknown'
-    print('[!] Warning, you need git installed for code version to be saved with archive json!')
+    print('[!] Warning: unable to determine git version, is git installed and in your $PATH?')

+CHROME_VERSION = 'unknown'
+try:
+    chrome_vers_str = run([CHROME_BINARY, "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    CHROME_VERSION = [v for v in chrome_vers_str.strip().split(' ') if v.replace('.', '').isdigit()][0]
+except Exception:
+    if USE_CHROME:
+        print('[!] Warning: unable to determine chrome version, is chrome installed and in your $PATH?')
+
+WGET_VERSION = 'unknown'
+try:
+    wget_vers_str = run(["wget", "--version"], stdout=PIPE, cwd=REPO_DIR).stdout.strip().decode()
+    WGET_VERSION = wget_vers_str.split('\n')[0].split(' ')[2]
+except Exception:
+    if USE_WGET:
+        print('[!] Warning: unable to determine wget version, is wget installed and in your $PATH?')
+
+WGET_USER_AGENT = WGET_USER_AGENT.format(GIT_SHA=GIT_SHA[:9], WGET_VERSION=WGET_VERSION)

 if sys.stdout.encoding.upper() not in ('UTF-8', 'UTF8'):
    print('[X] Your system is running python3 scripts with a bad locale setting: {} (it should be UTF-8).'.format(sys.stdout.encoding))
@@ -107,3 +130,4 @@ if sys.stdout.encoding.upper() not in ('UTF-8', 'UTF8'):
    print('')
    print('    Alternatively, run this script with:')
    print('        env PYTHONIOENCODING=UTF-8 ./archive.py export.html')
+
@@ -20,10 +20,12 @@ Parsed link schema: {
 import re
 import json
 import urllib
+from collections import OrderedDict
 import xml.etree.ElementTree as etree

 from datetime import datetime

+from config import ANSI
 from util import (
    domain,
    base_url,
@@ -37,22 +39,29 @@ from util import (
 def get_parsers(file):
    """return all parsers that work on a given file, defaults to all of them"""

-    return {
-        'pocket': parse_pocket_export,
-        'pinboard': parse_json_export,
-        'bookmarks': parse_bookmarks_export,
-        'rss': parse_rss_export,
-        'pinboard_rss': parse_pinboard_rss_feed,
-        'medium_rss': parse_medium_rss_feed,
-        'plain_text': parse_plain_text,
-    }
+    return OrderedDict([
+        ('Pocket HTML', parse_pocket_html_export),
+        ('Pinboard JSON', parse_pinboard_json_export),
+        ('Netscape HTML', parse_netscape_html_export),
+        ('RSS', parse_rss_export),
+        ('Pinboard RSS', parse_pinboard_rss_export),
+        ('Shaarli RSS', parse_shaarli_rss_export),
+        ('Medium RSS', parse_medium_rss_export),
+        ('Plain Text', parse_plain_text_export),
+    ])

 def parse_links(path):
    """parse a list of links dictionaries from a bookmark export file"""
    
    links = []
    with open(path, 'r', encoding='utf-8') as file:
-        for parser_func in get_parsers(file).values():
+        print('{green}[*] [{}] Parsing new links from output/sources/{} and fetching titles...{reset}'.format(
+            datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            path.rsplit('/', 1)[-1],
+            **ANSI,
+        ))
+
+        for parser_name, parser_func in get_parsers(file).items():
            # otherwise try all parsers until one works
            try:
                links += list(parser_func(file))
@@ -62,10 +71,12 @@ def parse_links(path):
                # parser not supported on this file
                pass

-    return links
+    print()
+
+    return links, parser_name


-def parse_pocket_export(html_file):
+def parse_pocket_html_export(html_file):
    """Parse Pocket-format bookmarks export files (produced by getpocket.com/export/)"""

    html_file.seek(0)
@@ -81,15 +92,15 @@ def parse_pocket_export(html_file):
                'url': fixed_url,
                'domain': domain(fixed_url),
                'base_url': base_url(fixed_url),
-                'timestamp': str(datetime.now().timestamp()),
+                'timestamp': str(time.timestamp()),
                'tags': match.group(3),
-                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or fetch_page_title(url),
+                'title': match.group(4).replace(' — Readability', '').replace('http://www.readability.com/read?url=', '') or fetch_page_title(fixed_url),
                'sources': [html_file.name],
            }
            info['type'] = get_link_type(info)
            yield info

-def parse_json_export(json_file):
+def parse_pinboard_json_export(json_file):
    """Parse JSON-format bookmarks export files (produced by pinboard.in/export/, or wallabag)"""
    json_file.seek(0)
    json_content = json.load(json_file)
@@ -166,7 +177,49 @@ def parse_rss_export(rss_file):

        yield info

-def parse_bookmarks_export(html_file):
+def parse_shaarli_rss_export(rss_file):
+    """Parse Shaarli-specific RSS XML-format files into links"""
+
+    rss_file.seek(0)
+    entries = rss_file.read().split('<entry>')[1:]
+    for entry in entries:
+        # example entry:
+        # <entry>
+        #   <title>Aktuelle Trojaner-Welle: Emotet lauert in gefÃ¤lschten Rechnungsmails | heise online</title>
+        #   <link href="https://www.heise.de/security/meldung/Aktuelle-Trojaner-Welle-Emotet-lauert-in-gefaelschten-Rechnungsmails-4291268.html" />
+        #   <id>https://demo.shaarli.org/?cEV4vw</id>
+        #   <published>2019-01-30T06:06:01+00:00</published>
+        #   <updated>2019-01-30T06:06:01+00:00</updated>
+        #   <content type="html" xml:lang="en"><![CDATA[<div class="markdown"><p>&#8212; <a href="https://demo.shaarli.org/?cEV4vw">Permalink</a></p></div>]]></content>
+        # </entry>
+
+        trailing_removed = entry.split('</entry>', 1)[0]
+        leading_removed = trailing_removed.strip()
+        rows = leading_removed.split('\n')
+
+        def get_row(key):
+            return [r.strip() for r in rows if r.strip().startswith('<{}'.format(key))][0]
+
+        title = str_between(get_row('title'), '<title>', '</title>').strip()
+        url = str_between(get_row('link'), '<link href="', '" />')
+        ts_str = str_between(get_row('published'), '<published>', '</published>')
+        time = datetime.strptime(ts_str, "%Y-%m-%dT%H:%M:%S%z")
+
+
+        info = {
+            'url': url,
+            'domain': domain(url),
+            'base_url': base_url(url),
+            'timestamp': str(time.timestamp()),
+            'tags': '',
+            'title': title or fetch_page_title(url),
+            'sources': [rss_file.name],
+        }
+        info['type'] = get_link_type(info)
+
+        yield info
+
+def parse_netscape_html_export(html_file):
    """Parse netscape-format bookmarks export files (produced by all browsers)"""

    html_file.seek(0)
@@ -184,7 +237,7 @@ def parse_bookmarks_export(html_file):
                'url': url,
                'domain': domain(url),
                'base_url': base_url(url),
-                'timestamp': str(datetime.now().timestamp()),
+                'timestamp': str(time.timestamp()),
                'tags': "",
                'title': match.group(3).strip() or fetch_page_title(url),
                'sources': [html_file.name],
@@ -193,7 +246,7 @@ def parse_bookmarks_export(html_file):

            yield info

-def parse_pinboard_rss_feed(rss_file):
+def parse_pinboard_rss_export(rss_file):
    """Parse Pinboard RSS feed files into links"""

    rss_file.seek(0)
@@ -217,7 +270,7 @@ def parse_pinboard_rss_feed(rss_file):
            'url': url,
            'domain': domain(url),
            'base_url': base_url(url),
-            'timestamp': str(datetime.now().timestamp()),
+            'timestamp': str(time.timestamp()),
            'tags': tags,
            'title': title or fetch_page_title(url),
            'sources': [rss_file.name],
@@ -225,7 +278,7 @@ def parse_pinboard_rss_feed(rss_file):
        info['type'] = get_link_type(info)
        yield info

-def parse_medium_rss_feed(rss_file):
+def parse_medium_rss_export(rss_file):
    """Parse Medium RSS feed files into links"""

    rss_file.seek(0)
@@ -242,7 +295,7 @@ def parse_medium_rss_feed(rss_file):
            'url': url,
            'domain': domain(url),
            'base_url': base_url(url),
-            'timestamp': str(datetime.now().timestamp()),
+            'timestamp': str(time.timestamp()),
            'tags': '',
            'title': title or fetch_page_title(url),
            'sources': [rss_file.name],
@@ -251,7 +304,7 @@ def parse_medium_rss_feed(rss_file):
        yield info


-def parse_plain_text(text_file):
+def parse_plain_text_export(text_file):
    """Parse raw links from each line in a text file"""

    text_file.seek(0)
@@ -261,6 +314,7 @@ def parse_plain_text(text_file):
            urls = re.findall(URL_REGEX, line)
            
            for url in urls:
+                url = url.strip()
                info = {
                    'url': url,
                    'domain': domain(url),
@@ -1,3 +1,5 @@
+from sys import maxsize
+from itertools import islice
 from collections import deque

 _marker = object()
@@ -1,54 +1,86 @@
 #!/usr/bin/env python3
-import argparse
+
 import re
+from argparse import ArgumentParser
+from os.path import exists, join
+from shutil import rmtree
 from typing import List

 from archive import parse_json_link_index
-from config import OUTPUT_DIR
-from index import write_json_links_index
+from config import ARCHIVE_DIR, OUTPUT_DIR
+from index import write_html_links_index, write_json_links_index


-def cleanup_index(patterns: List[str], yes=False):
-    regexes = [re.compile(p) for p in patterns]
-
-    index = parse_json_link_index(OUTPUT_DIR)
-    links = index['links']
+def cleanup_index(regexes: List[str], proceed: bool, delete: bool) -> None:
+    if not exists(join(OUTPUT_DIR, 'index.json')):
+        exit('index.json is missing; nothing to do')

+    compiled = [re.compile(r) for r in regexes]
+    links = parse_json_link_index(OUTPUT_DIR)['links']
    filtered = []
    remaining = []
+
    for l in links:
        url = l['url']
-        for r in regexes:
+        for r in compiled:
            if r.search(url):
                filtered.append((l, r))
                break
        else:
            remaining.append(l)

+    if not filtered:
+        exit('Search did not match any entries.')
+
+    print('Filtered out {}/{} urls:'.format(len(filtered), len(links)))

-    print("Filtered out {}/{} urls:".format(len(filtered), len(links)))
    for link, regex in filtered:
        url = link['url']
-        print(" {url} via {regex}".format(url=url, regex=regex.pattern))
+        print(' {url} via {regex}'.format(url=url, regex=regex.pattern))

-    proceed = False
-    if yes:
-        proceed = True
-    else:
-        res = input("Remove {} entries from index? [y/n] ".format(len(filtered)))
-        proceed = res.strip().lower() in ('y', 'yes')
+    if not proceed:
+        answer = input('Remove {} entries from index? [y/n] '.format(
+            len(filtered)))
+        proceed = answer.strip().lower() in ('y', 'yes')

-    if proceed:
-        write_json_links_index(OUTPUT_DIR, remaining)
-    else:
-        exit('aborting')
+    if not proceed:
+        exit('Aborted')
+
+    write_json_links_index(OUTPUT_DIR, remaining)
+    write_html_links_index(OUTPUT_DIR, remaining)
+
+    if delete:
+        for link, _ in filtered:
+            data_dir = join(ARCHIVE_DIR, link['timestamp'])
+            if exists(data_dir):
+                rmtree(data_dir)


 if __name__ == '__main__':
-    p = argparse.ArgumentParser('Index purging tool')
-    p.add_argument('--regex', '-r', action='append', help='Python regex to filter out')
-    p.add_argument('--yes', action='store_true', default=False, help='Do not propmpt for confirmation')
+    p = ArgumentParser('Index purging tool')
+    p.add_argument(
+        '--regex',
+        '-r',
+        action='append',
+        help='Regular expression matching URLs to purge',
+    )
+    p.add_argument(
+        '--delete',
+        '-d',
+        action='store_true',
+        default=False,
+        help='Delete webpage files from archive',
+    )
+    p.add_argument(
+        '--yes',
+        '-y',
+        action='store_true',
+        default=False,
+        help='Do not prompt for confirmation',
+    )

    args = p.parse_args()
-    regexes = args.regex
-    cleanup_index(regexes, yes=args.yes)
+    if args.regex:
+        cleanup_index(args.regex, proceed=args.yes, delete=args.delete)
+    else:
+        p.print_help()
@@ -15,49 +15,46 @@
            }
            header {
                background-color: #aa1e55;
-                color: white;
+                color: #1a1a1a;
                padding: 10px;
                padding-top: 0px;
                padding-bottom: 15px;
-                height: 100px;
+                /*height: 40px;*/
            }
            header h1 {
-                font-size: 38px;
+                margin: 7px 0px;
+                font-size: 35px;
                font-weight: 300;
-                color: black;
-                padding-top: 14px;
-                line-height: 1.4;
-                width: 100%;
+                color: #1a1a1a;
            }
-            header h1 small {
-                color: white;
-                font-size:0.45em;
-                margin-left: 10px;
-                display: block;
+            header h1 img {
+                height: 44px;
+                vertical-align: bottom;
            }
-            header h1 small a {
-                text-decoration: none;
-                color: orange;
-                opacity: 0.6;
-                font-weight: 300;
-            }
-            header h1 small a:hover {
-                opacity: 1;
-            }
-            header + div {
-                margin-top: 10px;
+            header a {
+                text-decoration: none !important;
+                color: #1a1a1a;
            }
            .header-center {
-                width: 100%;
+                margin: auto;
+                float: none;
                text-align: center;
+                padding-top: 6px;
+            }
+            .header-center small {
+                color: #eaeaea;
+                opacity: 0.7;
            }
            .header-left {
                float: left;
-                width: 50px;
-                height: 60px;
-                text-align: center;
-                padding: 20px;
-                margin-right: -70px;
+            }
+            .header-right {
+                float: right;
+                padding-top: 17px;
+                padding-right: 10px;
+            }
+            header + div {
+                margin-top: 10px;
            }
            #table-bookmarks_length, #table-bookmarks_filter {
                padding: 0px 15px;
@@ -105,23 +102,19 @@
    </head>
    <body>
        <header>
+            <div class="header-right">
+                <a href="https://github.com/pirate/ArchiveBox/wiki">Documentation</a> &nbsp; | &nbsp; 
+                <a href="https://github.com/pirate/ArchiveBox">Source</a> &nbsp; | &nbsp; 
+                <a href="https://archivebox.io">Website</a>
+            </div>
            <div class="header-left">
                <a href="?" title="Reload...">
-                    <img src="static/archive.png" style="height: 100%;"/>
-                </a>
-                <br/>
-                <a href="https://github.com/pirate/ArchiveBox">
-                    Github
+                    <h1><img src="static/archive.png"/> ArchiveBox: Index</h1>
                </a>
            </div>
            <div class="header-center">
-                <h1>
-                    &nbsp;Archived Sites
-                    <br/>
-                    <small>
-                        <a href="?">Last updated $time_updated</a><br/>
-                    </small>
-                </h1>
+                Archived Sites<br/>
+                <small>Last updated $time_updated</small>
            </div>
        </header>
        <table id="table-bookmarks">
@@ -41,7 +41,7 @@
                margin-right: 13px;
                color: black;
                height: 53px;
-                margin-top: 7px;
+                margin-top: 12px;
                margin-left: 10px;
            }
            .nav-icon img:hover {
@@ -0,0 +1,12 @@
+<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Pinboard Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL>
+<p>
+
+<DT><A HREF="https://github.com/trailofbits/algo" ADD_DATE="1542616733" PRIVATE="1" TOREAD="1" TAGS="vpn,scripts,toread">Algo VPN scripts</A>
+<DT><A HREF="http://www.ulisp.com/" ADD_DATE="1542374412" PRIVATE="1" TOREAD="1" TAGS="arduino,avr,embedded,lisp,toread">uLisp</A>
+
+</DL>
+</p>
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" xmlns:syn="http://purl.org/rss/1.0/modules/syndication/" xmlns:admin="http://webns.net/mvcb/">
+  <channel rdf:about="http://pinboard.in">
+    <title>Pinboard (private aaronmueller)</title>
+    <link>https://pinboard.in/u:aaronmueller/private/</link>
+    <description></description>
+    <items>
+      <rdf:Seq>
+        <rdf:li rdf:resource="https://mehkee.com/"/>
+        <rdf:li rdf:resource="https://qmk.fm/"/>
+      </rdf:Seq>
+    </items>
+  </channel>
+
+  <item rdf:about="https://mehkee.com/">
+    <title>Mehkee - Mechanical Keyboard Parts &amp; Accessories</title>
+    <dc:date>2018-11-08T21:29:32+00:00</dc:date>
+    <link>https://mehkee.com/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>keyboard gadget diy</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:gadget"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:diy"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+  <item rdf:about="https://qmk.fm/">
+    <title>QMK Firmware - An open source firmware for AVR and ARM based keyboards</title>
+    <dc:date>2018-11-06T22:36:21+00:00</dc:date>
+    <link>https://qmk.fm/</link>
+    <dc:creator>aaronmueller</dc:creator>
+    <dc:subject>firmware keyboard</dc:subject>
+    <dc:source>http://pinboard.in/</dc:source>
+    <dc:identifier>http://pinboard.in/u:aaronmueller/b:xxx/</dc:identifier>
+    <taxo:topics>
+      <rdf:Bag>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:firmware"/>
+        <rdf:li rdf:resource="http://pinboard.in/u:aaronmueller/t:keyboard"/>
+      </rdf:Bag>
+    </taxo:topics>
+  </item>
+</rdf:RDF>
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+    <posts user="aaronmueller">
+<post href="https://github.com/trailofbits/algo" time="2018-11-19T08:38:53Z" description="Algo VPN scripts" extended="" tag="vpn scripts" hash="18d708f67bb26d843b1cac4530bb52aa"  shared="no" toread="yes" />
+<post href="http://www.ulisp.com/" time="2018-11-16T13:20:12Z" description="uLisp" extended="" tag="arduino avr embedded lisp" hash="2a17ae95925a03a5b9bb38cf7f6c6f9b"  shared="no" toread="yes" />
+</posts>
@@ -0,0 +1,2 @@
+[{"href":"https:\/\/github.com\/trailofbits\/algo","description":"Algo VPN scripts","extended":"","meta":"62325ba3b577683aee854d7f191034dc","hash":"18d708f67bb26d843b1cac4530bb52aa","time":"2018-11-19T08:38:53Z","shared":"no","toread":"yes","tags":"vpn scripts"},
+{"href":"http:\/\/www.ulisp.com\/","description":"uLisp","extended":"","meta":"7bd0c0ef31f69d1459e3d37366e742b3","hash":"2a17ae95925a03a5b9bb38cf7f6c6f9b","time":"2018-11-16T13:20:12Z","shared":"no","toread":"yes","tags":"arduino avr embedded lisp"}]
@@ -3,12 +3,13 @@ import re
 import sys
 import time
 import json
+import signal
 import urllib.request

 from decimal import Decimal
 from urllib.parse import quote
 from datetime import datetime
-from subprocess import run, PIPE, DEVNULL
+from subprocess import TimeoutExpired, Popen, PIPE, DEVNULL, CompletedProcess, CalledProcessError
 from multiprocessing import Process

 from config import (
@@ -28,8 +29,7 @@ from config import (
    FETCH_SCREENSHOT,
    FETCH_DOM,
    FETCH_FAVICON,
-    FETCH_AUDIO,
-    FETCH_VIDEO,
+    FETCH_MEDIA,
    SUBMIT_ARCHIVE_DOT_ORG,
 )

@@ -43,7 +43,7 @@ base_url = lambda url: without_scheme(url)  # uniq base url used to dedupe links

 short_ts = lambda ts: ts.split('.')[0]

-URL_REGEX = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'
+URL_REGEX = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))[^<\""]+'


 def check_dependencies():
@@ -93,7 +93,7 @@ def check_dependencies():
            print('    See https://github.com/pirate/ArchiveBox for help.')
            raise SystemExit(1)

-    if FETCH_AUDIO or FETCH_VIDEO:
+    if FETCH_MEDIA:
        if run(['which', 'youtube-dl'], stdout=DEVNULL).returncode or run(['youtube-dl', '--version'], stdout=DEVNULL).returncode:
            print('{red}[X] Missing dependency: youtube-dl{reset}'.format(**ANSI))
            print('    Run ./setup.sh, then confirm it was installed with: {} --version'.format('youtube-dl'))
@@ -121,11 +121,10 @@ def progress(seconds=TIMEOUT, prefix=''):
    if not SHOW_PROGRESS:
        return lambda: None

-    chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
-    chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
-
-    def progress_bar(seconds=seconds, prefix=prefix):
+    def progress_bar(seconds, prefix):
        """show timer in the form of progress bar, with percentage and seconds remaining"""
+        chunk = '█' if sys.stdout.encoding == 'UTF-8' else '#'
+        chunks = TERM_WIDTH - len(prefix) - 20  # number of progress chunks to show (aka max bar width)
        try:
            for s in range(seconds * chunks):
                progress = s / chunks / seconds * 100
@@ -159,16 +158,18 @@ def progress(seconds=TIMEOUT, prefix=''):
            print()
            pass

-    p = Process(target=progress_bar)
+    p = Process(target=progress_bar, args=(seconds, prefix))
    p.start()

    def end():
        """immediately finish progress and clear the progressbar line"""
-        nonlocal p
-        if p is None: # protect from double termination
-            return

-        p.terminate()
+        # protect from double termination
+        #if p is None or not hasattr(p, 'kill'):
+        #    return
+        nonlocal p
+        if p is not None:
+            p.terminate()
        p = None

        sys.stdout.write('\r{}{}\r'.format((' ' * TERM_WIDTH), ANSI['reset']))  # clear whole terminal line
@@ -181,6 +182,20 @@ def pretty_path(path):
    return path.replace(REPO_DIR + '/', '')


+def save_source(raw_text):
+    if not os.path.exists(SOURCES_DIR):
+        os.makedirs(SOURCES_DIR)
+
+    ts = str(datetime.now().timestamp()).split('.', 1)[0]
+
+    source_path = os.path.join(SOURCES_DIR, '{}-{}.txt'.format('stdin', ts))
+
+    with open(source_path, 'w', encoding='utf-8') as f:
+        f.write(raw_text)
+
+    return source_path
+
+
 def download_url(url):
    """download a given url's content into downloads/domain.txt"""

@@ -218,8 +233,9 @@ def fetch_page_title(url, default=True):
        default = url

    try:
-        html_content = urllib.request.urlopen(url).read().decode('utf-8')
-
+        sys.stdout.write('.')
+        sys.stdout.flush()
+        html_content = urllib.request.urlopen(url, timeout=10).read().decode('utf-8')
        match = re.search('<title>(.*?)</title>', html_content)
        return match.group(1) if match else default or None
    except Exception:
@@ -518,3 +534,39 @@ def derived_link_info(link):
            'title': '{title} ({type})'.format(**link),
        })
    return link_info
+
+
+def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs):
+    """Patched of subprocess.run to fix blocking io making timeout=innefective"""
+
+    if input is not None:
+        if 'stdin' in kwargs:
+            raise ValueError('stdin and input arguments may not both be used.')
+        kwargs['stdin'] = PIPE
+
+    if capture_output:
+        if ('stdout' in kwargs) or ('stderr' in kwargs):
+            raise ValueError('stdout and stderr arguments may not be used '
+                             'with capture_output.')
+        kwargs['stdout'] = PIPE
+        kwargs['stderr'] = PIPE
+
+    with Popen(*popenargs, **kwargs) as process:
+        try:
+            stdout, stderr = process.communicate(input, timeout=timeout)
+        except TimeoutExpired:
+            process.kill()
+            try:
+                stdout, stderr = process.communicate(input, timeout=2)
+            except:
+                pass
+            raise TimeoutExpired(popenargs[0][0], timeout)
+        except BaseException as err:
+            process.kill()
+            # We don't call process.wait() as .__exit__ does that for us.
+            raise 
+        retcode = process.poll()
+        if check and retcode:
+            raise CalledProcessError(retcode, process.args,
+                                     output=stdout, stderr=stderr)
+    return CompletedProcess(process.args, retcode, stdout, stderr)
@@ -10,6 +10,7 @@ echo "        - Chromium Browser   (see README for Google-Chrome instructions in
 echo "        - python3"
 echo "        - wget"
 echo "        - curl"
+echo "        - youtube-dl"
 echo ""
 echo "    You may follow Manual Setup instructions in README.md instead if you prefer not to run an unknown script."
 echo "    Press enter to continue, or Ctrl+C to cancel..."
@@ -33,7 +34,7 @@ if which apt-get > /dev/null; then
        apt install chromium-browser -y
    fi
    echo "[+] Installing python3, wget, curl..."
-    apt install -y python3 python3-distutils wget curl
+    apt install -y python3 python3-distutils wget curl youtube-dl

 # On Mac:
 elif which brew > /dev/null; then   # 🐍 eye of newt
@@ -56,7 +57,7 @@ elif which brew > /dev/null; then   # 🐍 eye of newt
        chmod +x /usr/local/bin/chromium-browser
    fi
    echo "[+] Installing python3, wget, curl  (ignore 'already installed' warnings)..."
-    brew install python3 wget curl
+    brew install python3 wget curl youtube-dl
 else
    echo "[X] Could not find aptitude or homebrew! ‼️"
    echo ""
@@ -0,0 +1,21 @@
+version: '3'
+
+services:
+    archivebox:
+        build: .
+        stdin_open: true
+        tty: true
+        environment:
+            - USE_COLOR=False
+            - SHOW_PROGRESS=False
+        volumes:
+            - ./data:/data
+        command: bash -c 'echo "https://github.com/pirate/ArchiveBox" | /bin/archive; tail -f /dev/null'
+
+    nginx:
+        image: 'nginx'
+        ports:
+            - '8098:80'
+        volumes:
+            - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf
+            - ./data:/var/www
@@ -0,0 +1,49 @@
+# Config file uses python syntax but can also be sourced as a .env
+
+### To disable specific archive methods, uncomment the line and change it to False
+# FETCH_WGET=True
+# FETCH_WGET_REQUISITES=True
+# FETCH_PDF=True
+# FETCH_SCREENSHOT=True
+# FETCH_DOM=True
+# FETCH_WARC=False
+# FETCH_MEDIA=False
+# FETCH_GIT=True
+# FETCH_FAVICON=True
+# SUBMIT_ARCHIVE_DOT_ORG=True
+
+### To only download new links, and never attempt to update old ones, uncomment this line:
+# ONLY_NEW=True
+
+### To change your ouput folder permissions, uncomment and edit this line:
+# OUTPUT_PERMISSIONS=755
+
+### To change how much time is allowed before marking a link as failed, uncomment and edit these lines:
+# MEDIA_TIMEOUT=3600
+# TIMEOUT=60
+
+### To ignore SSL certificate errors, uncomment this line:
+# CHECK_SSL_VALIDITY=False
+
+### To customize the screenshot resolution, uncomment and change this line:
+# RESOLUTION="1440,2000"
+
+### To customize which domains we attempted to `git clone`, uncomment and change this line:
+# GIT_DOMAINS="github.com,bitbucket.org,gitlab.com"
+
+### To lie and pretend to be Google Chrome when archiving, uncomment this line:
+# WGET_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
+
+### To share a chrome data directory with your main browser, uncomment one of these lines:
+# CHROME_USER_DATA_DIR="~/Library/Application Support/Google/Chrome/Default"
+# CHROME_USER_DATA_DIR="~/.config/google-chrome/Default"
+
+### To use a custom chrome binary, uncomment and set to binary name or full path:
+# CHROME_BINARY="chromium-browser"
+
+### To use a custom wget binary, uncomment and set to binary name or full path:
+# WGET_BINARY="wget"
+
+### To customize Text displayed in html output footer, uncomment and edit this line:
+# FOOTER_INFO="Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests."
+
@@ -1,10 +0,0 @@
-server {
-    listen                      80 default_server;
-    server_name                 _;
-
-    root                        /opt/ArchiveBox/output/;
-    index                       index.html;
-    autoindex                   on;
-
-    try_files                   $uri $uri/ $uri.html =404;
-}
@@ -0,0 +1,45 @@
+user www-data;
+pid                                 /var/run/nginx.pid;
+
+worker_processes                    auto;
+timer_resolution                    100ms;
+
+worker_rlimit_nofile                40000;
+events {
+    worker_connections              40000;
+    use epoll;
+    multi_accept on;
+}
+
+http {
+    sendfile on;
+    tcp_nopush on;
+    tcp_nodelay on;
+    server_tokens off;
+    send_timeout 20;
+    keepalive_timeout 65;
+    types_hash_max_size 2048;
+    client_max_body_size 25m;
+
+    include                     mime.types;
+    default_type                application/octet-stream;
+
+    access_log                  /dev/stdout;
+    error_log                   stderr;
+
+    reset_timedout_connection on;
+
+    server_names_hash_bucket_size   64;
+
+    server {
+        listen                      80 default_server;
+        server_name                 _;
+
+        root                        /var/www;
+        index                       index.html;
+        autoindex                   on;
+
+        try_files                   $uri $uri/ $uri.html =404;
+    }
+}
+
Author	SHA1	Message	Date
Nick Sweeting	2496564750	Update README.md	2019-02-06 15:40:53 -08:00
Nick Sweeting	ce257949b4	fix double parsing by accident	2019-02-04 22:08:08 -08:00
Nick Sweeting	1ca27a60b2	show dots as progress bar for titles fetching	2019-02-04 22:07:59 -08:00
Nick Sweeting	ee93807a0a	tweak wording of parser cli output	2019-02-04 22:07:48 -08:00
Nick Sweeting	ca8f57ef5c	add dedicated shaarli rss parser	2019-02-04 21:18:42 -08:00
Nick Sweeting	522b6288c1	hide frequent youtubedl incompatibility	2019-02-04 21:18:17 -08:00
root	fb00f1f336	fix flickering progress bars	2019-02-05 04:45:36 +00:00
Nick Sweeting	bc6de29a5a	recreate chunks on each new progress bar	2019-02-04 20:41:35 -08:00
Nick Sweeting	78ffa57c06	ignore hsts as well	2019-02-04 20:37:19 -08:00
Nick Sweeting	637f63dc80	chmod media after downloading	2019-02-04 20:15:34 -08:00
Nick Sweeting	756f31a50b	hide common youtubedl failures	2019-02-04 20:08:54 -08:00
Nick Sweeting	2b45792ad8	new test files	2019-02-04 20:04:03 -08:00
Nick Sweeting	265e1bd830	fix parsing links from stdin with whitespace	2019-02-04 20:03:32 -08:00
Nick Sweeting	794c043fa4	fix chrome data dir in wrong folder	2019-02-04 20:03:16 -08:00
Nick Sweeting	9f5441c4f3	quieter media downloads	2019-02-04 20:03:06 -08:00
Nick Sweeting	4e5b466bb2	fix broken wget user agents	2019-02-04 20:02:54 -08:00
Nick Sweeting	d8c7c98c01	fix exception format string	2019-02-04 20:02:40 -08:00
Nick Sweeting	705abda1a5	send ArchiveBox user agent to archive.org	2019-02-04 20:02:33 -08:00
Nick Sweeting	8a9b4d6975	dont git clone urls with hashes	2019-02-04 19:21:08 -08:00
Nick Sweeting	b8e5e1d5bd	skip media when folder exists, even if empty	2019-02-04 19:21:00 -08:00
Nick Sweeting	901666bae6	dont show wget errors unless all files failed to download	2019-02-04 19:20:36 -08:00
Nick Sweeting	3ac0efb619	dont parse quotes as part of urls	2019-02-04 18:54:02 -08:00
Nick Sweeting	af8b9b5fdf	use parsed timestamps instead of now	2019-02-04 18:53:54 -08:00
Nick Sweeting	21c62043e3	new logo	2019-02-04 18:45:46 -08:00
Nick Sweeting	543d2fc010	tweak index visual header style	2019-02-04 18:45:36 -08:00
Nick Sweeting	c969451a1c	dont clobber pocket timestamps	2019-02-04 18:45:25 -08:00
Nick Sweeting	c32a385e8f	show which format file was parsed as	2019-02-04 18:45:18 -08:00
Nick Sweeting	ae0c20dc76	fix flake8 errors	2019-02-04 17:52:53 -08:00
Nick Sweeting	eeb89ae2cc	Merge pull request #138 from luoliyan/improve-purge-script Improve index purge script	2019-02-04 17:49:22 -08:00
luoliyan	fee4565194	Improve index purge script	2019-02-05 06:48:49 +09:30
Nick Sweeting	a38da8d0d1	typo	2019-02-04 10:59:19 -08:00
Nick Sweeting	ae8445ad21	Update README.md	2019-02-04 10:21:52 -08:00
Nick Sweeting	6de7832c4f	Update README.md	2019-02-04 10:18:36 -08:00
Nick Sweeting	aa8b518fe5	Update README.md	2019-02-04 10:18:02 -08:00
Nick Sweeting	96d356d860	Update README.md	2019-02-04 10:17:02 -08:00
Nick Sweeting	aad0207d6e	Update README.md	2019-02-04 10:16:35 -08:00
Nick Sweeting	1f4d0d2d16	Update README.md	2019-02-04 10:16:00 -08:00
Nick Sweeting	276f2fd532	Update README.md	2019-02-04 10:14:51 -08:00
Nick Sweeting	e21227976a	Update README.md	2019-02-04 10:12:57 -08:00
Nick Sweeting	ad7038e031	catch json parse errors in link archiving	2019-02-04 08:00:08 -08:00
Nick Sweeting	c37941efd1	dont match closing tags in full text	2019-01-31 19:46:34 -08:00
Nick Sweeting	ee796b287b	Update README.md	2019-01-30 00:48:28 -08:00
Nick Sweeting	e9a5f19f7c	Update README.md	2019-01-30 00:47:58 -08:00
Nick Sweeting	319de95915	Update README.md	2019-01-29 20:11:10 -08:00
Nick Sweeting	c79e0cc0a6	Update README.md	2019-01-29 20:09:38 -08:00
Nick Sweeting	2156c6951b	Update README.md	2019-01-29 20:07:48 -08:00
Nick Sweeting	9eb79258bb	check chrome version on startup if using chrome	2019-01-29 17:08:15 -08:00
Nick Sweeting	95ba5a90ee	Update docker-compose.yml	2019-01-29 16:55:50 -08:00
Nick Sweeting	81240cf58d	update docs	2019-01-29 16:37:51 -08:00
Nick Sweeting	cd9e20d199	Update README.md	2019-01-28 01:52:04 -08:00
Nick Sweeting	b6a212a887	Update README.md	2019-01-28 01:45:22 -08:00
Nick Sweeting	64eb6ff86b	Update README.md	2019-01-28 01:42:18 -08:00
Nick Sweeting	d2d001fe7a	Update README.md	2019-01-28 01:40:42 -08:00
Nick Sweeting	c3e7057bbd	Update README.md	2019-01-28 01:39:21 -08:00
Nick Sweeting	b53251fe45	fix parsers	2019-01-26 15:09:58 -08:00
Nick Sweeting	6a46e54282	bump node image to latest version	2019-01-26 15:09:58 -08:00
Nick Sweeting	b369a6df58	Update README.md	2019-01-25 21:45:44 -08:00
Nick Sweeting	d317baf42e	Update README.md	2019-01-25 21:44:44 -08:00
Nick Sweeting	c4b9d60f84	Update README.md	2019-01-25 21:44:24 -08:00
Nick Sweeting	c940dcc0db	Update README.md	2019-01-25 21:43:41 -08:00
Nick Sweeting	1d16cd2bf6	Update README.md	2019-01-25 21:39:13 -08:00
Nick Sweeting	7924028476	Update README.md	2019-01-25 21:38:38 -08:00
Nick Sweeting	48b373a4a1	Update README.md	2019-01-25 21:30:18 -08:00
Nick Sweeting	63ec4cf498	Update README.md	2019-01-25 21:29:33 -08:00
Nick Sweeting	ff125d990c	fix order of parsers not enforced on python 3.5	2019-01-25 21:02:25 -08:00
Nick Sweeting	e60070dbb2	add youtubedl to help str	2019-01-25 17:38:47 -08:00
Nick Sweeting	6e4728e193	Update README.md	2019-01-25 14:31:01 -08:00
Nick Sweeting	544de68314	Update README.md	2019-01-23 05:33:30 -05:00
Nick Sweeting	556ffc9ad4	Update feature_request.md	2019-01-23 05:10:41 -05:00
Nick Sweeting	d4ed256d19	Update README.md	2019-01-23 03:06:40 -05:00
Nick Sweeting	e5385bd95f	Update README.md	2019-01-23 03:04:34 -05:00
Nick Sweeting	98dd4a69c2	Update README.md	2019-01-23 02:53:53 -05:00
Nick Sweeting	cd53202b16	Update README.md	2019-01-23 02:21:01 -05:00
Nick Sweeting	fe2c8cb792	Update README.md	2019-01-23 02:17:39 -05:00
Nick Sweeting	e41657480f	Update README.md	2019-01-23 02:16:28 -05:00
Nick Sweeting	3798f5dd8e	Update README.md	2019-01-23 02:15:29 -05:00
Nick Sweeting	ed17034f68	Update README.md	2019-01-23 02:13:18 -05:00
Nick Sweeting	6aae581705	Update README.md	2019-01-23 02:09:11 -05:00
Nick Sweeting	47596664ed	Update README.md	2019-01-23 02:08:29 -05:00
Nick Sweeting	20de451515	add linux config example	2019-01-23 01:48:04 -05:00
Nick Sweeting	95301c9306	better default config	2019-01-23 01:42:55 -05:00
Nick Sweeting	a5fe91da9d	add default config	2019-01-23 01:30:10 -05:00
Nick Sweeting	2c30cd9468	better closure	2019-01-23 01:08:23 -05:00
Nick Sweeting	db0461ec4e	add timeout to title fetching	2019-01-23 01:08:23 -05:00
Nick Sweeting	e1be96e597	working docker-compose with google chrome	2019-01-23 01:08:23 -05:00
Nick Sweeting	2c375eab49	Update README.md	2019-01-22 23:53:46 -05:00
Nick Sweeting	28611bc11f	Update README.md	2019-01-22 23:53:01 -05:00
Nick Sweeting	1f4cff95a3	Update README.md	2019-01-22 23:52:29 -05:00
Nick Sweeting	c325a8f312	Update README.md	2019-01-22 23:51:37 -05:00
Nick Sweeting	3e2b3421eb	Update README.md	2019-01-22 23:50:36 -05:00
Nick Sweeting	d0948c7902	Update README.md	2019-01-22 23:49:30 -05:00
Nick Sweeting	c9c7e9dc67	Update README.md	2019-01-22 23:47:29 -05:00
Nick Sweeting	5d50e0dcd8	Update README.md	2019-01-22 23:46:49 -05:00
Nick Sweeting	af0f7bad63	Update README.md	2019-01-22 23:36:37 -05:00
Nick Sweeting	ec48549fff	Update README.md	2019-01-22 23:24:30 -05:00
Nick Sweeting	9347cc7001	Update README.md	2019-01-22 23:22:02 -05:00
Nick Sweeting	1a0b38c583	Update README.md	2019-01-22 23:21:28 -05:00
Nick Sweeting	6fdfa18013	Update README.md	2019-01-22 23:20:00 -05:00
Nick Sweeting	cc8b1620cb	Update README.md	2019-01-22 23:19:01 -05:00
Nick Sweeting	66100f45d5	Update README.md	2019-01-22 23:13:38 -05:00
Nick Sweeting	dd7d383e81	Update README.md	2019-01-22 23:12:48 -05:00
Nick Sweeting	a8c8793693	Update README.md	2019-01-22 23:09:33 -05:00
Nick Sweeting	ead4142b7b	Update README.md	2019-01-22 23:09:06 -05:00
Nick Sweeting	7e791e143f	Update README.md	2019-01-22 23:07:47 -05:00
Nick Sweeting	948f9e5350	Update README.md	2019-01-22 23:07:10 -05:00
Nick Sweeting	d32c6b3a55	Update README.md	2019-01-22 23:01:13 -05:00
Nick Sweeting	74256b7a71	Update README.md	2019-01-22 22:59:52 -05:00
Nick Sweeting	7e8962dc36	Update README.md	2019-01-22 22:58:04 -05:00
Nick Sweeting	680f85e068	Update README.md	2019-01-22 22:56:17 -05:00
Nick Sweeting	f8a617be81	Update README.md	2019-01-22 22:55:42 -05:00
Nick Sweeting	0cfe1b0b1c	Update README.md	2019-01-22 22:53:05 -05:00
Nick Sweeting	110934cef0	Update README.md	2019-01-22 22:36:42 -05:00
Nick Sweeting	01274568bc	Update README.md	2019-01-22 22:36:07 -05:00
Nick Sweeting	58e5bb3030	Update README.md	2019-01-22 22:35:38 -05:00
Nick Sweeting	cc86bf7379	Update README.md	2019-01-22 22:35:00 -05:00
Nick Sweeting	2482bbb820	Update README.md	2019-01-22 22:31:17 -05:00
Nick Sweeting	2366fdb6d2	Update README.md	2019-01-22 22:29:23 -05:00
Nick Sweeting	5d3e8236c0	Update README.md	2019-01-22 22:27:38 -05:00
Nick Sweeting	d80cb3e9bf	Update README.md	2019-01-22 22:26:46 -05:00
Nick Sweeting	07b98c7cda	Update README.md	2019-01-22 22:25:06 -05:00
Nick Sweeting	529c1610ab	Update README.md	2019-01-22 22:22:38 -05:00
Nick Sweeting	35419015d9	Update README.md	2019-01-22 22:19:40 -05:00
Nick Sweeting	757ee2e4de	Update README.md	2019-01-22 22:18:59 -05:00
Nick Sweeting	73ed3fc79a	Update README.md	2019-01-22 22:14:38 -05:00
Nick Sweeting	6bfa7d2023	Update README.md	2019-01-22 22:13:52 -05:00
Nick Sweeting	350fddf58b	Update README.md	2019-01-22 22:13:25 -05:00
Nick Sweeting	52149e18d5	Update README.md	2019-01-22 22:12:10 -05:00
Nick Sweeting	63c1f5f1b4	Update README.md	2019-01-22 22:11:38 -05:00
Nick Sweeting	71f70b8115	Update README.md	2019-01-22 22:10:29 -05:00
Nick Sweeting	b7afc9a09e	Update README.md	2019-01-22 22:09:33 -05:00
Nick Sweeting	29a5724889	Update README.md	2019-01-22 22:09:08 -05:00
Nick Sweeting	e94cfdc90d	Update README.md	2019-01-22 22:08:27 -05:00
Nick Sweeting	1f80535038	Update README.md	2019-01-22 22:06:25 -05:00
Nick Sweeting	e82f2fdbe9	Update README.md	2019-01-22 22:05:11 -05:00
Nick Sweeting	db0078d430	Update README.md	2019-01-22 22:04:40 -05:00
Nick Sweeting	c21c2aa34a	Update README.md	2019-01-22 22:00:21 -05:00
Nick Sweeting	e34f1c87ca	Update README.md	2019-01-22 21:59:06 -05:00
Nick Sweeting	67faaeb0c7	Update README.md	2019-01-22 21:58:24 -05:00
Nick Sweeting	21e87d975f	Update README.md	2019-01-22 21:56:19 -05:00
Nick Sweeting	9de17e1a21	reorder readme	2019-01-22 21:48:48 -05:00
Nick Sweeting	1213fcdda2	Update README.md	2019-01-22 21:44:14 -05:00
Nick Sweeting	5d397ad18b	Update README.md	2019-01-22 21:40:23 -05:00
Nick Sweeting	bf48dacbbe	add motivation section	2019-01-22 21:31:42 -05:00
Nick Sweeting	5b58f612c1	badges galore	2019-01-22 21:01:37 -05:00
Nick Sweeting	2c76f29b78	better issue template	2019-01-22 19:16:09 -05:00
Nick Sweeting	74e925da4b	update docs version	2019-01-20 14:08:49 -05:00
Nick Sweeting	f25be8bc24	add chrome user agent example in config	2019-01-20 14:08:33 -05:00
Nick Sweeting	071b39b944	nicer error formats	2019-01-20 14:08:33 -05:00
Nick Sweeting	de6eb649e0	disable wget timestamping if using warcs	2019-01-20 14:08:33 -05:00
Nick Sweeting	ce07a6c738	patch subprocess.run to have better timeout handling	2019-01-20 14:08:33 -05:00
Nick Sweeting	e620722bb9	fix timeouts hanging the progressbar	2019-01-20 14:08:33 -05:00
Nick Sweeting	7ea36c4adb	bump timeouts and improve curl archive method	2019-01-20 14:08:33 -05:00
Nick Sweeting	370adb56bf	Update README.md	2019-01-18 17:45:48 -05:00
Nick Sweeting	ea905d68bb	fix etc nginx and Docker config	2019-01-16 09:20:15 -05:00
Nick Sweeting	d24724ed79	Update Dockerfile	2019-01-16 08:05:01 -05:00
Nick Sweeting	0cb915cd24	Update README.md	2019-01-15 18:35:11 -05:00
Nick Sweeting	cc8611de83	Update config.py	2019-01-14 22:46:35 -05:00
Nick Sweeting	c42fcd42d7	fetch warc file inline with wget instead of as separate step	2019-01-14 22:43:14 -05:00
Nick Sweeting	2d86d4dfe0	update docs	2019-01-14 22:43:14 -05:00
Nick Sweeting	3186f24867	Update README.md	2019-01-14 19:12:08 -05:00
Nick Sweeting	45c499dc9e	Update README.md	2019-01-14 18:19:30 -05:00
Nick Sweeting	300b5c6182	put ArchiveBox and wget version in user agent	2019-01-14 18:17:30 -05:00
Nick Sweeting	39f5e12364	allow passing import list via stdin	2019-01-14 18:11:48 -05:00
Nick Sweeting	cb60bad1d7	disable WARC by default	2019-01-12 20:19:17 -05:00
Nick Sweeting	321cdeda11	add mirror to git clone	2019-01-12 20:13:13 -05:00
Nick Sweeting	c5dcb68a6d	install youtubedl in setup	2019-01-12 03:14:07 -05:00
Nick Sweeting	6189b1a8af	fix invalid old config vars	2019-01-12 03:13:55 -05:00
Nick Sweeting	b650c663a0	re-enable checking SSL by default	2019-01-11 22:48:09 -05:00
Nick Sweeting	f83750c545	cleanup options and make cli flags better for chrome headless timeouts	2019-01-11 22:38:50 -05:00
Nick Sweeting	911b84a79b	make wget cli options easier to read	2019-01-11 22:13:51 -05:00
Nick Sweeting	b77476cb83	fix user agent quoting	2019-01-11 19:54:43 -05:00
Nick Sweeting	33942d37cb	Update README.md	2019-01-11 07:57:46 -05:00
Nick Sweeting	4609b11d15	Update README.md	2019-01-11 07:56:24 -05:00
Nick Sweeting	5ea8a1ad9c	Update README.md	2019-01-11 07:21:30 -05:00