Browse Source

Redo the README w/ installation instructions

Perception 5 months ago
1 changed files with 16 additions and 14 deletions
  1. +16

+ 16
- 14 View File

@ -1,20 +1,22 @@
# Daily Stormer Utilities
# Daily Stormer Search
This is a little project to help me get out of my funk.
A Search Engine for the Daily Stormer
## Goals
# Installation
* Crawl all dailystormer articles.
* Save crawl results as JSON files.
* Be able to run the crawler again to get new articles.
* This implies that I can keep track of what I've already crawled.
* Make it possible to do a full text search on those articles.
In the root of the project, do the following.
## Questions
# Install node modules
npm i
### Crawling
# Initialize database
sqlite3 main.db < sql/ds.sql
sqlite3 main.db < sql/ds.001.sql
The recursive crawl to get old articles and the crawl to update should be different. The updater
can assume a big crawl has already been done, and just pull articles down from the RSS feed. However,
the recursive crawl doesn't have that luxury, because it must find past articles and handle newly
discovered tags and categories as new crawl targets.
# Scrape articles; This may take a few days if done from scratch.
# Run the server
micro-dev -p 1984 site.js