Daily Stormer Search
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Perception 58ee7fe140 Make headers smaller 5 months ago
bin Use ds.db() 5 months ago
routes Add routes and templates for tag pages 5 months ago
sql Merry Christmas 2020 5 months ago
templates Make headers smaller 5 months ago
.eslintrc.js Merry Christmas 2020 5 months ago
.gitignore Ignore *.db instead of just main.db 5 months ago
.jshintrc Merry Christmas 2020 5 months ago
.repl.js Initialize global.db in repl 5 months ago
README.md Merry Christmas 2020 5 months ago
index.js Add more tag-related functions 5 months ago
package-lock.json Add qs and ucfirst 5 months ago
package.json Add qs and ucfirst 5 months ago
page.js Setup EJS template rendering 5 months ago
site.js Add dotenv initialization 5 months ago

README.md

Daily Stormer Utilities

This is a little project to help me get out of my funk.

Goals

  • Crawl all dailystormer articles.
  • Save crawl results as JSON files.
  • Be able to run the crawler again to get new articles.
  • This implies that I can keep track of what I've already crawled.
  • Make it possible to do a full text search on those articles.

Questions

Crawling

The recursive crawl to get old articles and the crawl to update should be different. The updater can assume a big crawl has already been done, and just pull articles down from the RSS feed. However, the recursive crawl doesn't have that luxury, because it must find past articles and handle newly discovered tags and categories as new crawl targets.