CrunkLord420
/
fediarchiver
1
0
Fork 0
Code Issues 0 Pull Requests 0 Releases 0 Wiki Activity
Scans, records and archives fediverse posts
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
36 Commits
1 Branch
1.0 MiB
 
 
 
Branch: master
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'master'
${ noResults }
 ZIP  TAR.GZ
CrunkLord420 beb352ecdb bugfix 1 year ago
static basic front end 1 year ago
systemd added systemd service, logging tweaks 1 year ago
tmpl I forget 1 year ago
.gitignore basic front end 1 year ago
LICENSE initial 1 year ago
README.md added hub_workers option, will try to optimize this bottleneck 1 year ago
archive.go I forget 1 year ago
config.go added hub_workers option, will try to optimize this bottleneck 1 year ago
config.json-example added hub_workers option, will try to optimize this bottleneck 1 year ago
db.go added a mechanism to pause connections to servers that give certain 1 year ago
db_types.go big spidering update, last sqlite version 1 year ago
go.mod initial streaming API support 1 year ago
go.sum initial streaming API support 1 year ago
hub.go initial streaming API support 1 year ago
main.go big spidering update, last sqlite version 1 year ago
process.go initial streaming API support 1 year ago
rest.go bugfix 1 year ago
rest_test.go renamed FediRequester to DomainWorker 1 year ago
routes.go I forget 1 year ago
streaming.go initial streaming API support 1 year ago
views.go I forget 1 year ago
web.go bugfix 1 year ago
web_db.go big spidering update, last sqlite version 1 year ago

README.md

fediarchiver

About

  • Scans and records fediverse posts
  • Scans can be limited to specific users, or be ran in spider mode
  • Submits scanned posts to archival services (archive.is, archive.org)
  • Web front-end for easy searching available

Installation

  1. Create new PostgreSQL user and database for fediarchiver
  2. git clone https://git.kiwifarms.net/CrunkLord420/fediarchiver.git && cd fediarchiver
  3. go build
  4. cp config.json-example config.json and edit it
  5. ./fediarchiver

Notes

  • UserIDs can be obtained by getting a status eg. 9koPCTgOSOoI3Wcj9k from https://kiwifarms.cc/api/v1/statuses/9mP0jdcQ3ahnpAZqYC
  • rate_limit is milliseconds
  • http_timeout is seconds
  • You will want to increase your File Descriptor Limit if running in spider mode
  • Early rapid dev work in progress, expect bugs, non-sane data and DB schema changes. Watch the git log.
  • Proxy can be set via environmental variable HTTP_PROXY eg. HTTP_PROXY=socks5://127.0.0.1:9050 ./fediarchiver
  • Only tested with Pleroma, Gab-fork and Mastodon. Please report bugs for others.

Requirements

  • Golang
  • PostgreSQL

License

AGPLv3