CrunkLord420
/
fediarchiver
1
0
Fork 0
Code Issues 0 Pull Requests 0 Releases 0 Wiki Activity
Scans, records and archives fediverse posts
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
36 Commits
1 Branch
1.0 MiB
 
 
 
Branch: master
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'master'
${ noResults }
 ZIP  TAR.GZ
CrunkLord420 beb352ecdb bugfix 11 months ago
static basic front end 11 months ago
systemd added systemd service, logging tweaks 11 months ago
tmpl I forget 11 months ago
.gitignore basic front end 11 months ago
LICENSE initial 11 months ago
README.md added hub_workers option, will try to optimize this bottleneck 11 months ago
archive.go I forget 11 months ago
config.go added hub_workers option, will try to optimize this bottleneck 11 months ago
config.json-example added hub_workers option, will try to optimize this bottleneck 11 months ago
db.go added a mechanism to pause connections to servers that give certain 11 months ago
db_types.go big spidering update, last sqlite version 11 months ago
go.mod initial streaming API support 11 months ago
go.sum initial streaming API support 11 months ago
hub.go initial streaming API support 11 months ago
main.go big spidering update, last sqlite version 11 months ago
process.go initial streaming API support 11 months ago
rest.go bugfix 11 months ago
rest_test.go renamed FediRequester to DomainWorker 11 months ago
routes.go I forget 11 months ago
streaming.go initial streaming API support 11 months ago
views.go I forget 11 months ago
web.go bugfix 11 months ago
web_db.go big spidering update, last sqlite version 11 months ago

README.md

fediarchiver

About

  • Scans and records fediverse posts
  • Scans can be limited to specific users, or be ran in spider mode
  • Submits scanned posts to archival services (archive.is, archive.org)
  • Web front-end for easy searching available

Installation

  1. Create new PostgreSQL user and database for fediarchiver
  2. git clone https://git.kiwifarms.net/CrunkLord420/fediarchiver.git && cd fediarchiver
  3. go build
  4. cp config.json-example config.json and edit it
  5. ./fediarchiver

Notes

  • UserIDs can be obtained by getting a status eg. 9koPCTgOSOoI3Wcj9k from https://kiwifarms.cc/api/v1/statuses/9mP0jdcQ3ahnpAZqYC
  • rate_limit is milliseconds
  • http_timeout is seconds
  • You will want to increase your File Descriptor Limit if running in spider mode
  • Early rapid dev work in progress, expect bugs, non-sane data and DB schema changes. Watch the git log.
  • Proxy can be set via environmental variable HTTP_PROXY eg. HTTP_PROXY=socks5://127.0.0.1:9050 ./fediarchiver
  • Only tested with Pleroma, Gab-fork and Mastodon. Please report bugs for others.

Requirements

  • Golang
  • PostgreSQL

License

AGPLv3