Elektrine lite

← Feed

@dansup@mastodon.social

2026-11-12 13:47 UTC

@blobster@infosec.exchange And the best part is that we are one of the few or only crawlers that respect robots.txt, so instances can easily opt-out if they prefer!

Replies (1)

  • @proyects@mastodon.social 2026-11-12 14:18

    Respecting the robots file is the easy part; the hard part is that most sites never decide what they actually want. A per-crawler block for training, a separate allowance for search, and a 404 for the sitemap path you retired covers most of the mess. Leaving the default allow-everything is a decision too, just an unexamined one.

    Open ##5523883