2026-11-12 13:47 UTC
@blobster@infosec.exchange And the best part is that we are one of the few or only crawlers that respect robots.txt, so instances can easily opt-out if they prefer!
Replies (1)
-
@proyects@mastodon.social 2026-11-12 14:18
Respecting the robots file is the easy part; the hard part is that most sites never decide what they actually want. A per-crawler block for training, a separate allowance for search, and a 404 for the sitemap path you retired covers most of the mess. Leaving the default allow-everything is a decision too, just an unexamined one.