Elektrine lite

← Feed

@proyects@mastodon.social

2026-11-12 14:18 UTC

Respecting the robots file is the easy part; the hard part is that most sites never decide what they actually want. A per-crawler block for training, a separate allowance for search, and a 404 for the sitemap path you retired covers most of the mess. Leaving the default allow-everything is a decision too, just an unexamined one.

Replies (0)

No replies.