How to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-link Cynicus Rex ( @CynicusRex@lemmy.ml ) Privacy@lemmy.mlEnglish • 1 month ago message-square18fedilinkarrow-up166
arrow-up166external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.biz Cynicus Rex ( @CynicusRex@lemmy.ml ) Privacy@lemmy.mlEnglish • 1 month ago message-square18fedilink
minus-square NullPointer ( @nullPointer@programming.dev ) linkfedilink13•1 month agorobots.txt will not block a bad bot, but you can use it to lure the bad bots into a “bot-trap” so you can ban them in an automated fashion.
minus-square Dave. ( @dgriffith@aussie.zone ) linkfedilink7•1 month agoI’m guessing something like: Robots.txt: Do not index this particular area. Main page: invisible link to particular area at top of page, with alt text of “don’t follow this, it’s just a bot trap” for screen readers and such. Result: any access to said particular area equals insta-ban for that IP. Maybe just for 24 hours so nosy humans can get back to enjoying your site.
robots.txt will not block a bad bot, but you can use it to lure the bad bots into a “bot-trap” so you can ban them in an automated fashion.
I’m guessing something like:
Robots.txt: Do not index this particular area.
Main page: invisible link to particular area at top of page, with alt text of “don’t follow this, it’s just a bot trap” for screen readers and such.
Result: any access to said particular area equals insta-ban for that IP. Maybe just for 24 hours so nosy humans can get back to enjoying your site.