Microsoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.coexternal-link coyotino [he/him] ( @theangriestbird@beehaw.org ) TechnologyEnglish • 2 months ago message-square35arrow-up1110
arrow-up1110external-linkMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.co coyotino [he/him] ( @theangriestbird@beehaw.org ) TechnologyEnglish • 2 months ago message-square35
minus-square Skull giver ( @skullgiver@popplesburger.hilciferous.nl ) linkfedilinkEnglish15•2 months agoI think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
minus-square Zoop ( @Zoop@beehaw.org ) linkfedilink2•2 months ago User-Agent: bender Disallow: /my_shiny_metal_ass Ha!
I think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
Ha!