• Hirom ( Hirom@beehaw.org ) Banned from community
    link
    fedilink
    English
    arrow-up
    6
    ·
    edit-2
    2 years ago

    In my experience with bots, a portion of them obey robots.txt, but it’s tricky to find the user agent string that some bots react to.

    So I recommend having a robots.txt that not only target specific bots, but also tell all bots to avoid specific paths/queries.

    Example for dokuwiki

    User-agent: *
    Noindex: /lib/
    Disallow: /_export/
    Disallow: /user/
    Disallow: /*?do=
    Disallow: /*&do=
    Disallow: /*?rev=
    Disallow: /*&rev=
    
      • froztbyte ( froztbyte@awful.systems ) 
        link
        fedilink
        English
        arrow-up
        10
        ·
        2 years ago

        yes, you can match on user agent, and then conditionally serve them other stuff (most webservers are fine with this). nepenthes and iocaine are the current preferred/recommended servers to serve them bot mazes

        the thing is that the crawlers will also lie (openai definitely doesn’t publish all its own source IPs, I’ve verified this myself), and will attempt a number of workarounds (like using residential proxies too)

        • Hirom ( Hirom@beehaw.org ) Banned from community
          link
          fedilink
          English
          arrow-up
          4
          ·
          2 years ago

          Generating plausible-looking gibberish require resources. Giving any kind of response to these bots is a waste of resources, even if it’s giberish.

          My current approach is to have a robots.txt for bots than honor it. And drop all traffic during 24h for IPs used by bots that ignore robots.txt or misbehave.