Hi, I’m building a personal website and I don’t want it to be used to train AI. In my robots.txt
file I blocked:
- ChatGPT-User
- GPTBot
- Google-Extended
- FacebookBot
What bots should I also add? Are there any other ways to block AI bots?
IMPORTANT: I don’t want to block search engine crawlers, only bots that are used to train AI.
Luke ( @lukecooperatus@lemmy.ml ) 47•8 months agoFYI, bots and crawlers can simply ignore your robots.txt entirely. This is probably common knowledge around these parts, but I’ve run into clients at work who thought it was a law or something.
I do like the idea of intentionally polluting the data robots will see, as suggested by this comment. There’s no reliable way to block them without also blocking humans, so making the crawled data as useless as possible is a good option.
Just be careful not to also confuse screen readers with that tactic, so that accessibility is maintained for humans. It should be easy enough if you keep your
aria
attributes filled out appropriately, I imagine. Illecors ( @Illecors@lemmy.cafe ) English23•8 months agoGood luck.
bbbhltz ( @bbbhltz@beehaw.org ) 7•8 months agoI have a personal site.
It isn’t great. Don’t even have a domain name. My robots.txt is here
https://bbbhltz.codeberg.page/robots.txt
Why bother? I just don’t agree with AI.
Saganastic ( @Saganastic@kbin.social ) 5•8 months agoSpecifically what about AI don’t you agree with?
bbbhltz ( @bbbhltz@beehaw.org ) 6•8 months agoMostly the hype and because artists and creators are being hurt by its existence.
I feel as though using AI is a cop-out. If I want to do something good, I also want to be proud of it. So I would rather not take that away from myself by doing it with AI. However, progress marches on, and I am neither an expert nor an authority on the subject. Asking someone like myself that question is nearly a trap. If I tell you that Generative AI is a bubble, like cryptocurrency and the Metaverse, that is just my gut feeling.
Saganastic ( @Saganastic@kbin.social ) 2•8 months agoHow about a bubble like the internet? 90% of dotcoms failed in the 90s but the internet is alive and strong today. AI is just a tool, and from my experience an extremely useful one.
bbbhltz ( @bbbhltz@beehaw.org ) 2•8 months agoI get that argument. Perhaps the fact that I’m a professor influences my thinking. And, since we are in a privacy community, something like ChatGPT and privacy don’t mix.
Meredith Whittaker (Signal) says[1]:
The Venn diagram of privacy concerns and AI concerns is a circle
(I do keep on eye on their progress because it is interesting https://benchmarks.llmonitor.com/)
Saganastic ( @Saganastic@kbin.social ) 2•8 months agoAgreed that privacy can be a concern. Ideally it will be possible to run LLMs locally in the near future, but we’ll see.
folkrav ( @folkrav@lemmy.ca ) 1•8 months agoI was about to ask the same question. It’s one thing to think of the potential impacts of AI technology, but to be “against AI” in the most general sense is, to me, a weird concept, especially considering AI is so many things.
Nice, thats what I am looking for!
bbbhltz ( @bbbhltz@beehaw.org ) 2•8 months agoI don’t remember what all of those are for so you might want to look them up.
I did, most of them are used for AI or business search engines. I copied everything except Yandex.
Mubelotix ( @Mubelotix@jlai.lu ) 6•8 months agoBlock everyone but the crawlers you like. Blacklists are less reliable than whitelists
Oliver Lowe ( @otl@lemmy.sdf.org ) 4•8 months agoMaybe there’s some IP address ranges to try block?
It’s difficult because, for example, blocking the addresses OpenAI’s crawlers use may inadvertently block addresses from Azure used by Bing or whatever.
jlow (he/him) ( @jlow@beehaw.org ) 3•8 months agoI don’t really understand the reasoning behind doing any of this, they didn’t give a fuck about stealing clearly copyrighted content in the first place, why would they care about you (not OP specifically) begging them not to steal your stuff. (As long as theres no laws about this which afaik there aren’t).
wagoner ( @wagoner@infosec.pub ) 1•8 months agoSo that leaves two options then. Leave the front door wide open, don’t bother with any locks. Or shut down the web site. I’m for at least closing the door with the right robots.txt
jlow (he/him) ( @jlow@beehaw.org ) 1•8 months agoThe analogy should be either having the door open or having the door open but putting a note on the door saying to please not steal anything. I’m not saying you shouldn’t do it, I just don’t think it’s gonna do anything, so I’m not going to bother.
Xirup ( @Xirup@lemmy.dbzer0.com ) 1•8 months agoPehaps the user (or in this case the bot) will not go directly to your website, but first to some method of captcha verification or something like that, or like those pages (SteamDB for example) that do not open directly but first open a blank page to verify your network and browser with a captcha.
Bldck ( @Bldck@beehaw.org ) English1•8 months agoEasy. Add a section to your robots.txt file.