FOSS infrastructure is under attack by AI companies https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ Please boost for awareness, reach and to public shame Microsoft, Meta, OpenAI, Perplexity and other such AI companies.
#opensource #programming #Developers
The worst part? These companies are ignoring well-established robots.txt standards. They have been caught ignoring the robots.txt file multiple times and are now creating issues for the rest of us. They use residency proxy services even if you block all data center ranges. It is like a war for them to collect all code, images, PDFs, books, videos, and stuff on the Internet to train their LLM without paying for anything. It is wild how they are getting away with copyright violations.
@nixCraft@mastodon.social I remember someone being mad that their crawler got trapped by something malicious but only because the crawler ignored the robots.txt
Hilarious
@nixCraft@mastodon.social
I might be wise to keep records of their traversal of your site in hopes that in the future (if we’re lucky) a court case will require them to license all of the content.
@nixCraft@mastodon.social We had to implement firewall level blocks on all these AI bots - at one point it was like a DDOS attack on our IDP service and generating enormous logs on our side
@nixCraft@mastodon.social Big Tech AI is a culture of kleptocracy. Their GenAI bullshit generators can’t generate bullshit without tons of stolen goods.
@nixCraft@mastodon.social It’s probably time to try something like this
@nixCraft@mastodon.social I always wondered why people really thought that companies would be honest and follow robots.txt without cheating. To me it’s a good reminder that you can never ever trust companies to do the right thing if the wrong thing is possible.
@nixCraft@mastodon.social
We have collected 460K IP of a malicious scraping botnet. We share them. Use them.
@nixCraft@mastodon.social “robots.txt is a sign not a cop” 😔
@nixCraft@mastodon.social because trash can violate copyright but if we do the same to them nothing happens
@nixCraft@mastodon.social They’re afraid they’ll loose the fair use argument. They should, but I doubt they will. At any rate, once it’s all in they can claim that it would be too difficult to comply, blah blah, and it’s a second chance for a judge to let it slip through.
We are all going to have to use PKI to communicate and use white lists. They’re not going to let us have this.
@nixCraft@mastodon.social
And it’s not just about copyright violations, but also about real money lost. It’s like if someone stole your cash - in some cases, the AI “attacks” might raise hosting costs for the website owner.
Think about the website owner who paid ~500/month and suddenly get a 5k bill bc of an AI company. And if their services are off, they will face loss of revenue.
Record everything that is happening in your systems. If you can’t file a complaint for some reason, share the info with others.