Beehaw
  • Communities
  • Create Post
  • search
    Search
  • Login
  • Sign Up
sabreW4K3 ( sabreW4K3@lazysoci.al )  to TechnologyEnglish · 1 year ago

The Open-Source Software Saving the Internet From AI Bot Scrapers

www.404media.co

external-link
message-square
22
link
fedilink
  • cross-posted to:
  • opensource@programming.dev
  • opensource@lemmy.ml
147
external-link

The Open-Source Software Saving the Internet From AI Bot Scrapers

www.404media.co

sabreW4K3 ( sabreW4K3@lazysoci.al )  to TechnologyEnglish · 1 year ago
message-square
22
link
fedilink
  • cross-posted to:
  • opensource@programming.dev
  • opensource@lemmy.ml
Anubis, which block AI scrapers from scraping websites to death, has been downloaded almost 200,000 times.
  • Chris Remington ( remington@beehaw.org ) 
    link
    fedilink
    arrow-up
    3
    ·
    1 year ago

    Would you edit your post and add the following archive link to the body, please?

    https://archive.is/VcoE1

    • who ( who@feddit.org ) 
      link
      fedilink
      English
      arrow-up
      7
      ·
      edit-2
      1 year ago

      Unfortunately, archive.is seems to have moved behind a big corporate CAPTCHA service, subjecting readers to having their reading habits (both the articles and the referring communities) tracked at a large scale.

      I suggest this archive link instead:

      https://web.archive.org/web/20250707135819/https://www.404media.co/the-open-source-software-saving-the-internet-from-ai-bot-scrapers/

      • Chris Remington ( remington@beehaw.org ) 
        link
        fedilink
        arrow-up
        1
        ·
        1 year ago

        Unfortunately, archive.is has moved behind Cloudflare, subjecting readers to having their reading habits (both the articles and the referring communities) tracked at a large scale.

        How do you know this?

        What about https://ghostarchive.org/?

        • who ( who@feddit.org ) 
          link
          fedilink
          English
          arrow-up
          6
          ·
          1 year ago

          Sorry; I shouldn’t have written Cloudflare specifically. Their CAPTCHA page now contains scripts from Google, not Cloudflare. I have corrected my comment.

          How do you know this?

          Because a couple months ago, archive.is/archive.today started showing me CAPTCHA pages instead of the archived articles when I use Firefox with scripts disabled. The current page contains scripts hosted by Google, which I won’t enable, so I can’t read the archived articles.

          What about https://ghostarchive.org/?

          I haven’t used that site enough to have a consistent picture of what it’s doing. When I tried it a few minutes ago, it directed me to a CAPTCHA wall when trying to submit an article, but not when searching for an archived article. I’ll try to remember to look at it again periodically, to be able to answer this question in the future.

          • Chris Remington ( remington@beehaw.org ) 
            link
            fedilink
            arrow-up
            3
            ·
            1 year ago

            Thanks. I appreciate the info and effort.

    • sabreW4K3 ( sabreW4K3@lazysoci.al ) OP
      link
      fedilink
      arrow-up
      5
      ·
      1 year ago

      To be honest with you, I refuse on moral grounds. 404 are independent and do good work. You’ve already linked a pay wall bypass in the comments, if anyone would like to find it, it’s not hard to scroll.

      • Chris Remington ( remington@beehaw.org ) 
        link
        fedilink
        arrow-up
        4
        ·
        1 year ago

        OK. Fair enough.

Technology

technology

Subscribe from Remote Instance

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@beehaw.org

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

  • Free and Open Source Software
  • Programming
  • Operating Systems

This community’s icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 97 users / day
  • 816 users / week
  • 2.2K users / month
  • 5.1K users / 6 months
  • 5.11K local subscribers
  • 43.7K subscribers
  • 6.42K Posts
  • 107K Comments
  • Modlog
  • mods:
  • Chris Remington ( remington@beehaw.org ) 
  • alyaza [they/she] ( alyaza@beehaw.org ) 
  • TheRtRevKaiser ( TheRtRevKaiser@beehaw.org ) 
  • gyrfalcon ( gyrfalcon@beehaw.org ) 
  • rs5th ( rs5th@beehaw.org ) 
  • coldredlight ( coldredlight@beehaw.org ) 
  • Leigh ( SemioticStandard@beehaw.org ) 
  • TheRtRevKaiser ( TheRtRevKaiser@kbin.social ) 
  • BE: 0.19.15
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code