My favorite kind of graphs are ones where an entire axis is unlabeled:
You see this a lot with marketing graphs. They say nothing, but they’re designed to convince you that the graphs mean something.
Anyway, it’s neat they found and fixed, supposedly, some real bugs. I’m curious how many fake reports they had to sift through to find any real ones.
He writes 100 and that he did not yet sift through them.
But I am wondering the same. How often he had to run the same command and how long to find the right prompt
I can’t report because I haven’t validated them yet… I’m not going to send [the Linux kernel maintainers] potential slop
That’s worth pointing out IMO
Though that quote is followed by this, which indicates at least five of those vulnerabilities were real:
I searched the Linux kernel and found a total of five Linux vulnerabilities so far that Nicholas either fixed directly or reported to the Linux kernel maintainers, some as recently as last week:
I wonder how true that is. The author of this blog post seems to just be taking this guy’s word for it. Did Anthropic actually confirm the bug exists by trying to trigger it on real systems, or are they assuming it’s real because it looks plausible? The report claims you cam do it with two cooperating NFS clients, so did they actually do that, or are they just assuming it’ll work?
Those are five bugs the kernel maintainers have reviewed and decided to patch (the links are to the commits), not just five bug reports. I think that leans towards “they tested it” or at least “proofed the formal logic in their minds successfully”.
Real vulnerability or hallucinated?
Given Nicholas Carlini work at anthropic I would wait for another person to confirm this.
The research method is just pointing file by file and asking an LLM if any vulnerability exist and reminds me of the person who bugged ffmpeg devs with vulnerabilities on niche non enabled codec decryption.
reminds me of the person who bugged ffmpeg devs with vulnerabilities on niche non enabled codec decryption.
That was google.
If I ever received a vuln report from an AI, or other such glorified spreadsheet, I would promptly dismiss it then wait for a human to organically discover it on its own to consider that as proof of actual existence.
This would be meaningful if the findings were not produced by the corp trying to sell you the product being hyped. Big tech has a history of “faking it till you make it” and I can’t help but doubt that this is really just Claude Code mostly autonomously finding issues.
I’m really scared about what AI is going to do to the world, but I think it’s here to stay.
Hopefully it’s actually finding real bugs
