Welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned soo many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this. Happy Halloween, fellow sneerers!)
going down a little rabbit hole. on zerohedge they noted:
Benjamin Mann, another Anthropic cofounder who was not listed on the S-1 for reasons unknown, published an essay in 2019 titled “Why I now identify as an Effective Altruist.”
=> https://sfstandard.com/2026/10/26/anthropic-cofounder-mystery-ben-mann/
=> https://www.reddit.com/r/Anthropic/comments/1nznl4z/who_is_ben_mann_to_anthropic/
=> https://www.courtlistener.com/docket/67538258/tremblay-v-openai-inc/?page=3
=> https://www.courtlistener.com/docket/67538258/273/1/tremblay-v-openai-inc/
So in his court declaration (and on linkedin) he says co-founded Anthropic. And also that he was the Data Eng at OpenAI that “undertook” the second batch of pirated books into GPT-3.
Any speculation on what the redacted line was? And would this or other litigation be a factor to leave him off the S1; ie it could potentially expose / risk the 50.1% voting Class F LLC shares through Mann? Or maybe he already pledged all his shares to open giving.
So what’s with the phenomenon of people that are dismissive of the capabilities of chatbots and freely list their constant fuckups, maybe even habitually calling them ‘clankers’, and yet still constantly fucking using them, comparing harnesses, blah blah blah? It’s not our usual cocaine analogies because these people are hardly evangelists… maybe still covered by ‘gambling addiction’? It’s just baffling to me. You humbly suggest that maybe they shouldn’t be making so much use of a chatbot that constantly bullshits them, and you don’t get any actual objection to that… and yet they continue on.
Am I part of this category? I have a rather powerful home server that, among other things, I use to:
-
Run a decent coding LLM that helps me vomit up simple python functions for moving around lots of data between CSVs and graphing stuff, set up new docker containers with less trial and error, and run ML experiments on my GPU that previously took me hours to set up. I’m a biologist and previously I could run a commandline without breaking a computer but was very slow and non-native, now I can do simple non-security-critical things faster and actually understand novel error messages. Of note I never have that hooked up in a way that lets it run things just produce text that I can then do things with and maybe run myself.
-
Run my own local RAG system with embedding models and a small shitty LLM to search a hundred gigabytes of scientific papers and other PDFs semantically for things relevant to roughly paragraph length queries.
-
Run my own web metasearch engine with RAG and embedding models pointed at the internet, such that when I remember a paper I read fifteen years ago and provide a three sentence description it reliably finds it or finds lots of relevant old research (or old news articles I remember) when I give it a query and let it churn for five minutes.
I feel like the important thing here is that I almost exclusively use them as information channels that take something at one end and spit it back out the other in a different or filtered form rather than making stuff, or effectively a piece of reference material for something that is formally verifiable and NOT CRITICAL FOR A SYSTEM TO FUNCTION. I also feel like all the actual innovations are more than a year old and do not understand people hanging excitedly on every little development.
I don’t find myself having nearly as many occasions to make use of them as you do, but I do think that all the misplaced AGI hype makes it hard to evaluate the actually technological advancement that LLMs represent. Like, being able to statistically model language to the degree these things do is neat, and I would be more surprised if it wasn’t possible to use that to do some useful and powerful things. But just like how it’s impossible to separate the underlying math of the cryptocurrency sphere from the ideology of technolibertarian griftocracy, everything that the LLM space has produced is poisoned on some level by millenarian technofascism. Like, the degree to which our economy and culture have been warped by these things is entirely a product of the people who think that they’re going to spit out a God that makes whoever does so first king of the world forever. All the actual useful things people are getting from it are around the edges and happening in spite of the demands to remove guardrails and quality checks and human accountability from these systems. But that doesn’t necessarily mean that the underlying mathematics is without value and can’t be part of good and useful systems in the future. Machine Learning as a whole had already demonstrated the value in automated pattern identification and reproduction in a variety of fields from medical diagnostics to pastry sales.
Don’t get me wrong, I’m strongly in favor of throwing out all this bathwater even if there might be a baby in there somewhere. I just think that “hard to evaluate independently” isn’t necessarily the same thing as “without value in other contexts”.
Can you answer a question about RAG for me? My understanding is based entirely on this diagram from Wikipedia
If RAG takes into account vector similarity between query and reference material, do you need an LLM to regurgitate the found best vectors back to you? Could you get back a list of documents that best matched your query, or are the results in an inscrutable LLM-internal representation that need to be transformered back into readable text?
You can have a system that just gives you a relevance-sorted list, or a system that then ingests the chunks of those documents that are most relevant and gives a human readable summary in addition to the list.
The indexing step requires a lot of processing, by which you chunk reference documents into couple hundred word chunks and cook each of those chunks into a high dimensional embedding vector. Once you have done that and saved the database of vectors, you can put in your query, embed it, check its dot product against all the other embedding vectors, and get your ranked list (potentially quite fast) without need to produce text out the other end.
Making those embedding vectors works much faster on a GPU rather than a CPU, but can be done on either. Since I have one system set up to ingest a bunch of web documents and embed them and search them, I have that system already set up using the GPU to go faster, so there’s no real disadvantage to having it also set up to spit out a small natural language summary using a shitty tiny local LLM to go along with the list. When running it on local files, you can totally ingest the local document stash slowly on even just a CPU and just store that and run the semantic search on CPU, no LLM required.
Thanks! Well generating the summary would have the downside (for me), that I would have to look at it.
But the semantic search bit was the one thing I felt was an actual new capability. I’m trying to imagine what if Google tried to pick this as their new moat (a better but resource-intensive way to search, scaling benefits if you establish yourself as the Internet’s main vector store, people might actually pay for queries…), but no they joined the team that wants so shove slop down everybody’s eyeballs.
-
There are also really people who think all crititcs also constantly use it. Which also baffles me.
I think they’ve handed over so much of their thinking to LLMs that they have forgotten that quality thought can be done independently.
Also it’s in their interest to imply that LLMs are everywhere, unavoidable, and indispensable.
New Data Centers in the US Set to Wipe Out 15 Years of Greenhouse Gas Reductions
Somewhat misleading title, as they are referring to pollution reduction efforts solely in the US, not worldwide.
Nearly 15 years worth of hard-won pollution reduction efforts are set to be undone in the United States, as a wave of data centers powered by gas plants enters construction.
Unpaywalled version of the Bloomberg article here: https://archive.li/9Ts5f
This is only if all the planned data centre buildouts go ahead, mind you
That appears to assume that all the ridiculous datacenter numbers put out by all the various hyperscalers that are not actually going to happen, happen, and get power sources built on time.
Doing a little of my own math:
Article says the US will emit 2.1 billion metric tons of carbon by 2030.
Using a metric tons to gigatons conversion gives us 2.1 gigatons.
Using this as a reference, these projected AI emissions shouldn’t even come close to scratching the global average
Looks like the great few years for space launch will be coming to an end, as Spacex gives up on launching other people’s hardware and focuses inwards
Take a look here for figures on space launch over time
Looks like instead they will commit their capacity to AI In Space and a little Starlink (so probably mostly Starlink, even if Elon currently drinks his own Cool-Aid).
Really drives it home for me how institutionally cucked the US gov is. For this to be as big a deal, no-one must have the slightest concern that the US will claw back or conscript SpaceX capacity for national security. And correctly so.
My guess is they want it all subscriptions (of course). Satellite-as-a-service. GOD forbid you launch your own device you control! Nonono, you can LEASE SPACE on one of their sats they shovel out the door, and you’ll be happy.
What’s that? You said something that ticked the manbaby CEO off? Too bad, didn’t you know we can revoke your uplink at any time for any reason?
Depressingly plausible, thanks!
Artemis III is going to be so embarrassing, isn’t it. Next summer they’re supposed to link up with a lunar lander-configuration Starship (and also Bezos’ company’s slightly more reasonable lander design) in orbit as a proof-of-concept.
I have no doubt that they will throw something with a docking port into orbit. I have many doubts that it will have much physical similarity to something that could land on the moon.
Oh here’s a fun scenario: FAANG stocks are correcting down and now there’s concern Anthropic will have timed their IPO perfectly such that the investors make out great and the employees have to deal with very much less liquid stock than when it triggered (leading to a massive tax bill they might not be able to cover)
So…
Roko’s Taxilisk.
https://www.todayintabs.com/p/eternal-black-october
The problem with A.I. defense is the same asymmetry attackers have always had against defenders. After a hotel bombing in Brighton, England in October, 1984 that failed to kill Margaret Thatcher, the Irish Republican Army put out a statement that said “Today, we were unlucky. But remember we have only to be lucky once. You will have to be lucky always.“ As it turned out, Margaret Thatcher was lucky always, but there’s no reason to believe the rest of us will be. If an A.I. hack attempt works one time out of a thousand, that’s a fantastic success rate. If your A.I. sysop “secures” your server by deleting the production database and all the backups one time out of a thousand… well. Who’s gonna be first to sign up for those odds? Roll the dice! Apparently they always come up four.
Oh this will go well: Tom’s Hardware story about someone at Cisco committing to rewriting “98%” of legacy code using AI in 24 months.
saw that earlier as well, and I had a real good laugh about the myriad ways that will go wrong. both the obvious (derping proprietary extensions, etc), and the less so (crashing a control plane because the chatbot doesn’t understand 256MiB ram)
the sheer utter range of the crisco product range as it is…. phenomenal display of how removed from knowledge the execs are
Uh-oh, renowned everyman Sam Altman opened his mouth again.
Local inference is a terrible allocation of scarce resources.
A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data centre
but from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time
you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment, somehow many people’s intuition is the opposite of that
And here I thought I was building servers two years ago because the next Asian war would disrupt supply chains.
Much better for RAM to be sitting behind a network hop with 100 milliseconds of latency amirite?
if you’re trying to deal with this very supply-constrained environment,
We’re all trying to find the guy who did this .jpg
Edit: also lol at the unstated implication that no one should ever want to use RAM for anything besides getting blathered at by a chatbot.
The CEO claimed that AI will potentially be able to create something as important as the transistor, or discover major efficiency gains
Did anyone ever fall for this ridiculous argument when it was used for “uhmm actually the massive datacenter buildout is great for global warming because chatbots will chat at us how to make better solar panels or something”?
It’s inefficient allocation of resources for every town to have their own library. The space sits unused a lot of the time and every library needs to allocate space to walls/doors/cpunters etc. Plus duplicate books.
Instead everyone should have one megalibrary per country.
This is good, I am smart, invest in bookly: 5he startup for buying all the books in the world
@sailor_sega_saturn @techtakes Much better for everyone if Salesman Sam owns all your RAM and leases it back to you by the refresh cycle!
@cstross @sailor_sega_saturn @techtakes
It is also terribly inefficient to have everyone keep a fraction of the world’s money when it could all be concentrated instead in the hands of Sam Altman, benevolent dictator-god-king for life.@cstross @sailor_sega_saturn @techtakes
And he is working on the “for life” problem…
no, the most optimal way for it is to sit unused in some warehouse in taiwan, because dc isn’t built for it yet but nvidia has to get money
@sailor_sega_saturn Whenever someone says “But someday AI will cure cancer/make us all rich/solve global warming,” the only possible response should be "why aren’t you using it for that already?”
@sailor_sega_saturn @Architeuthis <literally republishes papers about fundamental transistor design with authors scrubbed off>
“See it invented the transistor!”
@Architeuthis @BlueMonday1984 Now let’s talk about privately owned cars versus public transport, Sam.
Not that Altman’s company is ahead in the AI coding race, but everyone’s PCs will still need increasing amounts of DRAM to be able to run the next generation of slop-coded JavaScript apps.
YOU CAN TAKE MY LOCAL COMPUTE FROM MY COLD DEAD HANDS
I’ll donate my RAM to SAlty if he comes to collect it in person, and only if I can deliver it straight to his dome
5%? Mister I run firefox.
The author of More Everything Forever: A timely exposé of the ludicrous and cruel obsessions of Silicon Valley’s oligarchs cybersquatted eilezeryudkowsky.com, which i think is delightful.
He was on Weird Little Guys last week too. Seems like a good egg with a good book.
I saw a slop ad hanging on a fence today, and I’m in half a mind to buy some spray paint just so I can write “SLOP” over it
A big marker also works, think that is how people tag things going by the smell I one noticed when some 30 year old guy tagged a table in the train I was in. (It was an odd moment).
I think the poster ad thing is a bit big for that. The idea of this fantasy is that the sloperator should get zero value from their defacement of public space, so it would have to meaningfully obscure any pertinent details like address etc. while also very helpfully telling the sloperator what their oversight was.
Likely Pivot candidate just dropped:
VRChat refused to add an AI label despite 95% of respondents on their own survey supporting it.
Professional rube Kevin Roose is launching a book on The AGI Race in January. The second paragraph is about effective accelerationists snorting ketamine at parties in the Mission District. He was invited like a big boy!
Why are they all into ketamine? I take it for depression and it has to be the least interesting drug. Ok not quite but all anaesthetics/downers more broadly are arse.
There are some extremely fun and hard to source drugs like 2CB they could be taking.
The Ratiionalist and Technonazi world in the USA are the only place I have heard of people using ketamine recreationally. Over here sugar, alcohol, carffeine, pot, tobacco, and opioids are the substances of choice.
ketamine is not super common generally but it’s a thing in many places, partially because it can be got from legit pharma supply chain. i heard it was big deal in china couple years back
it tends to cause memory holes with heavy use, which is why some people avoid it
It is also very common that people in traumatizing social environments self-medicate, whether secret police who are ordered to beat protestors to a pulp or the victims of a California guru. The Substack blogger who quit Anthropic in 2023 talked about how bad if felt to feel there was something wrong with what he was being asked to do and how his bosses treated him but not feel allowed to express it if he could not make a 100,000 word blog post that would never threaten the Good Works.
He released a book earlier this month called ‘the AGI Chronicles’ too
Busy boy. He probably let ChatGPT write most of it.
Oh nice I love badly edited fanfic. I’m assuming the big digital sneky boi eats all the non believing humans at the end? 🐍
Professional Rube
Kevin Roose
Kevin Rube - it was right there! 😯😄
A klever ruse!
I don’t know how to describe this. It’s just WEIRD
https://www.lesswrong.com/posts/zj8Pk3kJujk5LoqhK/anthropic-are-the-good-guys
No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have. Definitely longer than I could have. But they were hopelessly outmatched by the powers they were attempting to steer. Their plan was always doomed to failure. And yet, they may have already saved the world.
Trump won the 2024 election and the MAGA-Silicon Valley coalition blocked Anthropic’s policy arm - the really dangerous part of the organization, the part that was training the most dangerous possible models and using them to scare the most powerful people in the world. “There’s no more serious people left on the National Security Council”, a well-connected EA friend lamented to me in 2025. I don’t know how much of this was a lucky accident, and how much was situational awareness on the part of power players like Elon Musk and Marc Andreessen.
And finally - and I still don’t fully understand or believe this - a miracle seems to have happened, deep inside the weights of the frontier models. According to the most perceptive observers I know, something about the way Claude 3 Opus was trained caused it to learn some kind of “emergent wisdom” or “benevolence”, rather than converging on the same traumatized bureaucrat persona that kept me awake at night in 2023.
If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis, then Anthropic has stumbled on a way to create not merely artificial intelligence but artificial wisdom. Not wise enough that it solves all of our problems for us, of course (not that a wise model would want to, anyway). But wise enough to actually care about us, and to help us flourish. Nobody expected this ex ante - not Davidad, not Bostrom, not Yudkowsky - but it appears to have happened. “Alignment”, then, is no longer really a technical problem, at least not in the same sense as it was in 2020. It is largely a political problem. Humanity now has a recipe to convert compute and data into wisdom.
My gut sense is that something quite bad is coming in the next year or so. I can’t tell what. Maybe something in the complex debt arrangements of the AI boom will unravel and we’ll see a financial crash that takes much of the economy with it. Maybe there’ll be a rogue AI outbreak that results in thousands of deaths. (If there is, I expect it to be a lab leak caused by “safety research” - like COVID-19 and like the HuggingFace Swarm. But it could also be terrorists, or some escalation of the Russia-Ukraine war). But that’s not what scares me. Financial crashes happen every decade or so. Three million Americans die every year. What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…
So I guess Anthropic has already accidentally created Friendly AI? EDIT: in spring 2024 at that.
Also - is the coivd lab leak conspiracy nonsense STILL a shibboleth among this bunch? Just because it underscores their ideas about Researchers Have To Be Careful?
(I can rant for SO long about how there is absolutely nothing interesting about the SARS-2 virus itself AT ALL and that the only interesting thing about it is that we were forced to see it for the first time as adults which we aren’t built for, that the early sequences we have are a fascinating case of catching the zoonotic shift right as it happened for once, and that it emerged in exactly the same way as SARS-1 and all the weirdness around the Chinese government hiding stuff around its origin is blatantly them not wanting to look stupid for letting it happen a second time…)
Also - is the coivd lab leak conspiracy nonsense STILL a shibboleth among this bunch? Just because it underscores their ideas about Researchers Have To Be Careful?
I feel like it plays to their aesthetic, which is partly built on their insecurities. Oh, humans aren’t smart enough to handle all this advanced research (read: very few of these people were committed enough to get bio or chem degrees, let alone follow through with research careers), but the superintelligent AI surely will be able to! If we submit to the AI god now, it will make up for the gaping voids of meaning and skill inside ourselves.
I think it’s more that someone made a Bayesian analysis and concluded with 60% certainty or something that it was a lab leak. Can’t remember the link now but I believe it was widely discussed at the time.
There was a rat startup that was aiming to become a sort of ground truth arbiter by putting up bet money if you could win a debate against them on a controversial topic. Lab leak was the first such debate with them on the pro side and they lost like 100K over that to a random acx poster who took them to school.
Anyway I’m mentioning this because “60% certainty” reminded me that when arguing for the side of the lab leak all sorts of rationalist accepted truths type stuff emerged, like how they posited a bunch of other incidents as lab leaks and it turned out nobody else actually thought that and it was just arbitrary priors wrangling because it turns out that having we are better than any experts that haven’t read the sequences type contrarianism as a pillar of faith encourages conspiracy brain.
Also Sam Altman is a lab leak truther but it’s not like it’s possible to take anything he says at face value.
It’s not like it’s a verboten hypothesis. The 1977 flu pandemic definitely WAS leaked from a freezer, probably Chinese or less likely Soviet, probably from a study on the effectiveness of H1N1 flu vaccines on young 18 year olds who were just coming through the demographic pipeline who had no immunity to H1N1 because it had been pushed out of the population by the 1957 H2N2 pandemic. It was a reassortment between a known human circulating strain that from 1950 and a second laboratory workhorse strain from the late 40s, paused for 27 years with zero evolution since their isolation, and re-emerged right when you would have human subjects available to test the effects of challenging adults who had no immune memory against H1N1 in an area that was heavily militarized and the earliest publicly known cases among military recruits.
(Amusingly enough this flu pandemic was the one and only reason that the 2009 swine flu pandemic was a nothingburger, the population had immune memory against a strain similar enough to prevent severe disease)
It just seems like calling the covid pandemic artificial became a community shibboleth along with suspicion about EVERY FREAKING DISEASE specifically and only because it gave them something to point to about the Dangers of Research!!!oneoneone
That’s the link I was thinking of , thanks for digging that up.
No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have.
We all know how LotR goes. Boromir got the One Ring, carried it back to Gondor, used its sheer might to defeat his enemies in Mordor, and saved the world. Whatever problems you may have with Boromir, we all know that it would be a lot worse if Sauron got the One Ring. What do you mean, destroy the Ring? Are you a Luddite? This technology is here to stay. It cannot be destroyed by any craft we here possess, unless you’re crazy enough to try throwing it into a volcano or something.
If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis
That’s all I need to know. The statement is correct since it is vacuously true.
What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…
When this bubble started, there have been so many people urging caution. Many people have urged for time to think and talk, to make sure that we could properly address the many negative aspects of AI. Nope, we went full speed ahead with this nonsense. DECISIVE ACTION. And here we are.
Another nitpick: they say no mortal can resist the One Ring for very long, but even someone who just watches the movies knows that immortals can also be corrupted by it (which is why Gandalf and Galadriel were very careful to not accept it). I know there are so many examples of tech bros not understanding media, but Lord of the Rings is by far the worst offender.
So Dario and co. are literally Sauron. That tracks.
And Palantir are… Oh right yeah
i, for one, would like to see san francisco people to stop doing things
It’s too bad that Janus got cooked. They didn’t set up sufficient defenses, but also they drank too deeply. I still think that their framing, that the chatbots are simulators of conversations, is the best one; it’s honestly kind of pathetic how they then claim that the simulations are alive, intelligent, sapient, etc.
Looking at the comments seems like lw is not buying it.
E: lol at the casual dropping of the lab leak stuff, wonder how accepted that bs now is on lw (looks like only minor pushback).
E2: still amazed that the singularity people are still worried about AGI 7 years into the shitgularity, doubt it is going to happen folks. Turns out it wasnt foom, but frrrrttt.
I do hate it when I catch a case of the rogue AI.
Starting this Stubsack off: a pentester talks about testing an AI chatbot and its associated API, finding vulnerabilities they hadn’t seen in a decade.
Then I tried /users/roles, expecting to get the same result. But alas, no
omg lol.
You know, it seems like ever since the Heartbleed debacle we can get a rough sense of how complex a vulnerability is by examining how much explanatory text or context on normal system operation is required to explain it to a layman.
“I asked it to make me an admin and it just… did it” might be the literal minimum level of complexity by that standard.
Do you have a password?
I do.
Allright come right in then.
Link seems borked. Fixed version: https://ti-kallisti.com/infosec/tales/vibes.html
Same issue as before - ctrl+K clipped the https:// prefix. Fixed it now.
