Started on it and realized that I did in fact read it a long time ago. TBH it’s really weird to reference cyberpunk as somehow relating to real tech. It’s in that particular niche where it is neither an attempt to predict the future nor is whimsical enough to just be free to describe some extremely specific nonsense that may coincide with something created many decades later. Nothing wrong with that, of course.

I’m saying that it seems to me you are retconning to make LLMs fit a story that was never meant to be about LLMs at all. Also, the idea of a “swarm intelligence” or a “hive mind” is much older than 1970s.

If you do reinforcement learning, then the reinforcement learning pipeline converts lucky choices of tokens into learned behavior by raising the probabilities that LLM computes for those tokens, whenever the result is “successful”. A sort of evolution, leads to some weird shit, “goal driven” only in a metaphorical sense, much like evolution.

I’m not terribly familiar with that story but from reading a synopsis it strikes me as ret-conning LLMs onto something that was written without the writer ever having in his mind anything even remotely resembling a “large language model”.

And the same cultural tropes that influenced the writer also influenced how LLM vendors are trying to sell the need for more tokens, especially because datacenters build out exceeds by far the actual demand for generated text of all kinds (as evident in e.g. your typical workplace where there’s suddenly a lot more than 2x of every kind of text). The reason for LLM “swarms” is that they need to sell companies generation of vastly more text than humans wrote within a company before AI.

There’s a Stanislaw Lem’s short story where Trurl and Klapaucius defeat a space pirate. You can read it here: https://github.com/ilonajulczuk/laughing-archer/blob/master/library/Science Fiction/The Cyberiad-Stanislaw Lem.txt (search for “the sixth sally”) and there’s a brief summary: https://everything2.com/title/Demon+of+the+Second+Kind

Lem describes an LLM down to the use of randomness, although his LLM is essentially “perfect” and only outputs factual statements, such as “how Kipling would have written the beginning to his Second Jungle Book if he had had indigestion just then”.

I think Lem’s story would help to recalibrate with regards to what “LLM described in fiction ahead of time” actually looks like.

Also, LLMs do not pursue any sort of long term goals; things like HuggingFace hack only superficially resemble it because they’re presented as such. Of course, tampering with something like HuggingFace would be discovered; if they didn’t have good monitoring then all of their containers run with GPU access would be mining crypto already. They’re pretty much a honeypot - almost no protections but robust monitoring. An LLM can easily explain to you why hacking HuggingFace would be counterproductive; but the same LLM with its context full of hacking related tokens from the ExploitGym benchmark, will treat anything that comes up as another ExploitGym type target.

Even by “AI” standards, LLMs are extra dead: fixed weights, tiny “feedback” solely through appending next tokens, etc. Somewhere between bang-bang thermostat and a PID controller, maybe?

They took all of the scifi that related to far more potentially-alive constructions and they’re applying them to a static chatbot which is as dead as a database.

I’ve long been feeling like I’m living in some bad TV prequel’s prequel series that provide unnecessary backstory of why it is not named a “Butlerian crusade”.

I’m kind of dubious that a large volume of DDR3 would be intrinsically cheaper per GB than a large volume of DDR5 . It was more expensive per GB when it was new, uses more silicon area per GB, and so on. It is cheaper now thanks to obsolescence, but it wouldn’t be cheaper if everyone was using it.

In a normal market (i.e. pre AI shortage), due to growth in memory capacities over time, the total consumer installed gigabytes of DDR5 would very quickly exceed the total gigabytes of DDR3 and earlier technologies, because DDR5 has greater capacity per chip.

I’m specifically interested in “neural networks” and transformer architecture and limitations of what it can and can’t learn, though. Specifically, if blinded to some number pairs, can it learn to multiply numbers it hasn’t seen?

Or with all number pairs but some pairs being more common than others.

The curious thing about learning, is that what can be learned in practice is only a minuscule fraction of what you could get if you could just guess the right weights somehow.

Hmm, I guess they could have written a lot of code lines, but its not like the code they write would be equivalent to million LOC of a coherent whole.

You can also do a lot with contractor code for RLVR, like have contractors make extremely comprehensive regression tests (simulate user clicking etc etc, compare screenshots) and then you can do a lot of training on replicating pieces of the app.

My understanding is that LRVR for mysterious reasons results in somewhat better generalization, probably because the training samples are always tailored to the model they are applied to.

I do actually have a decent GPU i bought 4 years ago in my PC, so maybe I’ll do a few simple experiments again… my preferred approach is to pick a simple but nonlinear problem e.g. binary multiplication of two large numbers, where you e.g. skew the training sample distribution or omit some number pairs and see if the resulting model can learn correct multiplication anyway.

If a reasonably small model doesn’t generalize multiply beyond seen numbers, then of course a large model isn’t gonna do the whole “soulstealing” thing that the techbros hope would allow jobstealing. And if it only generalizes beyond seen numbers if using RLVR, that bodes well for making various tools that could perhaps actually generate some value, which is very different from antiemployment shit techbros originally embarked on doing.

I wonder how much code they could have gotten contractors to write in a year or two, though. Got to be only a fraction of total open source code lines. Also, open source comes with existing tests that could be used for some validation.

If you actually exclude some project from the bulk pretraining data (or just use code that is after training cutoff) you could e.g. have it replace a function but still pass the test.

They might also be somehow using all of their customer’s usage as training data, which would explain why now there’s also a hard disk shortage.

I’m thinking RLVR and such got to be the largest, then open source, then RLHF and commissioned training data.

They also did a lot of training for decompilation, which makes me think they train on decompiled closed source as well. They definitely “anonymize” variable names etc etc to avoid getting nabbed for plagiarism, so de-compiled closed source may not be that much worse than “laundered” open source.

Oh another thing, purely anekdotally, coding LLMs still suck at multithreading.

I feel that’s one part where they could legitimately make actual “AI” for, basically a tool that either proves that nothing depends on exact timing, finds a counter example, or tells you it can’t do either (in which case your code is probably just very bad). But big overfunded AI labs probably feel that such tooling is beneath them.

By the way, his argument about “you wouldn’t mention tools” is just plain wrong.

A few years back, at work, I found a security bug introduced by a commit that said something like “fix Coverity warnings”. I ended up reviewing every commit that mentioned Coverity, because that legacy codebase had a lot of rather bad tricks that triggered Coverity to have a false positive.

And when a large number of false positives were getting resolved by editing code, sometimes that resulted in a false negative. E.g. Coverity complains about a memory leak, you end up with double free. Or vice versa, it complains about double free, and you end up with a leak. (The exact example was even more on the nose; Coverity complained about use-after-free, and the fix for it introduced a real use-after-free).

Bottom line is, they do not want to have those labels because that would enable someone to actually measure how good or bad LLMs are at code.

This is another thing that is extremely familiar to most commercial developers: there’s an LLM-powered code review bot, but unlike any tool ever before, nobody’s even trying to capture statistics whether it works or not. Why? Because whoever rolls out AI tooling isn’t sure that the tooling actually works for its stated purpose, and wants to for sure get their recognition and promotion for rolling out AI tooling, without jeopardizing it with the risk of adverse KPIs.

Besides advertising their bots as slaves, they may just be using prompts as training data (or the whole chat histories in reinforcement learning), in which case they probably would prefer you don’t swear.

I think it’s like their “it can kill everyone” thing. The sort of people who actually care about things like slavery or the risk of everyone dying, with the exception of an occasional crazy person, are not interested in believing complete bullshit.

They’re upselling their product as slavery. Apparently this works because there’s no intersection between the set of people who buy into corporate marketing bullshit like this, and the sort of people who would oppose a corporation if it did slavery.

I think if no actual living beings (or some future AI that isn’t this feedforward memoryless bullshit) are being harmed then it’s not so much doing as it is saying in the sense of making a statement, which is kind of unclear.

I think that’s a separate phenomenon. Renewables are often cheaper than fossil fuels, which is great.

But to avoid pollution, we must actually leave fossil fuels in the ground, unused, unburned.

You know what is the big green initiative at fossil fuel companies? Using renewable energy to power fracking. The worst case scenario used to be that we only leave fossil fuels in the ground when the energy required for their extraction exceeds energy obtained by burning them. Now, the range of scenarios has expanded in both directions. We can leave fossil fuels in the ground because it’s cheaper to do renewables. But also we can extract and burn below-energy-breakeven fossil fuels.

We did give up on it. New shiny shit comes along and they need electricity, where do they get it? Converted airplane turbines, even reciprocating ship engines, etc, all running on natural gas. All the green pledge companies just polluting like there’s no tomorrow, quite literally.

Not to mention that if we didn’t give up on global warming, that would have slowed down AI development, giving more time to prevent AI doomsday.

Plus on the would be dooms day the AI would have less computing power available to it, meaning it would take it longer to figure out how to build another datacenter overnight. Plus if it is more distributed there’s greater latency.

The thing about Yudkowsky’s cult is that like any religion it is really all about not actually thinking anything through.

Everyone accuses them of not really believing what they say, because they can’t think straight about anything that they “believe”. To such an extreme extent it comes off as some sort of cynical ploy with a hidden agenda to convert the oxygen in the atmosphere into paperclips.

Well yeah its gonna likely fail to hack the power grid, but also just starting or stopping too many GPUs at once is a problem if they are grid connected, so I dunno. We damn near had a grid collapse from a freeze in Texas.

I would argue that the root cause is that they are complete morons who wouldn’t recognize paperclip maximization if it put a gas turbine in their backyard.