I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
This alarm’s being rung for over a year now, so “calling it now” means finally reading the writing on the wall
Let it be known that the first person to call it was actually Sam Altman when OpenAI’s paper on AI Scaling Laws in 2020 subtly showed that the diminishing returns will stop showing improvement with infinite power, compute time, and data before 94% accuracy is reached.
yeah I was kinda like. calling it?
They are not the automated from 0 to 100 coders that some people claim them to be. But they are quite capable, definitely much more capable than what anyone could have imagined ten years ago. Given well defined problems they can excel at even relatively complex tasks. I pointed Claude at a latex file of a somewhat complicated nonparametric statistical estimate calculation to look for any mistakes and it was actually able to find some. I then pointed it at a code that replicates the calculations and it was also able to correctly identify some issues with the code. I think this is the way one should use LLMs, not let it loose on coding tasks. In the former way you won’t even be able to burn through your first tier account quota where as in the latter the LLM will likely end up getting in weird loops burning tokens like there is no tomorrow. Also this method of sane usage of LLMs is much more suitable for open local LLMs. I don’t think there is any doubt anymore that LLMs can be very useful tools, not just for doing stuff but learning it too. People should move past the stage of invalid criticisms like “they are just stochastic parrots” and move to more serious matters like environmental impact, greedy fucking CEOs pretending LLMs are replacements for humans, degredation of skills, getting lazy at checking AI code, ethics of capitalizing on collective human knowledge and the unsustainable AI bubble that tech companies are pushing for.
This is very obvious unless you are in tech leadership, in which case your job is now to push this at all costs and suppress dissenting voices.
if it’s broken in a way that can’t be detected, is it actually broken?
all software is broken in some way. if the rate of bugs generated by llm and the severity of those bugs drops below the rate you would expect from a human programming team, then llm is offering something competitive.
It will eventually be detected, but it passes tests before hitting production, that is the problem.
deleted by creator
No, humans make less mistakes. Less. That’s the key here, statistical models are trained on human data so by pure logic can never, ever, under any circuimstance, reach 100% accuracy. With current understanding of LLMs with a focus on AI Scaling Laws, and more importantly of natural human language adaptation, they will never reach 94% accuracy with infinite power and infinite training. That’s what the curve shows us in OpenAI’s 2020 research paper on AI Scaling Laws and later Deepmind’s paper correcting their math, that the diminishing returns will hit a limit far before convergence.
In addition to that, the AI also cannot detect subtle changes to established problems or any new unaccounted for variables, because they’re a statistical model and not capable of actual thought. They also lack any sense of responsibility for their actions for the same reason.
You fucking sloppers always try to say “HuMAnS mAkE misTAKeS, TOO!” Yeah and the fucking slopbots are trained on those mistakes and make them again but worse.
But you’re forgetting the key difference that makes it so much worse - we can fix human mistakes especially if we can talk to the human to figure out how. With an llm we have no external reference, only poorly designed code where the comments are there to guide the writing, not describe what was written. So it’s much harder to debug an output, and the llm cannot be trusted to clean it up either.
You can totally fix AI-written code with AI. You tell it something is wrong, it tries to fix it.
I did a recent experiment with AI writing a document format converter and that’s exactly what I did. It wrote some code, I checked the output, found a formatting issue or similar, asked it to fix it, repeat. It works unreasonably well and with Fable the final code isn’t even bad.
You can fix problems, if you know they are there and there is a model of that problem being fixed.
You can’t fix problems you don’t know are there, or do not have modeling.
That’s true for humans too surely? How would anyone fix problems they don’t know exist?
Humans generally don’t hallucinate libraries or documentation. If there is a bug or error on a human maintaine repo the human in charge will generally know what went wrong and how to fix it, the AI will just gaslight your ass because the AI has no idea.
This is true. But it doesn’t invalidate any of my points. Humans have unique failure modes too.
To add to the other response - it is much more difficult to work with Ai to debug inconsistent issues or similar unless you can understand the code and step through with a debugger to check for race conditions or similar.
Recently I was working with an Ai tool for some c code that depending on machine ran wildly differently. The Ai was unable to identify any issues, and kept recommending fixes for hardcoding values or similar that I had to revert. The fix ended up needing to use valgrind to create a different enough environment to see how a race condition was made to properly have one async call delay for the other.
AI can be powerful, and humans can be dumb. But if the code was human made, I would not have needed 3 hours to find a problem, and I wouldn’t have tried to turn to AI for a simple fix because I’d know what I was looking for to start with.
Also, unlike say, writing articles, like 75% of coding is copy.pasting blindly from elsewhere anyway.
You know the feeling that you want to rewrite a project? But you know that most rewrites are a bad idea.
Be it your own, old code. Or code you inherited.
There is a small chance that the world realizes that they went in the wrong direction and nothing can get fixed. That will be the time of rewrites.
No, I don’t expect this to be very likely. The agent code will remain, and human programmers get yelled at for not fixing it fast enough.
It’s so nice to see some people speaking reason. If only any of those people ran multibillion dollar companies.
Let them fail and then scramble to rehire.
Agents cannot program
This is just factually incorrect. Difficult to get past a false assertion of this magnitude.
They are a highly sophisticated statistical model designed to mimic the distribution of programming.
I thought we had got over the stochastic parrot nonsense by now.
You can totally have objections about the ability of AI to program - how good it is, poor failure modes, high cost, technical debt, knowledge debt, broken social contracts, etc. All valid.
But if you’re still in the “It’s just a next word predictor! It can’t really think!” stage of denial, even now… Sorry you’re an idiot.
Um… but it is just a sophisticated statistical model… that’s literally what the math underpinning machine learning models is… and all it can do is make associations based on correlations within the field of the training data. That’s what it does.
This is like saying “but it is just a sophisticated network of neurons. All it can do is transform input signals into output signals. That’s what it does.”
Not really.
A machine learning model is a computer program. It is fundamentally a math equation, which we understand completely.
A living brain is not fundamentally a math equation, and is not purely a statistical model, at least not in any empirically demonstrable way. We don’t understand completely how it works, but we do know that it’s more complex than what you’re trying to imply.
The comparison is not valid. Machine learning models are not an equivalent to a biological brain.
A machine learning model is a computer program. It is fundamentally a math equation, which we understand completely.
Lol you couldn’t be more wrong about this. One of the most widely commented things about DNNs is that we don’t really understand them completely. I don’t know how you would miss that if you knew anything about AI at all.
A living brain is not fundamentally a math equation
It is. A very complicated one, sure. Which part of the brain do you think is impossible to simulate with maths?
One of the most widely commented things about DNNs is that we don’t really understand them completely. I don’t know how you would miss that if you knew anything about AI at all.
No one who actually works on digital neural networks thinks this. We may not be able to predict the behavior of a particular neural network with certainty (because there are a lot, like millions, of variables), but that does not mean that we don’t understand how they work.
Which part of the brain do you think is impossible to simulate with maths?
simulation ≠ reality
simulation ≠ reality
Yes it is, if your simulation is sufficiently accurate. Let me rephrase. Which part of the brain do you think has behaviour that is required for intelligence/consciousness and that behaviour cannot be replicated with sufficient accuracy on a computer (of arbitrary capacity)?
Um… but it is just a sophisticated statistical model…
The mistake has been thinking this implies LLMs can never do X task, and using it as a catch-all argument for any value of X, but it isn’t a good argument because it has been wrong for most of those.
The mistake has been thinking this implies LLMs can never do X task
As this article points out, an LLM can spit out chunks of regurgitated code that it scraped from the internet, but that does not make the LLM a programmer. The resulting output is an attempt to find an existing pattern in the database which fits with what the user has asked for, but it is not a product of actually understanding the use case for the code. It is just statistical correlation.
So, sure, an LLM can be set up to generate output related to X task. If you can collect and clean data that can be used to train the kind of output you want, it should be able to produce an approximate facsimile of the results you want. Is that valuable for your use case? Maybe.
We’re still just talking about what is essentially a complex search function. The statistical model returns results from its database that correlate most closely to your input. That does not mean it returns the right answer. If there is no good correlation, it will still return a result.
As long as you understand that the result you get is just a correlation based on your input and may or may not be relevant to your specific problem, and you are not fooled into believing that the LLM actually understands what you’re asking and produced a result by “thinking” about it, then you might be able to use an LLM as an effective tool - to search a large collection of information for something that is relevant(ish) to what you’re asking for.
The real mistake has been broad misunderstanding of what LLMs actually do, and trying to use them as general-purpose problem solving tools (or worse, as accurate and reliable sources of information).
Language models are not databases and they are not markov bots (similar function but work directly using statistical word association maps). The big difference is that those things are algorithms someone wrote and can fully comprehend what they do, but machine learning models are large algorithms built by another algorithm processing training data. There is much more uncertainty about what is going on under the hood.
There is also great uncertainty about what concepts like understanding or thinking might mean in computer science terms. The main thing we can really know is that ultimately a human mind is a computer, which means that understanding and thinking have some yet unknown mathematical representation, and therefore a comparison can be made. We should eventually be able to quantify whether or to what extent a given algorithm thinks. But you said in another comment that you don’t believe minds can be represented mathematically; this should mean that such comparisons would be apples to oranges, but you’re making them anyway for some reason, and implying they have predictive power for the limitations of LLMs.
Certainly they do have limitations, at least individually and possibly as a technology. There are things given models are bad at, there are things they initially seem to be able to do well as humans but fail in different ways that suggest over-reliance on pattern matching. But these have been determined empirically through testing. The idea that they are “just statistical models” and this knowledge can be used to say what is impossible for them from philosophical first principles keeps getting repeated but has never worked in practice. The reality is that no one knows enough to say for sure where the line is.
Language models are not databases and they are not markov bots (similar function but work directly using statistical word association maps).
Except that it’s been demonstrated multiple times that original training data can be extracted from a language model, so it is completely valid to talk about the model as a database, because the training data is stored within it.
Here’s a broad survey of more than 100 research papers demonstrating this: Training Data Extraction From Pre-trained Language Models: A Survey
There is much more uncertainty about what is going on under the hood.
So, this is a good anology in this case.
See, I know how an internal combustion engine works. I don’t know, by looking at the hood of a particular vehicle, how exactly a specific car’s engine operates (maybe it has 4 cylinders, or 6 or 8, maybe it has fuel injectors, maybe it has a carburetor, etc). However, I do know that the principles are the same for all internal combustion engines, and that just because I don’t know the details of how a particular engine operates, that does not mean that its operation is beyond my understanding.
The same is true for machine learning models. There may be uncertainty as to how a particular model operates “under the hood”, but the principles of operation are the same for all, and are not incomprehensible.
The main thing we can really know is that ultimately a human mind is a computer
We actually don’t know this. This is called computationalism. It is speculative, there are several alternative theories, and little in the way of experimental evidence supporting any particular theory.
The idea that they are “just statistical models” and this knowledge can be used to say what is impossible for them from philosophical first principles keeps getting repeated but has never worked in practice. The reality is that no one knows enough to say for sure where the line is.
You have to understand, the current branch of machine learning models grew out of algorithms whose purpose was processing large data sets with thousands or millions of variables and optimizing for areas in the data set where many of those variables were maximized (or minimized). Here’s a better explanation:
Hill Climbing Algorithm & Artificial Intelligence - Computerphile
How these tools perform their optimization, and what they optimize for, has been recombined in different ways to produce different types of models, and the search space of variables has been expanded with increased computing power, but the underlying operating principles are still the same. This is not a tool that can comprehend what it is doing, it can’t be self-aware. It can only process large amounts of input data and attempt to maximize for particular dimensions. This seems vague to humans because the amount of variables being handled at any given time is far more than a human mind can focus on, but that doesn’t make the optimization routine intelligent or conscious. It’s just doing a lot of number crunching really fast, optimizing for specific aspects as directed by its developers.
Except that it’s been demonstrated multiple times that original training data can be extracted from a language model, so it is completely valid to talk about the model as a database, because the training data is stored within it.
It’s been demonstrated that some more prominent pieces of training data can be reproduced, the majority of it cannot. This shows that those particular pieces of data are represented in some form within the model, it does not show that the way it works is equivalent to database lookups. If I can write down the lyrics of a song from memory, it shows that those lyrics are encoded as data in some form in my brain, but that doesn’t mean it’s valid to talk about my brain as a literal database, especially not in the sense that the limitations in the capabilities of a database can be ascribed to me (or its strengths, I cannot remember the exact lyrics of most songs I’ve heard, even if I can remember some).
Hill Climbing Algorithm & Artificial Intelligence - Computerphile
This video literally starts out by describing evolution as a similar optimization algorithm. If you know the basic mechanism of evolution, does that mean you can use that to then say with certainty and specificity what biological life in its vast diversity of techniques is not capable of? The “underlying operating principles” of evolution don’t “understand” chemistry or deception, but they still produce organisms capable of photosynthesis and camouflage. It’s an algorithm that produces other algorithms, which is what puts those resulting algorithms in a different category of comprehensibility than fixed algorithms that were explicitly written by someone. We are very far from having a comprehensive understanding of biological systems, despite knowing how evolution works.
This is not a tool that can comprehend what it is doing, it can’t be self-aware. It can only process large amounts of input data and attempt to maximize for particular dimensions. This seems vague to humans because the amount of variables being handled at any given time is far more than a human mind can focus on, but that doesn’t make the optimization routine intelligent or conscious. It’s just doing a lot of number crunching really fast, optimizing for specific aspects as directed by its developers.
This is like saying evolution is only a simple mechanism taking in the world as data, which, yeah, obviously, but that property doesn’t carry forward to what it produces. The bigger problem here though is, again, concepts like comprehension, consciousness, and intelligence are not well defined in computational terms, and it is unclear what statements involving them mean in any practical sense. These sorts of claims are non-falsifiable and don’t make testable predictions about the boundaries of AI capability.
Our apologies. Let us issue a correction:
“Agents should never be allowed to program”
There.
part of the issue as well is that when they get something completely broken, people just re roll the output until they get something that’s broken in ways they don’t notice. Or re roll parts of it, or tell the system to judge if the output is broken and re roll the parts that it judges are broken automatically. Or increase the size of the context window to get it closer to that upper limit of accuracy.
All this together can get a more functional output with less effort, and as people find these tricks it gives them the illusion of an upward trend in capability, like this is all solvable issues that will improve as time goes on. Big problem with that though, theses tricks and methods explode the compute cost rapidly. That’s all fine and dandy when everyone is getting their compute costs for these tools subsidized by these model providers, but eventually they will need to charge the real cost of running this. The compute providers that host the model providers are also running at a loss, trying to help grow the market segment and maximize their market share. And then places that have the datacenters in them are giving tax breaks and discount utilities to attract new construction.
Everyone except the people making the chips is selling at a loss, and as people pile on usage to make up for the fundamental limitations of these systems, the demand balloons, validating to the providers at all levels that this is a growing market they should invest more in to.
But eventually… they need to make money. The bill comes due on all the debt and investment. What happens to the people who have fully embraced these to run their businesses? Or to all the people who have built their skill set around using these systems? It’s a crisis, a series of crisis, each time a debt wall gets hit by someone in the supply chain. A half decade of technical debt that just got really expensive to deal with, and not enough experienced people to handle it, since all the grey beared retired and not enough new people got brought in to replace them because the entry level work was automated.
It’s going to be a wonderful time to be a Freelance Senior Developer and above in a few years.
But they do work, maybe not as a full replacement but my god the amount of boilerplate I can avoid in creating unit tests from scratch. Extracting and finding information in the code base is also useful, not everything is an easy text search of tracing a few code paths. It’s an incredible tool for these kinds of work.
If it becomes harder to tell the difference then it also means it’s closer to matching reality. And todays AI can do very impressive “reasoning”, managing to debug complex issues I have had.
The most important part is that you as developer is fully responsible and can stand behind what they do and deliver using AI agents.
Right? The bottle has opened. It has taken so much mundane work out of programming. Also, I feel like a human is just as likely to create great looking code changes with a possible flaw. You just have to review the code. Whether it’s a person or a bot, “lgtm” can only be used sparingly.
If my job mandates me to use ai agents, idgaf, I’ll use it. But my every oss contributions will be clanker-free
Calling them a statistical model is misrepresenting their functional understanding of concepts.
This guy does not get it.
