I feel this – we had a junior dev on our project who started using AI for coding, without management approval BTW (it was a small company and we didn’t yet have a policy specifically for it. Alas.)
I got the fun task, months later, of going through an entire component that I’m almost certain was ‘vibe coded’ – it “worked” the first time the main APIs were called, but leaked and crashed on subsequent calls. It used double- and even triple-pointers to data structures, which the API vendor’s documentation upon some casual reading indicated could all be declared statically and re-used (this was an embedded system); needless arguments; mallocs and frees everywhere for no good reason (again due to all of the un-needed dynamic storage involving said double/triple pointers to stuff). It was a horrible mess.
It should have never gotten through code review, but the senior devs were themselves overloaded with work (another, separate problem) …
I took two days and cleaned it all up, much simpler, no mem leaks, and could actually be, you know, used more than once.
Fucking mess, and LLMs (don’t call it “AI”) just allow those who are lazy and/or inexperienced to skate through short-term tasks, leaving huge technical debt for those that have to clean up after.
If you’re doing job interviews, ensure the interviewee is not connected to LLMs in any way and make them do the code themselves. No exceptions. Consider blocking LLMs from your corp network as well and ban locally-installed things like Ollama.
It should have never gotten through code review, but the senior devs were themselves overloaded with work
Ngl, as much as I dislike AI, I think this is really the bigger issue. Hiring a junior and then merging his contributions without code reviewing is a disaster waiting to happen with or without AI.
It used double- and even triple-pointers to data structures
(old song, to the tune of My Favourite Things)
🎶 “Pointers to pointers to pointers to strings,
this code does some rather unusual things…!” 🎶
The study was centered on bugfixing large established projects. This task is not really the one that AI helpers excel at.
Also small number of participants (16) , the participants were familiar with the code base and all tasks seems to be smaller in completion time can screw results.
Thus the divergence between studio results and many people personal experience that would experience increase of productivity because they are doing different tasks in a different scenario.
familiar with the code base
Call me crazy but I think developers should understand what they’re working on, and using LLM tools doesn’t provide a shortcut there.
You have to get familiar with the codebase at some point. When you are unfamiliar, in my experience, LLMs can provide help understanding it. Copying large portions of code you don’t really understand and asking for an analysis and explanation.
Not so far ago I used it on assembly code. It would have taken ages to decipher what it was doing by myself. The AI sped up the process.
But once you are very familiar with a established project you had work a lot with, I don’t even bother asking LLMs anything, as in my experience, I come up with better answers quicker.
At the end of the day we must understand that a LLM is more or less an statistical autocomplete trained on a large dataset. If your solution is not on the dataset the thing is not going to really came up with a creative solution. And the thing is not going to run a debugger on your code either, afaik.
When I use it the question I ask myself the most before bothering is “is the solution likely to be on the training dataset?” or “is it a task that can be solved as a language problem?”
I talked to Microsoft Copilot 3 times for work related reasons because I couldn’t find something in documentation. I was lied to 3 times. It either made stuff up about how the thing I asked about works or even invented entirely new configuration settings
In fairness the msdn documentation is prone to this also.
By “this” I mean having what looks like a comprehensive section about the thing you want but the actual information you need isn’t there, but you need to read the whole thing to find out.
Reading the paper, AI did a lot better than I would expect. It showed experienced devs working on a familiar code base got 19% slower. It’s telling that they thought they had been more productive, but the result was not that bad tbh.
I wish we had similar research for experienced devs on unfamiliar code bases, or for inexperienced devs, but those would probably be much harder to measure.
I don’t understand your point. How is it good that the developers thought they were faster? Does that imply anything at all in LLMs’ favour? IMO that makes the situation worse because we’re not only fighting inefficiency, but delusion.
20% slower is substantial. Imagine the effect on the economy if 20% of all output was discarded (or more accurately, spent using electricity).
I’m not saying it’s good, I’m saying I expected it to be even worse.
Does that imply anything at all in LLMs’ favour?
Yes it suggest lower cognitive load.
1% slowdown is pretty bad. You’d still do better just not using it. 19% is huge!
Would ai coders even get faster over time or just stay stagnant since they aren’t learning anything about what they’re doing
I wish AI was never invented, but surely this isn’t ture.
I’ve been able to solve coding issues that usually took me hours in minutes.
Wish it wasn’t so, but it’s been my reality
LLMs making you code faster means your slow not LLMs fast
I doubt anyone can write complex regex in ~30 seconds, LLM’s can
yes they can. I regularly do. Regexes aren’t hard to write, their logic is quite simple. They’re hard to read, yes, but they are almost always one-offs (ex, substitutions in nvim).
Regexes aren’t hard to write, their logic is quite simple.
He did say complex regex. A complex regex is not simple.
yes, “complex” regexes are quite simple too. Complex regexes are long, not difficult. They appear complex because you have to “inline” everything. They really are not that hard.
This is stupid pedantry. By that logic literally nothing is complex because everything is made up of simple parts.
cryptic != complex. Are they cryptic? yes. Are they complex? not really, if you can understand “one or more” or “zero or more” and some other really simple concepts like “one of these” or “not one of these” or “this is optional”. You could explain these to a child. It’s only because they look cryptic that people think they are complex. Unless you start using backreferences and advanced concepts like those (which are not usually needed in most cases) they are very simple. long != complex
I’m not trusting a regex written by AI
You don’t have to. You can read it.
That’s why you write tests
tests can never prove correctness of code. All they can prove is “the thing hasn’t failed yet”. Proper reasoning is always needed if you want a guarantee.
If you had the llm write the regex for you, I can practically guarantee that you won’t think of, and write tests for, all the edge cases.
Proper reasoning is always needed if you want a guarantee.
You formally verify your regexes? Doubtful.
Depends on what you need to match. Regex is just another programming language. It’s more declarative than traditional languages though (it’s basically pattern matching).
Pattern matching is something I already do a lot of in my code, so regexes aren’t that much different.
Regardless, the syntax sucks. It takes some time to get familiar with it, but once you get past that, it’s really simple.
You have definitely never worked with a regex guru.
No, but not everyone is a regex guru.
If AI can write code half as good and fast as a regex guru, it’s going to increase the average dev’s productivity a lot
If you find yourself writing regexes often enough that speeding up that process would increase your productivity by “a lot”, then you should get good at writing them yourself which means practicing without an LLM. If its something that you don’t do often enough to warrant getting good at then the productivity increase is negligible.
I think the main benefit here isn’t really productivity but developer comfort by saving them from having to step out of their comfort zone.
does the regex search for what you wanted to? Does it work in all cases? Can I be confident that it will find all instances i care about, or will I still have to comb the code manually?
🎵Had a great day out,
Callin’ my name like Ferris Bueller,
Time to wrap this up,
I’m getting 19℅ slower! 🎵
Well that’s a strangely deep cut Ken Ashcorp ref
I am honestly shocked to see a reference in the wild to Ken Ashcorp.
I’m honestly shocked that multiple people got the reference.
Removed by mod
