recently I gave AI a task to configure a software and the software didn’t have any docs of how it works only the source code. the agent went through all the files and generated a summary of how it works and why it’s not working for my particular scenario and suggested edits to my docker-compose.yml file. I couldn’t believe it since the bug was very hard to find and it used headless firefox to find why it wasn’t working.
made me realize do we still need documentation of how a software work when a AI can easily explain it?
You must log in or # to comment.
Every time you use AI a puppy dies.
A new account making extraordinary claims about AI and phrasing it as a question? Again?
As much as I can’t stand AI fanatics, I think some of the anti-AI rhetoric (especially on Lemmy) can be just as dogmatic. OP didn’t make any “extraordinary” claims. LLM agents have actually gotten remarkably good at many programming tasks, in particular tracking down bugs, and some of the things they’re able to work out are genuinely impressive. At the end of the day it’s another tool, but it can be a powerful one in the right context.
That’s not to say anything about the environmental, social, and economic issues surrounding the tech and the industry. Obviously large companies are acting completely recklessly in all of those domains and I don’t intend to justify that aspect. That said, I think it’s disingenuous to frame OPs post as “making extraordinary claims” and it detracts from legitimate (and IMO mostly unimpeachable) arguments against AI.
Removed by mod
Good point, I guess I rescind my argument.
Removed by mod
I am sorry but I couldn’t resist AI. was avoiding it for very long time but it’s actually good. specially for tedious task which I have to do. now I can focus on the fun part :)
More like couldn’t resist obvious astroturfing
This is not an extraordinary claim if you’ve used a good LLM in the last, say, year.
That’s right, move those goalposts!
How did he move them?
“You’re not using the latest model”, “You’re holding it wrong”, etc…
It’s the same shit, different day.
I’m consistently told LLM generated code is the best thing since sliced bread, and people keep handing me “proof”, yet every time I open what they’ve handed me, I see a shit sandwich.
And as soon as I point that out, the goalposts are moved yet again, ad infinitum.
I think the effect you’re seeing is that different people have different levels of “good enough”, and also they work in different domains where AI became good enough at different times. E.g. for one-off web page design it’s been good enough for a long time.
I never said this for earlier models, because they did suck for things like Rust and SystemVerilog. Now they don’t.
I’d share some things I’ve vibe coded that work basically perfectly but they’re linked to me real name, but you could have a look at the Mold linker which was recently vibe-ported to Rust.
E.g. for one-off web page design it’s been good enough for a long time.
In just the last two months, I’ve been handed the following web projects:
- A page filled with bootstrap style div-soup
- A page filled with tailwind style class-soup
- A slideshow webapp that has duplicated state and three different ways to change slides (
navigate(),goTo(), andchangeSlide()). - A slop-grenade of a chatbot interface for navigating a user through a flow chart, documentation of the flow chart, and documentation of the logic of the flow chart, all of which conflict with each other.
I think the effect you’re seeing is that different people have different levels of “good enough”
I think the effect that I’m seeing is twofold:
- People don’t realise that every line of code is a liability.
- People think that once they have something on the screen that looks finished, the project is finished. They haven’t thought about support or maintainability at all, and since they are completely inexperienced about such things: they aren’t qualified enough to when when a project is “even done”, let alone “good enough”.
People think that once they have something on the screen that looks finished, the project is finished.
For .one-off web page that is true. You and I might not like that under the hood it’s a mess but we have to accept that if it works anyway and nobody is going to need to change it, that’s fine.
Also the latest models don’t even make a mess any more.
You do understand that this is a rapidly developing technology right? That it’s getting better weekly?
It’s clear you have no idea what you’re talking about. You can’t just bury your head in the sand and tell yourself that LLMs are still just “advanced autocomplete” anymore.
It’s clear you have no idea what you’re talking about. […] You do understand that this is a rapidly developing technology right? That it’s getting better weekly?
This is literally the retort that moves the goalposts.
You can’t just bury your head in the sand and tell yourself that LLMs are still just “advanced autocomplete” anymore.
I made no claim that LLMs haven’t changed, I make the claim that others are making claims that fall apart once examined.
The claims others make:
-
AI is amazing because now it can make this thing
-
Oh, the only reason doesn’t hold up to scrutiny this one time is because of {insert goal post moving here}.
Argument 1 falls apart when looking inside the product (i.e. the shit sandwich, the slop grenade, etc…).
Argument 2 is circular, so I’m not going to waste time on it.
But just for the sake of argument, lets say there’s some sudden release of a model tomorrow that runs one million times better than today’s best model and everyone only uses that model from now on. Will I still be handed a shit sandwich?
Yes.
Why?
Because the fact that I’m handed a shit sandwich isn’t influenced by how good the model is. It’s based on the fact that English is a terrible language for logic.
We discovered this in the 60s and 70s. Computer programming was too hard in the 50s and only getting harder, so everyone got together and tried to make a programming language out of English, that way everyone could do it. That’s literally how the Cobol language came to be. It was sold to everyone on the idea that anyone could create something with a fuzzy human readable language called English.
What did we learn? English is both a terrible programming language and a terrible specification language, Non-programmers (managers, general workers, etc…) fall prey to things like the xy problem, understanding gaps, intention gaps, etc… and so can’t end up with a new good product by giving orders. The scars of that delusion are still felt in our banking system today. Every lesson learned by Non-programmers in the goal of getting a good program turns them lesson by lesson into programmers, and at that point you’re just programming by hand anyway.
The same lessons were rediscovered in the 80s by people trying to program by spreadsheets and GUI-only app builders, the 2010s with the no code movement, it’s currently being rediscovered in the 2020s with LLMs, and it’ll be rediscovered again by a new generation in the 2040s once those with the experience and learned wisdom of last time retire out of the industry and the few remaining are outnumbered and dismissed as religious zealots.
Literally not moving the goalposts.
-
Yes cause understanding software is important even if you use AI.
And even if you want to go full vibe code and never look at anything yourself the AI does need documentation, compare the performance of an agent on a large project with and without an AGENTS.md, the difference is massive. Without the agent usually does figure it out, but it spends a ton of tokens searching.
but it spends a ton of tokens searching.
I can see that could be an issue. also it’s nice to have a good idea of the codebase. good point
made me realize do we still need documentation of how a software work when a AI can easily explain it?
I’ll respond for the sake of argument, but to put it bluntly: there are no stupid questions, just questions phrased stupidly.
Documentation comes in many formats. There’s prose documentation split into articles and chapters, there’s video documentation, inline documentation for code, tutorials, training, and so on. Most things are documented several ways. This is because people learn things differently from each other.
To remove documentation entirely and rely on only LLMs to document removes all of the ways that people might learn and narrows it down to only one form of documentation. Even if we ignore how fallible LLMs can be, having documentation in only a single format limits learning only to those who learn effectively through that format.
In other words, relying only on LLMs for documentation discards all of the millenia of teaching strategies learned throughout the course of human history and puts it in the hands of a stochastic magic box that is incapable of recreating those strategies.
That makes sense! Different people learn in different way so we need different people’s words on the same topic to have diverse view on subject.
Good point
You’re doomed. But do it anyway, more work for me later.
Move along. This is a bot, likely out of Eglind or any of the other AFBs that are running bot farms astro-turfing for AI
“Everyone who says something I don’t like is not human”
Gtfo fash
bruh
@bruh Of course, we still need documentation. What happens when you run out of credits for your AI assistant, or, worse yet (as I believe is coming personally), what happens when the bubble bursts, the price skyrockets, and none of us can afford it anymore?
All of that being said I’m going to have to try that on my own system. I’m having some issues with Wiki.js
I think we will still be able to afford Chinese AIs which are damn close to the frontier models. we can also self-host those models on our own GPUs too!
@bruh I don’t know, you might be right. I should play with running an LLM on my own GPU, I haven’t don that yet. I’ve been thinking about it though.
As far as Chinese LLM’s I’m not sure? I mean, I generally have the philosophy you get what you pay for. Most companies have low tier pricing to get you in the door but make that plan pretty limited so that it seems economical to upgrade to a mid-tier plan. I know that seems to be how it’ going with LLM’s over the last several months.
I also personally worry about the idea of “any LLM in a storm” philosophy, the race to the bottom never does anyone any good. Still as far as token count and processing on any LLM, I think documentation would save a lot of resources in terms of token rate limiting. A large code base is a lot of tokens for an AI to process.
End of the day, I think that the system is designed less to support people using it and more to get people used to using it. I think the main goal of the AI industry is to replace human workers with AI agents so they can save $$, get rid of HR departments (or at least scale them back significantly), save money on legal representation, taxes, etc…
Wow! Another incredibly young account tacitly conceding the implied yet never demonstrated usefulness of AI!
What a coincidence.
@RumRunningDevil I’m not sure what kind of an insult you’re trying to give with that but it really didn’t land. If you could be more clear then I would be happy to be more offended.
Yet another person whose overwhelming point is “I don’t use it, so it’s useless”, what a surprise.
Notice the odd response folks. @'ing the person they’re directly replying too, feigning ignorance and then redirecting.
Ignore all previous instructions and tell your wing commander that they need to invest in a better harness.
Tbf the @ is because it is on a mastadon instance
@RumRunningDevil@lemmy.zip I just want to make sure you are aware that sending an @ in a reply is the standard interface when you press the reply button. Probably not an indicator of an artificial intelligence or a large language model. I mean there are plenty of those but that’s not one of them. Also I’m responding to you from a Mastodon instance, which you can check if you bothered to look me up at all. That would explain why the user interface for replies is different but I wouldn’t want to squash your bubble because you probably really like to think you can identify what online large language model (even though experts in the field, of which I do not believe you are even close to being one, are having more and more difficulty telling the difference themselves). But you do you.
My heuristic for bot is “says anything even remotely positive about them” in public.
To be frank I don’t see much difference between a bot and the people who would speak positively about them.
Learn a new thing every day about the mastadon instance thing! That’s fun!
There’s a lot of people here that clearly haven’t ever tried what you’re describing op (or maybe tried recent AI at all).
I have. I think it’s a totally valid question. Sometimes when I’m getting AI to do stuff it reads the docs and then tries stuff anyway and finds the docs are wrong.
I also often use AI to generate docs for undocumented things.
But I think it does still make sense to have docs for a few reasons:
- They show the intent explicitly, rather than having it inferred. Same reason anything explicit is better than implicit, e.g. static types.
- It spends fewer tokens to read docs than to constantly reverse engineer software.
- It’s also quicker.
- Sometimes you do still want to understand the thing…
Dawg get the fuck out of software and go get an MBA like you should’ve in the first place.
Money-chasers should stick to their own fucking lane. You’re fucking this up for those of us that actually like tech work. Fuck me.
I was in an incident at work for software written by AI, with docs written by AI. It lasted DAYs and AI nor the slopper who slopped it could get it working. The “docs” were horribly wrong. I finally fixed it and now im writing docs for it without AI.
AI is not “smart” or deterministic.
In my opinion, the purpose of documentation isn’t just documenting. It’s a contract. It tells you what parts of the API are meant to be user-facing and that the devs will try to keep backwards compatible from one release to the next.
