Unit tests have been around for a long time. They tended to be under-utilized in the past because they’re such a maintenance headache - changes / improvements in the production code that used to be a simple 5 minute thing can easily balloon to an hour or more of headache in necessary adjustments to perfectly good unit tests for “the old way” that happened to be impacted by the change.
Unit tests are a good way of keeping under-experienced programmers from accidentally breaking old things as they work on new things. They are VERY appropriate tools for LLM agent assisted development.
Boeing’s executives had motivated reasoning to skimp on actual engineering and give the results the exec wanted over the objections of actual engineers.
Sound familiar?
Frankly, no. LLMs are so sychophantic, there’s little evidence they would have flagged a warning that what he executives wanted was dangerous.
And no, LLMs are not just some tool. Never in human history has a hammer given it’s users psychosis.
that they would have articulated the reasoning of the systems engineers who would have been lobbying for better human factors, training, and visibility of the potential problems. If the systems engineers hadn’t been downsized that cycle, it’s a here today gone tomorrow discipline at many companies.
Frankly, I think the Star Wars trope of humans disrespecting droids was prophetic. We’re coming to a world where an LLM will spout off far more about a topic than you want to hear, and your attention just wanders off of them - rudely if they were an organic humanoid - without a further thought, until you want them to talk to the navicomputer or something else that they’re suited for.
The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
There are all kinds of aspects to software development. Under-training pilots and then having software lead them to crash a plane full of passengers counts as a software development fail in my book. Also a human factors fail, marketing fail, a management fail, a quality systems fail, probably a regulatory oversight fail… it’s up there with ignoring the guy telling the Space Shuttle mission control not to launch when it has been under 32F the night before, which wasn’t a software fail at all, but was a broad cultural failure. Boeing’s MAX-10 situation just happened to have software in the crux of the matter.
I do software development. When I see something, I say something. I’m frequently ignored. Several times over the years, I’ve had the people that ignored me come back asking for help fixing the issue, after they realized that it was more important than they thought it was when they initially ignored me. Luckily, none of the people who died in those situations died (or were harmed in any significant way) because of the problem, but people dying “near the problem” brought the seriousness of it to a level where it finally got addressed.
Unit tests have been around for a long time. They tended to be under-utilized in the past because they’re such a maintenance headache - changes / improvements in the production code that used to be a simple 5 minute thing can easily balloon to an hour or more of headache in necessary adjustments to perfectly good unit tests for “the old way” that happened to be impacted by the change.
Unit tests are a good way of keeping under-experienced programmers from accidentally breaking old things as they work on new things. They are VERY appropriate tools for LLM agent assisted development.
Absolutely not. In a world where hallucinating machines are writing code, humans should be writing the tests at a bare minimum.
AI writing code to pass tests written by AI will get people killed.
Boeing managed to kill plenty of people without a single bit of help from AI / LLMs in their MAX 10 software.
Used properly, LLMs could have articulated why their approach would lead to multiple crashes.
Used improperly, any tool is dangerous, from a chainsaw to a plastic spoon, the potential for injuries and deaths are unlimited.
Boeing’s executives had motivated reasoning to skimp on actual engineering and give the results the exec wanted over the objections of actual engineers.
Sound familiar?
Frankly, no. LLMs are so sychophantic, there’s little evidence they would have flagged a warning that what he executives wanted was dangerous.
And no, LLMs are not just some tool. Never in human history has a hammer given it’s users psychosis.
This is a completely nonsense argument.
that they would have articulated the reasoning of the systems engineers who would have been lobbying for better human factors, training, and visibility of the potential problems. If the systems engineers hadn’t been downsized that cycle, it’s a here today gone tomorrow discipline at many companies.
Frankly, I think the Star Wars trope of humans disrespecting droids was prophetic. We’re coming to a world where an LLM will spout off far more about a topic than you want to hear, and your attention just wanders off of them - rudely if they were an organic humanoid - without a further thought, until you want them to talk to the navicomputer or something else that they’re suited for.
Never tell me the odds.
The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
There are all kinds of aspects to software development. Under-training pilots and then having software lead them to crash a plane full of passengers counts as a software development fail in my book. Also a human factors fail, marketing fail, a management fail, a quality systems fail, probably a regulatory oversight fail… it’s up there with ignoring the guy telling the Space Shuttle mission control not to launch when it has been under 32F the night before, which wasn’t a software fail at all, but was a broad cultural failure. Boeing’s MAX-10 situation just happened to have software in the crux of the matter.
I do software development. When I see something, I say something. I’m frequently ignored. Several times over the years, I’ve had the people that ignored me come back asking for help fixing the issue, after they realized that it was more important than they thought it was when they initially ignored me. Luckily, none of the people who died in those situations died (or were harmed in any significant way) because of the problem, but people dying “near the problem” brought the seriousness of it to a level where it finally got addressed.