- howrar ( @howrar@lemmy.ca ) 41•20 days ago
We have models that are specifically made to be good at these kinds of tasks. Why would you choose the ones that aren’t and then make generalizing claims about how AI sucks in this domain?
- spaduf ( @spaduf@slrpnk.net ) 9•20 days ago
Yeah this is probably just straight up misinformation. By no means is a diagnosis going to be made by a generalist multimodal LLM. Diagnosis is a literally a binary classification (although that is an oversimplification) and on medical CV you are optimizing on that directly.
- snooggums ( @snooggums@midwest.social ) English6•20 days ago
They did not use a LLM.
In a recent experiment, they set out to determine how reliable LMMs are in medical diagnosis — asking both general and more specific diagnostic questions — as well as whether models were even being evaluated correctly for medical purposes.
Curating a new dataset and asking state-of-the-art models questions about X-rays, MRIs and CT scans of human abdomens, brain, spine and chests, they discovered “alarming” drops in performance.
- Thorry84 ( @Thorry84@feddit.nl ) 12•20 days ago
You’ve quoted them stating they used LLMs while claiming they did not use a LLM? What am I missing here?
- everett ( @everett@lemmy.ml ) 5•20 days ago
What am I missing here?
“L” “M” “M”
- spaduf ( @spaduf@slrpnk.net ) 5•20 days ago
Which in this context just means multimodal LLM, correct?
- blindsight ( @blindsight@beehaw.org ) 6•20 days ago
Correct.
large language models (LLM) vs. large multi-modal models (LMM)
Regardless, they both use an LLM as the main driver. Multi modal just means that the LLM is interfaced with generative and/or predictive AIs for other types of content like images, sound, video, etc.
This is using a generalist tool for a specialized job. I’d expect the limit for LMMs is telling you if your picture is a heart or a kidney… Maybe. With low accuracy. Diagnosing? lol, hell no.
- ResoluteCatnap ( @ResoluteCatnap@lemmy.ml ) English26•20 days ago
As others have said, you don’t need (and shouldn’t use) a LLM for a classification task like this. There are machine learning models that can handle this and identify underlying patterns that humans can not easily detect. And yes, they can get accuracy and precision scores much higher than 50%
What an incredibly stupid article.
- Umbrias ( @Umbrias@beehaw.org ) 11•20 days ago
Correct, you shouldn’t use llm for this task.
Which is literally the point of the paper, because various techbros have been trying to claim that they are good at these tasks.
- Thorry84 ( @Thorry84@feddit.nl ) 18•20 days ago
This is pretty dumb, machine learning algorithms (fuck off with calling it AI) are especially good at seeing signs of disease in data such as xrays, CT and MRI scans. It’s the one place they really help save time and prevent mistakes. And even if it’s just to flag shit for a second opinion by a doctor and not to replace the doctor, that’s still super useful. Pattern recognition is hard and these kinds of algorithms are very good at them if provided the right source data to work off.
If only the media and big corps would stop claiming LLMs are general AI, then maybe people would stop using them for stuff it’s clearly not good at and not meant for.
- jsomae ( @jsomae@lemmy.ml ) 10•20 days ago
This isn’t dumb. This is a very good study as it is helping to remind people that these fancy new tools aren’t good at everything. The media reporting on this is doing a service.
Edit: my bad making two responses
- spaduf ( @spaduf@slrpnk.net ) 5•20 days ago
By casting doubt on a related but fundamentally different bit of medical tech? Yeah that’s what we need: more folks questioning medicine based on pop science understandings of the technology.
- Umbrias ( @Umbrias@beehaw.org ) 1•20 days ago
A study debunking the usage of llm in medicine has almost no impact on general machine learning applications in medicine. This is textbook concern trolling.
- jsomae ( @jsomae@lemmy.ml ) 1•20 days ago
Good point
- jsomae ( @jsomae@lemmy.ml ) 4•20 days ago
Can’t stop people calling it AI. People have called video game bots AI since the 90s, even in industry. Any algorithm is a form of artificial intelligence, really. LLMs and machine vision are multipurpose, though I agree that general-purpose is still a stretch.
- 0ops ( @0ops@lemm.ee ) 4•20 days ago
Seriously, the field of artificial intelligence has been around since the beginning of computer science, since Alan Turing founded it after coming up with the modern computer. Frankly, if you ask me, anyone complaining about LLMs being referred to as AI has been watching too many movies. AI != Human-but-metal and it never has. Going by the Wikipedia article, to be considered AI, a machine just has to perceive it’s environment and learn - degree notwithstanding.
Of course this definition is pretty vague, so in practice AI tends to refer to the cutting edge of flexible computer algorithms. Many now-mundane algorithms much simpler than today’s LLMs (like A* and genetic algorithms) were once considered AI for their flexible logic. At some point the Internet decided that it doesn’t count unless it’s literally Jarvis, but that’s a very stingy definition of a very broad field.
- xep ( @xep@fedia.io ) 3•20 days ago
Why wouldn’t agents in video games be AI, though? Things like are pathfinding, search, and behaviour trees are commonly used for agents in games, and in computer science these are widely considered to be artificial intelligence techniques. It’s unlikely that you would find a CS textbook calling the Fast Fourier Transform AI though, or things like Bresenham’s Line Drawing algorithm.
- jsomae ( @jsomae@lemmy.ml ) 2•19 days ago
Absolutely. I wouldn’t call Bresenham AI. In some contexts, like games, I might call A* search AI. But to someone from the Victorian era who paid people to compute taylor series by hand, something basic and flexible like a microprocessor which can run bresenham or FFT or etc. etc. … might have been seen as artificial intelligence. Using a machine to solve a problem that normally requires human brainpower.
- Match!! ( @match@pawb.social ) English13•20 days ago
Coincidentally, I trained a CNN to tell dogs from cats and it does a godawful job diagnosing cancer