We have models that are specifically made to be good at these kinds of tasks. Why would you choose the ones that aren’t and then make generalizing claims about how AI sucks in this domain?
Yeah this is probably just straight up misinformation. By no means is a diagnosis going to be made by a generalist multimodal LLM. Diagnosis is a literally a binary classification (although that is an oversimplification) and on medical CV you are optimizing on that directly.
In a recent experiment, they set out to determine how reliable LMMs are in medical diagnosis — asking both general and more specific diagnostic questions — as well as whether models were even being evaluated correctly for medical purposes.
Curating a new dataset and asking state-of-the-art models questions about X-rays, MRIs and CT scans of human abdomens, brain, spine and chests, they discovered “alarming” drops in performance.
large language models (LLM) vs. large multi-modal models (LMM)
Regardless, they both use an LLM as the main driver. Multi modal just means that the LLM is interfaced with generative and/or predictive AIs for other types of content like images, sound, video, etc.
This is using a generalist tool for a specialized job. I’d expect the limit for LMMs is telling you if your picture is a heart or a kidney… Maybe. With low accuracy. Diagnosing? lol, hell no.
We have models that are specifically made to be good at these kinds of tasks. Why would you choose the ones that aren’t and then make generalizing claims about how AI sucks in this domain?
Yeah this is probably just straight up misinformation. By no means is a diagnosis going to be made by a generalist multimodal LLM. Diagnosis is a literally a binary classification (although that is an oversimplification) and on medical CV you are optimizing on that directly.
They did not use a LLM.
You’ve quoted them stating they used LLMs while claiming they did not use a LLM? What am I missing here?
“L” “M” “M”
Which in this context just means multimodal LLM, correct?
Correct.
large language models (LLM) vs. large multi-modal models (LMM)
Regardless, they both use an LLM as the main driver. Multi modal just means that the LLM is interfaced with generative and/or predictive AIs for other types of content like images, sound, video, etc.
This is using a generalist tool for a specialized job. I’d expect the limit for LMMs is telling you if your picture is a heart or a kidney… Maybe. With low accuracy. Diagnosing? lol, hell no.