There is huge excitement about ChatGPT and other large generative language models that produce fluent and human-like texts in English and other human languages. But these models have one big drawback, which is that their texts can be factually incorrect (hallucination) and also leave out key information (omission).
In our chapter for The Oxford Handbook of Lying, we look at hallucinations, omissions, and other aspects of “lying” in computer-generated texts. We conclude that these problems are probably inevitable.
Could the same not describe the human brain?
It knows the probabilities that one word or sentence will follow another, based on the data it was trained on. That’s it. It’s like typing exclusively through a smartphone keyboard’s predictive text, just much more complex and trained on a much larger dataset.
It has no way of connecting any word or sentence to what that word or sentence means or represents in the real world, and it’s absolute pants at making up anything new, which is why AI generated stories are cliches from top to bottom.
Like, it knows what words and phrases are associated with the word cat, or with a description prompt like “a cute story about your cat”, so it can imitate that text, or imitate it and mix it with imitations of other text patterns (" give me a cute story about your cat riding a dinosaur in a sarcastic tone"), but it’s just a mirror that reflects back what humans put into it.
If it must be compared to a human brain, it’s more like exclusively broca’s area of the human brain - the part that handled language. If broca’s area is damaged in a head injury, a human can end up with aphasia, unable to speak coherently but maybe still able to understand others perfectly well , or able to speak but unable to understand language from others, or both, but in all other ways the person can be completely fine and still able to live and solve puzzles and interact with other humans normally. (Granted, I am not a neurologist, and I’d imagine head injuries like these can probably get much more complicated.)
Anyway, hypothetically, if you were to take broca’s area out of a human brain and keep it alive in a jar on its own, without the rest of the brain: I don’t think it could be described as sentient or thinking on its own. Although the human brain is so complex and interconnected and poorly understood (beyond what area is generally associated with what function) and malleable/plastic based on experience (e.g. the brains of blind people re-purpose areas normally used for sight to he used for other purposes instead, absent visual input) that I could be wrong / it could be arguable.
The experiences of people who have had the two halves of their brain separated (this used to be a treatment for seizures, apparently), who then seem to have the two halves thinking and operating independently, or of people who manage to live largely normal lives despite missing half or more of their brain entirely, would suggest that maybe even a tiny piece of a brain could be a thinking being, I suppose.
There was an experiment a while back… MIT, maybe? Like a decade or more ago. Where they made little robots that had a kind of scaffold/medium in which they planted and grew neurons sourced from mouse brains. The robot brains, iirc, changed in reaction to the input the robots were given/their environment and sensors, and they showed personality differences in the way they behaved.
LLM’s don’t do anything like that, though, so far as I know. They don’t restructure themselves in response to input like an organic brain can. They’re a really complicated pile of if-thens, and while the inputs and therefore outputs change, the mechanism that turns inputs into outputs never does.
This comment is like 90% tangent at this point. Oh well.
Anyway, there’s a really good classic metaphor for this that explains it much better, about a man in a locked room who receives mail in a language he can’t speak through a slot in the wall, but he is still able to sort the mail in complex ways even though he can’t understand the content of any of it. I can’t remember the whole thing or the name of it off the top of my head though. I’ll try to remember.
Thank you so much for your thoughtful response. I’m sorry for not seeing it for so long! If you can believe it, I just discovered the “inbox” in my lemmy app and am going through all the things people said to me over the past month.
This whole topic is really interesting to me. I hear what you’re saying and imagine the distinctions you’re drawing between these models and real brains are significant. I can’t help but wonder, though, if we, as humans, might be poorly equipped to recognize the characteristics of emerging intelligence in the systems we create.
I am reminded vaguely of the Michael Crichton book Andromeda Strain (it has been many years since I read it, granted) wherein an alien lifeforms based on silicon, rather than carbon, was the major plot object. It is interesting to think that something like an alien intelligence might emerge in our own networked systems without our noticing. We are waiting for our programs to wake up and pass the Turing test. Perhaps, when they wake up, no one will even see because we are measuring the wrong set of things…
My issue is using humanizing language which adds a lot of baggage and implied assumptions which are miss leading. Bottom line is that these things just give crazy results at times through design and the data you put it and even at best they are rather dumb. Maybe useful but the user has to know their limitations and verify their results.