Abstract:

Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucina- tion is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hal- lucinate. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

  • That’s not how LLMs work.

    Super short version is that LLMs probabilistically determine the next word most likely to occur in a sequence. They do this using Statistical Models (like what your cell phone’s auto complete uses); Transformers (rating the importance of preceding words, so the model can “focus” on the most important words); and Relatedness (a measure of how closely linked different words/phrases are to reach other in meaning).

    With increasingly large models, LLMs can build a more accurate representation of Relatedness across a wider range of topics. With enough examples, LLMs can infinitely generate content that is closely Related to a query.

    So a small LLM can make sentences that follow writing conventions but are nonsense. A larger LLM can write intelligibly about topics that are frequently included in the training materials. Huge LLMs can do increasingly nuanced things like “explain” jokes.

    LLMs are not capable of evaluating truth or facts. It’s not part of the algorithm. And it doesn’t matter how big they get. At best, with enough examples to build a stronger Relatedness dataset, they are more likely to “stay on topic” and return results that are actually similar to what is being asked.

    • No, I’ve used LLMs to do exactly this, and it works. You prompt it with a statement and ask “is this true, yes or no?” It will reply with a yes or no, and it’s almost always correct. Do this verification through multiple different LLMs and it would eliminate close to 100% of hallucinations.

      EDIT

      I just tested it multiple times in chatgpt4, and it got every true/false answer correct.

      • This can only work in contexts where the LLM already has a strong Relatedness database for the topic. LLMs are incapable of assessing the accuracy of any information they weren’t trained on with sufficient examples to build that database.

        This paper, if I’m understanding it correctly, is saying the same thing: LLMs will always hallucinate and are incapable of identifying hallucinations in certain contexts.

        The problem we face is that it’s hard to identify hallucinations in the exact context when LLMs are most likely to make them: in content that’s not widely known and understood with many examples in the training data.

        Or maybe I’m off base and I need to read the full study.

        • That’s not a problem at all, I already use prompts that allow the LLM to say they don’t know an answer, and it does take that option when it’s unable to find a correct answer. For instance I often phrase questions like this “Is it known whether or not red is a color in the rainbow?” And for questions where it doesn’t know the answer it now will tell you it doesn’t know.

          And to your other point, the systems may not be capable of discerning their own hallucinations, but a totally separate LLM will be able to do so pretty easily.

        • Well, LLMs can and do provide feedback about confidence intervals in colloquial terms. I would think one thing we could do is have some idea of how good the training data is in a given situation - LLMs already seem to know they aren’t up to date and only know stuff to a certain date. I don’t see why this could not be expanded so they’d say something much like many humans would - i.e. I think bla bla but I only know very little about this topic. Or I haven’t actually heard about this topic, my hunch would be bla bla.

          Presumably like it was said, other models with different data might have a stronger sense of certainty if their data covers the topic better, and the multi cycle would be useful there.

          • The problem isn’t just that llms can’t say “I don’t know”, it’s also that they don’t know if they know something or not. Confidence intervals can help prevent some low-hanging fruit hallucinations but you can’t eliminate hallucinations entirely since they will also hallucinate about how correct they are about a given topic.