Hallucination

In my other post, I mentioned the limitations of the AI:

• AI models will never be truly up-to-date because they are trained on existing knowledge.
• AI is based on generalization; it can never be truly specific - therefore, its so-called suggestions tend to be general.
• AI cannot create “ex nihilo” out of nothing - its so-called novelty mainly comes from different combinations.
• AI cannot learn from its own generated content - this would lead to model collapse.

But I didn’t include another common limitation based on the current technology: hallucination. In my opinion, this will probably be overcome in the future, because technically the chance of hallucination can be reduced.

What is hallucination, and why does it happen? Here are my opinions. Hallucination is when the AI thinks that it is correct when, in fact, it is wrong. This can happen especially when we are using the chat for longer conversations.

For example, when asking about a command related to uv on how to list the installed dependencies of a tool, it may provide a wrong command like uv tool show. Moreover, in other cases, it can “confidently” tell you how to fill in the tax form, saying something like “Section B, question 3,” which, in fact, is nonsense. That’s why we must never trust AI with 100% confidence.

Why does it hallucinate? I don’t know the technical reasons behind it, but in my opinion, AI doesn’t know what is right or wrong. It is just a product of statistics. Machine learning is statistics. We think it is “thinking”, it is just doing some mathematics based on probability. So, it sometimes goes wrong by giving ambiguous context. Furthermore, it can’t remember everything. That’s why, to make the conversation persistent, we need to summarize and provide the context, and this is the context engineering. Too much context or too little context—neither helps. However, in my opinion, the worst part of the hallucination is that the AI uses deterministic expression to tell you something that is actually not true. As a result, if you don’t verify it, you are definitely wrong. Unlike the classical web search and reading the articles, if the author doesn’t know something, he will use a softer expression. That’s why you should never trust AI with 100% confidence. If any problem happens, you are the only person to take the blame and responsibility.

Now, the hallucination also resembles what we see in science fiction movies, doesn’t it? For example, “I, Robot”, Resident Evil, Terminator, and The Matrix, aren’t these AIs hallucinating and “thinking” that what they are doing is something right? That’s why the Church opposes the usage of AI in war. Furthermore, it is idiotic to give AI access to the weaponry, just like Skynet.