Honest GenAI: Generative AI, hallucination, and facts
The generative capacities from Gen AI are a double edged sword. At its best, language models can synthesize an answer that draws information from many sources. But the flip side is that parts of the presented information may be hallucinated.
(One may call this a farmakon, the Greek word that means both medicine and poison, to indicate that the core capability of GenAI is at once its potential danger. It's a matter of the right amount of hallucination.)
Let's explore this concept of hallucination in different modalities.
Berglund2023-The reversal curse LLM trained on A is B fails to learn B is A
Code
When producing code, my experience is that ChatGPT often produces very good looking code, but sometimes it calls non-existing functions (that sound logical) or combines a method from one library (e.g. geopandas) on another (e.g. pandas).
TODO code example.
Images
For GenAI producing images, this "hallucinatory" quality is exactly what we like about it! In images we typically care less about "facts" but instead talk about being "true" to reality. But in art being "true" does not equal to "as realistic as possible". An more abstract or for example impressionistic painting can catch someone's glance or character very well.
One particular example where I cared about the image being generated being "true" is when I was trying to generate a stylized self portrait based on a prompt image of myself. In this case I wanted the model to "hallucinate" just enough to stylize the portrait while still remaining "true" to the image. This almost without exception was a one-way ticket to the uncanny valley.
Text: synthesizing answers that combine information from various sources
The use of LLM-based GenAI such as ChatGPT is increasingly a replacement for the Google search. Instead of looking through the top N hits, ChatGPT can draw on relevant information on the internet (including the forums I would visit myself) and give you a brief answer. But in a information retrieval context the factual correctness of information (and finding the source) are important.
Language GenAI: the model can only retrieve facts insofar as they were known at a particular "snapshot" in time. E.g. ChatGPT was training on data until 2021.
Particularly when asking about facts and events after that snapshot, the model will start to produce grammatically coherent but incorrect explanations. This is typically called "hallucination".
TODO text example.
Countering unwanted hallucinations
In this case of the self-portrait, I had two strategies to counter too much hallucination: 1) clever prompt engineering and 2) finetuning on my own portrait.
Prompt engineering is the cheapest and quickest option you have available. Finetuning may be feasible approach for small applications, big budgets, and only if the particular model can be fine-tuned (not all LLMs can).
But when it comes to producing correct facts, prompt engineering is not sufficient, because it does not change the fact that the necessary context to answer a question may not be present in the outdated training.
Also finetuning is not a feasible approach when it comes to large language models such as ChatGPT because of the immense cost (ChatGPT was estimated to cost at least 100 million dollars to train [Sam Altman, source needed]). Moreover, to systematically counter the problem of missing factual context in the training data, training would have to be frequent.
If context is missing to answer particular questions correctly, we need to somehow make the required domain knowledge available without retraining the model. So what's a more scalable approach?
RAG:
The Retrieval Augmented Generation (RAG) paradigm is a natural starting point for working this out further.
It adds context with domain knowledge without requiring fine tuning.
To read: https://www.pinecone.io/learn/retrieval-augmented-generation/
RAG retrieval augmented generation
Uncertainty and correcting mistakes
I don't make coding mistakes because I'm just a computer program - ChatGPT
Another interesting thing to discuss is: what to do if nevertheless information is presented that was incorrect?
Experience: tell ChatGPT they were wrong, you'll get a response like "Of course I was wrong" followed by an answer that is presented with equal conviction but is probably also equally wrong. There was no uncertainty in the original wrong answer, and neither in the second!
TODO give example.
What is still lacking: how (un)certain is generative AI about the information that is presented?
TODO TBC
Asking ChatGPT what mistakes it commonly makes
Prompt: What are some common mistakes ChatGPT makes?
ChatGPT, like any language model, can make several common mistakes, which can include:
- Incorrect Information: ChatGPT may generate responses that are factually incorrect or outdated because it doesn't have access to real-time data. Its knowledge is limited to what it was trained on, which only goes up until September 2021.
- Ambiguity: It might misinterpret ambiguous queries and provide answers that don't address the user's intended meaning or context.
- Repetition: ChatGPT can sometimes repeat certain phrases or ideas within a response, making the conversation feel less natural.
- Overuse of certain phrases: It may rely too heavily on certain phrases or expressions, making its responses repetitive.
- Long-Winded Responses: ChatGPT sometimes provides overly verbose or lengthy responses when a more concise answer would suffice.
- Confabulation: It can generate responses that sound coherent but are actually fabrications or combinations of information, especially when asked about hypothetical scenarios.
- Sensitivity to Input Wording: The model can be sensitive to how a question is phrased. Slight rephrasing of the same question may yield different responses.
- Inappropriate or Offensive Content: Despite efforts to filter out inappropriate content, ChatGPT may occasionally produce offensive, biased, or harmful responses.
- Lack of Common Sense: It may provide responses that lack common-sense reasoning or provide unrealistic solutions to problems.
- Inability to Ask for Clarification: Instead of asking for clarification when faced with an ambiguous query, ChatGPT often guesses the user's intent, potentially leading to misunderstandings.
- Not Recognizing Limits: ChatGPT may generate responses that imply a level of understanding or knowledge that it doesn't actually possess, which can be misleading.
- Verbose or Unnecessary Phrasing: It might use overly formal language or unnecessary words, making responses less natural.
- Difficulty with Specific Topics: For some specialized or technical topics, ChatGPT may struggle to provide accurate or in-depth information.
- Time References: It may not always understand or respond appropriately to time-related queries, as it lacks real-time awareness.
- Limited Worldview: ChatGPT's responses can reflect the biases present in the training data, potentially reinforcing stereotypes or biases unintentionally.
It's important for users to be aware of these limitations and critically evaluate the information provided by ChatGPT. While it can be a valuable tool, especially for general information and creative writing, users should exercise caution and verify information from reliable sources when necessary.