RAG retrieval augmented generation
- format data in chunks and encode as vectors; less token consumption of LLM
- metadata per chunk is used for offering citation of source material in answers
- regularly update your data
- evaluation: evaluating LLMs is very hard, but because you can now run them on your own data you have a ground truth
- in AzureML you can build a "fact checking" component. How?
Summary of AzureML RAG:
Azure Machine Learning allows you to incorporate RAG in your AI using the Azure AI Studio or using code with Azure Machine Learning pipelines. It offers several value additions like the ability to measure and enhance RAG workflows, test data generation, automatic prompt creation, and visualize prompt evaluation metrics. It enables the integration of RAG workflows into MLOps workflows using pipelines. You can also use your data with open source offerings like LangChain.
Steps
- Chunk data
- Wat is de beste chunking methode?
- Generate embeddings
- Store in vector index
- FAISS based vector index compatiable with LangChain
- Azure Cognitive Search
- Wat is de retrieval performance?
- RAG als je geen relevante context in de index hebt: dan gaat hij most similar (maar totaal irrelevante) context meegeven. Kan dit juist de kwaliteit verlagen?
- Metric to track: query density
- Standard ranking metrics
- User feedback
Use Cases
- Q & A Talk to your documents, e.g. PDF
- Summarization
- Conversational AI with more factually correct
- Fact checking component?
Vragen:
- Evaluatie met ground truth data (die hebben we zelf)
- Wij weten het juiste antwoord op de vraag: Wie was in 2017 de CEO van NS etc.
- Experimenten en testen opzetten. Wat kunnen we zeggen over de factual correctness van de LLM?
- Als we evaluatie hebben hebben we ook metrics over: welk model is het beste?
-
Data management: hoe zorg je dat informatie up to date blijft? Wat werkt goed?
-
Bijv. je wilt informatie over oude CAOs erin houden, maar als iemand naar de salarisschaal van nu vraagt wil je naar de recente CAO gestuurd worden. Of iig dat hij naar meerdere CAOs verwijst in de trant van: zie hier die van 2023, 2022, 2021
-
Is er bijv. informatie over de "freshness" van data.
-
Limitations of RAG in Azure OpenAI studio
From MS Learn page:
The system message, for example, is a useful reference for instructions for the model and is included with every call. While there's no token limit for the system message, when using your own data the system message gets truncated if it exceeds 200 tokens. The response from the model is also limited when using your own data is 1500 tokens. Due to these token limitations, it's recommended that you limit both the question length and the conversation history length in your call. Prompt engineering techniques such as breaking down the task and chain of thought prompting can help the model respond more effectively.