RAG Evaluation and Trustworthy Agents
Ragas, LLM-as-a-judge, synthetic tests, guardrails, observability, retrieval diagnostics, agent eval, full audit
What you'll learn
- ✓Explain why 'accuracy' is not one number and split retrieval failures from generation failures
- ✓Run Ragas metrics and read a per-sample table to diagnose a RAG pipeline
- ✓Design an LLM-as-a-judge rubric with G-Eval and recognise judge bias
- ✓Generate and validate synthetic evaluation data
- ✓Implement input/output guardrails and test them against false positives and holes
- ✓Diagnose retrieval with hit@k, MRR and per-failure-class analysis; evaluate agent trajectories
- ✓Audit a RAG application end to end and write the report
About this course
Wiring an LLM into a product takes an afternoon. Knowing whether it is right takes a discipline — and this course teaches that discipline end to end.
We start with why evaluation is business-critical and the diagnostic questions every RAG owner should be able to answer. Then the tools: quantitative RAG evaluation with Ragas (context precision, faithfulness, answer relevancy) and how to read the per-sample table; LLM-as-a-judge with G-Eval, rubric design and judge bias; synthetic test data for the cold-start problem and how to validate it; input and output guardrails and how to know a guardrail is lying; observability and feedback loops with LangSmith. Two sessions go deeper than the usual cookbook: retrieval diagnostics (query-planning failures, temporal retrieval, multi-hop with and without planning, hit@k and MRR per failure class) and agent evaluation (tool-call correctness, trajectory scoring, judge rubrics, cost and latency budgets). The capstone audits a small customer-support RAG app with three planted defects: build the evaluation set, run the metric battery, find and fix each defect, and write the audit report.
Every session is a notebook shown on screen with the narration explaining what each metric can and cannot tell you, what not to automate, and how to tell when a score is not trustworthy.
Curriculum9 sections · 18 lectures · 2h 45m
Section 0. Welcome
Free previewSection 1. Why Evaluation Is Business-Critical
Free previewSection 2. RAG Evaluation with Ragas
Free preview- ▶dataset and ragas metrics9:24
- 🔒scores per sample diagnosis10:24
Section 3. LLM-as-a-Judge and G-Eval
- 🔒rubric and judge prompt8:54
- 🔒compare batch judge bias7:54
Section 4. Synthetic Test Data
- 🔒cold start and generation8:06
- 🔒validation and ragas format7:48
Section 5. Guardrails
- 🔒input guardrails7:42
- 🔒output guardrails and pipeline9:36
Section 6. Observability
- 🔒langsmith tracing7:48
- 🔒feedback loop and capstone7:48
Section 7. Retrieval Diagnostics and Agent Evaluation
- 🔒traps metrics per class table10:06
- 🔒planning time multihop14:06
- 🔒agent trajectories judges budgets12:42
Section 8. Capstone RAG Audit
- 🔒audit brief and checklist10:24
- 🔒eval set battery and guardrail audit11:36
- 🔒fix rerun report course finale9:30
Requirements
- · A computer with Python 3.10+ (a free Google Colab account is enough for most sessions)
- · Comfort reading Python code; you do not need to be an expert
- · An OpenAI API key (a LangSmith key for the observability session)
- · Experience building at least one RAG or LLM application
Who this is for
- · Engineers who have shipped a RAG or agent and are asked 'how do you know it works?'
- · Tech leads setting up evaluation and monitoring for LLM features
- · QA and data teams responsible for LLM quality
- · Anyone who wants trustworthy AI features rather than impressive demos
Read alongside the course
The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work.