LLM Evals Course: Complete Course Recap
This is my recap of Hamel and Shreya's LLM evaluation course. I'm hoping I come back here in the future every time I need to remind myself of how to do this the right way.
Evaluation
LLM Evals Lesson 8: Improving LLM Products
Notes from the final lesson of Hamel and Shreya's LLM evaluation course - practical strategies for improving accuracy and reducing costs through prompt refinement, architecture changes, fine-tuning, and model cascades.
LLM Evals Course Lesson 7: Interfaces for Human Review
Notes from lesson 7 of Hamel and Shreya's LLM evaluation course - interface design principles and strategic sampling.
LLM Evals Course Lesson 6: Complex Pipelines and CI/CD
Notes from lesson 6 of Hamel and Shreya's LLM evaluation course - debugging agentic systems, handling complex data modalities, and implementing CI/CD for production LLM applications.
LLM Evals Course Lesson 5: How to Evaluate Complex Architectures
Notes from lesson 5 of Hamel and Shreya's LLM evaluation course - evaluating retrieval quality, generation quality, and common pitfalls in RAG systems.
LLM Evals Course Lesson 4: Multi-turn and Collaborative Evaluation
Notes from lesson 4 of Hamel and Shreya's LLM evaluation course - handling multi-turn conversations and building evaluation criteria through collaboration.