LLM Evals Course Lesson 4: Multi-turn and Collaborative Evaluation
Notes from lesson 4 of Hamel and Shreya's LLM evaluation course - handling multi-turn conversations and building evaluation criteria through collaboration.
Artificial intelligence
View All
Building Domain-Specific Annotation Tools with FastHTML: Lessons from Isaac Flath
How Isaac Flath built a medical flashcard annotation tool for AnkiHub using FastHTML, and why custom annotation tools beat generic ones for complex domains.
LLM Evals Course Lesson 3: Building Automated Evaluators
Notes from lesson 3 of Hamel and Shreya's LLM evaluation course - implementing automated evaluators, building reliable LLM-as-judge systems, and avoiding common pitfalls.
Recent Post
Error Analysis for Improving LLM Applications
A systematic approach to analysing and improving large language model applications through error analysis.
Why we need Experiment-based Roadmaps in the AI Product Era
Why evaluation-driven experimentation creates better roadmaps in AI products.
The M×N Problem in Software Architecture
Understanding the combinatorial complexity problem that plagues many software systems, and how modern architectures solve it.
Jackie Bavaro on Strategy
Bavaro's approach to strategy: Vision, Strategic Framework, and Roadmap.
Three Ways I Think Frameworks are Good (actually)
Are frameworks actually useful? Exploring how they enable communication, engagement, and focused thinking
The challenges of mastering LLMs, and their role as cyborg enhancement
Simon Willison was a guest on Logan Kilpatrick's Google podcast. Topics covered: AI as a 'cyborg enhancement', the non-intuitive challenges of mastering LLM use, and the legitimate need for uncensored language models in fields like journalism.