
Test your own product's failures, not leaderboard scores.
Steal this
- Collect your product's real failures and test against those.
The lesson and the takeaways are our summary. Everything below is theirs, with where they said it.
How they work
Factual- Their skills walk a coding agent through building evals for your product. (Hamel Husain and Shreya Shankar)
Where they talk and write
Interviews 53
- Look at Your Data Before You Write a Single Eval | Hamel Husain, Evals @ OpenAITwoSetAI: Decoding AI Builders ·
- Can data agents do real analysis?The Analytics Engineering Roundup (dbt Labs) ·
- The Rise of the AI ScientistVanishing Gradients ·
- Data Agents with Shreya Shankar - Weaviate Podcast #135!Weaviate Podcast ·
- How to Do AI Evals Step-by-Step with Real Production Data | Tutorial by Hamel Husain and Shreya ShankarThe Growth Podcast ·
- Bridge Atlas Episode 5: VerifiabilityProtocolized ·
47 older
- Evals, error analysis, and better prompts: A systematic approach to improving your AI productsHow I AI ·
- Debugging AI Products: From Data Leakage to Evals with Hamel HusainJust Now Possible (Product Talk) ·
- Episode 60: 10 Things I Hate About AI Evals with Hamel HusainVanishing Gradients ·
- AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel HusainBehind the Craft (Peter Yang) ·
- Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)Lenny's Podcast ·
- Ep. 29: Shreya Shankar, Berkeley EECS PhD Student and DocETL CreatorDelta Podcast ·
- Shreya Shankar on AI for Corporate Data ProcessingGenerative AI in the Real World ·
- Getting Beyond the Demo with Hamel HusainGenerative AI in the Real World ·
- Episode 57: AI Agents and LLM Judges at Scale: Processing Millions of Documents (Without Breaking the Bank)Vanishing Gradients ·
- Mastering AI Evals: The missing skill in AI product management | Hamel HusainProduct Founder with Amir Rezaei ·
- Mindset Over Metrics: How to Approach AI Engineering | Hamel HusainChain of Thought | AI Agents, Infrastructure & Engineering ·
- The PM’s Role in AI Evals: Step-by-StepThe Growth Podcast ·
- Unlocking Unstructured Data with LLMsThe Data Exchange with Ben Lorica ·
- Episode 50: A Field Guide to Rapidly Improving AI Products -- With Hamel HusainVanishing Gradients ·
- Beyond the Demo: Building AI Systems That Actually WorkThe Data Exchange with Ben Lorica ·
- 📆 ThursdAI - May 1- Qwen 3, Phi-4, OpenAI glazegate, RIP GPT4, LlamaCon, LMArena in hot water & more AI newsThursdAI - The top AI news from the past week ·
- Ep. #79, AI and Otel: Look at your Data with Hamel HusainO11ycast ·
- [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya ShankarLatent Space ·
- Episode 45: Your AI application is broken. Here’s what to do about it.Vanishing Gradients ·
- The Future of Data Engineering and the Evolving Landscape of Enterprise AI AdoptionThe Founder to Fortune Podcast ·
- Two-Shot: {Shreya Shankar}{PhD Student, UC Berkeley EECS}Two-Shot: Artificial Intelligence | Technology | DevTools | Research + (light humor) ·
- AI Agents for Data Analysis with Shreya Shankar - 703The TWIML AI Podcast ·
- Why Your AI Product Needs Evals with Hamel HusainHigh Agency (Humanloop) ·
- Episode 33: What We Learned Teaching LLMs to 1,000s of Data ScientistsVanishing Gradients ·
- Episode 32: Building Reliable and Robust ML/AI PipelinesVanishing Gradients ·
- Building Real-World LLM Products with Fine-Tuning and MoreThe TWIML AI Podcast ·
- Episode 29: Lessons from a Year of Building with LLMs (Part 1)Vanishing Gradients ·
- Episode 30: Lessons from a Year of Building with LLMs (Part 2)Vanishing Gradients ·
- #12 - Shreya Shankar: HCI for ML engineers, Stanford, and the role of ChatGPT in educationThinking About Computers ·
- Episode 22: LLMs, OpenAI, and the Existential Crisis for Machine Learning EngineeringVanishing Gradients ·
- Episode 21: Deploying LLMs in Production: Lessons LearnedVanishing Gradients ·
- Shreya Shankar: Machine Learning in the Real WorldThe Gradient: Perspectives on AI ·
- 🤖 GPTs & Large Language Models in production with Hamel HusainThe MLOps Podcast ·
- 🏃♀️Moving Fast and Breaking Data with Shreya ShankarThe MLOps Podcast ·
- Grounded Research: From Google Brain to MLOps to LLMOps — with Shreya Shankar of UC BerkeleyLatent Space ·
- Shreya Shankar — Operationalizing Machine LearningGradient Dissent: Conversations on AI ·
- 9: AutoML, Literate Programming, and Data Tooling Cargo CultsVanishing Gradients ·
- Towards Observability for ML Pipelines // Shreya Shankar // MLOps Coffee Sessions #75Agentic Conversations (formally mlops.community) ·
- Exploring the FastAI Tooling Ecosystem with Hamel HusainThe TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) ·
- Deploying Machine Learning Models Safely and SystematicallyThe Data Exchange with Ben Lorica ·
- Shreya Shankar, ML Engineer @Viaduct on Applied ML research & moreJay Shah Podcast ·
- A day in the life on an Applied Machine Learning Researcher - Shreya Shankar, Stanford UniversityJay Shah Podcast ·
- Shreya Shankar: Lessons learned after a year of putting ML into productionMachine Learning Engineered ·
- Machine Learning OpsRasa Chats ·
- Hamel Husain — Building Machine Learning ToolsGradient Dissent: Conversations on AI ·
- Hamel Husain | Fastpages, Open Source | ML at Github | fastaiChai Time Data Science ·
- Semantic search at GithubData Skeptic ·
Writing 47
- How To Use Decision Models for Evalshamel.dev ·
- Evals for AI EngineersBook · O'Reilly Media ·
- Claude’s new auto eval toolhamel.dev ·
- Advanced evals: How to find (and fix) hidden AI failures in your productLenny's Newsletter ·
- AI Evals: Everything You Need to Knowhamel.dev ·
- AI Product Engineering Noteshamel.dev ·
41 older
- Do Automated Evals Work?Parlance Labs ·
- “It’s Hard to Eval” Is a Product Smellhamel.dev ·
- Exploring Agent-Assisted Qualitative Analysissh-reya.com ·
- The Revenge of the Data Scientisthamel.dev ·
- Can AI Agents Answer Your Data Questions? A Benchmark for Data AgentsarXiv ·
- Evals Skills for Coding Agentshamel.dev ·
- Why I Stopped Using nbdevhamel.dev ·
- Task Cascades for Efficient Unstructured Data ProcessingarXiv ·
- Multi-Objective Agentic Rewrites for Unstructured Data ProcessingarXiv ·
- A pragmatic guide to LLM evals for devsThe Pragmatic Engineer ·
- On the Consumption of AI-Generated Content at Scalesh-reya.com ·
- Selecting The Right AI Evals Toolhamel.dev ·
- Building eval systems that improve your AI productLenny's Newsletter ·
- In Defense of AI Evals, for Everyonesh-reya.com ·
- Stop Saying RAG Is Deadhamel.dev ·
- Inspect AI, An OSS Python Library For LLM Evalshamel.dev ·
- Writing in the Age of LLMssh-reya.com ·
- Steering Semantic Data Processing With DocWranglerarXiv ·
- PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model PipelinesarXiv ·
- RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation PipelinesarXiv ·
- A Field Guide to Rapidly Improving AI Productshamel.dev ·
- Thoughts On A Month With DevinAnswer.AI ·
- Using LLM-as-a-Judge For Evaluation: A Complete Guidehamel.dev ·
- DocETL: Agentic Query Rewriting and Evaluation for Complex Document ProcessingarXiv ·
- An Open Course on LLMs, Led by Practitionershamel.dev ·
- Data Flywheels for LLM Applicationssh-reya.com ·
- Short Musings on AI Engineering and "Failed AI Projects"sh-reya.com ·
- What We’ve Learned From A Year of Building with LLMsApplied LLMs ·
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesarXiv ·
- Debugging AI With Adversarial Validationhamel.dev ·
- Comparing LLMs on "Real-World" Retrievalsh-reya.com ·
- Your AI Product Needs Evalshamel.dev ·
- Is Fine-Tuning Still Valuable?hamel.dev ·
- Fuck You, Show Me The Prompt.hamel.dev ·
- How To Debug Axolotlhamel.dev ·
- SPADE: Synthesizing Data Quality Assertions for Large Language Model PipelinesarXiv ·
- Tokenization Gotchashamel.dev ·
- Tools for curating LLM Datahamel.dev ·
- vLLM & Large Modelshamel.dev ·
- Optimizing LLM latencyhamel.dev ·
- Operationalizing Machine Learning: An Interview StudyarXiv ·
Each link goes to the original page, newest first.