AI Evaluations Crash Course in 50 Minutes (2025) | Hamel Husain

AI Evaluations Crash Course in 50 Minutes (2025) | Hamel Husain

Author: Peter Yang September 28, 2025 Duration: 52:29

Today, I want to share a new episode with Hamel Husain.


Hamel has trained 2,000+ PMs and engineers from companies like OpenAI, Anthropic, and Google on how to run AI evals. In my new episode, he shares a free master class on how to build evals for a real AI agent in just 50 minutes using a simple spreadsheet. I learned a lot from Hamel and I think you will too.


Hamel and I talked about:

(00:00) What the most valuable part of evals is

(01:25) Live walkthrough: Analyzing 100 real production traces

(09:50) Creating the eval criteria using a simple spreadsheet

(24:44) Why binary pass/fail ratings beat 1-5 scores every time

(28:52) The agreement metric trap that fools most PMs

(30:08) True positive and negative rates explained

(36:00) How to set up continuous evals in production


Get the takeaways: https://creatoreconomy.so/p/ai-evaluations-crash-course-in-50-minutes-hamel-husain


Where to find Hamel:

X: https://x.com/HamelHusain

Website: https://hamel.dev/


📌 Subscribe to this channel – more interviews coming soon!


For anyone building the future, Behind the Craft is a conversation with Peter Yang that moves beyond theory and into the tangible details of creation. This podcast lives in the messy, rewarding space where ideas become real products. Each episode is built on candid interviews with experts who have been in the trenches, dissecting the pivotal decisions, the unexpected hurdles, and the hard-won lessons that rarely make it into a polished case study. You’ll hear the unvarnished stories behind the features and companies shaping our world, focusing on the practical frameworks and mental models that effective product leaders and creators rely on daily. It’s about understanding the craft from the inside out-the strategic shifts, the team dynamics, the user insights that truly move the needle. Tune in for a direct, no-fluff dialogue designed to accelerate your own journey, providing actionable guidance you can apply immediately to level up your own work. This is where the blueprint meets the build.
Author: Language: English Episodes: 50

Behind the Craft
Podcast Episodes
OpenAI PM Reveals How He Uses Codex to Do Product Work | Rohan Varma [not-audio_url] [/not-audio_url]

Duration: 39:21
As a product manager for OpenAI Codex, Rohan has a firsthand look at how one of the most AI-native teams in the world operates. He showed me how he uses Codex to trigger work from Slack replies, prototype designs with im…
Agentic Engineering, Clearly Explained in 50 Min | Matt Van Horn [not-audio_url] [/not-audio_url]

Duration: 47:45
Matt has no CS degree, doesn’t know how to code, and doesn’t even read the plans that AI writes. Despite this, he’s built an incredible open source track record with projects like last30days (#1 trending on GitHub with 4…
Inside Anthropic: How Claude Actually Gets Built | Alex Albert [not-audio_url] [/not-audio_url]

Duration: 35:03
Alex is a research PM at Anthropic building the next Claude model. He gave me a rare inside look at how the research team operates — including how they turn user feedback into model training, how they decide which capabi…