Complete Beginner's Course on AI Evaluations: Step by Step (2025) | Aman Khan

Complete Beginner's Course on AI Evaluations: Step by Step (2025) | Aman Khan

Author: Peter Yang August 24, 2025 Duration: 51:47

Today, I want to share a new episode with Aman Khan.The best way to learn about AI evaluations is to watch 2 PMs build them live from scratch. In our new episode, Aman and I walk through creating evals for an AI customer support agent — from labeling a golden dataset to aligning LLM judges. This is the complete beginners AI eval course you've been waiting for.Aman and I talked about:

(00:00) What are AI evals and how to get good at them

(02:52) The 4 types of AI evaluations everyone should know

(06:08) Live demo: Building evals for a customer support agent

(10:29) Using Anthropic's console to generate great prompts

(15:13) Creating the evaluation criteria

(17:40) Adding human labels to the golden dataset

(31:05) Scaling evals with LLM-judge prompts

(38:21) How to align LLM judges with human judgmentGet the takeaways: https://creatoreconomy.so/p/complete-beginner-course-on-ai-evaluations-aman-khanWhere to find Aman:

X: https://www.linkedin.com/in/amanberkeley/

Website: https://arize.com/📌 Subscribe to this channel – more interviews coming soon!


For anyone building the future, Behind the Craft is a conversation with Peter Yang that moves beyond theory and into the tangible details of creation. This podcast lives in the messy, rewarding space where ideas become real products. Each episode is built on candid interviews with experts who have been in the trenches, dissecting the pivotal decisions, the unexpected hurdles, and the hard-won lessons that rarely make it into a polished case study. You’ll hear the unvarnished stories behind the features and companies shaping our world, focusing on the practical frameworks and mental models that effective product leaders and creators rely on daily. It’s about understanding the craft from the inside out-the strategic shifts, the team dynamics, the user insights that truly move the needle. Tune in for a direct, no-fluff dialogue designed to accelerate your own journey, providing actionable guidance you can apply immediately to level up your own work. This is where the blueprint meets the build.
Author: Language: English Episodes: 100

Behind the Craft
Podcast Episodes
Why You Should Push Yourself to Do Hard Things | Nat Eliason (Author) [not-audio_url] [/not-audio_url]

Duration: 50:13
My guest today is Nat Eliason, one of my favorite writers online. Nat is the author of Crypto Confidential, a real-life thriller about how he made and lost millions in crypto. We also talked about how Nat uses AI to edit…
Proven Rituals of Great Product Teams | Lane Shackleton (CPO Coda) [not-audio_url] [/not-audio_url]

Duration: 33:08
My guest today is Lane Shackleton, Chief Product Officer of Coda. Lane has interviewed hundreds of product teams from Figma, Uber, Spotify, and more as the chief product officer of Coda. He shared with me his favorite ri…
Unlock Your Storytelling Superpower | Robbie Crabtree (Coach) [not-audio_url] [/not-audio_url]

Duration: 36:31
My guest today is Robbie Crabtree, founder of Competitive Storytelling. Robbie is a top storytelling coach who has helped hundreds of founders raise $600M+ through this skill. Before that, he perfected his storytelling c…