Complete Beginner's Course on AI Evaluations: Step by Step (2025) | Aman Khan

Complete Beginner's Course on AI Evaluations: Step by Step (2025) | Aman Khan

Author: Peter Yang August 24, 2025 Duration: 51:47

Today, I want to share a new episode with Aman Khan.The best way to learn about AI evaluations is to watch 2 PMs build them live from scratch. In our new episode, Aman and I walk through creating evals for an AI customer support agent — from labeling a golden dataset to aligning LLM judges. This is the complete beginners AI eval course you've been waiting for.Aman and I talked about:

(00:00) What are AI evals and how to get good at them

(02:52) The 4 types of AI evaluations everyone should know

(06:08) Live demo: Building evals for a customer support agent

(10:29) Using Anthropic's console to generate great prompts

(15:13) Creating the evaluation criteria

(17:40) Adding human labels to the golden dataset

(31:05) Scaling evals with LLM-judge prompts

(38:21) How to align LLM judges with human judgmentGet the takeaways: https://creatoreconomy.so/p/complete-beginner-course-on-ai-evaluations-aman-khanWhere to find Aman:

X: https://www.linkedin.com/in/amanberkeley/

Website: https://arize.com/📌 Subscribe to this channel – more interviews coming soon!


For anyone building the future, Behind the Craft is a conversation with Peter Yang that moves beyond theory and into the tangible details of creation. This podcast lives in the messy, rewarding space where ideas become real products. Each episode is built on candid interviews with experts who have been in the trenches, dissecting the pivotal decisions, the unexpected hurdles, and the hard-won lessons that rarely make it into a polished case study. You’ll hear the unvarnished stories behind the features and companies shaping our world, focusing on the practical frameworks and mental models that effective product leaders and creators rely on daily. It’s about understanding the craft from the inside out-the strategic shifts, the team dynamics, the user insights that truly move the needle. Tune in for a direct, no-fluff dialogue designed to accelerate your own journey, providing actionable guidance you can apply immediately to level up your own work. This is where the blueprint meets the build.
Author: Language: English Episodes: 100

Behind the Craft
Podcast Episodes
How to Become an AI First PM and Designer | Paul Adams (CPO Intercom) [not-audio_url] [/not-audio_url]

Duration: 50:55
My guest today is Paul Adams, Chief Product Officer of Intercom. When ChatGPT launched, Paul and the Intercom team blew up their roadmap to go all in on AI. Since then, they’ve built an AI-first customer service suite th…
The Best Product Designer I Ever Worked With | Alex Cornell (Meta AI) [not-audio_url] [/not-audio_url]

Duration: 48:06
My guest today is Alex Cornell, Director of Design at Meta AI. Alex is one of the best product designers I’ve ever worked with. He helped design Facebook, Substack, Linear, and an app for family and friends called Cocoon…
Inside How Figma Builds AI Products | Yuhki Yamashita (CPO Figma) [not-audio_url] [/not-audio_url]

Duration: 35:55
My guest today is Yuhki Yamashita, Chief Product Officer of Figma. Yuhki and I talked about how Figma: - Created a culture where employees love their craft - Builds AI products to solve real user problems - Plans to make…
Building the World's Most Wearable AI | Dan Siroker (CEO Limitless) [not-audio_url] [/not-audio_url]

Duration: 22:44
My guest today is Dan Siroker, CEO of Limitless. In just 24 hours after launch, Dan and team sold over 10,000 AI pendants to help people augment their memory with what they’ve seen, said, and heard - making Limitless arg…
How the Best AI Products Grow | Brian Balfour (CEO Reforge) [not-audio_url] [/not-audio_url]

Duration: 39:14
My guest today is Brian Balfour, CEO of Reforge. Brian and I did a deep dive into how 4 of the best AI products grew so fast. We spoke about: Why ChatGPT is investing in custom GPTs How Midjourney grew to $200M ARR despi…
How to Crack Your Product Strategy | Jackie Bavaro (PM Author) [not-audio_url] [/not-audio_url]

Duration: 35:37
My guest today is Jackie Bavaro, author of Cracking the PM Career and Cracking the PM Interview, two of the most popular books on product management out there. Jackie was also head of PM at Asana. I spoke to Jackie about…
How the PM Role is Changing in 2024 | PM Expert Panel [not-audio_url] [/not-audio_url]

Duration: 51:27
My guests today are Sanchan (ex-general manager at Airbnb), Nickey (Product Director at Duolingo) and John (Senior Director at Toast). We talked about: 1. How the PM role is changing 2. Our favorite PM use cases for AI 3…
The journey from SVP to solopreneur | Elena Verna (Dropbox) [not-audio_url] [/not-audio_url]

Duration: 36:25
Elena Verna is a solopreneur who has served as head of growth at Dropbox, Amplitude, Miro, and more. Elena and I talked about: 1. Why she left a full-time role to work for herself 2. The five laws of growth and how they…

«1...678910