How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

Author: Peter Yang August 23, 2026 Duration: 53:52

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com

🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so


Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂


P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG


We talked about:

(00:00) Why AI evals are completely different now with the latest models

(02:30) Demo: Live audit of the evals that I built for my AI skills

(05:14) Top down vs. bottom up evals and why AI sucks at the latter

(08:16) Spinning up AI agents to grade each eval criteria

(14:32) Demo: Using Shreyas AI skill to run evals in Claude

(23:46) How to review output labels to identify failures

(34:37) How to turn failures into reusable evals

(43:19) Where human judgement is still needed


Resources mentioned:

Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills

Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/


Thanks to our sponsors:

Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang


📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel


Where to find Shreya and Hamel:

Shreya: https://www.sh-reya.com/

Hamel: https://hamel.dev/


Subscribe to this channel - more interviews coming soon!


For anyone building the future, Behind the Craft is a conversation with Peter Yang that moves beyond theory and into the tangible details of creation. This podcast lives in the messy, rewarding space where ideas become real products. Each episode is built on candid interviews with experts who have been in the trenches, dissecting the pivotal decisions, the unexpected hurdles, and the hard-won lessons that rarely make it into a polished case study. You’ll hear the unvarnished stories behind the features and companies shaping our world, focusing on the practical frameworks and mental models that effective product leaders and creators rely on daily. It’s about understanding the craft from the inside out-the strategic shifts, the team dynamics, the user insights that truly move the needle. Tune in for a direct, no-fluff dialogue designed to accelerate your own journey, providing actionable guidance you can apply immediately to level up your own work. This is where the blueprint meets the build.
Author: Language: English Episodes: 50

Behind the Craft
Podcast Episodes
How OpenAI's Codex Team Builds with Codex (43 Min) | Alex & Romain [not-audio_url] [/not-audio_url]

Duration: 43:16
Alex is the product lead for Codex and Romain runs developer experience. They gave me a rare inside look at how the Codex team operates, including how they build products with Codex and how they ship without traditional…
Master Claude Code + Figma MCP for Design in 50 Min | Felix Lee [not-audio_url] [/not-audio_url]

Duration: 50:15
Felix is an expert at using Claude Code and Figma MCP to design and build apps. Watch as he demos how you can build apps that look exactly like their Figma designs, build a game from a FigJam flow cart, and convert code…