Product Metrics are LLM Evals // Raza Habib CEO of Humanloop // #320

Product Metrics are LLM Evals // Raza Habib CEO of Humanloop // #320

Author: Demetrios June 3, 2025 Duration: 53:06

Raza Habib, the CEO of the LLM Eval platform Humanloop, talks to us about how to make your AI products more accurate and reliable by shortening the feedback loop of your evals. Quickly iterating on prompts and testing what works, along with some of his favorite Dario from Anthropic AI Quotes.


// Bio

Raza is the CEO and Co-founder at Humanloop. He has a PhD in Machine Learning from UCL, was the founding engineer of Monolith AI, and has built speech systems at Google. For the last 4 years, he has led Humanloop and supported leading technology companies such as Duolingo, Vanta, and Gusto to build products with large language models. Raza was featured in the Forbes 30 Under 30 technology list in 2022, and Sifted recently named him one of the most influential Gen AI founders in Europe.


// Related Links

Websites: https://humanloop.com


~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

MLOps Swag/Merch: [https://shop.mlops.community/]


Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Raza on LinkedIn: /humanloop-raza


Timestamps:

[00:00] Cracking Open System Failures and How We Fix Them

[05:44] LLMs in the Wild — First Steps and Growing Pains

[08:28] Building the Backbone of Tracing and Observability

[13:02] Tuning the Dials for Peak Model Performance

[13:51] From Growing Pains to Glowing Gains in AI Systems

[17:26] Where Prompts Meet Psychology and Code

[22:40] Why Data Experts Deserve a Seat at the Table

[24:59] Humanloop and the Art of Configuration Taming

[28:23] What Actually Matters in Customer-Facing AI

[33:43] Starting Fresh with Private Models That Deliver

[34:58] How LLM Agents Are Changing the Way We Talk

[39:23] The Secret Lives of Prompts Inside Frameworks

[42:58] Streaming Showdowns — Creativity vs. Convenience

[46:26] Meet Our Auto-Tuning AI Prototype

[49:25] Building the Blueprint for Smarter AI

[51:24] Feedback Isn’t Optional — It’s Everything


Hosted by Demetrios, MLOps.community is a space for honest, meandering talks about the real work of making artificial intelligence systems actually work. This isn't about hype or theoretical papers; it's about the messy, practical, and often surprising journey of taking models from a notebook into a live environment. You'll hear from engineers and practitioners who are in the trenches, discussing the tools, the frustrations, and the occasional breakthroughs that define the day-to-day. The conversations are deliberately relaxed, covering everything from traditional machine learning pipelines to the new world of large language models and even the intangible "vibes" of team culture and process. Each episode peels back a layer on what "production" really means, whether that involves deploying a predictive service, managing an agentic system, or maintaining reliability as everything scales. Tuning into this podcast feels like grabbing a coffee with colleagues who aren't afraid to dig into the technical nitty-gritty while keeping the tone conversational and accessible. It's for anyone who builds, manages, or is just curious about the operational backbone that allows AI to deliver value, offering a grounded perspective often missing from the broader conversation.
Author: Language: en-us Episodes: 100

MLOps.community
Podcast Episodes
Best AI Hackathon Project Ever? [Bite Size Episode] [not-audio_url] [/not-audio_url]

Duration: 9:08
AI Conversations Powered by Prosus Group Unicorn Mafia won the recent hackathon at Raise Summit and explained to me what they built, including all the tech they used under the hood to make their AI agents work. Winners:C…
Are Evals Dead? [not-audio_url] [/not-audio_url]

Duration: 25:24
AI Conversations Powered by Prosus Group Your AI agent isn’t failing because it’s dumb—it’s failing because you refuse to test it. Chiara Caratelli cuts through the hype to show why evaluations—not bigger models or fanci…
The DuckLake Lakehouse Format // Hannes Mühleisen // #339 [not-audio_url] [/not-audio_url]

Duration: 57:24
The DuckLake Lakehouse Format // MLOps Podcast #339 with Hannes Mühleisen, Co-founder and CEO of DuckDB Labs.Join the Community: https://go.mlops.community/YTJoinInGet the newsletter: https://go.mlops.community/YTNewslet…
How LiveKit Became An AI Company By Accident [not-audio_url] [/not-audio_url]

Duration: 32:34
AI Conversations Powered by Prosus Group Russ d'Sa shares how LiveKit went from a small open-source project during the pandemic to powering voice interfaces for giants like OpenAI. He talks about the turning point when L…
Economics of Building Data Centers, GPU Clouds, Sovereign AI [not-audio_url] [/not-audio_url]

Duration: 45:59
AI Conversations Powered by Prosus Group. Craig Tavares, COO of Buzz High Performance Compute, shares lessons from building GPU cloud infrastructure worldwide. He stresses the role of sovereign mandates, renewable power,…
The Era of AI Agents in Marketing // Joel Horwitz // #337 [not-audio_url] [/not-audio_url]

Duration: 48:56
The Era of AI Agents in Marketing // MLOps Podcast #337 with Joel Horwitz, Growth Engineer at Neoteric3D.Join the Community: https://go.mlops.community/YTJoinInGet the newsletter: https://go.mlops.community/YTNewsletter/…