The AI Skill That Will Define Your PM Career in 2025 | Aman Khan (Arize)

The AI Skill That Will Define Your PM Career in 2025 | Aman Khan (Arize)

Author: Peter Yang January 12, 2025 Duration: 46:26

My guest today is Aman Khan.

Aman is the Director of Product at Arize AI. Last year, we both heard the Chief Product Officers of OpenAI and Anthropic share that AI evaluations will be the most important skill for PMs in 2025. Aman gave me a crash course on how to build this critical skill in our interview.

 

Timestamps:

  • (00:00) Evals force you to get into your user's shoes

  • (03:42) 5 skills to build right now to become a great AI PM

  • (07:43) How curiosity leads to better AI products

  • (10:20) You can build AI without the job title

  • (17:00) A deep dive into why AI evals are so important

  • (24:53) Example of running evals for an AI customer support agent

  • (31:32) When humans need to be in the loop for evals

  • (35:46) How to get better at writing evals right now

  • (39:39) My personal prompt and eval for transcripts

  • (44:23) Catching the AI wave is alot like surfing

     

    Get the takeaways: https://creatoreconomy.so/p/the-ai-skill-that-will-define-your-pm-career-aman-khan

     

    Where to find Aman:

    LinkedIn: https://www.linkedin.com/in/amanberkeley/

    Arize: https://arize.com/


    📌 Subscribe to this channel – more interviews coming soon!


    For anyone building the future, Behind the Craft is a conversation with Peter Yang that moves beyond theory and into the tangible details of creation. This podcast lives in the messy, rewarding space where ideas become real products. Each episode is built on candid interviews with experts who have been in the trenches, dissecting the pivotal decisions, the unexpected hurdles, and the hard-won lessons that rarely make it into a polished case study. You’ll hear the unvarnished stories behind the features and companies shaping our world, focusing on the practical frameworks and mental models that effective product leaders and creators rely on daily. It’s about understanding the craft from the inside out-the strategic shifts, the team dynamics, the user insights that truly move the needle. Tune in for a direct, no-fluff dialogue designed to accelerate your own journey, providing actionable guidance you can apply immediately to level up your own work. This is where the blueprint meets the build.
    Author: Language: English Episodes: 100

    Behind the Craft
    Podcast Episodes
    Inside Anthropic: How Claude Actually Gets Built | Alex Albert [not-audio_url] [/not-audio_url]

    Duration: 35:03
    Alex is a research PM at Anthropic building the next Claude model. He gave me a rare inside look at how the research team operates — including how they turn user feedback into model training, how they decide which capabi…
    How OpenAI's Codex Team Builds with Codex (43 Min) | Alex & Romain [not-audio_url] [/not-audio_url]

    Duration: 43:16
    Alex is the product lead for Codex and Romain runs developer experience. They gave me a rare inside look at how the Codex team operates, including how they build products with Codex and how they ship without traditional…