Why Your AI Bill Will Double Before It Gets Better

Why Your AI Bill Will Double Before It Gets Better

Author: Demetrios August 3, 2026 Duration: 32:24

In this episode, we're joined by Josh Collier, FinOps Lead at Superhuman (formerly Grammarly), to explore what it really costs to run AI at scale and why the rules of the game changed faster than anyone expected.


We discuss how AI token costs dropped 80% in two years, why that trend has sharply reversed with frontier models doubling in price, and how Josh rebuilt a single LLM workflow that cost $400k a month down to $80k by rethinking the architecture. He also shares how a cost calculator built in 15 minutes transformed the way his team estimates spend before running experiments, and why research-led optimization is the only kind that works without degrading the product.


Along the way, we cover hidden costs most teams miss, the trade-off between Azure reserved capacity and OpenAI Priority Processing, why fixed subscription pricing is broken in an AI-native world, vendor lock-in risk, and what OpenAI's Guaranteed Capacity announcement really signals about where vendor relationships are heading next.


Superhuman: https://superhuman.com


Josh Collier: https://www.linkedin.com/in/josh-collier-945b7029/

Demetrios: https://www.linkedin.com/in/dpbrinkm


Timestamps:

[00:00] OpenAI Guaranteed Capacity: what's really going on

[01:04] Josh's path into AI FinOps

[02:48] Token costs: the 80% price drop

[04:16] Why costs will only go up

[05:06] External LLMs as financial risk

[07:16] Why subscription pricing is dead

[08:22] The data residency fee nobody notices

[09:33] The cost calculator built in 15 minutes

[10:24] How it changed dev team speed

[13:00] Tracking costs by service and team

[15:33] $400k workflow rebuilt for $80k

[17:13] Why only research can optimize tokens

[20:00] Speculative decoding win

[23:11] One bad query, $40k gone

[26:00] Why Azure PTU was exhausting

[28:59] Shadow traffic load testing

[29:07] Priority processing: no brainer

[31:10] Guaranteed capacity: lock-in signal?

[32:18] The danger of multi-year AI deals

[33:28] Vendor-agnostic proxy as exit strategy


Hosted by Demetrios, MLOps.community is a space for honest, meandering talks about the real work of making artificial intelligence systems actually work. This isn't about hype or theoretical papers; it's about the messy, practical, and often surprising journey of taking models from a notebook into a live environment. You'll hear from engineers and practitioners who are in the trenches, discussing the tools, the frustrations, and the occasional breakthroughs that define the day-to-day. The conversations are deliberately relaxed, covering everything from traditional machine learning pipelines to the new world of large language models and even the intangible "vibes" of team culture and process. Each episode peels back a layer on what "production" really means, whether that involves deploying a predictive service, managing an agentic system, or maintaining reliability as everything scales. Tuning into this podcast feels like grabbing a coffee with colleagues who aren't afraid to dig into the technical nitty-gritty while keeping the tone conversational and accessible. It's for anyone who builds, manages, or is just curious about the operational backbone that allows AI to deliver value, offering a grounded perspective often missing from the broader conversation.
Author: Language: en-us Episodes: 50

Agentic Conversations (formally mlops.community)
Podcast Episodes
We Cut LLM Latency by 70% in Production [not-audio_url] [/not-audio_url]

Duration: 1:05:20
Maher Hanafi is an engineering leader who went from zero AI experience to self-hosting LLMs at enterprise scale — managing GPU costs, optimizing inference with TensorRT LLM, and building an AI platform for HR tech. In th…
Getting Humans Out of the Way: How to Work with Teams of Agents [not-audio_url] [/not-audio_url]

Duration: 50:30
Rob Ennals is the creator of Broomy, an open-source IDE designed for working effectively with many agents in parallel. He previously worked at Meta, Quora, Google Search, and Intel Research. He has a PhD in Computer Scie…
Fixing GPU Starvation in Large-Scale Distributed Training [not-audio_url] [/not-audio_url]

Duration: 52:48
Kashish Mittal is a Staff Software Engineer at Uber, working on large-scale distributed systems and core backend infrastructure.Fixing GPU Starvation in Large-Scale Distributed Training // MLOps Podcast #367 with Kashish…
arrowspace: Vector Spaces and Graph Wiring [not-audio_url] [/not-audio_url]

Duration: 56:01
Lorenzo Moriondo is a Technical Lead for AI at tuned.org.uk, working on AI agent protocols, graph-based search, and production-grade LLM systems.arrowspace: Vector Spaces and Graph Wiring // MLOps Podcast #365 with Loren…
Agentic Marketplace [not-audio_url] [/not-audio_url]

Duration: 51:26
Donné Stevenson is a Machine Learning Engineer at Prosus, working on scalable ML infrastructure and productionizing GenAI systems across portfolio companies.Pedro Chaves is a Data Science Manager at OLX Group, working on…
Durable Execution and Modern Distributed Systems [not-audio_url] [/not-audio_url]

Duration: 1:00:36
Johann Schleier-Smith is the Technical Lead for AI at Temporal Technologies, working on reliable infrastructure for production AI systems and long-running agent workflows. Durable Execution and Modern Distributed Systems…