Better K8s Prometheus Alerts with Robusta

Better K8s Prometheus Alerts with Robusta

Author: Bret Fisher March 4, 2023 Duration: 51:47

🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship program for engineers wanting to learn the latest CI/CD automation and dive into Agentic DevOps. Meetups are happening now, with new course videos dropping every few weeks. Join the Guild and become your team's leader in AI for infrastructure automation https://www.bretfisher.com/theguild 🍾


Bret is joined by Natan Yellin, the co-founder of Robusta.dev to talk Kubernetes and Prometheus monitoring, alerting, and maybe some CPU limit ranting.

Robusta tries to fill the gap left by Kubernetes' own AlertManager which has a very specific and not so helpful way of describing events in your cluster. This makes it hard to diagnose the cause of the event and you're left with Google, StackOverflow and an awful lot of head-scratching. Robusta acts as a proxy between AlertManager and your notification platform of choice.

In the show we talk about what Robusta is, how to deploy it in your clusters, and Natan also details some of the enhancements in their cloud offering that you can layer on top of that, which has a generous free tier.

Streamed live on YouTube on January 5, 2023.


Unedited live recording of this show on YouTube (Ep. #197). Includes demos.

Topics★
Robusta Website
Robusta on GitHub
KubeCon - Building a Runbook Automation System for Prometheus and Kubernetes
Stop using K8s CPU limits
Recommended Pod Spec
Send Push notifications to your phone
Prometheus AlertManager
Grafana Labs
Kubewatch

★Natan Yellin★
Natan on Twitter
Natan on LinkedIn

Join my Community
New live course on CI automation and gitops deployments
Best coupons for my Docker and Kubernetes courses
Chat with us and fellow students on our Discord Server DevOps Fans
Grab some merch at Bret's Loot Box

Homepage bretfisher.com

  • (00:00) - DDT MAIN
  • (00:04) - Intro
  • (03:10) - In today's episode
  • (05:16) - Main show
  • (05:44) - Introducing Natan
  • (06:10) - Alert fatigue
  • (06:46) - Where did the idea for Robusta come from?
  • (10:33) - Someone has to do the job
  • (11:34) - What does Robusta offer?
  • (12:42) - Proxying the alerts and providing context
  • (13:47) - Saving 10 to 30 minutes
  • (16:05) - The open source Robusta repo
  • (16:27) - The need to de-aggregate event data
  • (17:26) - Example or demo
  • (17:56) - Question about observability for microservices
  • (20:55) - Tip 1 Consider using silences
  • (22:06) - Tip 2 Monitor outcomes
  • (22:40) - Don't ignore alerts because of fatigue
  • (25:30) - Sending to different channels based on priority
  • (26:59) - Question about sending messages to destinations
  • (28:34) - Question
  • (29:06) - Installing Robusta
  • (29:59) - Demo set up commands
  • (30:11) - Questions
  • (30:28) - Demo Kubernetes-specific
  • (31:22) - Multi-cluster question
  • (33:49) - What does the SaaS platform do?
  • (35:01) - Demo with SaaS
  • (35:54) - kubectl not recommended
  • (37:20) - Breaking the glass
  • (40:32) - Question about notifications
  • (42:31) - Getting started
  • (43:41) - CPU limiting
  • (44:32) - Soft limits on CPU in Kubernetes
  • (46:52) - Bret's pod spec
  • (51:39) - Outro

You can also support this podcast by subscribing to my YouTube channel and my weekly newsletter at bret.news!

Grab the best coupons for my Docker and Kubernetes courses.
Join my cloud native DevOps community on Discord.
Grab some merch at Bret's Loot Box
Homepage bretfisher.com


Bret Fisher hosts DevOps and Docker Talk: Cloud Native Interviews and Tooling alongside Nirmal Mehta, pulling conversations directly from their live audience sessions. This isn't a theoretical lecture series; it's grounded in the practical, often messy realities of building and running modern systems. Each episode digs into the specific tools and philosophies shaping infrastructure today, from the foundational elements like Docker and container orchestration with Kubernetes or Swarm to the broader cultural shifts of DevOps, SRE, and platform engineering. You'll hear discussions that span the entire software lifecycle, touching on the integration of security with DevSecOps, the automation promises of GitOps, and what "Cloud Native" truly means for development teams. The dialogue stays focused on actionable insights and real-world use cases, offering a genuine sense of what works and what doesn't in the field. For anyone navigating the container and cloud ecosystem, this podcast serves as a regular, insightful check-in with practitioners and experts, hosted by someone deeply embedded in the community. The natural, interview-driven format makes complex topics accessible, ensuring listeners come away with more than just buzzwords-they get context and clarity.
Author: Language: English Episodes: 100

DevOps and Docker Talk: Cloud Native Interviews and Tooling
Podcast Episodes
Docker AI, what’s new with MCP, Agents, Sandboxes, and more [not-audio_url] [/not-audio_url]

Duration: 1:18:38
Michael Irwin of Docker joins me to run through Gordon AI improvements, Docker Hardened Images and what's now free, Docker Sandboxes for running agents in proper isolation, Model Runner updates including MLX support on M…
Backup S3, Google Drive, iCloud, Notion with Plakar [not-audio_url] [/not-audio_url]

Duration: 1:09:10
Bret is joined by the founders of Plakar - Julien Mangeard and Gilles Chehade - to nerd out over backup engineering. The kind where you're building your own file formats and cryptographic layers, not just wiring up cron…
Your Images are Out of Date (probably) - The Silent Rebuilds problem [not-audio_url] [/not-audio_url]

Duration: 38:59
Container base images (like Official Docker Hub images) are often updated without new tag versions. I call this Silent Rebuilds. There's no way to know this happens without image digest-checking automation like Dependabo…
AI Wins and Misses for 2025 [not-audio_url] [/not-audio_url]

Duration: 1:17:14
I'm joined by Nirmal Mehta of AWS and Viktor Farcic from Upbound, to go through our 2025 year in review. We look into the AI tools that consumed us this year, from CLI agents to terminal emulators, IDEs, AI browsers - wh…
Find Your K8s Happy Path with RawKode Academy [not-audio_url] [/not-audio_url]

Duration: 53:29
I talk with David Flanagan, aka Rawkode, about his new opinionated Tech Matrix that helps you navigate the overwhelming CNCF landscape. https://rawkode.academy/technology/matrix🙌 The Agentic DevOps Guild has launched! It…
Move K8s Stateful Pods Between Nodes [not-audio_url] [/not-audio_url]

Duration: 46:59
Bret is joined by Philip Andrews and Dan Muret of Cast AI to discuss pod live migration between nodes in a Kubernetes cluster. 🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship program for…
Swarm 2030. Let's Go! [not-audio_url] [/not-audio_url]

Duration: 26:05
Bret discusses exciting news about Swarm being maintained until 2030.🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship program for engineers wanting to learn the latest CI/CD automation an…
Is Docker Building the Best AI Stack? [not-audio_url] [/not-audio_url]

Duration: 1:07:45
Bret and Nirmal are joined by Michael Irwin to discuss Docker's comprehensive AI toolkit, covering everything from local model deployment to cloud-based container orchestration across multiple interconnected tools and se…
Could AI End Human QA? [not-audio_url] [/not-audio_url]

Duration: 54:59
Bret is joined by Andrew Tunall, the President and Chief Product Officer at Embrace, to discuss his prediction that we’ll all start shipping non-QA'd code (buggier code in production) and QA will need to be replaced with…
AI Agents Running Containers [not-audio_url] [/not-audio_url]

Duration: 57:31
The Daytona founders - Ivan Burazin and Vedran Jukic - discuss their pivot to an AI agent cloud. We dig into the new infrastructure requirements of developing agents that need their own sandboxes to operate in.A year ago…