SE Radio 717: Eric Tschetter on Decoupling Observability

In this episode, host Amey Ambade sits with Eric Tschetter, co-founder of Apache Druid and Chief Architect at Imply, to dissect the critical move toward Decoupling Observability. To begin, they define three pillars—logs, metrics, and traces—and consider why the rise of microservices has made traditional, tightly coupled stacks a major source of pain. Such coupled systems can lead to issues such as vendor lock-in, prohibitive scaling costs, and operational complexity.

Drawing parallels to the Business Intelligence world's separation, Tschetter presents an architectural solution with four distinct layers: Ingest/Route, Data Storage, Query/Compute, and Visualization. This framework aims to provide flexibility to combat the limitations of monolithic observability tools. The conversation moves into the practical challenges and significant benefits of this decoupled model, focusing heavily on data portability and the role of technologies such as OpenTelemetry in standardizing schemas so that data can flow freely between multiple back-ends. A significant portion of the discussion is dedicated to the Query/Compute layer, specifically how Apache Druid addresses the unique demands of real-time analytics on observability data, including indexing strategies and unifying results across hot and cold storage. They also delve into operational survival, covering critical topics like smart sampling to preserve high-value signals, best practices for buffering and backpressure, and the governance models required for multiple teams to safely access the same data lake.

The episode concludes with an honest look at the complexity trade-offs and a roadmap for organizations considering a migration from a coupled vendor stack.