Observability

Monitoring answers questions you knew to ask. Observability lets you ask new ones during an incident, without shipping code.

The three signals. Metrics are cheap aggregates, ideal for dashboards and alerts, and they cannot tell you about one specific request. Logs are detailed per-event records, expensive at volume, and invaluable once you know where to look. Traces follow one request across services and are the fastest way to answer "which hop was slow" in a distributed system.

Percentiles, not averages. An average latency hides everything that matters. If p50 is 40ms and p99 is 4s, one percent of requests are terrible — and for a page making 100 backend calls, nearly every page view hits at least one. Alert on p95 and p99; treat the average as decoration.

The four golden signals — latency, traffic, errors, saturation — cover most of what you need from a service dashboard. Alert on symptoms, not causes. "Checkout error rate above 2%" tells you users are hurting. "CPU above 80%" might be perfectly healthy, and paging a human for it teaches them to ignore pages, which is how real alerts get missed.

Cardinality is the hidden cost. Every distinct label combination on a metric creates a new time series, so adding a user id as a label can multiply your metrics bill by millions. High-cardinality data belongs in logs and traces.

Service level objectives turn all of this into a decision rule. An SLO states a target — 99.9% of checkout requests served under 300ms over 30 days — and the error budget is whatever remains: 0.1%, or roughly 43 minutes a month. Spending it gradually is normal; spending it in one afternoon is a signal to stop shipping features and fix reliability instead. The value is that it converts an argument about whether the service is fast enough into arithmetic both sides already agreed to.

Make the data usable before you need it. Log structured events rather than prose, so a field can be filtered instead of grepped for. Attach a request id to everything and propagate it, or you cannot join a log line to the trace it belongs to. And sample deliberately: keeping every trace at volume is unaffordable, while a flat 1% throws away the rare slow request you most wanted, so sample errors and slow requests far more heavily than successes.