Why Your P95 Latency Explodes Before Your CPU Does

P95 or P99 latency often spikes while CPU still looks "fine." Databases rarely fail exactly at 100% CPU, they degrade when queueing, contention, and coordination overhead begin compounding under load. Here's what's happening under the hood:

1. Queueing starts before saturation: As concurrency rises, requests wait on locks, buffers, worker slots, or the CPU run queue. Even small increases in queue depth amplify tail latency well before average CPU looks critical.
2. Lock contention hides inside normal utilization: Hot rows, gap locks, or long transactions serialize access and create lock wait chains. A small number of blocked queries can cascade into tail latency spikes without materially increasing average CPU.
3. Disk and memory stalls hit latency first: Checkpoints, I/O flushes, temp file spills, cache churn, or page faults introduce unpredictable pauses. These stalls disproportionately inflate tail latency even when throughput appears steady.
4. Planner and executor overhead compounds under concurrency: Even with plan caching, queries still incur executor, memory, and latch overhead. As concurrency rises, coordination costs and context switching push latency up faster than CPU utilization suggests.
5. Replication and network effects surface in the tail: Cross-zone hops, network jitter, and connection pool exhaustion add variance that disproportionately impacts P95. Replica lag and replay pressure can further amplify tail latency without shifting averages much.

Tail latency is a leading indicator. When P95 climbs, it usually reflects excess work and coordination pressure building beneath the surface before CPU saturates. Readyset removes redundant query execution at the SQL layer, so increasing concurrency does not compound coordination overhead into tail latency spikes.

🔍 What We're Watching in the Database Ecosystem

Databases Built for AI Agents

This post explores how, as AI agents become the primary database users, architecture must shift toward machine-speed responsiveness, elastic compute over persistent clusters, and isolation-by-default environments that can be created and torn down on demand. When workloads are bursty, combinatorial, and short-lived, the real challenge becomes sustaining high concurrency without coordination overhead and redundant execution compounding under load.

OpenAI Raises $110B in Historic Funding Round

OpenAI's $110B round signals that frontier AI is now an infrastructure race, measured in gigawatts of training and inference capacity. As models scale and stateful runtimes move closer to production workloads, the bottleneck shifts downstream to data systems that can sustain high-concurrency, low-latency access without becoming the limiting factor.

In Case You Missed It

New on our blog: Replication Internals: Decoding the MySQL Binary Log - Part 1: Introduction and Data Types

Marcelo walks through the raw structure of MySQL's binary log, decoding event formats and value encodings byte by byte. If you're building CDC tooling, debugging replication behavior, or want a first-principles understanding of how changes move through MySQL, this is foundational reading.