Readyset Docs

Day-2 operations for QueryPilot, including config validation and reload, graceful shutdown, logging, monitoring, and pool management.

This page covers running QueryPilot after deployment. The Router is the SQL proxy (binary querypilot). Everything here assumes the metrics endpoint is enabled ([metrics] enabled = true), which serves Prometheus metrics on port 9090 and the Router's admin API on 9091 by default.

Config validation

Both binaries take --check: load the config, validate it, print the result, and exit without serving traffic.

querypilot --check -c /etc/sqp/querypilot.toml

The Router prints Configuration is valid on success and a parse or validation error otherwise. Its schema is strict: unknown keys fail. Run --check in CI or a deploy hook before rolling a config change; a config that passes --check will not fail startup on schema grounds.

Online config reload

Router

When the metrics endpoint is enabled, the Router serves a reload API on the admin port:

curl -X POST http://localhost:9091/api/config/reload
curl http://localhost:9091/api/config/status

Reloadable without a restart:

routing.rules and routing.pattern_rules
routing.read_write_split, routing.read_pool, routing.write_pool, routing.default_pool, routing.mysql_default_pool
routing.cache.enabled, routing.cache.max_size
[[translation.rules]] (translations are cleared and re-registered from config; translations registered at runtime via the API are ephemeral and do not survive a reload)
logging.level

Not reloadable: listeners, pools, tls, and the rest of general. Those require a restart, and reloaded rules can only reference pools that existed at startup.

Reload validates before applying: the TOML must parse, regexes must compile, and pool references must exist. On failure it returns 400 identifying the offending rule and field, and the old config stays active. On success the change set is applied atomically and the routing-decision cache is cleared. The response reports what changed (rule counts, whether the read/write split, cache config, or log level moved).

Graceful shutdown and connection draining

The Router shuts down on SIGTERM or Ctrl+C (docker stop and Kubernetes pod termination both send SIGTERM):

Listeners stop accepting new connections.
The Router waits for active client connections to drain, up to a fixed 5-second window.
Connections still open when the window expires are force-closed, with a warning log that includes the remaining count.
Plugins shut down last, after traffic has stopped.

Plan rolling restarts around the 5-second drain: with transaction pooling, in-flight transactions shorter than the window complete; long-running queries are cut off. Drain the pod from your load balancer before sending SIGTERM if you need zero interruption. The shutdown_timeout key under [general] parses but the drain window is fixed at 5 seconds in this release.

Logging

Both servers configure logging from the [logging] section: level, and format = "pretty" (human-readable, for terminals) or "json" (one JSON object per line, for log pipelines). The provided deployment configs use json.

Override the level at launch without editing the config:

QUERYPILOT_LOG_LEVEL=debug querypilot

Precedence is CLI flag (--log-level) over environment variable over config file. The Router's config level is a closed list (trace/debug/info/warn/error); the env/CLI override accepts full tracing filter directives such as per-module levels.

RUST_LOG is not read by either binary; exporting it does nothing. Both servers reload logging.level via POST /api/config/reload as described above. See environment variables.

Monitoring

Both components serve Prometheus metrics on GET /metrics at their [metrics] bind address. The full catalog is in the metrics reference; ready-made Grafana dashboards and alert rules are in Extras. Start by alerting on these:

Metric	Alert when	Meaning
`sqp_backend_health`	`== 0` for any `pool`/`backend`	A backend failed health checks and is out of rotation
`sqp_pool_utilization_ratio`	Sustained near 1.0	Pool at its `max_connections` cap; requests will queue
`sqp_pool_timeouts_total`	Increasing	Sessions gave up waiting for a connection (`acquire_timeout` expired)
`sqp_query_errors_total`	Rate above your baseline, by `error_type`	Query failures (deadlocks, syntax, connection loss, timeouts)

Useful companions: sqp_pool_queue_depth and sqp_pool_wait_duration_seconds (leading indicators of exhaustion), sqp_queries_total and sqp_query_duration_seconds (traffic and latency baselines), sqp_routing_decisions_total by pool (verifies traffic splits the way you configured), and sqp_auth_failures_total.

Pool status management

Take a pool out of rotation without a restart, for example ahead of backend maintenance:

curl http://localhost:9091/api/pools                       # list pools and statuses
curl -X PUT http://localhost:9091/api/pools/replica/status \
  -H "Content-Type: application/json" -d '{"status": "offline_soft"}'

Statuses are online, offline_soft, and shunned. Both non-online statuses cause acquire attempts against the pool to fail immediately as if no healthy backends exist, so make sure routing has somewhere else to send that traffic before you flip a pool offline. Set {"status": "online"} to restore it. Status overrides live in memory and reset to online on restart. See the admin API reference.

Operations