Operations and metrics

The admin listener should remain on loopback or behind authenticated network policy. It intentionally exposes no credentials, but backend names, shard names, health, and capacity are operationally sensitive.

Sensitive-data contract

DBProxy metrics never use SQL text, prepared parameters, Redis keys or values, shard-key values, usernames, passwords, authentication payloads, client addresses, or operation IDs as labels. Labels are limited to configured backend and shard names, roles, TLS modes, query/operation classes, outcomes, and capability names. This keeps cardinality bounded and avoids placing application data in Prometheus.

Structured logs similarly omit SQL text, prepared values, Redis command arguments and payloads, usernames, passwords, tokens, and shard-key values. Redis command names can appear in validation errors, but their keys and values do not. Logs can contain random operation IDs, client peer addresses, listener endpoints, configured backend/shard/table names, configuration revisions, and sanitized error descriptions. Treat those fields as operationally sensitive: restrict log access and retention, and keep debug disabled unless it is needed for a bounded investigation. Secret-bearing configuration structures deliberately do not implement Debug, invalid TOML diagnostics do not echo source lines or values, and credential-bearing URLs are rejected or sanitized before errors are logged.

Backend and operating-system errors may still reveal infrastructure details such as hostnames, ports, certificate paths, or database object names. The admin endpoints expose more topology detail than metrics, including backend addresses. Neither logs nor observability endpoints should be published as a public service.

Use proxy.tls.mode = "required" for client traffic and backends.tls.mode = "verify_identity" for database traffic in production. Mount private keys as read-only secrets and restrict them to the DBProxy process identity. The required backend mode provides encryption but deliberately skips certificate authentication, so it should be treated as a migration mode.

Readiness

/healthz is process liveness. /readyz requires every configured shard primary to have passed a health check and to remain below the configured consecutive failure threshold. Replicas do not gate readiness because reads can fall back to their shard primary.

When atomic MySQL fanout is enabled, readiness also requires dbproxy_xa_in_doubt_transactions == 0. A post-decision participant failure withdraws readiness, and restart recovery must resolve the durable decision before the listener becomes available.

At startup, readiness is false until the first successful primary probes. When PostgreSQL is enabled, readiness also requires every configured shard primary to pass its configured probe. The default is TCP transport. A per-backend readiness_probe authenticates with a dedicated role and executes SELECT 1; its password can be supplied through environment expansion. /postgres/backends reports the probe mode, effective readiness, TCP transport, optional SQL readiness, and connection counters. SQL probes currently use plaintext PostgreSQL unless readiness_probe.tls.enabled = true. TLS probes verify both the trust chain and hostname and can use an explicit CA and client identity.

When Redis is enabled, readiness also requires TCP transport health to every configured shard primary. Sentinel mode also requires the most recent refresh to reach its configured address quorum. Cluster mode instead requires a complete current 16,384-slot snapshot. /redis/backends reports dynamic Sentinel primary addresses plus transport and connection counters. Static transport probes do not authenticate, issue PING, validate replication role, or measure replica lag.

Prometheus groups

Per-backend metrics use bounded backend, shard, and role labels sourced only from static configuration. SQL text, usernames, client addresses, and database values are never used as labels.

Initial alerts

• dbproxy_backend_healthy{role="primary"} == 0 for longer than two health intervals: shard unavailable or not yet proven ready.
• increase(dbproxy_query_timeouts_total[5m]) > 0: backend saturation, lock contention, or network loss.
• dbproxy_backend_pool_waiters > 0 sustained for several minutes: pool pressure; correlate with in-use connections and query latency.
• dbproxy_backend_reserved_pool_waiters > 0: transaction or pinned-session concurrency exceeded the isolated primary reserve; increase it only after checking the database's total connection budget.
• Any increase in dbproxy_write_fanout_failures_total or dbproxy_write_fanout_timeouts_total: possible cross-database divergence.
• Any increase in dbproxy_scatter_failures_total: a participating shard failed, exceeded the shared result budget, or returned incompatible metadata.
• dbproxy_xa_in_doubt_transactions > 0: a durable commit or abort decision is not resolved everywhere. Do not replay the application write; preserve the journal volume and restore participant connectivity.
• A rising ratio of dbproxy_replica_fallbacks_total to dbproxy_queries_replica_total: unhealthy or undersized replicas.
• Any increase in dbproxy_forced_shutdowns_total: drain deadline too short or sessions stuck in long operations.
• Sustained client I/O timeouts: slow or abandoned clients, packet loss, or an undersized downstream consumer.
• dbproxy_postgres_backend_transport_healthy{role="primary"} == 0: the PostgreSQL/TimescaleDB primary cannot accept TCP connections.
• Sustained dbproxy_postgres_pool_waiters > 0 or increasing dbproxy_postgres_pool_acquire_timeouts_total: a transaction pool is saturated or its backend is slow to connect.
• Any increase in dbproxy_postgres_pool_reset_failures_total: an unsafe connection was discarded because its state could not be reset.
• dbproxy_postgres_backend_healthy{role="primary"} == 0 while transport is healthy: authenticated SQL readiness, credentials, or database selection is failing.
• Rising dbproxy_postgres_replica_fallbacks_total: unavailable replicas or an incorrect read-only topology.
• dbproxy_redis_backend_transport_healthy{role="primary"} == 0: the Redis primary cannot accept TCP connections.
• Rising dbproxy_redis_replica_fallbacks_total: unavailable Redis replicas or an incorrect read-only topology.
• Any dbproxy_redis_sentinel_refresh_failures_total increase: Sentinel endpoints did not agree at quorum; readiness is withdrawn.
• Any dbproxy_redis_cluster_refresh_failures_total or dbproxy_redis_cluster_redirect_failures_total increase: the native slot map or a redirection could not be proven safe. When MOVED responses rise without a successful refresh, check for planned resharding and unreachable announced nodes.
• Any sustained increase in dbproxy_postgres_shard_routing_failures_total or dbproxy_redis_shard_routing_failures_total: missing client initialization, invalid range values, or a configuration/client shard-map mismatch.
• Any increase in dbproxy_redis_partial_write_failures_total: a non-atomic cross-shard write may have changed only a subset of target shards; reconcile from the application's source of truth.
• Rising dbproxy_redis_command_routing_failures_total: unsupported command shapes, cross-shard operations disabled by policy, or incorrect key counts.
• Rising dbproxy_redis_read_only_rejections_total: clients are sending writes or non-whitelisted commands to a command-mode read-only listener.
• Any increase in dbproxy_cache_invalidation_failures_total: Redis/Valkey did not advance a committed table/shard generation; stale entries remain bounded by TTL, but investigate cache availability immediately.

Tune thresholds against normal traffic rather than alerting on raw cumulative counters.

Performance tuning

• Run cargo bench --bench routing after parser, classifier, or sharding changes.
• Keep write_fanout.max_concurrency below the connection capacity available on every target shard. It bounds concurrency per client operation, not globally.
• postgres.relay_buffer_bytes and redis.relay_buffer_bytes allocate two buffers per active transparent connection. The accepted range is 4 KiB through 1 MiB and the default is 32 KiB.
• Use streaming MySQL results where bounded memory is more important than returning a clean error before any rows have been emitted.
• Treat a sustained increase in SQL-analysis duration as a signal to inspect statement complexity, deny-rule count, and CPU saturation.