Observability

The canonical Observe interface, configured monitor credentials and the monitoring starter require v0.3.2 or later. Earlier releases expose only the legacy metrics described under compatibility.

MQLite exposes one canonical observation through AdminService/Observe, the Go SDK, CLI, MCP, console and Prometheus /metrics. Queue snapshots and committed message effects originate in the engine. HTTP request measurements originate in the server; pure embedded use marks that domain not_applicable. Grafana derives rates, time windows and percentiles from these values instead of maintaining another set of message counters.

Prometheus scraping is disabled by default. Enable authenticated /metrics on the existing API port with MQLITE_METRICS=on or mqlite serve --metrics:

MQLITE_DB=file:/data/mq.db \
MQLITE_TOKENS="$ADMIN_TOKEN" \
MQLITE_MONITOR_TOKENS="$MONITOR_TOKEN" \
MQLITE_METRICS=on \
mqlite serve --addr 127.0.0.1:6754

This local example exposes both the API and metrics on loopback. The exporter shares the broker's process, engine and counters; it needs no extra port or VM. The Go SDK uses WithMetrics(true) alongside WithTokens(...) and optionally WithMonitorTokens(...). Enabling metrics without administrator auth is rejected.

When enabled, GET / automatically includes "metrics":"/metrics"; clients resolve this against the same origin. When disabled, discovery omits the field and /metrics returns 404 even with valid credentials.

In production, use a private broker address or an ingress that blocks /metrics from the public Internet while allowing private collectors. API and metrics share a port, so a port-level firewall cannot separate them. Bearer authentication protects the data but does not hide a publicly reachable route. The monitor credential remains required on private networks.

For a runnable broker, Prometheus and Grafana stack, see the local observability demo. For an existing platform, use the cloud and Kubernetes guide.

Maintenance failure diagnosis

Brokers running v0.3.3 or later add optional maintenance[].diagnostics and storage.operations[].last_error fields to Observe. Older snapshots remain valid; missing diagnostics mean not reported, not healthy. Existing Prometheus labels and cumulative error buckets are unchanged.

Each maintenance task reports its latest outcome (never, ok, failed, or interrupted), last run time, consecutive failed passes, recovery time, and the latest eight failed steps. One pass can fail at several steps but increments the failure counter only once. Canceled/closed operations remain interruptions. A successful pass clears the consecutive streak and records recovery; it does not clear cumulative failures or history. All snapshot counters/history reset when the broker process restarts. A failed poll makes the console's previous data stale.

Error details include a timestamp, generated explanation, category, HTTP status or SQLite code where available, and an opaque fingerprint. Categories distinguish provider authentication, rate limits, provider availability, DNS, TLS, timeouts, protocol, constraints, schema, and storage failures. These descriptions summarize the observed error; an HTTP 503 alone cannot establish the provider's root cause. Unknown commit outcomes remain explicitly ambiguous and must be reconciled before retrying. Diagnostic changes do not broaden write retries.

A storage bucket's count covers all occurrences in that legacy bucket; last_error describes only its latest failure. Neither a historical count nor a successful maintenance pass proves message integrity or high availability. Empty queues still run background database maintenance, so failures can occur while no clients are sending or receiving.

Private diagnostic log

Structured stderr events maintenance.failure, maintenance.interrupted, maintenance.recovered, storage.interrupted, and storage.failure record the task/step or operation. Failure events include the same fingerprint as Observe and a bounded provider explanation with common credentials and URLs redacted. Public monitor responses contain generated explanations only, never the raw provider/SQL error. Private logs may still contain sensitive database error context; keep them operator-only.

To retain evidence beyond the platform's stderr retention, enable a separate file:

MQLITE_DIAGNOSTIC_LOG=/data/mqlite-diagnostics.jsonl mqlite serve
# Equivalent: mqlite serve --diagnostic-log /data/mqlite-diagnostics.jsonl

The parent directory must exist. The file is disabled by default; an invalid configured path fails startup. It appends across restarts, uses mode 0600, syncs each record, and rotates at 4 MiB with three archives (.1 through .3, newest first): at most 16 MiB of retained diagnostic records, not a time-based retention promise. Routine requests and successful maintenance passes are excluded. A successful recovery is logged once per failure streak. A disk write failure still emits the original event to stderr plus one warning per logging failure streak.

Mount the file on persistent storage to survive machine replacement/deployment; Fly's ephemeral root filesystem is insufficient. Export archives elsewhere if evidence must survive volume loss or sustained errors filling the rotation budget. The file is independent of the queue database, and is not replayed into Observe history after restart. It is diagnostic evidence, not a message backup or failover mechanism. Embedded applications can supply their own logger with WithLogger.

Read-only monitoring access

Configure a separate scraper credential in MQLITE_MONITOR_TOKENS, alongside configured administrator credentials in MQLITE_TOKENS. Monitoring credentials can read /metrics and AdminService/Observe. They cannot send, receive, peek at message bodies, manage queues, inspect administrative configuration or issue keys. Administrator credentials and managed manage keys retain monitoring access.

Configured monitor credentials do not query the database during authentication, so they can observe storage failure. Managed keys still require their current revocation and expiry state to be checked; no monitoring cache bypasses that check. Do not reuse an administrator token as a monitor token. Configuration changes require a broker restart; the demo's startup wrapper reads separate secret files.

A Prometheus job should reference a mounted secret file:

scrape_configs:
  - job_name: mqlite
    # The native listener is plain HTTP. Use https only when a TLS proxy
    # terminates TLS in front of the private broker.
    scheme: http
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    authorization:
      type: Bearer
      credentials_file: /run/secrets/mqlite-monitor.token
    static_configs:
      - targets: [mqlite.internal.example:6754]

If a TLS proxy fronts the listener, use the correct TLS CA rather than disabling certificate checks. Grafana connects to Prometheus and does not need the MQLite token. See the Prometheus authorization reference.

Availability and freshness

The native snapshot includes sampled_at_ms, a collection status, last successful collection time and elapsed collection duration. A failed queue read is unavailable, not an empty list of healthy queues. Prometheus responds with available in-process measurements and mqlite_collection_success 0, omitting queue gauges that could not be collected. Failure before the endpoint, including invalid scrape credentials, causes Prometheus up to become zero instead.

Queue counts are actual retained rows from a grouped database query. They describe stored states, not guaranteed immediate delivery eligibility: visibility, TTL, lease expiry awaiting maintenance and ordering rules can affect claims. Different measurement domains are sampled at different instants. The observation is not an atomic transaction encompassing every database, HTTP and in-process counter.

/healthz is open process liveness. mqlite_storage_read_available reports a real storage read probe; success does not prove writes, end-to-end delivery or data integrity. Failed read latency is absent, not zero. Local storage size is unavailable for memory/remote backends, not a fabricated zero-byte database.

Collection cost and interval

Each observation collects the full queue inventory with one grouped message scan and a storage read probe. The scan includes retained rows, so its cost grows with backlog as well as queue count. A local SQLite database uses the same single-writer connection for application work and collection; monitoring is not free or isolated from message-operation latency.

Start production scraping at 15 seconds, measure collection duration and actual send/receive latency under the expected retained backlog, then adjust the scrape and native/console polling intervals. Each native observation or Prometheus scrape collects its own snapshot. Grafana dashboard refreshes query Prometheus instead of adding direct broker scans. The local demonstration uses a faster interval to make transitions visible; choose production cadence from measured workload needs.

RPC/result and storage/error dimensions have fixed vocabularies, but their zero-initialized counters and histogram buckets still produce many series. Queue dimensions grow with the configured inventory. Measure scrape response size, duration and series count before increasing inventory or polling frequency; a working small demo does not establish a large-deployment capacity limit.

Metric contract

Prometheus uses text exposition 0.0.4. Labels are bounded categories except for configured entity names. Credentials, key IDs, message IDs, payloads, expressions, raw URLs and free-text errors are never labels. Durations use seconds; timestamps use Unix seconds in Prometheus and epoch milliseconds in native JSON.

Histograms serialize finite cumulative _bucket samples, a +Inf bucket, _count and _sum. Whole-request duration currently supplies a sum and request count, not a histogram: their ratio is a mean, never a percentile.

Storage operations are read, write and transaction; outcomes are ok, error, rejected and outcome_unknown. rejected with error code application means the operation was rejected by application/transaction callback logic; it is not evidence of a database outage. Counts represent logical engine storage-wrapper calls, not individual messages or every SQL statement inside a transaction. Retries are counted separately. Internal maintenance and observation reads are included; collecting a snapshot itself contributes read/probe operations. Error categories are empty on success or one of canceled, closed, busy, connection, full, corrupt, io, application, outcome_unknown, other. Fixed storage outcome/error combinations and registered RPC/result combinations are initialized to zero, so the first event can be detected after an earlier successful scrape. Use rate/increase across observed samples; a first scrape after an event cannot reconstruct its time. Maintenance tasks are locks, scheduled, ttl, dedup, receipts, retention, reclaim. Check enabled status and each task's actual cadence before alerting on its last-success time. Canceled passes during normal shutdown are interrupted, not successful passes or maintenance errors; they do not advance the last-success timestamp. Filter counters cover compilation/evaluation failures while routing published messages. Invalid Subscribe configuration is an RPC invalid_argument result; deliberate TestFilter validation is not a routing failure.

Message effects

Counters increase for confirmed effects, after transaction commit. A rollback, transaction retry or exact receive/settlement replay does not create a second committed effect. An uncertain remote outcome is not classified as a known rollback. Counters are process-local and reset at restart; unobserved activity before a crash can be lost. They are operational measurements, not a durable billing or business audit ledger. Custom raw SQL through EngineTx.SQL is not attributed to message-effect events. Queue gauges still report the actual retained database rows, including unexpected states; do not use event counters to reconstruct changes made outside the normal message operations.

Some events overlap. For example, rejection can also enter the DLQ; do not sum all events into a message ledger. A claim does not prove a response reached the client, and completion means the consumer reported success, not independently verified business-side processing. Broker redelivery, client HTTP retry and storage retry are separate concepts.

Compatibility from v0.3.1

Prometheus scraping is now disabled by default. On upgrade, enable MQLITE_METRICS=on or --metrics to keep scraping the same API port, and configure a monitor token for the collector. All five pre-existing families remain available when enabled. These compatibility views read canonical values; they do not maintain separate counters:

The ambiguous queue_total and millisecond age names are deprecated for new consumers; use the canonical names. They are retained until the next minor version (v0.4.0), when any removal requires an explicit migration. No independent lifetime completion counter is introduced.

Dashboard and useful queries

The provisioned dashboard offers instance/queue selection, traffic, backlog, request/authentication results, storage, maintenance and active alerts. Empty samples remain “No data” or “Not available”; they are not replaced with green zeros.

# Stored queue states, not immediate claim eligibility
mqlite_queue_messages{state="active"}

# Observed completion throughput; rate handles observed counter resets
sum by (instance, queue) (rate(mqlite_message_events_total{event="completed"}[5m]))

# Handler p95; Receive includes intentional long-poll waiting
histogram_quantile(0.95, sum by (instance, rpc, le) (
  rate(mqlite_rpc_duration_seconds_bucket[5m])))

# Whole-request arithmetic mean, including authentication
sum by (instance, rpc) (rate(mqlite_rpc_request_duration_seconds_total[5m]))
/
sum by (instance, rpc) (rate(mqlite_rpc_requests_total[5m]))

# Newly observed dead-letter transitions, not net DLQ depth change
increase(mqlite_message_events_total{event="dead_lettered"}[5m])

Apply rate/increase to counters, not queue depth gauges. delta on depth is net retained-state change and cannot measure failure throughput. Do not subtract sends and receives to infer loss, count locked rows as live consumers, or sum the same database snapshot from duplicate collectors. Requests may contain batches. Alert thresholds should match the application's SLA and minimum request volume.

Runbooks

Scrape or collection failure

Inspect the Prometheus target's lastError first. A down target can mean a stopped broker, route/TLS failure, invalid scraper credentials or an endpoint failure. With a working configured monitor credential, up=1 and collection_success=0 means metrics are reachable but the queue snapshot failed. Check bounded storage errors and broker logs; do not treat omitted gauges as zero. Preserve evidence before intervention. After recovery, verify a fresh successful snapshot and a controlled send/receive/complete canary.

Queue backlog and dead letters

Compare oldest active/locked age, depth, delivery and completion trends with the application's intended schedule. Check consumers, lease duration and ordering. Scheduled/deferred work may be intentional. Inspect dead-letter reasons and fix the failure before redrive. Retention, purge and redrive can reduce depth while new failures continue. Unexpected retained states require investigation, not deletion just to silence the graph. Filter failure is distinct from a normal non-match.

Authentication and integration

Use aggregate outcomes to distinguish missing/invalid credentials, revoked or expired keys and insufficient permissions. Client connection failures that never reach MQLite need client telemetry. Grafana datasource failure is a different hop from Prometheus scrape failure. Rotate the configured monitor token by allowing both old/new distinct monitoring tokens temporarily, restarting the broker, updating the scraper file, confirming at least two successful scrapes, then removing the old token and restarting. Never grant the scraper send/manage privileges to fix a routing error.

Storage and maintenance

Check the read probe, operation outcome, error category, retry count and pool waiting. outcome_unknown requires idempotent reconciliation; a blind resend can produce duplicates. A healthy read does not prove write availability. Check maintenance enabled status and cadence before declaring a stalled task. Reuse host/container monitoring for free disk, CPU, OOM and volume health. For disk-full, restore or integrity incidents, follow operations.md.

Logs, alerts and validation

Request logs retain operation/status context, queue and item counts; empty Receive responses are debug noise by default. Logs can help diagnose a slow nonempty request without placing payloads, credentials or unbounded errors into metric labels. Normal long polling can raise Receive latency without a storage problem.

The example rules cover sustained scrape, collection/storage failure, stale collection, unexpected states, backlog age, retained DLQ, repeated auth failures, uncertain outcomes, maintenance and filter failures. Tune them before production. Prometheus evaluates conditions; configure Alertmanager or the platform's notification receiver separately. The local demo's short MQLiteDemoDeadLetter rule is explicitly a test rule, not production policy.

The real stack verifier checks authenticated scraping, monitor isolation, canonical/native agreement, actual message identity and body, replay, redelivery, scheduled/deferred work, DLQ firing/resolution, TTL discard/dead-letter handling, retention deletion, configuration/routing filter errors, credential outcomes, Grafana provisioning and every panel query. Deliberate invalid queries and missing datasources verify that the checker rejects failures. --faults also interrupts only that isolated stack to verify scrape-auth recovery, Grafana connection failure/recovery and persistence/counter reset after broker restart. Managed-cloud/Kubernetes deployment validation remains environment-specific.

DeleteQueue is an additive RPC observation extension. New brokers include its request counters and the queue_not_empty and queue_changed outcomes. The required decoder baseline is unchanged so new clients continue to accept observations from older brokers. An occupied-queue rejection is an application outcome, not a storage failure.