mqlite main — full documentation bundle. Source version: v0.3.3. Latest recorded release: v0.3.3. These docs track main; consult release notes for published artifacts and upgrade requirements. Source of truth: https://github.com/mqlitehq/mqlite/tree/main/docs # Core concepts & subscription filters > Goal: explain MQLite's model once, clearly, and place it against the comparable > services from Azure Service Bus, AWS (SQS + SNS) and GCP Pub/Sub. MQLite is designed > from first principles (one SQLite file, embeddable, agent-friendly), but conceptually > it sits closest to Azure Service Bus, so ASB is the primary reference frame and the > differences are called out where they matter. Sections 1–5 are the model; sections > 6–11 are the **subscription-filter language** (`expr`). --- ## 1. The whole picture in one diagram ``` ┌──────────────────────────── two delivery targets ─────────────────────────┐ │ │ │ A) send straight to a Queue B) publish to a Topic, fan out │ │ │ │ producer producer │ │ │ send "orders" │ publish "events" │ │ ▼ ▼ │ │ ┌────────────┐ ┌──── topic "events" ────┐ │ │ │ queue │ │ (just a set of subs; │ │ │ │ orders │ │ stores nothing itself)│ │ │ │ ┌────────┐ │ └─┬─────────┬──────────┬─┘ │ │ │ │ active │ │ filter match│ match │ no match│ │ │ │ ├────────┤ │ ▼ ▼ ✗ (no copy) │ │ │ │ locked │ │ sub "audit" sub "billing" │ │ │ ├────────┤ │ ┌────────┐ ┌────────┐ │ │ │ │ DLQ │ │ │backing │ │backing │ ← each subscription │ │ │ └────────┘ │ │ queue │ │ queue │ is an independent │ │ └────────────┘ └────────┘ └────────┘ queue (+ own DLQ) │ │ consumer consumer consumer │ └────────────────────────────────────────────────────────────────────────────┘ ``` One sentence to remember: **a Topic stores nothing — it is only a routing rule that copies a message into a number of subscription queues. The thing that actually stores messages, can be consumed, and can dead-letter is always a Queue** (including the backing queue behind a subscription). --- ## 2. The five concepts ### Queue A named message container, and the only entity that actually stores messages. A message runs this state machine — a row is in exactly one state at a time; below `count` = `delivery_count`, `max` = the queue's `max_delivery_count`, and `✗ removed` means the row is deleted from the database: ``` ┌───────────┐ Send(scheduled) │ scheduled │ scheduler, when due ─► active ───────────────► └─────┬─────┘ cancel ─► ✗ removed (never-delivered only) Abandon(delay_ms>0) ───┘ (backoff parking: holds its group on ordered queues) │ ▼ Send ┌──────────┐ ◄── requeue: immediate abandon / lock expiry, count < max ───────────────► │ active │ ◄── redrive: from dead_lettered (count → 0) └────┬─────┘ │ receive / claim (count++) ▼ ┌──────────┐ ──── defer ──────► ┌──────────┐ │ locked │ ◄─ ReceiveDeferred │ deferred │ └────┬─────┘ (by seq) └──────────┘ │ complete ─► ✗ removed │ renew ─► (stays locked) │ reject, or abandon / lock-expiry, count ≥ max: ▼ ┌───────────────┐ ◄── TTL expiry (active/locked/deferred/scheduled; │ dead_lettered │ ✗ removed if dead_letter_on_expire = 0) └───────┬───────┘ │ purge / DLQ retention ─► ✗ removed ``` Every transition, with its trigger and the condition under which it fires: | From | Trigger | To | When | |---|---|---|---| | _(new)_ | `Send` | active | delivered immediately | | _(new)_ | `Send` scheduled | scheduled | enqueue time is in the future | | scheduled | scheduler loop | active | enqueue time reached | | scheduled | `Cancel` | ✗ removed | cancel before it activates; never-delivered rows only (`delivery_count=0`) | | active | `Receive` / claim | locked | `count`++; lock acquired | | locked | `Complete` | ✗ removed | matching `lock_token` and unexpired lease | | locked | `Abandon` | active | `count < max`, no backoff delay (immediate redelivery) | | locked | `Abandon` | scheduled | `count < max` with `delay_ms > 0`: backoff parking — auto-reactivated when due, holds its group on ordered queues | | locked | `Abandon` | dead_lettered | `count ≥ max` | | locked | lock expiry (reaper) | active | `count < max`; auto-redelivered | | locked | lock expiry (reaper) | dead_lettered | `count ≥ max` (`MaxDeliveryCountExceeded`) | | locked | `Reject` | dead_lettered | explicit, with a reason | | locked | `Defer` | deferred | set aside; fetched later by seq | | locked | `Renew` | locked | extends the lock lease | | deferred | `ReceiveDeferred` | locked | fetched by seq; `count`++ | | active / locked / deferred / scheduled | TTL expiry | dead_lettered | queue has `dead_letter_on_expire=1` (`TTLExpired`) | | active / locked / deferred / scheduled | TTL expiry | ✗ removed | queue has `dead_letter_on_expire=0` (discard) | | dead_lettered | `Redrive` | active | back to this (or a target) queue; `count` reset to 0 | | dead_lettered | `Purge` | ✗ removed | manual deletion | | dead_lettered | DLQ retention | ✗ removed | background janitor bounds (age / count / bytes); touches **only** dead-letter rows | - **The DLQ is not a separate queue.** It is messages in the same queue with `state='dead_lettered'`. A message lands there when its `delivery_count` reaches `max_delivery_count` (i.e. `≥`), on an explicit `reject`, or on TTL expiry (when the queue's `dead_letter_on_expire=1`). Inspect with `peek state=dead_lettered`, send back with `redrive`, or delete with `purge`. - **Lease deadlines fence every settlement and renewal.** A new effect requires the current token and `locked_until_ms > now`; equality is expired, even before the reaper runs. Renew before that deadline. A live receipt can replay an already committed settlement only for the same queue, sequence, token, operation and effect-bearing arguments, including after the original lease expires. - **Two redelivery paths.** `Abandon` is an explicit, client-driven settlement; **lock expiry** is automatic — the reaper (~1s) reclaims a lock held past `lock_duration` without settling. Both redeliver while `count < max` and dead-letter once `count ≥ max`. - **Ordering survives lock expiry.** On ordered paths (`group_fifo`, `strict_fifo`, and grouped messages on a `standard` queue — they share the same claim rule) an expired-but-not-yet-reaped lock still blocks its group (strict: the whole queue): successors are never delivered ahead of the expired head. Once the reaper runs, the head is redelivered first, in id order — or dead-lettered there and then if it has exhausted `max_delivery_count`, which also unblocks the group. The cost is that a consumer timeout stalls its group for up to one reaper interval (~1s) — the same trade SQS FIFO makes. A slow-but-alive consumer should `Renew` its lock rather than let it lapse. An explicit `Abandon` releases the head immediately; with a `delay_ms` backoff the head parks in `scheduled` until the backoff lapses and — like a scheduled send — keeps blocking its group (strict: the whole queue) until it is redelivered first, so order holds across a backoff (MQLITE-66). - You can have any number of independent plain queues. ### Topic A **logical name** standing for "the set of subscriptions attached to it". - A topic is **not an explicitly created/stored object** — its existence is exactly "there are rows in `subscriptions` with `topic=`". **A topic with zero subscriptions does not exist.** - A topic **holds no messages of its own**: no inbox, no buffer, no retention. ### Subscription The relationship binding a Topic to a backing queue, with an optional Filter. - Each subscription has its **own independent backing queue** (own consumption, own `seq_number`s, own DLQ). - A subscription belongs to exactly **one** topic (guarded by `ErrNameConflict`). - "A topic has many queues" is, precisely: **each subscription owns one queue, and the topic fans out through those subscriptions.** ### Filter A **single expr-lang boolean expression** on a subscription, evaluated per message at publish. - Empty expression = match everything. - e.g. `subject_parts[0] == "orders" && properties["tier"] == "gold"`. - Compiled and type-checked at `Subscribe`; a bad one is rejected (`ErrInvalidFilter` → 400). See §6–11 below for the full filter language and expression environment. ### Provisioning — what you create vs. what is created for you | Entity | How it comes to exist | Primitive | |---|---|---| | Plain Queue | **You create it explicitly**; `Send` to a non-existent queue errors (no auto-create) | `CreateQueue(name)` | | Topic | **Implicit** — no create call; springs into existence with its first subscription, gone when the last is removed | *(no `CreateTopic`)* | | Subscription **+ its backing queue** | **Created together in one call** — `Subscribe` provisions the backing queue for you | `Subscribe(topic, name, filter)` | - There is **no "create topic" operation** and **no same-named queue behind a topic**. The backing queue is named after the **subscription**, not the topic. - You never pre-create a queue to attach a subscription to — one `Subscribe` call builds the backing queue and the fan-out mapping. - **One flat namespace, disjoint by construction.** Plain-queue names, subscription names (= their backing-queue names) and topic names never overlap; conflicts fail loud with `ErrNameConflict` (HTTP 409) at creation time, in both directions: `Subscribe` rejects a **topic** that names an existing queue or subscription, a **subscription name** that belongs to a plain queue, to another topic's subscription, or to a live topic, and the degenerate `topic == name`; `CreateQueue` rejects a name that is live as a topic, and a **cross-kind upsert** (a plain `CreateQueue` over a subscription's backing queue, or `kind=subscription` over a plain queue) — a deliberate same-kind reconfig stays open. (ASB behaves the same way: queues and topics share one entity namespace, so a same-name create conflicts instead of silently rerouting sends.) The guards apply to creations from this version on; a pre-existing colliding pair in an old DB keeps resolving topic-first and now fails loud on any attempt to re-create either side. - Name resolution at publish/send: a name with subscription rows → **topic** (fan-out); otherwise it must be an **existing queue**; otherwise the call errors. Because the namespace is disjoint, this resolution is **unambiguous** — a name can never be a topic and a queue at the same time, so a `Send` is never silently rerouted. ``` You only ever do two kinds of "create": CreateQueue(name) -> one independent work queue (must exist before Send) Subscribe(topic, sub, filter) -> auto-creates backing queue=sub + attaches to topic (the topic comes into being as a side effect) ``` > Difference from ASB: in ASB both the Topic and the Subscription are created > explicitly; MQLite makes "create topic" implicit and "create the subscription's > queue" automatic — two fewer steps. ### Message ID vs Sequence Number (the two ids people conflate) | | Who assigns it | When fanned out to N subs | Purpose | |---|---|---|---| | `message_id` | optionally supplied by the app | **identical in every copy** | idempotency/dedup key (dedup is per-queue: `(queue, message_id)`) | | `seq_number` | assigned by the broker (globally monotonic) | **different in every copy** | the handle to locate + settle on consume; the ASB SequenceNumber analogue | --- ## 3. Direct answers to common questions **Q: A message fans out to two subscriptions — are the IDs the same?** The `message_id` is **the same** (copied verbatim); the `seq_number` is **different** (the broker allocates each copy a distinct id from one database-wide monotonic sequence). The two are fully independent rows, settled separately with their own `seq_number` + `lock_token`. **Q: I add a filtered subscription under a topic — do historical messages get routed to it?** **No.** Fan-out happens **only at publish time**, against the subscriptions that exist then. There is no replay, no offset rewind. A new subscription only receives messages published **after** it was created (and matching its filter). (Unlike Kafka; same as ASB / SNS+SQS.) **Q: A topic gets a message with no subscriptions at the time — dropped? buffered? a default inbox?** **No inbox; the message is not retained.** Two cases: ① subscriptions exist but none match → the message is **silently dropped** (not an error); ② no subscription names that target at all → it falls back to "must be an existing queue", delivered if one exists, otherwise an error. **To retain a message you must have a matching subscription before it is published.** --- ## 4. Comparison with cloud services ### Concept mapping | MQLite | Azure Service Bus | AWS | GCP Pub/Sub | |---|---|---|---| | Queue | Queue | SQS Queue | (Topic + single subscription, roughly) | | Topic | Topic | SNS Topic | Topic | | Subscription | Subscription | SNS→SQS subscription | Subscription | | Filter (expr-lang) | SQL/Correlation Filter | SNS Filter Policy | Subscription Filter | | peek-lock + complete | Peek-Lock + Complete | Visibility Timeout + Delete | ack / nack | | `seq_number` | SequenceNumber | (none) | (none, only ackId) | | DLQ (sub-state) | DLQ (sub-queue) | Redrive Policy → DLQ | Dead-letter Topic | | dedup window | Duplicate Detection | SQS FIFO dedup (5min) | none native (DIY) | | group_id ordering | Session | SQS FIFO MessageGroupId | Ordering Key | ### Behavioral differences (the ones that bite) | Dimension | MQLite | ASB | SNS+SQS | GCP Pub/Sub | Kafka | |---|---|---|---|---|---| | New sub gets history | ❌ | ❌ | ❌ | ⚠️ seek + retention can replay | ✅ from offset 0 | | Topic stores messages | ❌ routing only | ❌ | SNS ❌ (SQS does) | ✅ topic retention | ✅ log is the store | | No matching subscriber | dropped | dropped | dropped | dropped | still in the log | | Fan-out copies independent | ✅ own seq+DLQ | ✅ | ✅ | ✅ | ⚠️ one log, many groups | | Deployment shape | **single SQLite file / embeddable** | managed cluster | managed | managed | cluster | | Ordering model | group_id FIFO (optional) | Session | FIFO queue | Ordering Key | per-partition | ### One-line positioning - **Like ASB**: peek-lock + DLQ + Topic/Subscription + Filter + SequenceNumber + dedup map almost one-to-one. If you know Azure Service Bus the learning curve is near zero. - **Unlike Kafka / parts of GCP Pub/Sub**: MQLite is **subscription-based, keeps no log, cannot replay**. It is a "work queue + topic fan-out", not a "message log / stream". For replay / event sourcing choose something like Kafka. - **MQLite's own trade-off**: the whole broker is a single SQLite file — embeddable in-process, zero-ops, agent-friendly (ships with an MCP server). The price is exactly the "no replay / no topic retention" simplifications above — deliberate, not missing. --- ## 5. Speedbumps (easy to trip on) - Treating a Topic as "an inbox that stores my messages" — it doesn't; subscribe first. - Assuming fan-out copies share one `seq_number` — they don't; settle each separately. - Assuming a new subscription back-fills history — it can't. - Treating the DLQ as a queue you create separately — it is a state of the same queue. - Publishing to a zero-subscription topic name expecting it to buffer — it actually looks for a same-named queue, or errors. --- # Subscription filters (expr) A **subscription filter** decides, per message, whether *this* subscription receives a copy of a published message. A filter is a single [expr-lang](https://expr-lang.org) boolean expression — for example: ``` subject_parts[0] == "orders" && properties["tier"] == "gold" ``` An **empty filter matches every message**. The expression is **type-checked and compiled once when you subscribe** (a bad one is rejected immediately with `400 invalid_argument`) and **run once per message at publish time** (the per-message gate in the topic fan-out from §1–§3): ``` Subscribe ──► compile + type-check ──► cache the program (bad expr → 400) Publish ──► for each subscription: run program(message) ──► route when true ``` The filter environment contains the message's own fields and timestamps. Use `enqueued_at` as the publish-time reference for repeatable routing decisions. The upstream `now()` builtin is also available and reads the wall clock, so an expression using it can produce different results for the same message. ## 6. Setting a filter | surface | how | |---|---| | CLI | `mqlite subscribe orders orders-gold --expr 'properties["tier"]=="gold"'` | | Go SDK | `cli.Subscribe(ctx, "orders", "orders-gold", &mqlite.Filter{Expr: "properties[\"tier\"]==\"gold\""})` | | HTTP | `POST /mqlite.v1.AdminService/Subscribe` body `{"topic":"orders","name":"orders-gold","filter":{"expr":"..."}}` | Re-subscribing with the same name and a new `expr` replaces the filter (recompiled on the next publish). Omit the filter (or use an empty `expr`) to receive everything. ## 7. The message environment The variables a filter can reference. Unknown names are a compile error (so a typo is caught at `Subscribe`, not silently at runtime). ### Core | variable | type | notes | |---|---|---| | `subject` | string | the routing label (= ASB Label) | | `properties` | map | custom string headers — `properties["k"]`, `"k" in properties` (absent key → `""`) | | `group_id` | string | ordering/session key | | `correlation_id` | string | | | `reply_to` | string | | | `message_id` | string | dedup/idempotency id | | `content_type` | string | e.g. `application/json` | ### Time Both are the message's own timestamps (epoch-derived `time.Time`, UTC), not a wall-clock read. | variable | type | value | |---|---|---| | `enqueued_at` | time | when the message was published — and since fan-out runs at publish, this *is* "now" | | `visible_at` | time | when it becomes deliverable: equal to `enqueued_at` for an immediate send, the scheduled time for a delayed/`Schedule`d one (never null) | Compute a delay by subtraction (`time - time` → duration): ``` visible_at - enqueued_at > days(1) # only route significantly-delayed messages enqueued_at.Hour() >= 9 && enqueued_at.Hour() <= 21 # business-hours publish window ``` ### Derived Computed from the message; always defined (cannot error on absence). | variable | type | example | |---|---|---| | `subject_parts` | []string | `"orders.eu.new"` → `["orders","eu","new"]` — MQTT-style hierarchy: `subject_parts[0] == "orders"` | | `body_size` | int | byte length — `body_size < 4096` (route by size, not content) | | `property_keys` | []string | sorted property names — `len(property_keys) > 0`, `"tier" in property_keys` | ### Body content Route on the payload itself. These are **projected only when your filter references them**, so filters that don't touch the body pay nothing. | variable | type | example | |---|---|---| | `body_text` | string | the raw body as text — `body_text contains "urgent"` (always defined; `""` for an empty body) | | `body_json` | map | the body decoded as a JSON object — `body_json.amount > 100`, `body_json["tier"] == "gold"`, `"k" in body_json` | `body_json` is **only decoded when `content_type` looks like JSON** (or is unset); an explicit non-JSON type, an empty body, invalid JSON, or a non-object JSON (array/scalar) all yield an empty object `{}`. So `body_json` itself is never null — but reaching into an **absent field** (`body_json.amount` when there is no `amount`) yields `nil`, and comparing that (`nil > 100`) is a runtime error → **fail-closed** (the message isn't routed to that subscription, logged). Guard with a presence check when the field may be missing: ``` "amount" in body_json && body_json.amount > 100 ``` ## 8. Language Standard expr operators and builtins are available: - comparison `== != < <= > >=`, boolean `&& || !` (or `and`/`or`/`not`), membership `in` - strings: `startsWith`, `endsWith`, `contains`, `matches` (regex) - collections: `len()`, `all()`, `any()`, `none()`, `filter()`, `map()`, indexing `x[0]` ### Durations For the time fields, durations are spelled with **type-checked helpers** (recommended, unambiguous) or an extended `duration()`: | form | meaning | |---|---| | `seconds(n)` `minutes(n)` `hours(n)` `days(n)` `weeks(n)` | a duration; `days`/`weeks` are fixed (24h / 7d) | | `duration("90m")` `duration("1d12h")` `duration("2w")` | string form; Go's units **plus** `d` (=24h) and `w` (=7d) | There is no month/year unit — calendars make them ambiguous as fixed spans. For "older than a month", use `days(30)`. ``` visible_at - enqueued_at == hours(2) visible_at - enqueued_at > duration("1d") ``` ### expr-lang reference The filter engine **is** [`expr-lang/expr`](https://github.com/expr-lang/expr) (pinned at **v1.17.8**), so its full language — every operator, literal form, and built-in — is available and documented upstream. The **message environment** in §7, the duration helpers above and the fail-closed boolean contract in §10 are MQLite-specific; everything else is stock expr. When in doubt about a builtin's exact behavior, the upstream reference is authoritative. | What | Where | |---|---| | Source / project | | | Language reference (canonical) | | | Operators | | | String builtins | | | Array/collection builtins | | | Date/duration builtins | | | Number builtins | | | Type-conversion builtins | | | Docs home | | > The evaluator exposes no file or network I/O APIs and is sandboxed > (memory-safe, side-effect-free, always-terminating; see §10). This does not remove > the upstream `now()` builtin: prefer message-derived timestamps when routing > decisions must be repeatable. The duration helpers > `days()`/`weeks()`/`duration("…d…w")` are MQLite extensions on top of expr's date > functions, not part of stock expr. ## 9. Filter examples ``` subject_parts[0] == "orders" # topic hierarchy properties["tier"] == "gold" && properties["region"] == "eu" "priority" in properties && properties["priority"] == "high" subject startsWith "payment." and not (properties["test"] == "true") body_size < 4096 # small messages only visible_at - enqueued_at > days(1) # delayed > 1 day len(subject_parts) >= 2 && subject_parts[1] == "eu" "amount" in body_json && body_json.amount > 100 # route on payload (guarded) body_text contains "urgent" ``` ## 10. Filter safety Filters evaluate the message environment and available expr builtins. No file or network I/O APIs are exposed; `now()` can read the wall clock: - expr is **memory-safe, side-effect-free, and always-terminating** by design (no file or network access, no unbounded loops). - A filter must be a **boolean** (`expr.AsBool`); a non-bool expression is rejected at `Subscribe`. - Resource bounds: the source length and compiled AST size are capped. - **Fail-closed:** if a filter errors at runtime (e.g. `subject_parts[3]` on a two-part subject) or panics, the message is **not** routed to that subscription and the error is logged — never a broker crash, never a silent match. ## 11. Further reading - **expr language** — source [`github.com/expr-lang/expr`](https://github.com/expr-lang/expr), full reference at (see §8 for the per-category builtin links). - **Why expr** — [dependencies.md](https://github.com/mqlitehq/mqlite/blob/main/docs/dependencies.md) explains why `expr-lang/expr` is a stable long-term dependency (pinned v1.17.8). - **Normative invariants** — the conformance spec ([conformance.md](https://github.com/mqlitehq/mqlite/blob/main/docs/conformance.md) §11) for the filter rules MQLite guarantees. # How mqlite works — building a reliable queue on one SQLite file > This is the teaching document: how mqlite is actually built, table by table and > decision by decision — and *why* this construction yields a queue you can trust. > Nothing here is required to use mqlite; all of it helps you predict what it will > do. Companion reading: [concepts.md](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md) (the model from the outside), > [conformance.md](https://github.com/mqlitehq/mqlite/blob/main/docs/conformance.md) (every promise below, pinned as a test). --- ## 1 · The bet: a queue is rows plus discipline Strip any message queue to its skeleton and you find three ideas: 1. a **durable list** of messages, 2. a **claim** operation that hands a message to exactly one consumer at a time, 3. **rules for failure** — what happens when that consumer never comes back. Kafka builds this on a replicated log. RabbitMQ builds it on a custom broker with its own storage engine. mqlite's bet is that for a huge class of workloads — background jobs, outboxes, fan-out, pipelines — all three ideas fit inside something you already trust: **a SQLite database with ACID transactions**. A message is a row. A state is a column. A transition is one `UPDATE` guarded by a `WHERE` clause — which under a transaction is an atomic compare-and-swap. Get the schema and the transitions right, and the reliability story stops being clever code and becomes **database invariants**. The rest of this document is those invariants, in the order they earn their keep. ## 2 · The schema: eight tables, one of which matters most Everything lives in eight `STRICT` tables (`engine/schema.go`). Seven carry the machinery; one carries your data: | Table | Job | |---|---| | `messages` | **the queue itself** — every message, in every state | | `queues` | per-queue config: lock duration, retry bound, TTL, dedup window, ordering mode, DLQ retention | | `subscriptions` | the topic fan-out roster (`topic → subscription` + optional filter) | | `dedup` | duplicate-detection window | | `settlement_receipts` | makes acknowledgements idempotent (§7) | | `receive_attempts` | makes receives idempotent (§7) | | `meta` | one row: the schema guard token (§9) | | `access_keys` | SHA-256 token digests, permission sets, expiry and revocation metadata (v0.3.1) | The heart, abridged: ```sql CREATE TABLE messages ( id INTEGER PRIMARY KEY AUTOINCREMENT, -- SQLite rowid = the sequence number queue TEXT NOT NULL REFERENCES queues(name), state TEXT NOT NULL DEFAULT 'active' CHECK (state IN ('active','locked','deferred', 'scheduled','completed','dead_lettered')), visible_at INTEGER NOT NULL DEFAULT 0, -- delivery time (epoch ms) locked_until INTEGER NOT NULL DEFAULT 0, -- lock lease expiry lock_token TEXT, -- the fencing token (§4) delivery_count INTEGER NOT NULL DEFAULT 0, expires_at INTEGER NOT NULL DEFAULT 0, -- TTL deadline; 0 = none body BLOB NOT NULL, -- ... message_id, group_id, subject, properties, dead_letter_reason ... ) STRICT; ``` Three deliberate choices hide in this table: - **`id INTEGER PRIMARY KEY AUTOINCREMENT` *is* SQLite's rowid** — strictly increasing and never reused, even after the row is deleted (it may gap). That single column gives mqlite its sequence numbers, its FIFO order, *and* its fastest possible index, for free. - **State is data, not memory.** A message being "locked" is a row value, not a broker-side object or a TCP connection. Kill the process at any instant and the queue's exact state survives in the file — recovery (§6) is a `SELECT`, not a reconstruction. - **Every timestamp is epoch milliseconds** in plain `INTEGER` columns, and the clock is injectable — which is why the failure paths in the test suite run deterministically instead of with `sleep()`. ## 3 · The single-writer model: concurrency by subtraction Here is the design decision everything else leans on. SQLite allows one writer at a time — most systems fight that; mqlite **builds on it**: - **In-process, the connection pool has size one** (`SetMaxOpenConns(1)`). `database/sql` queues every caller onto that connection, so *all writes are serialized before SQLite ever sees them*. There is no lock contention, no `SQLITE_BUSY` retry loop, no interleaving to reason about. Two goroutines cannot half-claim the same message because their statements cannot overlap. - **Across processes, an advisory lock says no.** An embedded local file also takes an OS lock on a `.lock` sidecar (keyed on the *canonical* path, so `./mq.db` and a symlinked spelling collide). A second process opening the same file fails fast with `ErrDBLocked` instead of corrupting claims. Want multiple processes? Run the broker — one broker *is* the one writer. - **The file is in WAL mode**, so at the file level readers never block the writer and never see torn state. (In embedded mode reads still funnel through the same single connection; WAL is what makes that safe and cheap.) `synchronous=NORMAL` is the default durability point; `FULL` is a knob (`MQLITE_SYNC`) when an fsync per commit is worth ~4–11× of your local throughput (on remote Turso it's a no-op — the server owns durability). - **Remote Turso/libSQL** relaxes the pool to 4 connections because the Turso primary serializes writes server-side; it is also the only path that retries transient errors — locally a failed statement never ran, so there is nothing to retry. The punchline: mqlite doesn't *implement* mutual exclusion between consumers — it **arranges to never need it**. Concurrency control by subtraction. ## 4 · Claiming: Peek-Lock is one atomic UPDATE The hot path — a consumer asking for work — is a single statement (`engine/claim.go`, abridged): ```sql UPDATE messages SET state = 'locked', locked_until = :now + :lock_duration, lock_token = :random_128_bit_token, delivery_count = delivery_count + 1 WHERE id = ( SELECT m.id FROM messages m WHERE m.queue = :queue AND m.state = 'active' AND m.visible_at <= :now AND (m.expires_at = 0 OR m.expires_at > :now) -- head-of-line probes for grouped messages (§5) go here ORDER BY m.id ASC LIMIT 1) RETURNING id, body, delivery_count, ...; ``` One statement, three jobs: 1. **Select** the eligible head (oldest `active`, already visible, not expired). 2. **Claim** it — flip to `locked`, stamp a lease and a token, count the attempt. 3. **Return** the full message to the caller. Because the single writer executes this atomically, "select then claim" can never race: whoever's UPDATE runs first gets the row; the next one gets the next row. A batch `Receive` runs this in a loop **inside one transaction** — one commit, one fsync, for the whole batch (that change alone took a 1-vCPU broker from 591 to 1,507 msg/s of sustained drain). **The lock is a lease plus a fencing token.** `locked_until` is a timestamp — not a connection, not a heartbeat — so a dead consumer's lock simply *expires*. `lock_token` is 128 random bits handed only to the claimer, and every acknowledgement must present it: ```sql DELETE FROM messages WHERE id = :seq AND queue = :q AND lock_token = :token AND state = 'locked' AND locked_until > :now; ``` The current token and an unexpired lease are both required. Equality at the deadline is already expired, including before the background reaper runs. The broker samples time after local writer admission and afresh for every settlement statement and remote retry. A stale consumer's new settlement affects zero rows and returns `lock_lost`. After redelivery, the changed token also fences the previous owner. `Renew` and `RenewBatch` require the same live lease and never revive an expired one. A live exact-request receipt can replay an already committed settlement (§7). Receive-and-delete removes the claimed row in the same transaction. It grants no lease: both its token and deadline are cleared before the response or attempt record is created. The SDK therefore returns a zero `LockedUntil` time in that mode. ## 5 · Staying fast: partial indexes and an O(n²) war story A claim must find "the oldest active message" without caring how many locked, scheduled or dead-lettered rows sit around it. The tool is SQLite's **partial indexes**: ```sql CREATE INDEX idx_msg_active ON messages(queue, id) WHERE state = 'active'; ``` Only active rows live in this index, pre-sorted by `(queue, id)` — so the claim's inner SELECT is one index seek, O(log n), even with a 200k-row backlog behind a lock. Each background sweep gets its own partial index the same way (locked by expiry, scheduled by due time, DLQ by queue, TTL by deadline). The war story that shaped this (MQLITE-22): grouped messages — on the FIFO modes, and on `standard` queues whenever a message carries a `group_id` — must also check that no *earlier* message of the same group is still in flight: a head-of-line probe. Written the obvious way — one big `NOT EXISTS (... WHERE ((b.state='locked' AND b.locked_until > :now) OR b.state IN ('deferred','scheduled')) AND b.id < m.id)` — SQLite's planner can't match the `OR`/`IN` mixture to any partial index and falls back to a **backward rowid scan**: O(n) per candidate, O(n²) to drain a blocked backlog. On a 49k-message queue that was 1.29 seconds *per claim*. The fix is mechanical once you see it — **one `EXISTS` per state**, each a single `state = '...'` equality the planner can match to a partial index: ```sql ... AND NOT ( EXISTS (SELECT 1 FROM messages b WHERE b.queue=m.queue AND b.group_id=m.group_id AND b.state='deferred' AND b.id now` (§4). The store needs an atomic compare-on-field update, or fencing collapses and a stale consumer can acknowledge someone else's delivery. - **A serialization story for the claim path.** mqlite gets atomicity by funneling all writes through one connection (§3). Your store either offers the same single-writer discipline, or true serializable transactions — snapshot isolation with write skew is *not* enough for "exactly one claimer wins". - **Auxiliary writes in the same transaction.** The idempotency receipts (§7) only work because the receipt commits *atomically with* the operation it protects. A store that can't put the receipt in the same commit turns "idempotent" into "usually idempotent". - **A tunable durability point** (§3's `NORMAL`/`FULL`) and a **multi-process guard** (§3's advisory lock) — or a documented equivalent for each. **The good news: your migration test suite already exists.** The [conformance TCK](https://github.com/mqlitehq/mqlite/blob/main/docs/conformance.md) is written against behavior, not against SQLite — a store that passes it (with the race detector on) has proven G1–G7. Two things do *not* port and must be re-invented per engine: the plan-pinning tests (§5's `EXPLAIN QUERY PLAN` assertions are SQLite's dialect — you owe your new engine an equivalent performance pin, or the O(n²) class returns silently) and the schema golden test (pin whatever your layout artifact is). **How the usual candidates grade against this bar:** - **Another embedded SQL database** — clears everything by construction; the work is porting SQL dialect and the performance pins. - **Embedded KV/LSM stores** (bbolt, Badger, Pebble) — G1/G2/G6 are fine; G3/G4 mean hand-building every index the schema got for free; and G5's second half fails: your application cannot put *its* tables inside *their* transaction, so **the outbox — the killer feature — is lost**, and on LSM engines G7's space story needs care. - **An external store** (Redis, Postgres-as-a-service, a broker) — may clear G1–G7, but fails G8, and with it the outbox and the "one file you can `cp`" operations story. At that point you don't want mqlite — a Postgres-native queue (pgmq, River) fits better. - **A self-built log/BTree** — every guarantee above lands on your desk at once, including the ones you only discover under `kill -9`. You would be rewriting the hardest 20 % of SQLite to save the easiest 80 %. That grading is why mqlite runs on SQLite today: in a weighted evaluation of six candidates it was the only one delivering all eight guarantees — and uniquely the two differentiators, the shared-transaction outbox and the Turso remote. The engine keeps the question open behind a narrow store boundary (enqueue / claim / settle / recover / in-tx), but the price of admission is this appendix, with the conformance suite as the judge. --- *Want the view from outside? [concepts.md](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md). The wire-level details? [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md). The receipts for the performance claims? [benchmark.md](https://github.com/mqlitehq/mqlite/blob/main/docs/benchmark.md).* # mqlite HTTP API reference The broker speaks **Connect-style JSON over HTTP**. Every operation is a single `POST` to `/mqlite.v1./` with a JSON body — curl-able by construction. This document is generated from the wire contract in [`wire/wire.go`](https://github.com/mqlitehq/mqlite/blob/main/wire/wire.go) (the single source of truth shared by the broker and the Go SDK, so the two can't drift) and the server's error mapping. ## Conventions - **Transport:** HTTP `POST`, `Content-Type: application/json`. Discovery and health use open `GET` endpoints; opt-in metrics uses authenticated `GET` on the same port. - **`body`** is **base64** in JSON (Go marshals `[]byte` as base64). - **Timestamps** are **epoch milliseconds** (UTC) integers. Message durations use `*_ms`; observability durations explicitly named `*_seconds` retain fractional seconds. - **Sequence numbers** (`seq_number`) are broker-assigned integers, strictly increasing and **never reused** (they may gap) — **the handle you settle with**, valid only while the message lives. You never set them. - **At-least-once:** consumers must be idempotent. A message is delivered at least once and never silently dropped (see [retention.md](https://github.com/mqlitehq/mqlite/blob/main/docs/retention.md) for what is and isn't auto-deleted). - **One settlement verb per outcome** — `Complete` / `Abandon` / `Reject` / `Defer`, no aliases. New effects require the `lock_token` from `Receive` and an unexpired lease; exact-request receipt replays are described below. ### Size limits The message **body** is capped per message, and the whole request body is capped per RPC call (so an unbounded batch can't OOM the broker before per-message checks run). There is **no enforced length limit** on the string metadata fields (`subject`, `message_id`, `group_id`, `correlation_id`, `reply_to`, `content_type`, `properties` keys/values) — they are stored as SQLite `TEXT`; keep them reasonable (they count toward the JSON request you send, not toward the body cap). | limit | default | knob | |---|---|---| | `body` size | **1 MiB** → `413 message_too_large` over it | `MQLITE_MAX_MESSAGE_BYTES` (serve) / `Options.MaxMessageBytes` (embedded) | | request body per RPC call | **32 MiB** → `413 message_too_large` | `server.MaxBodyBytes` (bodies are base64; split very large batches) | | `max_messages` per `Receive` | 256 (max) | request field | | `wait_time_ms` long-poll | 20000 (max) | request field | | `Peek` `max` | 32 default, 1000 max | request field | | filter `expr` source | capped length + AST node count | internal (see [concepts.md](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#subscription-filters-expr)) | ## The base URL Every example below is written against `$BASE_URL`, because the port is the one thing a reader cannot guess — and guessing it wrong is the whole reason the examples say it out loud now. ```bash # Talking to the broker directly. mqlite serves JSON-over-HTTP on TCP 6754 and terminates # no TLS of its own, so this is plain http. When enabled, /metrics uses the # same listener; terminate TLS in a reverse proxy when required. BASE_URL=http://127.0.0.1:6754 # Behind a reverse proxy. Public HTTPS on 443 (which the URL omits, as usual) terminates at # the proxy, which forwards to the broker's internal http://…:6754. BASE_URL=https://mq.example.com ``` 6754 is **not** a second, binary protocol — there is no binary protocol. It is the port the same JSON-over-HTTP API listens on when nothing is proxying for you. The discovery card's `endpoints` are route **paths**, deliberately relative: they resolve against the origin that served the card, so they inherit its scheme, host and port. Fetch the card from `$BASE_URL` and the paths are usable as-is. The optional `metrics` field is the same-origin path `/metrics` when enabled; it is omitted by default. Resolve it against `$BASE_URL` like the other paths. ## Auth Use one `Authorization: Bearer ` header. The broker accepts configured administrator tokens in `MQLITE_TOKENS` (comma-separated), plus managed access keys stored in its database. **Secure by default:** if `MQLITE_TOKENS` is **unset**, `mqlite serve` generates and prints an administrator token at startup. Newly generated tokens always use `mqk_` plus 64 lowercase hexadecimal characters: 256 random bits, 68 ASCII characters in total. Existing configured tokens remain valid without changing their format. Token values are case-sensitive. Set `MQLITE_TOKENS=off` to explicitly disable auth (loopback by default; a remote bind needs the explicit insecure override). In that mode, the existing queue API remains open and key-management RPCs are forbidden. An embedded `Serve` without `WithTokens` has the same explicit auth-off behavior. With auth enabled, missing, malformed, unknown, expired or revoked credentials return `401 unauthenticated`; a valid key without the required permission returns `403 permission_denied`. A key-store failure is a server error, never an auth bypass. Bare tokens, other authentication schemes and duplicate Authorization headers are rejected. ### Permissions Permissions apply to **all queues and subscriptions in this broker**. Use `send`, `listen`, both, or `manage`; `manage` includes the other two and may issue new administrator keys. This is consumption permission, not a read-only observer: receiving, settling and renewing change message state. | Required permission | Operations | |---|---| | `send` | `QueueService/Send`, `Schedule`, `Cancel` | | `listen` | `QueueService/Receive`, `ReceiveDeferred`, `Peek`, `Stats`, `Complete`, `CompleteBatch`, `Abandon`, `Reject`, `Defer`, `Renew`, `RenewBatch` | | `manage` | All `AdminService` methods, all `AuthService` methods, and `/metrics` | | configured monitor | `AdminService/Observe` and `/metrics` only | Both receive modes require `listen`, including receive-and-delete. Batch and idempotent replay paths use the same permission checks as the original operation. The console's administration views require `manage`. A configured monitor token opens only its observability views. The public static page alone grants no API access. In v0.3.2 and later, `MQLITE_MONITOR_TOKENS` configures comma-separated read-only monitoring credentials alongside administrator authentication. The embedded equivalent is `WithMonitorTokens`. Monitor and administrator credentials must be distinct; configuring monitor tokens with authentication disabled is rejected. Monitor credentials authenticate without querying the key database, so collection failures remain observable. They cannot peek at bodies, receive, send, change configuration or issue keys. They are rotated through configuration and a broker restart, not the managed-key API; `monitor` is not a managed-key permission. See [observability](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) for a ready-to-run monitoring stack. Configured administrators and managed `manage` keys have identical operation rights. Configured credentials are checked first: explicitly placing an existing managed token in `MQLITE_TOKENS` grants it administrator rights independently of its database permissions, expiry or revocation. Remove it from configuration to withdraw that static grant. Database key operations never alter environment configuration. ### Observe Available in **v0.3.2 and later**; older brokers do not implement this route. `POST /mqlite.v1.AdminService/Observe` with `{}` returns the canonical broker observation used by `/metrics`, the SDK `Observe`, CLI `observe`, MCP `observe` and console. Requires `manage` or a configured monitor credential. | Field | Meaning | |---|---| | `sampled_at_ms`, `started_at_ms`, `backend`, `version` | Queue sampling instant, engine lifetime and non-sensitive identity | | `access` | Request capability: `manage`, `monitor` or `anonymous`; SDK embedded mode uses `embedded` | | `collection` | Queue collection `state`, `last_success_at_ms`, `duration_seconds`, bounded `error_code` | | `queue_count`, `subscription_count`, `queues` | Actual retained state counts and age; counts are usable only when collection is `available` | | `runtime` | Schema version, read-probe availability/latency and local footprint availability/bytes; no DSN or path | | `messages` | Engine-lifetime `{queue,event,count}` effects, recorded after confirmed commits | | `storage` | Bounded operation outcomes, durations, retries and database connection pool measurements | | `maintenance`, `filters` | Maintenance enablement, results/durations/last success and filter failures | | `http` | Request outcomes/full elapsed time, final authentication outcomes and post-auth handler histograms | Successful authentication returns HTTP 200 even if queue collection is `unavailable`: `queues` is then `null`, with an explicit failure code and the last successful collection time. In-memory counters remain available. Never interpret absent queues or unavailable sizes/probes as zero. Authentication itself still fails closed; a managed key may be unusable while its database is unavailable. Pure embedded observations mark the HTTP domain `not_applicable`. The observation is sampled over an interval, not a transaction spanning every runtime counter. Its queue gauges share one aggregate SQL read and one engine clock value. Counters reset on engine/process restart; they are not a durable message ledger. Histograms contain cumulative finite buckets and a total count that is also the Prometheus `+Inf` bucket. The existing handler histogram excludes authentication; whole-request duration includes it and normal Receive long polls. The shared decoder bounds observations to 32 MiB and rejects incomplete objects, inconsistent availability and malformed histogram data. The complete JSON schema is represented by [`wire/observability.go`](https://github.com/mqlitehq/mqlite/blob/main/wire/observability.go) and [`engine/observability.go`](https://github.com/mqlitehq/mqlite/blob/main/engine/observability.go); metric semantics and compatibility aliases are specified in [observability](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md). ```bash curl -fsS "$BASE_URL/mqlite.v1.AdminService/Observe" \ -H "Authorization: Bearer $MQLITE_TOKEN" \ -H 'Content-Type: application/json' -d '{}' ``` ### CreateKey / ListKeys / RevokeKey These methods are available in v0.3.1 and later; v0.3.0 does not implement them. All paths are `/mqlite.v1.AuthService/` and require `manage` with authentication enabled. | Method | Request | Response | |---|---|---| | `CreateKey` | `id`, `name`, `permissions`, optional `expires_at_ms` | `{"key": , "token": "mqk_…"}` | | `ListKeys` | optional `after_id`, `limit`, `sort` | `{"keys": [], "next_after_id": "…"}`; cursor omitted at the end | | `RevokeKey` | `id` | `{"ok": true}`; already revoked records succeed again | `AccessKey` metadata contains `id`, `name`, `permissions`, `created_at_ms`, `expires_at_ms` and `revoked_at_ms`. Zero expiry means no expiry; zero revocation time means not revoked. Expired and revoked records remain visible to managers. Metadata never contains a token or its digest. Lists include only managed keys; limit defaults to 100 and cannot exceed 1000. Omitted or empty `sort` means `id_asc` (public ID ascending). Use `sort: "created_desc"` for newest first: creation time descending, then public ID descending for equal timestamps. Pass the previous page's `next_after_id` as `after_id`, retaining the same sort. For `created_desc`, the cursor must identify an existing key; an unknown cursor or unsupported sort returns `400 invalid_argument`. Revocation and expiry do not remove records or change their position. Return to the first page to see records newer than the current cursor. Pagination is not a snapshot of concurrent inserts. The `id_asc` mode also accepts a valid ID that is not present, which supports looking up a retained creation ID using its immediate predecessor and `limit: 1`. Before creation, generate and retain a public ID: exactly 32 lowercase hexadecimal characters representing 128 random bits. The SDK provides `GenerateKeyID()`; the CLI generates one if `--id` is omitted. This ID is separate from the secret and lets you identify a creation whose response was lost. Names are nonempty UTF-8 labels up to 128 bytes without leading/trailing whitespace or NUL. Names need not be unique. `permissions` accepts `send`, `listen`, or `manage`; any combination containing `manage` is normalized to `["manage"]`. Expiry is epoch milliseconds, either zero or a future instant. The server generates the secret and stores only its SHA-256 digest. Save the creation response securely: no later operation can recover the same secret. A duplicate ID returns `409 key_conflict` rather than producing or revealing a key. Do not automatically retry a creation after an ambiguous response. Reconcile the retained ID, revoke an undelivered key when confirmed, and use a new ID for its replacement. An absent row does not prove that an outstanding request cannot still commit. Never guess which same-named key to revoke. Revocation rejects subsequent authentication attempts immediately. Requests already authorized, including a long-poll Receive, may finish. Subsequent settlement and renewal still require a valid key. Issued keys are independent: revoking their creator does not revoke them. Change permissions by issuing a replacement key, moving clients, then revoking the old key. ## Open endpoints (no auth) | Method | Path | Returns | |---|---|---| | `GET` | `/` | JSON discovery card (see below) | | `GET` | `/healthz` | `200 ok` (liveness) | | `GET` | `/ui` | Embedded admin console (when enabled — see below) | ```bash curl "$BASE_URL/" # what is this? (no auth) # → {"name":"mqlite","version":"","description":"...","status":"ok", # "auth":"bearer","docs":"https://github.com/mqlitehq/mqlite", # "endpoints":["/mqlite.v1.QueueService/Send", ...every RPC route...], # "health":"/healthz","ui":"/ui"} curl "$BASE_URL/healthz" # ok ``` The card's `auth` is `"bearer"` when RPCs require a token or `"none"` when auth is off, so an agent can branch on it directly. `endpoints` is the **complete** list of RPC route paths (it always matches what the broker serves — a pinned contract). `health` is the relative liveness path. The optional `metrics` field is `"/metrics"` when `--metrics` or `MQLITE_METRICS=on` enables the exporter on the API port. It is omitted when disabled, and the route returns `404` even with valid credentials. Enabling metrics requires administrator auth; the advertised route still requires a monitor or administrator credential. No separate URL configuration is needed. `/ui` serves the **embedded admin console** (a static single-page app baked into the binary) and is auth-exempt — the page itself loads without a token; its API calls carry one you paste in. It's on by default; set `MQLITE_UI=off` to run headless, in which case `/ui` 404s and the card's optional `ui` field is omitted. ## Other endpoints | Method | Path | Auth | Returns | |---|---|---|---| | `GET` | `/metrics` (when enabled) | Bearer with `manage` or configured monitor | Prometheus text: `mqlite_queue_messages{queue,state}` gauges | ## QueueService All paths are `/mqlite.v1.QueueService/`. ### Send Enqueue one or more messages in one transaction. - **Request** `SendRequest`: `queue` (string), `messages` ([Message]), `scheduled_enqueue_time_ms` (int, optional), `ttl_ms` (int, optional). - **Response** `SendResponse`: `seq_numbers` ([int]) — one per input message, positional. - **Dedup:** in a multi-message batch, a slot that hits a dedup conflict (same `message_id`, different body) comes back as `0` (skipped) while the rest commit. A **single-message** Send instead returns `409 already_exists`. ```bash curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data "{\"queue\":\"orders\",\"messages\":[{\"body\":\"$(printf hi|base64)\"}]}" \ "$BASE_URL/mqlite.v1.QueueService/Send" ``` ### Receive Peek-Lock (default) or Receive-and-Delete; long-polls up to `wait_time_ms`. Receive-and-Delete removes the row before returning it and grants no lease: `lock_token` is empty and `locked_until_ms` is zero (omitted when empty), including replays with the same receive attempt ID. The SDK exposes a zero `LockedUntil` time. - **Request** `ReceiveRequest`: `queue`, `max_messages` (int, default 1, max 256), `wait_time_ms` (int, long-poll; max 20000), `receive_mode` (int: `0` peek-lock, `1` receive-and-delete), `receive_attempt_id` (string, optional idempotency key — a retry replays the same batch / same lock tokens). - **Response** `ReceiveResponse`: `messages` ([Message]) — each carries a `lock_token` to settle with (peek-lock mode). ```bash curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"queue":"orders","max_messages":1,"wait_time_ms":5000}' \ "$BASE_URL/mqlite.v1.QueueService/Receive" # → {"messages":[{ # "seq_number":42,"lock_token":"lt_9f3a…","delivery_count":1, # "body":"aGk=","message_id":"order-42","group_id":"cust-7", # "enqueued_at_ms":1750000000000,"locked_until_ms":1750000030000 # }]} # settle it: POST .../Complete {"queue":"orders","seq_number":42,"lock_token":"lt_9f3a…"} ``` ### Complete / Abandon / Reject / Defer / Renew All take `SettleRequest` and return `SettleResponse` `{ "ok": true }`. Settling with an expired or wrong `lock_token` → `409 lock_lost`, unless a live settlement receipt matches the exact request: `queue`, `seq_number`, `lock_token`, operation, and the arguments that change its effect (`delay_ms`, or Reject's reason/description). This replay rule applies to Complete/Abandon/Reject/Defer; Renew extends a live lease. New settlement and renewal effects require `locked_until_ms > now`: the exact deadline already counts as expired, even if the background reaper has not run. The broker samples time after local writer admission and afresh for each statement and remote retry. A live exact-request receipt can replay a previously committed settlement after its original lease deadline; it cannot authorize a different request or revive a lease. | Method | Extra fields | Effect | |---|---|---| | `Complete` | — | delete the message (done) | | `Abandon` | `delay_ms` (int, optional) | release the lock → redelivered after `delay_ms`; with a delay the head keeps holding its group's line on ordered queues | | `Reject` | `dead_letter_reason`, `dead_letter_description` (optional) | move to the DLQ | | `Defer` | — | set aside; retrieve later by seq via `ReceiveDeferred` | | `Renew` | — | extend the lock by the queue's lock duration | `SettleRequest`: `queue`, `seq_number`, `lock_token` (+ the extras above). > **Defer + ordering.** A deferred message keeps its head-of-line position: in a > `group_fifo`/`strict_fifo` (or otherwise grouped) queue the messages behind it — its > group, or the **whole queue** under `strict_fifo` — stay blocked until it is retrieved > via `ReceiveDeferred` and settled (other groups proceed). Normal `Receive` never > returns a deferred message, and with no TTL there is no auto-recovery, so recover a > lost `seq_number` via `Peek` with `state=deferred`. > **Abandon + ordering.** With `delay_ms > 0` the message parks as `scheduled` until the > backoff lapses and — like a deferred head — holds its head-of-line position on ordered > queues: its group (the whole queue under `strict_fifo`) is not delivered ahead of it. > With no delay it is redelivered immediately. ### CompleteBatch Complete many messages in **one round-trip** — settle a whole received batch without a Complete-per-message N+1 (the cheap path for draining). - **Request** `CompleteBatchRequest`: `queue`, `messages` ([{`seq_number`, `lock_token`}]). - **Response** `CompleteBatchResponse`: `results` ([{`seq_number`, `ok`}]) — `ok=false` for a stale/expired lock on that item (not fatal to the batch). Each item is fenced + idempotent exactly like `Complete`. ### RenewBatch Extend the lock lease of a whole received batch in **one round-trip**. Renewing message-by-message costs one request per message, so on a slow link a renewal pass can take **longer than the lease it is renewing** — a 64-message batch at 50 ms per call needs 3.2 s against a 2 s lock, and most of the locks expire mid-pass. - **Request** `RenewBatchRequest`: `queue`, `messages` ([{`seq_number`, `lock_token`}]). **At most 512 messages per call** — more returns `400 invalid_argument`. - **Response** `RenewBatchResponse`: `results` ([{`seq_number`, `ok`, `locked_until_ms`}]). `ok=false` for a stale/expired lock on that item (not fatal to the batch); each item is fenced on its own `lock_token`, exactly like `Renew`. `locked_until_ms` is the deadline the broker actually committed — renew again before it. And `ok=true` means the lease was live **when the broker's statement finished** — not when the answer reaches you. The result still has to be mapped, encoded and travel back, so on a very short lease it can lapse on the way. **`locked_until_ms` is the authoritative fact: compare it against your own clock.** (If the write itself outlived the lock, you get `ok=false` rather than a lock you never really held.) > **Why 512?** Renewal is a claim about the *future* ("this lease is live"), and that claim is > only keepable within a single statement: across several, the first batch's lease can expire — > and be reaped — while a later one is still running. `Receive` hands out at most 256 messages, > so a consumer never meets the limit by accident. (`CompleteBatch` has no such limit, because > completion is terminal: it reports what already happened.) ### ReceiveDeferred Lock previously-`Defer`'d messages by sequence number. - **Request** `ReceiveDeferredRequest`: `queue`, `seq_numbers` ([int]). - **Response** `ReceiveResponse`: `messages` ([Message]). ### Schedule Enqueue for future delivery (same wire shape as `Send`). - **Request** `SendRequest` with `scheduled_enqueue_time_ms` set (epoch ms). - **Response** `SendResponse`: `seq_numbers`. ### Cancel Delete a not-yet-activated scheduled message. - **Request** `CancelRequest`: `queue`, `seq_number`. **Response** `SettleResponse`. ### Peek Browse without locking or settling (triage; recover a deferred seq). - **Request** `PeekRequest`: `queue`, `from_seq` (int, optional), `state` (string, optional: `active`/`locked`/`deferred`/`scheduled`/`dead_lettered`), `max` (int, default 32, max 1000). - **Response** `PeekResponse`: `messages` ([Message], includes `state` + `dead_letter_reason`/`description`). ### Stats - **Request** `MetricsRequest`: `queue`. - **Response** `MetricsResponse`: `queue`, `active`, `locked`, `deferred`, `scheduled`, `dead_lettered`, `total`, `oldest_message_age_ms`. The age is measured from the oldest active or locked message; it is zero when neither state is present. ```bash curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"queue":"orders"}' "$BASE_URL/mqlite.v1.QueueService/Stats" # → {"queue":"orders","active":128,"locked":3,"deferred":0,"scheduled":5, # "dead_lettered":2,"total":138,"oldest_message_age_ms":41200} ``` ## AdminService All paths are `/mqlite.v1.AdminService/`. ### DeleteQueue Available in **v0.3.3 and later**. Removes an **empty plain queue**. Requires administrator/manage permission. - POST `/mqlite.v1.AdminService/DeleteQueue` - Read the queue's `generation` from `ListQueues`, then send `{"name":"orders","generation":""}`. Response: `{"deleted":true}`. Missing generation returns HTTP 400. - The generation must still match inside the deletion transaction. HTTP 409, `queue_changed`, means the name now identifies a replacement; the request did not delete it. Preserve the original generation when retrying/reconciling. - All message states block deletion, including locked, scheduled, deferred and dead-lettered messages: HTTP 409, `queue_not_empty`. Drain or settle them first. - Subscription backing queues are rejected with HTTP 400, `invalid_argument`; this operation does not remove topic subscriptions. - Missing queues return HTTP 404, `not_found`. Queue configuration and per-queue deduplication, settlement receipts and receive-attempt records are removed in the same transaction. Recreating the name starts with fresh auxiliary records; message sequence numbers are not reset. Process event counters remain historical. - Stop producers and consumers before retiring a queue. The transactional emptiness check prevents deletion of a concurrent acknowledged send. Requests still addressing a retired name fail; an existing long poll is woken. - If the result is `outcome_unknown` or the response is lost, inspect `ListQueues` before retrying. Queue names are reusable. Keep the original generation; never automatically look up a replacement generation and retry a previous deletion against it. Go SDK: `client.DeleteQueue(ctx, name)` / `embedded.DeleteQueue(ctx, name)`. These resolve the current generation once per invocation. For retained requests, pass `DeleteQueueOpts{Generation: originalGeneration}` to fence later retries. `ListQueues` adds the optional `generation` field; existing response fields and all prior RPC contracts remain unchanged. New queue generations are opaque IDs stored in the existing `meta` table. Existing queues use a stable legacy identity until deleted; reconfiguration preserves identity. No schema-version change or migration is required. The console exposes **delete queue** in a plain queue's detail page and requires an exact name confirmation. Older brokers report the operation as unsupported. ### CreateQueue Create a queue, or **update** one (idempotent **upsert** on `name`: calling it again with the same name **and the same `kind`** overwrites that queue's config — existing messages are untouched). Names are one flat namespace shared with subscriptions and topics: a `name` that is live as a **topic**, or that belongs to an entity of a **different kind** (e.g. a plain create over a subscription's backing queue), is rejected with `409 name_conflict` instead of silently taking over ([concepts → provisioning](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#provisioning--what-you-create-vs-what-is-created-for-you)). **Response** `{}` (Empty). **Request** `CreateQueueRequest`: `name` (string) + `config` (`QueueConfigJSON`, all fields optional — omit one to take its default). Every duration is **epoch milliseconds**. Config is **queue-level**: it's set here, not per `Send`/`Receive`. | Field | Type | Default | Meaning | |---|---|---|---| | `kind` | string | `queue` | `queue` (normal) or `subscription` (a topic's backing queue — normally created via `Subscribe`, not by hand). | | `lock_duration_ms` | int (ms) | `30000` (30s) | the **peek-lock window**: how long a `Receive`'d message stays invisible to other consumers before its lock expires and it's redelivered. `Renew` extends a held lock by this same amount. There is **no per-`Receive` override** — it's fixed by the queue. `0` → the 30s default. | | `max_delivery_count` | int | `10` | how many delivery attempts before a message is dead-lettered (`reason=MaxDeliveryCountExceeded`). Reached at `>=` (the Nth delivery is the last). `0` → the default of 10. | | `default_ttl_ms` | int (ms) | `0` = unlimited | per-message time-to-live applied when a `Send` doesn't set its own `ttl_ms`. On expiry the message goes to the DLQ or is discarded — see `dead_letter_on_expire`. | | `dead_letter_on_expire` | bool | `true` | on TTL expiry: `true` → move to the DLQ; `false` → discard silently. | | `dedup_window_ms` | int (ms) | `0` = **off** | duplicate-detection window. `0` disables dedup entirely (then `message_id` is just metadata). `>0` collapses repeats of the same `message_id` within the window — see [The `Message` object → `message_id`, dedup & uniqueness](#message_id-dedup--uniqueness). | | `ordering_mode` | string | `standard` | `standard` (per-`GroupID` FIFO, cross-group parallel; ungrouped messages unordered) · `group_fifo` (same, but every send **must** carry a `group_id` → else `400 group_required`) · `strict_fifo` (one global head-of-line FIFO for the whole queue). See [conformance §5](https://github.com/mqlitehq/mqlite/blob/main/docs/conformance.md#5--ordering-modes). | | `dlq_max_age_ms` | int (ms) | `0` = inherit | per-queue DLQ retention override: drop dead letters older than this. `0` inherits the broker default, `-1` = unbounded. See [retention.md](https://github.com/mqlitehq/mqlite/blob/main/docs/retention.md). | | `dlq_max_count` | int | `0` = inherit | keep at most N dead letters (oldest dropped first). `0` inherits, `-1` = unbounded. | | `dlq_max_bytes` | int | `0` = inherit | cap total dead-letter **body** bytes. `0` inherits, `-1` = unbounded. | ```bash curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"name":"orders","config":{ "lock_duration_ms":60000, "max_delivery_count":5, "dedup_window_ms":300000, "ordering_mode":"group_fifo" }}' \ "$BASE_URL/mqlite.v1.AdminService/CreateQueue" # → {} ``` ### Subscribe Register subscription `name` under `topic` (creates its backing queue). - **Request** `SubscribeRequest`: `topic`, `name`, `filter` (optional — `{"expr": ""}`, an [expr boolean over the message](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#subscription-filters-expr); empty/omitted matches all). A malformed expr is rejected with `400 invalid_argument`. **Response** `{}`. - Name conflicts are rejected with `409 name_conflict`: a `topic` that names an existing queue or subscription, a `name` that belongs to a plain queue / another topic's subscription / a live topic, and `topic == name`. Re-subscribing the same `(topic, name)` is the idempotent filter-update path and succeeds. ### ListQueues - **Request** `{}`. **Response** `ListQueuesResponse`: `queues` ([{`name`, `kind`, `lock_duration_ms`, `max_delivery_count`, `default_ttl_ms`, `dedup_window_ms`}]). ### ListSubscriptions - **Request** `{}`. **Response** `{subscriptions: [{topic, name, expr}]}` — every subscription with its topic and [filter expression](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#subscription-filters-expr) (`expr` empty = match all). `ListQueues` shows the backing queues; this exposes the topic + filter it omits. ### TestFilter - **Request** `{expr, message?}` — dry-run a [filter expression](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#subscription-filters-expr): it compiles `expr` and, if `message` (a sample, body base64) is given, evaluates it exactly as publish-time fan-out would (nothing is enqueued). - **Response** `{valid, error?, ran, matched}` — `valid` = compiled; `error` = the precise compile error or a runtime/eval error; `ran` = a sample was evaluated; `matched` = the sample would route to a subscription using this filter. ### Redrive Move dead-lettered messages back to active (or to another queue). - **Request** `RedriveRequest`: `queue`, `target` (string, optional — cross-queue), `max` (int), `older_than_ms` (int), `rate_per_sec` (int). **Response** `RedriveResponse`: `moved` (int). ### Purge Permanently delete dead-lettered messages. - **Request** `PurgeRequest`: `queue`, `max` (int), `older_than_ms` (int; both zero purges the whole DLQ). **Response** `PurgeResponse`: `purged` (int). ### Status A desensitized runtime snapshot for an ops view — never includes a connection string or auth token. - **Request** `{}`. **Response** `StatusResponse`: `version`, `backend` (`memory` | `local file` | `remote libSQL/Turso`), `remote` (bool), `location` (a local path, or a masked remote host like `libsql://***.turso.io`), `schema_version`, `ping_ms` (a `SELECT 1` read round-trip; `-1` if it failed), `db_size_bytes` (local on-disk footprint: db + WAL + shm; `0` for memory/remote), `queues`, `subscriptions`, `uptime_ms`, `auth` (bool). ## The `Message` object One shape for both send **input** and receive/peek **output** (fields are omitted when empty). "Dir" = whether *you set it on send* (**in**) or *the broker sets it on* *receive/peek* (**out**). | Field | Type | Dir | Meaning · constraints | |---|---|---|---| | `body` | base64 | in | the payload, **opaque** to the broker (it never parses it). The only size-capped field — **≤ 1 MiB** by default (`413` over it). Empty body is allowed. | | `message_id` | string | in | optional **dedup / idempotency key**. No effect unless the queue has dedup on (`dedup_window_ms > 0`); empty → the body's SHA-256 is used. Not required to be unique. **See the next section.** | | `group_id` | string | in | **ordering / session key** — messages with the same `group_id` are FIFO among themselves (= SQS MessageGroupId / ASB SessionId). **Required** on every send to a `group_fifo` queue (else `400 group_required`); `strict_fifo` is one global FIFO and does **not** need a `group_id`. Not a consumer group. | | `subject` | string | in | free-form **routing label** (= ASB Label). **Not unique**, may repeat or be empty. Split on `.` into `subject_parts` for filters (`"orders.eu.new"` → `["orders","eu","new"]`). | | `correlation_id`, `reply_to`, `content_type` | string | in | free-form ASB-style metadata; the broker stores but never interprets them (`content_type` only hints filter `body_json` decoding). | | `properties` | object | in | custom **string→string** headers, stored verbatim; visible to filters as `properties["k"]`. Values must be strings. | | `seq_number` | int | out | broker-assigned, strictly increasing, **never reused** (may gap); the handle you settle/peek with while the message lives. | | `delivery_count` | int | out | how many times this message has been delivered (claimed). | | `lock_token` | string | out | fencing token from peek-lock `Receive`; settle with it. | | `state` | string | out | `Peek` only: `active`/`locked`/`deferred`/`scheduled`/`dead_lettered`. | | `enqueued_at_ms`, `expires_at_ms`, `visible_at_ms`, `locked_until_ms` | int | out | epoch-ms timestamps. | | `dead_letter_reason`, `dead_letter_description` | string | out | populated for DLQ messages. | ### `message_id`, dedup & uniqueness The single most-asked question: **can the sender set `message_id`, and what happens on a collision?** - **Yes, the sender sets it** (it's an optional input field). It is **not** required to be unique, and the broker assigns nothing here — leave it empty and the field just stays empty (the *dedup* path, if active, falls back to the body's SHA-256). - **What the broker does with it depends entirely on the queue's `dedup_window_ms`:** | queue dedup | effect of `message_id` | |---|---| | **off** (`dedup_window_ms = 0`, the **default**) | purely informational metadata. **No uniqueness is enforced and a collision is impossible** — send the same `message_id` a thousand times and you get a thousand independent messages (each its own `seq_number`). Use it as an idempotency hint your *own* consumer checks. | | **on** (`dedup_window_ms > 0`) | within the sliding window, keyed by **`(queue, message_id)`**:
• same id **+ same body** → collapses to one enqueue (the duplicate is silently dropped; you get the original `seq_number` back).
• same id **+ different body** → a **conflict**: a single `Send` returns `409 already_exists` (`ErrDedupConflict`); in a multi-message `Send` that one slot comes back as `seq 0` (skipped) while the rest commit.
• empty id → the body's SHA-256 is the key (content-addressed dedup). | - **Dedup is per-queue.** The key is `(queue, message_id)`, so the same `message_id` in two different queues — or fanned out to two subscriptions' backing queues — never collides; each is independent. - **So "can self-specifying cause a problem?"** Only if (a) the queue has dedup **on** and (b) you reuse an id within the window **with a different body** — that's the deliberate conflict signal. With dedup off (default), or with identical bodies, there is no problem. ## Errors Errors are a JSON envelope `{"code": "...", "message": "..."}` with an HTTP status: | HTTP | `code` | When | |---|---|---| | 400 | `invalid_argument` | malformed JSON, an **unknown field**, trailing data after the object, a bad filter expression, an empty queue name, or an unknown enum (`kind` / `ordering_mode`). The decoder is strict — a typo'd field is a 400, not a silently-dropped no-op (`mqlite.ErrInvalidArgument`). | | 400 | `group_required` | a `group_fifo` send with no `group_id` | | 401 | `unauthenticated` | missing/invalid Bearer token (auth is on by default; only `MQLITE_TOKENS=off` disables it) | | 403 | `permission_denied` | a valid key lacks permission, or key management was attempted with auth disabled (`mqlite.ErrPermissionDenied`) | | 404 | `not_found` | the queue or message doesn't exist; or an unknown path | | 409 | `already_exists` | single-message dedup conflict (same id, different body) | | 409 | `name_conflict` | a queue/subscription/topic name collision at creation time — the namespace is one and disjoint (`Subscribe`/`CreateQueue`) | | 409 | `key_conflict` | a managed key ID or token digest is already registered (`mqlite.ErrKeyConflict`); no existing secret is returned | | 409 | `lock_lost` | settle with an expired or wrong `lock_token` | | 413 | `message_too_large` | body over `MaxMessageBytes` (default 1 MiB) | | 499 | `canceled` | the client canceled the request | | 503 | `outcome_unknown` | a remote (Turso/libSQL) write lost its commit acknowledgement — it **may or may not** have applied. **Do not blindly retry**; reconcile messages by `message_id`/dedup, or key operations by their retained public key ID (`mqlite.ErrOutcomeUnknown`). Only occurs against a remote store, never a local file/`:memory:`. | | 500 | `internal` | unexpected server error | The Go SDK re-exports the sentinel errors (`mqlite.ErrLockLost`, `ErrDedupConflict`, `ErrInvalidArgument`, `ErrOutcomeUnknown`, …) so `errors.Is` works in both embedded and client mode. # CLI reference (`mqlite`) One binary, **two modes** — the same commands work against a local SQLite DB (embedded) or a running broker (client), chosen by environment: - **Embedded:** set `MQLITE_DB` (e.g. `file:./mq.db`, `:memory:`, `libsql://…`). The command opens the DB in-process. - **Client:** set `MQLITE_ENDPOINT` (+ `MQLITE_TOKEN`). The command talks to that broker over HTTP. **Client mode wins if `MQLITE_ENDPOINT` is set.** ```bash mqlite [flags] [args] ``` ## Connection (environment) | Env | Meaning | |---|---| | `MQLITE_DB` | embedded DB DSN: `file:./mq.db` / `:memory:` / `libsql://.turso.io` | | `MQLITE_DB_AUTH_TOKEN` | auth token for a remote libSQL/Turso DSN | | `MQLITE_ENDPOINT` + `MQLITE_TOKEN` | client mode: a running broker + its Bearer token | | `MQLITE_MONITOR_TOKENS` | optional comma-separated read-only credentials for `observe` and `/metrics`; requires administrator auth and credentials distinct from administrators | | `MQLITE_TOKENS` | broker (`serve`) administrator Bearer tokens; **unset → a `mqk_…` token is generated + printed**, `=off` disables auth; additional managed keys live in the database | | `MQLITE_METRICS` | authenticated Prometheus `/metrics` on the API port; `on`/`true`/`1` enables, `off`/`false`/`0` disables (default); requires administrator auth | | `MQLITE_DIAGNOSTIC_LOG` | optional private rotating JSONL failure/recovery log path for `serve`; parent must exist; use persistent storage | | `MQLITE_SYNC` | `NORMAL` (default) / `FULL` / `OFF` / `EXTRA` durability (embedded/serve). An unrecognized value is **rejected at startup** — a typo never silently downgrades to `NORMAL`. | | `MQLITE_DLQ_MAX_AGE` · `MQLITE_DLQ_MAX_COUNT` · `MQLITE_DLQ_MAX_BYTES` | broker DLQ retention (`serve`); on by default, disable with `MQLITE_DLQ_RETENTION=off` | The DB string is **never compiled in** — it comes from the environment. The endpoint comes from the environment too, or from `--endpoint` (below), which overrides it. ## Global flags (data + admin commands) | Flag | Meaning | |---|---| | `--endpoint ` | client mode: a running broker (overrides `MQLITE_ENDPOINT`) | | `--token ` | Bearer token for `--endpoint` (overrides `MQLITE_TOKEN`) | | `--output text\|json` | `text` (default, human) or `json` (machine-readable, for scripting) | They apply to the data and admin commands. **`serve`, `version` and `help` take none of them** (a broker is not a client), and **`vacuum` is local-only** — it rejects `--endpoint` outright, because reclaiming pages means opening the DB file directly. An ambient `MQLITE_TOKEN` is sent only to the endpoint that environment names. If `--endpoint` points at a *different* broker, the token is withheld (with a warning) rather than leaked to another host — pass `--token` to authenticate there. `--token=` (empty) explicitly sends no credential at all. Two spellings of the same broker (a trailing slash, `mqlite://h` vs `http://h:6754`) are the same target and keep the token. ### `--output json` uses the wire's field names Every message the CLI emits in `--output json` **is** the HTTP API's message object — the same struct, so the same field names and encodings (`seq_number`, `body` base64, epoch-ms timestamps). The CLI never invents or renames a key. Two things are the CLI's own, and are *not* claims about the HTTP response: - **The envelope.** `receive`/`peek` print a bare JSON **array** of messages; the HTTP API returns a `{"messages": [...]}` object. - **The lock token.** It is emitted only where you need it to settle later — `receive --no-ack` and `receive-deferred`. A default auto-ack `receive` has already Completed the messages, so it omits `lock_token`; raw Peek-Lock Receive always returns one. The CLI also doesn't expose *every* HTTP operation (batch send, attempt ids). For the complete contract — all routes, all fields — see [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md). > **Settling across invocations needs a broker.** `receive --no-ack` locks a message and > prints its `lock-token`; you settle it later with `complete/abandon/reject/defer/renew > `. This works against a running broker (client mode), which holds > the lock between calls. In **embedded** mode each invocation opens the DB fresh and > reclaims orphaned locks, so a `--no-ack` lock does not survive to the next command — use > a broker (`mqlite serve` + `--endpoint`) when you need manual settle-later. ## Commands ### `serve` — run the broker ```bash MQLITE_DB=file:/data/mq.db MQLITE_TOKENS=mqk_dev mqlite serve --addr :6754 ``` | Flag | Default | | |---|---|---| | `--addr` | `:6754` | listen address | | `--diagnostic-log ` | `MQLITE_DIAGNOSTIC_LOG` (otherwise disabled) | private failure/recovery log; 4 MiB × 4 files; see [diagnostics](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md#private-diagnostic-log) | | `--metrics` | `false` | enable authenticated `/metrics` on the API port; overrides `MQLITE_METRICS`; `--metrics=false` disables | | `--insecure-allow-remote` | `false` | with auth disabled, allow a non-loopback bind (otherwise refused) | The listen address may also come from **`MQLITE_ADDR`** (precedence: `--addr` > `MQLITE_ADDR` > `:6754`; a blank value is rejected). With auth disabled (`MQLITE_TOKENS=off`) the broker **refuses a non-loopback bind** unless `--insecure-allow-remote` is passed, and **`MQLITE_CORS` defaults to off**. Serves the RPC API, the open `/` + `/healthz`, and — unless `MQLITE_UI=off` — the embedded admin console at `/ui`. Enable Prometheus on the same API port with one flag: ```bash MQLITE_DB=file:./mq.db MQLITE_TOKENS=mqk_dev mqlite serve \ --addr 127.0.0.1:6754 --metrics ``` Equivalent environment configuration is `MQLITE_METRICS=on`. Values `on`/`true`/`1` enable metrics; `off`/`false`/`0` disable it. Values are case-insensitive; unset or empty disables metrics. Invalid values fail startup. `--metrics=false` overrides an environment-provided enabled value. Administrator auth must remain enabled. Scrapes accept configured administrators, managed `manage` keys, or configured monitor credentials; send/listen keys cannot scrape metrics. When enabled, the open discovery card includes `"metrics":"/metrics"`, which resolves against the same origin. When disabled, the field is omitted and the route returns `404` even with valid credentials. No separate address or URL configuration is needed. If the API is publicly reachable, so is this route; keep metrics off public ingress with path filtering or a private-only broker. ### `key create|list|revoke` — manage persistent access keys Available in v0.3.1 and later. In client mode, use an environment administrator or a managed `manage` key as `MQLITE_TOKEN`. The broker can keep running while keys are created or revoked. Embedded mode directly manages the configured database and requires exclusive ownership; it cannot open a running local broker's DB. ```bash # MQLITE_ENDPOINT and MQLITE_TOKEN already select the broker and administrator. mqlite key create --name producer --permissions send --output json mqlite key create --name worker --permissions listen --output json mqlite key create --name processor --permissions send,listen --output json mqlite key create --name operator --permissions manage --output json mqlite key list --limit 100 --output json mqlite key list --sort created_desc --limit 25 --output json mqlite key revoke --id ``` | Subcommand | Flags | Result | |---|---|---| | `create` | required `--name`, `--permissions`; optional `--id`, `--expires-at-ms` | key metadata plus its one-time secret | | `list` | optional `--after-id`, `--limit` (default 100, maximum 1000), `--sort id_asc\|created_desc` (default `id_asc`) | metadata and a continuation cursor; no secrets or digests | | `revoke` | required `--id` | idempotently revokes an existing database key | `--id` is a public 32-character lowercase hex ID; when omitted, the CLI generates it before contacting the broker. Creation failures report that ID so an ambiguous result can be reconciled without guessing from a name. The secret is a separate `mqk_` token with 64 lowercase hex characters. Save it securely when returned; it cannot be retrieved again. `--expires-at-ms` uses epoch milliseconds; zero (the default) means no expiry. Key names need not be unique. Lists default to ascending public ID order. `--sort created_desc` lists the newest-created keys first across all pages, with descending ID as the tie-breaker for identical creation timestamps. Pass `next_after_id` as `--after-id` and keep the same `--sort` on every page; text output includes the full continuation command. In creation order, the cursor must be an existing key ID (revoked and expired keys remain valid cursors). ID order also accepts absent predecessor IDs for reconciling a lost create response. `send` permits sending, scheduling and cancelling scheduled messages; `listen` permits consumption, settlement, renewal, peek and queue stats; `manage` includes both and all administration, including issuing other `manage` keys. Permissions apply to the whole broker. A 403 is a permission failure, not an invalid token; ordinary send/listen keys cannot use `key`, `list`, `status` or the admin console. Existing configured administrator tokens retain their original format and rights. To rotate a managed key, create its replacement, move the application to the new token, then revoke the old ID. No broker restart is needed. See [the authentication contract](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md#auth) and [backup/restore implications](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#consistent-backups). ### `create-queue ` — create/update a queue ```bash mqlite create-queue orders --lock 30s --max-delivery 5 --dedup 10m --ordering strict_fifo ``` | Flag | Default | | |---|---|---| | `--lock` | 0 → 30s | Peek-Lock duration (`0` inherits the engine default, 30s) | | `--max-delivery` | 0 → 10 | deliveries before a message goes to the DLQ (`0` inherits 10) | | `--ttl` | 0 (unlimited) | default message TTL | | `--dedup` | 0 (off) | dedup window | | `--ordering` | `standard` | `standard` / `group_fifo` / `strict_fifo` | | `--dlq-max-age` | 0 (inherit) | per-queue DLQ retention: drop dead letters older than this | | `--dlq-max-count` | 0 (inherit) | per-queue: keep at most N dead letters (`-1` = unbounded) | | `--dlq-max-bytes` | 0 (inherit) | per-queue: cap dead-letter body bytes (`-1` = unbounded) | ### `subscribe ` — create a subscription *(alias: `create-subscription`)* ```bash mqlite subscribe orders eu-orders --expr 'subject_parts[0]=="orders" && properties["region"]=="eu"' ``` | Flag | | | |---|---|---| | `--expr` | "" | subscription filter — an [expr](https://github.com/mqlitehq/mqlite/blob/main/docs/concepts.md#subscription-filters-expr) boolean predicate; empty = match all | ### `send ` — enqueue a message `body` of `-` reads stdin; or use `--file`. ```bash mqlite send orders '{"id":1}' --group order-1 --message-id m1 echo '{"id":2}' | mqlite send orders - ``` | Flag | | | |---|---|---| | `--file` | read body from a file | | `--message-id` | dedup / idempotency key | | `--group` | group id (ordering / session key) | | `--subject` | subject (label) | | `--reply-to` | reply-to address | | `--ttl` | per-message TTL | ### `receive ` — consume (Peek-Lock) By default it receives and auto-Completes. Use `--no-ack` to leave messages locked and print each `lock-token` (settle them later — see settlement commands below), or `--delete` for at-most-once. ```bash mqlite receive orders --max 10 --wait 5s ``` | Flag | Default | | |---|---|---| | `--max` | 1 | max messages | | `--wait` | 0 | long-poll wait (e.g. `5s`) | | `--no-ack` | false | leave messages locked (don't Complete) | | `--delete` | false | receive-and-delete (at-most-once, no lock) | **Auto-ack never acknowledges output you can't see.** A message is only Completed *after* its body is written to stdout successfully. If the write fails (broken pipe) the message is left locked and redelivered; if stdout cannot deliver at all (closed at exec, or `/dev/null`) the command refuses up front — nothing is claimed. Use `--delete` to drain explicitly, or `--no-ack` if you only want the lock tokens. ### `peek ` — browse without locking ```bash mqlite peek orders --state dead_lettered --max 20 ``` | Flag | Default | | |---|---|---| | `--state` | "" | `active`/`locked`/`deferred`/`scheduled`/`dead_lettered` | | `--from` | 0 | start sequence number | | `--max` | 16 | max messages | ### `metrics ` — queue counters Prints a human line by default; use `--output json` for the JSON object. (**Changed in 0.3.0:** `metrics` previously printed JSON unconditionally — scripts that parsed the old default must add `--output json`.) ```bash mqlite metrics orders # active / locked / deferred / scheduled / dead_lettered / total mqlite metrics orders --output json # machine-readable ``` ### `list` — list queues & subscriptions ```bash mqlite list ``` ### `redrive ` — move dead letters back to active ```bash mqlite redrive orders --max 100 # back to source mqlite redrive orders --to orders-replay # to another queue ``` | Flag | Default | | |---|---|---| | `--to` | "" | target queue (default: back to source) | | `--max` | 0 (all) | max messages | | `--older-than` | 0 | only messages older than this | ### `purge-dlq ` — delete dead letters Destructive. An **unbounded** purge requires an explicit `--all`; otherwise pass a bound (`--max`/`--older-than`). Negative bounds are rejected (they would otherwise mean "no limit"). Exactly one positional `` — a stray argument is an error, not ignored. ```bash mqlite purge-dlq orders --older-than 24h # bounded mqlite purge-dlq orders --all # delete the entire DLQ (explicit) ``` | Flag | Default | | |---|---|---| | `--max` | 0 | max messages to delete (0 = unbounded → `--all` required) | | `--older-than` | 0 | only delete messages older than this (**minimum 1ms** — a sub-millisecond value would round to "no bound") | | `--all` | false | delete the entire DLQ (required to run unbounded) | ### `vacuum` — reclaim free DB pages to the OS Local maintenance (embedded; **stop the broker first** — the single-writer lock will reject it otherwise). New local DBs use `auto_vacuum=INCREMENTAL`, so the default is a no-lock `PRAGMA incremental_vacuum`; `--full` runs a full `VACUUM` (rewrites the file, global lock). Not applicable to a remote Turso/libSQL store. ```bash MQLITE_DB=file:/data/mq.db mqlite vacuum # incremental MQLITE_DB=file:/data/mq.db mqlite vacuum --full # full rewrite ``` ### `complete` / `abandon` / `reject` / `defer` / `renew` — settle a message Settle a message received earlier with `receive --no-ack` (which printed its `lock-token`). Positional args are always ` `. Client mode only in practice (a broker holds the lock between calls — see the note under Global flags). ```bash mqlite complete orders 42 lt_9f3a… # done mqlite abandon orders 42 lt_9f3a… --delay 60s # release for retry, after a backoff mqlite reject orders 42 lt_9f3a… --reason bad # dead-letter it (--detail …) mqlite defer orders 42 lt_9f3a… # set aside for receive-deferred mqlite renew orders 42 lt_9f3a… # extend the lock lease ``` ### `schedule ` — enqueue for future delivery ```bash mqlite schedule orders '{"id":1}' --at 30m # a duration from now mqlite schedule orders '{"id":1}' --at 2027-01-02T15:04:05Z # or an absolute RFC3339 time (must be future) ``` | Flag | | | |---|---|---| | `--at` | required | delivery time: RFC3339 or a duration (`30m`, `2h`) | | `--file` · `--message-id` · `--group` · `--subject` | | same as `send` | Cancel a scheduled message before it activates with **`cancel `**. ### `receive-deferred --seq …` — fetch deferred messages by seq ```bash mqlite receive-deferred orders --seq 42,57 # re-locks them and prints tokens to settle ``` ### `observe` — canonical observation Available in **v0.3.2 and later**. Remote mode requires a v0.3.2 or later broker; configured monitor credentials are also new in v0.3.2. ```bash mqlite observe --output json MQLITE_ENDPOINT=http://127.0.0.1:6754 MQLITE_TOKEN="$MONITOR_TOKEN" mqlite observe ``` Both output modes print the complete canonical JSON snapshot, including collection availability/freshness, queue gauges, committed message effects, storage operations, maintenance, filter failures, and HTTP request/authentication measurements. It is the same data contract as SDK `Observe`, MCP `observe`, and the HTTP `Observe` route; `/metrics` serializes these measurements for Prometheus. Check `collection.state` before interpreting queue counts. An unavailable collection has `queues: null`, not zero messages; process counters remain usable when the configured credential can authenticate. Durations use seconds, timestamps use epoch milliseconds, and process counters reset on restart. Embedded mode marks HTTP as `not_applicable`. Runtime `read_available` and `db_size_available` distinguish unknown or unsupported measurements from real zeroes. See [observability.md](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md). A configured `MQLITE_MONITOR_TOKENS` credential can call only `observe` and `/metrics`. It also opens the console's read-only overview/metrics views. `manage` includes observation; managed `send`/`listen` keys do not gain broker-wide monitoring. Monitor credentials are configured at startup and cannot be created through `key create`. ### `status` — backend snapshot ```bash mqlite status # backend, location, schema, ping, size, queue/subscription counts ``` ### `list-subscriptions` — subscriptions with topic + filter ```bash mqlite list-subscriptions ``` ### `test-filter ` — dry-run a subscription filter Compiles the expression and, with a sample, reports whether it would match (nothing is enqueued). ```bash mqlite test-filter 'subject == "ord.created"' # just validate mqlite test-filter 'subject == "ord.created"' --subject ord.created # validate + test a sample ``` | Flag | | | |---|---|---| | `--subject` · `--group` · `--prop k=v,k=v` · `--file` | | build the sample message | ### `version` / `help` ```bash mqlite version mqlite help ``` ## Examples end to end ```bash # embedded: produce + consume against a local file, no broker export MQLITE_DB=file:./mq.db mqlite create-queue jobs mqlite send jobs "hello" mqlite receive jobs --wait 2s # client: same commands against a running broker export MQLITE_ENDPOINT=https://your-mqlite.fly.dev MQLITE_TOKEN=mqk_prod_xxx mqlite list mqlite metrics jobs ``` See also: [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md) (HTTP), [mcp.md](https://github.com/mqlitehq/mqlite/blob/main/docs/mcp.md) (agents), [examples.md](https://github.com/mqlitehq/mqlite/blob/main/docs/examples.md) (Go SDK). # Access keys Runtime access keys are available in **v0.3.1 and later**. Version v0.3.0 supports configured administrator tokens only. ## Choose a permission | Permission | Operations | |---|---| | `send` | Send, schedule and cancel scheduled messages | | `listen` | Receive, settle, renew, peek and read queue statistics | | `send` + `listen` | Both application roles | | `manage` | Both application roles, queue/topic administration, console, metrics, and issuing/revoking keys, including other `manage` keys | Permissions apply across the broker, not to individual queues. A `listen` key can delete consumed messages through completion or receive-and-delete; it is not a read-only auditing role. A `send` key can cancel scheduled messages from other producers. Use separate brokers when applications need a stronger isolation boundary. The keys in `MQLITE_TOKENS` remain administrators. Keep one configured administrator available for recovery. Existing configured token strings remain valid; every newly generated token uses `mqk_` followed by 64 lowercase hexadecimal characters. The broker stores only the SHA-256 digest of managed tokens, together with their public ID, name, permissions and lifecycle timestamps. Names need not be unique. For read-only monitoring in **v0.3.2 and later**, configure distinct `MQLITE_MONITOR_TOKENS` alongside administrator authentication. These credentials grant only `Observe` and `/metrics`, including the console's monitoring views; they cannot read message bodies, consume, publish or manage keys. `monitor` is a configuration capability, not a managed-key permission. It works independently of the key database during a storage outage. See the [monitoring starter](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) for configuration and rotation. Managed authentication performs one indexed database query for every request, without caching credentials. On Turso this includes a remote round trip; measure the application's request latency and throughput with its intended credentials and storage location before sizing a deployment. ## Create and use a key Start the broker with authentication enabled and connect the CLI using an administrator credential from your secret manager: ```sh export MQLITE_ENDPOINT=http://127.0.0.1:6754 # Set MQLITE_TOKEN securely to a configured administrator or managed manage key. mqlite create-queue orders mqlite key create --name orders-producer --permissions send --output json mqlite key create --name orders-worker --permissions listen --output json ``` Save each returned `token` securely. It is shown only on successful creation and cannot be recovered through listing or by repeating creation. Give the producer its `send` token as `MQLITE_TOKEN`, then run `mqlite send orders 'hello'`. Give the worker its `listen` token, then run `mqlite receive orders`. The worker command prints and completes the message by default. Each creation has a separate public `id` (32 lowercase hex characters). The CLI generates it before making the request, or accepts `--id`. An optional `--expires-at-ms` sets a future UTC epoch-millisecond expiry; zero means no expiry. Use an administrator for `mqlite key list --output json`. Pagination uses `--after-id` with the previous `next_after_id`; the default page limit is 100, maximum 1000. Listings include expired and revoked metadata, never secrets or digests. Configured tokens are managed in configuration and do not appear here. The CLI defaults to public ID order. Use `mqlite key list --sort created_desc --output json` for newest first, retaining `--sort created_desc` on subsequent pages. Equal creation timestamps are ordered by public ID descending. ## Go SDK Both `Client` and `Embedded` expose `CreateKey`, `ListKeys`, `ListKeysWithOptions` and `RevokeKey`. Embedded administration is trusted local access and needs exclusive ownership of a local database. Use `Client` to manage a running broker. ```go id := mqlite.GenerateKeyID() // retain this public ID before the request created, err := admin.CreateKey(ctx, mqlite.CreateKeyOptions{ ID: id, Name: "orders-producer", Permissions: []string{"send"}, }) if err != nil { // Preserve id; reconcile an ambiguous result before issuing a replacement. return fmt.Errorf("create key %s: %w", id, err) } // Store created.Token in your secret manager; never log the result struct. producer, err := mqlite.Open(ctx, endpoint, mqlite.WithToken(created.Token)) if err != nil { return err } defer producer.Close() _, err = producer.SendOne(ctx, "orders", mqlite.OutMessage{Body: []byte("hello")}) ``` Pass `page.NextAfterID` to the next `ListKeys(ctx, afterID, limit)` call until it is empty. This method keeps public ID order. For newest-first pagination, use `ListKeysWithOptions(ctx, mqlite.ListKeysOptions{Sort: "created_desc", Limit: 100})` and set `AfterID` to the returned cursor for later pages, keeping the same `Sort`. Revoke a managed key with `RevokeKey(ctx, id)`. The SDK returns `ErrUnauthenticated` for missing/invalid/revoked/expired credentials and `ErrPermissionDenied` for insufficient permissions. `Receiver.Run` stops on these permanent authentication/authorization failures, including settlement and renewal. ## HTTP, console and MCP The three HTTP operations are `AuthService/CreateKey`, `ListKeys` and `RevokeKey`. See the complete [request/response contract](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md#createkey--listkeys--revokekey). All require `Authorization: Bearer `. Authentication disabled with `MQLITE_TOKENS=off` does not enable anonymous key administration. The console at `/ui/` accepts a configured administrator or managed `manage` key for administration; configured monitor credentials open its observability views. Open **Access keys** to create, inspect and revoke credentials. Save the newly created secret before dismissing its one-time display. The console explains when an older broker does not support the feature. Application `send`/`listen` keys belong in clients, not the administrator console. The **managed keys** list shows runtime-issued access keys newest first across all pages. Creating a key returns to the first page; equal creation timestamps use public ID descending as a stable tie-breaker. Configured `MQLITE_TOKENS` administrators are changed through configuration and a broker restart. Expiry uses local time in `yyyy-MM-dd HH:mm` format with an English calendar, independent of browser language; list timestamps include seconds. The MCP server exposes `create_key`, `list_keys` and `revoke_key`. Configure its `MQLITE_TOKEN` with the permission required by each tool. Key creation needs a public ID allocated before the call. See [MCP usage](https://github.com/mqlitehq/mqlite/blob/main/docs/mcp.md); do not paste new secrets into prompts, tool logs or generated source files. ## Rotate, revoke and recover 1. Create a replacement key and store its secret. 2. Move the application to the replacement and verify a send/receive/settle cycle. 3. Revoke the old public ID with `mqlite key revoke --id `. No restart is needed. Revocation is permanent and repeated revocation succeeds. New authentication fails immediately after a successful revocation, but a request already authorized, including a long poll, may finish. Later settle/renew requests authenticate again. Keys issued by another key are independent: revoking their issuer does not revoke them. An administrator can revoke itself; retain another administrator before doing so. If a creation response is lost, keep the original public ID and list keys as an administrator to find it. If it exists without a saved secret, revoke it and create a replacement with a new ID. A repeated ID returns `409 key_conflict` and never replays the secret. A timeout or `unavailable` response is not proof that creation failed; clients must not blindly retry with a new ID. A token explicitly present in `MQLITE_TOKENS` is always an administrator, even if a matching managed record is restricted, revoked or expired. Remove that static configuration and restart to remove the configured grant. To rotate configured tokens, temporarily accept old and new values, move clients, then remove the old value and restart again. Backups include managed key state. Restoring an older snapshot can restore an old revocation state or discard newer credentials. Review keys using a configured administrator before reopening traffic. See [backup and upgrade procedures](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md). # Deploying mqlite mqlite is **one pure-Go binary** (or one container image) over **one SQLite file** (or a remote Turso DB). Run it embedded in your process, or as a network broker — this guide covers the broker (`mqlite serve`). Pick a target: - [Docker / GHCR](#docker--ghcr) — quickest; the published multi-arch image. - [Fly.io](#flyio-minimal-cost) — a minimal-cost, scale-to-zero recipe. - [systemd](#systemd-bare-metal) — a single binary on a VM. - [Turso](#turso-remote-libsql) — remote replicated storage instead of a local volume. - [Production operations](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md) — backups, restoration, upgrades and incident actions. ## Configuration (all targets) Everything is read from the environment — the DB string is never compiled in. | Env | Meaning | |---|---| | `MQLITE_DB` | `file:/data/mq.db` (local) or `libsql://.turso.io` (remote) | | `MQLITE_DB_AUTH_TOKEN` | auth token for a remote libSQL/Turso DSN | | `MQLITE_TOKENS` | comma-separated administrator Bearer tokens (**set this in production**) | | `MQLITE_MONITOR_TOKENS` | v0.3.2+: optional comma-separated credentials for `Observe` and `/metrics` only; requires auth and distinct administrator credentials | | `MQLITE_METRICS` | v0.3.2+: opt-in authenticated `/metrics` on the API port; `on`/`true`/`1` enables, `off`/`false`/`0` disables (default); requires administrator auth | | `MQLITE_SYNC` | durability: `NORMAL` (default) / `FULL` / `OFF` / `EXTRA` (local file only); an unrecognized value is rejected at startup | | `MQLITE_DLQ_MAX_AGE` · `MQLITE_DLQ_MAX_COUNT` · `MQLITE_DLQ_MAX_BYTES` | DLQ retention bounds (defaults 14d / 1,000,000 per queue; byte cap off; `MQLITE_DLQ_RETENTION=off` to disable) — see [retention.md](https://github.com/mqlitehq/mqlite/blob/main/docs/retention.md) | | `MQLITE_MAX_MESSAGE_BYTES` | reject larger bodies (default 1 MiB) | | `MQLITE_UI` | serve the embedded admin console at `/ui` (default on; `off` runs headless) | > **Auth (secure by default):** if `MQLITE_TOKENS` is **unset**, `serve` **generates a > random `mqk_…` token and prints it at startup** — the broker is never silently open. > Set `MQLITE_TOKENS` to your own token(s) for a stable value (rotate by updating it), > or `MQLITE_TOKENS=off` to explicitly disable auth — but then the broker **refuses a > non-loopback bind**: bind `127.0.0.1` explicitly, or pass `--insecure-allow-remote` to > expose it (and `MQLITE_CORS` defaults to off while auth is off). The `/` > discovery and `/healthz` endpoints stay open, as does the static `/ui` console when > enabled (its API calls still carry a token); everything else needs > `Authorization: Bearer `. > **Admin console:** the broker bakes a static web console into the binary and serves > it at `/ui` (e.g. `http://localhost:6754/ui`) — no separate process or Node runtime. > Paste your broker URL + token in the console to drive every queue operation. Set > `MQLITE_UI=off` to disable it for headless deployments. The broker listens on `:6754` by default (`mqlite serve --addr :6754`). Full endpoint and error reference: [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md). ## Docker / GHCR > **Version and upgrade compatibility.** > These instructions target **v0.3.3**, with default port **6754** and schema token **5**. > Existing v0.3.0/v0.3.1/v0.3.2 databases remain compatible; back up before upgrading and > retain a configured administrator. Existing Prometheus scrapers require > `MQLITE_METRICS=on` after upgrading; metrics is now disabled by default. > Rolling back to v0.3.1 also requires its > previous monitoring configuration; v0.3.0 additionally ignores managed keys. > **v0.2.0 uses port 8080 and schema token 2.** Its database cannot be opened by > v0.3.3: preserve the old binary/database pair and follow the > [upgrade and rollback procedure](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#upgrade-and-rollback) before replacing it. The published image is multi-arch (amd64 + arm64): ```bash # v0.3.3, default port 6754 docker run -d --name mqlite -p 6754:6754 \ -v mqlite-data:/data \ -e MQLITE_DB=file:/data/mq.db \ -e MQLITE_TOKENS=mqk_prod_CHANGEME \ -e MQLITE_MONITOR_TOKENS=mqk_monitor_CHANGEME \ -e MQLITE_SYNC=FULL \ ghcr.io/mqlitehq/mqlite:0.3.3 ``` - The named volume `mqlite-data` persists the SQLite file across restarts. - Pin a version tag in production; `:0.3` tracks patches, `:latest` the newest release. Images `>= 0.3.0` listen on `6754`; **`0.2.x` and earlier listen on `8080`** — if you pin an older tag, publish the port it actually uses. - Verify: `curl http://localhost:6754/` (discovery card) and `/healthz`. Metrics is disabled by default. For a private broker or an ingress that blocks public `/metrics`, add `-e MQLITE_METRICS=on` and give the collector a monitor token. Scraping uses the same API port; enabled discovery includes `"metrics":"/metrics"`. ## Fly.io (minimal cost) A scale-to-zero, single-machine recipe — cheapest way to run it online. Save as `fly.toml`: ```toml app = "your-mqlite" primary_region = "sin" # pick a region near you [build] image = "ghcr.io/mqlitehq/mqlite:0.3.3" # pinned image, no build on Fly [env] MQLITE_DB = "file:/data/mq.db" # SQLite on the persistent volume MQLITE_SYNC = "FULL" # sync every acknowledged local commit [[mounts]] source = "data" # the volume created below destination = "/data" [http_service] internal_port = 6754 force_https = true auto_stop_machines = "stop" # fully stop when idle (cheapest) auto_start_machines = true # start on the next request min_machines_running = 0 # scale to zero [[vm]] size = "shared-cpu-1x" # lowest shared CPU memory = "256mb" # lowest memory ``` ```bash fly apps create your-mqlite fly volume create data --size 1 --region sin # 1 GB SQLite volume (region-bound) fly secrets set MQLITE_TOKENS=mqk_prod_CHANGEME \ MQLITE_MONITOR_TOKENS=mqk_monitor_CHANGEME # broker secrets, not in fly.toml fly deploy --ha=false # single machine curl https://your-mqlite.fly.dev/ # discovery card ``` The public recipe leaves metrics disabled. Enabling `MQLITE_METRICS=on` also makes `/metrics` reachable through this public HTTP service; authentication protects its contents but does not make the route private. For private monitoring without another Fly Machine, remove `[http_service]` and any public `[[services]]` entries, set `MQLITE_ADDR=fly-local-6pn:6754` and `MQLITE_METRICS=on`, and connect clients and the collector through Fly's private network. From your workstation, `fly proxy 6754:6754 --app your-mqlite` opens a loopback tunnel; Prometheus can scrape `http://127.0.0.1:6754/metrics` using the monitor token. Start the Machine before connecting: direct private access does not provide the public recipe's autostart behavior. If the API must stay public, enable metrics only when your existing ingress actually blocks public `/metrics` while preserving private collector access; otherwise leave it disabled. See [cloud and Kubernetes monitoring](https://github.com/mqlitehq/mqlite/blob/main/docs/observability-cloud.md#flyio) and Fly's [private networking](https://fly.io/docs/networking/private-networking/) and [proxy](https://fly.io/docs/flyctl/proxy/) references. **Cost:** with `auto_stop_machines="stop"` + `min_machines_running=0` the machine runs only while serving requests (cold-starts in seconds, stops when idle), so the idle storage charges remain while the machine is stopped. The historical workloads in [benchmark.md](https://github.com/mqlitehq/mqlite/blob/main/docs/benchmark.md) measured roughly 25–34 MB RSS; these are not memory limits for other workloads or versions. Size the current broker using your message sizes, concurrency, queue count and monitoring load, and measure peak memory under that traffic. Continuous requests, including monitoring, can keep the machine active. ## systemd (bare metal) Build or download the binary, then run it as a service: ```bash go build -o /usr/local/bin/mqlite ./cmd/mqlite # or grab a release binary ``` `/etc/systemd/system/mqlite.service`: ```ini [Unit] Description=mqlite broker After=network.target [Service] # Bind loopback: the reverse proxy below is the only public entry point, so the broker # is not reachable on the LAN even without a firewall rule. Use --addr :6754 only if you # deliberately want it on all interfaces (and then add a firewall rule). ExecStart=/usr/local/bin/mqlite serve --addr 127.0.0.1:6754 Environment=MQLITE_DB=file:/var/lib/mqlite/mq.db Environment=MQLITE_TOKENS=mqk_prod_CHANGEME Environment=MQLITE_SYNC=FULL Restart=on-failure DynamicUser=yes StateDirectory=mqlite # creates/owns /var/lib/mqlite [Install] WantedBy=multi-user.target ``` ```bash systemctl daemon-reload && systemctl enable --now mqlite curl http://localhost:6754/healthz ``` Put a TLS-terminating reverse proxy (Caddy/nginx) in front for anything public; it connects to the broker on `127.0.0.1:6754`, which is the only interface the broker binds above — so the proxy (with TLS + whatever access control you add) is the single entry point, not a bypassable layer over an all-interfaces socket. Runtime-managed keys in v0.3.1 and later can be created and revoked without a restart. Use a configured administrator or managed `manage` key for console and key administration; use `send`/`listen` for applications. In v0.3.2 and later, use distinct configured `MQLITE_MONITOR_TOKENS` for a read-only metrics scraper or console monitoring views. These credentials are changed through configuration and a broker restart. See [key commands](https://github.com/mqlitehq/mqlite/blob/main/docs/cli.md#key-createlistrevoke--manage-persistent-access-keys) and [key rotation](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#key-rotation). ## Backup, restore and upgrades Follow the [production runbook](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#consistent-backups) for read-only online snapshots, offline directory copies, isolated restore validation and rollback. Use a fresh restore directory so an old WAL/SHM cannot attach to the snapshot. **v0.2.0 databases use schema 2; v0.3.0 through v0.3.3 use schema 5.** There is no in-place migration from schema 2. Before upgrading from v0.2.0, account for retained work in every state, keep the old binary/database pair, and create the v0.3.3 database separately. Upgrading from v0.3.0 reuses the existing database and adds managed-key storage; upgrading from v0.3.1/v0.3.2 leaves the schema unchanged. Take a consistent backup first. `Observe` and configured monitor credentials require v0.3.2; restore compatible monitoring configuration when rolling back. See [upgrade and rollback](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#upgrade-and-rollback). ## Turso (remote libSQL) Use replicated remote storage instead of a local volume — no disk to manage, and the same engine. Point `MQLITE_DB` at the libSQL URL and pass the token: ```bash MQLITE_DB=libsql://.turso.io \ MQLITE_DB_AUTH_TOKEN= \ MQLITE_TOKENS=mqk_prod_CHANGEME \ mqlite serve ``` The same env works in Docker (`-e MQLITE_DB=... -e MQLITE_DB_AUTH_TOKEN=...`) or Fly (`fly secrets set MQLITE_DB_AUTH_TOKEN=...`); drop the `[[mounts]]` volume since data lives in Turso. Durability is the Turso server's responsibility — `MQLITE_SYNC` is ignored for remote DSNs. See [turso.md](https://github.com/mqlitehq/mqlite/blob/main/docs/turso.md). ## Verify any deploy ```bash curl https:/// # discovery: {name, version, status, ...} (open) curl https:///healthz # ok (open) # authed round-trip: T=mqk_prod_CHANGEME curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"name":"orders","config":{}}' \ https:///mqlite.v1.AdminService/CreateQueue ``` ## Production checklist - [ ] `MQLITE_TOKENS` set (auth on); rotate by updating it. - [ ] HTTPS in front (Fly `force_https`, or a reverse proxy for systemd/Docker). - [ ] Persistent storage sized for the **peak** backlog — the file grows to peak, then a background janitor returns freed pages to the OS (`incremental_vacuum`) so it shrinks back gradually as the queue drains ([retention.md](https://github.com/mqlitehq/mqlite/blob/main/docs/retention.md)). - [ ] DLQ retention left on (default) or tuned for your volume. - [ ] Pinned image/binary version and recorded checksum/digest. - [ ] `MQLITE_SYNC=FULL` for acknowledged local commits that must survive power loss. - [ ] One active broker per database; no overlapping replacement. - [ ] Verified backup and rehearsed restore meet the application's RPO/RTO. - [ ] Free-disk, restart/error and queue-age alerts; an authenticated write/consume canary. - [ ] Enable `MQLITE_METRICS=on` for authenticated Prometheus scraping on the API port; use private routing or public-ingress path filtering to keep `/metrics` private — see [observability.md](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md). # Observability The canonical `Observe` interface, configured monitor credentials and the monitoring starter require **v0.3.2 or later**. Earlier releases expose only the legacy metrics described under [compatibility](#compatibility-from-v031). MQLite exposes one canonical observation through `AdminService/Observe`, the Go SDK, CLI, MCP, console and Prometheus `/metrics`. Queue snapshots and committed message effects originate in the engine. HTTP request measurements originate in the server; pure embedded use marks that domain `not_applicable`. Grafana derives rates, time windows and percentiles from these values instead of maintaining another set of message counters. Prometheus scraping is disabled by default. Enable authenticated `/metrics` on the existing API port with `MQLITE_METRICS=on` or `mqlite serve --metrics`: ```sh MQLITE_DB=file:/data/mq.db \ MQLITE_TOKENS="$ADMIN_TOKEN" \ MQLITE_MONITOR_TOKENS="$MONITOR_TOKEN" \ MQLITE_METRICS=on \ mqlite serve --addr 127.0.0.1:6754 ``` This local example exposes both the API and metrics on loopback. The exporter shares the broker's process, engine and counters; it needs no extra port or VM. The Go SDK uses `WithMetrics(true)` alongside `WithTokens(...)` and optionally `WithMonitorTokens(...)`. Enabling metrics without administrator auth is rejected. When enabled, `GET /` automatically includes `"metrics":"/metrics"`; clients resolve this against the same origin. When disabled, discovery omits the field and `/metrics` returns `404` even with valid credentials. In production, use a private broker address or an ingress that blocks `/metrics` from the public Internet while allowing private collectors. API and metrics share a port, so a port-level firewall cannot separate them. Bearer authentication protects the data but does not hide a publicly reachable route. The monitor credential remains required on private networks. For a runnable broker, Prometheus and Grafana stack, see [the local observability demo](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/README.md). For an existing platform, use the [cloud and Kubernetes guide](https://github.com/mqlitehq/mqlite/blob/main/docs/observability-cloud.md). ## Maintenance failure diagnosis Brokers running v0.3.3 or later add optional `maintenance[].diagnostics` and `storage.operations[].last_error` fields to Observe. Older snapshots remain valid; missing diagnostics mean **not reported**, not healthy. Existing Prometheus labels and cumulative error buckets are unchanged. Each maintenance task reports its latest outcome (`never`, `ok`, `failed`, or `interrupted`), last run time, consecutive failed passes, recovery time, and the latest eight failed steps. One pass can fail at several steps but increments the failure counter only once. Canceled/closed operations remain interruptions. A successful pass clears the consecutive streak and records recovery; it does not clear cumulative failures or history. All snapshot counters/history reset when the broker process restarts. A failed poll makes the console's previous data stale. Error details include a timestamp, generated explanation, category, HTTP status or SQLite code where available, and an opaque fingerprint. Categories distinguish provider authentication, rate limits, provider availability, DNS, TLS, timeouts, protocol, constraints, schema, and storage failures. These descriptions summarize the observed error; an HTTP 503 alone cannot establish the provider's root cause. Unknown commit outcomes remain explicitly ambiguous and must be reconciled before retrying. Diagnostic changes do not broaden write retries. A storage bucket's count covers **all** occurrences in that legacy bucket; `last_error` describes only its latest failure. Neither a historical count nor a successful maintenance pass proves message integrity or high availability. Empty queues still run background database maintenance, so failures can occur while no clients are sending or receiving. ### Private diagnostic log Structured stderr events `maintenance.failure`, `maintenance.interrupted`, `maintenance.recovered`, `storage.interrupted`, and `storage.failure` record the task/step or operation. Failure events include the same fingerprint as Observe and a bounded provider explanation with common credentials and URLs redacted. Public monitor responses contain generated explanations only, never the raw provider/SQL error. Private logs may still contain sensitive database error context; keep them operator-only. To retain evidence beyond the platform's stderr retention, enable a separate file: ```sh MQLITE_DIAGNOSTIC_LOG=/data/mqlite-diagnostics.jsonl mqlite serve # Equivalent: mqlite serve --diagnostic-log /data/mqlite-diagnostics.jsonl ``` The parent directory must exist. The file is disabled by default; an invalid configured path fails startup. It appends across restarts, uses mode `0600`, syncs each record, and rotates at 4 MiB with three archives (`.1` through `.3`, newest first): at most 16 MiB of retained diagnostic records, not a time-based retention promise. Routine requests and successful maintenance passes are excluded. A successful recovery is logged once per failure streak. A disk write failure still emits the original event to stderr plus one warning per logging failure streak. Mount the file on persistent storage to survive machine replacement/deployment; Fly's ephemeral root filesystem is insufficient. Export archives elsewhere if evidence must survive volume loss or sustained errors filling the rotation budget. The file is independent of the queue database, and is not replayed into Observe history after restart. It is diagnostic evidence, not a message backup or failover mechanism. Embedded applications can supply their own logger with `WithLogger`. ## Read-only monitoring access Configure a separate scraper credential in `MQLITE_MONITOR_TOKENS`, alongside configured administrator credentials in `MQLITE_TOKENS`. Monitoring credentials can read `/metrics` and `AdminService/Observe`. They cannot send, receive, peek at message bodies, manage queues, inspect administrative configuration or issue keys. Administrator credentials and managed `manage` keys retain monitoring access. Configured monitor credentials do not query the database during authentication, so they can observe storage failure. Managed keys still require their current revocation and expiry state to be checked; no monitoring cache bypasses that check. Do not reuse an administrator token as a monitor token. Configuration changes require a broker restart; the demo's startup wrapper reads separate secret files. A Prometheus job should reference a mounted secret file: ```yaml scrape_configs: - job_name: mqlite # The native listener is plain HTTP. Use https only when a TLS proxy # terminates TLS in front of the private broker. scheme: http metrics_path: /metrics scrape_interval: 15s scrape_timeout: 10s authorization: type: Bearer credentials_file: /run/secrets/mqlite-monitor.token static_configs: - targets: [mqlite.internal.example:6754] ``` If a TLS proxy fronts the listener, use the correct TLS CA rather than disabling certificate checks. Grafana connects to Prometheus and does not need the MQLite token. See the [Prometheus authorization reference](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). ## Availability and freshness The native snapshot includes `sampled_at_ms`, a collection status, last successful collection time and elapsed collection duration. A failed queue read is unavailable, not an empty list of healthy queues. Prometheus responds with available in-process measurements and `mqlite_collection_success 0`, omitting queue gauges that could not be collected. Failure before the endpoint, including invalid scrape credentials, causes Prometheus `up` to become zero instead. Queue counts are actual retained rows from a grouped database query. They describe stored states, not guaranteed immediate delivery eligibility: visibility, TTL, lease expiry awaiting maintenance and ordering rules can affect claims. Different measurement domains are sampled at different instants. The observation is not an atomic transaction encompassing every database, HTTP and in-process counter. `/healthz` is open process liveness. `mqlite_storage_read_available` reports a real storage read probe; success does not prove writes, end-to-end delivery or data integrity. Failed read latency is absent, not zero. Local storage size is unavailable for memory/remote backends, not a fabricated zero-byte database. ### Collection cost and interval Each observation collects the full queue inventory with one grouped message scan and a storage read probe. The scan includes retained rows, so its cost grows with backlog as well as queue count. A local SQLite database uses the same single-writer connection for application work and collection; monitoring is not free or isolated from message-operation latency. Start production scraping at 15 seconds, measure collection duration and actual send/receive latency under the expected retained backlog, then adjust the scrape and native/console polling intervals. Each native observation or Prometheus scrape collects its own snapshot. Grafana dashboard refreshes query Prometheus instead of adding direct broker scans. The local demonstration uses a faster interval to make transitions visible; choose production cadence from measured workload needs. RPC/result and storage/error dimensions have fixed vocabularies, but their zero-initialized counters and histogram buckets still produce many series. Queue dimensions grow with the configured inventory. Measure scrape response size, duration and series count before increasing inventory or polling frequency; a working small demo does not establish a large-deployment capacity limit. ## Metric contract Prometheus uses text exposition 0.0.4. Labels are bounded categories except for configured entity names. Credentials, key IDs, message IDs, payloads, expressions, raw URLs and free-text errors are never labels. Durations use seconds; timestamps use Unix seconds in Prometheus and epoch milliseconds in native JSON. | Family | Type | Labels / meaning | | --- | --- | --- | | `mqlite_info` | gauge, value 1 | `version`, `backend`, `schema_version` | | `mqlite_start_time_seconds` | gauge | Engine start timestamp | | `mqlite_sample_timestamp_seconds` | gauge | Snapshot timestamp | | `mqlite_collection_success` | gauge | 1 when queue collection succeeds, otherwise 0 | | `mqlite_collection_status` | gauge, value 1 | `state`, bounded `error_code` | | `mqlite_collection_last_success_timestamp_seconds` | gauge | Last successful collection, 0 before one succeeds | | `mqlite_collection_duration_seconds` | gauge | Queue collection elapsed time | | `mqlite_entities` | gauge | `kind=queue|subscription`; queue count excludes subscriptions and topic routing names | | `mqlite_queue_messages` | gauge | `queue`, `state=active|locked|deferred|scheduled|dead_lettered` | | `mqlite_queue_retained_messages` | gauge | `queue`; all retained rows including unexpected states | | `mqlite_queue_unexpected_messages` | gauge | `queue`; retained rows outside the five normal states | | `mqlite_queue_oldest_message_age_seconds` | gauge | `queue`; original enqueue age of oldest active/locked row, 0 if none | | `mqlite_message_events_total` | counter | `queue`, `event`; confirmed effects defined below | | `mqlite_rpc_requests_total` | counter | `rpc`, `code`; registered RPC requests including authentication rejection | | `mqlite_rpc_request_duration_seconds_total` | counter | Same labels; cumulative whole-request elapsed time, including authentication | | `mqlite_rpc_duration_seconds` | histogram | `rpc`; post-authentication handler latency, preserving the existing timing scope | | `mqlite_authentication_total` | counter | `outcome=success|missing|invalid|expired|revoked|permission_denied|backend_error` | | `mqlite_storage_operations_total` | counter | `operation`, `outcome`, `error_code` | | `mqlite_storage_operation_duration_seconds` | histogram | Same labels; storage-operation elapsed time | | `mqlite_storage_retries_total` | counter | Same labels; safe retry attempts within the observed operation | | `mqlite_storage_pool_connections` | gauge | `state=max_open|open|in_use|idle` | | `mqlite_storage_pool_waits_total` | counter | Connection-pool waits | | `mqlite_storage_pool_wait_duration_seconds_total` | counter | Cumulative pool waiting time, distinct from SQL execution | | `mqlite_storage_read_available` | gauge | Storage read probe succeeded, 0 or 1 | | `mqlite_storage_ping_seconds` | gauge | Read round trip; omitted on failure | | `mqlite_storage_size_available` | gauge | Local footprint can be reported, 0 or 1 | | `mqlite_storage_size_bytes` | gauge | Local DB plus WAL/shared-memory footprint; omitted if unavailable | | `mqlite_maintenance_enabled` | gauge | `task`; disabled/inapplicable task is 0 | | `mqlite_maintenance_runs_total` | counter | `task`, `outcome=success|error|interrupted` | | `mqlite_maintenance_duration_seconds` | histogram | `task`; maintenance pass elapsed time | | `mqlite_maintenance_last_success_timestamp_seconds` | gauge | `task`; last successful pass, 0 before success | | `mqlite_filter_failures_total` | counter | Routing failures, `stage=compile|evaluate`; normal filter mismatch is not an error | Histograms serialize finite cumulative `_bucket` samples, a `+Inf` bucket, `_count` and `_sum`. Whole-request duration currently supplies a sum and request count, not a histogram: their ratio is a mean, never a percentile. Storage operations are `read`, `write` and `transaction`; outcomes are `ok`, `error`, `rejected` and `outcome_unknown`. `rejected` with error code `application` means the operation was rejected by application/transaction callback logic; it is not evidence of a database outage. Counts represent logical engine storage-wrapper calls, not individual messages or every SQL statement inside a transaction. Retries are counted separately. Internal maintenance and observation reads are included; collecting a snapshot itself contributes read/probe operations. Error categories are empty on success or one of `canceled`, `closed`, `busy`, `connection`, `full`, `corrupt`, `io`, `application`, `outcome_unknown`, `other`. Fixed storage outcome/error combinations and registered RPC/result combinations are initialized to zero, so the first event can be detected after an earlier successful scrape. Use `rate`/`increase` across observed samples; a first scrape after an event cannot reconstruct its time. Maintenance tasks are `locks`, `scheduled`, `ttl`, `dedup`, `receipts`, `retention`, `reclaim`. Check enabled status and each task's actual cadence before alerting on its last-success time. Canceled passes during normal shutdown are `interrupted`, not successful passes or maintenance errors; they do not advance the last-success timestamp. Filter counters cover compilation/evaluation failures while routing published messages. Invalid `Subscribe` configuration is an RPC `invalid_argument` result; deliberate `TestFilter` validation is not a routing failure. ### Message effects Counters increase for confirmed effects, after transaction commit. A rollback, transaction retry or exact receive/settlement replay does not create a second committed effect. An uncertain remote outcome is not classified as a known rollback. Counters are process-local and reset at restart; unobserved activity before a crash can be lost. They are operational measurements, not a durable billing or business audit ledger. Custom raw SQL through `EngineTx.SQL` is not attributed to message-effect events. Queue gauges still report the actual retained database rows, including unexpected states; do not use event counters to reconstruct changes made outside the normal message operations. | Event | Meaning | | --- | --- | | `enqueued` | New queue copies; topic fanout counts actual copies, not producer requests | | `scheduled` | Enqueued copies initially scheduled; a subset of enqueue activity | | `deduplicated`, `dedup_conflict` | Suppressed duplicates and conflicting dedup attempts | | `delivered`, `redelivered` | Committed claims; redelivery is a subset, not additional unique messages | | `completed`, `receive_deleted` | Successful completion removals and receive-and-delete removals | | `abandoned`, `deferred`, `rejected` | Confirmed settlement effects | | `dead_lettered` | Transitions into the DLQ, including automatic paths | | `ttl_discarded`, `retention_deleted` | TTL discard and dead-letter retention deletion | | `purged`, `canceled`, `redriven` | Administrative DLQ purge, scheduled cancellation and redrive effects | | `lock_expired`, `recovered`, `activated` | Expired-lease handling, startup recovery and scheduled activation | Some events overlap. For example, rejection can also enter the DLQ; do not sum all events into a message ledger. A claim does not prove a response reached the client, and completion means the consumer reported success, not independently verified business-side processing. Broker redelivery, client HTTP retry and storage retry are separate concepts. ### Compatibility from v0.3.1 Prometheus scraping is now disabled by default. On upgrade, enable `MQLITE_METRICS=on` or `--metrics` to keep scraping the same API port, and configure a monitor token for the collector. All five pre-existing families remain available when enabled. These compatibility views read canonical values; they do not maintain separate counters: | Existing family | Canonical mapping | | --- | --- | | `mqlite_queue_messages` | Same queue state gauges | | `mqlite_queue_total` | Same value as `mqlite_queue_retained_messages`; still a gauge | | `mqlite_queue_oldest_message_age_ms` | Canonical seconds multiplied by 1000 | | `mqlite_messages_completed_total` | `mqlite_message_events_total{event="completed"}` | | `mqlite_rpc_duration_seconds` | Same post-authentication handler histogram | The ambiguous `queue_total` and millisecond age names are deprecated for new consumers; use the canonical names. They are retained until the next minor version (v0.4.0), when any removal requires an explicit migration. No independent lifetime completion counter is introduced. ## Dashboard and useful queries The provisioned dashboard offers instance/queue selection, traffic, backlog, request/authentication results, storage, maintenance and active alerts. Empty samples remain “No data” or “Not available”; they are not replaced with green zeros. ```promql # Stored queue states, not immediate claim eligibility mqlite_queue_messages{state="active"} # Observed completion throughput; rate handles observed counter resets sum by (instance, queue) (rate(mqlite_message_events_total{event="completed"}[5m])) # Handler p95; Receive includes intentional long-poll waiting histogram_quantile(0.95, sum by (instance, rpc, le) ( rate(mqlite_rpc_duration_seconds_bucket[5m]))) # Whole-request arithmetic mean, including authentication sum by (instance, rpc) (rate(mqlite_rpc_request_duration_seconds_total[5m])) / sum by (instance, rpc) (rate(mqlite_rpc_requests_total[5m])) # Newly observed dead-letter transitions, not net DLQ depth change increase(mqlite_message_events_total{event="dead_lettered"}[5m]) ``` Apply `rate`/`increase` to counters, not queue depth gauges. `delta` on depth is net retained-state change and cannot measure failure throughput. Do not subtract sends and receives to infer loss, count locked rows as live consumers, or sum the same database snapshot from duplicate collectors. Requests may contain batches. Alert thresholds should match the application's SLA and minimum request volume. ## Runbooks ### Scrape or collection failure Inspect the Prometheus target's `lastError` first. A down target can mean a stopped broker, route/TLS failure, invalid scraper credentials or an endpoint failure. With a working configured monitor credential, `up=1` and `collection_success=0` means metrics are reachable but the queue snapshot failed. Check bounded storage errors and broker logs; do not treat omitted gauges as zero. Preserve evidence before intervention. After recovery, verify a fresh successful snapshot and a controlled send/receive/complete canary. ### Queue backlog and dead letters Compare oldest active/locked age, depth, delivery and completion trends with the application's intended schedule. Check consumers, lease duration and ordering. Scheduled/deferred work may be intentional. Inspect dead-letter reasons and fix the failure before redrive. Retention, purge and redrive can reduce depth while new failures continue. Unexpected retained states require investigation, not deletion just to silence the graph. Filter failure is distinct from a normal non-match. ### Authentication and integration Use aggregate outcomes to distinguish missing/invalid credentials, revoked or expired keys and insufficient permissions. Client connection failures that never reach MQLite need client telemetry. Grafana datasource failure is a different hop from Prometheus scrape failure. Rotate the configured monitor token by allowing both old/new distinct monitoring tokens temporarily, restarting the broker, updating the scraper file, confirming at least two successful scrapes, then removing the old token and restarting. Never grant the scraper send/manage privileges to fix a routing error. ### Storage and maintenance Check the read probe, operation outcome, error category, retry count and pool waiting. `outcome_unknown` requires idempotent reconciliation; a blind resend can produce duplicates. A healthy read does not prove write availability. Check maintenance enabled status and cadence before declaring a stalled task. Reuse host/container monitoring for free disk, CPU, OOM and volume health. For disk-full, restore or integrity incidents, follow [operations.md](https://github.com/mqlitehq/mqlite/blob/main/docs/operations.md#incident-actions). ## Logs, alerts and validation Request logs retain operation/status context, queue and item counts; empty Receive responses are debug noise by default. Logs can help diagnose a slow nonempty request without placing payloads, credentials or unbounded errors into metric labels. Normal long polling can raise Receive latency without a storage problem. The [example rules](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/prometheus/rules.yml) cover sustained scrape, collection/storage failure, stale collection, unexpected states, backlog age, retained DLQ, repeated auth failures, uncertain outcomes, maintenance and filter failures. Tune them before production. Prometheus evaluates conditions; configure Alertmanager or the platform's notification receiver separately. The local demo's short `MQLiteDemoDeadLetter` rule is explicitly a test rule, not production policy. The [real stack verifier](https://github.com/mqlitehq/mqlite/blob/main/test/observability/verify.py) checks authenticated scraping, monitor isolation, canonical/native agreement, actual message identity and body, replay, redelivery, scheduled/deferred work, DLQ firing/resolution, TTL discard/dead-letter handling, retention deletion, configuration/routing filter errors, credential outcomes, Grafana provisioning and every panel query. Deliberate invalid queries and missing datasources verify that the checker rejects failures. `--faults` also interrupts only that isolated stack to verify scrape-auth recovery, Grafana connection failure/recovery and persistence/counter reset after broker restart. Managed-cloud/Kubernetes deployment validation remains environment-specific. DeleteQueue is an additive RPC observation extension. New brokers include its request counters and the `queue_not_empty` and `queue_changed` outcomes. The required decoder baseline is unchanged so new clients continue to accept observations from older brokers. An occupied-queue rejection is an application outcome, not a storage failure. # Cloud and Kubernetes monitoring These examples require **MQLite v0.3.2 or later**, including configured monitor credentials and the canonical metric families. MQLite exposes native Prometheus metrics on the API port when `MQLITE_METRICS=on` is set; it is disabled by default. The same dashboard and metric definitions work with a private Prometheus collector or a compatible managed service. Keep the broker's metrics authenticated: configure a distinct `MQLITE_MONITOR_TOKENS` credential, give only that credential to the collector, and keep administrator credentials out of monitoring configuration. See [observability.md](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) for the metric contract and the runnable local stack. ## Common integration ```text MQLite :6754/metrics <-- private route + monitor Bearer token -- collector | Prometheus-compatible storage | existing Grafana + rules ``` 1. Locate the collector in the broker's private network, or provide a TLS route reachable only from approved collectors. Configure the correct CA; do not disable TLS certificate verification. Enable `MQLITE_METRICS=on`. API and metrics share a port, so keeping `/metrics` off the public Internet requires either a private-only broker or public-ingress path filtering. A port-level firewall cannot separate HTTP paths, and Bearer auth does not hide the route. 2. Configure `/metrics` on the broker's API port (default `6754`), a 15-second starting interval and a shorter scrape timeout. Use a secret file or your collector's secret reference; do not put the credential into target URLs, labels or dashboard JSON. 3. Verify `up{job="mqlite"} == 1` and `mqlite_collection_success == 1`. Scrape success alone does not establish database availability or successful business processing. 4. Import the [standard dashboard](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/grafana/dashboards/mqlite.json), select the platform datasource, and adopt the [rule groups](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/prometheus/rules.yml). Keep the `job="mqlite"` label or deliberately adjust the rules to your collector. 5. Configure the platform's notification receiver and test delivery separately. Firing rules prove condition evaluation, not that an email or page was delivered. The [existing-broker job](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/prometheus/existing-broker.example.yml) is a reusable starting point. Prometheus supports Bearer authorization through `credentials_file`; a cloud console may expose a different secret-binding mechanism. Do not paste a local filesystem path into a managed service unless that service actually mounts that file. [Prometheus configuration reference](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). ## Alibaba Cloud: ECS or ACK For ECS, Managed Service for Prometheus supports application endpoints reachable by its collectors. Place the collector in the correct VPC and configure custom metric collection for the private broker address and port. Limit security-group access to the collector's actual source network; a publicly reachable metrics port is unnecessary. The official guide also covers reachable non-ECS machines in the same VPC. [Alibaba Cloud application metric collection](https://www.alibabacloud.com/help/en/cms/cloudmonitor-2-0/collect-application-metrics-of-ecs-deployment-to-the-prometheus-instance). Use your service's custom scrape job and credential support to supply the monitor Bearer token. Confirm the generated scrape configuration, target status and actual query result. If the selected managed collector cannot securely attach the header, run a small private Prometheus collector with the supplied job and use the platform's supported remote-write integration; do not switch off MQLite auth. ECS custom job discovery and update procedures are documented in [service discovery rules](https://www.alibabacloud.com/help/en/arms/prometheus-monitoring/manage-service-discovery-rules-for-ecs-environments). For ACK, use your installed collector's supported ServiceMonitor/PodMonitor or custom-job mechanism. The [Kubernetes examples](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/kubernetes/README.md) target upstream Prometheus Operator. Check the actual CRD version, namespace selectors and Secret access in your ACK installation before applying them. ## Fly.io Use the existing Fly Machine for MQLite and run Prometheus/Grafana locally or in an existing monitoring platform. No second Fly Machine is required for export. For private-only access: 1. Remove public `[http_service]` and `[[services]]` entries from `fly.toml`. 2. Set `MQLITE_ADDR=fly-local-6pn:6754` and `MQLITE_METRICS=on`, preserving administrator auth and a distinct configured monitor token. 3. Connect your collector through Fly's 6PN or a WireGuard peer and scrape `your-mqlite-app.internal:6754/metrics` with the monitor token. For a local inspection tunnel, run `fly proxy 6754:6754 --app `; a local Prometheus process can use `127.0.0.1:6754` as its target. A Docker collector needs a deliberate route to the host tunnel; its own `127.0.0.1` is the container. Fly's private network is scoped by organization/network configuration. It adds a network boundary; it does not replace MQLite authentication. These connections require the Machine to be running. See [private networking](https://fly.io/docs/networking/private-networking/), [app services](https://fly.io/docs/networking/app-services/) and [`fly proxy`](https://fly.io/docs/flyctl/proxy/). If the broker API must remain public, turning on metrics makes its route reachable through that same service. Use an existing ingress with an actual public `/metrics` deny rule while permitting private collection; otherwise leave metrics disabled. A monitor token restricts data access, but does not make a public path private. The examples do not assume Fly's public proxy filters paths. Fly's automatic custom-metrics stanza documents a port and path but no per-target MQLite Bearer credential. Adding `[metrics]` alone is therefore not a verified way to scrape this authenticated endpoint. Use the reusable authenticated Prometheus job and validate a real scrape. [Fly custom metrics](https://fly.io/docs/monitoring/metrics/). If you use Fly's managed Prometheus-compatible query endpoint in Grafana, its Fly access token authenticates Grafana to that service. It is separate from the MQLite monitor token used for scraping. Follow Fly's documented token-type authorization scheme. Fly's managed metrics service does not supply alert notifications; configure your Grafana/Prometheus alerting path explicitly. The official metrics guide explains these boundaries and its retention limits. [Fly metrics authentication and alerting](https://fly.io/docs/monitoring/metrics/). Keep the broker's single-writer SQLite volume model. Ensure the monitored Machine is running when a private collector scrapes it; do not assume a stopped Machine is healthy or that every private request uses Fly Proxy auto-start. Monitoring cadence and scale-to-zero policy need a deliberate choice for the actual routing setup. ## Azure: VM or AKS A private VM collector can use the same TLS/Bearer job. For AKS with Azure managed Prometheus, use its documented custom scrape mechanism and namespace-scoped Secret access. Azure's managed ServiceMonitor/PodMonitor CRDs use a different API group from upstream Prometheus Operator, so the supplied upstream manifest cannot be copied unchanged. Check the collector's supported fields, RBAC and Kubernetes version-specific requirements. [Azure custom scrape CRDs and authentication](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-scrape-crd). For an existing upstream operator installation on AKS, use the upstream [Kubernetes examples](https://github.com/mqlitehq/mqlite/blob/main/ops/observability/kubernetes/README.md). Do not run duplicate scrape jobs unless their replica labels and downstream deduplication are designed; otherwise dashboards may double-count process counters or duplicate database gauges. ## Production handoff checklist | Concern | Required check | | --- | --- | | Credentials | Monitor token differs from admin; no body access, queue management or key issuance; secret never appears in labels | | Networking | Collector can reach the selected private/TLS endpoint; public access remains restricted | | Persistence | Broker database and monitoring history have suitable volumes, retention and backup policies | | Rules | Thresholds fit application SLA; notification routing was actually tested | | Freshness | Missing scrape, failed queue collection and unavailable backend measurements remain distinguishable from zero | | Scale | Record scrape duration and series count; queue labels scale with configured queues, not message count | | Host health | Reuse cloud/node/container monitoring for disk space, CPU, memory, OOM and volume health | | Upgrades | Pin reviewed images; validate exporter protocol, dashboard queries and alerts after upgrade | No Alibaba Cloud, Fly.io, Azure managed service or Kubernetes cluster is created by the local demo. Provider setup above is reusable guidance, with environment-specific validation steps. Local integration evidence must not be presented as a managed-cloud deployment test. # Production operations This runbook covers v0.3.3 and upgrades from v0.3.0/v0.3.1/v0.3.2 (port 6754, schema token 5). Runtime-managed access keys require v0.3.1; v0.3.0 supports configured tokens only. Unified `Observe` and configured monitor credentials require v0.3.2. Upgrading from v0.2.0 changes both its default port (8080) and schema token (2); follow [upgrade and rollback](#upgrade-and-rollback) before replacing the broker. See [deployment](https://github.com/mqlitehq/mqlite/blob/main/docs/deployment.md) for installation and [observability](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) for metric definitions and alerts. ## Operating boundaries - Run one active broker per database. A local file has one MQLite owner and one SQL connection; use HTTP clients for other processes. A second broker sharing a Turso database is also unsupported: startup recovery assumes the previous owner has stopped. Do not perform a rolling overlap against the same database. - Keep local storage on a persistent local filesystem or attached block volume, with working flush semantics. SQLite WAL is unsuitable for a network filesystem. Size for the peak backlog, WAL, temporary operations, backups and log retention. - Use `MQLITE_SYNC=FULL` for production when an acknowledged commit must survive an OS or power failure. The default `NORMAL` protects against process crashes but can lose recent commits after power loss. `FULL` still depends on the storage stack honoring flushes; SIGKILL tests do not simulate a power failure. - Set stable `MQLITE_TOKENS`, terminate HTTPS, and keep the broker behind the proxy or a private network. Configured tokens are administrators; runtime keys can restrict sending or listening across the broker. There are no per-queue roles. Keep credentials in a restricted environment/secret file. - Consumers must be idempotent. Restarts, lost responses and restoring an older snapshot can repeat work whose external side effects already happened. Reuse stable `MessageID`/body, receive `AttemptID`, and the exact settlement request when retrying within their documented retention windows; do not treat an HTTP timeout as proof that a write failed. Choose and record the application's recovery objectives before launch: acceptable snapshot age (RPO), recovery time (RTO), backlog age, and capacity limits. Measure restore time with representative data. A successful backup command alone is not a restore test. ## Startup and routine checks Pin the binary checksum or image digest and record its source commit, Go version, MQLite version and schema token. Use the same configuration when rehearsing and running the service, especially durability, message limits and DLQ retention. `GET /healthz` is a process liveness check; its handler does not query storage. Use authenticated `AdminService/Status` to inspect the backend (`ping_ms >= 0`), then a dedicated canary queue to verify a complete write/consume/settle cycle. A successful read does not prove that disk space remains for a write. ```bash export MQLITE_ENDPOINT=http://127.0.0.1:6754 # Set MQLITE_TOKEN from your secret manager; do not paste it into shell history. mqlite status --output json mqlite create-queue ops-canary mqlite send ops-canary "canary-$(date -u +%s)" mqlite receive ops-canary mqlite metrics ops-canary --output json ``` The CLI `receive` completes messages after printing them unless `--no-ack` is set. The final canary queue total must be zero. Run one canary consumer at a time and keep it separate from business traffic. A periodic canary should check the returned body against its own sent identity and fail on an unexpected or missing message. Monitor process/container restarts, memory, free disk, DB plus WAL size, recent backup age, RPC errors and queue age/depth. See [observability](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) for the difference between backlog gauges and the completion counter. Review DLQ messages before retention removes them; retention is a bounded failure buffer, not an archive. Leave space for a new backup and WAL growth during a long read. ## Key rotation Use a separate managed key for each producer, consumer or administrator. A `manage` key can issue or revoke keys, including other administrators; a `send` or `listen` key cannot. Console administration requires `manage`; use distinct configured `MQLITE_MONITOR_TOKENS` for a read-only metrics scraper or monitoring view. These credentials do not rely on the key database. See the [monitoring starter](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md) and [cloud integration guide](https://github.com/mqlitehq/mqlite/blob/main/docs/observability-cloud.md). Create managed keys with `mqlite key create`, move clients to the replacement, then `mqlite key revoke --id `. These changes persist in the database and take effect without restarting the broker. Revocation rejects new authentication attempts; requests already authorized may finish. Keys issued by another key are independent, so revoking their issuer does not revoke them. Configured administrator rotation still uses `MQLITE_TOKENS=old,new`, a broker restart, client/scraper cutover, then removal of the old configured token and another restart. Quiesce clients during each replacement and verify the canary before resuming traffic. A token explicitly configured here remains an administrator even if a matching database key has expired or been revoked; remove that configuration to revoke the static grant. Keep a configured administrator available for recovery, and store secrets outside command history. ## Consistent backups These procedures are for a local SQLite file. Turso backups and point-in-time recovery belong to the remote service; rehearse that provider's restore process and keep one active MQLite owner during the switchover. Protect backups like the live database: they include message bodies, properties, managed key digests/permissions/revocations and any embedded outbox business tables. Keep a verified copy outside the database's failure domain, together with the matching binary/image identity and configuration needed to restore it. Store plaintext credentials separately. Restoring a snapshot also restores its key state: newer keys may disappear and keys revoked after the snapshot may become valid again. Before reopening client traffic, use a configured administrator to review key metadata and revoke or replace credentials whose state changed after the backup. ### Online snapshot Use a read-only connection to the live source and a new destination filename: ```bash umask 077 MQ_DB=/var/lib/mqlite/mq.db MQ_BACKUP_DIR=/var/backups/mqlite mkdir -p "$MQ_BACKUP_DIR" MQ_SNAPSHOT="$MQ_BACKUP_DIR/mq-$(date -u +%Y%m%dT%H%M%SZ).db" sqlite3 -readonly "$MQ_DB" '.timeout 10000' "VACUUM INTO '$MQ_SNAPSHOT'" test "$(sqlite3 -readonly "$MQ_SNAPSHOT" 'PRAGMA integrity_check')" = ok test -z "$(sqlite3 -readonly "$MQ_SNAPSHOT" 'PRAGMA foreign_key_check')" sqlite3 -readonly "$MQ_SNAPSHOT" "SELECT value FROM meta WHERE key='schema_version'" sha256sum "$MQ_SNAPSHOT" > "$MQ_SNAPSHOT.sha256" ``` Run the commands with error checking (`set -e` in a script). A failed or interrupted snapshot is not a backup; preserve the error and choose a new filename for the next attempt. Verify the checksum after copying the snapshot to backup storage. `VACUUM INTO` makes a transactionally consistent snapshot, including business and auxiliary tables. Do not copy only the main file while the broker is running, and do not write or run an external checkpoint against the live source. Keep the backup connection read-only and short-lived. SQLite's documented [WAL-reset issue](https://www.sqlite.org/wal.html#the_wal_reset_bug) requires multiple connections concurrently writing/checkpointing; MQLite's single connection and this backup restriction avoid that topology. See [dependency policy](https://github.com/mqlitehq/mqlite/blob/main/docs/dependencies.md). ### Offline directory copy Stop producers and consumers, then stop the broker and confirm it has exited. Copy the complete data directory while no process can change it; include the main DB and any remaining `-wal`/`-shm` files. A clean shutdown normally checkpoints and removes these files, but their absence must not be assumed after a crash. Keep the matching set together. Do not delete a WAL file to make a backup look self-contained. Restart the original service only after the directory copy finishes. For the systemd recipe, `systemctl stop mqlite` must finish before copying `/var/lib/mqlite/`. For Docker, stop the container and copy its named volume using a helper with a read-only source mount. A volume snapshot taken while the broker is still writing needs its own documented consistency guarantees. ## Restore into a fresh directory 1. Keep the original data and backup unchanged. Verify the backup checksum and run `integrity_check` and `foreign_key_check` on the saved snapshot or a working copy. Record its schema token and the binary version that created it. 2. Create an empty restore directory or volume. An online `VACUUM INTO` snapshot is a standalone DB; copy that file into the empty directory. For an offline copy, restore the complete matching directory. Never place a snapshot beside WAL/SHM files from another database lifetime. 3. Give the broker account read/write access to that directory. Start the matching binary against it on an isolated endpoint, with authentication and `FULL`, while business clients remain stopped. Do not attach the original database as well. 4. Verify configuration, subscriptions, message identities and body hashes against recorded expectations. On startup every orphaned lock is reclaimed: below the queue's delivery limit it becomes active; at the limit it goes to DLQ with `MaxDeliveryCountExceeded`. Delivery count and payload survive, and the old lock token is invalid. Old receive-attempt snapshots may replay old handles; that does not give those handles ownership of a recovered message. 5. Check deferred/scheduled/DLQ work and topic filters, then run the authenticated canary. A restore replays the snapshot's state: later completions, sends, dedup records and external business effects require reconciliation against the application's own records. TTL and retention clocks continue to matter. 6. Stop the isolated broker, point the service at the restored directory and start its single owner. Resume clients only after validation. Record actual recovery time and recovered snapshot age; retain the original for investigation. The repeatable [restore drill](https://github.com/mqlitehq/mqlite/blob/main/test/production/restore/README.md) verifies all schema tables and columns, complete payloads, auxiliary records, outbox data and the expected recovery transformations. Run it for a candidate and rehearse the same procedure on representative operational data before relying on a backup. ## Upgrade and rollback Check schema tokens before replacing a binary. Version numbers alone do not establish database compatibility, and MQLite does not run schema migrations. | Situation | Procedure | |---|---| | Same schema token | Quiesce clients, stop the old owner, take a consistent backup, start the candidate against a restored copy, validate, then replace the owner. Keep the old binary and snapshot for rollback. | | Different schema token | Keep the old database with its old binary. Account for all retained work, then create a new database using the candidate. Do not edit the schema token or point the candidate at the old file as a migration. | | Rollback before candidate accepts business writes | Stop the candidate and restore the matched old binary plus old snapshot/configuration; verify before resuming clients. | | Rollback after candidate accepts business writes | First preserve candidate data and reconcile new messages and external effects. Blindly restoring the old snapshot would discard those new writes. | For **v0.3.2 → v0.3.3**, schema token **5**, port **6754**, message data and managed keys remain compatible. Queue generations use additive rows in the existing metadata table; no table migration is required. New diagnostics do not change the storage retry policy. If enabled, keep the private diagnostic log on persistent storage and retain it separately from database backups. Rolling back to v0.3.2 preserves data but does not support DeleteQueue or detailed diagnostics; remove `--diagnostic-log` from older CLI invocations. Stop clients before retiring a queue name and keep the original deletion generation when reconciling uncertain outcomes. For **v0.3.1 → v0.3.2**, schema token **5**, port **6754**, message data and managed-key storage remain unchanged. Follow the same-schema procedure above; do not overlap brokers. Validate existing key permissions and a complete canary cycle, then check `Observe`, the monitor credential and the new dashboard/rules. Prometheus scraping is now disabled by default: set `MQLITE_METRICS=on` or use `--metrics` to retain `/metrics` on the API port, and keep the route off public ingress. Configure the collector with its distinct monitor token. Process counters reset at restart; existing Prometheus history is separate from the broker database. When rolling back to v0.3.1, stop the candidate before starting the old binary. Existing managed keys retain their permissions, but v0.3.1 has no `Observe` endpoint or configured monitor credentials. Pause v0.3.2-only observers and restore the previous authorized scraper/rule configuration; the old `/metrics` endpoint requires an administrator. Keep administrator authentication enabled. Preserve writes accepted by the candidate rather than blindly restoring the pre-upgrade snapshot. The two deprecated metric aliases remain available in v0.3.2; see [metric compatibility](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md#compatibility-from-v031). For **v0.3.0 → v0.3.1 or v0.3.2**, schema token **5** and the port remain unchanged. The candidate adds the independent `access_keys` table without rewriting existing queue or message tables. Rehearse opening a restored copy first. Rolling back to v0.3.0 preserves that extra table, but the old binary cannot authenticate managed keys or enforce their permissions: reconnect clients with configured administrator tokens, or keep them stopped until v0.3.1 or later is restored. A database with an unrelated, incompatible table named `access_keys` is rejected without modifying it. For **v0.2.0 → v0.3.0**, schema **2 → 5** and default port **8080 → 6754** both change. Before the cutover, stop producers and account for **every queue and subscription in every state**, including future scheduled work, deferred work and DLQ messages. Drain retained messages with the old version, or make an explicit application-level replay plan with stable business identities. A queue that has no active messages may still contain scheduled, locked, deferred or DLQ work. Create the candidate database in a new volume/directory; update the port, reverse proxy, clients and probes together. Preserve the old binary, configuration and snapshot. The new binary must reject a real schema-2 file and the old binary must reject a schema-5 file without changing its logical contents; the restore drill checks both directions. There is no general message export/import migration tool in this release, so do not improvise a direct SQL table copy between schemas. ## Incident actions | Symptom | First actions | Verify before resuming | |---|---|---| | Full disk / `SQLITE_FULL` | Stop ingress, preserve logs and acknowledged message identities. Expand storage or remove unrelated expired logs/backups; never remove the live DB or WAL. A full `VACUUM` needs working space and is unsuitable as an emergency first step. | Backend status, a fresh canary, successful replay/reconciliation of ambiguous requests, and retained-message integrity. | | Repeated broker crashes | Stop restart loops long enough to preserve the complete DB directory and logs. Check disk, memory/OOM and configuration. Restart only one owner. | Recovery of orphaned locks, DLQ reason/count, canary and consumer idempotency. | | Old backlog / growing DLQ | Check consumers, lock expiry/renewal and handler failures. Inspect reasons and payloads; fix the cause before bounded redrive. | Backlog age falls and verified redriven identities complete. | | Storage/remote timeouts | Check backend reachability and latency. Treat lost write responses as ambiguous; keep retries within the idempotency contract. | Authenticated status plus a write/consume canary. | | `ErrSchemaVersionMismatch` | Stop the attempted cutover. Recover the matching binary/database pair. | Schema token and the correct old or new port; no token restamping. | [SQLite backup semantics](https://www.sqlite.org/lang_vacuum.html) and [WAL file handling](https://www.sqlite.org/wal.html#the_wal_file) describe the storage rules behind these procedures. # MCP server (`mqlite-mcp`) `mqlite-mcp` is a [Model Context Protocol](https://modelcontextprotocol.io) server that exposes the mqlite broker as **agent tools** — so an AI agent (Claude, etc.) can observe broker health, create queues, send, receive, settle messages, and manage access keys without writing any HTTP. It is a thin, **dependency-free** forwarder: it speaks MCP (JSON-RPC 2.0 over stdio) and turns each tool call into one HTTP POST to the broker. Stdlib + the in-repo `wire` contract only — no MCP SDK, no CGO (the same ethos as the rest of mqlite). ## Build & run ```bash make build # → bin/mqlite-mcp (also: go build ./cmd/mqlite-mcp) ``` It's a stdio server: an MCP host launches it and talks over stdin/stdout. Configure it with the broker endpoint + token: | Env | Default | Meaning | |---|---|---| | `MQLITE_ENDPOINT` | `http://127.0.0.1:6754` | the broker base URL | | `MQLITE_TOKEN` | — | configured administrator, monitor credential, or managed broker access key; tool calls follow its permissions | ## Connect an agent host Most MCP hosts take a JSON server entry. For example: ```json { "mcpServers": { "mqlite": { "command": "/path/to/bin/mqlite-mcp", "env": { "MQLITE_ENDPOINT": "https://your-mqlite.fly.dev", "MQLITE_TOKEN": "mqk_prod_xxx" } } } } ``` The agent then sees the tools below and can drive the broker directly. ## Tools Minimal — selected core routes map 1:1 (kept small on purpose — over-specified tools make models misuse them): | Tool | Does | |---|---| | `observe` | complete canonical observation; takes no arguments; requires monitor or manage | | `list_queues` | list queues/subscriptions | | `create_queue` | create/update a queue by name | | `send` | send a message (`queue`, `body`, optional `message_id`/`group_id`) | | `receive` | peek-lock messages → returns `seq_number` + `lock_token` | | `complete` | acknowledge a message (`queue`, `seq_number`, `lock_token`) | | `abandon` | release a message for redelivery | | `reject` | dead-letter a message | | `renew` | extend a message's lock so long-running work isn't redelivered | | `defer` | set a message aside for later retrieval by `seq_number` | | `receive_deferred` | retrieve deferred messages by their `seq_numbers` | | `peek` | browse without locking (optionally by `state`) | | `stats` | queue counters by state | | `redrive` | move dead letters back to active | | `purge` | permanently delete dead letters | | `create_key` | create a managed key (`id`, `name`, `permissions`, optional `expires_at_ms`); returns the token once | | `list_keys` | list public key metadata, including revoked/expired keys (`after_id`, `limit`, `sort`) | | `revoke_key` | revoke a managed key by public `id`; repeated revocation is safe | Settlement is by `lock_token` from `receive` — delivery is at-least-once, so an agent should treat handlers as idempotent. Full HTTP semantics: [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md). ## Read-only observation The `observe` tool requires **v0.3.2 or later** of both `mqlite-mcp` and the broker. Configured monitor credentials also require a v0.3.2 or later broker. Call `observe` with `{}` to get the same canonical JSON snapshot as the HTTP route, Go SDK and CLI: availability/freshness, queue gauges, process message effects, storage operations, maintenance, filters and HTTP measurements. The tool does not recompute counters. An unavailable queue collection remains explicit even when the HTTP request succeeds; inspect `collection.state` and runtime availability flags. Process counters reset when the broker restarts. See [observability.md](https://github.com/mqlitehq/mqlite/blob/main/docs/observability.md). Use a credential configured in the broker's `MQLITE_MONITOR_TOKENS` as this process's `MQLITE_TOKEN` for a read-only monitoring agent. The tool inventory stays the same, but the broker permits only `observe` for this credential; other tool calls return `permission_denied`. Managed `send`/`listen` credentials cannot observe the whole broker, and monitor is not a managed-key permission. ## Access key management The three key tools require `manage`; both an environment administrator token and an active managed `manage` key may issue further administrator keys. `send` and `listen` keys receive `permission_denied` for these tools. The MCP server forwards `MQLITE_TOKEN` on every call, and broker authorization decides what it may do. Before calling `create_key`, choose and retain a unique 32-character lowercase hexadecimal `id`. The `id` is public metadata, distinct from the secret token: ```json { "name": "create_key", "arguments": { "id": "e734810a9f634d75bd8a029c1f7e5062", "name": "order-producer", "permissions": ["send"] } } ``` `permissions` accepts `send`, `listen`, their combination, or `manage` (which includes both). `expires_at_ms` is a future UTC epoch-millisecond timestamp; omit it or use zero for no expiry. The response has `key` metadata and a one-time `token` in the `mqk_` + 64 lowercase hex format. Store the token securely; it will be visible in the successful MCP tool result and may be retained by your MCP host. List responses never contain secrets or token hashes. Creation is not retried automatically. An error includes the retained valid public ID; after an uncertain result, use `list_keys` and `revoke_key` for that ID before issuing a replacement with a new ID. Reusing an ID returns `key_conflict` and cannot recover a secret. Pass `next_after_id` from `list_keys` back as `after_id` until the cursor is absent; the default limit is 100, maximum 1000. The optional `sort` is `id_asc` (default) or `created_desc` (newest-created first, then descending public ID for timestamp ties). Keep the same sort on every page. Creation-order cursors must identify an existing key, including revoked or expired keys. The default ID order also accepts absent predecessor IDs when reconciling a lost create response. Environment tokens are not listed or revocable through these tools. Key tools are unavailable when broker authentication is explicitly disabled. # Examples For authenticated producer/consumer credentials, Go SDK key management, console and MCP workflows, see [Access keys](https://github.com/mqlitehq/mqlite/blob/main/docs/access-keys.md) (v0.3.1 and later). Copy-runnable examples for the four ways to use mqlite: **embedded** in-process, the **transactional outbox** (the killer feature), a **remote** client with a hands-off consumer, and raw **curl**. Full API: [api-reference.md](https://github.com/mqlitehq/mqlite/blob/main/docs/api-reference.md). ## 1. Embedded (in-process, no broker) One SQLite file, no network, no second process. ```go package main import ( "context" "fmt" "log" "time" "github.com/mqlitehq/mqlite" ) func main() { ctx := context.Background() mq, err := mqlite.OpenEmbedded(ctx, "file:./mq.db") if err != nil { log.Fatal(err) } defer mq.Close() if err := mq.CreateQueue(ctx, "orders", mqlite.QueueConfig{ LockDuration: 30 * time.Second, MaxDeliveryCount: 5, }); err != nil { log.Fatal(err) } // produce if _, err := mq.SendOne(ctx, "orders", mqlite.OutMessage{Body: []byte(`{"id":42}`)}); err != nil { log.Fatal(err) } // consume (Peek-Lock): receive, do work, then Complete msgs, err := mq.Receive(ctx, "orders", mqlite.RecvOpts{Max: 10, Wait: 5 * time.Second}) if err != nil { log.Fatal(err) } for _, m := range msgs { fmt.Printf("got: %s\n", m.Body) if err := m.Complete(ctx); err != nil { // or m.Abandon(ctx) to retry, m.Reject(ctx) -> DLQ log.Fatal(err) } } } ``` ## 2. Transactional outbox (the killer feature) Commit a business write **and** the enqueue in **one** SQLite transaction — the message is enqueued if and only if the business change commits. No dual-write race, no separate outbox poller. Embedded-only (same DB). ```go package main import ( "context" "log" "github.com/mqlitehq/mqlite" "github.com/mqlitehq/mqlite/engine" // EngineTx + engine.OutMessage are used inside Tx ) func main() { ctx := context.Background() mq, err := mqlite.OpenEmbedded(ctx, "file:./app.db") if err != nil { log.Fatal(err) } defer mq.Close() _ = mq.CreateQueue(ctx, "order-events", mqlite.QueueConfig{}) // Your app tables live in the same DB. Tx gives you that transaction. err = mq.Tx(ctx, func(tx *engine.EngineTx) error { // 1) the business write, on the SAME transaction. Use tx.Context(), NOT the ctx you // passed to Tx: on a local store, cancelling a statement mid-transaction leaks the // SQLite connection and wedges the database. if _, err := tx.SQL().ExecContext(tx.Context(), `INSERT INTO orders(id, total) VALUES(?, ?)`, 42, 1999); err != nil { return err // rollback: nothing enqueued } // 2) enqueue in the same tx — commits atomically with the write above: _, err := tx.SendOne("order-events", engine.OutMessage{Body: []byte(`{"order":42}`)}) return err }) if err != nil { log.Fatal(err) } // Either both happened, or neither did. } ``` For local transactions, `tx.Context()` is deliberately not cancellable. Raw `tx.SQL()` statements can still execute after the original caller cancels: check `ctx.Err()` between business statements and return promptly. Mqlite-owned statements such as `tx.SendOne` have their own checks. Cancellation observed before commit rolls back all the transaction's writes, even if the callback returns nil. See the [cancellation contract](https://github.com/mqlitehq/mqlite/blob/main/docs/conformance.md#13--cancellation-context-deadlines). ## 3. Remote client + hands-off consumer Talk to a running broker over HTTP. The `Receiver` runs a managed consume loop: return `nil` → Complete, return an error → Abandon (with auto lock-renewal and configurable concurrency). ```go package main import ( "context" "log" "github.com/mqlitehq/mqlite" ) func main() { ctx := context.Background() cli, err := mqlite.Open(ctx, "https://your-mqlite.fly.dev", mqlite.WithToken("mqk_prod_xxx")) if err != nil { log.Fatal(err) } defer cli.Close() // produce if _, err := cli.SendOne(ctx, "orders", mqlite.OutMessage{Body: []byte("hi")}); err != nil { log.Fatal(err) } // consume: nil -> Complete, error -> Abandon; auto-renew the lock; 4 in flight. err = cli.Receiver("orders", mqlite.WithAutoRenew(), mqlite.WithConcurrency(4)). Run(ctx, func(ctx context.Context, m *mqlite.Message) error { return process(m.Body) // MUST be idempotent (at-least-once) }) if err != nil { log.Fatal(err) } } func process(body []byte) error { log.Printf("got: %s", body); return nil } ``` ## 4. curl (raw HTTP) Any language can drive the broker — it's one POST per operation. (`body` is base64.) ```bash HOST=https://your-mqlite.fly.dev T=mqk_prod_xxx curl $HOST/ # discovery (open, no auth) curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"name":"orders","config":{}}' \ $HOST/mqlite.v1.AdminService/CreateQueue curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data "{\"queue\":\"orders\",\"messages\":[{\"body\":\"$(printf hi | base64)\"}]}" \ $HOST/mqlite.v1.QueueService/Send # receive -> returns seq_number + lock_token curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"queue":"orders","max_messages":1,"wait_time_ms":5000}' \ $HOST/mqlite.v1.QueueService/Receive # complete with the lock_token from the receive curl -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ --data '{"queue":"orders","seq_number":1,"lock_token":""}' \ $HOST/mqlite.v1.QueueService/Complete ```