Core concepts & subscription filters

Goal: explain MQLite's model once, clearly, and place it against the comparable services from Azure Service Bus, AWS (SQS + SNS) and GCP Pub/Sub. MQLite is designed from first principles (one SQLite file, embeddable, agent-friendly), but conceptually it sits closest to Azure Service Bus, so ASB is the primary reference frame and the differences are called out where they matter. Sections 1–5 are the model; sections 6–11 are the subscription-filter language (expr).

1. The whole picture in one diagram

  ┌──────────────────────────── two delivery targets ─────────────────────────┐
  │                                                                            │
  │   A) send straight to a Queue        B) publish to a Topic, fan out        │
  │                                                                            │
  │   producer                           producer                              │
  │      │ send "orders"                    │ publish "events"                 │
  │      ▼                                  ▼                                   │
  │   ┌────────────┐                ┌──── topic "events" ────┐                 │
  │   │ queue      │                │ (just a set of subs;   │                 │
  │   │  orders    │                │  stores nothing itself)│                 │
  │   │ ┌────────┐ │                └─┬─────────┬──────────┬─┘                 │
  │   │ │ active │ │       filter match│  match  │  no match│                  │
  │   │ ├────────┤ │                  ▼         ▼          ✗ (no copy)         │
  │   │ │ locked │ │            sub "audit"  sub "billing"                     │
  │   │ ├────────┤ │            ┌────────┐   ┌────────┐                        │
  │   │ │  DLQ   │ │            │backing │   │backing │  ← each subscription   │
  │   │ └────────┘ │            │ queue  │   │ queue  │    is an independent   │
  │   └────────────┘            └────────┘   └────────┘    queue (+ own DLQ)   │
  │      consumer                  consumer     consumer                       │
  └────────────────────────────────────────────────────────────────────────────┘

One sentence to remember: a Topic stores nothing — it is only a routing rule that copies a message into a number of subscription queues. The thing that actually stores messages, can be consumed, and can dead-letter is always a Queue (including the backing queue behind a subscription).

2. The five concepts

Queue

A named message container, and the only entity that actually stores messages. A message runs this state machine — a row is in exactly one state at a time; below count = delivery_count, max = the queue's max_delivery_count, and ✗ removed means the row is deleted from the database:

                   ┌───────────┐
  Send(scheduled)  │ scheduled │   scheduler, when due ─► active
  ───────────────► └─────┬─────┘   cancel ─► ✗ removed (never-delivered only)
  Abandon(delay_ms>0) ───┘  (backoff parking: holds its group on ordered queues)
                         │
                         ▼
  Send             ┌──────────┐ ◄── requeue: immediate abandon / lock expiry, count < max
  ───────────────► │  active  │ ◄── redrive: from dead_lettered (count → 0)
                   └────┬─────┘
                        │ receive / claim (count++)
                        ▼
                   ┌──────────┐ ──── defer ──────► ┌──────────┐
                   │  locked  │ ◄─ ReceiveDeferred  │ deferred │
                   └────┬─────┘    (by seq)         └──────────┘
                        │ complete ─► ✗ removed
                        │ renew ─► (stays locked)
                        │ reject, or abandon / lock-expiry, count ≥ max:
                        ▼
                   ┌───────────────┐ ◄── TTL expiry (active/locked/deferred/scheduled;
                   │ dead_lettered │      ✗ removed if dead_letter_on_expire = 0)
                   └───────┬───────┘
                           │ purge / DLQ retention ─► ✗ removed

Every transition, with its trigger and the condition under which it fires:

• The DLQ is not a separate queue. It is messages in the same queue with state='dead_lettered'. A message lands there when its delivery_count reaches max_delivery_count (i.e. ≥), on an explicit reject, or on TTL expiry (when the queue's dead_letter_on_expire=1). Inspect with peek state=dead_lettered, send back with redrive, or delete with purge.
• Lease deadlines fence every settlement and renewal. A new effect requires the current token and locked_until_ms > now; equality is expired, even before the reaper runs. Renew before that deadline. A live receipt can replay an already committed settlement only for the same queue, sequence, token, operation and effect-bearing arguments, including after the original lease expires.
• Two redelivery paths. Abandon is an explicit, client-driven settlement; lock expiry is automatic — the reaper (~1s) reclaims a lock held past lock_duration without settling. Both redeliver while count < max and dead-letter once count ≥ max.
• Ordering survives lock expiry. On ordered paths (group_fifo, strict_fifo, and grouped messages on a standard queue — they share the same claim rule) an expired-but-not-yet-reaped lock still blocks its group (strict: the whole queue): successors are never delivered ahead of the expired head. Once the reaper runs, the head is redelivered first, in id order — or dead-lettered there and then if it has exhausted max_delivery_count, which also unblocks the group. The cost is that a consumer timeout stalls its group for up to one reaper interval (~1s) — the same trade SQS FIFO makes. A slow-but-alive consumer should Renew its lock rather than let it lapse. An explicit Abandon releases the head immediately; with a delay_ms backoff the head parks in scheduled until the backoff lapses and — like a scheduled send — keeps blocking its group (strict: the whole queue) until it is redelivered first, so order holds across a backoff (MQLITE-66).
• You can have any number of independent plain queues.

Topic

A logical name standing for "the set of subscriptions attached to it".

• A topic is not an explicitly created/stored object — its existence is exactly "there are rows in subscriptions with topic=<name>". A topic with zero subscriptions does not exist.
• A topic holds no messages of its own: no inbox, no buffer, no retention.

Subscription

The relationship binding a Topic to a backing queue, with an optional Filter.

• Each subscription has its own independent backing queue (own consumption, own seq_numbers, own DLQ).
• A subscription belongs to exactly one topic (guarded by ErrNameConflict).
• "A topic has many queues" is, precisely: each subscription owns one queue, and the topic fans out through those subscriptions.

Filter

A single expr-lang boolean expression on a subscription, evaluated per message at publish.

• Empty expression = match everything.
• e.g. subject_parts[0] == "orders" && properties["tier"] == "gold".
• Compiled and type-checked at Subscribe; a bad one is rejected (ErrInvalidFilter → 400). See §6–11 below for the full filter language and expression environment.

Provisioning — what you create vs. what is created for you

• There is no "create topic" operation and no same-named queue behind a topic. The backing queue is named after the subscription, not the topic.
• You never pre-create a queue to attach a subscription to — one Subscribe call builds the backing queue and the fan-out mapping.
• One flat namespace, disjoint by construction. Plain-queue names, subscription names (= their backing-queue names) and topic names never overlap; conflicts fail loud with ErrNameConflict (HTTP 409) at creation time, in both directions: Subscribe rejects a topic that names an existing queue or subscription, a subscription name that belongs to a plain queue, to another topic's subscription, or to a live topic, and the degenerate topic == name; CreateQueue rejects a name that is live as a topic, and a cross-kind upsert (a plain CreateQueue over a subscription's backing queue, or kind=subscription over a plain queue) — a deliberate same-kind reconfig stays open. (ASB behaves the same way: queues and topics share one entity namespace, so a same-name create conflicts instead of silently rerouting sends.) The guards apply to creations from this version on; a pre-existing colliding pair in an old DB keeps resolving topic-first and now fails loud on any attempt to re-create either side.
• Name resolution at publish/send: a name with subscription rows → topic (fan-out); otherwise it must be an existing queue; otherwise the call errors. Because the namespace is disjoint, this resolution is unambiguous — a name can never be a topic and a queue at the same time, so a Send is never silently rerouted.

You only ever do two kinds of "create":
  CreateQueue(name)             -> one independent work queue (must exist before Send)
  Subscribe(topic, sub, filter) -> auto-creates backing queue=sub + attaches to topic
                                    (the topic comes into being as a side effect)

Difference from ASB: in ASB both the Topic and the Subscription are created explicitly; MQLite makes "create topic" implicit and "create the subscription's queue" automatic — two fewer steps.

Message ID vs Sequence Number (the two ids people conflate)

3. Direct answers to common questions

Q: A message fans out to two subscriptions — are the IDs the same? The message_id is the same (copied verbatim); the seq_number is different (the broker allocates each copy a distinct id from one database-wide monotonic sequence). The two are fully independent rows, settled separately with their own seq_number + lock_token.

Q: I add a filtered subscription under a topic — do historical messages get routed to it? No. Fan-out happens only at publish time, against the subscriptions that exist then. There is no replay, no offset rewind. A new subscription only receives messages published after it was created (and matching its filter). (Unlike Kafka; same as ASB / SNS+SQS.)

Q: A topic gets a message with no subscriptions at the time — dropped? buffered? a default inbox? No inbox; the message is not retained. Two cases: ① subscriptions exist but none match → the message is silently dropped (not an error); ② no subscription names that target at all → it falls back to "must be an existing queue", delivered if one exists, otherwise an error. To retain a message you must have a matching subscription before it is published.

4. Comparison with cloud services

Concept mapping

Behavioral differences (the ones that bite)

One-line positioning

• Like ASB: peek-lock + DLQ + Topic/Subscription + Filter + SequenceNumber + dedup map almost one-to-one. If you know Azure Service Bus the learning curve is near zero.
• Unlike Kafka / parts of GCP Pub/Sub: MQLite is subscription-based, keeps no log, cannot replay. It is a "work queue + topic fan-out", not a "message log / stream". For replay / event sourcing choose something like Kafka.
• MQLite's own trade-off: the whole broker is a single SQLite file — embeddable in-process, zero-ops, agent-friendly (ships with an MCP server). The price is exactly the "no replay / no topic retention" simplifications above — deliberate, not missing.

5. Speedbumps (easy to trip on)

• Treating a Topic as "an inbox that stores my messages" — it doesn't; subscribe first.
• Assuming fan-out copies share one seq_number — they don't; settle each separately.
• Assuming a new subscription back-fills history — it can't.
• Treating the DLQ as a queue you create separately — it is a state of the same queue.
• Publishing to a zero-subscription topic name expecting it to buffer — it actually looks for a same-named queue, or errors.

Subscription filters (expr)

A subscription filter decides, per message, whether this subscription receives a copy of a published message. A filter is a single expr-lang boolean expression — for example:

subject_parts[0] == "orders" && properties["tier"] == "gold"

An empty filter matches every message. The expression is type-checked and compiled once when you subscribe (a bad one is rejected immediately with 400 invalid_argument) and run once per message at publish time (the per-message gate in the topic fan-out from §1–§3):

 Subscribe ──► compile + type-check ──► cache the program     (bad expr → 400)
 Publish   ──► for each subscription: run program(message) ──► route when true

The filter environment contains the message's own fields and timestamps. Use enqueued_at as the publish-time reference for repeatable routing decisions. The upstream now() builtin is also available and reads the wall clock, so an expression using it can produce different results for the same message.

6. Setting a filter

Re-subscribing with the same name and a new expr replaces the filter (recompiled on the next publish). Omit the filter (or use an empty expr) to receive everything.

7. The message environment

The variables a filter can reference. Unknown names are a compile error (so a typo is caught at Subscribe, not silently at runtime).

Core

Time

Both are the message's own timestamps (epoch-derived time.Time, UTC), not a wall-clock read.

Compute a delay by subtraction (time - time → duration):

visible_at - enqueued_at > days(1)        # only route significantly-delayed messages
enqueued_at.Hour() >= 9 && enqueued_at.Hour() <= 21   # business-hours publish window

Derived

Computed from the message; always defined (cannot error on absence).

Body content

Route on the payload itself. These are projected only when your filter references them, so filters that don't touch the body pay nothing.

body_json is only decoded when content_type looks like JSON (or is unset); an explicit non-JSON type, an empty body, invalid JSON, or a non-object JSON (array/scalar) all yield an empty object {}. So body_json itself is never null — but reaching into an absent field (body_json.amount when there is no amount) yields nil, and comparing that (nil > 100) is a runtime error → fail-closed (the message isn't routed to that subscription, logged). Guard with a presence check when the field may be missing:

"amount" in body_json && body_json.amount > 100

8. Language

Standard expr operators and builtins are available:

• comparison == != < <= > >=, boolean && || ! (or and/or/not), membership in
• strings: startsWith, endsWith, contains, matches (regex)
• collections: len(), all(), any(), none(), filter(), map(), indexing x[0]

Durations

For the time fields, durations are spelled with type-checked helpers (recommended, unambiguous) or an extended duration():

There is no month/year unit — calendars make them ambiguous as fixed spans. For "older than a month", use days(30).

visible_at - enqueued_at == hours(2)
visible_at - enqueued_at > duration("1d")

expr-lang reference

The filter engine is expr-lang/expr (pinned at v1.17.8), so its full language — every operator, literal form, and built-in — is available and documented upstream. The message environment in §7, the duration helpers above and the fail-closed boolean contract in §10 are MQLite-specific; everything else is stock expr. When in doubt about a builtin's exact behavior, the upstream reference is authoritative.

The evaluator exposes no file or network I/O APIs and is sandboxed (memory-safe, side-effect-free, always-terminating; see §10). This does not remove the upstream now() builtin: prefer message-derived timestamps when routing decisions must be repeatable. The duration helpers days()/weeks()/duration("…d…w") are MQLite extensions on top of expr's date functions, not part of stock expr.

9. Filter examples

subject_parts[0] == "orders"                              # topic hierarchy
properties["tier"] == "gold" && properties["region"] == "eu"
"priority" in properties && properties["priority"] == "high"
subject startsWith "payment." and not (properties["test"] == "true")
body_size < 4096                                          # small messages only
visible_at - enqueued_at > days(1)                        # delayed > 1 day
len(subject_parts) >= 2 && subject_parts[1] == "eu"
"amount" in body_json && body_json.amount > 100           # route on payload (guarded)
body_text contains "urgent"

10. Filter safety

Filters evaluate the message environment and available expr builtins. No file or network I/O APIs are exposed; now() can read the wall clock:

• expr is memory-safe, side-effect-free, and always-terminating by design (no file or network access, no unbounded loops).
• A filter must be a boolean (expr.AsBool); a non-bool expression is rejected at Subscribe.
• Resource bounds: the source length and compiled AST size are capped.
• Fail-closed: if a filter errors at runtime (e.g. subject_parts[3] on a two-part subject) or panics, the message is not routed to that subscription and the error is logged — never a broker crash, never a silent match.

11. Further reading

• expr language — source github.com/expr-lang/expr, full reference at https://expr-lang.org/docs/language-definition (see §8 for the per-category builtin links).
• Why expr — dependencies.md explains why expr-lang/expr is a stable long-term dependency (pinned v1.17.8).
• Normative invariants — the conformance spec (conformance.md §11) for the filter rules MQLite guarantees.