DLQ + Replay

Catches the operations that fail because a dependency was down, at the call sites that opt in, with the context needed to run them again, so nothing is silently lost — and once the dependency recovers, the backlog replays.

What is it?

When an operation in your app fails because something it relies on is unavailable (a database timeout, a payment provider returning 503, a webhook endpoint that won't answer), that unit of work usually just disappears. The request errors out, maybe a line lands in a log file, and the actual work (the payment, the job, the message) is lost.

A dead letter queue (DLQ) is the standard name for a holding area where failed work is parked instead of thrown away. Think of the "undeliverable mail" bin at a post office: a letter that couldn't be delivered isn't shredded; it's set aside so it can be looked at and sent again later. Replay is the act of taking that parked work and running it again. In Baldur these two halves are a single feature: DLQ + Replay.

Why it matters

Without a dead letter queue, an outage doesn't only cause errors while it is happening — it causes permanent data loss. Every payment, job, or message that failed during the outage is gone, and recovering it means digging through logs and reconstructing the work by hand, if it can be recovered at all.

DLQ + Replay turns that permanent loss into a recoverable backlog:

No silent loss. At a call site that opts in (dlq=True), a call that fails for good is captured together with the forensic context (what was being done, the request data, the failure reason, the per-attempt retry history) needed to understand and re-run it. That includes the calls an already-open circuit breaker rejected before they ran, and the entries still buffered in memory when a worker process exits. The failures that skip the queue, each by a stated rule, are listed under How it works in Baldur.
Recover on your schedule. When the dependency comes back, replay the backlog instead of rebuilding lost work from log files.
Catch-up can be automatic. Once its prerequisites are in place (see Closing the loop), Baldur replays queued work when a tripped circuit breaker for that dependency recovers, so the backlog drains itself without an operator watching the clock.
Even repeat failures aren't a dead end. A failure that keeps failing isn't replayed forever or silently dropped. Once it exhausts its replay budget it is parked for review instead of looping. After you fix the underlying cause you can deliberately re-drive it, so even a "poison-pill" failure is recoverable rather than lost.
The queue protects itself. Size limits (overall and per-domain) plus an overflow strategy keep a failure storm from filling your storage and dragging the rest of the system down with it.

How it works in Baldur

When an operation Baldur protects with dlq=True fails for good, it is captured as an entry in the dead letter queue, recording the context needed to replay it later. That holds whichever of these ended the call: retries that ran out, retry switched off (BALDUR_RETRY_ENABLED=false), a TenacityBridgePolicy as retry=, Baldur's own timeout= cutting the call off (retry or not), or a full bulkhead refusing it. The entry is filed under the name the call is protected under, in its domain form: lowercased, with a character such as - turned into _, so a Payment-API call is filed as payment_api. A replay handler is filed the same way, so one declared as Payment-API replays those entries. The one exception is a RetryPolicyConfig that names its own domain=, whose retry stage files the failures it ends under that domain. A name with no domain form of its own, such as one that starts with a digit (3ds-auth), is filed under the shared catch-all OTHER_DOMAIN (logged as a WARNING domain.input_rejected), where a handler declared under that name does not find it. A call that never ran because its circuit breaker was already open is captured as well: the breaker rejects it in microseconds, but the work that call carried is parked under the breaker's own name with the failure type CIRCUIT_BREAKER_OPEN, so an outage's fast-rejected calls are recoverable alongside the ones that failed. (This capture is on by default and can be switched off.)

Some calls are not parked, by rule. Each one leaves a log record (within the rate limit described below the table), and several of those records sit below the default BALDUR_LOG_LEVEL=WARNING, so each rule names the record and its level — the level to turn on when you are looking for a call that is not in the queue:

A failed call is not parked when…	Its record
a `fallback=` answered it (the caller got a value)	`policy_chain.fallback_applied` (WARNING)
the call's own idempotency guard (`idempotency_key=`) refused it — already completed, still in flight, or unverifiable	`idempotency.duplicate_blocked`, `idempotency.execution_blocked`, `idempotency.guard_check_failed` (WARNING)
a preset pipeline's error-budget guard refused it (PRO)	`policy_pipeline.execution_rejected` (WARNING)
Baldur runs observe-only (dry-run, shadow, evaluation)	`execution_mode.intervention_suppressed` with `action="dlq_store"` (INFO)
open-circuit capture is switched off (`BALDUR_DLQ_OPEN_CIRCUIT_CAPTURE_ENABLED=false`)	`dlq_sink.capture_skipped` with `reason="open_circuit_capture_disabled"` (DEBUG)
its `retry=` is a `RetryPolicy` you built with `enable_dlq=False` and that stage ended it (a call `timeout=` cuts off first is parked)	`dlq_sink.capture_skipped` with `reason="stage_declined"` (DEBUG)
the queue is switched off (`BALDUR_DLQ_ENABLED=false`)	`dlq.store_skipped_disabled` (DEBUG) and `dlq_sink.create_dlq_entry_failed` (ERROR)
the queue is full and the overflow strategy is `reject`	`dlq.store_rejected_overflow` (WARNING)

Like every Baldur log line below ERROR, these records are rate-limited: past ten of one event in ten seconds, the rest of that window's are dropped, and the next one carries their count. Under observe-only the would-store record shares its event with the would-have records the circuit breaker and the retry stage write for every call, so in a dry run of more than a few calls per ten seconds most would-store records are among those dropped.

Two switches exclude every call of a process at once rather than one call, so neither writes a per-call record of the exclusion: the kill switch, which makes Baldur step aside and is logged once per process when it changes, and BALDUR_PROTECT_ENABLED=false, under which a protected function runs bare. An async call cancelled from outside, by a task cancel or by the caller's own asyncio.wait_for running out, is not parked either and leaves no record: the cancellation passes through Baldur untouched. Bound the call with Baldur's timeout= when its expiry should be parked. Capturing a failure is designed to stay off the request's critical path: by default the write to the store happens in the background, so the call that already failed does not wait for storage; it pays only for masking and size-capping the entry and handing it over. If the queue's storage backend is itself unreachable at capture time, the entry falls back to a local on-disk record (and, as a last resort, to the process's error stream) instead of being silently lost. Nothing moves such a record back into the queue once the store returns; recovering it from that host's disk is a manual step. Each entry then moves through a lifecycle you can watch in the Web Console DLQ panel or query over the REST API:

stateDiagram-v2
    [*] --> PENDING: operation fails, captured
    PENDING --> REPLAYING: replay triggered
    REPLAYING --> RESOLVED: replay succeeds
    REPLAYING --> PENDING: replay fails, attempts remain
    REPLAYING --> REQUIRES_REVIEW: replay attempts exhausted
    REPLAYING --> REQUIRES_REVIEW: automatic on-recovery replay fails
    REPLAYING --> PENDING: held past the stale-release cutoff, attempts remain
    REPLAYING --> REQUIRES_REVIEW: interrupted on its last allowed attempt
    REQUIRES_REVIEW --> REPLAYING: operator force-redrives after a fix
    PENDING --> EXPIRED: expiry window passes
    RESOLVED --> ARCHIVED: aged out / cleaned up

You have three ways to replay the queued work:

Targeted replay. Pick a single entry and retry it from the Web Console or the REST API, useful when you want to confirm a fix before draining everything. The same single-entry surface also lets you resolve an entry by hand. Force-redriving an entry that is parked for review is a REST-only admin action; the console does not expose it.
Batch replay by failure type. Replay the queued entries of one failure type from code, using the replay API (batch_replay_by_failure_type, up to 100 entries per call by default): for example, the database-timeout failures after the database recovers. A batch started from the replay service can also run in an adaptive mode (replay_batch(..., use_adaptive=True), off by default): instead of a fixed batch size, Baldur watches the success rate of each batch and adjusts the next one, shrinking the batch when too many replays are still failing and growing it again after several clean batches, staying between a floor and a ceiling you set. With PRO active, batch replay is also a one-click Web Console action and a REST endpoint; both select pending entries rather than a failure type, 50 per call by default, and the REST endpoint can narrow them to one domain, named in its stored form (payment_api; Payment-API selects nothing). Either way the batch runs its entries directly, one after another; PRO additionally ships a standalone replay queue with rate limiting and backpressure that your own code can pace replay work through.
Automatic on recovery. When a dependency's circuit breaker closes again after an outage, Baldur sweeps the queued failures tied to it (the calls its open breaker rejected, and the failure types you mapped to it) and replays them, so recovery and catch-up happen together. Not every outage closes a breaker — one too short to open it, a batch that ended while it was open, a job with no breaker at all — so Baldur also runs a recovery trial: about once a minute it replays one parked job per job name, taken from the failures that name's sweep would replay, and when that job succeeds it sweeps the rest. A sweep works in passes of a bounded size and keeps going, pass after pass, until the backlog is drained or a bound stops it. How far one recovery goes, how the trial is paced, and how a recovery tells you it stopped early are covered under Closing the loop below.

When a failure can't be replayed successfully (the dependency is still down, or the work itself is broken), the entry goes back to waiting, and every replay attempt spends part of a configurable replay budget. An entry that exhausts that budget is neither retried forever nor discarded: it converges to a terminal needs-review state, where it stays queryable so an operator can investigate. The sweep a breaker's closing starts is stricter than the operator-driven and code paths: an entry whose job runs and fails during that sweep is parked for review right away, without spending the rest of its budget. On the automatic paths and in replay from code, a replay whose job never began (its own circuit breaker refused the call, or its own idempotency key kept it from starting) spends nothing and is not parked; the console and REST actions count it as an attempt. A function protected with @baldur.protected(name, replay=True), which Baldur re-runs from its stored arguments, reports this by itself; a hand-written handler says so in its result (see step 1 under Closing the loop). A replay that was interrupted (its worker died, or its job stopped waiting on a run that is still going: its own timeout= gave up, or a Celery soft time limit or another interruption cut that wait short) stays in REPLAYING until the stale release returns it, once it has been held 30 minutes (the default; never less than 10): back to waiting, or to needs-review when that was its last allowed attempt. The stale release runs as a Celery task: Beat queues it every 15 minutes on the maintenance queue and each recovery-trial tick runs it too, so without a Celery worker an interrupted entry stays in REPLAYING. A replay you start yourself is held the same way: a single-entry retry or force-redrive whose job may still be running answers with the entry's status replaying (or with an error, when the job raised), and a console/REST batch replay (PRO) leaves each such entry replaying and counts it failed; the entry does not come back sooner when the job ends, and retrying it again is refused until the stale release. The hold has a limit: a job still running when the stale release returns its entry can be started again beside itself, so give the job's own outbound calls a timeout that ends a stuck run well inside that window. PRO's throttle-recovery replay, which replays entries a throttle rejected, runs them without taking them and holds none. Once the root cause is fixed, an operator can deliberately force-redrive the parked entry, an admin-level action (recorded in the audit trail when PRO is active) that grants it a fresh replay budget and sends it back through replay. If the underlying problem still isn't fixed, the entry spends that fresh budget the same way as the first one and re-converges to needs-review, so a force-redrive can never turn a poison-pill into an endless loop.

With PRO active and its scheduled retention running, an entry captured more than 72 hours ago is marked expired at the next retention run (every six hours) and can no longer be replayed or force-redriven. On the Redis store only waiting entries expire; on the in-memory and SQL stores entries parked for review expire too.

What you observe	When it happens
A failed operation appears in the queue with its failure reason and request data	a `dlq=True` call fails for good and no `fallback=` answered it
A call appears in the queue as a `CIRCUIT_BREAKER_OPEN` failure without ever having run	the dependency's circuit breaker is open and rejects the call at a `dlq=True` call site with no `fallback=`
You retry or resolve a single entry	an operator action from the Web Console DLQ panel or the REST API
You force-redrive an entry parked for review	an admin action over the REST API
A batch of queued entries replays in one call	`batch_replay_by_failure_type` from code (one failure type, 100 entries by default), or the console/REST batch replay (PRO; pending entries, 50 by default, optionally of one domain over REST)
Queued work drains on its own	a dependency's circuit breaker recovers, or a recovery trial finds the dependency answering again, and an automatic replay sweep runs
One parked job of a name is replayed on its own while the others wait, first after a minute, then less and less often	the recovery trial testing whether the job's dependency answers again (one job per job name; 60 seconds, doubling up to 9 minutes while trials keep failing)
A drain stops with work still queued, and says why	the recovery's continuation bound was reached, a circuit for that domain re-opened, the integrity gate blocked replay, governance blocked automation (PRO), a pass made no progress, or a pass errored (a `DLQ_REPLAY_BLOCKED` event whose `block_reason` names which)
A batch replay grows or shrinks batch by batch	adaptive batch sizing was opted in (`use_adaptive=True`) and the recent replay success rate changes
An entry stops being retried and is parked in a needs-review state	its replay attempts are exhausted, its job ran and failed during the sweep a breaker's closing started, or it was interrupted on its last allowed attempt
An entry expires and can no longer be replayed	PRO: 72 hours after capture, while the scheduled retention runs
New failures displace the oldest entries, or are rejected	the queue hits its size limit and the overflow strategy applies
Buffered entries are written out, or spilled to the local fallback, as a process exits	a worker stops or is recycled while its outbox still holds entries

When the queue reaches its size limit, the overflow strategy decides what gives:

drop_oldest evicts the oldest entries to make room for new failures (the default; unless PRO is active, the eviction happens synchronously, inside the store call).
reject refuses new entries so nothing already queued is displaced. On the defaults the per-domain limit is checked only on every tenth store while the queue as a whole is under 80% of its size limit, so until then a domain at its own limit still takes about nine of every ten new failures.
compress_oldest (PRO) summarizes the oldest entries into a compact record before evicting them, so an aggregate trace of what failed survives even after the raw entries are gone. That holds on the Redis and in-memory stores; on the SQL store it evicts them with no summary. The summaries are grouped by domain, failure type, and error code, stay queryable over the REST API, and move through their own lifecycle (ACTIVE, then STALE, then ARCHIVED); an archived summary is kept, not deleted. Without PRO, compress_oldest falls back to drop_oldest, with a one-time warning the first time the queue overflows.

The queue lives in one of three stores, chosen once at startup: in-memory, Redis, or SQL. BALDUR_DLQ_BACKEND names the store explicitly; left unset, Baldur takes Redis when a Redis URL is configured, else SQL when a database DSN is, else memory. An unrecognized value logs a warning and falls back to that same probe order. Capture flows into the store through a non-blocking in-memory outbox that keeps it off the request hot path. The outbox watches its own pressure in every tier: it reports how many entries are waiting in its buffer and how long they wait before being written (the leading signs of a buffer under stress), counts every entry it drops in a metric, and raises a warning and an event when its drop rate crosses a threshold, so you learn the outbox is shedding instead of discovering it after the fact.

An in-memory buffer would normally die with its process. Baldur therefore tears the outbox down on every exit path (a signalled stop, a gunicorn or Celery worker recycle, and a plain interpreter exit when a script returns or calls sys.exit(), through an atexit hook the outbox registers when it starts) under one time budget, BALDUR_DLQ_OUTBOX_JOIN_TIMEOUT_SECONDS (5 seconds by default): buffered entries are flushed to the store, the writer is joined, and whatever is still unwritten at that point is spilled to the local on-disk fallback (the manual-recovery record above). Keep that budget below the process watchdog that will kill the worker anyway (gunicorn --timeout, Kubernetes terminationGracePeriodSeconds): a teardown the watchdog cuts short loses its tail with no report, whereas a teardown that runs out of its own budget reports exactly what it lost. If the deadline passes with entries still unwritten, they are gone with the process, and a CRITICAL dlq_outbox.shutdown_dump_incomplete log line states how many. The guarantee is no unreported loss. Under gunicorn, a worker that stops or is recycled runs this teardown from Baldur's gunicorn hooks when they are wired, and from the atexit hook when they are not; the gunicorn graceful-shutdown runbook shows the two ways to wire the hooks and how the budget fits gunicorn's own timeouts.

With PRO active, the Meta-Watchdog daemon actively probes the liveness of that background writer, so a stalled drain is detected rather than silently backing up.

Trace continuity: from the original failure to its replay

When you run distributed tracing (OpenTelemetry into Jaeger, Datadog, Tempo, and the like), the DLQ sits across a deliberate gap in time: an operation fails now, is parked, and is replayed later — sometimes seconds later, sometimes weeks. Baldur links the two ends so that "original failure → DLQ capture → replay" reads as one connected story instead of two unrelated traces.

How the link is made. At capture time Baldur records the failing request's trace on the entry (its origin_trace_id, plus the full W3C trace/span ids when an OpenTelemetry span is active). When that entry is later replayed — by any path: a targeted retry, a batch, an automatic on-recovery sweep, a recovery trial, or a force-redrive — Baldur re-attaches that origin:

when OpenTelemetry is active and the entry carries its origin span ids, every replay path wraps the replay in a dlq.replay span carrying a span link back to the original failure's span, plus a searchable baldur.dlq.origin_trace_id attribute,
the log lines reporting the replay carry an origin_trace_id field, and
the replay paths that write an audit record (a batch replay, the automatic sweep, a recovery trial, a force-redrive, with PRO's audit trail active) record the origin trace there too; a targeted single-entry retry keeps its trail in the log line and the span instead of a per-entry audit record.

The link is additive: the replay keeps its own trace (the operator request or circuit-breaker recovery that triggered it) as its primary trace, and the origin is attached alongside. The two answer different questions — "under what request did this replay run?" versus "what failure is this replay recovering?" — so both are kept rather than one overwriting the other.

Finding the connection in your tools.

In an OpenTelemetry backend (Jaeger/Datadog/Tempo): open the dlq.replay span and follow its span link to jump to the original failure's trace. You can also search spans by the baldur.dlq.origin_trace_id attribute.
Without OpenTelemetry (plain structured logs): the origin id is stored on the entry itself and stamped on the replay's log lines, so an origin_trace_id=<id> query across your logs returns the replay activity for that failure; the entry's detail view (console or REST) is the capture-side record. Note that a bare trace_id=<origin> search will not match the replay lines — because the replay keeps its own trigger trace, the origin rides the separate origin_trace_id field. Search that field, not trace_id.

Expected limits. Both of the following are inherent to store-and-forward tracing, not defects:

Mixed-mode capture. If a failure was captured while OpenTelemetry was not producing a span, only the display origin_trace_id is stored — there is no full span id to build an OTEL span link from at replay time. Nothing is lost: an operation with no capture-time span had no origin span to link to in the first place. Replaying such an entry creates no dlq.replay span either; the origin id still travels in the replay's log line and audit record.
APM retention mismatch. Because the DLQ can outlive any trace-retention window by design, an entry replayed weeks later may link to an origin trace your APM store has already aged out (Jaeger/Datadog typically keep ~14–30 days). The span link then resolves to an expired trace — an orphan span or a "Trace Not Found" in the UI. The origin_trace_id remains recorded on the entry and in the logs/audit regardless.

Entries captured before this linkage existed, and captures that had no active trace, simply replay without an origin link — the absence is the normal state, not an error.

Why capture is opt-in per call

@baldur.protected does not send a call's final failures, or the calls its open circuit rejects, to the dead letter queue unless the call site asks for it with dlq=True (the @dlq_protect preset pins it on, together with retry and the circuit breaker, and replay=True implies it). The queue itself is ready with no wiring — storage resolves to your configured backend or the in-memory fallback — so the flag is not about infrastructure. It is about what gets stored.

To be replayable, an entry has to carry the work itself: capture records a snapshot of the failing call's arguments — the request data a replay re-runs. Only arguments of simple types are snapshotted (numbers, strings, ids, dates and the like). A dict, list or object argument is left out, so a call that takes one carries that work only if you pass context_from= a function that returns the PolicyContext with that payload in its extra["request_data"]. That snapshot is your domain data: order ids, amounts, user ids. Sensitive-looking fields are masked by rule-based patterns, but no generic rule set can know which of your values are sensitive, so domain values survive masking by design. Persisting that data is therefore a decision Baldur leaves to you, made explicitly per call site, rather than something an upgrade switches on silently.

There is deliberately no "capture without the payload" middle ground. Replay re-runs the captured request data, so an entry whose payload the write-side size cap cut off is refused replay rather than re-run half-blind — a payload-less capture would demote the queue from a work queue to a log. For call sites that should record the failure but must not snapshot arguments (say, a password-verification path), keep dlq=True and pass context_from=False: the failure is still captured, without the auto-extracted argument payload — accepting that such entries need an operator (or a custom replay handler with its own data source) rather than hands-off replay.

BALDUR_DLQ_ENABLED (below) is a different switch: it governs whether the capture service accepts entries at all. It defaults to on so that every dlq=True call site just works — it does not put any call into the queue by itself.

Configuration

The knobs an operator sets most often. The full list lives in the API reference.

Env Var	Default	What it controls
`BALDUR_DLQ_ENABLED`	`true`	Whether failed operations are captured into the dead letter queue at all
`BALDUR_DLQ_MAX_SIZE`	`100000`	Maximum total entries the queue holds before the overflow strategy applies
`BALDUR_DLQ_BACKEND`		Which store holds the queue: `memory`, `redis`, or `sql`; leave it empty and startup picks the first one the environment offers (Redis, then SQL, then memory)
`BALDUR_DLQ_OUTBOX_ENABLED`	`true`	Capture failures through a non-blocking outbox so recording a failure stays off the request hot path
`BALDUR_DLQ_OUTBOX_JOIN_TIMEOUT_SECONDS`	`5.0`	Total budget an exiting process spends flushing its outbox to the store and spilling the rest to the local fallback; size it below your process watchdog
`BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED`	`true`	Automatic replay of queued failures when a circuit breaker recovers
`BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_ITEMS`	`100`	Entries one on-recovery pass replays
`BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_CONTINUATIONS`	`100`	Passes one recovery may chain while work is still reachable; multiplied by the pass size, the most one recovery drains
`BALDUR_REPLAY_AUTOMATION_RECOVERY_TRIAL_ENABLED`	`true`	The recovery trial: one parked job per job name replayed periodically to find out whether its dependency answers again; it also needs `BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED`

If you don't use automatic replay, turn it off rather than leaving it half-configured: with BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED=false, recovery events skip the replay dispatch entirely, the recovery trial stops, and the WARNING that Celery is missing, raised on a recovery that leaves work parked, disappears with it (the arming surface reports disabled).

Closing the loop — making automatic replay actually drain

Automatic replay — on a circuit breaker's recovery and by the recovery trial — is on by default, but it only drains your backlog once its prerequisites are in place. Until they are, a recovery leaves the entries parked — captured and safe, but not replayed. The Web Console DLQ panel and the GET /dlq/cleanup/stats/ payload report an armed / disarmed / unverified state and name the prerequisite behind it, so you can tell at a glance whether the loop is live.

Three lanes can drain the queue, and they need different things: the open-circuit sweep replays what an open breaker rejected, the mapped sweep replays the failure types you listed for the recovered service, and the recovery trial replays one parked job per job name, from those same failures, until one succeeds. Four prerequisites are shared — the enable flag, the Celery extra, a worker on the queue, and a registered handler — and each lane adds one of its own: the mapped sweep needs a service_failure_type_map entry (map_unconfigured when it has none), the open-circuit sweep needs open-circuit capture on (open_circuit_capture_disabled when it is off), and the recovery trial needs its switch on (recovery_trial_disabled when it is off). armed is true when any lane has everything it needs; the lanes block says which:

Register a replay handler per domain. Baldur captures the failed work, but only your code knows how to re-run it. Register a handler for each domain you want replayed:
```
from baldur.services.replay_service import register_replay_handler, ReplayHandler, ReplayResult

class PaymentReplayHandler(ReplayHandler):
    @property
    def domain(self) -> str:
        return "payment"

    def can_replay(self, failed_op) -> tuple[bool, str]:
        return True, ""

    def replay(self, failed_op) -> ReplayResult:
        # re-run the captured work; return succeeded/failed
        return ReplayResult.succeeded(failed_op.id, "reprocessed")

register_replay_handler(PaymentReplayHandler())
```
A function protected with @baldur.protected(name, replay=True) registers its own handler, which re-runs it from its stored arguments (see LLM API rate limits).

Without a registered handler, nothing replays that domain's entries automatically: the sweeps and the recovery trial select nothing for that domain, and batch replay from code asks the domain's handler before it takes an entry, which the default handler refuses, so the entries wait untouched. A single-entry retry from the console or the REST API takes the entry first and spends one replay attempt failing. The arming surface reports handler_missing only while no handler at all is registered; it does not check that each domain you capture has one.

When your handler's call was refused by the job's own circuit breaker before the work began, say so in the result, so the automatic paths and replay from code give the replay attempt back and the entry waits for the next replay: ReplayResult(success=False, dlq_id=failed_op.id, error="...", data={"job_started": False, "rejected_by_breaker": True}) ("job_started": False alone for work that did not begin for another reason). A result without these flags counts as work that ran. Registration is per process: the on-recovery sweep runs inside the Celery worker, so the worker must register the handler too, and the arming surface can only vouch for the process that answers it.
Map recovered services to their failure types. When a circuit breaker closes, Baldur needs to know which captured entries the recovered dependency is responsible for. Configure that mapping with BALDUR_REPLAY_AUTOMATION_SERVICE_FAILURE_TYPE_MAP (see Environment Variables). A mapped type is matched only under the mapped service's own domain — the name in its stored form, payment_api for Payment-API — and only when a replay handler is registered for that domain. A handler can also name the types of its own domain to replay, in its auto_replay_failure_types property, with no map entry (the arming surface still reports map_unconfigured). A dlq=True failure is typed MAX_RETRIES_ plus its exception's class name (MAX_RETRIES_TIMEOUTERROR, as an entry's detail view shows it); an entry whose type is neither mapped nor named by the handler stays parked through a recovery without a signal. A recovery whose breaker name gets no lane at all (no replay handler for its domain, or no domain of its own) and that leaves entries parked is surfaced as a blocked-with-signal event naming which is missing, not a silent no-op; with no mapping, the arming surface reports map_unconfigured either way.

Entries captured because the circuit was open are the one exception: the circuit that just closed is the very one that rejected them, so they need no map entry. On recovery they are swept for that breaker's domain whenever the breaker's name resolves to a domain of its own (not the unclassified catch-all) with a replay handler registered for it, map or no map. That is the open-circuit lane: with the shared prerequisites in place, a plain dlq=True deployment on the shipped defaults with an empty mapping reads armed: true, with lanes.mapped reporting map_unconfigured on its own — the sweep that drains its captures runs, the one that has nothing to select by does not.
Run a Celery worker on the dlq_processing queue. On-recovery replay and the recovery trial run as Celery tasks. A worker must be consuming the dlq_processing queue for them to run:
```
celery -A your_app worker -Q dlq_processing
```
If Celery itself is absent, the arming surface reports celery_missing, and a recovery that leaves work parked logs a WARNING naming this remediation. If Celery is present but no worker is consuming the queue, the dispatch itself succeeds and the recovery logs nothing unusual; only the arming surface reports worker_missing (it asks the broker which queues the live workers consume). You can still drain the backlog manually with the single-entry Retry action.

If the broker cannot be reached at all and nothing else is missing, the surface answers armed: null — unverified — with unverified_link: "worker_missing", rather than claiming the loop is armed on a question it could not ask. The baldur_dlq_auto_replay_armed gauge follows: it is 1 only when a sweep verified everything it needs, and 0 both for a missing prerequisite and for an unverified one. The gauge moves within the worker-probe cache TTL plus one metric-collection interval plus the probe's own budget, roughly 80 seconds on the defaults.
Keep on-recovery replay enabled. BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED is true by default; the arming surface reports disabled when it is turned off.

How far one recovery drains. A recovery does not replay everything in one go. The sweep runs in passes of BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_ITEMS entries (100 by default), and after each pass it queues the next one for the same service as long as work is still reachable: the pass filled its quota, ran up against the replay task's time limit, or stopped scanning before it had looked at every candidate. The chain runs to at most BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_CONTINUATIONS passes (100 by default), so one recovery clears up to 10,000 entries on the defaults; a domain that parks more than that wants a higher continuation count rather than a bigger pass. Both values are read once, when the recovery is dispatched, so a change you make while a drain is running applies to the next recovery. A pass that a replay's own circuit breaker refused ends there, with that entry waiting, and the next pass is queued 30 seconds later instead of at once.

Every pass starts by reading the breakers of its domain from the breaker store. Live traffic can re-trip a breaker between passes, and replaying into a dependency that has just failed again would only park each entry for review, so the chain stops instead: a sweep a breaker's closing started stops when any breaker of the domain is not closed, and a sweep a recovery trial started stops on a breaker that refuses calls (its replays go through the job's own breaker, which admits a half-open probe). Every pass also asks the integrity gate and, with PRO active, governance (the kill switch, Emergency Mode, the error budget). The chain also stops when the continuation bound runs out, when a pass ends without moving forward, or when a pass raises. A drain that ends for any of these reasons with work still queued announces it: a WARNING log and a DLQ_REPLAY_BLOCKED event whose block_reason is circuit_reopened, integrity_blocked, continuation_bound_reached, pass_made_no_progress, pass_errored or the governance check that refused (kill_switch, for example), on the same channel that reports a recovery with no lane to replay its parked work. A drain that simply finds nothing left ends without an announcement, and so does one an operator's hold stops (below).

Read capped on the dlq_replay_batch_completed event as a per-pass fact. It says that pass filled its quota or hit its time limit, not that the drain as a whole is over, and the last pass of a fully drained queue can carry it too. Whether more is coming is answered by the absence of a DLQ_REPLAY_BLOCKED stop, not by capped.

A chain that stopped early is picked up by the recovery trial: once its job name is due, a trial replays one of the parked jobs and, when it succeeds, sweeps the rest. An entry the stopped chain was replaying stays in REPLAYING until the stale release returns it. To drain sooner, replay the remainder yourself (the single-entry Retry action, batch replay from code, or the console/REST batch replay with PRO active).

The recovery trial. A breaker closing is evidence that a dependency answers again, but many outages never close one: a burst too short to open the breaker, a batch that ended while it was open (an open breaker closes only after successful calls), a job with no breaker. So about once a minute Baldur takes, for each job name with parked work and a replay handler, the next parked entry that name's sweep would select and its handler would replay, and replays it — the only call that can show the job's dependency answers is the job itself. The sweep selects only mapped or handler-named failure types and calls an open breaker rejected, so for a job with no breaker, map or name its failure types (step 2) or the trial finds nothing to take. A trial that succeeds resolves that job and starts the sweep for the rest — unless the trial itself closed the job's breaker, whose closing starts its own sweep. The trial's sweep leaves a replay whose job runs and fails waiting for its next attempt rather than parking it for review, since one success is weaker evidence than a breaker closing. After a trial that fails, the next trial of that job name waits 60 seconds, doubling up to 9 minutes, so a recovery starts within about ten minutes of the dependency answering, at about six trials an hour per job name once the wait is at its longest. The trials take a job name's parked jobs in turn, so a parked job that keeps failing for a reason of its own (a bad input, a bug in the job) takes its turn too: each such job the turn reaches before a good one adds up to nine more minutes. A trial whose dependency still fails, whose own timeout= cuts it off, or whose own breaker refuses the call spends none of the entry's replay attempts; a trial whose worker dies counts as one, so a job that kills its worker reaches needs-review at its cap instead of being tried forever. No trial starts while a breaker of the domain refuses calls or carries an operator's hold, while the kill switch is pulled or governance blocks automation, or while the integrity gate blocks replay. The trial runs on the dlq_processing worker, queued every minute by Celery Beat (configure_baldur_celery(app) installs the schedule) and by Baldur's own scheduler, so a deployment without Beat gets it too. Turn it off with BALDUR_REPLAY_AUTOMATION_RECOVERY_TRIAL_ENABLED=false.

An operator's hold. Closing a breaker by hand without replay (the console's Allow, or force_close(..., trigger_replay=False)) pins it for the manual-override window, 90 minutes by default. For as long as the pin lasts, nothing replays that domain's parked work automatically: no trial runs, and a running chain stops at its next pass, logging dlq.circuit_recovery_held at INFO with no warning, event or audit record. When the pin lapses or you hand the breaker back to automatic control, the next trial picks the backlog up. Closing with replay (trigger_replay=True) starts the sweep even when the breaker was already closed, and your own pin does not stop the sweep you asked for.

Recommended alert: the example alert rules in the Baldur repository ship DLQAutoReplayDisarmed (baldur_dlq_auto_replay_armed == 0 for 10 minutes); the for: clause is what keeps a short broker blip from paging. Because a dispatched replay only runs when a worker is actually consuming dlq_processing, alert on the depth of that queue (broker-side) as well. A queue that grows without draining means the dispatch is succeeding but no worker is consuming it.

What belongs in automatic replay — and what doesn't

Replay re-runs the original operation, so it is a recovery tool for work that is safe to run again and a hazard for work that isn't. The same caution governs retries. Two properties of the captured work decide whether it belongs in the automatic on-recovery path.

Is it safe to repeat? A replayed operation executes a second time, so replaying one that can't safely repeat may double its effect: a second charge, a duplicate shipment, a repeated email. Map a failure type into automatic replay only when re-running it is a no-op once the work has already completed. The handler's can_replay(failed_op) check refuses an entry on every replay path: the automatic sweeps, the recovery trial and batch replay from code ask it before they take the entry, so a refused entry keeps waiting with its replay attempts untouched, and the operator-driven paths (the single-entry retry and force-redrive actions, and the console/REST batch replay, PRO) ask it after taking the entry. It decides whether an entry may run, not whether it already ran, so replay() itself should still detect work that has already completed and report success without re-running it. Making an operation safe to repeat is a separate guarantee Baldur provides (see Idempotency), and money-equivalent operations should anchor their dedup on a database uniqueness constraint rather than a cache. The what-belongs-in-Baldur-vs-your-database boundaries runbook covers where that line sits.

Is anyone still waiting on it? Automatic replay fires minutes or hours after the failure, long after the original caller has gone. That fits backend work whose goal is simply to finish: settling a payment the gateway already accepted, writing an order your database rejected, delivering a webhook, reconciling state. It does not fit a synchronous, user-facing action such as a checkout authorization. A customer who abandoned a failed payment should not be charged when the dependency recovers an hour later. Protect that path with the circuit breaker, which fails fast so the user gets an immediate retry instead of a hang, and with idempotency, so the user's own retry can't double-act. Replay is not the tool for that path.

This is why the failure types you map to on-recovery replay are the transient, your-side ones: a database timeout, a dropped connection, a dependency returning 503, where the work is sound and only the infrastructure faltered. A business rejection such as a declined card is not one of them. It will fail again on replay, so it does not belong in the mapping.

Tier behavior

DLQ + Replay runs in every tier. Capture and recovery are the OSS core; PRO layers the operate-at-scale surface on top.

In OSS: the full failure-preservation loop — capture with forensic context (including the calls an open circuit rejected), your choice of in-memory, Redis, or SQL store, the non-blocking outbox with its own pressure signals (buffer depth, write-wait latency and the drop-rate warning) and its budgeted drain on process exit, the local on-disk fallback when the store is unreachable, size limits with the drop_oldest / reject overflow strategies, the Web Console DLQ panel and the read REST endpoints (list, detail, facet counts, cleanup stats), the single-entry actions (retry and resolve from the console or REST, force-redrive over REST), batch replay by failure type from code with opt-in adaptive (success-rate-driven) batch sizing, and automatic replay on circuit-breaker recovery and by the periodic recovery trial, with its armed / disarmed / unverified surface.
With PRO active: batch replay becomes a one-click console action and REST endpoint, a standalone replay queue with rate limiting and backpressure is available to your code, the compress_oldest overflow strategy and its compressed summaries become available, evictions move off the capture path to a background water-level worker, Meta-Watchdog probes the outbox drain worker's liveness, scheduled retention expires, archives and purges old entries (an expired entry can no longer be replayed), and synthetic test entries can be created for debugging and load tests. The background eviction, the expiry, the compressed summaries' aging and the scheduled purge run only as Celery Beat tasks from Baldur's schedule (configure_baldur_celery(app) installs it): without Beat and a worker consuming the maintenance queue, a PRO queue past its size limit under drop_oldest or compress_oldest is never trimmed.