Skip to content

PRO Release Notes

Release notes for Baldur PRO — the licensed tier. The open-source core has its own changelog in the baldur-framework repository. Each entry describes an operator-visible capability; a PRO license is required.

The format follows Keep a Changelog.

[Unreleased]

Changed

  • A licensed production boot checks its signing key and incident store against the licence it actually runs with. The free core used to require BALDUR_SQL_DSN (or Django DATABASES) and both BALDUR_SECRETS_* keys on every production deployment, so a free-tier deployment that followed the getting-started pages could not start. From the free core at 1.17.0 those requirements follow what writes: BALDUR_SECRETS_AUDIT_SIGNING_KEY while the audit trail is on — which an active entitlement arranges — and a SQL or Django store under an active entitlement, because the licensed tier writes its incident records there. For a licensed deployment the requirements are unchanged in substance, and they are now read after startup has validated the licence, so a licence token that reaches the environment from inside the application (a dotenv file, Django settings) can no longer leave them waived by an earlier cached verdict. BALDUR_SECRETS_ENCRYPTION_KEY is no longer required. Restart after installing a licence into a running deployment: the requirements are checked at startup, and a process that became licensed without one can keep incident records in per-process memory and chain them without a key until it restarts.

  • ha_pipeline(max_retries=0) now raises ValueError when the pipeline is built. max_retries counts every attempt, so 0 ran the call zero times and ended in a rejection that stored nothing; pass max_retries=1 for a single attempt with no retry. This comes with the free core at 1.17.0, which refuses an attempt count below one on every retry stage — the package's dependency floor requires it.

Fixed

  • A forked worker no longer reports the leader scheduler and the leader elector dead. Both run only in the process that started them — under gunicorn --preload or a Celery prefork worker, the master or worker main — and every forked worker inherited their registrations without their threads, so each worker's watchdog reported Scheduler-* (and, with leader election on, LeaderElector-*) DEAD from its first probe and raised a CRITICAL daemon_worker.died; with worker respawn enabled it also started a second scheduler loop inside the worker. Forked workers now skip those two and keep reporting every worker they do use — one whose thread is not running in the worker is still reported DEAD. A real scheduler death in a pre-fork master is not paged by the watchdog, because the master runs none: read it from the master's own admin /metrics (baldur_daemon_worker_alive). The licensed tier's own locks are also repaired in a forked worker, so a worker forked while a background thread held one no longer blocks on its first use. This needs the free core at 1.17.0, which ships the fork repair and the flag the watchdog reads — the package's dependency floor requires it.

  • An emergency level set in one process now reaches every process, and a process holding an older copy no longer overwrites it. Each worker used to answer from its own copy of the level and refresh it only on some reads: every worker except the one that activated emergency mode kept admitting all traffic to HTTP load shedding, and an automatic activation in a worker with an older copy could replace a higher level set elsewhere — lifting the LEVEL_3 breaker hold across the fleet without the recovery gate. Every process now re-reads the level on the emergency interval (thirty seconds by default), readers never touch the store, and every change is a versioned write decided on the stored level: an automatic activation never lowers or repeats a stored level, an operator's activation during a gradual recovery ends that recovery instead of being walked back down, and a stop-recovery request served by any worker ends the recovery whichever worker runs it. An emergency change the state store did not confirm is in force nowhere: the console and POST /emergency/* answer 503 with persisted (false, or null when the outcome is decided by the next read), and an automatic activation logs a warning instead of degrading the one worker that made it. Governance reads the kill switch and the level straight from those copies instead of its own cache, so a level change reaches governance decisions within the same interval. The chaos safety guard now refuses an experiment when it cannot read the kill switch, or when its process has not read the switch within the last three refresh intervals — it used to admit every experiment on a failed read. This needs the free core at 1.17.0, which ships the per-process refresher and the versioned writes — the package's dependency floor requires it.

  • A governance refusal during emergency mode now names when the level lifts. While an emergency level blocked automation, the refusal stated the countdown of the governance operating mode's STRICT declaration, which an emergency activation never sets — so with no STRICT declared it read Auto-restore in ~0.0h, even for a security violation's LEVEL_2, which lifts thirty minutes later by default. The remaining time now comes from the emergency level that made the refusal: a thirty-minute activation reads ~0.5h, and a level with no expiry — one that holds until an operator lifts it — no longer promises an automatic restore and reads Emergency mode <level> is active: operations blocked. When automation is refused is unchanged; only the time the refusal states is.

  • ha_pipeline() now parks a call its full bulkhead refuses, and files its dead-letter entries under the service name. The high-availability preset runs its retry stage around a bulkhead, and a call the full bulkhead refused on every attempt ended as a BulkheadFullError rejection with no dead-letter entry — during saturation, exactly the work the dead letter queue exists to keep. It is now parked once, as MAX_RETRIES_BULKHEADFULLERROR with the retry stage's attempt count, and so is a failure with retry switched off (BALDUR_RETRY_ENABLED=false) or behind a retry_policy= that makes no store decision of its own, such as a tenacity bridge. Its retry-exhaustion entries now go under the service_name you passed instead of default (a domain= you named is kept), so a replay handler registered for default no longer receives them. Under sustained saturation every refused call becomes an entry, bounded by the queue's size cap and overflow strategy. A call a configured fallback answered, or the error-budget guard refused, is still not parked. This needs the free core at 1.17.0, which ships the capture — the package's dependency floor requires it.

  • A thread-pool bulkhead compartment gives back every seat a call took, and admits its workers plus its queue on every path. A call that timed out while it waited in the compartment's queue used to keep its waiting seat for good: after as many such timeouts as the queue had seats (10 on the built-in external_api compartment), the compartment rejected every call with BulkheadFullError although no worker was busy, until the process restarted. A call cut off before any worker started it — by its execution timeout, by your own future.cancel(), or by a shutdown that cancels queued work — now gives its seat back, and so does a submit the pool refuses. Every entry now counts against one capacity, workers plus queue: execute and submit calls from admission until they end, and holders from acquire, try_acquire and async callers until they release, so queue_size=0 now runs as many calls at once as there are workers and rejects the next, where it used to reject every call. The status surface reports that capacity: max_concurrent (and the baldur_bulkhead_max_concurrent gauge) is workers plus queue, active_count counts every seat taken — running, queued or held — waiting_count the queued calls, and queue_size is reported too, so utilization never passes 100%. A function that raises TimeoutError itself inside the compartment now gets that error back, not BulkheadTimeoutError. This needs the free core at 1.17.0, which ships the shared seat count and the status field — the package's dependency floor requires it.

  • FastAPI admission control no longer blocks the event loop while a request waits for its tier's slot. With a critical or standard tier saturated, each admission waited for a slot (up to the tier's bulkhead timeout, 50 ms and 30 ms by default) on the event loop's own thread, so the loop served no other request meanwhile. FastAPI now awaits the slot, and other requests on the same loop run while it waits; the tier timeouts, and the micro-bursts they absorb, are unchanged, and Django and Flask keep their per-request worker thread. A released slot now returns to the compartment it was taken from, even if that tier's compartment was registered again while the request ran. This needs the free core at 1.17.0, which ships the awaited admission — the package's dependency floor requires it.

  • A critical security violation's log record now says what its handling did to the emergency level. Each process that handled a critical violation logged event_bus.security_violation_emergency_activated with emergency_level=level_2 whatever the activation did — also when a level at or above LEVEL_2 was already in force (set by another trigger, or by another process before this one's copy caught up), when the kill switch was pulled, and when the state store did not confirm the write; with the Redis event bus, every process the violation reached logged an activation, so one activation read as one per process in an incident timeline. The record now names the outcome: event_bus.security_violation_emergency_activated only when this handling wrote the level, event_bus.security_violation_emergency_already_active when a level at or above LEVEL_2 was already in force, and event_bus.security_violation_emergency_not_raised when the level this process acts on stayed below it — the emergency mode warning logged just before names the cause. All three are WARNING and carry requested_level and current_level (the level the process acts on when the handling returns) in place of emergency_level, and event_bus.security_violation_emergency_failed now carries the violation type, the incident id, the reason and the requested level as well. The activation reason gains a per-handling suffix, [call <12 hex>], which also shows in the stored reason, the audit record and the escalation notification and joins a stored activation to the record of the process that wrote it. A write whose outcome the store could not confirm is recorded as not raised even if it lands later. This needs the free core at 1.17.0, which ships the new records — the package's dependency floor requires it.

  • A thread-pool compartment whose wait is cut short keeps holding what the call keys or replays until its task ends. A call waiting on a compartment's task that a Celery soft time limit, a gevent timeout or Ctrl-C interrupted used to leave the task running with nothing recording it: the call's idempotency key was released, and a replay that ran the job completed its entry, while the task still ran — so a same-key call or the next replay could start the work again beside it. The interrupted wait is now recorded like the compartment's own timeout: the key stays held until the task ends and is then released as before, and a replay leaves its entry replaying until the stale-replay release. The interruption itself still reaches the caller unchanged. This needs the free core at 1.17.0 — the package's dependency floor requires it.

  • A console or REST batch replay leaves an entry replaying while its job may still be running. A batch replay whose job was cut off by its own timeout= (or whose wait an interruption cut short) used to return the entry to pending, or to review at its last attempt, at once — so any replay could start the job again while the first run was still going. Such an entry now stays replaying until the stale-replay release returns it, and the batch counts it failed and audits the replay as failed. A batch replay whose job ended keeps today's behaviour. This needs the free core at 1.17.0, which reports whether a replayed job may still be running — the package's dependency floor requires it.

Removed

  • Traffic-aware replay is removed, with its two runtime-config fields. The lane replayed every pending entry of a domain once every circuit breaker of that domain read closed — which a breaker that never opened reads while its dependency is still down — whatever the entries' failure type, and it ignored an operator's close-without-replay. The free core at 1.17.0 replaces it with the recovery trial: about once a minute it replays one parked job per job name to find out whether its dependency answers again, a trial that finds the dependency still failing costs the job none of its replay attempts, and a successful trial replays the rest. The console no longer offers traffic_aware_enabled and traffic_aware_max_items, and BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_ENABLED and BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_MAX_ITEMS are reported at startup as unknown variables; unset them. The daily report counts a recovery trial's replays on its Auto-replay line. This needs the free core at 1.17.0, which ships the recovery trial — the package's dependency floor requires it.

  • The opt-in disk-durable DLQ outbox mode (BALDUR_DLQ_OUTBOX_DURABLE) is removed. With it on, the outbox writer copied each entry it took from the buffer to a local disk buffer before writing it to the store, but nothing read those copies back into the queue after a restart or deleted them once the store write succeeded, so the mode protected no entry and only grew the disk buffer. Unset the variable; Baldur now reports it at startup as an unknown variable. Capture is unchanged: an entry still reaches the store through the outbox, and an exiting process still flushes and spills what is left.

[1.5.0] - 2026-09-16

Added

  • Automatic staged recovery now runs on the shipped schedule. The recovery coordination lane — the trigger check, the mid-recovery health monitor, the stale-approval sweep and session cleanup — has always shipped with a beat schedule, and nothing composed it: a PRO deployment got the coordinator, its REST surface and its console actions, but a recovery started from any of them had no worker executing its steps unless the operator had wired the lane by hand, and the health monitor that can abort a recovery on re-failure had never run. The lane is now composed by the same beat composition every other lane uses (configure_baldur_celery / get_baldur_beat_schedule), on declared queues: the trigger check, step execution, resume and health monitor on baldur.critical, the stale sweep and cleanup on maintenance. A worker started with an explicit -Q must add baldur.critical for recoveries to advance; a worker started without -Q consumes every declared queue already. What starts, on every PRO deployment with a beat: each minute the trigger check reads the emergency level and the measured error rate, and when the level is elevated, the rate is under the stability threshold and manual approval is not required, a recovery session starts and its steps execute; every 30 s during an active recovery the health monitor evaluates the measured rate and re-escalation can abort the recovery. The coordinator never lowers the emergency level itself — the recovery gate still decides that. Set include_recovery=False on the composition to keep automatic recovery off the schedule, or require_manual_approval to keep a human in the loop. Every started session — from the console, the REST surface, the runbook or the trigger — now dispatches its own first step; a step whose publish the broker refused is marked on the session and re-dispatched by the next trigger cycle instead of stranding the session. This needs the free core at 1.14.0, which ships include_recovery.

Fixed

  • A LEVEL_3 stand-down with an open breaker has a gated exit. LEVEL_3 holds every circuit breaker where it is, so a breaker that was open when the level was declared could not close on its own, its refused calls kept the system-wide error rate above the gate's threshold, and the one exit the gate exists to avoid — force=true — was the only one left; the refusal named a rate, not the hold. A gradual recovery now stands down from LEVEL_3: its first step, the one that lets held breakers probe again, compares CPU and the error rate over the dependencies the hold does not cover and names the held breakers it left out; every later step measures the full rate as before. A refused release (POST /emergency/release, 409) now names the open breakers and the two gated exits — the gradual recovery, or a force-close of each breaker — instead of steering to force=true, and the console's Release and Gradual recovery forms say the same. Runbooks gain emergency.start_gradual_recovery (target level NORMAL, LEVEL_1 or LEVEL_2) and a force parameter on emergency.deactivate; a refused emergency.deactivate fails its step with EMERGENCY_RECOVERY_BLOCKED and the open breakers, not as a transient. The Adaptive Throttle's Full Stop now clears on any de-escalation below LEVEL_3, not only on a drop to NORMAL, so the console no longer reports Full Stop active while requests are admitted at the LEVEL_2 limit. A breaker an operator force-closed in the shared store is no longer counted from a worker's stale OPEN mirror. The walk also reads the shared state back before each step, so a release or level change made on another worker while it waited is honoured instead of written over. This needs the free core at 1.14.0, which ships the evidence fields — the package's dependency floor requires it; against an older core the gate's live read fails on the missing fields and refuses every release and recovery step until the core is upgraded (force=true remains the exit).

  • The recovery coordinator's stability checks now measure the error rate instead of assuming it. The trigger check reported stable: True, error_rate: 0.0 unconditionally, the in-session health-check step assumed success, and the approval notification rendered that 0.0% as a measurement. All three now read the same system-wide circuit-breaker failure rate the recovery gate reads. A rate that cannot be measured is reported as such: the trigger refuses to start a recovery, the health-check step fails (the coordinator compensates and the lane schedules a bounded resume), and the notification reads not measured. Expect automatic recoveries to be refused during a shared-store outage — a manual start from the REST surface stays open.

  • Emergency Mode's recovery gate now reads the error rate your protect() calls actually produce. The gate withholds recovery while the fleet-wide circuit-breaker failure rate is above its threshold, and it reads that rate from the process-shared breaker service. Every breaker built by protect(), @circuit_breaker and the preset pipelines used to keep its own private service, so on a deployment whose outbound traffic ran through those — the ordinary shape — the shared rate was measured over no calls at all and read 0.0: the error-rate half of the gate could never hold, and only the CPU half stood between an active emergency and its release. Those breakers now record on the shared service, so the rate the gate compares is the one your protected calls produced. Expect recovery to be withheld where it was released before — a deployment in emergency mode whose protected dependencies are still failing above the threshold now stays in it until they recover. A dependency whose breaker has tripped counts as failing for as long as it stays open, in whichever worker it tripped: the calls the open breaker refuses count as failed calls, and a breaker another worker opened counts at least the failures that tripped it, so a dependency that is fully down holds the gate through the error-rate half and not only through CPU. And a gate that cannot read the shared breaker store — Redis unreachable, the store quarantined — treats the error rate as unmeasured and refuses, with a reason that names the store; it no longer reads that outage as a healthy 0.0. An operator who must exit regardless still forces the release. This needs the free core at 1.14.0, which ships the shared-service binding and the open-period rate — against an older core the gate keeps reading 0.0.

[1.4.0] - 2026-09-07

Added

  • BALDUR_RATE_LIMIT_THROTTLE_INTEGRATION_RECOVERY_STRATEGY is now read. It has always been documented as the control over how the Adaptive Throttle gives back the limit a 429 took, and nothing consulted it: every deployment got the gradual walk. Set it to immediate and the cooldown end restores the limit in one step instead of over the recovery dampening interval, which is the answer when the gradual walk's first step costs you more protected-tier capacity than the 429 did. Neither strategy can lower the limit, so immediate is safe to leave on; the global BALDUR_THROTTLE_RECOVERY_DAMPENING_ENABLED still applies to every recovery cause, while this one is scoped to the 429 path alone.

Changed

  • Without an active licence, PRO behaviour reached by direct import is now withheld instead of performed. Registration has always been gated on the entitlement, but a deployment whose licence had lapsed — or that never had one — could still reach the paid behaviour by importing and calling it, which is how the notification hub, the scheduled and graceful configuration paths, the incident-record writers and background dead-letter overflow eviction were reached in practice. Each of those now checks the entitlement at its own entry point. A notification returns the same shape the hub already returns for a cooldown or a maintenance window — the call succeeds, nothing is delivered, and the reason reads not_entitled — so no caller changes and the existing suppressed-notification metric counts it. The incident-record writers and their auto-sealing lane refuse; readers stay open on purpose, because a history captured while licensed is your data. Three things keep working deliberately: circuit-breaker OPEN and CLOSED alerts fall back to the free core's own push (set BALDUR_META_WATCHDOG_SLACK_WEBHOOK_URL for it — that push carries no deduplication, so a flapping breaker can repeat), records already written while licensed stay readable, and an emergency mode activated while licensed still expires and restores on its own rather than stranding the deployment in it. On a lapsed licence, expect silence from the paid channels rather than delivery — the free core's alert push is the fallback worth configuring before you need it. This needs the free core at 1.11.0, which carries the entitlement check every gate calls, so the minimum supported core moved up with this release.

  • A licensed boot that names a Redis URL now selects the cross-host audit hash chain by itself. The hash chain's default mechanism is a file lock, which does not span hosts — so on the paid tier's own documented topology, a cluster of two or more pods, each pod kept an independent chain with its own genesis and its own sequence space, and on a shared read-write volume they could overwrite each other's sequence numbers outright. The setting that fixes this has existed and been documented all along; what changes is that the obligation is no longer discharged by asking an operator to read a page. There is no pod-count signal to gate on, so startup gates on the only affirmative fact available: whether anyone named the Redis URL this chain would actually dial. Where that is true and the entitlement is active, the distributed chain is selected. An explicit setting of either polarity still wins — BALDUR_AUDIT_DISTRIBUTED_HASH_CHAIN=false keeps the file-locked chain, and a value you set stays distinguishable afterwards from one the product inferred. Two things to know before upgrading: the chain now dials the Redis you named, so it must be reachable from every host that writes the trail; and, paired with this release's free core, an explicit true with no Redis client resolvable fails startup instead of quietly building the local chain you did not ask for.

  • Shadow-budget reconciliation no longer freezes circuit breakers. A pending reconciliation larger than 5 % used to flip the circuit breaker's Freeze Mode — in whichever web worker happened to compute the budget, so what it produced was a process-local freeze on one worker rather than a fleet decision. Freeze Mode is now the breakers' own view of Emergency Level 3, and a deployment-freeze signal is not that; the chaos safety guard already reads this service's own deployment verdict for the question the flip was standing in for. The threshold, the warning and the audit event on that path are unchanged, so reconciliation still reports what it found. If you were relying on a large pending adjustment holding breaker transitions, that is Emergency Level 3's job now — and, with this release's free core, holding at Level 3 is what breakers actually do.

Fixed

  • The Adaptive Throttle no longer ramps back up while another worker's 429 cooldown is still in force. RATE_LIMIT_COOLDOWN_END used to be a prediction: each worker armed a timer for the expiry it had learned, and a peer's later 429 — the ordinary shape of a storm, not an edge — extended the shared cooldown through the store, reaching that worker through no event at all. Its timer then fired on time, the throttle dropped its 429 reduction, recovery dampening started, and outbound throughput climbed back into a provider that was still rate-limiting the fleet; on the Redis event bus one early announcer released every worker's throttle at once. The all-clear is now read from the shared store before it is published, so a worker that missed the extending 429 learns the new expiry and stays reduced instead. While the store cannot be read at all, every announcement in that process is held rather than guessed — the throttle keeps its reduction through the outage and releases at most one probe interval late, never early. This needs the free core version that ships the verified announcer; against an older core the timer behaviour is unchanged. Two consequences worth knowing: on the Redis event bus each throttle now receives one all-clear per worker that observed the key's 429 — the state after each is identical, but the audit trail records a recovery start per copy — and an operator clear() releases the inbound reduction only where the cleared key's 429 was observed, so on a multi-worker deployment the reduced limit can stand until the observing worker's original expiry.
  • The 429 kill switch now covers the whole 429 path. With BALDUR_RATE_LIMIT_THROTTLE_INTEGRATION_ENABLED=false the handler that reduces the limit stopped, but the handler that recovers it did not check anything — so the first cooldown end still moved the limit to 80 % of its configured start, a fifth of your outbound throughput given away on a throttle nothing had reduced. What the code called a recovery was, on an untouched limit, a reduction. The recovery now runs only where this instance actually took something, so the switch means what it says: with it off, no 429-driven path touches the limit at all. On an instance that is reduced, an operator flipping the switch mid-episode still gets the reduction back at the cooldown end, unchanged.
  • Consecutive 429s from the same provider now compound. Each cooldown end restored the limit to 80 % of the configured start rather than to what that 429 episode had taken, so a three-in-a-row storm ended no lower than a single 429 at the steepest ratio would have put it — the escalating REDUCTION_RATIO_1/2/3 ladder had no cumulative effect on any service whose cooldowns expire between retries, which is the ordinary shape. Each episode now records its own take-point and gives back exactly that, so a storm backs you off further each time as configured. Expect a lower steady-state limit during a sustained storm than previous releases produced; the gradient walks it back up as response times recover.
  • A recovery can no longer lower the limit. Every step of the 429 recovery now writes at most an increase, so a misconfigured BALDUR_THROTTLE_RECOVERY_STEP_1_PERCENT — the three step percentages are validated individually and never against each other, so 0.1 is accepted — can no longer turn each cooldown end into a further 90 % cut. The same clamp settles two orderings that used to lose capacity silently: a 429 landing mid-recovery no longer discards where that recovery was heading, and an emergency, kill-switch or error-budget release arriving over a 429 recovery no longer steps the limit down to its own, lower endpoint. No recovery ends below where an earlier one would have ended it.
  • The Audit Trail now records the throttle decisions it was always meant to record. Six families of throttle event — every 429-driven limit reduction, every recovery start and completion, and every error-budget limit change — were assembled and then dropped on the way to the trail: the recording call carried fields the recorder had no place for, and the failure was swallowed by the same fail-open guard that keeps an audit problem from breaking live traffic. The only trace was one debug line naming a type error. On a licensed deployment, where the trail is on by default, a compliance timeline of automatic throttle decisions was therefore missing exactly the events an incident review goes looking for, with nothing in the trail to indicate the gap. Those records now land, each with its own fields — the rate-limit key, the reduction actually applied, the limit the episode is set to give back, and the error-budget percentage that drove a change. Expect new rows in the throttle portion of your trail after upgrading; the volume follows your own 429 and error-budget activity, and the per-event sampling controls still apply.
  • Throttle audit records now carry a severity someone chose. Six of the throttle actions the trail records had no severity assigned and fell back to info, so an operator pulling the kill switch was filed at the same level as a routine limit adjustment, while a hard stop — the same class of deliberate, system-wide halt — was filed as critical. An error-budget critical sat at info beside an identically severe SLA critical. Anyone reviewing an incident by filtering the trail on severity, which is the first thing anyone does with a trail this size, was reading past exactly the entries that mattered. Both kill-switch actions and the error-budget critical now file as critical, the error-budget warning as warning, and the remaining two keep the info they already had — now as a stated value rather than a default. Nothing else about these records changes: the same events are recorded, at the same time, with the same fields, and none of them became an immediately synced cascade event. A severity-based alert or filter over your trail will start matching kill-switch and error-budget entries it silently skipped before.
  • The chaos safety guard's "70 % of breakers are OPEN" block can now block an experiment. It built a fresh threshold monitor for each check, so the consecutive-trigger count the block required could never get past the first of the two ticks it needed — in every release that shipped it, a chaos experiment launched into a collapsing fleet was permitted by a guard whose panic block could not fire. The check now reads the shared monitor's instantaneous verdict, which is the right reading for a pre-flight safety check: it needs the condition now, not two ticks of hysteresis, and taking the reading no longer advances the count it is reading. A fleet state that cannot be read at all is no longer treated as a fleet that is fine — with the guard's fail-safe setting at its default, an unreadable breaker store blocks the experiment and says so, where before it fell into the fail-open branch and a store blip permitted the run. Expect experiments to be refused where they were quietly allowed, both when the fleet really is 70 % open and when the breaker store cannot be reached; the block message tells the two apart.
  • The governance sweep no longer reports a page it withheld as delivered. The emergency-expiry sweep is the one lane deliberately left running without a licence — it is the only remaining exit from a persisted emergency state — so its notification half is suppressed there by design. Both governance notification helpers branched on success alone, and a suppression is reported as a success; one of them additionally named slack as the channel when nothing had been sent at all. A lapsed deployment therefore recorded an auto-restore notification as sent, to a channel, for a page nobody received. Suppressed, failed and delivered are now told apart at both sites, so the log line and the trail agree with what happened.

[1.3.0] - 2026-09-04

Added

  • An entitled boot now says what it actually registered. Every step of PRO service registration is guarded on its own, so a licensed module that fails to import leaves its slot on the free-core default and startup carries on — an operator paying for thread-pool bulkhead isolation could be running semaphore isolation, with nothing to show for it but one warning that scrolled past at boot. Startup now ends on a single verdict: a entitlement.pro_registration_failed warning naming each step that did not land when anything failed, and the existing informational line when everything did. baldur_entitlement_registration_failures carries the count for the life of the process, so the condition reaches a dashboard rather than living only in a boot log nobody re-reads, and the startup report's extensions.reports block carries the same verdict for anything reading it programmatically. The verdict is derived from the steps that actually ran — no expected-slot list is written down anywhere — so it cannot drift from the registration path as services are added. A deployment without Celery is not reported degraded, because a task surface absent by design is not a failed step; a task module that is present and broken is one, and is named. This needs the free core at 1.10.0, which ships the counter and keeps each startup hook's report: against an older core the report is discarded and there is no counter to write, so the minimum supported core moved up with this release.

Fixed

  • A kill switch flipped on one server now reaches the governance gate on every server. The gate's own enabled-check read a process-local mirror that nothing refreshed, so staleness across processes was unbounded: automation on a second node kept passing the pre-action gate indefinitely after someone pulled the switch, and the free core's newly published flip event had nothing to land on. The check now re-reads the shared state backend on a cache miss, so the delay is bounded by one cache window — or immediate wherever the event bus delivers the flip. Both halves are required for that bound: pair this release with free core 1.10.0, which publishes the flip and invalidates the gate's cache. Two related corrections ride along. A Full Stop is no longer lifted by the re-enable event that is now genuinely reachable — the throttle's deactivation handler used to start recovery dampening while a Full Stop was active, lifting the limit off zero with nothing able to re-apply it, and a Full Stop again ends only where it is meant to (its "recovery started" log line no longer fires when no recovery was started). And concurrent misses on one key now perform one backend read per process instead of one per racing thread, which matters most when the backend is slow — exactly when the read is serialized behind the lock every kill-switch reader takes. One limit worth knowing: the shipped state backends swallow read errors and return nothing, so a backend that has gone silently unreadable leaves the gate serving this process's last known value for a full cache window without raising; the bound above holds while the backend is readable.

Removed

  • TTLCache — withdrawn from the governance surface. The gate's result cache was a hand-rolled class re-exported from that package; it is now the framework's own shared TTL cache, which is what made the single-read-per-process behavior above possible. Nothing in the documented governance surface referenced the class.

[1.2.0] - 2026-08-31

Added

  • PagerDuty incident close — the escalation transport can now send the Events API resolve verb for an incident it opened, using the same routing key and dedup key as the trigger that opened it. Its first user is the escalation channel self-test, which closes its own synthetic incident in the same call, so verifying a channel no longer leaves a real incident open for someone to remember. If the close does not land, the self-test result names the cause and says to close it by hand. Component-failure incidents are unchanged and remain manual-close.

  • The Runtime Config editor now tells you whether a change reaches your running workers. Applying an edit reported success and nothing else, so a circuit_breaker change that only takes effect at the next worker start looked identical to one that took effect immediately — and mid-incident that difference is the whole question. Every response that reports a configuration change (the three read endpoints, all three apply strategies, reset-to-defaults, and a history rollback) now carries a runtime_apply block with a mode, a convergence bound, and one sentence; the console renders it as a badge beside the apply strategy. The mode is derived from each domain's actual invalidation wiring rather than written by hand, so it cannot claim an effect that did not happen: circuit_breaker reports live with its convergence bound wherever delivery is running, and every other domain reports unverified — the value is stored, and a running worker keeps the old one until it restarts. Where the Audit Trail is enabled, each CONFIG_CHANGE record carries the same statement as an application field, so a compliance timeline no longer reads a stored value as one that was in effect from the moment it was written.

  • A circuit_breaker edit now reaches workers that are already running. Raising recovery_timeout mid-incident, or tightening failure_threshold during a rolling deploy, used to mean restarting workers to act on your own decision — and an autoscale in between left the fleet split across two configurations for one service. Every process re-reads the stored section on a poll (BALDUR_RUNTIME_CONFIG_WATCH_INTERVAL_SECONDS, 30 seconds by default) and rebuilds its breakers in place, so the change lands within that bound without discarding in-flight protection state: open circuits stay open, recorded failure rates survive, and per-service threshold overrides are untouched. The interval is the same number the runtime_apply block reports, so the promise and the mechanism cannot drift apart — set it to 0 and the domain honestly drops back to reporting itself stored-only. Where the event bus runs on Redis a peer usually picks a change up within a second, but the poll is what carries the guarantee. Deployments where the delivery does not run — a gunicorn worker with no Baldur hooks, a Celery prefork child, the poll switched off — say so rather than claim a bound they cannot keep. A new baldur_runtime_config_installed_fingerprint gauge answers the fleet-wide question the per-process response cannot: workers serving the same configuration publish the same number, so any spread across your scrape is real divergence.

  • The canary watchdog now runs. Canary Recovery's supervision jobs existed but were never scheduled, so the work they were meant to do quietly did not happen. On an entitled install they now run on their own — on Celery deployments off the beat schedule, elsewhere off the in-process scheduler — and the always-on work is deliberately non-mutating, so the activation changes no rollout by itself. Every five minutes each live rollout's config-type lock is renewed, closing a real gap: the lock used to lapse at its 30-minute timeout while the rollout was still under supervision, reopening the door to a second rollout on the same config type; a renewal that finds the lock in a different rollout's hands raises a lock-conflict alert instead of absorbing it. A stalled rollout now produces a delivered alert — routed through the notification hub, deduplicated per rollout — rather than relying only on the Meta-Watchdog's health probe, and rollout metrics are collected on a two-minute cadence. The two mutating actions are separate opt-ins that ship off. BALDUR_CANARY_WATCHDOG_ENABLE_AUTO_PROMOTE promotes a stage whose observation window has elapsed through the same governance and health gates as a manual promote; only stages created with auto-promotion marked participate, and a promotion attempted from a stale view of the rollout is refused, so each observed window advances at most one stage. While governance blocks the sweep, that is now visible: baldur_canary_governance_blocked_total counts the block with its reason, and baldur_canary_pending_promotion gauges the rollouts waiting behind it — both metrics existed but could never record before this release. BALDUR_CANARY_WATCHDOG_ENABLE_AUTO_ROLLBACK rolls back a rollout already judged stalled once it stays stuck past BALDUR_CANARY_WATCHDOG_AUTO_ROLLBACK_AFTER_MINUTES (60 by default), audited under the system's own identity as a flagged governance bypass. Two things before you turn anything on: the worker process must call baldur.init() — a worker without it skips the jobs with a warning naming the fix — and resolve rollouts that are already stalled first, because the first scan alerts once per still-stalled rollout and, with auto-rollback on, any of them past the timer is rolled back on the next scan.

Changed

  • The Audit Trail now turns itself on. With an active entitlement, BALDUR_AUDIT_ENABLED no longer has to be set: startup switches the audit subsystem on and selects the hash-chain backend, so a licensed deployment records a tamper-evident trail of configuration changes and healing decisions from first boot rather than recording nothing until someone finds the flag. The cost is real and worth planning for before you upgrade — a write-ahead log, a background sync worker, and a hash-chained audit_<date>.jsonl growing under BALDUR_AUDIT_LOG_DIR, whose default logs/audit is a relative path that a container wipes on every restart; point it at a persistent volume first. The chain is keyed by BALDUR_SECRETS_AUDIT_SIGNING_KEY, which a production deployment already had to provide — that secret gate is unchanged and fires whether or not audit is on. Setting BALDUR_AUDIT_ENABLED yourself still wins in both directions: the hook acts only when the variable is unset, so an explicit false keeps a staging clone or a data-residency hold silent, and an install that already sets true sees no change. Each step of the switch-on yields rather than forces — a backend your application selected is left alone, an unwritable log directory rolls the change back instead of parking audit on a backend that cannot be constructed, and any failure is a warning that leaves the rest of startup untouched. Without an entitlement nothing changes: audit stays off, and turning the switch on there still requires selecting a backend yourself, which startup now says plainly when it is missing.

  • Automatic watchdog recovery can lift an operator's manual circuit-breaker block, and this release is where that starts to matter. The core now enforces a manual Block on the traffic path, so a Block that was previously advisory actually rejects requests — but the Meta-Watchdog's circuit-breaker recovery, which runs only if you set BALDUR_META_WATCHDOG_RECOVERY_ENABLED=true (off by default), force-closes the breakers it finds OPEN without asking whether an operator pinned them. A single breaker sitting OPEN for five minutes is enough to start a pass, and that pass then force-closes up to five OPEN breakers, oldest first — a manual Block among them included, whatever its own age. What replaces the Block is not a return to automatic protection but a force-Allow held for BALDUR_CB_MANUAL_OVERRIDE_TTL_MINUTES (90 minutes by default): traffic returns to the dependency you just took out of rotation, and the breaker will not trip on its own until that pin expires. If you rely on manual blocks, leave the flag at its default; with it on, treat a Block as something to re-affirm rather than as something that holds. Teaching recovery to skip operator-pinned rows is a known gap, and it is not closed in this release.

  • Two circuit-breaker fields are no longer editable in the console: rate_limit_distributed and cluster_state_propagation_enabled. Both are deploy-time topology flags whose consumers read the environment — one gates a subscription set up once at startup, the other latches on first use and stops re-reading — so a value stored through the editor was never read by anything, restart or not, while the widget updated and the audit trail recorded a change that could not take effect. Set them through BALDUR_CB_RATE_LIMIT_DISTRIBUTED and BALDUR_CB_CLUSTER_STATE_PROPAGATION_ENABLED instead; every other circuit-breaker field is unchanged.

  • Background per-domain DLQ eviction now covers the domains you actually use. The 60-second sweep decided which domains to trim by reading the metric-label registry, which in practice held only Baldur's six built-in domain names — so for a real application the per-domain sweep never acted, and each domain was held near 100 % of max_size_per_domain by the store-time check alone. It now enumerates the domains that have entries, straight from the DLQ store, and trims each one over the limit down to 70 % of it — the same watermark the rest of the eviction path uses. Two consequences on the first run after upgrade: per-domain trimming reaches application domains for the first time, so it deletes more than the previous release did; and DLQ retention no longer moves when you change max_registered_domains, which is a metric-cardinality setting and never should have decided which rows are deleted. If you do not want the trim, raise max_size_per_domain before upgrading.

  • The throttle-replay TTL-expiry counter now carries the same domain label as every other DLQ metric. baldur_dlq_throttle_replay_ttl_expired_total wrote the stored domain string straight onto the label, so one logical domain could appear under two label values — the canonical one everywhere else, the raw spelling here — and the counter was outside the max_registered_domains cardinality ceiling entirely. It now resolves the domain like the rest of the DLQ families: a registered domain keeps its own series under the canonical spelling, and anything past the cap collapses into the shared OTHER_DOMAIN series.

  • Escalation metrics now tell a delivered page from a logged one. baldur_watchdog_escalation_total stamped result="sent" whenever escalate() reported success — but the notification seam substitutes a logging adapter whenever the configured transport cannot be resolved, and that adapter reports success unconditionally. The Meta-Watchdog's own channel recovery registers exactly that fallback the moment your real channel probes unhealthy, so sent overstated precisely when it mattered most; dry_run_mode short-circuits to success with the same shape. A delivery that only reached this process's log, or that dry-run accepted, now records the new result value result="logged", and sent means a channel that leaves the host — Slack, Teams, PagerDuty or a webhook — accepted the page. A dashboard or alert selecting {result="sent"} will stop counting log-only and dry-run installs: that is the correction, and it is the value the Web Console's new "humans paged" counter reads. fallback (every configured channel was tried and failed) and suppressed (cooldown, dedup or maintenance) are unchanged, as is the Meta-Watchdog status payload's own escalation count — that one counts pages raised, so on a log-only install it legitimately exceeds the delivered count.

  • Escalation PagerDuty severity now follows the event's level. At the default BALDUR_META_WATCHDOG_PAGERDUTY_SEVERITY=critical, each escalation is sent at its own level — component pages stay critical, while the informational channel self-test is sent as info instead of raising a critical-severity page for a test. Setting the variable to any other value keeps its previous meaning exactly: a forced global severity applied to every event, self-test included.

  • Breaking: the canary watchdog's two automatic actions now default to off. enable_auto_promote and enable_auto_rollback shipped as true, which was moot while nothing scheduled the watchdog — but a deployment that wired the watchdog up by hand loses both actions on upgrade until it sets BALDUR_CANARY_WATCHDOG_ENABLE_AUTO_PROMOTE=true / BALDUR_CANARY_WATCHDOG_ENABLE_AUTO_ROLLBACK=true. Off is the deliberate posture: activating the lane must not start promoting or rolling back rollouts on installs that never had either. One pairing note: the auto-promote path now hands the canary service a stage-index guard (the stale-view refusal above), so the lane needs this release's baldur-pro beside it — with an older one, the opt-in path fails loudly in the task's per-rollout errors rather than promoting unguarded.

Removed

  • ThrottleLuaScripts, RedisThrottleLimitManager, SafeOpenConfig, SafeOpenFallbackManager, ServiceSafeLimitState, RedisConnectionState, get_safe_open_fallback_manager and reset_safe_open_fallback_manager — withdrawn from the Adaptive Throttle service, together with the shared-limit claims the documentation made for them: the Adaptive Throttle guide's BALDUR_REDIS_URL row, and the rate-limiting guide's fleet-scope table, which listed the limit value as shared across your fleet. What you admit is unchanged: the adaptive limit has always lived in each worker's own memory, and the retry cooldown the Rate Limit Coordinator shares across workers is unaffected.

  • BALDUR_CANARY_WATCHDOG_SLACK_CHANNEL — it accepted a Slack channel name, but the value sat on a path that expects a channel type, so it never reached a delivery target and the alerts it claimed to route degraded to a log line. Watchdog alerts now route through the notification hub as operations-category events; to give them a dedicated Slack channel, set it under the operations key of BALDUR_CHANNEL_ROUTING_CATEGORY_SLACK_TARGETS.

Fixed

  • The Daily Report's "Auto-replay" line now counts only replays the system started on its own. A batch replay you launched from the console or the REST API was counted there as well, so a digest section framed as unattended recovery work included work you had just done by hand — and the same batches inflated the Auto-Processing "Recovered" count they feed. Operator-initiated batches no longer appear on that line; they remain visible in the digest through the DLQ resolved count, and in the console and audit trail as before. Automatic replays (circuit-close recovery, traffic-aware, throttle-aware, and the DLQ consumer) are unaffected.
  • Throttle-recovery replays now record their own provenance. The automatic sweep that replays throttle-rejected entries after a recovery stamped each resolved entry with manual_replay, so the compliance record attributed unattended work to an operator; it now stamps throttle_replay, matching the single-entry throttle-aware replay path.
  • Throttle-recovery DLQ replays now leave the same forensic trail as every other replay path. The automatic sweep that replays throttle-rejected entries after a recovery previously wrote no replay audit record and did not stamp the original failure's trace onto its log lines — a replayed (or failed-to-replay) entry from that unattended path was invisible in the audit trail and unlinkable to its origin. Each attempt now records the replay audit (success, handler failure, and exception alike) and carries the origin trace on its per-entry logs, matching the manual and batch replay paths.

[1.1.0] - 2026-07-17

Added

  • Runtime Config editor — a web console panel to view and change runtime-tunable settings (retry attempts, circuit-breaker thresholds, DLQ limits, and more) from the browser, with a per-section apply-strategy selector, a current → proposed diff confirmation before applying, an out-of-range clamp notice, and an audit-status badge. Replaces hand-crafting PUT calls with curl or editing env vars and restarting.
  • Runtime-config REST surface — read responses now include a per-section version, and PUT /config/{section} accepts an optional expected_version for optimistic-concurrency control. A stale write returns HTTP 409 with the expected/actual versions and the current config so the client can merge and retry; omitting expected_version preserves the previous unconditional write behavior.
  • DLQ auto-replay arming surface — you can now see at a glance whether automatic replay on circuit-breaker recovery is actually armed, and if not, which prerequisite is missing. A new baldur_dlq_auto_replay_armed gauge, an auto_replay block on GET /dlq/cleanup/stats, and a Web Console DLQ-panel badge name the first missing link (PRO/entitlement absent, on-recovery replay disabled, no Celery worker on the dlq_processing queue, unconfigured service→failure-type map, or no registered replay handler), plus a baldur_dlq_replay_dispatch_total{outcome} counter for each recovery dispatch — so a self-healing loop that would otherwise drain nothing is visible before an incident rather than after.
  • Optional reason on a manual DLQ replay (POST /dlq/replay) and retry (POST /dlq/{id}/retry), with matching Web Console inputs; when provided it is recorded in the entry's resolution note and the audit trail.
  • DLQ trace continuity — a failed operation's trace is now linked to its later replay, so "original failure → DLQ capture → replay" reads as one connected story in your tracing tools instead of disconnected traces. The failing request's trace is captured onto the entry and, on every replay path (targeted retry, batch, automatic on-recovery sweep, force-redrive), surfaced on the replay's log line, folded into its audit record, and — when OpenTelemetry is active — attached as a span link (with a searchable baldur.dlq.origin_trace_id attribute) on a dlq.replay span back to the original failure. The link is additive: the replay keeps its own trigger trace, and the origin is attached alongside. Always on and fail-open — a linkage failure never affects a store or replay.

Changed

  • Replay-automation environment variables and config fields renamed for clarity so each name says what it tunes: BALDUR_REPLAY_AUTOMATION_TRACK1_ENABLED → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED, BALDUR_REPLAY_AUTOMATION_TRACK1_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK2_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ADAPTIVE_INITIAL_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK3_ENABLED → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_ENABLED, and BALDUR_REPLAY_AUTOMATION_TRACK3_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_MAX_ITEMS, along with the matching REST config fields. Defaults are unchanged. Breaking: a stored runtime-config override or environment variable set under an old TRACK-ordinal name is ignored and reverts to the field default — re-set it under the new name.

Removed

  • track2_enabled runtime-config field and the scheduled-batch DLQ replay it advertised. The periodic beat job behind it was inert — it dispatched an incompatible task signature to a queue no worker consumes — and the config field had no backing setting, so setting it never had any effect. Automatic replay on circuit-breaker recovery and manual/operator batch replay are unaffected and remain the DLQ drain paths.
  • HedgingResultValidator / ResultMismatchRecord — removed from the Hedging service. No hedging execution path ever constructed or invoked the validator, so the hedged-result divergence detection it implied never ran; removing it makes the hedging surface honest.

Fixed

  • Meta-Watchdog probe and escalation metrics for the daemon_workers component — the catch-all that watches every Baldur background worker thread — are now labeled with the component's own name. They were previously folded into the catch-all other label, so a dashboard could not break out probes or pages for the watchdog's widest component. Metric history recorded before the fix stays under other; new series start under daemon_workers.
  • Live canary evaluation no longer computes promotion verdicts on simulated data. With BALDUR_CONFIG_SHADOW_LIVE_EVALUATION_ENABLED=true but no time-series metrics source connected via set_metrics_provider(), the promotion gate previously evaluated the built-in simulation provider's synthetic series as if they were real canary metrics — a promotion could pass or block on fiction — and the rollout metrics collector fed the same synthetic numbers into stage health metrics. Both now honestly skip with a WARNING (reason: no_metrics_provider): the evaluation gate defers to the other promotion gates and the collector reports no metrics, matching the documented opt-in contract ("comes online once a time-series metrics source is connected"). Once a source is registered, the live path runs as before.
  • Error Budget Gate alerts no longer suppress one another. All three gate alert types (fail-open activation, rate-limit exceeded, circuit-breaker open) previously shared one notification-hub cooldown window, so a warning-level fail-open page could swallow a later critical circuit-open page for the rest of the window. Each alert type now cools down under its own hub key, aligned one-to-one with the gate's local per-type cooldown — a missed page stays diagnosable from either layer's suppression log. The gate's default cooldown also now follows the alert_cooldown_seconds setting instead of a hardcoded 300 seconds.
  • Error-budget fail-safe alerts now cool down per component instead of globally. One component's fail-safe alert no longer silences a different component's within the shared window; two threads racing the same alert can no longer both page; and a failed send no longer starts a cooldown that would suppress the immediate retry.
  • Recovery resume retries now apply jitter to their exponential backoff, so simultaneously failed recoveries no longer retry in lockstep. The schedule's base and cap are tunable via BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_BASE_SECONDS / BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_MAX_SECONDS (previously hardcoded 30 s doubling to a 300 s cap).
  • Error-budget-gate configuration changes now survive a restart. A gate config change made through the REST API or console returned success and took effect in memory, but was written under a store key that nothing read back on boot, so the change silently reverted on the next worker restart. The gate now persists to its own runtime-config section — with version tracking and an audit record — so a tuned threshold, cache TTL, or fail-open policy is retained across restarts.
  • Canary config rollouts now actually take effect. A started or promoted canary rollout previously wrote its config change only to per-cluster cache keys that nothing read, so on the single-deployment setup the running process never saw the change — stage health checks evaluated the unchanged system and reported success, and a rollback reverted nothing. Rollout start, stage promotion, and rollback now apply through the runtime-config manager (the same in-process surface as a console config edit), so the change takes effect, is audited, and is faithfully reverted on rollback — including a rollback of the first-ever change on a domain, which restores the values captured at create time. Creating a rollout for an unknown config type or field is now rejected with a clear error naming the valid vocabulary instead of silently applying nothing.
  • A runtime-config edit made while a canary rollout is in progress can no longer be silently clobbered. A rollout holds a per-section lock for its lifetime and reverts the whole section to its pre-rollout snapshot on rollback — so an editor/REST edit to that section mid-rollout would previously apply and then be silently overwritten by the rollback. Such an edit is now rejected with HTTP 409 (error_code: ROLLOUT_CONFLICT, naming the owning rollout) instead; the Web Console editor also shows a locked badge on the section and disables its Apply. The rollout's own start/promote/rollback still apply — only foreign edits are blocked. Enforcement covers every config writer (editor, SLO, governance, drift, chaos-safety, full/section resets, delayed/graceful scheduled changes, and the chaos-component config persists), and the create-time snapshot is now captured under the lock so it cannot miss a concurrent edit. The escape hatch is to roll back or cancel the rollout, or wait for it to finish.
  • Gradual recovery from Emergency Mode no longer reports itself as an unhealthy background worker. Shortly after a gradual recovery started, its worker was misjudged as stalled — turning the Meta-Watchdog daemon_workers health red for the entire recovery, and paging on-call where an escalation channel is configured — even while the recovery was progressing normally. The worker's health is now judged against its actual step cadence, so a healthy recovery stays healthy while a genuinely hung recovery is still detected.
  • Delayed and graceful runtime-config changes now actually apply. Editing a section whose default apply strategy is delayed (circuit breaker, DLQ, retry, idempotency, security) and clicking Apply previously returned "scheduled" but the value never changed and no clamp or audit was recorded; the change now takes effect after its delay, with the out-of-range clamp surfaced and the change audited.
  • Config history rollback — rolling back a saved configuration version now applies correctly for every section (retry, circuit breaker, DLQ, and the rest), instead of failing for every Pydantic-backed section and silently leaving the configuration unchanged.
  • Daily Report — the Slack digest now delivers the full multi-section report instead of a body cut off at 500 characters and padded with unreadable raw-metadata fields; sections for features not in the current release are no longer shown, and operator-defined custom metrics render under their own heading instead of trailing the error list.
  • Audit trail — a delayed or graceful configuration change is now attributed to the operator who requested it, instead of being recorded as an internal worker. The "who changed what" record is now accurate for every configuration change regardless of apply strategy.
  • Audit trail — an audit record that the central store rejects during recovery (e.g. one oversized or invalid entry the store refuses while accepting its batch neighbors) is no longer silently lost. The background sync worker now holds its position at the failed entry, so its local write-ahead-log file is retained and the entry is retried every cycle instead of being skipped and the file deleted. A persistently-rejected entry raises a CRITICAL audit_sync_worker.cursor_stalled alert and sets a new baldur_wal_sync_cursor_stalled gauge (1 while stalled, 0 once it lands) so the operator can fix the rejection at its source; the entry is never auto-discarded.
  • Audit trail — audit records buffered locally during a central-store outage are now delivered correctly when the store recovers. On recovery replay the background sync worker previously handed the store a raw internal record instead of a proper audit entry, which — depending on the configured store — silently discarded the record (hash-chain store), left the backlog permanently undrained (stdout / database store), or wrote it in the wrong on-disk shape (file store). Replayed records are now delivered as real audit entries with their original action and timestamp preserved, so a compliance trail survives a central outage intact.
  • Concurrent runtime-config edits are no longer silently lost. Previously, two operators (or pods) editing the same configuration section within one read→write window resolved last-writer-wins with no signal — the first operator's change vanished. Edits are now guarded by an optimistic-concurrency version: a stale edit is rejected with HTTP 409 (carrying the current version and values to retry against) instead of overwriting a fresher change, and a scheduled/rollback apply that races an unrelated change auto-retries rather than dropping. A pending (delayed/graceful) change created while another change is being applied is also no longer dropped.
  • Traffic-aware DLQ replay is now deliverable. Its scheduled job was routed to a dlq queue that nothing registers on and no worker consumes, so even with the feature enabled it drained nothing; it now targets the canonical dlq_processing queue that the DLQ worker serves. The feature is off by default, so default deployments are unaffected.
  • DLQ auto-replay on circuit-breaker recovery no longer goes silently inert. When it cannot run — no Celery worker consuming the dlq_processing queue, the service_failure_type_map unconfigured, or no replay handler registered for the domain — the recovery now logs a WARNING naming the remediation and the arming surface reports disarmed, instead of a DEBUG-level skip that left operators believing auto-replay was on. The prerequisites for closing the loop are now documented in the DLQ + Replay guide.
  • Replay provenance is now trigger-accurate. Every successful replay was recorded with resolution type auto_replay regardless of how it started, so an operator's console/API replay was indistinguishable from a system-automatic one in the compliance record. Replays now record their actual trigger (manual, circuit-close auto-replay, traffic-aware, throttle-aware, scheduled batch, or DLQ consumer) and the acting principal in the audit trail.
  • BALDUR_REPLAY_AUTOMATION_* settings (on-recovery enable / max-items, and the service→failure-type map) are now honored even when the runtime-config service is not present, instead of falling back to hardcoded defaults that ignored the environment variables.
  • The Web Console Runtime Config editor's remaining settings domains now take effect. Editing the sla, security, idempotency, notification, forensic, or metrics section previously persisted and displayed the new value, but no running consumer read it — so the edit was inert even after a worker restart. Every consumer of these domains now reads the same layered-settings surface the console writes, so an edit is observed on the next read (idempotency within a ~30s per-process cache; a few construction-captured fields — e.g. notification channel targets, idempotency cache-TTLs — still take effect after a worker restart, noted in the runtime-config-change runbook). Separately, 15 editor widgets that had no backing consumer at all — the forensic collection/masking toggles and the security request-counting/failed-login detection fields — were removed from the editor; their BALDUR_* settings and env vars remain.

[1.0.0] - 2026-06-23

The inaugural PRO release. Entries require a PRO license.

Added

  • Bulkhead — isolates each dependency in its own fixed slice of capacity.
  • Dead Letter Queue + Replay — captures operations that fail on a downed dependency with the context needed to re-run them, and replays the backlog once it recovers.
  • Emergency Mode — sheds non-critical traffic in deliberate steps under stress and holds until the system stabilizes.
  • Adaptive Throttle — caps admitted requests and moves the cap up or down automatically as the service speeds up or slows down.
  • Canary Recovery — rolls a configuration change out to a small slice of the fleet first and automatically restores the previous config if the rollout degrades or stalls.
  • Governance — a safety gate every automated recovery action must pass before running, recording why each action was allowed or blocked.
  • Meta-Watchdog — monitors Baldur's own healing subsystems and pages a human if they stall or go quiet.
  • Daily Report — rolls a day of self-healing activity (recoveries, circuit-breaker trips, errors) into a single digest, formatted for Slack.
  • Unified Notification — one hub that routes, deduplicates, rate-limits, and records every alert Baldur raises, delivering to Slack.
  • Audit Trail — tamper-evident, append-only record of every configuration change and healing decision (who, what, when, why) with masked client IPs.