Skip to content

PRO Release Notes

Release notes for Baldur PRO — the licensed tier. The open-source core has its own changelog in the baldur-framework repository. Each entry describes an operator-visible capability; a PRO license is required.

The format follows Keep a Changelog.

[Unreleased]

Added

  • PagerDuty incident close — the escalation transport can now send the Events API resolve verb for an incident it opened, using the same routing key and dedup key as the trigger that opened it. Its first user is the escalation channel self-test, which closes its own synthetic incident in the same call, so verifying a channel no longer leaves a real incident open for someone to remember. If the close does not land, the self-test result names the cause and says to close it by hand. Component-failure incidents are unchanged and remain manual-close.

  • The Runtime Config editor now tells you whether a change reaches your running workers. Applying an edit reported success and nothing else, so a circuit_breaker change that only takes effect at the next worker start looked identical to one that took effect immediately — and mid-incident that difference is the whole question. Every response that reports a configuration change (the three read endpoints, all three apply strategies, reset-to-defaults, and a history rollback) now carries a runtime_apply block with a mode, a convergence bound, and one sentence; the console renders it as a badge beside the apply strategy. The mode is derived from each domain's actual invalidation wiring rather than written by hand, so it cannot claim an effect that did not happen: circuit_breaker reports live with its convergence bound wherever delivery is running, and every other domain reports unverified — the value is stored, and a running worker keeps the old one until it restarts. Where the Audit Trail is enabled, each CONFIG_CHANGE record carries the same statement as an application field, so a compliance timeline no longer reads a stored value as one that was in effect from the moment it was written.

  • A circuit_breaker edit now reaches workers that are already running. Raising recovery_timeout mid-incident, or tightening failure_threshold during a rolling deploy, used to mean restarting workers to act on your own decision — and an autoscale in between left the fleet split across two configurations for one service. Every process re-reads the stored section on a poll (BALDUR_RUNTIME_CONFIG_WATCH_INTERVAL_SECONDS, 30 seconds by default) and rebuilds its breakers in place, so the change lands within that bound without discarding in-flight protection state: open circuits stay open, recorded failure rates survive, and per-service threshold overrides are untouched. The interval is the same number the runtime_apply block reports, so the promise and the mechanism cannot drift apart — set it to 0 and the domain honestly drops back to reporting itself stored-only. Where the event bus runs on Redis a peer usually picks a change up within a second, but the poll is what carries the guarantee. Deployments where the delivery does not run — a gunicorn worker with no Baldur hooks, a Celery prefork child, the poll switched off — say so rather than claim a bound they cannot keep. A new baldur_runtime_config_installed_fingerprint gauge answers the fleet-wide question the per-process response cannot: workers serving the same configuration publish the same number, so any spread across your scrape is real divergence.

Changed

  • Automatic watchdog recovery can lift an operator's manual circuit-breaker block, and this release is where that starts to matter. The core now enforces a manual Block on the traffic path, so a Block that was previously advisory actually rejects requests — but the Meta-Watchdog's circuit-breaker recovery, which runs only if you set BALDUR_META_WATCHDOG_RECOVERY_ENABLED=true (off by default), force-closes the breakers it finds OPEN without asking whether an operator pinned them. A single breaker sitting OPEN for five minutes is enough to start a pass, and that pass then force-closes up to five OPEN breakers, oldest first — a manual Block among them included, whatever its own age. What replaces the Block is not a return to automatic protection but a force-Allow held for BALDUR_CB_MANUAL_OVERRIDE_TTL_MINUTES (90 minutes by default): traffic returns to the dependency you just took out of rotation, and the breaker will not trip on its own until that pin expires. If you rely on manual blocks, leave the flag at its default; with it on, treat a Block as something to re-affirm rather than as something that holds. Teaching recovery to skip operator-pinned rows is a known gap, and it is not closed in this release.

  • Two circuit-breaker fields are no longer editable in the console: rate_limit_distributed and cluster_state_propagation_enabled. Both are deploy-time topology flags whose consumers read the environment — one gates a subscription set up once at startup, the other latches on first use and stops re-reading — so a value stored through the editor was never read by anything, restart or not, while the widget updated and the audit trail recorded a change that could not take effect. Set them through BALDUR_CB_RATE_LIMIT_DISTRIBUTED and BALDUR_CB_CLUSTER_STATE_PROPAGATION_ENABLED instead; every other circuit-breaker field is unchanged.

  • Background per-domain DLQ eviction now covers the domains you actually use. The 60-second sweep decided which domains to trim by reading the metric-label registry, which in practice held only Baldur's six built-in domain names — so for a real application the per-domain sweep never acted, and each domain was held near 100 % of max_size_per_domain by the store-time check alone. It now enumerates the domains that have entries, straight from the DLQ store, and trims each one over the limit down to 70 % of it — the same watermark the rest of the eviction path uses. Two consequences on the first run after upgrade: per-domain trimming reaches application domains for the first time, so it deletes more than the previous release did; and DLQ retention no longer moves when you change max_registered_domains, which is a metric-cardinality setting and never should have decided which rows are deleted. If you do not want the trim, raise max_size_per_domain before upgrading.

  • The throttle-replay TTL-expiry counter now carries the same domain label as every other DLQ metric. baldur_dlq_throttle_replay_ttl_expired_total wrote the stored domain string straight onto the label, so one logical domain could appear under two label values — the canonical one everywhere else, the raw spelling here — and the counter was outside the max_registered_domains cardinality ceiling entirely. It now resolves the domain like the rest of the DLQ families: a registered domain keeps its own series under the canonical spelling, and anything past the cap collapses into the shared OTHER_DOMAIN series.

  • Escalation metrics now tell a delivered page from a logged one. baldur_watchdog_escalation_total stamped result="sent" whenever escalate() reported success — but the notification seam substitutes a logging adapter whenever the configured transport cannot be resolved, and that adapter reports success unconditionally. The Meta-Watchdog's own channel recovery registers exactly that fallback the moment your real channel probes unhealthy, so sent overstated precisely when it mattered most; dry_run_mode short-circuits to success with the same shape. A delivery that only reached this process's log, or that dry-run accepted, now records the new result value result="logged", and sent means a channel that leaves the host — Slack, Teams, PagerDuty or a webhook — accepted the page. A dashboard or alert selecting {result="sent"} will stop counting log-only and dry-run installs: that is the correction, and it is the value the Web Console's new "humans paged" counter reads. fallback (every configured channel was tried and failed) and suppressed (cooldown, dedup or maintenance) are unchanged, as is the Meta-Watchdog status payload's own escalation count — that one counts pages raised, so on a log-only install it legitimately exceeds the delivered count.

  • Escalation PagerDuty severity now follows the event's level. At the default BALDUR_META_WATCHDOG_PAGERDUTY_SEVERITY=critical, each escalation is sent at its own level — component pages stay critical, while the informational channel self-test is sent as info instead of raising a critical-severity page for a test. Setting the variable to any other value keeps its previous meaning exactly: a forced global severity applied to every event, self-test included.

Fixed

  • The Daily Report's "Auto-replay" line now counts only replays the system started on its own. A batch replay you launched from the console or the REST API was counted there as well, so a digest section framed as unattended recovery work included work you had just done by hand — and the same batches inflated the Auto-Processing "Recovered" count they feed. Operator-initiated batches no longer appear on that line; they remain visible in the digest through the DLQ resolved count, and in the console and audit trail as before. Automatic replays (circuit-close recovery, traffic-aware, throttle-aware, and the DLQ consumer) are unaffected.
  • Throttle-recovery replays now record their own provenance. The automatic sweep that replays throttle-rejected entries after a recovery stamped each resolved entry with manual_replay, so the compliance record attributed unattended work to an operator; it now stamps throttle_replay, matching the single-entry throttle-aware replay path.
  • Throttle-recovery DLQ replays now leave the same forensic trail as every other replay path. The automatic sweep that replays throttle-rejected entries after a recovery previously wrote no replay audit record and did not stamp the original failure's trace onto its log lines — a replayed (or failed-to-replay) entry from that unattended path was invisible in the audit trail and unlinkable to its origin. Each attempt now records the replay audit (success, handler failure, and exception alike) and carries the origin trace on its per-entry logs, matching the manual and batch replay paths.

[1.1.0] - 2026-07-17

Added

  • Runtime Config editor — a web console panel to view and change runtime-tunable settings (retry attempts, circuit-breaker thresholds, DLQ limits, and more) from the browser, with a per-section apply-strategy selector, a current → proposed diff confirmation before applying, an out-of-range clamp notice, and an audit-status badge. Replaces hand-crafting PUT calls with curl or editing env vars and restarting.
  • Runtime-config REST surface — read responses now include a per-section version, and PUT /config/{section} accepts an optional expected_version for optimistic-concurrency control. A stale write returns HTTP 409 with the expected/actual versions and the current config so the client can merge and retry; omitting expected_version preserves the previous unconditional write behavior.
  • DLQ auto-replay arming surface — you can now see at a glance whether automatic replay on circuit-breaker recovery is actually armed, and if not, which prerequisite is missing. A new baldur_dlq_auto_replay_armed gauge, an auto_replay block on GET /dlq/cleanup/stats, and a Web Console DLQ-panel badge name the first missing link (PRO/entitlement absent, on-recovery replay disabled, no Celery worker on the dlq_processing queue, unconfigured service→failure-type map, or no registered replay handler), plus a baldur_dlq_replay_dispatch_total{outcome} counter for each recovery dispatch — so a self-healing loop that would otherwise drain nothing is visible before an incident rather than after.
  • Optional reason on a manual DLQ replay (POST /dlq/replay) and retry (POST /dlq/{id}/retry), with matching Web Console inputs; when provided it is recorded in the entry's resolution note and the audit trail.
  • DLQ trace continuity — a failed operation's trace is now linked to its later replay, so "original failure → DLQ capture → replay" reads as one connected story in your tracing tools instead of disconnected traces. The failing request's trace is captured onto the entry and, on every replay path (targeted retry, batch, automatic on-recovery sweep, force-redrive), surfaced on the replay's log line, folded into its audit record, and — when OpenTelemetry is active — attached as a span link (with a searchable baldur.dlq.origin_trace_id attribute) on a dlq.replay span back to the original failure. The link is additive: the replay keeps its own trigger trace, and the origin is attached alongside. Always on and fail-open — a linkage failure never affects a store or replay.

Changed

  • Replay-automation environment variables and config fields renamed for clarity so each name says what it tunes: BALDUR_REPLAY_AUTOMATION_TRACK1_ENABLED → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED, BALDUR_REPLAY_AUTOMATION_TRACK1_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK2_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ADAPTIVE_INITIAL_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK3_ENABLED → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_ENABLED, and BALDUR_REPLAY_AUTOMATION_TRACK3_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_MAX_ITEMS, along with the matching REST config fields. Defaults are unchanged. Breaking: a stored runtime-config override or environment variable set under an old TRACK-ordinal name is ignored and reverts to the field default — re-set it under the new name.

Removed

  • track2_enabled runtime-config field and the scheduled-batch DLQ replay it advertised. The periodic beat job behind it was inert — it dispatched an incompatible task signature to a queue no worker consumes — and the config field had no backing setting, so setting it never had any effect. Automatic replay on circuit-breaker recovery and manual/operator batch replay are unaffected and remain the DLQ drain paths.
  • HedgingResultValidator / ResultMismatchRecord — removed from the Hedging service. No hedging execution path ever constructed or invoked the validator, so the hedged-result divergence detection it implied never ran; removing it makes the hedging surface honest.

Fixed

  • Meta-Watchdog probe and escalation metrics for the daemon_workers component — the catch-all that watches every Baldur background worker thread — are now labeled with the component's own name. They were previously folded into the catch-all other label, so a dashboard could not break out probes or pages for the watchdog's widest component. Metric history recorded before the fix stays under other; new series start under daemon_workers.
  • Live canary evaluation no longer computes promotion verdicts on simulated data. With BALDUR_CONFIG_SHADOW_LIVE_EVALUATION_ENABLED=true but no time-series metrics source connected via set_metrics_provider(), the promotion gate previously evaluated the built-in simulation provider's synthetic series as if they were real canary metrics — a promotion could pass or block on fiction — and the rollout metrics collector fed the same synthetic numbers into stage health metrics. Both now honestly skip with a WARNING (reason: no_metrics_provider): the evaluation gate defers to the other promotion gates and the collector reports no metrics, matching the documented opt-in contract ("comes online once a time-series metrics source is connected"). Once a source is registered, the live path runs as before.
  • Error Budget Gate alerts no longer suppress one another. All three gate alert types (fail-open activation, rate-limit exceeded, circuit-breaker open) previously shared one notification-hub cooldown window, so a warning-level fail-open page could swallow a later critical circuit-open page for the rest of the window. Each alert type now cools down under its own hub key, aligned one-to-one with the gate's local per-type cooldown — a missed page stays diagnosable from either layer's suppression log. The gate's default cooldown also now follows the alert_cooldown_seconds setting instead of a hardcoded 300 seconds.
  • Error-budget fail-safe alerts now cool down per component instead of globally. One component's fail-safe alert no longer silences a different component's within the shared window; two threads racing the same alert can no longer both page; and a failed send no longer starts a cooldown that would suppress the immediate retry.
  • Recovery resume retries now apply jitter to their exponential backoff, so simultaneously failed recoveries no longer retry in lockstep. The schedule's base and cap are tunable via BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_BASE_SECONDS / BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_MAX_SECONDS (previously hardcoded 30 s doubling to a 300 s cap).
  • Error-budget-gate configuration changes now survive a restart. A gate config change made through the REST API or console returned success and took effect in memory, but was written under a store key that nothing read back on boot, so the change silently reverted on the next worker restart. The gate now persists to its own runtime-config section — with version tracking and an audit record — so a tuned threshold, cache TTL, or fail-open policy is retained across restarts.
  • Canary config rollouts now actually take effect. A started or promoted canary rollout previously wrote its config change only to per-cluster cache keys that nothing read, so on the single-deployment setup the running process never saw the change — stage health checks evaluated the unchanged system and reported success, and a rollback reverted nothing. Rollout start, stage promotion, and rollback now apply through the runtime-config manager (the same in-process surface as a console config edit), so the change takes effect, is audited, and is faithfully reverted on rollback — including a rollback of the first-ever change on a domain, which restores the values captured at create time. Creating a rollout for an unknown config type or field is now rejected with a clear error naming the valid vocabulary instead of silently applying nothing.
  • A runtime-config edit made while a canary rollout is in progress can no longer be silently clobbered. A rollout holds a per-section lock for its lifetime and reverts the whole section to its pre-rollout snapshot on rollback — so an editor/REST edit to that section mid-rollout would previously apply and then be silently overwritten by the rollback. Such an edit is now rejected with HTTP 409 (error_code: ROLLOUT_CONFLICT, naming the owning rollout) instead; the Web Console editor also shows a locked badge on the section and disables its Apply. The rollout's own start/promote/rollback still apply — only foreign edits are blocked. Enforcement covers every config writer (editor, SLO, governance, drift, chaos-safety, full/section resets, delayed/graceful scheduled changes, and the chaos-component config persists), and the create-time snapshot is now captured under the lock so it cannot miss a concurrent edit. The escape hatch is to roll back or cancel the rollout, or wait for it to finish.
  • Gradual recovery from Emergency Mode no longer reports itself as an unhealthy background worker. Shortly after a gradual recovery started, its worker was misjudged as stalled — turning the Meta-Watchdog daemon_workers health red for the entire recovery, and paging on-call where an escalation channel is configured — even while the recovery was progressing normally. The worker's health is now judged against its actual step cadence, so a healthy recovery stays healthy while a genuinely hung recovery is still detected.
  • Delayed and graceful runtime-config changes now actually apply. Editing a section whose default apply strategy is delayed (circuit breaker, DLQ, retry, idempotency, security) and clicking Apply previously returned "scheduled" but the value never changed and no clamp or audit was recorded; the change now takes effect after its delay, with the out-of-range clamp surfaced and the change audited.
  • Config history rollback — rolling back a saved configuration version now applies correctly for every section (retry, circuit breaker, DLQ, and the rest), instead of failing for every Pydantic-backed section and silently leaving the configuration unchanged.
  • Daily Report — the Slack digest now delivers the full multi-section report instead of a body cut off at 500 characters and padded with unreadable raw-metadata fields; sections for features not in the current release are no longer shown, and operator-defined custom metrics render under their own heading instead of trailing the error list.
  • Audit trail — a delayed or graceful configuration change is now attributed to the operator who requested it, instead of being recorded as an internal worker. The "who changed what" record is now accurate for every configuration change regardless of apply strategy.
  • Audit trail — an audit record that the central store rejects during recovery (e.g. one oversized or invalid entry the store refuses while accepting its batch neighbors) is no longer silently lost. The background sync worker now holds its position at the failed entry, so its local write-ahead-log file is retained and the entry is retried every cycle instead of being skipped and the file deleted. A persistently-rejected entry raises a CRITICAL audit_sync_worker.cursor_stalled alert and sets a new baldur_wal_sync_cursor_stalled gauge (1 while stalled, 0 once it lands) so the operator can fix the rejection at its source; the entry is never auto-discarded.
  • Audit trail — audit records buffered locally during a central-store outage are now delivered correctly when the store recovers. On recovery replay the background sync worker previously handed the store a raw internal record instead of a proper audit entry, which — depending on the configured store — silently discarded the record (hash-chain store), left the backlog permanently undrained (stdout / database store), or wrote it in the wrong on-disk shape (file store). Replayed records are now delivered as real audit entries with their original action and timestamp preserved, so a compliance trail survives a central outage intact.
  • Concurrent runtime-config edits are no longer silently lost. Previously, two operators (or pods) editing the same configuration section within one read→write window resolved last-writer-wins with no signal — the first operator's change vanished. Edits are now guarded by an optimistic-concurrency version: a stale edit is rejected with HTTP 409 (carrying the current version and values to retry against) instead of overwriting a fresher change, and a scheduled/rollback apply that races an unrelated change auto-retries rather than dropping. A pending (delayed/graceful) change created while another change is being applied is also no longer dropped.
  • Traffic-aware DLQ replay is now deliverable. Its scheduled job was routed to a dlq queue that nothing registers on and no worker consumes, so even with the feature enabled it drained nothing; it now targets the canonical dlq_processing queue that the DLQ worker serves. The feature is off by default, so default deployments are unaffected.
  • DLQ auto-replay on circuit-breaker recovery no longer goes silently inert. When it cannot run — no Celery worker consuming the dlq_processing queue, the service_failure_type_map unconfigured, or no replay handler registered for the domain — the recovery now logs a WARNING naming the remediation and the arming surface reports disarmed, instead of a DEBUG-level skip that left operators believing auto-replay was on. The prerequisites for closing the loop are now documented in the DLQ + Replay guide.
  • Replay provenance is now trigger-accurate. Every successful replay was recorded with resolution type auto_replay regardless of how it started, so an operator's console/API replay was indistinguishable from a system-automatic one in the compliance record. Replays now record their actual trigger (manual, circuit-close auto-replay, traffic-aware, throttle-aware, scheduled batch, or DLQ consumer) and the acting principal in the audit trail.
  • BALDUR_REPLAY_AUTOMATION_* settings (on-recovery enable / max-items, and the service→failure-type map) are now honored even when the runtime-config service is not present, instead of falling back to hardcoded defaults that ignored the environment variables.
  • The Web Console Runtime Config editor's remaining settings domains now take effect. Editing the sla, security, idempotency, notification, forensic, or metrics section previously persisted and displayed the new value, but no running consumer read it — so the edit was inert even after a worker restart. Every consumer of these domains now reads the same layered-settings surface the console writes, so an edit is observed on the next read (idempotency within a ~30s per-process cache; a few construction-captured fields — e.g. notification channel targets, idempotency cache-TTLs — still take effect after a worker restart, noted in the runtime-config-change runbook). Separately, 15 editor widgets that had no backing consumer at all — the forensic collection/masking toggles and the security request-counting/failed-login detection fields — were removed from the editor; their BALDUR_* settings and env vars remain.

[1.0.0] - 2026-06-23

The inaugural PRO release. Entries require a PRO license.

Added

  • Bulkhead — isolates each dependency in its own fixed slice of capacity.
  • Dead Letter Queue + Replay — captures operations that fail on a downed dependency with the context needed to re-run them, and replays the backlog once it recovers.
  • Emergency Mode — sheds non-critical traffic in deliberate steps under stress and holds until the system stabilizes.
  • Adaptive Throttle — caps admitted requests and moves the cap up or down automatically as the service speeds up or slows down.
  • Canary Recovery — rolls a configuration change out to a small slice of the fleet first and automatically restores the previous config if the rollout degrades or stalls.
  • Governance — a safety gate every automated recovery action must pass before running, recording why each action was allowed or blocked.
  • Meta-Watchdog — monitors Baldur's own healing subsystems and pages a human if they stall or go quiet.
  • Daily Report — rolls a day of self-healing activity (recoveries, circuit-breaker trips, errors) into a single digest, formatted for Slack.
  • Unified Notification — one hub that routes, deduplicates, rate-limits, and records every alert Baldur raises, delivering to Slack.
  • Audit Trail — tamper-evident, append-only record of every configuration change and healing decision (who, what, when, why) with masked client IPs.