Skip to content

Your Celery task failed after max_retries. Where did it go?

A task calls something outside your process: a payment provider, a model API, an email service. Tonight that dependency is down for forty minutes. Every task that runs in the window fails, retries, fails again, and after the third retry Celery gives up — max_retries defaults to 3, and once it is exceeded the current exception is re-raised (or MaxRetriesExceededError, when retry() was called without one). The task's state becomes FAILURE. The worker acknowledges the message. The next task starts.

Here is what Celery keeps of that task:

  • With a result backend (and ignore_result off): the FAILURE state and the traceback, filed under the task id. Not the arguments — and you need the id to look it up.
  • Without a result backend, or with ignore_result=True: nothing.
  • In the broker: nothing either. The message that carried the arguments was acknowledged, and an acknowledged message is gone.

So the list of what you lost this morning is your log file, if you logged the arguments. grep MaxRetriesExceededError gives you a count, not a list.

Why retry did not cover it

Retry is for blips. retry_backoff is off by default, so the three attempts land within seconds of each other; turn it on and retry_backoff_max caps the wait at 600 seconds, so the whole budget still fits inside half an hour. A forty-minute outage outlasts every task's retries — and each task spends them alone. Two hundred queued tasks make six hundred calls into an API that is down, and then two hundred tasks are gone. Retry answers "was that a blip?" It has no answer to "the dependency is down; hold this and run it later."

What Celery gives you for this, honestly

  • Reject and a dead-letter exchange. Raise Reject(requeue=False) and a RabbitMQ broker with a dead-letter exchange configured will file the message there. That is broker-level, RabbitMQ-only (a Redis broker has no equivalent), and it takes you as far as "the message is somewhere" — replay is still re-publishing by hand.
  • acks_late and task_reject_on_worker_lost. These protect a task whose worker died mid-run. A task that raised is acknowledged either way.
  • The result backend. A record that it failed, not a copy you can re-run.
  • A transactional outbox. Write the task's name and arguments to a table in the same transaction as the business data and publish from that table, and the task is published if and only if the data committed — the fix for a .delay() that ran before the commit, or never ran because the broker was down at that moment. The contract ends at the broker. Once the message is out, a task that raises after max_retries is acknowledged exactly as above, and the outbox has already done its part. To cover this failure the row would have to stay open until the task succeeds — at which point the table is a job queue, with a poller, row locking across workers, a rule for the row that never succeeds, and the same open question of when the provider is back.
  • Building it yourself. An on_failure hook that writes the task name, args and kwargs to a table; a management command that re-enqueues them; and some way of deciding that the provider is back, so the replay does not run into the same outage. It is a weekend of work, and it is the part of the codebase that never gets a test.

Capturing every task that fails for good

One call in the module that builds your Celery app:

from celery import Celery
from baldur.adapters.celery import setup_baldur_signals

app = Celery("myproject")

setup_baldur_signals(
    app=app,
    task_domain_mapping={
        "myproject.tasks.send_invoice": "billing",
        "myproject.tasks.summarize_document": "llm",
    },
)

That connects Baldur to Celery's task signals. From then on, a failure that arrives after the retries are spent (retries >= max_retries, or from a task with no retry budget at all) is captured with the task name and id, its args and kwargs, the exception type and message, the traceback, a failure-type classification read off the exception (NETWORK_ERROR, TIMEOUT, RATE_LIMITED, EXTERNAL_SERVICE_ERROR, VALIDATION_ERROR, AUTH_ERROR, …), and a recommended action. Intermediate retry failures are not captured — only the attempt Celery gave up on. Nothing changes per task; @app.task stays as it is.

The same signals feed a circuit breaker per domain, so billing has a live state you can read. The signal path records; it does not stop tasks from running while the breaker is open. For the calls you want held back as well, put @baldur.protected("billing") under @app.task — the Celery quickstart shows the two together.

Seeing what you lost

baldur dlq list --pending
baldur dlq list --domain billing --json

Or open the built-in web console's dead-letter panel. Either way, each entry is the task, the arguments it was called with, why it failed, and what to do about it — the list the log file could not give you.

Getting it back

Baldur captured the work; only your code knows how to re-run a task. A replay handler for a Celery domain re-enqueues the task from the captured request:

from baldur.services.replay_service import (
    ReplayHandler,
    ReplayResult,
    register_replay_handler,
)
from myproject.tasks import send_invoice


class BillingReplayHandler(ReplayHandler):
    @property
    def domain(self) -> str:
        return "billing"

    def can_replay(self, failed_op) -> tuple[bool, str]:
        return True, ""

    def replay(self, failed_op) -> ReplayResult:
        request = failed_op.request_data or {}
        send_invoice.apply_async(
            args=request.get("args", []),
            kwargs=request.get("kwargs", {}),
        )
        return ReplayResult.succeeded(failed_op.id, "re-enqueued")


register_replay_handler(BillingReplayHandler())

Register it in a module the worker imports — registration is per process, and the worker is where replay runs. "Succeeded" here means re-enqueued; from there the task's outcome is Celery's again, retries included.

With the handler in place, two ways to drain the queue:

  • By hand, once you know the provider is back: the per-entry Retry action in the console. It runs your handler in the process that serves the console, so register the handler there too; a process without it fails the attempt and spends one of the entry's replay attempts.

  • Automatically, when the dependency recovers. Tell Baldur which failure types the outage produced for the domain, and run a worker on the dlq_processing queue the replay is dispatched to:

    export BALDUR_REPLAY_AUTOMATION_SERVICE_FAILURE_TYPE_MAP='{"billing": ["NETWORK_ERROR", "TIMEOUT", "EXTERNAL_SERVICE_ERROR"]}'
celery -A myproject worker -Q celery,dlq_processing

    When the billing breaker closes again, the captured entries with those failure types go back through your handler. The console's dead-letter panel reports whether the loop is armed and, if it is not, names the prerequisite that is missing — a handler, the map, or a worker on the queue.

What this is not

It is not "replay everything." A task that failed because its input was bad will fail the same way tomorrow. That is what can_replay is for, and why the map lists the failure types an outage produces rather than every type there is; a VALIDATION_ERROR entry waits for a person.

Replay is a bet that the first attempt did nothing. For a task that charged a card and then timed out, that bet is not free — the charge may have gone through and the confirmation never came back. Baldur's idempotency key does not settle that case: a call that raised after the charge went through (the client's read timeout) releases its key, so the replay charges again. The payment provider's own idempotency key does — derive it from the order ID, and the provider answers the repeat with the original charge. Baldur's key covers the other repeat, a run that already completed: the repeat raises IdempotencyDuplicateError instead of running, and Baldur does not hand back the first run's result. It has its own page: Idempotency keys for Python services.

Shared means shared storage. Out of the box the queue lives in process memory, and Celery almost always runs more than one process. Point Baldur at Redis before you start the second worker — BALDUR_REDIS_URL is the one variable — or the entries one worker captured are invisible to the others. The Celery quickstart has the production checklist.

Where to go next