Handle failure

Retries, which errors deserve one, per-attempt timeouts, and where a job goes when it has run out of chances.

Your signup pipeline works while the mail API is up. This page is about the other days.

Four things, in the order you meet them: a job fails and is retried; some failures should not be retried at all, which is error handling; a job that never returns needs a timeout; and a job that has genuinely run out of road has to go somewhere, which is the dead-letter queue. Get these four right and the rest of production is deployment.

Everything here is enforced by the Rust scheduler, not by your process. A worker that dies mid-backoff does not take the schedule with it.

Retries

A task that raises is retried automatically, with backoff — each wait longer than the last, so a struggling dependency is not hammered by every worker at once. Two settings: how many attempts, and how fast the waits grow.

python

@queue.task(queue="emails", max_retries=3, retry_backoff=2.0)
def send_welcome_email(user_id: int, address: str) -> str:
    ...

queue.task("sendWelcomeEmail", sendWelcomeEmail, {
  maxRetries: 3,
  retryBackoff: { baseMs: 2000, maxMs: 300_000 },
});

java

Task<Signup> SEND_WELCOME_EMAIL = Task.of("send_welcome_email", Signup.class)
        .queue("emails")
        .maxRetries(3)
        .retryPolicy(RetryPolicy.exponential(Duration.ofSeconds(2), Duration.ofMinutes(5)));

rust

#[flexiq::task(
    queue = "emails",
    max_retries = 3,
    retry_backoff_ms = "2s",
    retry_max_delay_ms = "5m"
)]
fn send_welcome_email(user_id: i64, address: String) -> flexiq::Outcome<String> {
    println!("welcome mail → {address}");
    Ok(format!("sent:{user_id}"))
}

With a 2-second base the waits run roughly 2s, 4s, 8s, plus jitter so a burst of identical failures does not retry in lockstep. Three retries buys about fourteen seconds of outage; a mail API that is down for ten minutes needs a bigger budget or a longer cap.

A retry re-runs the whole task body

Not the part that failed — all of it. If your task charges a card and then sends a receipt, a failure in the receipt re-charges the card. Make the body safe to repeat (idempotency), or checkpoint the parts that succeeded with durable steps.

Full behaviour, the backoff formula and per-job overrides: Retries.

Which errors deserve a retry

Retrying a timeout is sensible. Retrying a malformed email address is three more identical failures and a slower path to the same conclusion. Classify the error and the second kind fails fast instead.

python

@queue.task(max_retries=3, dont_retry_on=[ValueError])
def send_welcome_email(user_id: int, address: str) -> str:
    ...

retry_on is the whitelist form (only these retry); dont_retry_on is the blacklist. Set one, not both.

queue.task("sendWelcomeEmail", sendWelcomeEmail, {
  maxRetries: 3,
  retryOn: (error) => !(error instanceof ValidationError),
});

retryOn classifies the thrown error: false dead-letters immediately.

java

Task<Signup> SEND_WELCOME_EMAIL = Task.of("send_welcome_email", Signup.class)
        .maxRetries(3)
        .retryOn(error -> !(error instanceof IllegalArgumentException));

retryOn classifies the thrown exception: false dead-letters immediately.

rust

use flexiq::TaskError;

#[flexiq::task(queue = "emails", max_retries = 3)]
fn send_welcome_email(user_id: i64, address: String) -> flexiq::Outcome<String> {
    if !address.contains('@') {
        // Fatal: three more attempts find the same bad address.
        return Err(TaskError::fatal(format!("not an address: {address}")).into());
    }

    // Someone else's transient problem, so: worth another attempt.
    send_mail(&address).map_err(|e| TaskError::retryable(e.to_string()))?;
    Ok(format!("sent:{user_id}"))
}

There is no retry predicate to configure. The body classifies its own failure by which constructor it reaches for, and the macro will not guess: retryable gets the retry policy, fatal dead-letters immediately.

The rule of thumb: retry what is someone else's transient problem — network, rate limit, a 503. Do not retry what is wrong with the input, because no amount of waiting will change it.

More, including how to fail a job deliberately and how errors are recorded per attempt: Error handling.

Timeouts

A failure announces itself. A hang does not — it holds a worker slot until someone notices. Bound every attempt.

python

@queue.task(queue="emails", max_retries=3, timeout=30)
def send_welcome_email(user_id: int, address: str) -> str:
    ...

queue.task("sendWelcomeEmail", sendWelcomeEmail, {
  maxRetries: 3,
  timeoutMs: 30_000,
});

java

Task<Signup> SEND_WELCOME_EMAIL = Task.of("send_welcome_email", Signup.class)
        .maxRetries(3)
        .timeout(Duration.ofSeconds(30));

rust

#[flexiq::task(queue = "emails", max_retries = 3, timeout = "30s")]
fn send_welcome_email(user_id: i64, address: String) -> flexiq::Outcome<String> {
    println!("welcome mail → {address}");
    Ok(format!("sent:{user_id}"))
}

An omitted timeout is 300 000 ms, and .timeout_ms(..) on the call overrides whatever the attribute declared — five minutes is a ceiling, not a decision about this task.

A timeout bounds one attempt, not the job: it counts as a failure, consumes one retry, and the job comes back on the usual backoff. What the deadline can actually interrupt depends on the pool and the language — Timeouts is explicit about what stops and what merely stops being waited on.

Set one on every task you deploy — a fallback is not a bound you chose.

An omitted timeout falls back to 300 seconds, or the queue's default_timeout. Nothing wedges a slot forever, but five minutes is a ceiling rather than a decision about this task.

timeoutMs has no default. A task left unbounded holds its worker slot until it returns, however long that takes.

An unset timeout — or 0 — means no limit. A handler left unbounded holds its worker thread until it returns, however long that takes.

Know what the deadline actually does. Nothing interrupts a running body: every handler runs on a blocking thread the pool does not cancel, and the timeout is noticed afterwards by the stale-job reap, which records the failure and lets the job retry on the usual backoff. The abandoned attempt keeps its slot until the body returns of its own accord.

So the timeout is when you will be told, and when the job will be tried again — not when the work stops. Anything that could block forever needs its own bound inside the body: a client with a connect and read timeout, a bounded recv_timeout, a deadline you check between steps.

The dead-letter queue

When the retries are gone, the job does not vanish and it does not stay in the queue pretending it might still run. It moves to the dead-letter queue — a separate store that keeps the payload, the final error, and the error history from every attempt.

python

for entry in queue.dead_letters(limit=20):
    print(entry["task_name"], entry["error"])

queue.retry_dead(dead_id)   # re-enqueue with a fresh budget

const dead = await queue.deadLetters();
queue.retryDead(deadId); // re-enqueue with a fresh budget

java

List<DeadJob> dead = queue.listDead(50, 0);
String newJobId = queue.retryDead(deadId); // re-enqueue with a fresh budget

The dead-letter reads are not on FlexiQ. queue.storage() is the escape hatch to the Storage trait, which carries all of them:

rust

use flexiq::Storage;

for entry in queue.storage().list_dead(20, 0, None)? {
    println!("{} {}", entry.task_name, entry.error.as_deref().unwrap_or(""));
}

// Re-enqueue with a fresh budget; the new job id comes back.
let new_job_id = queue.storage().retry_dead(&dead_id, None)?;

The per-attempt history is the useful part: "failed the same way four times" is a bug in your code, "three timeouts then a different error" is an unhealthy dependency. Fix the cause before replaying — a retry against an unchanged cause just walks back into the DLQ.

Dead-letter queue covers listing, bulk purge, and letting the queue drain the DLQ on its own for failures that are transient by nature.

The finished task

Putting the four together, the signup mailer from the last page now says what it expects of the world:

python

@queue.task(
    queue="emails",
    max_retries=5,
    retry_backoff=2.0,
    timeout=30,
    dont_retry_on=[ValueError],
)
def send_welcome_email(user_id: int, address: str) -> str:
    ...

queue.task("sendWelcomeEmail", sendWelcomeEmail, {
  maxRetries: 5,
  retryBackoff: { baseMs: 2000, maxMs: 300_000 },
  timeoutMs: 30_000,
  retryOn: (error) => !(error instanceof ValidationError),
});

java

Task<Signup> SEND_WELCOME_EMAIL = Task.of("send_welcome_email", Signup.class)
        .queue("emails")
        .maxRetries(5)
        .retryPolicy(RetryPolicy.exponential(Duration.ofSeconds(2), Duration.ofMinutes(5)))
        .timeout(Duration.ofSeconds(30))
        .retryOn(error -> !(error instanceof IllegalArgumentException));

rust

use flexiq::TaskError;

#[flexiq::task(
    queue = "emails",
    max_retries = 5,
    retry_backoff_ms = "2s",
    retry_max_delay_ms = "5m",
    timeout = "30s"
)]
fn send_welcome_email(user_id: i64, address: String) -> flexiq::Outcome<String> {
    if !address.contains('@') {
        return Err(TaskError::fatal(format!("not an address: {address}")).into());
    }

    send_mail(&address).map_err(|e| TaskError::retryable(e.to_string()))?;
    Ok(format!("sent:{user_id}"))
}

One attempt plus five retries — six runs spread over about a minute (2+4+8+16+32 seconds, plus jitter) — for anything transient; immediate rejection for a bad address; thirty seconds before an attempt is abandoned; and a dead-letter entry with the full history if it still cannot succeed.

Next: Go to production — choosing a backend, running workers as a service, and watching the queue.