Troubleshooting

Turn on debug logs first — they show the worker claiming, running, and settling jobs:

import { setLogLevel } from "@byteveda/flexiq";
setLogLevel("debug");
// or: FLEXIQ_LOG_LEVEL=debug node app.js

Turn on debug logs first — they show the worker claiming, running, and settling jobs:

FlexiQLogger.setLevel(LogLevel.DEBUG);
// or: FLEXIQ_LOG_LEVEL=debug java -jar app.jar

Turn on debug logs first — the Rust core's own logging is wired through RUST_LOG, independent of Python's logging module:

RUST_LOG=flexiq_core=debug,flexiq_python=debug flexiq worker --app myapp:queue

Turn on debug logs first — the engine logs through the log facade, so a binary that installs no implementation prints nothing at all:

fn main() -> flexiq::Result<()> {
    env_logger::init(); // RUST_LOG=flexiq=debug,flexiq_core=debug
    // ...
    Ok(())
}

Any log implementation works; the filter names the crates, not the SDK.

timeout_ms is the internal, database-stored deadline in milliseconds; the timeout= kwarg above takes seconds and flexiq converts it under the hood.

timeout_ms is the database-stored deadline in milliseconds, and the timeout attribute is a duration literal that expands to it — a bare integer there is already milliseconds, so timeout = 30 is 30 ms, not 30 seconds. Per call, .timeout_ms(30_000) overrides it.

The deadline is enforced by the scheduler's stale-job sweep, not by interrupting the handler: the body runs to completion on its blocking thread and its late result is dropped by the attempt fence.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

queue.requeue_job(job_id)  # True if it was Running and is now Pending

requeue_job() releases the job's execution claim so a healthy worker can pick it up, and preserves the retry budget. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

queue.requeueJob(jobId); // true if it was running and is now pending

requeueJob() releases the job's execution claim so a healthy worker can pick it up, and preserves the retry budget. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

flexiq.requeueJob(jobId); // true if it was running and is now pending

requeueJob() releases the job's execution claim so a healthy worker can pick it up, and preserves the retry budget. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually. FlexiQ does not wrap this one either:

use flexiq::Storage;

// true if it was Running and is now Pending
queue
    .storage()
    .requeue_stuck(&job_id, flexiq::now_millis())?;

requeue_stuck releases the job's execution claim so a healthy worker can re-claim it, preserves the retry budget, and clears any pending cancel request. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

GIL-bound CPU task: a long-running CPU task is holding the GIL (Global Interpreter Lock — only one thread runs Python bytecode at a time), blocking the scheduler thread from dispatching new jobs. The scheduler runs in Rust, but it still needs the GIL to call Python functions.

Fix: switch to the prefork pool for CPU-bound tasks — it runs tasks in separate subprocesses instead of threads, so one task's GIL usage can't block the scheduler.

flexiq worker --app myapp:queue --pool prefork

Node handlers share one thread — the event loop. A long synchronous computation blocks every in-flight job, not just its own (see Execution Models). Offload CPU-bound work to a worker_threads pool (or a native addon) and await it, or set a timeoutMs so a stuck attempt is aborted — the timeout stops tracking the attempt, but a synchronous handler still won't yield the loop until it returns.

Handlers run on real JVM threads (see Execution Models), so one hung handler doesn't stall the others under the default cached pool — it only ties up its own thread. If you sized the pool with concurrency(n) though, enough hung handlers exhaust it and everything else queues up behind them.

close() drains in-flight handlers for 30 seconds, then interrupts and waits 30 more before closing the native worker — a stuck handler delays shutdown by up to a minute. Set a task timeout so a stuck attempt is reaped, and keep handlers interruptible.

Task bodies run on spawn_blocking threads behind a semaphore sized by num_workers, so one hung body ties up its own slot and nothing else — until enough of them hang that the semaphore is exhausted and dispatch stops. #[flexiq::task] refuses async fn for the same reason: every body is synchronous, so none of them can wedge a runtime thread the scheduler needs.

A body that panics is not lost. The dispatch catches the unwind and reports a retryable failure carrying the panic payload:

task `process_data` panicked: called `Option::unwrap()` on a `None` value

So a panic follows the retry policy like any other failure — which also means a panic over state that stays broken (a poisoned Mutex, say) retries into the same panic until the job dead-letters. That is the shape to look for when a task fails identically on every attempt.

Set a task timeout so a stuck attempt is reaped. A macro-expanded body receives its arguments and nothing else, so there is no cancellation flag for it to poll — a long body that must be interruptible has to carry its own.

Worker logs raise KeyError: "task 'myapp.tasks.process' not registered". Task names default to module.function_name; if you enqueue from one import path and run the worker with a different one, the names won't match.

# Check the task name stored in the job
job = queue.get_job(job_id)
print(job.to_dict()["task_name"])  # e.g. "myapp.tasks.process"

# Check what the worker has registered
print([t["name"] for t in queue.registered_tasks()])

Fix: use consistent import paths. If the task is myapp/tasks.py:process, always import it as myapp.tasks.process — not tasks.process (relative) or src.myapp.tasks.process (with an src prefix).

You can also set an explicit name to decouple the task name from the module path:

@queue.task(name="process-data")
def process(payload):
    ...

The worker dequeued a job whose task name was never registered on it — it throws TaskNotRegisteredError (No task registered with name "..."). Register the handler (queue.task(name, fn)) on the worker process.

The worker dequeued a job whose task name has no handler on it — the job fails with no handler registered for task '...'. Register the handler (handle(name, payloadType, fn)) on the worker process.

The worker dequeued a job whose task name it has no body for. The job fails fatally — no retry, because no amount of retrying makes an unregistered task runnable here:

task not registered on this worker: process_data

Declaring the task is not registering it. Every task the worker should run needs its own register call on the builder:

let worker = queue
    .worker()
    .register::<process_data>() // the name is the task, not a string
    .spawn()?;

A task's default name is the function name, not its module path, so two process_data functions in different modules are one task name. Registering both on one worker is refused at spawn() rather than silently dropping a body:

task `process_data` was registered twice on one worker: one of the two bodies would never run

Give one an explicit name to separate them — #[flexiq::task(name = "reports::process_data")].

OperationalError: database is locked

flexiq sets busy_timeout=5000ms to wait for locks, but heavy write loads can still cause contention.

Causes:

• Multiple processes writing to the same SQLite file simultaneously
• Very large batch inserts blocking the writer
• Long-running transactions from external tools (e.g. a DB browser) holding locks

Fixes:

• Avoid opening the database file with other tools while the worker is running
• Use enqueue_many() / task.map() for batch inserts — they use a single transaction
• If running multiple processes, ensure only one worker process writes at a time, or move to Postgres for multi-machine deployments

• Keep workers on one machine and moderate concurrency, or
• Move to Postgres or Redis for multi-process / multi-machine deployments

• Keep workers on one machine and moderate concurrency, or
• Move to Postgres or Redis for multi-process / multi-machine deployments

• Batch your writes — enqueue_batch is one transaction per batch, enqueue is one per job
• Keep workers on one machine and moderate num_workers, or
• Move to Postgres or Redis for multi-process / multi-machine deployments

The connection pool is 8 by default. FlexiQ::open takes that default; FlexiQ::from_storage is where another number goes, though the single-writer constraint caps write throughput regardless of pool size:

use flexiq::{FlexiQ, SqliteStorage, StorageBackend};

let storage = SqliteStorage::with_pool_size("/var/lib/myapp/flexiq.db", 16)?;
let queue = FlexiQ::from_storage(StorageBackend::Sqlite(storage))?;

Set result_ttl on the Queue to auto-purge going forward instead of purging manually every time:

queue = Queue(
    db_path="myapp.db",
    result_ttl=86400,  # Purge completed/dead jobs older than 24 hours
)

There's no built-in TTL option — run the purge above on a schedule instead (a cron job, or a periodic task on the queue itself).

There's no built-in TTL option — run the purge above on a schedule instead (a cron job, or a periodic task on the queue itself).

Retention is a scheduler setting, so a worker can sweep for you instead of you purging by hand. RetentionConfig carries one window per table — uniform(ms) sets them all to the same one:

use flexiq::{RetentionConfig, SchedulerConfig};

let worker = queue
    .worker()
    .register::<process_data>()
    .scheduler_config(SchedulerConfig {
        retention: Some(RetentionConfig::uniform(24 * 60 * 60 * 1_000)),
        ..SchedulerConfig::default()
    })
    .spawn()?;

RetentionConfig::recommended() is the per-table set instead of one window for everything — dead letters outlive task logs, which is usually what you want. The sweep runs on the scheduler's cleanup_interval tick, so it needs a worker running; a producer-only process cleans nothing.

Possible causes and fixes:

1. Scheduler poll interval too high: default is 50ms. Jobs can wait up to one poll interval before being picked up.

   python

   queue = Queue(scheduler_poll_interval_ms=10)  # Poll every 10ms

   Lower values increase CPU/DB usage. Balance based on your latency requirements.

2. Not enough worker threads: all workers are busy. Increase the pool size.

   python

   queue = Queue(workers=16)

3. Rate limiting: the task or queue has a rate limit active — rate-limited jobs are rescheduled 1 second into the future.

   python

   pending = queue.list_jobs(status="pending", limit=10)
   for job in pending:
       print(job.to_dict()["scheduled_at"])

4. Redis push-dispatch fallback window: on Redis, push-dispatch is on by default. Enqueues are announced, delayed ones included, and the worker's own retries and deferrals (rate limit, concurrency, circuit breaker) arm a local timer, so jobs usually dispatch on time. The rest — a resumed queue, a delayed job enqueued before the worker started, a deadline past what one large batch announces or the listener can buffer — falls back to a 2s timer rather than an immediate wake, longer than the ≤200ms polling ceiling it replaced. push_dispatch=False restores polling.

5. Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Possible causes and fixes:

1. Batch size too small: raise batchSize and channelCapacity on runWorker to claim and buffer more per poll — see the execution model.

   ts

   queue.runWorker({ queues: ["default"], channelCapacity: 256, batchSize: 16 });

2. Not enough worker processes: add more worker processes against shared storage.

3. Capped by a limit: check you aren't capped by a per-task maxConcurrent or rate limit.

4. Redis push-dispatch fallback window: on Redis, push-dispatch is on by default. Enqueues are announced, delayed ones included, and the worker's own retries and deferrals (rate limit, concurrency, circuit breaker) arm a local timer, so jobs usually dispatch on time. The rest — a resumed queue, a delayed job enqueued before the worker started, a deadline past what one large batch announces or the listener can buffer — falls back to a 2s timer rather than an immediate wake, longer than the ≤200ms polling ceiling it replaced. pushDispatch: false restores polling.

5. Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Possible causes and fixes:

1. Not enough concurrency: raise .concurrency(n), or let .autoscale(AutoscaleOptions.of(min, max)) grow the pool with queue depth — see Execution Models.

2. Batch size too small: raise .batchSize(n) and .channelCapacity(n) on the worker builder — see Batching.

3. Not enough worker processes: add more worker processes against shared storage.

4. A producer-side gate is deferring the enqueue: the Java SDK throttles on the producer side — a queue.gate(...) (including the built-in Recipes) can defer a job's creation into the future, which looks like latency but is actually delayed enqueue. Check for an active gate on the task; see Rate Limiting.

5. Redis push-dispatch fallback window: on Redis, push-dispatch is on by default. Enqueues are announced, delayed ones included, and the worker's own retries and deferrals (rate limit, concurrency, circuit breaker) arm a local timer, so jobs usually dispatch on time. The rest — a resumed queue, a delayed job enqueued before the worker started, a deadline past what one large batch announces or the listener can buffer — falls back to a 2s timer rather than an immediate wake, longer than the ≤200ms polling ceiling it replaced. pushDispatch(false) restores polling.

6. Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Possible causes and fixes:

1. Batch size too small: unset, SchedulerConfig::batch_size defaults to 8 on Redis and 1 — one job per round trip — on SQLite/Postgres. Set it explicitly, and set max_in_flight to the worker's own parallelism so a deep backlog isn't claimed by one process and starved from its peers.

   rust

   use flexiq::SchedulerConfig;
   
   let worker = queue
       .worker()
       .register::<process_data>()
       .num_workers(8)
       .scheduler_config(SchedulerConfig {
           batch_size: Some(16),
           max_in_flight: Some(8),
           ..SchedulerConfig::default()
       })
       .spawn()?;

2. Scheduler poll interval too high: poll_interval defaults to 50ms, so a job can wait that long before being picked up. Lower it for latency, at the cost of CPU and database load.

3. Not enough slots: every body is running and num_workers is the ceiling. Raise it, or add worker processes against shared storage.

4. Capped by a limit: check the task isn't held by a rate_limit, max_concurrent or max_in_flight_per_task on its #[flexiq::task] attribute — a rate-limited job is rescheduled into the future rather than run.

5. Redis push-dispatch fallback window: on Redis, push-dispatch is on by default. Enqueues are announced, delayed ones included, and the worker's own retries and deferrals (rate limit, concurrency, circuit breaker) arm a local timer, so jobs usually dispatch on time. The rest — a resumed queue, a delayed job enqueued before the worker started, a deadline past what one large batch announces or the listener can buffer — falls back to a 2s timer rather than an immediate wake, longer than the ≤200ms polling ceiling it replaced. .push_dispatch(false) on the worker builder restores polling.

6. Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Causes:

1. Large result payloads: task return values are stored in the database but also held in the scheduler's result buffer briefly. If tasks return large objects (images, dataframes), memory spikes.

   Fix: return a reference (file path, object key) instead of the data itself.

   python

   # Bad — large result stored in memory and DB
   @queue.task()
   def process_image(path: str) -> bytes:
       return open(path, "rb").read()
   
   # Good — return a path
   @queue.task()
   def process_image(path: str) -> str:
       out = path + ".processed"
       # ... write output to out ...
       return out

2. Accumulated job records: without result_ttl, the database grows unbounded. See Database growing too large above.

3. Resource leaks in tasks: a task opens a file or connection and never closes it. Use context managers.

Causes:

1. Large result payloads: return values are serialized and written to storage on every job. Returning large buffers or objects (images, parsed documents) repeatedly pressures the heap — return a reference (file path, object key) instead of the data itself.
2. Accumulated job records: without periodic purging, storage keeps growing. See Database growing too large above.
3. Resource leaks in handlers: a handler opens a file, socket, or worker_threads worker and never closes it. Each leaked handle accumulates for the life of the process — always release what you open.

Causes:

1. Large result payloads: return values are serialized and written to storage on every job. Returning large objects repeatedly pressures the JVM heap — return a reference (file path, object key) instead of the data itself.
2. Accumulated job records: without periodic purging, storage keeps growing. See Database growing too large above.
3. Resource leaks in handlers: a handler opens a connection or file directly instead of through a worker-scoped resource and never closes it. Prefer scoped resources — they're closed for you when the worker shuts down.

Causes:

1. Large result payloads: a return value is encoded and written to storage on every job, and Job::result hands it back as an owned Vec<u8>. Returning large buffers repeatedly is allocation churn on the way in and on the way out — return a reference (file path, object key) instead of the data itself.
2. Accumulated job records: without retention or periodic purging, storage keeps growing. See Database growing too large above.
3. Resource leaks in task bodies: a body takes its arguments and nothing else, so anything it opens is its own to close. A static cache or pool reached from a body lives for the process — bound it, or it grows for as long as the worker runs.

Rust has no garbage collector, so steady growth here is a leak you can name rather than pressure you wait out — one of the three above, or a container the body appends to and never trims.

• Serializer mismatch — producer and worker must use the same serializer too, not just the same codecs.
• Exact error: SerializationError: no codec registered named 'gz' (at dispatch time) or ValueError: no codec registered named 'gz' (at enqueue time).

• Serializer mismatch — producers and workers must use the same serializer, not just the same codecs.
• Exact error: SerializationError: no codec registered named "gz".

• Serializer mismatch — producers and workers must use the same serializer and codec chain.
• no codec registered named '...' — the task was enqueued with named codecs; register the same names with FlexiQ.builder().codec(name, codec) on the worker.
• Generic payload types — a non-Jackson serializer only supports plain Class payloads; use JsonSerializer/MsgpackSerializer (or a non-generic payload type) for TypeReference payloads.

There is no serializer to mismatch and no codec chain to register: the payload codec is the wire CBOR, fixed. The decode failures you can actually get are these, all fatal on the worker rather than retryable:

• unsupported payload codec: tag 0x.. — the envelope's first byte is not TAG_CBOR. The producer wrote a payload by hand instead of through <task>::call(..).
• empty payload: no codec tag — a zero-length payload.
• malformed call envelope: .. — the bytes are CBOR but not the [args, kwargs] call shape.
• task arguments do not match the handler: .. — wrong count, or a type the parameter cannot hold. Argument order is positional and declaration order is preserved, so this is a real signature mismatch.
• call carries N keyword argument(s) — a producer sent kwargs. A Rust task is positional-only, and the call is refused by count rather than silently dropping them.

ModuleNotFoundError: No module named 'flexiq._flexiq'

The compiled Rust extension isn't present for this platform or Python version. Reinstall from a wheel that matches both (pip install --force-reinstall flexiq), or rebuild it locally with uv run maturin develop if you're working from source.

Cannot find module '.../native/index.js'

The native addon isn't built. Run pnpm build:native. Workflows and mesh also require the addon to be built with their cargo features (included in build:native) — a "feature not enabled" error at queue.workflows means the addon was built without them.

UnsatisfiedLinkError: no bundled native library for platform '...'

The jar has no native binary for this OS/architecture. Build the native library for the platform and point -Dflexiq.native.lib=/path/to/library at it. On hardened hosts where /tmp is mounted noexec, extraction succeeds but loading fails — set -Dflexiq.native.workdir=/path/on/exec/volume instead.

There is nothing to load. The engine is a crate compiled into your binary, so this whole class of failure lands at build time instead: a flexiq::workflows or flexiq::mesh that won't resolve is a cargo feature that is off, not a library that failed to load.

flexiq = { version = "2", features = ["workflows"] }

postgres, redis, workflows and mesh are all opt-in this way, and cargo features are additive — turning one on never takes another away. SQLite is statically linked either way, which is also why a runtime image needs no database client library installed.

A failed job's error is the contract's JSON rather than a bare message, so a reader in any shell gets the same shape:

{ "errtype": "TaskError", "message": "card declined", "traceback": [] }

errtype is always "TaskError" and traceback is always [] — Rust has no exception type to name and no stack to attach, and [] is the contract's no-frames case. The message is whatever TaskError::retryable or TaskError::fatal was given, and which of the two the body chose is what decided whether the job retried or dead-lettered on the spot.