Prefork Worker Pool

Spawn separate child processes for true CPU parallelism. Each child has its own Python GIL (Global Interpreter Lock — only one thread runs Python bytecode at a time), so CPU-bound tasks don't block each other.

When to use

Getting started

queue = Queue(db_path="myapp.db", workers=4)
queue.run_worker(pool="prefork", app="myapp:queue")

flexiq worker --app myapp:queue --pool prefork

The app parameter tells each child process how to import your Queue instance. It must be an importable path in module:attribute format.

Mapping from Celery: --pool prefork / --concurrency N ≈ flexiq's --pool prefork / workers=N. The default pool differs, though — flexiq defaults to thread, while Celery defaults to prefork.

How it works

1. The Rust scheduler dequeues jobs from storage
2. PreforkPool frames each job as a JSON header line followed by the task payload's raw bytes, and writes it to the least-loaded child's stdin pipe
3. Each child decodes the frame, executes the task wrapper (with middleware, resources, proxies), and writes the result back as the same kind of frame on stdout
4. Reader threads parse results and feed them back to the scheduler
5. The scheduler updates job status in storage

Configuration

Migrating from thread pool

The thread pool is the default. To switch to prefork:

# Before (thread pool)
queue.run_worker()

# After (prefork)
queue.run_worker(pool="prefork", app="myapp:queue")

Everything else stays the same — task decorators, middleware, resources, events, and the scheduler all work identically. The only difference is where task code executes (child process vs. worker thread).

Debugging child processes

Children inherit the parent's stderr, so print() statements and Python logging appear in the parent's terminal.

Enable debug logging to see child lifecycle events:

import logging
logging.getLogger("flexiq.prefork.child").setLevel(logging.DEBUG)

Log output includes:

• child ready (app=..., pid=...) — child initialized and waiting for jobs
• executing task_name[job_id] — job received (DEBUG level)
• task task_name[job_id] failed: ... — task error
• shutdown received — clean shutdown
• resource teardown error — resource cleanup failure

Limitations

• Tasks must be importable: Each child process imports the app module independently. Tasks defined inside functions or closures cannot be imported.
• No shared state: Children are separate processes. In-memory caches, globals, or module-level state are not shared between children.
• Startup cost: Each child imports the full app module on start. This happens once per child, not per job.
• Resource re-initialization: Worker resources (DB connections, etc.) are initialized independently in each child.