Unique Tasks

Deduplicate active jobs via unique_key — partial unique index, atomic check-and-insert.

Deduplicate active jobs by key — if a job with the same unique_key is already pending or running, the existing job is returned instead of creating a new one:

Celery has no native dedup — you'd add celery-once plus a Redis lock. flexiq needs neither: deduplication is enforced in the queue database.

python

job1 = process.apply_async(args=("report",), unique_key="daily-report")
job2 = process.apply_async(args=("report",), unique_key="daily-report")
assert job1.id == job2.id  # Same job, not duplicated

Once the original job completes (or fails to DLQ — dead letter queue, a holding area for jobs that exhausted their retries), the key is released and a new job can be created with the same key.

Implementation

Deduplication uses a partial unique index: CREATE UNIQUE INDEX ... ON jobs(unique_key) WHERE unique_key IS NOT NULL AND status IN (0, 1) (0 = pending, 1 = running). Only pending and running jobs participate. The check-and-insert is atomic (transaction-protected), so concurrent calls with the same unique_key are handled gracefully without race conditions.