Server & wire
The network door in its own tier: whether to reach for flexiq-server at all, the three doors it opens, and what the wire guarantees a caller with no SDK.
The network door in its own tier: whether to reach for flexiq-server at all, the three doors it opens, and what the wire guarantees a caller with no SDK.
This tier sits beside Python, Node and Java rather than inside them, because the network door is not written in a language. Everything here holds whichever SDK — or none — is on the other end of it: one binary to run, three doors, one proto, one kind of credential.
No — this does not replace the mode you already use. Embedded/local mode —
one process, no daemon, straight to SQLite — stays the default and stays why
you'd pick FlexiQ over a broker-backed queue. Server mode is additive: a
separate flexiq-server process that holds the database credential and exposes
it over the network, for the cases where nothing in-process can reach that
credential at all.
| Local mode | Server mode | |
|---|---|---|
| Process shape | Your app opens storage directly | flexiq-server opens storage; a producer or executor that uses one of its doors never needs to |
| Who can enqueue | A process with the Rust core compiled into it — Python, Node or Java | Any process that can speak gRPC or JSON over HTTP — no native binding required |
| Who can run tasks | A worker in the same process, or an attached executor over TCP/Unix | The same executors, reachable now over a gRPC stream too |
| Reach for it when | The default — this is nearly always the right answer | A producer has no native binding to reach for, or the database credential must not live in every caller |
Writing that caller is its own page, one per door: Clients without an SDK for the producer side, Custom executors for the executor side.
Nothing forces an all-or-nothing switch. A deployment can put only its producer on the network door and leave its workers opening storage directly, same as today — the "Try it" section below points at an example of exactly that hybrid.
The prefork pool, and the worker pool underneath it, assume an OS process on
the same machine as the jobs it runs — that's where the CPU-parallelism story
lives. A remote executor, whether it dials in over the attach protocol today or
over flexiq.executor.v1 once a client speaks it, gets none of that: it is a
Celery-style remote consumer, one network hop from the scheduler, with the
capacity of whatever container it happens to run in. Server mode buys you a
network door; it does not buy you more cores.
flexiq-server puts three independent things on the network, not one door with
three names:
| Producer door | Executor door | Admin door | |
|---|---|---|---|
| Package | flexiq.v1 | flexiq.executor.v1 | flexiq.admin.v1 |
| Does | Enqueue, read, cancel work | Claim work, report on it | Pause queues, work the dead-letter queue, list workers, manage periodic tasks and overrides, mint and revoke API tokens |
| Reachable over | gRPC, and plain HTTP with JSON bodies on the same listener | gRPC only — no HTTP binding | gRPC, and plain HTTP with JSON bodies under /v1/admin/ |
| Scope | produce | execute | inspect to read, admin to change, tokens to manage tokens |
A produce token cannot open an executor stream or pause a queue, and an
execute token cannot enqueue — no scope is a hierarchy, and a credential
scoped to one package never opens another. See
the gRPC door for the producer RPCs,
the admin door for the operator's,
API tokens for the credential's
lifecycle, and
the executor door
for the Attach stream.
Every SDK's executor CLI still attaches over TCP or a Unix socket — the
protocol attached executors already
use. flexiq.executor.v1 is dialled by the remote clients — the Go client's
executor package and the Ruby flexiq-executor gem — and by a custom
executor written directly against the proto, in any language with a gRPC
library, which is the same "no native binding" pitch the producer door makes.
Custom executors is how.
raw versus structuredThe producer door takes a job's payload two ways. raw is the CBOR envelope
an SDK already builds, byte for byte — nothing lost, but it needs a CBOR
library. structured takes ordinary JSON values and lets the server build that
same envelope, so a shell script can enqueue with no codec at all. What it
cannot carry, it refuses rather than rounds: integers past 2^53 - 1,
non-finite numbers, and anything JSON has no syntax for. See the full treatment for the one thing it silently normalises instead
— object key order — and what that costs an auto: idempotency key.
examples/polyglot
has a gRPC-speaking variant beside the original shared-database one: a producer
that enqueues over the gRPC door with grpcurl, no SDK installed at all, and a
Node worker that attaches to the scheduler rather than opening the file. That
worker also shows what an executor gives up — it fans out to a second stage,
and with no storage to write the follow-up job into, it sends it back through
the same producer door. The Java worker still opens the database file directly.
| Cost | Mitigation |
|---|---|
| No CPU-parallelism story — a remote executor is a network hop from the scheduler | Colocate the executor with its scheduler when raw throughput matters more than isolation |
| A second process to deploy, watch and restart | Reach for it only when local mode's constraint — a native binding, or the credential itself — is the actual blocker |
No SDK's executor CLI dials the executor door yet | Use TCP/Unix attach for an SDK-based executor; the gRPC stream is for the Go and Ruby remote clients or a custom one |
| TLS is opt-in, not on by default | Set FLEXIQ_GRPC_TLS_CERT/_TLS_KEY to terminate it in process (TLS and mTLS), or put a proxy or mesh in front — either way it covers both doors, since they share one listener |