The gRPC door

flexiq-server binds one listener per role, and FLEXIQ_GRPC_LISTEN adds a fourth beside attach, the dashboard and the admission webhook. It is the network-side producer door — enqueue, read and cancel over gRPC rather than over a database connection — alongside grpc.health.v1 and server reflection.

FLEXIQ_DSN=postgres://user:pass@host/db \
FLEXIQ_NAMESPACE=prod \
FLEXIQ_GRPC_LISTEN=127.0.0.1:50051 \
flexiq-server

That listener refuses every call until a token exists, loopback included — grpc.health.v1 excepted, so a readiness probe answers on an unprovisioned door. Mint one first — API tokens is the whole lifecycle, and the rest of this page assumes $FLEXIQ_TOKEN holds one.

The address is parsed exactly like FLEXIQ_LISTEN: host:port, a bare :port (which binds loopback, the safe reading), or unix:/run/flexiq-grpc.sock. A Unix socket lands at mode 0660 through the same hardened bind the attach listener uses, so the shared group is a boundary in front of the credential — not a replacement for it. A Unix bind presents a token like any other.

The role ships behind a cargo feature. The published image has it compiled in; a binary you built yourself without --features grpc rejects FLEXIQ_GRPC_LISTEN at boot rather than ignoring it, because a deployment that looks configured and serves nothing on the port its clients dial is the worse failure.

It serves one namespace, and refuses to start without one

FLEXIQ_GRPC_LISTEN requires FLEXIQ_NAMESPACE. An unset namespace is not a neutral default inside the storage layer — it means three different things depending on the call: only the unnamespaced rows to a dequeue, every namespace to a read addressed by job id, and no filter at all to a listing. A network port that could express it would be one bug away from a cross-tenant read, so the gRPC door serves exactly one named namespace and never the ambiguous one.

The cost is worth stating plainly: jobs written without a namespace are invisible over gRPC. A deployment that wants this door sets FLEXIQ_NAMESPACE on the server and configures its producers with the same value. One line, paid once, in exchange for making a cross-tenant read structurally impossible rather than conditionally absent.

What it serves

flexiq.v1.ProducerService — Enqueue, EnqueueBatch, GetJob, ListJobs, CancelJob, QueueStats, SubmitWorkflow and GetWorkflowRun — beside grpc.health.v1 and reflection.

Writing a client against it, in a language with no FlexiQ SDK, is its own page.

# $FLEXIQ_TOKEN is a token you minted. Exported rather than prefixed onto the
# command: a `VAR=x cmd` assignment reaches that one process, and every call in
# this block needs the value.
grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  localhost:50051 list
# The health *call* is the exception and needs no credential. The token is here
# because grpcurl looks a method up through reflection before calling it, and
# reflection does — a probe carrying its own descriptor needs neither.
grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  localhost:50051 grpc.health.v1.Health/Check
grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  -d '{"task_name":"send_email","raw":"","options":{"queue":"emails"}}' \
  localhost:50051 flexiq.v1.ProducerService/Enqueue

A job enqueued this way is an ordinary row: the same scheduler dispatches it and an unmodified SDK worker runs it, with nothing about the job recording which door it came through.

Two things the producer surface deliberately does not offer. It has no admin methods — pausing queues, dead-letter operations, workers, periodic tasks and overrides are the admin door, a separate package behind the inspect, admin and tokens scopes, because an operator surface and a producer surface should not share a credential. And there is no way to name a namespace; it comes from the credential, as above.

Reflection, and what it hands out

The list above needed no .proto on hand, no checkout and no -protoset flag, because the wire contract is embedded in the binary — the same committed descriptor the buf gate builds, so it cannot drift from the schema CI checks. Against a bare image, that is the whole toolchain:

flexiq.admin.v1.AdminService
flexiq.executor.v1.ExecutorService
flexiq.v1.ProducerService
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection

Name a service and list narrows to its methods — the fully-qualified form the Enqueue line above passes as its last argument:

grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  localhost:50051 list flexiq.v1.ProducerService

flexiq.v1.ProducerService.CancelJob
flexiq.v1.ProducerService.Enqueue
flexiq.v1.ProducerService.EnqueueBatch
flexiq.v1.ProducerService.GetJob
flexiq.v1.ProducerService.GetWorkflowRun
flexiq.v1.ProducerService.ListJobs
flexiq.v1.ProducerService.QueueStats
flexiq.v1.ProducerService.SubmitWorkflow

describe answers with the type, comments included. The descriptor keeps source info, so what comes back is the .proto as written rather than a shape stripped of its reasons — which is why the two arms below explain themselves without a second page:

grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  localhost:50051 describe flexiq.v1.EnqueueRequest

flexiq.v1.EnqueueRequest is a message:
message EnqueueRequest {
  // The registered task to run. Not validated here; see the service comment.
  string task_name = 1;
  // The job's body.
  //
  // Setting no arm is an error: an absent body is not an empty one, and
  // `raw = ""` is a zero-length payload.
  //
  // Both arms end up as the same envelope in storage. They differ in what they
  // can express: `raw` carries anything the format does, `structured` carries
  // what JSON does. Pick `raw` if you have a CBOR library.
  oneof body {
    // The tagged envelope from BINDING_CONTRACT.md — one tag byte followed by
    // CBOR [args, kwargs]. Reaches storage untouched.
    bytes raw = 2;
    // Arguments the server encodes into that same envelope. Refuses the values
    // it cannot encode losslessly rather than rounding them, and does not
    // preserve object key order — see StructuredArgs.
    .flexiq.v1.StructuredArgs structured = 3;
  }
  .flexiq.v1.EnqueueOptions options = 4;
}

describe takes a service the same way, and on flexiq.v1.ProducerService it returns that service's whole comment block — the list of what the door deliberately does not offer and does not promise, which is worth reading once before building against it.

That is the loop, and it closes without a file: list to find the method, describe to learn the message, -d with a JSON body to call it. A build pipeline that wants the same contract without a running server downloads it instead — Getting the contract has the two curl lines, and they are the supported route.

Two doors for one payload, and only one of them is lossless

Job.payload is the tagged envelope from the binding contract: one tag byte, then CBOR [args, kwargs]. EnqueueRequest gives you two ways to supply it, and choosing between them is a precision decision rather than a matter of taste.

raw carries the envelope you built, byte for byte, all the way to storage. It loses nothing, and it needs a CBOR library — it is what an SDK sends.

structured takes the arguments as ordinary JSON values and lets the server build that same envelope, so a shell script or a curl one-liner can enqueue work with no codec at all. Nothing downstream can tell the two apart: the row is an ordinary row, and for a call with no multi-key object the bytes are the ones an SDK would have produced. Object key order is the one exception, below.

grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" -d '{
  "task_name": "charge",
  "structured": {"args": [{"order_id": "ord-0001", "amount_cents": 1000}]},
  "options": {"queue": "payments"}
}' localhost:50051 flexiq.v1.ProducerService/Enqueue

What structured cannot carry, it refuses — it never rounds:

• Integers beyond ±9007199254740991 (2^53 - 1). A JSON number is a double, so 9007199254740993 would arrive as 9007199254740992. The request is rejected with INVALID_ARGUMENT instead of being quietly rewritten.
• Non-finite numbers. NaN and the infinities have no JSON form.
• Byte strings, CBOR tags, and anything else JSON has no syntax for. There is no way to express them here at all, which is a refusal by construction.

And one thing it normalises rather than refuses: object key order is not preserved. Keys arrive in a protobuf map, which is unordered by definition, so they are encoded sorted. Nothing decodes differently — a CBOR map is unordered too — but the bytes differ from what an SDK would have produced for the same call, so the SDKs' automatic auto: idempotency key, which is a hash over those bytes, will not match across the two doors. Set unique_key yourself if you need deduplication; it is the only mechanism this door offers in any case.

Send raw when any of that matters. That both arms exist is the reason none of it has to be guessed at.

Two things it does not promise, worth stating before a client assumes otherwise. unique_key is not an idempotency key without an expiry: it dedupes against the active job, so once the original completes or dead-letters the key is released and the same request enqueues a second job. A client's total retry deadline therefore has to be shorter than the job's own life. And no write is safely retried blind: UNAVAILABLE, DEADLINE_EXCEEDED and CANCELLED on an Enqueue may all mean the write landed and the connection dropped afterwards, and no field on the wire can tell you which. Clients that retry set a unique_key; EnqueueBatch needs one per item, and neither is retried automatically.

The same door, without gRPC

structured arguments only help a client with no CBOR library if such a client has a door to knock on, and gRPC from a shell script is not one. So the producer RPCs are also served over ordinary HTTP with JSON bodies, on the same listener — same port, same credential, same handlers, no proxy and no second process.

curl -X POST http://localhost:50051/v1/jobs \
  -H "authorization: Bearer $FLEXIQ_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "taskName": "charge",
    "structured": {"args": [{"orderId": "ord-0001", "amountCents": 1000}]},
    "options": {"queue": "payments"}
  }'

GET is served for exactly the read-only RPCs and nothing else, so the method is never a judgement call. Everything else is a POST, Enqueue included: a unique_key makes one request idempotent, not the method, and a level that holds for only some requests is one a proxy will apply to the others.

Bodies and query strings are proto3 JSON. Field names are the lowerCamelCase ones, and the .proto's own snake_case names are accepted too; a name that is neither is refused rather than ignored, because a silently dropped typo enqueues a job you did not describe. Timestamps are RFC 3339, durations are seconds with an s ("30s", "1.500s"), bytes fields are base64, and 64-bit integers — every counter in QueueStats — are strings, since a JSON number is a double.

Failures carry the same error model gRPC does, in the standard JSON shape. The reason is what to branch on; the message is for humans and may be reworded in any release:

{"error": {
  "code": 429,
  "status": "RESOURCE_EXHAUSTED",
  "message": "queue `payments` is full",
  "details": [{"@type": "type.googleapis.com/google.rpc.ErrorInfo",
               "reason": "QUEUE_FULL", "domain": "flexiq.byteveda.org",
               "metadata": {"queue": "payments", "pending": "1001", "cap": "1000"}}]
}}

The HTTP status is derived from status by one fixed mapping, which has one consequence worth knowing in advance: a request body over the 4 MiB cap — the same cap the gRPC door applies, so the two cannot disagree about what is too large — comes back 400 with "status": "OUT_OF_RANGE", not 413.

Three things this door does not do. It serves the producer service and, under /v1/admin/, the admin service — nothing else; a token's scope decides which of the two a call reaches. It does not expose the executor API — a worker surface has different credentials and different failure modes, and no reason to be reachable from a browser. And it transcodes no streams: a stream has no request/response HTTP mapping. The one exception is WatchJobs, served as Server-Sent Events on a path of its own — see Without gRPC.

A request without a credential comes back 401 with "reason": "UNAUTHENTICATED" rather than a gRPC trailer. Where this listener terminates TLS (TLS and mTLS, below), the JSON facade is served over the same certificate — ALPN offers both h2 and http/1.1, so curl https:// reaches it with nothing extra configured.

The OpenAPI document

The table above is also published as a machine-readable document, so a client generator, a Postman import or an API gateway can read this door without anyone writing the paths out by hand: contracts/openapi.json.

It is OpenAPI 3.1, generated from the same .proto files the gRPC door is generated from, and committed the way contracts/descriptor.binpb is. The paths in it are not a separate description that could go stale: each one is a google.api.http option on its RPC, and the server's route table is tested against those options, so a path the document advertises is a path the binary serves.

Two things it deliberately does not describe. Response schemas are open — a later release may add a field, and a client that refuses one is a client that breaks on the next upgrade — while request bodies are closed, because that is what the door does. And a oneof is described as arms that exclude each other, not as one that must be set: "exactly one of raw and structured" is a rule the service enforces and states in its own prose, alongside the others it enforces on values.

GET /v1/jobs:watch is not in it. OpenAPI describes an event stream only loosely, and the document is generated from google.api.http options, which a server stream does not carry. The path is documented under Without gRPC instead.

Health, and which probe it belongs in

grpc.health.v1 is answered out of storage, the same question /readiness answers on the dashboard: storage is checked, the worker registry is not. A queue with no workers is still a queue, but a door that cannot reach its database can only fail every call it is handed.

In Kubernetes that makes the gRPC health check the right readiness probe and the wrong liveness one — a database outage should take a replica out of rotation, not restart it — which is how the Helm chart wires a gRPC-only release. The grpc probe field needs Kubernetes 1.24; on an older cluster set grpc.healthProbe=false and readiness falls back to a socket connect.

Check and Watch are answered without a credential, by name — an unknown method under grpc.health.v1 is authenticated like anything else. A Kubernetes grpc: probe cannot send metadata, so gating health would mean either no readiness probe or a token written literally into the Deployment spec, and what it publishes is one bit to something that has already reached the port.

Scraping it

GET /metrics is served on the gRPC listener, as ordinary HTTP on the same port, in the Prometheus text format. It exists because a release with grpc.enabled=true and dashboard.enabled=false has no other scrape target — the dashboard's /metrics is on the dashboard's listener, and a gRPC-only pod does not run one.

curl -H "authorization: Bearer $FLEXIQ_TOKEN" http://localhost:50051/metrics

It carries the same storage gauges the dashboard publishes — flexiq_jobs, flexiq_workers, and flexiq_executors / flexiq_executor_slots when this process is the one executors attach to — plus what only this door knows:

door is grpc or http, because the JSON facade is the same service by another spelling and both are counted under the RPC they reached. A refused call is counted too — a client failing every call for want of a credential is precisely what you would scrape to discover.

Two things the numbers do not say. The duration is time to the response head, which is the whole call for every unary RPC but only the time to open a stream for Attach, Health/Watch and reflection — and those three report OK whatever they end with, because their terminal status rides trailers. Watch flexiq_executors for stream health, not flexiq_grpc_requests_total.

The scrape is credentialled like everything else on this listener, but by no particular scope: produce and execute both open it. A path that is not a method this build serves — including anything unrouted — is counted as other, so no caller can mint a series by inventing a URL.

What to do with those numbers is Scaling.

Tuning the listener

Four variables, all optional, all whole numbers of seconds except the last:

The keepalive is an HTTP/2 ping, not a TCP one: this listener binds its own socket, and the TCP-level setting would be accepted and silently ignored. Its timeout is derived from the interval rather than configured separately — two knobs that must stay in a ratio are one knob plus a way to get it wrong.

The request deadline does not bound a stream. It races the response future, and Attach returns its response as soon as it has accepted the connection, so an executor stream lives for its rotation period however short the deadline is. The same holds for Health/Watch and reflection.

FLEXIQ_GRPC_MAX_CONCURRENT_REQUESTS is per connection, so one noisy client cannot crowd out another's. Raising it does not buy throughput on its own: every RPC here is answered out of a bounded storage pool, and more calls in flight than the pool can serve is a queue in front of it that nothing reports on.

Watching jobs

WatchJobs is a server stream that follows jobs as they change state, so a client waits for a job without polling GetJob. It watches either a set of up to 100 job ids, or a queue.

An id watch starts with the current state of every id: a SNAPSHOT, or not_found_job_id for an id that does not exist or belongs to another namespace. The two look the same on purpose. Live transitions follow. When a job's item has terminal set, that job is done. When every job is done the stream ends with OK. So a watch on a job that already finished sends one item and closes:

grpcurl -plaintext -H "authorization: Bearer $FLEXIQ_TOKEN" \
  -d '{"jobIds": {"jobIds": ["'"$JOB_ID"'"]}}' \
  localhost:50051 flexiq.v1.ProducerService/WatchJobs

Decide that a job is finished by terminal, not by status. FAILED is not terminal while a RETRYING or DEAD is still to come. The stream carries no payload or result; read the result with GetJob once the job is terminal.

A queue watch sends live transitions of the queue's jobs and no snapshot. It opens with a checkpoint, an item that carries only the cursor the watch starts from. Every item after it carries a cursor too. Keep the last one, checkpoint included, so a stream lost before its first transition resumes with no gap.

Where transitions come from

Transitions reach a watch in two ways:

• Instantly, from this process. The server taps the same event hub its scheduler and doors already announce on, so a job this process runs or cancels reaches the stream with no database read.
• Within the reconcile interval, from anywhere else. A job run by another replica, or by a worker that reads the database directly, changes nothing this process can hear. So one task per process re-reads every watched id in a single batched query every FLEXIQ_GRPC_WATCH_RECONCILE seconds and sends what moved on. That is one query per tick however many clients are watching.

The re-read covers id watches only. A queue watch sees only this process's transitions, because a queue has no bounded set of ids to re-read. Run the producer door on the same process as the scheduler, or watch ids.

Resuming a stream

• An id watch: open it again. The snapshot is read from storage, so a job that finished while you were disconnected comes back terminal.
• A queue watch: send the last cursor back as resumeCursor. The server keeps the last FLEXIQ_GRPC_WATCH_BUFFER transitions in memory, per process. A cursor from another replica, from before a restart, or older than that window fails FAILED_PRECONDITION with reason WATCH_CURSOR_EXPIRED. The transitions in that gap are gone. Watch again without a cursor, and read the jobs' current state with ListJobs if you need it.

Limits

On Kubernetes, two of these depend on the deployment. With more than one replica, most transitions reach a watch through the reconcile, so reconcileSeconds is their latency. maxPerToken is shared by every client pod that uses the same token. The chart refuses at helm template time a value the server would refuse at boot.

A token over its cap is refused RESOURCE_EXHAUSTED with reason WATCH_LIMIT. The server never slows down for a client that stops reading. When the client falls behind the buffer, or a send waits longer than the stall bound, the stream ends RESOURCE_EXHAUSTED with reason WATCH_OVERFLOW, and the client resumes as above. On shutdown every stream ends UNAVAILABLE with reason SHUTTING_DOWN, so clients reconnect to another replica.

FLEXIQ_GRPC_REQUEST_TIMEOUT does not bound a watch, for the same reason it does not bound Attach.

Without gRPC

The same stream is served as Server-Sent Events on the JSON facade, for curl, a browser or an HTTP-only gateway. Same listener, same handler, same token scope (read), same limits:

curl -N "http://localhost:50051/v1/jobs:watch?jobIds=$JOB_ID" \
  -H "authorization: Bearer $FLEXIQ_TOKEN"

The query takes jobIds (repeat it for more than one id) or queue, and resumeCursor for a queue watch. Each WatchJobsResponse is one event. Its data is the response as proto3 JSON, and its id is the cursor when there is one:

data: {"cursor":"b8R-DXP126kAAAAAAAAAAQ"}
id: b8R-DXP126kAAAAAAAAAAQ

data: {"cursor":"b8R-DXP126kAAAAAAAAAAg","transition":{"jobId":"01a0f0e7-…","kind":"JOB_TRANSITION_KIND_CANCELLED","terminal":true,…}}
id: b8R-DXP126kAAAAAAAAAAg

An id watch hands out no cursor, so its events carry no id, and it ends:

event: end
data: {}

• A refusal before the stream opens — a bad query, WATCH_LIMIT, WATCH_CURSOR_EXPIRED, no credential — is the facade's ordinary JSON error with its HTTP status.
• A failure after it opens — WATCH_OVERFLOW, SHUTTING_DOWN — is a final event: error whose data is that same error body. Then the connection closes.
• A watch that finishes — every watched id terminal — ends with event: end. Then the connection closes.
• An idle stream sends a : keepalive comment every 15 seconds, so a proxy with an idle timeout keeps it open.

A browser's EventSource resumes a queue watch with no code of its own: it sends the last event id back as Last-Event-ID when it reconnects, and the server takes that over any resumeCursor still in the URL. It also reconnects after every close, so call close() on end and on error:

const events = new EventSource("/v1/jobs:watch?queue=orders");
events.onmessage = (e) => handle(JSON.parse(e.data));
events.addEventListener("end", () => events.close());
events.addEventListener("error", (e) => {
  // A server `error` event has data; a dropped connection does not, and is
  // left to EventSource to retry.
  if (e.data) events.close();
});

EventSource cannot set an authorization header, so a browser reaches this path through something that adds one — a same-origin gateway or proxy.

Behind an HTTP load balancer, a queue watch sees only the replica that took the connection. That is the same rule as over gRPC (Where transitions come from), but a load balancer picks the replica per connection, and a reconnect can land somewhere else — where the cursor fails WATCH_CURSOR_EXPIRED. Watch ids, or pin the connection to one replica.

The executor door

The same listener also serves flexiq.executor.v1.ExecutorService — a fourth Transport beside the attach protocol's TCP and Unix-socket transports, not a second dispatcher. An executor placed over this stream is placed by the exact rules RemoteDispatcher::attach already applies, shows up in the same registry as a socket-attached one, and is compared against it by the same registry-divergence check. Nothing about a job's dispatch, retries or dead-lettering changes because the executor that ran it dialled in over gRPC.

Six RPCs, in two groups. Attach and Heartbeat are the attach topology; Settle, ExtendLease, ReportProgress and WriteTaskLog are for a push target reporting on work that outlived the request it arrived on. A push deployment serves only the second group — nothing attaches there, so Attach and Heartbeat answer FAILED_PRECONDITION and say which deployment you reached. See Custom executors for what the four carry.

Attach is one bidirectional stream for the lifetime of a connection: AttachRequest/AttachResponse are each a oneof over the worker frame protocol's frames — hello in, hello_ack out, then jobs out and results in until either side ends it. Heartbeat is a separate unary RPC, deliberately off the dispatch stream — a busy stream and a dead peer would otherwise be indistinguishable — and carries the session token the Attach response returned in the flexiq-attach-session-bin metadata, so an executor cannot shrink a peer's advertised capacity by naming itself.

FLEXIQ_DSN=/var/lib/flexiq/app.db \
FLEXIQ_NAMESPACE=prod \
FLEXIQ_GRPC_LISTEN=127.0.0.1:50051 \
FLEXIQ_LISTEN=127.0.0.1:7777 \
flexiq-server

The execute scope opens this package, on the same token mechanism API tokens covers — mint one with flexiq-server token create --scope execute and send it as authorization: Bearer $FLEXIQ_TOKEN gRPC metadata, exactly like a producer call. Attach is gated by the same tower layer as every other RPC on this listener, and checks it before the handshake — FLEXIQ_ATTACH_TOKEN is a different credential for a different listener (the TCP/Unix attach port) and does nothing here.

The stream is bounded on purpose, FLEXIQ_GRPC_EXECUTOR_STREAM_MAX_AGE seconds — 30 minutes by default. A gRPC stream cannot be load-balanced once it has started, so a stream-per-executor that never ended would pin every executor to whichever replica it first reached. Before ending one, the scheduler stops matching new work to it and waits for what it already holds, so rotation never costs a job in flight. A clean stream end means reconnect; a shutdown frame means stop. Both are ordinary outcomes, not failures — a client that treats a rotation as an error will reconnect anyway, but a log line that says so is one less incident to chase.

What it does not have, on purpose: no HTTP binding — the JSON facade above serves flexiq.v1 and flexiq.admin.v1, never this package — and no admin or producer methods, so an execute-scoped credential can run work and nothing else.

Security

Same posture as the attach listener:

• TLS is optional, and terminated in process when it is set. FLEXIQ_GRPC_TLS_CERT and FLEXIQ_GRPC_TLS_KEY turn it on; FLEXIQ_GRPC_TLS_CLIENT_CA turns on mTLS beside it. Unset, the listener is exactly the plaintext door above — the right shape behind a sidecar proxy or a service mesh, where TLS is configured once for the cluster rather than per listener. TLS and mTLS, below, covers the rest.
• The credential is a bearer token, and it is one factor. It proves the peer holds a string; it does not encrypt the connection. What it is not is a shared secret: each token is named, scoped, expiring and revocable on its own, so a leak costs you one client's access rather than a rotation across every client at once. Only its SHA-256 digest is stored, so the database does not hold anything that can be replayed.
• There is no anonymous path and no fallback credential. A door with no token minted refuses every call, loopback included, because a listener with no credential provisioned is a misconfiguration rather than a permission grant. grpc.health.v1 Check and Watch are the one exception, by name — see Health, and which probe it belongs in. The KEDA external scaler still needs a token, but reads it from its scalerMetadata as well as the header, since KEDA cannot send one — see External scaler.
• The operator surface is scoped apart. Pausing queues, dead-letter operations, workers, periodic tasks and overrides need inspect or admin, which no producer or executor token carries, so a produce token that leaks costs you an enqueue path, not a tenant. Token minting needs its own tokens scope and can never widen a grant; settings and webhook secrets are on no gRPC surface at all.
• The schema is behind the credential, but not behind a scope. Any live token reads every package this build carries, its own scope included and the other one too — see Reflection, and what it hands out. What that costs is small while the .proto is published anyway; what it means is that revocation, not scoping, is the control over who can enumerate this door.

TLS and mTLS

Three variables, all optional, all unset by default — which leaves the listener exactly as plaintext as it has always been:

A cert without a key, a key without a cert, or a client CA bundle with neither, refuses to start — a boot with half a TLS configuration is one nobody meant. So does a path that is not a readable file (the error names the variable), a key that does not match its certificate, and a PEM file that parses but holds no certificates.

FLEXIQ_GRPC_LISTEN=0.0.0.0:50051 \
FLEXIQ_GRPC_TLS_CERT=/etc/flexiq/grpc-tls/tls.crt \
FLEXIQ_GRPC_TLS_KEY=/etc/flexiq/grpc-tls/tls.key \
flexiq-server

logs gRPC listener on https://0.0.0.0:50051 (TLS) at boot — (TLS, client certificates required) once FLEXIQ_GRPC_TLS_CLIENT_CA is also set. Dial it as https://:

curl --cacert ca.pem -H "authorization: Bearer $FLEXIQ_TOKEN" https://grpc.example:50051/v1/stats

The JSON facade above rides the same certificate, on the same port — ALPN offers both h2 and http/1.1, and curl negotiates whichever it speaks. There is no separate TLS configuration for the two.

mTLS is defence in depth beside the token, never instead of it. The certificate says the connection may exist; the token — still checked on every call the same way — says which principal it is. A handshake that fails against FLEXIQ_GRPC_TLS_CLIENT_CA is refused before any credential on the request is even read. The two grpc.health.v1 methods stay exempt from the token as before; under mTLS a probe still needs a trusted client certificate to connect at all.

Each connection handshakes in its own task, with a 10 s timeout and at most 64 pending handshakes at once, so one slow or silent peer cannot block every other connection from being accepted. A plaintext client dialling a TLS listener is not served — there is no fallback.

Reload without a restart. The server polls the files' content every 30 seconds — not their modification time, because a projected Kubernetes Secret is a symlink swap that does not reliably move one — and on a change builds a new TLS configuration and swaps it in whole. A new connection gets the new certificate; a connection already open keeps the session it negotiated. A reload that fails, a half-written file mid-copy for instance, keeps serving the previous configuration and logs an error rather than dropping the listener, and tries again on the next poll. FLEXIQ_GRPC_TLS_CLIENT_CA reloads the same way — it is the same mechanism the admission webhook already uses for its own certificate.

Behind a mesh, set nothing. A cluster that already terminates TLS for every listener through a sidecar or a mesh gains nothing from setting these three here — TLS configured once, there, covers this door for free, and the plaintext posture described through the rest of this page still applies to the loopback hop between the sidecar and this process.

With Helm

grpc.enabled needs namespace and nothing else — the credential is not a chart value, because tokens live in the database. A release that still sets grpc.token or grpc.existingSecret is refused at helm template time rather than starting a door credentialled by a value nothing reads.

namespace: prod
grpc:
  enabled: true

Mint the first token against the running release:

kubectl exec deploy/flexiq-flexiq-server -- \
  flexiq-server token create --name my-producer --scope produce

The door gets its own ClusterIP Service — never merged with the dashboard's, for the same reason attach is separate — with appProtocol: grpc so a mesh or an Ingress keyed off it speaks HTTP/2 rather than downgrading and breaking every stream.

grpc.tls.secretName terminates TLS on this Service from a kubernetes.io/tls Secret, with the readiness-probe fallback that needs — see Kubernetes for the values and what they mount.