Scaling

flexiq-server holds no state of its own. Every replica reads and writes the same database, and the database is the coordination point — so adding a replica is safe by construction, and it is also, on its own, frequently useless. This page is about which number tells you which of those two it is.

What each role is bounded by

The caveat on the gRPC door is the executor stream: a gRPC stream cannot be load-balanced once it has started, so an executor stays on whichever replica it first reached until the stream rotates. That is why it rotates — FLEXIQ_GRPC_EXECUTOR_STREAM_MAX_AGE, 30 minutes by default. Adding a replica does not rebalance the executors already attached; it takes up to one rotation period. See The gRPC door.

The webhook is the one role worth running at two replicas for availability rather than throughput: it sits in the pod-creation path, and its default failurePolicy: Ignore means an outage silently stops injecting sidecars rather than failing loudly.

Exactly one maintenance owner

Retention sweeps delete the same rows on every replica that runs them, so exactly one process should. FLEXIQ_MAINTENANCE is per process and therefore per Deployment — a three-replica Deployment cannot enable it on one of its three identical pods — so the maintenance owner is a separate single-replica release and everything else sets it off. Kubernetes has the two Helm releases that express it.

Turning it off does not turn off dead-worker reaping or in-flight recovery. Those run in every replica on purpose: a job whose worker died is rescued in about 30 seconds instead of waiting out its full timeout, and a deployment that tied that to the maintenance flag would lose it everywhere but on one pod.

The numbers to scale on

Both /metrics endpoints — the dashboard's and the gRPC door's — publish the same storage gauges. Four series answer four different questions:

Read them in that order, because they diagnose in that order:

• Pending is flat and free slots are non-zero. The backlog is not a capacity problem. Look at what the jobs are waiting on — a rate limit, a per-task concurrency cap, a lock, a paused queue — before scaling anything.
• Pending is rising and free slots are at zero. Add executors. This is the common case, and it is the only one where adding replicas of the worker helps.
• Free slots are non-zero, pending is rising, and dispatch is not keeping up. Now a second server replica is the answer — or a larger FLEXIQ_WORKERS, if the slots are attached to one replica that is not using them.
• flexiq_grpc_request_duration_seconds is climbing while queue depth is flat. The producer door is the bottleneck, not the queue. That is a storage pool or a CPU limit, and it scales with gRPC replicas rather than executors.

flexiq_executors and flexiq_executor_slots are absent, not zero, on a process no executor attaches to — "none attached" and "not the attach point" are different answers, and a dashboard reading zero could not tell them apart. A recording rule that sums them across replicas has to tolerate the absence.

Autoscaling with KEDA

The Helm chart installs flexiq-server; it does not install KEDA or create a scaling object. Three starting manifests live under deploy/keda/ — Kubernetes covers which to copy, and KEDA autoscaling covers the trigger fields.

Two things about them that matter to a server deployment specifically:

flexiq scaler is an SDK command, not part of flexiq-server. The binary has exactly one subcommand, token. So scaled-object.yaml and scaled-job.yaml, which point a metrics-api trigger at a flexiq scaler Service, require an SDK process to run that endpoint. A deployment whose only FlexiQ process is the server has no such endpoint and should use the external scaler below, or the Prometheus trigger.

The Prometheus manifest's queries name SDK metrics. flexiq_queue_depth and flexiq_worker_utilization come from an SDK's Prometheus collector; the server publishes flexiq_jobs and flexiq_executor_slots. Scraping flexiq-server and applying that manifest unedited yields no data and therefore no scaling, which KEDA reports as a healthy zero. Against a server scrape, the queries are:

triggers:
  # Backlog on one queue.
  - type: prometheus
    metadata:
      serverAddress: "http://prometheus:9090"
      metricName: flexiq_pending
      query: sum(flexiq_jobs{queue="default",status="pending"})
      threshold: "10"
  # Capacity: scale up as busy slots rise toward the total.
  - type: prometheus
    metadata:
      serverAddress: "http://prometheus:9090"
      metricName: flexiq_busy_slots
      query: sum(flexiq_executor_slots{state="total"}) - sum(flexiq_executor_slots{state="free"})
      threshold: "8"

Both triggers count something that rises under load, which is not a style choice. threshold is an AverageValue target, so the HPA asks for ceil(metric / threshold) replicas — a query that falls as the queue fills scales in at exactly the moment it should scale out. Free slots are that query; busy slots are its inverse. Set the second threshold to the slot count one executor advertises, so a fully-occupied replica is the point at which another is asked for.

Scaling on the executor Deployment, not the server one, is almost always what you want: the server is the thing holding the slots, and the slots are what the backlog is waiting for.

External scaler

When the gRPC listener is on, flexiq-server serves KEDA's external scaler (externalscaler.ExternalScaler) on it. There is no separate scaler Deployment and no SDK process: KEDA dials the same Service your producers do.

Pick external when the server is your only FlexiQ process, or when the metric endpoint should be authenticated rather than trusted by network position. Pick metrics-api when an SDK scaler is already running and the network is your boundary.

Mint a token and hand it to KEDA

The scaler checks the token itself, because KEDA cannot send a header. Mint one with the inspect scope for the namespace being scaled (Tokens covers where to run it):

flexiq-server token create --name keda --scope inspect

Put it in a Secret and expose it on the container of the Deployment KEDA scales, as FLEXIQ_SCALER_TOKEN:

env:
  - name: FLEXIQ_SCALER_TOKEN
    valueFrom:
      secretKeyRef:
        name: flexiq-scaler-token
        key: token

The trigger then names the variable, not the secret value:

triggers:
  - type: external
    metadata:
      scalerAddress: "flexiq-flexiq-server-grpc:50051"
      tokenFromEnv: FLEXIQ_SCALER_TOKEN
      queue: default
      targetQueueDepth: "10"

flexiq-flexiq-server-grpc is the gRPC Service of the chart installed as helm install flexiq ...: <release>-flexiq-server-grpc, shortened to <release>-grpc when the release name already contains flexiq-server (so a release called flexiq-server gives flexiq-server-grpc), or <fullnameOverride>-grpc when that is set. 50051 is the default grpc.port.

KEDA resolves tokenFromEnv from the scale target's environment and sends the value in the trigger metadata. A token key puts it in the manifest instead, which is only for a throwaway cluster. The namespace measured is the token's own, so there is no namespace key.

A narrowed grant works: inspect:queue=emails may scale a trigger whose queue is emails, and is refused for any other queue and for a trigger with no queue. A grant narrowed to tasks is always refused, since the depth counts every task.

Metadata

A missing token is UNAUTHENTICATED; a token without inspect is PERMISSION_DENIED; a malformed value is INVALID_ARGUMENT naming the key. If the call carries an authorization header it wins over the metadata, and a bad header is not rescued by a good token. Requests are capped at 64 KiB.

The two numbers answer different questions. The metric KEDA sizes replicas on is the pending count. Whether the target runs at all is pending + running, so a job already claimed keeps the target up until it finishes. An unknown queue reports 0 rather than an error, as /api/scaler does.

The two doors draw that line differently. gRPC IsActive is pending + running > activationQueueDepth; the isActive field of /api/scaler is pending > 0, so it goes false while the last claimed jobs are still running.

On Redis, a namespaced count reads the per-namespace job index. A job enqueued by a process older than that index is not in it until the job finishes, so during a rolling upgrade from such a version the scaler can under-count and let the target scale to zero. Finish the upgrade before relying on scale-to-zero; namespace quotas have the same window.

The metric is named flexiq-<queue> (or flexiq-all), lowercased with every character outside [a-z0-9] turned into -. So emails_eu and emails-eu both become flexiq-emails-eu. That is harmless: KEDA prefixes each trigger's metric with s<index>-, so two triggers in one ScaledObject still differ.

Push: scale from zero on the enqueue

external-push also opens a StreamIsActive stream. The server answers once on open and again whenever the queue flips between idle and active, so a first job wakes the target without waiting for the polling interval:

triggers:
  - type: external-push
    metadata:
      scalerAddress: "flexiq-flexiq-server-grpc:50051"
      tokenFromEnv: FLEXIQ_SCALER_TOKEN
      queue: default

Replica counts still come from the polled metric. An enqueue through the server wakes an idle stream at once; everything else, including a drain, is seen on the periodic re-read, FLEXIQ_GRPC_WATCH_RECONCILE (5 seconds by default).

Streams count against FLEXIQ_GRPC_WATCH_MAX_PER_TOKEN (16) per token, separately from watches. KEDA opens about one per trigger.

The token is checked when a stream opens, not while it runs. Revoking it does not end a stream already open, as with WatchJobs; the stream keeps answering until it closes, and KEDA's reconnect is then refused. Polled calls fail on the next poll.

TLS

With FLEXIQ_GRPC_TLS_* set, give KEDA the CA through a TriggerAuthentication whose secretTargetRef sets the caCert parameter, and point the trigger's authenticationRef at it. KEDA documents enableTLS: "true" for the external trigger. external-push reads the same keys: KEDA builds both scalers from one metadata parser and one connection builder, which turns TLS on when enableTLS is true, a caCert parameter is set, or a client certificate (tlsClientCert with tlsClientKey) is loaded. For mTLS, add tlsClientCert and tlsClientKey to the same TriggerAuthentication. TLS and mTLS covers the server side.

Complete manifests are deploy/keda/scaled-object-external.yaml and deploy/keda/scaled-object-external-stream.yaml in the repository.

The storage ceiling

Every replica shares one database, so past a certain point the database is what you are scaling, not the server.

• SQLite does not replicate. One writer, one file, no network. The Helm chart refuses a SQLite DSN outright, because replicas cannot share a pod-local file safely. Its throughput limits are covered under Deployment.
• Postgres is the default answer for a replicated deployment. Watch the connection count: every replica opens a pool, and the pools multiply by the replica count long before the query load becomes interesting.
• Redis is single-threaded for command execution, so the ceiling is commands per second rather than connections.

Backends covers choosing between them and what each one costs.