KEDA Autoscaling

Scale flexiq worker replicas on Kubernetes by queue depth, using the bundled scaler server.

KEDA (Kubernetes Event-driven Autoscaling) scales a worker Deployment up and down based on queue depth. flexiq ships a dedicated scaler server that KEDA's metrics-api trigger polls directly — no separate metrics pipeline required.

Requires a Kubernetes cluster with the KEDA operator installed.

Running flexiq-server with its gRPC listener? It serves a native KEDA scaler (type: external, or external-push to scale from zero on the enqueue), authenticated with an inspect token, with no scaler Deployment to run. See the server's scaling guide.

Running on bare metal, Docker, or systemd without Kubernetes? Use the bare-metal autoscaler instead — it spawns and drains flexiq worker subprocesses with the same HPA-style formula.

The scaler's /api/scaler endpoint is unauthenticated by design — keep it reachable only from inside the cluster (ClusterIP, never a public LoadBalancer or Ingress).

flexiq scaler and serve_scaler() bind to 127.0.0.1 by default — see Security: Scaler bind address. Widen to 0.0.0.0 only behind a trusted network boundary (e.g. the ClusterIP Service below).

serveScaler binds 0.0.0.0 by default, which is what lets a Kubernetes Service reach it out of the box — bind to 127.0.0.1 explicitly if you ever run it outside a container on a shared host.

ScalerOptions.defaults() binds 0.0.0.0 by default, which is what lets a Kubernetes Service reach it out of the box — pass an explicit host if you ever run it outside a container on a shared host.

There is no bind default to inherit: FLEXIQ_DASHBOARD is the address, and without it the role does not start at all. The Service above needs it wide:

bash

FLEXIQ_DASHBOARD=0.0.0.0:9091
FLEXIQ_DASHBOARD_AUTH=off
FLEXIQ_ALLOW_INSECURE=1        # required — see below

The server refuses this combination without the third line

A non-loopback bind with FLEXIQ_DASHBOARD_AUTH=off is a startup error, not a warning — the same port serves cancel, replay, purge and pause, so reaching it from off-host has to be deliberate. FLEXIQ_ALLOW_INSECURE=1 is the acknowledgement.

FLEXIQ_DASHBOARD_AUTH=session clears the guard but breaks the trigger: only /health, /readiness, /metrics and the login routes bypass the session gate, and /api/scaler is not among them. KEDA's metrics-api trigger sends a plain GET with no cookie, so it would get a 401.

So the safe shape is a ClusterIP Service and a NetworkPolicy that admits only the KEDA operator — the network is the boundary here, not the port.

Scaler server

bash

flexiq scaler --app myapp:queue --port 9091

Or programmatically:

python

from flexiq.scaler import serve_scaler

serve_scaler(queue, host="0.0.0.0", port=9091, target_queue_depth=10)

import { Queue, serveScaler } from "@byteveda/flexiq";

const queue = new Queue({ backend: "postgres", dsn: process.env.DATABASE_URL });
serveScaler(queue, { port: 9091, targetQueueDepth: 10 });

Or from the CLI, over a backend:

bash

flexiq --backend postgres --dsn "$DATABASE_URL" scaler --port 9091

java

try (FlexiQ flexiq = FlexiQ.builder().postgres(System.getenv("DATABASE_URL")).open();
        Scaler scaler = Scaler.start(flexiq, ScalerOptions.onPort(9090))) {
    Thread.currentThread().join(); // serve until the process is killed
}

(main needs throws InterruptedException for the join().)

There's no bundled CLI for the scaler — embed Scaler.start(...) in a small entrypoint alongside (or instead of) your worker process.

The flexiq crate serves no HTTP and ships no CLI, so there is nothing to start in-process. /api/scaler comes from the standalone flexiq-server binary, whose dashboard role serves it; it is configured entirely from the environment, with no flags:

bash

export FLEXIQ_DSN="$DATABASE_URL"
export FLEXIQ_DASHBOARD=127.0.0.1:9091   # the role that serves /api/scaler
flexiq-server

Point the trigger below at /api/scaler on whatever address you gave FLEXIQ_DASHBOARD. The target depth is a fixed 10 in this server — there is no knob for it — but ?queue= still narrows the metric per request.

Options

Flag	Default	Description
`--app`	— (required)	Python path to the `Queue` instance, e.g. `myapp.tasks:queue`
`--host`	`127.0.0.1`	Bind address
`--port`	`9091`	Bind port
`--target-queue-depth`	`10`	Scaling target hint returned to KEDA

serve_scaler(queue, host=, port=, target_queue_depth=) takes the same options as keyword arguments.

Option	Default	Description
`port`	`9091`	Listen port
`host`	`"0.0.0.0"`	Bind address
`targetQueueDepth`	`10`	Target depth per replica, returned to KEDA
`queue`	all queues	Restrict the metric to one queue; overridable per request via `?queue=`

The CLI mirrors these as --port, --host, --target-queue-depth, and --queue, plus the global --backend/--dsn connection flags.

`ScalerOptions` field	Default	Description
`port`	`9090`	Bind port. `0` picks an ephemeral port — `scaler.port()` reports it.
`host`	`"0.0.0.0"`	Bind address
`targetQueueDepth`	`10`	Target depth per replica; must be `> 0`
`queue`	`null` (all queues)	Restrict the metric to one queue

ScalerOptions.defaults() sets all four; ScalerOptions.onPort(port) keeps the rest at their defaults.

flexiq-server takes no flags at all — the environment is the whole configuration, which is what --help prints instead of options. For the scaler, these are the ones that matter:

Variable	Default	Description
`FLEXIQ_DSN`	— (required)	Storage connection string
`FLEXIQ_BACKEND`	inferred from the DSN	`sqlite`, `postgres` or `redis`
`FLEXIQ_DASHBOARD`	off	Address of the role serving `/api/scaler`. Unset means no dashboard role, and no scaler endpoint.
`FLEXIQ_DASHBOARD_AUTH`	`off`	`off` or `session`. `session` gates `/api/scaler`.
`FLEXIQ_ALLOW_INSECURE`	unset	`1` permits an off-host bind with auth off
`FLEXIQ_NAMESPACE`	all namespaces	Scopes the counts the scaler reports
`FLEXIQ_DASHBOARD_METRICS_TOKEN`	unset	Bearer token for `/metrics` and `/readiness` — not for `/api/scaler`

There is no target-queue-depth option. targetQueueDepth in the response is a constant 10 compiled into the server. That costs nothing in practice: KEDA scales on targetValue from the trigger, and the response field is informational — set the number you want in the ScaledObject and ignore the one in the body.

The queue filter has no variable either; it is the ?queue= query parameter on the request, which means one endpoint serves as many scaled objects as you have queues.

Endpoints

Endpoint	Returns
`GET /api/scaler[?queue=<name>]`	Current queue depth and scaling target
`GET /health`	Liveness check — always `{"status": "ok"}`

Also serves GET /metrics in Prometheus text format when prometheus-client is installed (501 otherwise).

The dashboard role serves two more that a Deployment wants, and the gating differs per endpoint:

Endpoint	Gate
`GET /api/scaler[?queue=]`	The session gate. Open with `FLEXIQ_DASHBOARD_AUTH=off`; a `401` with `session`.
`GET /health`	Always public. `{"status":"ok"}`, no storage touched — a liveness probe, not a readiness one.
`GET /readiness`	Checks storage and the worker registry. Gated like `/metrics` unless `FLEXIQ_DASHBOARD_PUBLIC_READINESS=1`.
`GET /metrics`	Prometheus text, always available — no optional dependency. `FLEXIQ_DASHBOARD_METRICS_TOKEN` as a bearer token if set; public under `auth=off` otherwise.

/readiness deliberately ignores the worker count: an empty registry still answers 200, because readiness asks whether this process can serve the API and the workers are a different Deployment. Probe /health for liveness and /readiness for traffic; neither tells you whether anything is draining the queue.

Response shape

KEDA only reads the field named by valueLocation in the trigger config (metricValue below) — everything else in the response is informational.

json

{
  "metricName": "flexiq_queue_depth",
  "metricValue": 42,
  "isActive": true,
  "liveWorkers": 3,
  "totalCapacity": 12,
  "targetQueueDepth": 10,
  "workerUtilization": 0.583,
  "perQueue": {
    "default": { "pending": 42, "running": 7 }
  }
}

Filtering with ?queue=emails narrows metricValue, isActive, and metricName (suffixed flexiq_queue_depth_emails) to that queue — perQueue still reports every queue. There's no separate queueName field; the filter shows up in metricName instead.

json

{ "metricValue": 42, "targetValue": 10, "queueName": "*" }

metricValue is the pending count for queueName ("*" for all queues, or the name passed via ?queue=).

json

{ "metricValue": 42, "targetValue": 10, "queueName": "all" }

metricValue is pending + running (outstanding work), not pending-only — account for that when tuning targetValue if you're porting a threshold from Python or Node.

json

{
  "metricName": "flexiq_queue_depth",
  "metricValue": 42,
  "isActive": true,
  "liveWorkers": 3,
  "totalCapacity": 12,
  "targetQueueDepth": 10,
  "workerUtilization": 0.583,
  "perQueue": {
    "default": { "pending": 42, "running": 7 }
  }
}

totalCapacity counts the slots attached executors advertise — this server runs no in-process worker pool, so it reports 0 when nothing is attached, and workerUtilization is then omitted from the body entirely. Filtering with ?queue=emails narrows metricValue, isActive, and metricName (suffixed flexiq_queue_depth_emails) to that queue; perQueue still reports every queue.

Kubernetes deployment

Deploy the scaler as its own Deployment, exposed as a ClusterIP Service:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flexiq-scaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flexiq-scaler
  template:
    metadata:
      labels:
        app: flexiq-scaler
    spec:
      containers:
        - name: scaler
          image: your-image:latest
          command: [ ] # see "Container command" below
          ports:
            - containerPort: 9091
---
apiVersion: v1
kind: Service
metadata:
  name: flexiq-scaler
spec:
  selector:
    app: flexiq-scaler
  ports:
    - port: 9091
      targetPort: 9091

Run the scaler as its own small Deployment + Service (not in the worker pod) so scaling to zero workers doesn't take the metric source down with it.

Container command:

yaml

command: ["flexiq", "scaler", "--app", "myapp:queue", "--port", "9091", "--host", "0.0.0.0"]

The explicit --host 0.0.0.0 matters — flexiq scaler binds to 127.0.0.1 by default, which the Service above can't reach.

yaml

command: ["flexiq", "--backend", "postgres", "--dsn", "$(DATABASE_URL)", "scaler", "--port", "9091"]
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: flexiq-db
        key: dsn

yaml

command: ["java", "-cp", "app.jar", "com.example.ScalerMain"]
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: flexiq-db
        key: dsn

with a small entrypoint packaged into app.jar — bound to 9091 here to match the Service above (Java's own default is 9090, see Options):

java

public final class ScalerMain {
    public static void main(String[] args) throws InterruptedException {
        try (FlexiQ flexiq = FlexiQ.builder().postgres(System.getenv("DATABASE_URL")).open();
                Scaler scaler = Scaler.start(flexiq, new ScalerOptions(9091, "0.0.0.0", 10, null))) {
            Thread.currentThread().join(); // serve until the container is killed
        }
    }
}

Run the published server image instead of your own — its entrypoint is the flexiq-server binary and it takes no arguments, so this pod needs image and env rather than a command:

yaml

image: ghcr.io/byteveda/flexiq-server:2.0.0
env:
  - name: FLEXIQ_DASHBOARD
    value: "0.0.0.0:9091"
  - name: FLEXIQ_ALLOW_INSECURE
    value: "1"
  - name: FLEXIQ_DSN
    valueFrom:
      secretKeyRef:
        name: flexiq-db
        key: dsn

FLEXIQ_ALLOW_INSECURE=1 is required here and is not a formality: the same port also serves cancel, replay, purge and pause, so binding it off-host without FLEXIQ_DASHBOARD_AUTH=session is refused at startup unless you say so explicitly. Keep the Service ClusterIP.

ScaledObject (HTTP trigger)

Scale a long-running worker Deployment based on pending job count. This config is identical regardless of which SDK built the scaler — it only talks to the HTTP endpoint above:

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: flexiq-worker
  namespace: default
spec:
  scaleTargetRef:
    name: flexiq-worker        # your worker Deployment name
  pollingInterval: 15           # seconds between KEDA polls
  cooldownPeriod: 60            # seconds before scaling to zero
  minReplicaCount: 0            # scale to zero when idle
  maxReplicaCount: 10
  triggers:
    - type: metrics-api
      metadata:
        url: "http://flexiq-scaler:9091/api/scaler"
        valueLocation: "metricValue"
        targetValue: "10"

Filter to a specific queue name:

yaml

        url: "http://flexiq-scaler:9091/api/scaler?queue=emails"

Adjust the port to 9090 (the default for ScalerOptions) unless you overrode it when starting the scaler.

The port in url: is whichever one you put in FLEXIQ_DASHBOARD — there is no default to fall back to, so the two have to be written to match:

yaml

        url: "http://flexiq-scaler:9091/api/scaler"   # FLEXIQ_DASHBOARD=0.0.0.0:9091

targetValue is read from this manifest and nothing else, so the 10 the response reports has no bearing on it. Tune the number here.

One thing to know about what the trigger is reading: metricValue is pending only, not pending + running. Work already claimed by a worker does not keep the replica count up, which is what you want for scaling to zero — and which means cooldownPeriod is the only thing stopping a scale-down while long jobs are still executing. Set it longer than your longest task.

ScaledJob (ephemeral batch workers)

For batch/ETL workloads, use ScaledJob to create short-lived Kubernetes Jobs — one pod per N pending tasks — instead of scaling a long-running Deployment:

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: flexiq-batch-worker
  namespace: default
spec:
  jobTargetRef:
    template:
      spec:
        containers:
          - name: flexiq-worker
            image: your-image:latest
            command: [ ] # see "Worker container command" below
        restartPolicy: Never
  pollingInterval: 15
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  maxReplicaCount: 20
  scalingStrategy:
    strategy: default         # or "accurate" for 1:1 job-to-pod mapping
  triggers:
    - type: metrics-api
      metadata:
        url: "http://flexiq-scaler:9091/api/scaler"
        valueLocation: "metricValue"
        targetValue: "5"      # one pod per 5 pending jobs

Worker container command:

yaml

command: ["flexiq", "worker", "--app", "myapp:queue"]

yaml

command: ["flexiq", "run", "./worker.js"]

yaml

command: ["java", "-jar", "worker.jar"]

Package a main() that opens a FlexiQ, registers handlers with flexiq.worker(), and calls worker.awaitShutdown() — see Deployment.

yaml

command: ["/usr/local/bin/my-worker"]

There is no worker CLI to invoke — a Rust worker is a binary you build. Its main opens a FlexiQ, registers tasks through queue.worker(), and holds the returned WorkerHandle. WorkerHandle has only worker_id() and shutdown(), so park the main thread yourself and call handle.shutdown() on SIGTERM to drain in-flight work before the pod goes away.

Scaling with Prometheus

If you already have Prometheus scraping your workers, skip the scaler server and point KEDA's prometheus trigger at it directly:

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: flexiq-worker-prometheus
  namespace: default
spec:
  scaleTargetRef:
    name: flexiq-worker
  pollingInterval: 15
  cooldownPeriod: 60
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: "http://prometheus:9090"
        metricName: flexiq_queue_depth
        query: sum(flexiq_queue_depth{queue="default"})
        threshold: "10"
    - type: prometheus
      metadata:
        serverAddress: "http://prometheus:9090"
        metricName: flexiq_worker_utilization
        query: flexiq_worker_utilization{queue="default"}
        threshold: "0.8"

Both metrics come from PrometheusStatsCollector — see the Prometheus integration.

If you already have Prometheus scraping your workers, skip the scaler server and point KEDA's prometheus trigger at the queue-depth gauge directly:

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: flexiq-worker-prometheus
  namespace: default
spec:
  scaleTargetRef:
    name: flexiq-worker
  pollingInterval: 15
  cooldownPeriod: 60
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: "http://prometheus:9090"
        metricName: flexiq_queue_depth
        query: sum(flexiq_queue_depth{queue="default"})
        threshold: "10"

This metric comes from PrometheusStatsCollector — see the Prometheus integration. There's no flexiq_worker_utilization gauge yet, so a second trigger on utilization isn't available — queue depth alone is the supported signal.

There's no Prometheus contrib yet — use the ScaledObject or ScaledJob triggers above, which talk to the scaler endpoint directly and need no separate metrics pipeline.

The same server already exposes /metrics in Prometheus text format, from the same storage reads — so if Prometheus is scraping it, the trigger can point there instead. Mind the gauge names, which are not the ones a metricName lifted from elsewhere would use:

yaml

    - type: prometheus
      metadata:
        serverAddress: "http://prometheus:9090"
        metricName: flexiq_jobs_pending
        query: sum(flexiq_jobs{queue="default",status="pending"})
        threshold: "10"

Gauge	Labels
`flexiq_jobs`	`queue`, `status` — one of `pending`, `running`, `completed`, `failed`, `dead`, `cancelled`
`flexiq_workers`	none — rows in the worker registry
`flexiq_executors`	none — only emitted when an executor is attached
`flexiq_executor_slots`	`state` — `total` or `free`; same condition

There is no flexiq_queue_depth and no utilization gauge; sum() over flexiq_jobs with status="pending" is the queue depth. The two executor gauges are absent entirely when nothing is attached, so a query over them needs an or vector(0) rather than assuming a zero.

A worker Deployment scaled this way still needs the worker task to be doing the work — /metrics reports what storage holds, not what this process is running.

Deploy templates

Ready-to-use YAML templates are included in the repository under deploy/keda/:

File	Purpose
`scaled-object.yaml`	`ScaledObject` using the HTTP scaler endpoint
`scaled-object-prometheus.yaml`	`ScaledObject` using Prometheus metrics
`scaled-job.yaml`	`ScaledJob` for ephemeral batch workers
`scaled-object-external.yaml`	`ScaledObject` using `flexiq-server`'s external scaler
`scaled-object-external-stream.yaml`	The same over `external-push`, scaling from zero on the enqueue

Multi-replica storage

When using SQLite, all worker replicas must share the same database volume. For multi-replica Kubernetes deployments, use the Postgres backend — workers connect over the network and there's no shared-file constraint.

When using SQLite, all worker replicas must share the same database volume. For multi-replica Kubernetes deployments, use a networked Postgres or Redis backend instead — workers connect over the network and there's no shared-file constraint.

With SQLite every replica has to share one volume, which a Deployment scaled past one replica cannot do. Use a networked Postgres or Redis backend — replicas connect over the network and there is no shared file.

Choosing that backend is a compile-time decision in the worker image, not a runtime one. The variant only exists if the feature was on when the binary was built:

toml

flexiq = { version = "2", features = ["postgres"] }

rust

use flexiq::{FlexiQ, PostgresStorage, StorageBackend};

// `FlexiQ::open` takes a SQLite path, so a networked backend goes through
// `from_storage`.
let url = std::env::var("DATABASE_URL")?;
let queue = FlexiQ::from_storage(StorageBackend::Postgres(PostgresStorage::new(&url)?))?;

Build the scaler's server with the matching feature too — it reads the same storage, so a mismatch is a DSN it cannot open rather than a wrong number.

See also mesh scheduling for within-process dispatch locality — it composes with KEDA rather than replacing it, since KEDA scales the number of worker processes and mesh only changes how quickly a claimed job reaches an idle one.