Push dispatch
The scheduler POSTs claimed jobs to your endpoint instead of waiting for an executor to attach: the environment variables, the egress guard, and the limits worth knowing before you deploy it.
The scheduler POSTs claimed jobs to your endpoint instead of waiting for an executor to attach: the environment variables, the egress guard, and the limits worth knowing before you deploy it.
FLEXIQ_PUSH_TARGET_URL adds a way to run work that has nothing in common with
attach or the gRPC executor door: instead of an executor dialling in and
holding a connection, the scheduler dials out, once per job, to an
operator-configured URL. It exists for platforms that start a process from an
inbound request — Cloud Run, Lambda, and similar — which have nothing that
could hold an attach connection open in the first place.
FLEXIQ_DSN=postgres://user:pass@host/db \
FLEXIQ_PUSH_TARGET_URL=https://executor.internal/hook \
FLEXIQ_PUSH_TARGET_CAPACITY=10 \
FLEXIQ_PUSH_TARGET_ALLOW=executor.internal \
flexiq-serverWriting the target that receives that POST is
its own contract —
every header, the response table, and a worked example. This page is the
operator half: turning it on, and what it costs you.
Coming from Cloud Tasks or EventBridge, which already POST to your handlers?
From Cloud Tasks and EventBridge maps queues,
OIDC, retries and deadlines onto this page's settings.
Push is additive, not a replacement. If you can run a long-lived worker
process at all — a container, a VM, anything that stays up — attach is still
the better answer: it keeps the execution lease local to the process holding
the work, makes cancel() actually stop something, and puts no load balancer
or ingress between the scheduler and the job. Reach for push only when nothing
in your deployment can hold a connection open — a platform that hands you a
request and tears the process down when it is answered.
FLEXIQ_PUSH_TARGET_URL and FLEXIQ_LISTEN cannot both be set. A
Worker holds exactly one dispatcher, and running both would race two
dispatchers for the same queues — a routing policy that has to be designed,
not defaulted. Choose one per process; the gRPC producer door is
unaffected either way, and can run alongside push on the same process — it is
only the gRPC executor door that push has nothing to attach to, for the
same reason the socket-based attach listener doesn't.
Everything is optional except the first three, which turn the section on and are then required together.
| Variable | Default | Meaning |
|---|---|---|
FLEXIQ_PUSH_TARGET_URL | Unset (off) | Where the scheduler POSTs a claimed job, e.g. https://executor.internal/run |
FLEXIQ_PUSH_TARGET_CAPACITY | Required | Jobs the target may run at once |
FLEXIQ_PUSH_TARGET_ALLOW | Required | Comma-separated hosts and CIDRs the target's resolved address must match |
FLEXIQ_PUSH_TARGET_TIMEOUT | 60 | Seconds one dispatch may take before it is abandoned as failed |
FLEXIQ_PUSH_TARGET_CONNECT_TIMEOUT | 5 | Seconds the connection may take to establish |
FLEXIQ_PUSH_TARGET_DRAIN | 30 | Seconds shutdown waits for in-flight dispatches, then again for each to settle once abandoned — a shutdown runs to at most twice this |
FLEXIQ_PUSH_TARGET_MAX_REQUEST_BYTES | 8388608 (8 MiB) | Ceiling on one job's request body |
FLEXIQ_PUSH_TARGET_MAX_RESPONSE_BYTES | 1048576 (1 MiB) | Ceiling on one response body read back |
FLEXIQ_PUSH_TARGET_AUTH | none | none | bearer | hmac | oidc | sigv4 |
FLEXIQ_PUSH_TARGET_SETTLE | off | off | grpc — whether a 202 Accepted hands the job off to be settled later |
FLEXIQ_PUSH_TARGETS | Unset | Comma-separated names, for more than one target |
The three second-valued variables and the two byte caps each take a minimum of
1. None of the five is a "no limit" switch. For a duration, 0 is a deadline
that has already passed, so it would fail every dispatch — or, for DRAIN,
abort every in-flight one at shutdown. For a cap, 0 is a ceiling nothing fits
under, so every request is refused as oversized before it is sent, or every
answer with a body is, and neither refusal is retryable. In each case the
failure would look like something other than the setting you changed, so zero
is refused at boot, naming the variable.
Push dispatch is behind its own cargo feature, http-target, so a binary that
never dials out does not have to compile in an HTTP client and a TLS stack.
The published flexiq-server image carries it, alongside postgres,
redis and grpc, so there is nothing to rebuild to turn it on. A binary you
built yourself without the feature refuses FLEXIQ_PUSH_TARGET_URL at boot,
naming the missing feature rather than silently never dialling out.
The four auth schemes each read a few more variables, only when
FLEXIQ_PUSH_TARGET_AUTH selects them:
| Variable | For | Meaning |
|---|---|---|
FLEXIQ_PUSH_TARGET_TOKEN | bearer (required) | The bearer secret |
FLEXIQ_PUSH_TARGET_HMAC_SECRET | hmac (required) | The HMAC-SHA256 signing secret |
FLEXIQ_PUSH_TARGET_HMAC_KEY_ID | hmac (optional) | Sent alongside the signature, so a receiver holding two secrets mid-rotation knows which one to check against |
FLEXIQ_PUSH_TARGET_OIDC_SOURCE | oidc (required) | google | azure-imds | azure-app-service |
FLEXIQ_PUSH_TARGET_OIDC_AUDIENCE | oidc (required) | The aud claim the receiver checks |
FLEXIQ_PUSH_TARGET_AZURE_CLIENT_ID | oidc + azure-imds (optional) | A user-assigned identity's client id |
FLEXIQ_PUSH_TARGET_AZURE_OBJECT_ID | oidc + azure-imds (optional) | A user-assigned identity's object id |
FLEXIQ_PUSH_TARGET_AZURE_MSI_RES_ID | oidc + azure-imds (optional) | A user-assigned identity's Azure resource id |
FLEXIQ_PUSH_TARGET_AWS_SOURCE | sigv4 (optional) | default-chain (default) | environment | container | imds |
FLEXIQ_PUSH_TARGET_AWS_REGION | sigv4 (optional) | Overrides the region inferred from the target URL |
FLEXIQ_PUSH_TARGET_AWS_SERVICE | sigv4 (optional) | Overrides the service inferred from the target URL |
At most one of the three Azure identity selectors may be set — Azure IMDS takes one identity, and more than one leaves the server to guess which. The full scheme write-up, including the HMAC string-to-sign, is in the contract.
CAPACITY and ALLOW are requiredNeither has a default, and neither can be inferred.
CAPACITY because an attached executor announces its own slots in
hello, and a gRPC executor stream does the same — the scheduler reads
concurrency off the wire. A push target sends no hello at all; it is a URL,
not a connection, and nothing about it carries a number. Leave this unset and
there is nothing for the scheduler to size dispatch concurrency from.
ALLOW because the URL you configure is itself operator input — exactly
the kind of value a copy-paste error, a stale environment variable, or a
compromised deployment pipeline could point somewhere you did not intend. An
allowlist that defaulted to "permit whatever host the URL already names"
would not be a guard at all, just the same value written twice under two
names. Naming the destination separately, in its own variable, is what turns
this into an actual check rather than an echo.
A process can dispatch to several targets — a service per queue, as a Cloud
Tasks deployment often has. Name them in FLEXIQ_PUSH_TARGETS, and give each
a URL and the queues it serves:
FLEXIQ_PUSH_TARGETS=orders,billing \
FLEXIQ_PUSH_ORDERS_URL=https://orders-abc123-uc.a.run.app/flexiq \
FLEXIQ_PUSH_ORDERS_QUEUES=orders,refunds \
FLEXIQ_PUSH_ORDERS_OIDC_AUDIENCE=https://orders-abc123-uc.a.run.app \
FLEXIQ_PUSH_BILLING_URL=https://billing-def456-uc.a.run.app/flexiq \
FLEXIQ_PUSH_BILLING_QUEUES=invoices \
FLEXIQ_PUSH_BILLING_CAPACITY=5 \
FLEXIQ_PUSH_BILLING_OIDC_AUDIENCE=https://billing-def456-uc.a.run.app \
FLEXIQ_PUSH_TARGET_CAPACITY=20 \
FLEXIQ_PUSH_TARGET_ALLOW=.run.app \
FLEXIQ_PUSH_TARGET_AUTH=oidc \
FLEXIQ_PUSH_TARGET_OIDC_SOURCE=google \
flexiq-serverEvery setting in the tables above can be given per target by replacing
TARGET with the target's name — FLEXIQ_PUSH_BILLING_CAPACITY for
FLEXIQ_PUSH_TARGET_CAPACITY. A target that does not set one reads the
FLEXIQ_PUSH_TARGET_* value, so what the targets share is written once. Each
target is validated as a single one is, and a refusal names the target.
FLEXIQ_PUSH_<NAME>_URL and FLEXIQ_PUSH_<NAME>_QUEUES are required, and
FLEXIQ_PUSH_TARGET_URL is refused beside FLEXIQ_PUSH_TARGETS.FLEXIQ_QUEUES and FLEXIQ_WORKERS are refused beside named targets:
each target's queues and capacity already answer both.target and targets
are reserved, since FLEXIQ_PUSH_TARGET_* is already the single-target
spelling.Each target runs its own scheduler in the process, claiming only its own
queues and sized from its own capacity, so a full target never holds up
another's jobs. Each registers as its own worker, and a shutdown drains them
all at once, so the grace period below is the same for one target or several.
A Settle, ExtendLease or report on the executor door reaches whichever
target made the dispatch; the one-replica rule for settle callbacks still
applies to the process as a whole.
The Helm chart configures a single target; its push values have no named
targets yet.
A push target is no different from an attached executor in how its jobs are
retried and throttled: the scheduler applies the task and queue overrides set
through the admin door, the dashboard or the
CLI — retryBackoff, rateLimit and maxConcurrent — read when it starts. A
queue's rateLimit is the dispatch rate of every target serving it;
FLEXIQ_PUSH_TARGET_CAPACITY is the concurrency. With no override, a task
retries on the default curve, a ceiling that starts at one second and doubles
to five minutes.
Every push target is dispatched through the same guard, regardless of auth scheme, and it runs whether or not you think you need it:
FLEXIQ_PUSH_TARGET_ALLOW is a comma-separated list of
exact hostnames, .suffix entries for a name and its subdomains, and CIDR
networks. Nothing outside that list is reachable — an allowlist naming
nothing fails to start rather than quietly denying everything, so an empty
value is a configuration error, not a valid "deny all".api.example.com permits whatever api.example.com
resolves to, so whoever controls that record controls the destination —
bounded by the loopback, link-local and cloud-metadata refusals below, which
a name can never vouch past. That bound is what makes a name sufficient on
its own for a target whose address you do not control, such as
https://svc.run.app. If
you want the addresses constrained as well, list a CIDR beside the name: an
address outside it is refused however the name resolves.169.254.169.254
hands out cloud IAM credentials to anything that asks. There is no
environment variable that reopens this; the one that would ask for it,
FLEXIQ_PUSH_TARGET_ALLOW_PRIVATE, is rejected outright at boot, naming
itself, rather than silently ignored.https only. Every dispatch carries the job payload, the lease and the
idempotency key, and a bearer token, an OIDC identity token or a SigV4
signature all put credential material in a header; HMAC signs a request
without encrypting it, so it is no exception. An http:// target is refused
at boot rather than sent to in the clear. There is no relaxation on this
path: the one the library offers is for a loopback host under
allow_loopback, and FLEXIQ_PUSH_TARGET_ALLOW_PRIVATE — the variable that
would ask for it — is rejected outright, so a flexiq-server deployment has
no cleartext option at all.3xx is a refusal, never a followed hop — a redirect is
exactly the kind of indirection an allowlist cannot see through.FLEXIQ_PUSH_TARGET_MAX_REQUEST_BYTES
bounds what the scheduler will build as a request body; a response over
FLEXIQ_PUSH_TARGET_MAX_RESPONSE_BYTES is refused outright rather than read
in truncated form, because a partial body is not what the target said and
must not become a job result.By default the whole task runs inside the single request the scheduler is
holding open, for as long as FLEXIQ_PUSH_TARGET_TIMEOUT and the job's own
timeout allow. That ceiling is yours to set, but the platform in front of your
target sets one of its own, and it wins if it is the shorter of the two:
For a task that needs longer than the shortest of these, set
FLEXIQ_PUSH_TARGET_SETTLE=grpc. The target then answers 202 Accepted, the
request ends, and it reports the outcome afterwards:
scheduler → POST target job, attempt, lease
target → 202 Accepted "I have it, I will call back"
…work runs past the platform's deadline; the connection is gone…
target → ExtendLease(job, lease) optional, repeatable
target → Settle(job, lease, outcome) exactly once
Settle and ExtendLease are RPCs on the executor door, so a target that
uses this path is a gRPC client for those two calls and the deployment must
also run FLEXIQ_GRPC_LISTEN. That pairing is checked at boot; a deployment
that asks for callbacks without a door refuses to start rather than accepting
a hand-off it has no inbound surface for.
202
instantly would have the scheduler claim the whole queue.execute-scoped API token
like any other client of it. Both are required, and neither substitutes for
the other.The scheduler waits until the job's own timeout_ms, measured from when the
attempt started — the same deadline a non-202 dispatch gets. A target that
needs longer calls ExtendLease before it passes, and keeps calling: the
ceiling is one hour per call, deliberately not a total, so that asking is
what tells the scheduler the target is still alive. A request over the ceiling
is clamped rather than refused, and the response carries the deadline that was
actually stored — plan against that, not against what you asked for.
Settle answers FAILED_PRECONDITION, and a target must not retry it,
when the attempt was already settled — by the deadline, by an operator
requeue, or by a retry that ran elsewhere. This is the normal failure mode of
the design rather than an edge case: a target that runs long will eventually
lose a race to the deadline, and this is how it finds out its work was thrown
away.
A job that is accepted and never settled fails at its deadline with an error
carrying the push.accepted_not_settled prefix and saying the target accepted
it and never came back — distinct from an ordinary timeout on purpose, so
"accepted, never settled" is something an operator can see rather than infer.
The gRPC listener's /metrics also carries
flexiq_push_awaiting_settle, the number of dispatches this replica has
accepted and not yet settled.
202 stays a refusal: the job dead-letters after one attempt with a reason
naming the variable to set. That is deliberate — a framework answering 202
by default is exactly the bug the mandatory x-flexiq-outcome header exists to
catch, and silently waiting on it would cost the job its whole deadline instead
of failing fast.
Durable steps remain attach-only either way: there is no step session here to
resume, which is why x-flexiq-outcome: slept is refused on the request path
and Settle has no slept arm.
cancel() stops the result, not the workUnder attach, cancelling a job sends a cancel frame and the executor stops.
Under push, there is no connection into the target's process to send one on.
The scheduler picks the cancel up from storage within about a second, then:
Cancelled. The target sees its connection close, and whatever it answers
afterwards is fenced out.202 — it settles the attempt Cancelled, and the target's
next ExtendLease, ReportProgress, WriteTaskLog or Settle is refused
with reason JOB_CANCELLED.Either way the job ends Cancelled and is not retried. But the target is only
stopped if it stops itself: its side effects up to that point still happen.
A charge that was about to go through goes through, and an email that was about
to send, sends.
For a long-running target, that makes the reporting RPCs its cancel channel.
Call ExtendLease or ReportProgress at an interval you can afford to overrun
by, and stop on JOB_CANCELLED — that interval is how promptly a cancel reaches
the work. A target that never reports is still correct: it runs to the end and
its Settle is refused.
JOB_CANCELLED comes from the replica that dispatched the job, and only while
it still remembers the ending — its most recent 1024. Past that, or on another
replica, the refusal is a misrouted one instead. Both are FAILED_PRECONDITION,
and both mean stop. The rules are in the
push contract.
Push dispatch does not honour HTTP_PROXY, HTTPS_PROXY or ALL_PROXY. A
proxy would resolve the target's hostname itself, on the proxy's own
network path — which moves the one lookup this guard exists to check outside
the guard entirely. There is no way to vet an address a proxy resolves for
you, so this is refused rather than silently insecure:
FLEXIQ_PUSH_TARGET_PROXY is rejected outright at boot if set, naming itself
in the error. A deployment behind a mandatory egress proxy cannot use push
dispatch yet.
Every push request carries x-flexiq-lease when the scheduler held one. It is
an attempt-scoped capability — the token that identifies which execution claim
a job was dispatched under, and whoever holds it holds the ability to settle
that job's current attempt. It is narrow: it expires the moment the attempt
settles or is superseded, and the scheduler refuses a retired one outright. But
it is still a credential-shaped value, and some platforms (an API gateway's
access log, a debugging reverse proxy, a request logger enabled by default)
log every inbound request header without being asked to. Treat
x-flexiq-lease the way you would a bearer token in that logging surface, and
configure header redaction if your platform in front of the target offers one.
The dashboard reports execution capacity from the attach dispatcher, and a push deployment builds none, so every capacity reading is missing or zero:
GET /api/scaler — the KEDA external-scaler payload — reports
totalCapacity: 0 and omits workerUtilization altogether./metrics omits flexiq_executors and flexiq_executor_slots, the two
gauges derived from the same number.GET /api/executors reports nothing attached, which is simply true: under
push nothing attaches.An autoscaler that scales on utilization will therefore never scale a push
deployment — the field it reads is not in the body, and the capacity it would
divide by is zero. Scale on queue depth instead: metricValue in the same
payload, and the flexiq_jobs gauges, are read from storage and are correct
under push. Size the target itself with FLEXIQ_PUSH_TARGET_CAPACITY, which
nothing in the dashboard reflects today.
Queue depth, job counts, the job list, dead letters and every operator action are unaffected. It is capacity alone that the dashboard cannot see.
push.enabled needs a URL, a capacity and an allowlist — nothing else has a
sensible default, so the chart fails at helm template time rather than
starting a target that will not construct:
push:
enabled: true
url: https://executor.internal/hook
capacity: 10
allow: executor.internalMind the grace period: push.drain is spent twice, so a shutdown runs to at
most 2 × push.drain — 60 seconds at the default. The chart accounts for this
on its own: with push.enabled, an unset terminationGracePeriodSeconds
computes 2 × push.drain plus a 10-second margin for the pod's own teardown
(SIGTERM delivery, the runtime unwinding once the second drain returns, the
container runtime's own stop overhead) — 70 seconds at the default — instead
of leaving Kubernetes' 30-second default in force, which would cut the second
wait short and leave the abandoned attempts to the reaper.
An explicit terminationGracePeriodSeconds still wins over the computed one,
0 included — a deliberate "kill immediately, no grace". Any other explicit
value is refused at helm template time unless it clears 2 × push.drain,
the same way the chart refuses other combinations it already knows will not
boot:
push:
enabled: true
drain: 30
terminationGracePeriodSeconds: 75Setting drain without touching terminationGracePeriodSeconds still works —
the computed default tracks whatever push.drain is, so lowering drain
lowers it too.
push.enabled and attach.enabled are refused together, the same way the
binary refuses FLEXIQ_PUSH_TARGET_URL and FLEXIQ_LISTEN together. Every
auth-scheme variable beyond push.auth itself — a bearer token, an HMAC
secret, an OIDC audience — goes through extraEnv, reading a secret-bearing
one from a Secret you create via valueFrom.secretKeyRef; this chart does not
give each of the four schemes its own typed value.
There is no Service and no containerPort for this role. A push target
dials out; unlike every other role in this chart, it opens no listener for a
Service to front. A release running push alone has no HTTP or TCP surface at
all for Kubernetes to probe, so it carries no liveness or readiness probe
either — the pod is considered healthy once it is running, and restarts only
on a process exit. Enabling dashboard.enabled alongside push gives Kubernetes
/health and /readiness to probe, which answer on whether storage is
reachable — useful, but not a signal about the target: nothing probes the push
URL, and the dashboard cannot see the target's capacity either (above).