Audit trail
Which token enqueued a job, who paused a queue from the dashboard, who minted or revoked a token, and what a revoked token did while it was valid — a record for every thing a write touched, kept in the database.
Which token enqueued a job, who paused a queue from the dashboard, who minted or revoked a token, and what a revoked token did while it was valid — a record for every thing a write touched, kept in the database.
A token says who may call the gRPC door. The audit trail says who did: every
write a token makes — an enqueue, a cancel, a queue pause, a dead-letter replay
— leaves records in the database naming the token, what it did, to what, when,
and how the call ended. So does every change made from
the dashboard, named by the user who made it, and
every token minted or revoked from the dashboard, with flexiq-server token
or through the admin door's token RPCs. It answers the
questions a credential alone cannot:
A record never carries the token or its digest. It names the token by its
public id — the ID column of flexiq-server token list — and by the name it
was minted under.
Every call that needs a write scope is recorded: produce on the producer
door, admin on the admin door, and CreateToken and
RevokeToken under tokens. The same calls over the
JSON facade are recorded the
same way, under the same operation name.
A record is per target, not per call — count records as calls and a batch will mislead you. A call leaves one record for each thing it acted on, or a single untargeted record when it was refused before naming any.
| Field | Is |
|---|---|
time | When the call was answered |
principalKind | Who tokenId names: token (the gRPC door), user (a dashboard session), cli (flexiq-server token) or anonymous (a dashboard with auth off) |
tokenId | The token's public id, or the dashboard user's username. Empty for cli and anonymous |
principal | The token's name when the call was made — a label, since two tokens may share one — or the username |
operation | The RPC, e.g. flexiq.v1.ProducerService/Enqueue, or the dashboard route, e.g. dashboard POST /api/queues/{queue}/pause |
targetKind, target | What it acted on: job, workflow_run, queue, dead_letter, worker, periodic, task, namespace or token (the admin door's token RPCs) — and, from the dashboard, token, webhook, webhook_delivery, setting, topic, subscription or middleware — and its id or name |
outcome | The google.rpc.Code the call ended with: OK, PERMISSION_DENIED, NOT_FOUND, … CANCELLED means the call never answered — a deadline or a client that hung up — so its write may or may not have landed |
EnqueueBatch
writes one per job, and a dead-letter replay writes one for the entry and one
for the job it became, so "who enqueued this job" has an answer for a replayed
job too.
Refusals are recorded. A token that is valid but lacks the scope, or whose
grants do not
reach the queue, leaves a record with outcome PERMISSION_DENIED — and no target
when the call was refused before naming one. That is usually the record an
investigation is looking for.
What is not recorded, and why:
GetJob, ListJobs, WatchJobs and
every inspect method are dominated by polling, so by default the trail
answers "what did it change", not "what did it see". See
Recording reads.The flexiq-server dashboard records every change made through it: any
POST, PUT, PATCH or DELETE its API answers — pausing a queue, replaying
or purging dead letters, editing overrides, middleware, settings or webhooks,
minting or revoking a gRPC token. It is a rule, not a list, so a route added
later is recorded too.
principalKind user with their username.
With FLEXIQ_DASHBOARD_AUTH=open nobody signs in, so the change is recorded
as anonymous: what changed is still on the record, who is not.PERMISSION_DENIED against that session's
user. A request with no session at all — a login, an expired cookie — is not
recorded.dashboard DELETE /api/grpc-tokens/{id} — and each value in the path is a
target: the token id there, queue:emails for a pause. Minting a token names
the new token's id, though the route carries none.google.rpc.Code: OK,
INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND, …FLEXIQ_NAMESPACE, or default when it serves
none.Reads — opening a page, listing jobs — are not recorded from the dashboard.
flexiq-server token create and token revoke record too, as principalKind
cli, operation cli token create or cli token revoke, targeting the token.
The command has no identity of its own to record — whoever can run it can open
the database — so the record says that it happened and to which token. A
revoke naming no token is recorded NOT_FOUND. If the trail cannot take the
record, the command still succeeds and says so on stderr.
A dashboard an SDK serves in-process records the same changes, into the same
table, with the same row: who, operation, targets, outcome and namespace are
decided in the shared core, so a filter for dashboard POST /api/queues/{queue}/pause finds a pause whichever dashboard took it. An SDK
dashboard with the legacy shared-token gate names no one, and records as
anonymous. A route only flexiq-server serves — minting a gRPC token — is
recorded only there.
The write is off the request path, as on the server: an action is never
refused or slowed because the audit table is, and a record the table cannot
take goes to the log under flexiq::audit. The SDK dashboard also prunes its
namespace hourly; see retention.
With fq, on a token that carries inspect:
fq audit list # newest first; --all follows every page
fq audit list --job <JOB_ID> # who enqueued it, who cancelled it
fq audit list --token-id <TOKEN_ID> # everything one token did
fq audit list --principal-kind user # every dashboard change
fq audit list --target queue:emails # or --kind queue for every queue
fq audit list --since 2026-09-01T00:00:00Z --until 2026-09-02T00:00:00Ztime by principal operation target outcome
------------------------ ---------------------- ----------- ----------------------------------------- ---------------------------------------- -----------------
2026-09-29T09:52:10.118Z user:alice alice dashboard POST /api/queues/{queue}/pause queue:emails OK
2026-09-29T09:51:33.524Z token:10d08aa8296d8f38 ops flexiq.admin.v1.AdminService/PauseQueue queue:emails OK
2026-09-29T09:51:33.515Z token:665dbdce673017bc dashboard flexiq.v1.ProducerService/Enqueue - PERMISSION_DENIED
2026-09-29T09:51:33.474Z token:8a7fbf03e21cfa60 billing-api flexiq.v1.ProducerService/Enqueue job:01a0ec93-a261-75b4-806b-d6fb5c06c449 OKby reads kind:id, like the target; a cli or anonymous record shows the
kind alone. The filters combine: every one set must match. --token-id takes
the public id, not a credential — fq still reads its own token from
FLEXIQ_TOKEN — or a username, alongside --principal-kind user.
The same read is AdminService.ListAuditRecords over gRPC, and
GET /v1/admin/auditRecords over JSON, with tokenId, principalKind,
targetKind, target, since, until, pageSize and pageToken as query
parameters:
curl -H "authorization: Bearer $FLEXIQ_TOKEN" \
"http://localhost:50051/v1/admin/auditRecords?tokenId=8a7fbf03e21cfa60"It needs inspect. A produce token is refused SCOPE_DENIED: the trail names
every token in the namespace, and a producer has no business listing them.
The flexiq-server dashboard lists the trail under Configuration → Audit
trail, newest first, with the same filters: principal kind, token id or
username, target kind, target, and a since/until range. A row's principal
narrows the list to that principal; its target links to the page that shows it —
the job, the workflow run, the topic, the webhook's deliveries, the queue's jobs,
or the workers and gRPC tokens lists. On gRPC tokens, a token's What this
token did opens the trail filtered to it, revoked tokens included.
The page is for admins. With FLEXIQ_DASHBOARD_AUTH=session, a viewer is
refused 403 — the trail names every user and token and what each did, the same
reason a viewer cannot list the tokens — and the nav hides the entry. With auth
off there are no roles, so anyone who reaches the dashboard can read it. The
dashboard reads its own namespace, as it records into it.
Its JSON is GET /api/audit-records on the dashboard port, with token_id,
principal_kind, target_kind, target, since and until (Unix
milliseconds), limit (default 50, at most 500) and after (the previous
page's next_cursor). A dashboard an SDK serves has no such route and shows no
page.
The submitting token is also stamped on the job, as enqueuedBy — the same
public id as a record's tokenId. GetJob, ListJobs, WatchJobs,
fq jobs get --json and the dashboard show it without a second query, and it stays on the job after
the job finishes, dies or is archived, so a job outlives its audit records'
retention without losing who sent it.
fq jobs get <JOB_ID> --json | jq -r .enqueuedBy # 8a7fbf03e21cfa60
fq audit list --token-id 8a7fbf03e21cfa60 # and what else that token didThe server sets it from the token it authenticated; no request field, option or metadata key can. Which jobs carry it:
| Job | enqueuedBy |
|---|---|
Enqueue, EnqueueBatch, and every node of a SubmitWorkflow | The calling token |
An admin TriggerPeriodic | The calling token |
| A dead-letter replay | The original job's — the replay rebuilds the same job; the trail records who replayed it |
| A duplicate or debounced enqueue | Whatever the job it collapsed into already holds |
| An SDK enqueue in-process, a periodic firing, an HTTP trigger, a fan-out child, a dashboard job replay | Unset — no token was involved. A job replay's replayed_from metadata leads back to the original, which keeps its own |
Jobs enqueued before the server recorded it read back unset.
For a revoked third-party credential, "what did it see" is sometimes the
question. Set FLEXIQ_GRPC_AUDIT_READS=on (Helm: grpc.auditReads) and every
call that needs read or inspect is recorded too, as are GetToken and
ListTokens under tokens, in the same table, with the same fields. on, true, yes or 1 turns it on; off, false, no or
0 leaves it off, in any case. Any other value stops the server at startup
instead of quietly leaving reads unrecorded.
A read names what it asked for, not each row it got back:
| Read | Target |
|---|---|
GetJob | The job id |
ListJobs, ListDeadLetters, ListPeriodicTasks, ListOverrides | The queue and task filters, when set |
QueueStats, ListQueues, GetThroughput | The queue filter, when set |
GetWorkflowRun | The workflow run id |
WatchJobs | Each watched job id, or the queue |
GetDeadLetter | The dead-letter id |
GetPeriodicTask | The periodic task name |
GetToken | The token's public id |
ListWorkers, ListTokens, GetNamespaceQuota, ListAuditRecords | None — one untargeted record |
The target is named before the lookup, so a miss still says which id was tried
— its outcome is NOT_FOUND. A WatchJobs stream, over gRPC or
server-sent events, is recorded once, when
it opens; the transitions it then delivers are not.
Repeats are folded. A read that repeats one already recorded — same token,
same RPC, same target, same outcome — within
FLEXIQ_GRPC_AUDIT_READS_WINDOW seconds (default 60; Helm:
grpc.auditReadsWindow) leaves no second record. A client polling one job
every two seconds leaves one record a minute, not thirty. 0 records every
read. Writes are never folded.
The window is kept in each server's memory, so behind several replicas the same read can be recorded once per replica — the trail can over-count, never skip a read one replica served. If more distinct reads arrive inside one window than the server can track, the extra ones are recorded every time rather than dropped.
Reads share the trail's overflow path, and
by default its retention; they can be kept for less. Read volume
is far higher than write volume: watch
flexiq_audit_records_total{outcome="overflow"} after turning this on.
Records are kept for 90 days, then deleted. Set
FLEXIQ_AUDIT_RETENTION_DAYS (Helm: audit.retentionDays) to keep them longer
or shorter; it must be at least 1. There is no "forever": a table nobody
prunes grows without bound. The variable's earlier name,
FLEXIQ_GRPC_AUDIT_RETENTION_DAYS (Helm: grpc.auditRetentionDays, read when
audit.retentionDays is unset), is still read; setting both to different
values stops the server at startup. The chart sets the window whenever the
dashboard or the gRPC door runs.
Read records can be kept for less: set
FLEXIQ_AUDIT_READS_RETENTION_DAYS (Helm: audit.readsRetentionDays; default:
the write window). It must be at least 1 and no longer than the write window
— a read must not outlive the writes it relates to, so a longer value stops the
server at startup. Every record says which it is: a write, or a read.
Each gRPC listener prunes its own namespace once an hour, and so does a
dashboard running without one. An SDK's dashboard prunes its namespace too,
with its own window: its audit_retention_days / auditRetentionDays option,
else the same FLEXIQ_AUDIT_RETENTION_DAYS, else 90 days. The delete is
idempotent, so replicas and dashboards sharing a database prune without
electing a leader — the second one finds nothing left. Where several prune one
namespace with different windows, the shortest wins. The window is not part of
the per-queue retention an SDK configures for its jobs.
The door never waits on the audit write, and never fails a call because of it. Records go into an in-memory buffer and one writer appends them in batches; an enqueue does not get slower because the audit table is, and it is not refused because the audit table is down.
A record the buffer cannot take, or whose write fails, is written to the log
instead — one JSON line under the log target flexiq::audit — and counted on
/metrics:
| Series | Labels | Is |
|---|---|---|
flexiq_audit_records_total | outcome | Records written to the table, or sent to the log on overflow / write_failed |
Alert on anything but written rising. A record is degraded to the log stream,
never dropped silently — but the log stream is only an audit trail if something
keeps it. On a clean shutdown the listener and the dashboard each wait up to
ten seconds for their writer to append what is already buffered before the
process exits; what is still unwritten after that is logged as lost. The
dashboard's /metrics carries the same series.