User-facing guide to the waterfall trace viewer added in architect task
60ba09e4 (backend pass in 2c89844, frontend pass in this change). For the
underlying trace-propagation contract, authorization matrix, and residual-risk
statement, see docs/TRACING_RUNBOOK.md.
Account-setting caution — CloudWatch Transaction Search. AgentCore
Observability (the managed GenAI views for AgentCore-hosted agents, e.g. the
intake runtime) requires enabling CloudWatch Transaction Search — and that
account-wide switch redirects X-Ray traces to CloudWatch Logs, after which
the X-Ray APIs this viewer is built on return nothing, for every Lambda in
every stack. Before touching that account setting (or planning any AgentCore
agent tracing), read "Account tracing settings — Transaction Search" at the
top of docs/TRACING_RUNBOOK.md: it states the constraint, the two coherent
choices (port the viewer to Transaction Search span queries vs. stay on the
X-Ray APIs and accept that AgentCore agents remain invisible to tracing),
and what the intake container is still missing before it emits telemetry at
all.
Trace backend — a supported adopter choice. trace-query-handler.ts
dispatches on the TRACE_BACKEND env var (xray | spans), configured via
backend/.env — see backend/.env.example. The default is xray (no
prerequisites, correct out of the box); spans queries CloudWatch Logs
Insights over aws/spans instead of the X-Ray APIs, and shapes results into
the same TraceEntry/TraceSpan types so this page's behavior below is
identical either way. Selecting spans requires CloudWatch Transaction
Search to already be enabled in the target account/region — see "Account
tracing settings — Transaction Search" in docs/TRACING_RUNBOOK.md for the
cutover procedure and its schema-verification step before choosing spans
in any real account.
A one-click waterfall view of the X-Ray spans behind a workflow execution or an agent conversation: nested span tree, per-span durations drawn as proportional bars, and fault/error/throttle badges — so you can see exactly where time went and where something failed, without leaving the app.
There is no separate "search for a trace" step for normal use — the viewer is reached by clicking View trace:
- From an execution's detail sheet (Agent Apps → an app → Executions tab → open an execution) — opens the trace for that execution.
- From a project's conversation view (the chat header, next to the document panel toggle) — opens the trace(s) linked to that conversation.
Both deep links land on /observability/trace/:kind/:id, which resolves the
execution or conversation to your organization and fetches the matching
X-Ray trace(s). Non-admin users can only open traces for executions and
conversations owned by their own organization (server-enforced 403
otherwise — see the authorization matrix in docs/TRACING_RUNBOOK.md).
- Each trace block is one X-Ray trace (there can be more than one per execution/conversation — one Lambda/worker hop can produce its own trace).
- Rows are spans, indented by nesting depth, with a collapsible chevron when a span has children.
- The horizontal bar's position and width are proportional to the span's start offset and duration within its trace — read left-to-right against the duration ruler at the top of each trace block.
- A
fault,error, orthrottlebadge appears on the trace header and/or the individual span, in that precedence order.
The viewer never fakes data. If a trace can't be shown, you'll see one of:
- Loading — fetching in progress.
- Indexing — X-Ray hasn't finished indexing the trace yet (typically resolves within ~90 seconds of the execution/conversation completing); the page auto-retries once and offers a manual retry.
- Empty — no trace was recorded (a sampling miss, or the flow's tracing hasn't been stitched yet).
- Not authorized — you don't have access to this execution/conversation's trace (cross-org, or you tried the admin-only raw trace-id route without admin rights).
- Unavailable — the trace API isn't configured for this deployment.
Troubleshooting a browser-only failure that curl doesn't reproduce:
if FRONTEND_ORIGIN was not supplied at deploy time, the costHttpApi's
CORS AllowOrigins falls back to a blocking placeholder origin
(https://frontend-origin-not-configured.invalid). The API itself is
healthy and answers a direct curl request normally, but every browser
call — the trace routes above and the cost dashboards alike — fails
preflight as a generic network error, not a typed state the UI can
distinguish (it will not present as "Unavailable" or "Not authorized";
the request never completes). Fix: redeploy TelemetryStack with
FRONTEND_ORIGIN set to the actual deployed frontend origin (after this
branch's telemetry-stack is deployed).
Beyond the waterfall trace, opening an execution's detail view
(ExecutionDetailSheet) surfaces per-node telemetry directly in the UI:
duration, queue wait (dispatch → worker-start), cold start, and retry
count for each node, sourced from execution-resolver.ts off the same
NodeDurationMs/NodeQueueWaitMs/NodeColdStart values the platform
health dashboard's widgets read (see "Platform-Health Dashboard + SLO
Alarms" below) — one telemetry pipeline feeding both the aggregate
dashboard and the per-execution inspection view.
Administrators additionally see an Observability entry in the sidebar with a Raw trace ID (admin) sub-entry, and a raw-trace-id input directly on the Observability page. A raw X-Ray trace id has no organization it can be checked against (unlike an execution or conversation id), so looking one up directly is restricted to admins — this is the same reasoning used for other account-wide reads in the governance UI.
A "Download replay package" button appears next to "View trace" on
execution inspection, and above the waterfall for execution/conversation
deep links (never for the raw trace-id kind — a replay package always needs
an ownership entry key). Unlike the raw trace-id lookup above, replay
package download is not admin-only — every member of the owning org can
download it, since ownership is resolved the same way as
by-execution/by-conversation. See docs/REPLAY_PACKAGE.md for the full
envelope contract, the sanitisation/fail-closed-gate guarantee, and the
eval-ingestion contract.
The trace query routes (/traces/by-execution/{id}, /traces/by-conversation/{id},
/traces/{traceId}) were added to the same API Gateway HTTP API that
already serves the cost dashboards (TelemetryStack's costHttpApi), behind
the same Cognito JWT authorizer and CORS configuration. The frontend
traceService therefore reads the same configuration key the cost
service already uses — serverService.getConfig()?.costApiUrl (surfaced to
the frontend as aws_cost_api_url) — rather than introducing a new config
key or a new CDK output. If aws_cost_api_url isn't configured for a given
deployment, both the cost dashboards and the trace viewer report themselves
as unavailable with zero network calls.
This means aws_cost_api_url is a slight misnomer now that it also serves
tracing — a documented tradeoff, with an optional future rename to
aws_telemetry_api_url tracked as tech debt, not addressed in this change.
Two one-click links between a governance decision and the runtime trace it
was made inside of (architect task 9b3f4f78). Full contract in
.kiro/specs/governance-ui/graphql-contract.md; this section is the
user-facing summary.
From a governance finding -> "what actually ran" (Governance -> Tracer): the finding header shows a View runtime trace action.
- If the finding carries a stamped
traceId(X-Ray trace id captured at the moment the finding was written) and you're an admin, it deep-links straight to/observability/trace/traceId/<id>— the same admin-only raw trace-id route described above. - If you're not an admin, the
traceIdis shown as a copyable string instead (trace data is account-wide, so the raw-id route can't be ownership-checked for anyone) with a note explaining why, plus a View execution trace fallback using the finding'sworkflowIdthrough the ownership-gated by-execution route. - If the finding predates this feature (the ledger is write-once — old
rows can never be retro-stamped), the primary button is disabled with a
tooltip saying so, and the same
workflowIdfallback is offered. - If the trace API isn't configured for this deployment, the link is hidden entirely rather than shown broken.
From a runtime trace -> "why was this allowed?" (Tracer ->
Governance): the waterfall page shows a Governance decisions (N) panel
below the trace, populated from the loaded trace's execution_id
annotation (falling back to correlation_id). N=0 renders as "No
governance decisions recorded for this execution" — true whether none were
written or the execution ran ungoverned. A missing annotation renders
"Execution id unavailable on this trace — cannot look up governance
decisions" instead of attempting a lookup. A query failure renders inline
without breaking the waterfall.
Known limitation (read before treating the fallback/panel as
authoritative): the fallback/panel above both key off workflowId /
execution_id, which for supervisor-dispatched governance findings today is
the supervisor's own orchestrationId — a separate identifier from the
StepRunner executionId used by runtime traces. They are not guaranteed to
be the same value. The traceId primary link does not have this problem
(it's the trace the finding was actually written inside), but the
workflowId-keyed fallback and the runtime→decision panel may legitimately
come back empty/not-found even for a governed execution. Treat a hit as
signal, not an absence of a hit as proof nothing was governed.
BOUNDED SCAN as of Pass 2 (design §4, decision f1cbd5ef) — runId pivot:
getDecisionTrace additionally surfaces finding.runId and a
linkedExecutionId pivoted from it: when the finding carries a
server-minted runId (Pass 1), the resolver runs a bounded, paginated Scan
of the executions table (capped at 1000 items examined) looking for a row
whose own runId matches AND whose orgId matches the finding's org (a
cross-org match is skipped, never surfaced) — unlike the
workflowId/orchestrationId fallback above, runId is minted once per
dispatch and does not suffer the orchestrationId/executionId identifier
mismatch. linkedExecutionId is null, never an error, when: the finding
predates runId stamping (write-once ledger, no backfill), no execution row's
runId matches within the scan cap, or the only matching row belongs to a
different org. A null here means "not found within the bounded scan," not
a guarantee that no matching execution exists — it does not distinguish
"confirmed absent" from "cap-truncated" in the API response itself (the
String return type has no field for that); a cap-truncation is instead
logged at warn in the resolver so operators can tell the two apart in
CloudWatch. This pivot becomes a true guarantee only once a dedicated GSI on
runId (deferred per design — "+1 GSI findings" is future work) replaces
the capped Scan with an exact Query. The workflowId fallback remains
available as a secondary signal regardless of linkedExecutionId's value.
Still best-effort (pre-runId findings/executions, or no GSI-backed global
lookup): the runId pivot above is a bounded, capped Scan (no runId GSI
exists on either the governance ledger or the executions table — see
docs/TRACING_RUNBOOK.md's "Deferred" note); it degrades to null on any
lookup failure rather than propagating an error. Two GSIs (governance
ledger runId index; cost-ledger runId index) are the deferred follow-up
that would make a global "given only a runId, find everything" query
possible without a Scan — not implemented in this pass.
Owned by TelemetryStack (decision ab73ae1b: dashboards + alarms are a
telemetry-stack responsibility, kept out of backend-stack.ts to preserve
its resource headroom). One dashboard; per-stack detail dashboards are
deliberately deferred — CloudWatch's own drill-down (click a widget →
Metrics console → filter by dimension) already covers that need at the
current traffic scale.
Six sections, top to bottom (on-call reads the health strip first):
| # | Section | Widgets |
|---|---|---|
| 0 | Health strip | 1h SingleValue: workflow failures, AppSync 5XX, max DLQ depth, cost-reconciler windows-reconciled, escalations |
| 1 | API health | AppSync errors/latency/requests; cost + gateway HttpApi 5xx/4xx; published-app APIs via SEARCH() |
| 2 | Workflow health | NodeDurationMs/NodeQueueWaitMs p50/p90 (SEARCH()), NodeFailure breakdown, NodeColdStart, worker/supervisor/fabricator Lambda Errors/Throttles/Duration |
| 3 | Cost & reconciliation | AbsEstimateDriftPct (with a 25% SLO annotation), WindowsReconciled, UnmatchedLedgerModels, LedgerTokens vs MetricTokens, TierBActive |
| 4 | Governance | OffFrontierEscalations + an AlarmStatusWidget referencing the existing (arbiter-stack-owned) escalation alarm |
| 5 | DLQ / error budget | DLQ depth + oldest-message age (SEARCH(), citadel-*dlq*), DynamoDB throttles/errors, and the platform-health AlarmStatusWidget (all 6 new alarms) |
All Citadel/Workflows metric-name/namespace/dimension strings are imported
from backend/src/utils/metrics-constants.ts — never retyped. That module
was extended for this story to add METRIC_NODE_DURATION_MS,
METRIC_NODE_FAILURE, and METRIC_NODE_QUEUE_WAIT_MS, mirroring the
values already pinned in the Python arbiter tier's
arbiter/common/metrics_constants.py (same namespace, both languages write
into it; the TS side just didn't have these three yet). Cross-dimension
widgets use SEARCH(); cross-dimension alarms use CloudWatch
Metrics Insights, because SEARCH() results are not alarmable.
No new SNS topic was created. BackendStack.alarmTopic (previously a local
const) was promoted to public readonly and threaded into
TelemetryStackProps, alongside appSyncApiId (for the concrete
GraphQLAPIId dimension on A3). TelemetryStack already depends on
BackendStack, so this added zero new stack edges.
| # | Alarm name | Metric / expression | Threshold | Period × Eval (datapoints) | treatMissingData |
Action |
|---|---|---|---|---|---|---|
| A1 | citadel-workflow-node-failure-${env} |
Metrics Insights SUM(NodeFailure) across WorkflowId/AgentId |
≥ 1 | 5m × 3 (1) | notBreaching |
Trace viewer + DLQ check |
| A2 | citadel-workflow-queue-wait-${env} |
Metrics Insights MAX(NodeQueueWaitMs) |
> 30000 ms | 5m × 3 (3) | notBreaching |
Worker concurrency/throttles |
| A3 | citadel-appsync-5xx-${env} |
AWS/AppSync 5XXError (dim GraphQLAPIId) |
≥ 5 | 5m × 1 (1) | notBreaching |
Resolver logs / X-Ray |
| A4 | citadel-dlq-not-empty-${env} (see DLQ-alarm note below) |
Math expression: sum of ApproximateNumberOfMessagesVisible (Maximum) across the explicit, test-drift-guarded list of every citadel-*dlq* queue (CloudWatch rejects an Insights LIKE filter at alarm create) |
≥ 1 | 5m × 1 (1) | notBreaching |
Per-consumer redrive procedure: runbooks/DLQ_REDRIVE.md |
| A5 | citadel-cost-reconciler-stalled-${env} |
Citadel/CostReconciler WindowsReconciled (dim Environment) |
< 1 | 1h × 3 (3) | breaching — absence IS the failure |
Check reconciler Lambda/schedule |
| A6 | citadel-cost-drift-high-${env} |
Citadel/CostReconciler AbsEstimateDriftPct (dim Environment) |
> 25% | 1h × 3 (3) | notBreaching |
Pricing catalog freshness |
DLQ-alarm note (A4): the shared async-DLQ change (branch
feat/eventbridge-shared-dlqs, unmerged as of 2026-08-31) adds seven
per-stack Lambda async DLQs (citadel-<stack>-async-dlq-${env}) and splits
DLQ coverage into two alarms to stay under CloudWatch's metric-math operand
limit: citadel-dlq-not-empty-${env} keeps the seven pre-existing
work/stream/notifier DLQs, and a new citadel-dlq-not-empty-shared-${env}
covers the seven shared async DLQs — both action the same alarm topic. The
same change replaces the hand-maintained mirror-list drift guard with a
structural scan (backend/lib/__tests__/dlq-coverage-structural.test.ts)
that discovers every DLQ from the synthesized templates and fails if any is
unalarmed. Triage and redrive for both alarms:
runbooks/DLQ_REDRIVE.md.
Every alarm carries an in-code comment marking its threshold as a
dev-calibrated starting point. The existing OffFrontierEscalations
alarm (arbiter-stack) and the backend-stack Lambda error/throttle alarms are
retained unchanged — they are surfaced on the dashboard, not re-created here.
- Run for 2 weeks against real (or representative staging) traffic.
- Pull p90/p99 of
NodeDurationMs,NodeQueueWaitMs, and the AppSync 5XX rate from CloudWatch for that window. - Set each threshold to
baseline × agreed-multiplier(multiplier decided per-alarm, not a single global constant — e.g. queue-wait tolerates less headroom than cost-drift). - Record the change as an ADR referencing the baseline data pulled in step 2.
No threshold shipped in this change is final; treat the numbers in the table above as placeholders that unblock dev/staging sign-off.
Unit level (backend/lib/__tests__/telemetry-stack.test.ts): Template
assertions per alarm — name, threshold, comparison operator, evaluation
periods, datapoints-to-alarm, treatMissingData, and that AlarmActions
resolves to the shared alarm topic — plus dashboard-body assertions for the
four cross-stack namespaces and all six section titles. These are the
primary, CI-enforced proof that each alarm is wired correctly.
Operational (manual, one-time-per-environment sign-off — supplements, does not replace, the unit assertions above):
Prerequisite — configure alarm delivery first. The alarms publish to
the citadel-alarms-${env} SNS topic, but no human subscriber exists unless
ALARM_DELIVERY (one of email | slack | none, in backend/.env) was set
when the environment was deployed — see backend/.env.example for the full
explanation, including the one-time Slack/Chatbot OAuth console step. Without
that configuration the alarms fire into a topic with no subscriber: the state
transition and the SNS publish still succeed, and nobody is notified. For
email mode, delivery also requires the recipient to click the confirmation
link SNS sends — a PendingConfirmation subscription looks present in the
console but delivers nothing. Verify both before proceeding:
TOPIC_ARN=$(aws sns list-topics --region "$REGION" \
--query "Topics[?ends_with(TopicArn, ':citadel-alarms-${ENV}')].TopicArn" --output text)
aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" --region "$REGION" \
--query 'Subscriptions[].[Endpoint,SubscriptionArn]' --output table
# At least one subscription must be listed, AND its SubscriptionArn must be a
# real ARN. "PendingConfirmation" means the email confirmation link was never
# clicked — that subscription is silently equivalent to no subscription.Datapoint alarms (A1–A4, A6) — publish a synthetic datapoint and confirm the alarm actually fires:
aws cloudwatch put-metric-data --namespace Citadel/Workflows \
--metric-name NodeFailure --unit Count --value 1 \
--dimensions WorkflowId=inject-test,AgentId=inject-test --region "$REGION"
# wait ~5-15 min for the eval window, then:
aws cloudwatch describe-alarms --alarm-names "citadel-workflow-node-failure-${ENV}" \
--query 'MetricAlarms[0].StateValue' --output text # expect: ALARMAbsence-based alarm (A5) cannot be forced by adding data — either wait 3h with the reconciler schedule disabled, or (faster, and this also proves the SNS wiring for every alarm including the datapoint ones) force the state directly:
aws cloudwatch set-alarm-state --alarm-name "citadel-cost-reconciler-stalled-${ENV}" \
--state-value ALARM --state-reason "fault-injection wiring test" --region "$REGION"
# confirm the configured destination (ALARM_DELIVERY email/Slack — see
# prerequisite above) received the notification; if nothing arrives, re-check
# the subscription exists and is not PendingConfirmation, then:
aws cloudwatch set-alarm-state --alarm-name "citadel-cost-reconciler-stalled-${ENV}" \
--state-value OK --state-reason "reset" --region "$REGION"put-metric-data proves the metric → threshold path for datapoint alarms;
set-alarm-state proves the alarm → SNS → subscriber path for every alarm,
including the absence-based one — but only if a delivery destination is
configured and confirmed (the prerequisite above). SNS accepts the publish
even with zero subscriptions, so an alarm can reach ALARM state "successfully"
while notifying no one. Both checks, run against a configured destination, are
required before signing off a new environment's alarm wiring.