Smoke OTEL - 26058026448

[github-actions]

Executive Summary

Overall: FAIL

The local OTEL mirror and Grafana backend both show spans for run 26058026448. Sentry received no OTLP export errors but returned no span data (inconclusive). Datadog failed with a 404 Not Found on OTLP export, and the Datadog MCP server reports zero available tools.

Run URL: https://github.com/github/gh-aw/actions/runs/26058026448

---

Status Matrix

Backend	Write Config Present	Write Export Succeeded	Read Config Present	Read Query Succeeded	Overall
Local OTLP	✅	✅	—	—	✅
Sentry	✅	🔶	✅	🔶	🔶
Grafana	✅	✅	✅	✅	✅
Datadog	✅	❌	❌	❌	❌

---

Evidence

Local OTLP

• OTEL env vars: OTEL_EXPORTER_OTLP_ENDPOINT ✅, GH_AW_OTLP_ENDPOINTS ✅
• OTEL_SERVICE_NAME: gh-aw.smoke-otel-backends
• COPILOT_OTEL_FILE_EXPORTER_PATH: not set (file exporter path absent)
• Local mirror /tmp/gh-aw/otel.jsonl: 1 line — span gh-aw.agent.setup, traceId edcd32d72bf7843ef0c4d52bad00fecc, run_id=26058026448 ✅
• Configured OTLP backend hosts: o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, otlp-intake.***REDACTED***
• OTLP export error count: 1 → Datadog only (see Datadog section)

Sentry

• MCP connection: ✅ (find_organizations returned github org at https://us.sentry.io)
• Project found: ✅ (gh-aw project in github org)
• Current-run spans (github.run_id:26058026448): no results
• Fallback recent gh-aw events (last 14d): no results
• No OTLP export errors attributed to o205451.ingest.us.sentry.io
• Note: list_events queries the error-events dataset only. Span-level trace search is not available via the configured Sentry MCP tools in this workflow. Absence of results may reflect the error-vs-span dataset mismatch rather than missing ingestion.

Grafana

• MCP connection: ✅ (list_datasources returned 12 datasources)
• Tracing datasource: grafanacloud-ghaw-traces (uid grafanacloud-traces, type tempo) ✅
• Current-run trace: ✅ — traceID edcd32d72bf7843ef0c4d52bad00fecc, rootSpan gh-aw.pre_activation.setup, service gh-aw.smoke-otel-backends, 5 matched spans, duration 43718 ms
  • Sample span: spanID=d376b3318e3cb6e7, github.run_id=26058026448, duration 1436 ms
• No OTLP export errors attributed to otlp-gateway-prod-eu-west-2.grafana.net

Datadog

• Configured OTLP host: otlp-intake.***REDACTED*** ✅ (present in GH_AW_OTLP_ENDPOINTS)
• OTLP export error: ❌ — {"host":"otlp-intake.***REDACTED***","status":404,"reason":"Not Found"} from /tmp/gh-aw/otlp-export-errors.jsonl
• Datadog MCP tools: ❌ — datadog --help reports 0 available tools; no span or trace queries possible
• Current-run spans: ❌ — not queryable (no MCP tools) and export failed
• Fallback recent gh-aw spans: ❌ — same reasons

---

Failure Analysis

Sentry — Write Export Succeeded: 🔶 (inconclusive)

• Failing step: Could not confirm OTLP span delivery to o205451.ingest.us.sentry.io.
• Evidence: No OTLP export errors on the write path. However, list_events is restricted to the Sentry errors dataset; span/trace-level data is not queryable through the available MCP tools.
• Most likely root cause: MCP tool set is limited to the error-events API surface. Spans ingested via OTLP land in the performance/tracing dataset, which requires a different Sentry API endpoint (e.g. /api/0/organizations/{org}/spans/ or the Explore → Traces UI).
• Path: read path / MCP tool coverage gap.
• Next debug step: Use list_events with dataset=spans or dataset=performance if those parameters are supported, or add a Sentry MCP tool that calls the spans API. Alternatively, visit https://github.sentry.io/performance/ and filter by github.run_id=26058026448.

Datadog — Write Export Succeeded: ❌

• Failing step: OTLP export to otlp-intake.***REDACTED*** returned HTTP 404.
• Evidence: /tmp/gh-aw/otlp-export-errors.jsonl → {"host":"otlp-intake.***REDACTED***","status":404,"reason":"Not Found"}.
• Most likely root cause: The Datadog OTLP intake URL is incorrect or the endpoint path is missing. otlp-intake.***REDACTED*** is the EU Datadog site intake, but the full path must be /api/v2/otlp/v1/traces (or /v1/traces depending on agent vs. direct ingest mode). A bare hostname without the correct path returns 404.
• Path: write path / configuration.
• Next debug step: Verify the full URL in GH_AW_OTLP_ENDPOINTS for the Datadog entry includes the correct path (e.g. (otlpintake/redacted)***REDACTED***/api/v2/otlp). Also confirm the DD_API_KEY secret is set and the API key has OTLP ingest permissions for the ***REDACTED*** site.

Datadog — Read Config Present: ❌

• Failing step: Datadog MCP server returned 0 tools.
• Evidence: datadog --help output: (tool list unavailable).
• Most likely root cause: The Datadog MCP server bridge is not fully configured or the tool manifest at /home/runner/work/_temp/gh-aw/mcp-cli/tools/datadog.json is empty or missing.
• Path: read path / MCP configuration.
• Next debug step: Check the contents of /home/runner/work/_temp/gh-aw/mcp-cli/tools/datadog.json and ensure the Datadog MCP server is provisioned with DD_API_KEY, DD_APPLICATION_KEY, and the correct site (***REDACTED***).

Generated by 🧪 Smoke OTEL for issue #33124 · ● 4.6M · ◷

• expires on May 18, 2026, 10:24 PM UTC