Skip to content

[Feature] Support external OTLP collector for trace runtime - #2041

Open
matrix72c wants to merge 5 commits into
InternLM:mainfrom
matrix72c:feat/external-otel-collector
Open

[Feature] Support external OTLP collector for trace runtime#2041
matrix72c wants to merge 5 commits into
InternLM:mainfrom
matrix72c:feat/external-otel-collector

Conversation

@matrix72c

@matrix72c matrix72c commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Allow XTuner processes to export spans to an externally managed, network-reachable OTLP collector while preserving the existing driver-local collector and viewer defaults.

Motivation

The default trace path introduced in #1946 uses a collector on the driver's loopback address. In a multi-pod deployment, a remote Ray actor resolves 127.0.0.1 to its own pod, so it cannot reach the driver's collector and its spans are missing from the JSONL and Jaeger traces.

This PR allows the producers to use a routable collector Service instead:

driver / Ray actors in multiple pods
        | OTLP/gRPC
        v
external collector Service
        |-- file exporter --> shared traces.jsonl --> XTuner viewer
        `-- exporter ------> Jaeger or another backend

Configuration

TraceConfig(
    enabled=True,
    external_otlp_endpoint="http://otel-collector.namespace.svc:4317",
    external_trace_jsonl_path="/shared/traces/traces.jsonl",
    xtuner_viewer_enabled=True,
)
  • external_otlp_endpoint is propagated to Ray child processes so every process exports to the same reachable collector.
  • The external collector is deployed, configured, started, and stopped outside XTuner.
  • external_trace_jsonl_path only tells the driver-side viewer where to read the collector-owned JSONL file; it does not configure or start the collector and is not propagated to Ray child processes.
  • When the viewer is enabled, the path must refer to the same underlying file written by the external collector and must be visible from the driver, for example through a shared volume.
  • The viewer filters a shared JSONL file by service.name and XTuner run.id, so the file may contain spans from multiple runs.
  • If the external collector only exports to Jaeger or another backend, leave xtuner_viewer_enabled=False; no JSONL path is required.
  • Viewer JSONL loading is deferred until the first request so the external collector may create the file after the viewer process starts.

Compatibility

When external_otlp_endpoint is unset, local collector startup, per-run traces.jsonl output, viewer startup, and shutdown behavior remain unchanged. External collectors are never started or stopped by XTuner.

Tests

  • 8 focused tests covering local defaults, external collector lifecycle, endpoint propagation to Ray children, collector ownership of JSONL, deferred viewer loading, shared JSONL viewer configuration, and invalid configuration combinations.
  • ruff check.
  • compileall.
  • git diff --check.

@matrix72c
matrix72c force-pushed the feat/external-otel-collector branch from d6f6738 to 7681fde Compare August 24, 2026 14:09
@windreamer
windreamer requested a review from YanhuiDua August 24, 2026 23:17
@matrix72c matrix72c changed the title [Feat] Support external OTLP collector for trace runtime [Feature] Support external OTLP collector for trace runtime Sep 2, 2026
@matrix72c
matrix72c force-pushed the feat/external-otel-collector branch from 4c80db2 to fa952bf Compare September 7, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant