Experiment(secret-manager): add tracing transport logic to google-cloud-secret-manager

[chalmerlowe]

Problem

Client libraries currently lack built-in support for OpenTelemetry tracing interceptors. We need a flexible, explicit mechanism to inject these interceptors into the transport pipeline without tying the core low-level helpers too tightly to specific observability features.

Note

This PR is NOT intended to be merged. It is a proof-of-concept intended to inform the design and implementation of changes that need to be made in the GAPIC Generator templates.

Solution

This PR implements explicit interceptor injection in the SecretManagerServiceClient and its gRPC transport.

• Updated SecretManagerServiceClient to resolve OpenTelemetry interceptors using google-api-core helpers and explicitly pass them to the transport.
• Added unit tests to verify that interceptors are correctly injected and applied.

Notes to Reviewers

• Async Deferred: As per the design plan, Async gRPC updates are deferred to a separate PR to keep this implementation small and clean.
• Stacked PRs: This work builds on the foundational helpers introduced in the google-api-core.
• Explicit Injection: This demonstrates the "active orchestrator" role of the generated client, delegating resolution to core helpers but explicitly controlling the pipeline.

Fixes #18139 (partially, this is Phase 2 of the larger effort)

[gemini-code-assist]

Code Review

This pull request introduces OpenTelemetry (OTel) tracing support by adding a helper module _otel_helpers.py to resolve and instantiate OTel gRPC interceptors, extending ClientOptions to accept a tracer_provider, and updating the Secret Manager gRPC transport to inject these interceptors. The review feedback highlights Python compatibility issues in _otel_helpers.py due to the use of the | union operator, which is unsupported in older Python versions, and suggests using typing.Union instead. Additionally, it recommends adding a defensive None check for client_options to avoid calling getattr on a None object.

[chalmerlowe]

Warning

Do not merge label added because this is a prototype implementation to discover the needed edits to the GAPIC generator templates. This is not intended to be merged.

[chalmerlowe]

Happy to fine tune the tests with fixtures, reusable mocks and/or functions, etc, but prefer to wait on going there until we get some buy-in on the overall approach in the code to avoid premature optimization.

[daniel-sanche]

Do you need to modify any of these kwargs? If not, it might be best to keep this as a callable, and let the Transport pass in the rest of the channel-init arguments:

transport_kwargs["channel"] = functools.partial(
   _observability.create_channel_with_otel,
   channel_factory=SecretManagerServiceGrpcTransport.create_channel
   client_options=self._client_options
)

But this would only work if create_channel_with_otel supported passthrough positional args, along with the kwargs

[daniel-sanche]

A problem with this is that if anyone needs to use a custom channel (like bigtable), they lose the otel interceptors.

I was hoping there was a way that we could decouple the interceptors, instead of baking them into the channel. But it sounds like otel makes that difficult?

[daniel-sanche]

Here are a couple ideas on how to get around this:

A (preferred). Maybe when creating a channel, the interceptors argument could accept functions structured as Callable[[Channel], Channel], and then it can apply both styles of interceptor separately

B. If we really need to treat this otel interceptor as a special case, we could add another argument to the transport for this. Something like enable_otel, and if set, it will add the extra interceptor to whatever channel it sends up with

I haven't looked too deeply into this though, so maybe there are flaws, or maybe when we start thinking about async, that would cause even more complication

[daniel-sanche]

I left a comment here, suggesting that we may be able to accept Callable[[Channel], Channel] here to support otel's interceptor