Add safe context watcher API by florentinl · Pull Request #6227 · PyO3/pyo3

florentinl · 2026-07-23T11:16:27Z

Adds PyContext and safe context-watcher bindings for GIL-enabled CPython 3.14:

typed context-switch events
RAII watcher registration
panic- and error-safe callback trampoline

use pyo3::{context::ContextEvent, prelude::*};

fn context_changed(_py: Python<'_>, event: ContextEvent<'_, '_>) -> PyResult<()> {
    if let ContextEvent::Switched(context) = event {
        println!("current context: {context:?}");
    }
    Ok(())
}

fn main() -> PyResult<()> {
    Python::initialize();

    Python::attach(|py| {
        let _watcher = pyo3::register_context_watcher!(py, context_changed)?;
        py.run(
            c"import contextvars\ncontextvars.Context().run(lambda: None)",
            None,
            None,
        )
    })
}

Tested with unit tests, doctests, and Clippy on Python 3.14.

chirizxc · 2026-07-24T19:54:34Z

+/// storage. State can still be shared through safe static synchronization primitives.
+///
+/// Panics and returned [`PyErr`][crate::PyErr] values are reported as unraisable exceptions and
+/// never unwind across the C boundary.


Should we mention this as well?

/// # Thread Safety /// /// On GIL-enabled builds, callbacks are invoked synchronously. On free-threaded /// builds (when this API is available), callbacks may be invoked concurrently. /// Ensure your callback is thread-safe if needed.

That's true, It should definitely be mentioned if we enable the API on free threaded builds.

The problem is that PyContext_AddWatcher and PyContext_ClearWatcher while available on free threaded are not callable concurrently and require external synchronization, which is not something we can guarantee from an extension.

I was thinking of making this API only available on GIL-enabled builds for this reason.

I think we can should requireSend + Sync + 'static on the provided closure.

The API in this PR accepts a function path, which is coerced to a function pointer. Function pointers carry no captured state and are already Send + Sync + 'static, so there is nothing additional to enforce.

CPython’s watcher API does not expose a user-data pointer where we could store a closure, which is why I did not try to implement a closure registration api.

@bschoenmaeckers can I ask you to give it another look ? Thanks

The problem is that PyContext_AddWatcher and PyContext_ClearWatcher while available on free threaded are not callable concurrently and require external synchronization, which is not something we can guarantee from an extension.

Where do you conclude this? We should make a CPython issue for this if not already present.

chirizxc · 2026-07-28T13:21:33Z

LGTM, but im not a maintainer

florentinl · 2026-07-28T13:35:00Z

Still, thank you for taking a look.

## Description Add a generic CPython 3.14+ publisher for the internal `python.context.switch` event. It registers the CPython `PyContext_AddWatcher` API from the native extension and dispatches the event after each context switch. The C callback catches Rust panics at the FFI boundary and preserves any exception already pending when CPython invokes it. This PR is stacked on #19604, which adds the public `DD_TRACE_OTEL_CTX_ENABLED` configuration and activation-listener setup. The setting defaults to `true`; setting it to `false` disables both the listener and this Python 3.14 watcher. It also complements #19336, which independently provides the asyncio, uvloop, AnyIO, and greenlet emitters for older Python versions. ## Testing - `scripts/lint checks` - `reno lint` - CPython 3.14 native context watcher: 4 passed - Linux CPython 3.14.5 OTel thread context: 6 passed ## Risks Low ## Additional Notes - The PR contains raw external C bindings to CPython. I am currently upstreaming them to PyO3; once that work is merged and released, we can rely on the PyO3 bindings and simplify this code: - PyO3/pyo3#6204 - PyO3/pyo3#6227 Co-authored-by: florentin.labelle <florentin.labelle@datadoghq.com>

Fix the newsfragment's overclaim that PyContext requires GIL-enabled CPython 3.14+ (it's available on all supported versions; only the watcher API needs 3.14+), clarify the SAFETY comments on ContextWatcherGuard around single-interpreter attachment, and drop a redundant doctest cfg attribute already covered by the module-level gate in lib.rs.

codspeed-hq · 2026-08-20T13:44:01Z

Merging this PR will improve performance by 10.31%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 140 untouched benchmarks
⏩ 6 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`test_empty_class_init_py`	27.1 µs	24.5 µs	+10.31%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing florentinl:florentin.labelle/add-safe-wrappers-for-context-watching-api (cf95e72) with main (6b67723)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

davidhewitt · 2026-08-22T18:14:36Z

I would quite like to avoid the macro for this API, it seems unfortunate. I understand why it's there, however. I'm thinking about what we can do as alternatives.

davidhewitt

I opened florentinl#1 which has a suggestion how I'd like this API to look.

Also placed a few other comments here.

davidhewitt · 2026-08-22T14:59:26Z

+/// storage. State can still be shared through safe static synchronization primitives.
+///
+/// Panics and returned [`PyErr`][crate::PyErr] values are reported as unraisable exceptions and
+/// never unwind across the C boundary.


The problem is that PyContext_AddWatcher and PyContext_ClearWatcher while available on free threaded are not callable concurrently and require external synchronization, which is not something we can guarantee from an extension.

Where do you conclude this? We should make a CPython issue for this if not already present.

davidhewitt · 2026-08-22T17:28:52Z

+/// This guard is bound to the [`Python`] attachment used to create it. It therefore cannot be sent
+/// to another thread, moved outside that attachment, or moved into [`Python::detach`].


I think this is potentially awkward, we might want to consider the same strategy as PyBuffer here which automatically attaches on drop.

That said, a bound type is more efficient so maybe we want to support the ability to bind & unbind this and have two guard types? 🤔

I have a feeling that for most usecases the lifetime will be a problem.

davidhewitt · 2026-08-22T18:03:34Z

+    ///
+    /// - The thread must be attached to Python.
+    /// - `object` must follow the contract for the supplied `event`.
+    pub unsafe extern "C" fn context_watcher<Callback: ContextWatcherCallbackDef>(


It would be desirable to reduce the amount of unsafe code in here by using PyO3's higher-level APIs where possible.

davidhewitt · 2026-08-22T18:04:37Z

+                // Although normal PyO3 APIs return errors as `PyResult`, `PyErr::restore` can be
+                // called directly. Do not allow an Ok return with an exception still set.
+                if crate::PyErr::occurred(py) {
+                    return Err(crate::PyErr::fetch(py));
+                }


This seems like a general problem that would be worth addressing in all trampolines rather than special casing here

chirizxc reviewed Jul 24, 2026

View reviewed changes

Comment thread src/context.rs

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from eb0dff3 to 66ad857 Compare July 28, 2026 07:47

florentinl marked this pull request as ready for review July 28, 2026 08:19

florentinl requested a review from chirizxc July 28, 2026 08:19

chirizxc reviewed Jul 28, 2026

View reviewed changes

Comment thread src/context.rs

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from 7259b06 to 69b141c Compare July 28, 2026 10:40

florentinl requested a review from chirizxc July 28, 2026 11:20

florentinl requested a review from bschoenmaeckers July 29, 2026 15:19

florentinl mentioned this pull request Aug 3, 2026

feat(tracing): publish Python context switches natively DataDog/dd-trace-py#19431

Merged

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch 2 times, most recently from b253b69 to 2041732 Compare August 13, 2026 13:27

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from 9826318 to d0a8de6 Compare August 20, 2026 12:56

florentinl added 9 commits August 20, 2026 15:26

Add safe context watcher API

ec1f01f

Add newsfragment

885824d

Refine context API availability

62c5b41

Guard macro-dependent context tests

73d0abc

Refine context watcher safety and tests

4d6a364

add forgotten guard test

c6cec32

Isolate forgotten context watcher test

4bd9379

Simplify context watcher event conversion

b0d3446

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from d0a8de6 to cf95e72 Compare August 20, 2026 13:32

davidhewitt reviewed Aug 22, 2026

View reviewed changes

		/// This guard is bound to the [`Python`] attachment used to create it. It therefore cannot be sent
		/// to another thread, moved outside that attachment, or moved into [`Python::detach`].

Uh oh!

Conversation

florentinl commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florentinl Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florentinl Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chirizxc commented Jul 28, 2026

Uh oh!

florentinl commented Jul 28, 2026

Uh oh!

codspeed-hq Bot commented Aug 20, 2026

Merging this PR will improve performance by 10.31%

Performance Changes

Footnotes

Uh oh!

davidhewitt commented Aug 22, 2026

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florentinl commented Jul 23, 2026 •

edited

Loading

florentinl Jul 27, 2026 •

edited

Loading

florentinl Jul 29, 2026 •

edited

Loading