streaming-safe scalar updates for mne.viz.Brain

[payamsash]

Describe the new feature or enhancement

making mne.viz.Brain usable for real-time streaming applications (neurofeedback, BCI, live source imaging), as suggested in discussion with @larsoner.

Describe your proposed implementation

mne.viz.Brain uses pyvistaqt.BackgroundPlotter under the hood, but the
public API is not designed for repeated scalar updates from a background
thread:

brain.add_data() is designed to be called once, calling it repeatedly (once per analysis window) is not thread-safe because it recreates actors and scalar bars also brain.set_time() does not accept new data only works when the data was preloaded.

MNE-RT uses pyvistaqt.BackgroundPlotter directly, keep a single PolyData mesh, and update mesh["activity"] scalars in-place plus call p.render(), which BackgroundPlotter handles safely from a
non-Qt thread. see link:

Describe possible alternatives

Add a lightweight streaming update API to mne.viz.Brain, for example:

# One-time setup
brain = mne.viz.Brain("fsaverage", hemi="split", surf="inflated")
handle = brain.add_data(stc_empty, colormap="hot", clim=(0, 1))

# Called from any thread
handle.set_scalars(new_vertex_values)   # thread-safe scalar swap + render

Or expose the existing PolyData mesh directly so users can update scalars themselves:

brain.lh_mesh["activity"] = new_lh_values
brain.render()   # non-blocking

Additional context

No response

[larsoner]

Currently add_data does not return anything. My first thought was to make it return something that would allow these updates, maybe the PolyData.

However, under the hood, we actually have an overlay compositing class _LayeredMesh that takes care of mapping scalar values from multiple sources to a single mesh (this is designed to handle annoying coincident topology issues that happen if you just create multiple PolyData meshes with different data values). Under the hood, this class does eventually update a PolyData attribute to make its updates:

mne-python/mne/viz/_3d_overlay.py

Line 172 in 40ffe5d

self._polydata[self._default_scalars_name] = self._current_colors

So I think what we can do is:

1. Clean up the compositing class _LayeredMesh a bit (be careful about private vs public attributes, rename to be public, put in public API)
2. Expose it via some public Brain attribute, maybe brain.layered_meshes but there is probably a better name

Then what you want can be accomplished with some variant of

brain = mne.viz.Brain("fsaverage", hemi="lh")
data = np.random.default_rng(0).normal(size=(163842,))
brain.add_data(data)

plus a public-API-ified version of this:

>>> brain._layered_meshes["lh"].update_overlay("data", np.abs(data))
>>> brain.plotter.render()

We'll need to be a bit careful here because there's stuff like how the data gets upsampled to the high-resolution mesh, we'd want to make sure the upsampler was accessible to the user, etc. But I think it's doable!

[payamsash]

yes, but then the user should know about smooth_mat and know the overlay is named data. what about making add_data() return handle object, which holds the mesh reference, overlay name and smooth_mat so user only calls handle.update(new_array) and it should be sth like:

if self._smooth_mat is not None:
      array = self._smooth_mat.dot(array)
self._mesh.update_overlay("data", scalars=array)
self._brain.plotter.render()

[larsoner]

yes, but then the user should know about smooth_mat and know the overlay is named data

Hmm... I think either way we can abstract the smooth_mat stuff actually by changing _LayeredMesh to be the thing that stores the upsampler / smoother.

For the "data" key part, I think we probably want to expose a key="data" default kwarg in add_data to help take care of #11503. That would make it clear that there is a key associated with the data, its default is "data", and then interacting with the _LayeredMesh wouldn't be so bad.

One reason I shy away from returning a new object for just this one purpose (streaming updates) is that the Brain object will already need to keep track of these data-storing objects and manipulate them. It seems cleaner to expose an API for the user to access them (allowing Brain to keep the references) rather than create new objects, at least to start. I can also foresee with #11503 that we might want to make some sort of data container class that stores everything about how the data gets mapped (colormap, clim, data, upsampler/smoother), and the Brain will need to keep its own references to that.

I suggest we start by seeing how far we can get with the _LayeredMesh manipulation approach, and eventually if we do create some data-storing class that Brain holds, maybe we can eventually return that to the user in add_data to make it easier for them to access. I think building it this way will help ensure that

[payamsash]

Ok, I understand now. Should I go ahead and open a PR with those steps (so move smooth_mat into LayeredMesh, add key="data" to add_data() and make layered_meshes public)?

[larsoner]

Sure that would be a good start!