feat: Add catmaid spatially indexed skeletons (read-only) by afonsobspinto · Pull Request #119 · MetaCell/neuroglancer

afonsobspinto · 2026-01-26T10:24:44Z

Closes:

CATMAID Datasource and Spatially Indexed Skeletons

This document explains how the CATMAID datasource is wired up and how spatially
indexed skeletons are loaded, serialized, and rendered.

CATMAID datasource implementation (`src/datasource/catmaid/`)

Entry points and registration

register_default.ts registers the provider under the catmaid scheme. The
URL format is expected to be catmaid://<base_url>/<project_id>.

Credentials flow

credentials_provider.ts defines CatmaidCredentialsProvider, which fetches
an anonymous API token from <serverUrl>/accounts/anonymous-api-token.
This provider is used via fetchOkWithCredentials in the API client to retry
on 401/403 and request a refreshed token when needed.

API client (`api.ts`)

CatmaidClient wraps all CATMAID endpoints used by the data source.

Key operations:

listSkeletons() -> list of skeleton IDs
getSkeleton(skeletonId) -> full skeleton as nodes from
skeletons/<id>/compact-detail
fetchNodes(boundingBox, lod) -> spatially indexed nodes from
node/list using msgpack; supports multi-LOD data in the response
getDimensions() and getResolution() -> stack metadata, including voxel
size and translation
getGridCellSizes() -> grid cache configurations (from custom metadata object); defaults to the hard-coded
grid size if none is present

Provider wiring (`frontend.ts`)

CatmaidDataSourceProvider.get() resolves the source, builds the coordinate
space, and creates the sub-sources:

Parse URL
- Strips catmaid://.
- Splits base URL and project id.
- Ensures http prefix.
Create CatmaidClient and fetch metadata in parallel:
- dimensions, resolution, gridCellSizes, skeletonIds.
Coordinate space setup
- Converts resolution (nm per voxel) to meters.
- Uses stack translation and dimension to establish voxel-space bounds.
Create skeleton sources
- Spatially indexed: CatmaidMultiscaleSpatiallyIndexedSkeletonSource
  provides a multiscale slice-view chunk source for spatially indexed
  skeleton data.
- Complete: CatmaidSkeletonSource loads full skeletons by ID.
Create segment properties for skeleton labels
- Skeleton IDs are converted to BigInt and stored in a
  SegmentPropertyMap.
Subsources returned
- skeletons-chunked (spatially indexed)
- skeletons (full skeletons)
- properties (segment labels)
- bounds (data bounds)

Multiscale grid mapping

CatmaidMultiscaleSpatiallyIndexedSkeletonSource.getSources():

Sorts CATMAID cache grid cell sizes from smallest to largest.
For each grid size, builds a ChunkLayout and SliceViewChunkSpecification.
Builds a chunkToMultiscaleTransform that scales each grid size relative to
the smallest grid size.
Sets useChunkSizeForScaleSelection = true so the backend can bias scale
selection based on chunk sizes (rather than voxel size alone).

Spatially indexed skeletons (`src/skeleton/`)

Data model and chunk serialization

The spatially indexed path uses slice-view chunking but renders as a mesh-like
edge list.

Key data types:

SpatiallyIndexedSkeletonChunk (frontend and backend)
- vertexPositions (Float32 vec3)
- vertexAttributes (packed bytes)
- indices (Uint32 pairs, each edge is two vertex indices)
- missingConnections (parent edges that cross chunk boundaries)
- nodeMap (nodeId -> vertexIndex mapping)
SkeletonChunkData serialization packs vertex positions and attributes into a
single Uint8Array with vertexAttributeOffsets for GPU upload.

Serialization (skeleton_chunk_serialization.ts):

Positions are always the first attribute.
vertexAttributeOffsets describes byte offsets of each attribute.
missingConnections and nodeMap are serialized alongside.

Chunk download (CATMAID backend)

CatmaidSpatiallyIndexedSkeletonSourceBackend.download():

Computes a chunk-aligned bounding box from chunkGridPosition and
chunkDataSize.
Calls CatmaidClient.fetchNodes(bbox, currentLod).
Builds:
- vertexPositions: XYZ for each node.
- vertexAttributes: segment ID only (stored as float).
- indices: edge pairs from parent-child relations.
- nodeMap: maps CATMAID node IDs to vertex indices.

Backend chunk scheduling and LOD

SpatiallyIndexedSkeletonRenderLayerBackend:

Maintains skeletonLod as a shared watchable value.
Debounces LOD changes (300 ms) to avoid invalidating the source cache on every
slider move.
Uses a multi-scale selection algorithm in recomputeChunkPriorities():
- Derives pixel size from slice view parameters.
- Selects which scales to load based on:
  - renderScaleTarget,
  - effectiveVoxelSize,
  - and optionally chunkLayout.size if
    useChunkSizeForScaleSelection is enabled.
- Requests chunks using forEachVisibleVolumetricChunk and applies a scale
  priority offset (SCALE_PRIORITY_MULTIPLIER).

Rendering pipeline overview

Rendering is split into two paths:

Regular skeleton layer (SkeletonLayer)
- Draws full skeletons (one chunk per skeleton) using the same shader stack.
- Each chunk renders all visible segments using a single draw call per chunk.
Spatially indexed skeleton layer (SpatiallyIndexedSkeletonLayer)
- Draws skeletons that are chunked spatially across the volume.
- Supports multi-scale sources and renders each chunk per segment to ensure
  consistent color usage and visibility filtering.

Spatially indexed draw algorithm (per chunk, per segment)

SpatiallyIndexedSkeletonLayer.draw() performs per-segment rendering to ensure
correct color assignment and visibility filtering:

Build a global node map across all loaded chunks to resolve inter-chunk
parent references.
For each chunk:
- Read the segment ID attribute from the packed attribute buffer.
- Group edge indices by segment ID.
- For each segment in the chunk:
  - Check visibility (getVisibleSegments).
  - Apply objectAlpha or hiddenObjectAlpha based on visibility.
  - Build a compacted vertex list for just that segment.
  - Remap indices to the compact list.
  - Upload compacted attribute textures.
  - Draw edges and (optionally) nodes with uColor set to the segment color.
Per-segment caching
- The filtered vertex textures are cached per chunk per segment by
  generation, so repeated frames can reuse GPU textures.
Inter-chunk edges
- missingConnections are tracked but not rendered yet.
- There is a TODO to build combined buffers to render those edges.

Improvements:

1) Per-draw global node map construction

Detail: SpatiallyIndexedSkeletonLayer.draw() rebuilds a global node map
every frame across all loaded chunks.

Risk: This scales with total loaded vertices and can become expensive.

2) Per-segment compaction and buffer swapping

Detail: For each segment in each chunk, the code:

Filters vertices,
Remaps indices,
Builds compact textures,
Creates a new GL buffer,
Draws,
Then restores the original buffers.

Risk: This is CPU-heavy and generates many short-lived GL buffers, which can
cause stalls and GC pressure for large skeleton sets.

3) Inter-chunk edges are not rendered

Detail: missingConnections are tracked, but the renderer does not draw
edges that cross chunk boundaries.

Risk: Skeletons that cross chunk boundaries appear disconnected.

Copilot

Pull request overview

This PR introduces support for spatially indexed skeleton rendering in Neuroglancer, specifically adding a CATMAID data source integration for loading skeletons organized in spatial chunks.

Changes:

Adds spatially indexed skeleton sources that load skeleton data in spatial chunks (similar to volume rendering) with LOD support
Implements CATMAID data source provider for fetching skeleton data from CATMAID servers
Adds UI controls for hidden object opacity and skeleton LOD level

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/skeleton/skeleton_chunk_serialization.ts	New utility module for serializing skeleton chunk data for transfer between backend and frontend
src/skeleton/gpu_upload_utils.ts	New utility for uploading vertex attribute data to GPU as 1D textures
src/skeleton/frontend.ts	Extended with new spatially indexed skeleton layer classes and rendering logic
src/skeleton/base.ts	Added RPC IDs for spatially indexed skeleton layers
src/skeleton/backend.ts	Added backend implementation for spatially indexed skeleton chunks with LOD support
src/segmentation_display_state/frontend.ts	Added hiddenObjectAlpha field for controlling opacity of non-visible segments
src/layer/segmentation/layer_controls.ts	Added UI controls for hidden opacity and LOD
src/layer/segmentation/json_keys.ts	Added JSON keys for new configuration options
src/layer/segmentation/index.ts	Integrated spatially indexed skeleton layers into segmentation user layer
src/datasource/index.ts	Extended DataSubsource type to include new skeleton source types
src/datasource/graphene/frontend.ts	Fixed type checking for meshSource RPC access
src/datasource/enabled_frontend_modules.ts	Registered CATMAID frontend module
src/datasource/enabled_backend_modules.ts	Registered CATMAID backend module
src/datasource/catmaid/register_default.ts	CATMAID provider registration
src/datasource/catmaid/register_credentials_provider.ts	CATMAID credentials provider registration
src/datasource/catmaid/frontend.ts	Frontend implementation for CATMAID data source
src/datasource/catmaid/credentials_provider.ts	Credentials provider for CATMAID authentication
src/datasource/catmaid/base.ts	Base parameter classes for CATMAID sources
src/datasource/catmaid/backend.ts	Backend implementation for CATMAID skeleton fetching
src/datasource/catmaid/api.ts	CATMAID API client implementation
package.json	Added package.json exports for CATMAID modules

Comments suppressed due to low confidence (1)

src/datasource/catmaid/frontend.ts:1

The conditional logic for determining sources is inconsistent. If mesh instanceof CatmaidMultiscaleSpatiallyIndexedSkeletonSource is true, passing allSources[0] (an array) is incorrect as SpatiallyIndexedSkeletonLayer expects a single source or an array of SliceViewSingleResolutionSource. This should pass allSources[0][0].chunkSource in both cases or adjust the constructor signature.

/**

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/skeleton/frontend.ts

src/datasource/catmaid/frontend.ts

src/datasource/catmaid/api.ts

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/skeleton/frontend.ts

src/widget/render_scale_widget.ts

src/skeleton/frontend.ts

src/skeleton/skeleton_chunk_serialization.ts

src/layer/segmentation/index.ts

src/skeleton/api.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ionality is active

afonsobspinto · 2026-02-13T15:58:41Z

https://metacell.atlassian.net/wiki/spaces/NGLA/pages/1199767554/Neuroglancer+Catmaid+Spatially+Indexed+skeletons+diagrams

seankmartin

Reviewed the new catmaid datasource and the skeletons api, looks good! I will review the parts that are more intertwined with the rest of neuroglancer later after I mess around with the neuroglancer build as I think it will be clearer after that

seankmartin · 2026-02-26T09:29:15Z

src/datasource/catmaid/api.ts

+    [key: string]: any;
+}
+
+export interface CatmaidStackInfo {


Do we use all these optional properties in some cases or have docs we can link to about them? For e.g. I've seen orientation commonly as 0. What does it mean if non-zero, do we need to reject it? If we never use some of these optional properties we could likely remove them

seankmartin · 2026-02-26T09:44:21Z

src/datasource/catmaid/api.ts

+}
+
+export const credentialsKey = "CATMAID";
+const DEFAULT_CACHE_GRID_CELL_WIDTH = 25000;


These could maybe benefit from a comment about what using the default grid would mean (I think single LOD with chunks in this shape?)
Also is this default coming from CATMAID or decided by us? I wonder if we could/should suggest slightly more isotropic shaped chunks if we are picking these sizes

As an aside, we likely need to write more detailed docs later - so on the code side I do mean something fairly brief about it

seankmartin · 2026-02-26T09:46:03Z

src/datasource/catmaid/api.ts

@@ -0,0 +1,474 @@
+


Overall this catmaid API looks good to me! V small points:

Missing license at top of file.

In general I think a linting and formatting pass was missed. These commands are:

npm run lint:fix && npm run format:fix uvx nox -s lint format mypy

You can see on #120 I applied these with a git ignore revs so we can keep the blame clean (of course we can remove the ignore revs before making a PR against google main branch since there all the commits get squashed anyway).
When possible it would be good to run the linting/formatting on this and also on the editable skeletons

seankmartin · 2026-02-26T09:56:46Z

src/datasource/catmaid/credentials_provider.ts

@@ -0,0 +1,57 @@
+import {


Missing license

seankmartin · 2026-02-26T09:57:58Z

src/datasource/catmaid/backend.ts

@@ -0,0 +1,183 @@
+/**
+ * @license
+ * Copyright 2024 Google Inc.


Date tends to be 2024 on these files, would expect either 2025 or 2026 depending on file date in what we've made here. So we can do 2026

seankmartin · 2026-02-26T10:05:01Z

src/datasource/catmaid/frontend.ts

+import { Borrowed } from "#src/util/disposable.js";
+import "#src/datasource/catmaid/register_credentials_provider.js";
+
+const METERS_PER_NANOMETER = 1e-9;


Would suggest to just do this inline in coordinateScaleFactors later in this file, 1e-9 is used across the code base like that. And your comment above where this is used is clear enough to understand the 1e-9 usage. But if want to keep the var, suggest changing name to NANOMETER_TO_METER = 1e-9`

seankmartin · 2026-02-26T10:18:32Z

src/datasource/catmaid/register_credentials_provider.ts

@@ -0,0 +1,9 @@
+import { registerDefaultCredentialsProvider } from "#src/credentials_provider/default_manager.js";


Missing license

seankmartin · 2026-02-26T10:22:53Z

src/datasource/graphene/frontend.ts

-    const meshSource = this.getMeshSource();
+    // `DataSubsource.mesh` is widened for spatial skeleton support, but Graphene
+    // segment updates still target mesh sources.
+    const meshSource = this.getMeshSource() as MeshSource | undefined;


I think this would be more appropriate in getMeshSource instead

seankmartin · 2026-02-26T10:23:09Z

src/datasource/graphene/frontend.ts

          (subsource) => subsource.id === "mesh",
        )[0];
        if (meshSubsource) {
          return meshSubsource.subsource.mesh;


Here is where we could cast to MeshSource (the comment you have on 1362 is good as to why, I just think it's better to do it here in case getMeshSource is used elsewhere

As an aside, does graphene also work with multiscale mesh sources? Are we narrowing the type too much?

seankmartin · 2026-02-26T10:44:51Z

src/skeleton/gpu_upload_utils.ts

+  const vertexAttributeTextures: (WebGLTexture | null)[] = [];
+  const numAttributes = vertexAttributeOffsets.length;
+
+  for (let i = 0; i < numAttributes; ++i) {


Leaving a note to check how many attributes usually, could run into texture limits

afonsobspinto added 26 commits November 25, 2025 15:41

feat: WIP - Add catmaid data source

5937007

feat: WIP - Reshape catmaid datasource to segmentation layer

ea43e6a

feat: Load catmaid skeletons into neuroglancer

ac3e06d

feat: Disable application of catmaid stack info

9567de8

feat: WIP - Add spatially indexed skeletons

f52bd13

feat: Create coordinate space based on stack info

b91d088

feat: Add catmaid credentials provider

b1d810a

feat: Add chunk fetching via SliceViewSpatiallyIndexedSkeletonLayer

9579b1a

feat: Draw all nodes without filtering

f58a399

feat: WIP - Draw all nodes with segment colors

5783674

feat: WIP - Draw all nodes as lines

e0e1d3d

feat: WIP - Draw segments with correct colors

509743f

feat: Add 'non-chunked' skeleton source

90967bf

chore: Add info and debug logs

f2fed63

feat: WIP - Add inter-chunk connections

12c4558

fix: WIP - Correct node color in 2D

b82f5a5

feat: WIP - Make visible chunks zoom-relative

357d031

fix: Fix spatially indexed skeleton node colors in 3D viewer

16575dc

feat: Add spatially indexed skeletons to segmentation user controls

1ab935e

fix: Update skeleton shader to work with alpha values

03c778f

feat: Use msgpack instead of json

df3fe7b

feat: Skip visible segments when both sources are active

cd6b18f

feat: Add new hidden segment opacity slider

7dd5bdb

feat: Add LOD slider

5695843

feat: Add CatmaidMultiscaleSpatiallyIndexedSkeletonSource

f7a3bcf

feat: Read extraNodes and limitReached info from catmaid

e520694

afonsobspinto requested a review from Copilot January 26, 2026 10:24

Copilot AI reviewed Jan 26, 2026

View reviewed changes

afonsobspinto changed the title ~~Feature/na 550~~ feat: Add catmaid spatially indexed skeletons Jan 26, 2026

afonsobspinto changed the title ~~feat: Add catmaid spatially indexed skeletons~~ feat: Add catmaid spatially indexed skeletons (read-only) Jan 26, 2026

afonsobspinto added 3 commits February 11, 2026 13:52

feat: Use RenderScaleHistogram as base for catmaid resolution sliders

c91e58a

feat: Add special handling of catmaid's no provider errors

a9ec128

feat: Re-evaluated 2D source selection when renderScaleTarget changes.

cea3fae

afonsobspinto requested a review from Copilot February 11, 2026 16:16

Copilot AI reviewed Feb 11, 2026

View reviewed changes

chore: Remove logs

5bea93f

afonsobspinto force-pushed the feature/NA-550 branch from e6f163b to 5bea93f Compare February 12, 2026 12:42

afonsobspinto and others added 6 commits February 12, 2026 12:45

chore: Have catmaid respect SpatiallyIndexedSkeletonSource interface

d8e91af

Update src/skeleton/frontend.ts

0f7b436

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

chore: Remove expensive gl.getError() call

0ea1537

Update src/skeleton/skeleton_chunk_serialization.ts

48dd890

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

refactor: Make activateDataSubsources type-safe

81dac0d

fix: Update shader to branch alpha by view type

25c2dc1

MetaCell deleted a comment from chatgpt-codex-connector bot Feb 12, 2026

afonsobspinto added 9 commits February 12, 2026 16:20

refactor: Emit new segmentation keys only when spatial skeleton funct…

a83e865

…ionality is active

fix: Rollback opacity shader changes

7f8c36a

refactor: Replace named attribute by explicit flag

08f3349

refactor: Rollback behaviour changes for graphene

d747542

refactor: Add singlenton pattern to Unpackr and CatmaidClient

c13964e

refactor: Extract debounce time to a variable

ca5e345

feat: Maintain global node lookup cache until rebuild is needed

70b87bb

feat: Add debounce to slice view

14de40e

fix: Fix resolution sliders disappearing on skeleton subsource toggle

daa7879

afonsobspinto marked this pull request as ready for review February 13, 2026 15:58

afonsobspinto requested a review from seankmartin February 13, 2026 15:58

seankmartin reviewed Feb 26, 2026

View reviewed changes

		@@ -0,0 +1,9 @@
		import { registerDefaultCredentialsProvider } from "#src/credentials_provider/default_manager.js";

Conversation

afonsobspinto commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CATMAID Datasource and Spatially Indexed Skeletons

CATMAID datasource implementation (src/datasource/catmaid/)

Entry points and registration

Credentials flow

API client (api.ts)

Provider wiring (frontend.ts)

Multiscale grid mapping

Spatially indexed skeletons (src/skeleton/)

Data model and chunk serialization

Chunk download (CATMAID backend)

Backend chunk scheduling and LOD

Rendering pipeline overview

Spatially indexed draw algorithm (per chunk, per segment)

Improvements:

1) Per-draw global node map construction

2) Per-segment compaction and buffer swapping

3) Inter-chunk edges are not rendered

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

afonsobspinto commented Feb 13, 2026

Uh oh!

seankmartin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

afonsobspinto commented Jan 26, 2026 •

edited

Loading

CATMAID datasource implementation (`src/datasource/catmaid/`)

API client (`api.ts`)

Provider wiring (`frontend.ts`)

Spatially indexed skeletons (`src/skeleton/`)