feat: add screen-space ambient occlusion (SSAO) by hubbardp · Pull Request #993 · google/neuroglancer

hubbardp · 2026-05-15T14:49:46Z

Summary

Screen-space ambient occlusion (SSAO) simulates shadows on 3D mesh surfaces by darkening crevices and concavities where ambient light would be occluded. It adds depth cues that help us perceive shapes, and it makes the display more appealing. SSAO is an efficient post-processing effect applied to the perspective view after opaque geometry is drawn. See src/ssao/README.md for more details and example images.

Screenshots

Usage

Press q to toggle SSAO on and off.
Use sliders in the "Settings" panel to adjust intensity and radius (softness).

Known limitations

SSAO is disabled in any perspective view that contains a volume-rendering layer; a one-time status banner notifies the user.
Translucent annotations covering a mesh suppress SSAO at the covered pixels.

Performance

The first use of SSAO triggers the creation of the NORMAL attachment, retained thereafter.
The first use of SSAO triggers mesh shader recompilation.
All uses of SSAO add three additional full-screen passes.
No user-perceptible change in performance, even when tested on older laptops with only integrated graphics.

Algorithm

GTAO (Ground Truth Ambient Occlusion): Jimenez et al., "Realtime Strategies for Accurate Indirect Occlusion", SIGGRAPH 2016

Testing

Manually verified with various datasets from Janelia, OpenOrganelle.
New browser tests in src/ssao/shaders.browser_test.ts cover composite math, GTAO and composite sentinel paths, blurring pass.

chrisj · 2026-05-15T16:10:32Z

@hubbardp this is very exciting! Looks great from a quick look so far

flyem_fib-25 DEMO

chrisj · 2026-05-15T16:18:14Z

+  // World→UV scale: wClip is -P.z under perspective and 1 under ortho.
+  float wClip = uProjection[2][3] * P.z + uProjection[3][3];
+  float screenRadius = uRadius * uProjection[1][1] / (2.0 * wClip);
+  screenRadius = min(screenRadius, MAX_KERNEL_FRACTION);


In the link I posted, I can't see any visible difference when changing the radius value, I'm wondering if the value is always larger than MAX_KERNEL_FRACTION

Thanks for the observation. I checked in what I think is an improvement; see more with your later comment, below.

hubbardp · 2026-05-15T16:33:24Z

For those who want to try the new functionality without building the code, it's running here:
http://neuroglancer-ssao.janelia.org/

chrisj · 2026-05-15T16:50:01Z

@hubbardp neuroglancer github actions are set up to create a deployment for every PR, available if you click "view details", it's definitely not obvious:

That's what I used to create the link I posted.

jbms · 2026-05-15T21:18:59Z

What is the reason for disabling the effect on highlighted segments? It makes the highlighting rather jarring, though I haven't tested with the alternative behavior.

hubbardp · 2026-05-15T21:43:56Z

What is the reason for disabling the effect on highlighted segments? It makes the highlighting rather jarring, though I haven't tested with the alternative behavior.

I tried not disabling the SSAO darkening on highlighted segments, and they were too dark to see when highlighted. Personally, I like the brighter highlighting. I found that before SSAO, sometimes the randomly-chosen segment color was bright enough that the highlighting seemed barely distinguishable.

fcollman · 2026-05-16T01:14:02Z

this look really cool! i'm excited by it.

I immediately went to a very busy microns dataset to see what it would look like

link

and I see thing looks reasonable when viewed from the +/-xz, +/-yz view, but rotating it +/- xy everything looks dim and muddy.

YZ looks good here

XY all the cells in the middle look dulled, when i was expecting some to look brighter.

hubbardp · 2026-05-16T16:45:13Z

this look really cool! i'm excited by it.
I immediately went to a very busy microns dataset to see what it would look like
and I see thing looks reasonable when viewed from the +/-xz, +/-yz view, but rotating it +/- xy everything looks dim and muddy.

Thanks for the test case. I pushed a fix for the anisotropy. I think it looks better from all view angles.

I have not yet updated http://neuroglancer-ssao.janelia.org/ so use the version from the GitHub action.

fcollman · 2026-05-16T16:52:59Z

awesome thank you agreed looks better on my localhost version!

seankmartin

Thanks for the changes! My comments are more related to the integration in neuroglancer, as I'll admit that I haven't dug into the SSAO implementation compared to the paper. Are there any known simplifications or deviations taken here over the paper description?

seankmartin · 2026-05-19T15:11:38Z

+
+SSAO is limited to mesh surfaces because only `MeshLayer` and
+`MultiscaleMeshLayer` supply a view-space normal via the three-argument
+`emit(color, pickId, viewNormal)` form. All other opaque geometry (skeletons,


I wouldn't see it as blocking, but in particular for the single meshes, I think it could be useful to expand here why they are not supported. Since they have normaIs, I imagine it's because the overhead in making the shader in the single mesh layer shader emitter aware and some duplication of the normal handling basically mimicking the changes made in mesh/frontend.ts

Thanks for the suggestion. I checked in a revised README that expands on the reasons for omitting annotations, skeletons and single-mesh layers. For the latter, I could add support with some refactoring, so my preference would be to leave it for future work. Let me know what you think.

This is perfect, thank you for updating the docs, looks great. No need to add right now in my opinion, just wanted to document why it wasn't added.

seankmartin · 2026-05-19T15:24:31Z

+  // Referenced by glsl_perspectivePanelEmitWithNormals; declared here so any
+  // shader using this emitter (mesh, annotation, etc.) gets it without having
+  // to know the emit body's internal dependencies.
+  builder.addUniform("highp float", "uHighlighted");


I think the use of a uniform for highlighted to disable AO in the shader will up being a bit limited in scope. This assumes that in a single draw everything is either highlighted or unhighlighted, which is a bit limiting.

Perhaps this could move to the responsibility of the mesh layer if this functionality is needed. So meshes zero-out their view normal to the no AO sentinel version. And then the perspective shaders don't have to worry about this anymore.

If other layers want to disable AO dynamically, they can use the same pattern

Thanks for the suggestion. I checked in code that implements what I think is your idea.

Yes exactly what I had in mind, thank you very much for the change

seankmartin · 2026-05-19T15:29:49Z

+export const glsl_perspectivePanelEmitWithNormals = `
+void emit(vec4 color, highp uint pickId, vec3 viewNormal) {
+  out_color = color;
+  float zValue = 1.0 - gl_FragCoord.z;
+  out_z = vec4(zValue, zValue, zValue, 1.0);
+  float pickIdFloat = float(pickId);
+  out_pickId = vec4(pickIdFloat, pickIdFloat, pickIdFloat, 1.0);
+  // Highlighted objects collapse to the zero sentinel so SSAO leaves them alone.
+  vec3 packedNormal = (1.0 - uHighlighted) * (normalize(viewNormal) * 0.5 + 0.5);
+  out_normal = vec4(packedNormal, 1.0);
+}
+void emit(vec4 color, highp uint pickId) {
+  out_color = color;
+  float zValue = 1.0 - gl_FragCoord.z;
+  out_z = vec4(zValue, zValue, zValue, 1.0);
+  float pickIdFloat = float(pickId);
+  out_pickId = vec4(pickIdFloat, pickIdFloat, pickIdFloat, 1.0);
+  // Zero-RGB sentinel; alpha=1 so the source overwrites dst under
+  // blend(SRC_ALPHA, ONE_MINUS_SRC_ALPHA).
+  out_normal = vec4(0.0, 0.0, 0.0, 1.0);


I think it would be nice if we could reduce the duplication in the bodies here so changes propagate if ever needed and to clarify what's changing between emitters. Maybe something like

const glslEmitBase = ` out_color = color; float zValue = 1.0 - gl_FragCoord.z; out_z = vec4(zValue, zValue, zValue, 1.0); float pickIdFloat = float(pickId); out_pickId = vec4(pickIdFloat, pickIdFloat, pickIdFloat, 1.0);`;

and then it could be used in both the 2-arg and 3-arg emits, and also be replaced in glsl_perspectivePanelEmit. Don't feel strongly on the name of the var glslEmitBase

Thanks, I checked in this idea for factoring out shared code.

seankmartin · 2026-05-19T15:38:15Z

+    // Mixed mesh + volume scenes are not yet supported; the opaque pass and
+    // NORMAL writes already happened from the with-normals emitter (minor
+    // waste, not a correctness issue). Notify once per panel lifetime via
+    // both the console and a dismissable status banner.


I'm unsure why volume rendering disables SSAO. SSAO happens in opaque rendering only, so I don't see how the later transparent rendering is interfered with. Could be missing something of course

I did another test of the code modified to allow SSAO with volume rendering, and the result is not right. The problem is that SSAO involves a final compositing step that multiplies the existing color buffer by the AO darkening. By the time that step runs, the volume has been blended into the color buffer, so AO darkens the volume contribution, too, and not just the mesh underneath. The part of the volume in front of the mesh should hide those AO details, but instead it gets darkened along with them. It should be possible to restructure the pipeline so the volume rendering is added after the SSAO compositng, but that would add to the scale of the code changes. Let me know if you think it is worth doing.

Thanks @hubbardp, makes sense! In my opinion it's fine for now to leave it, I was just conceptually missing why they weren't compatible. We can make a note/issue about it and come back to restructuring the pipeline

seankmartin · 2026-05-19T15:42:58Z

+      );
+      const radius = ssaoRadius * this.navigationState.zoomFactor.value;
+
+      this.ssaoManager.render(


Just wanted to note that as the complexity of the panel.ts file has increased, moving part of the rendering responsibility out of this file, as done here, seems reasonable and could be a good idea for a later refactoring of the volume rendering path

seankmartin · 2026-05-19T15:45:04Z

+    // NORMAL writes already happened from the with-normals emitter (minor
+    // waste, not a correctness issue). Notify once per panel lifetime via
+    // both the console and a dismissable status banner.
+    if (ssaoRequested && hasVolumeRendering && !this.ssaoVolumeWarned) {


Related to https://github.com/google/neuroglancer/pull/993/changes#r3267587804, if we could leave AO on with volume rendering - maybe we could instead enable this warning message if you have an AO supported mesh layer but with transparent rendering for that layer?

Thanks for the suggestion. A common use of transparency is silhouette rendering for ROIs, and in my tests, it looks fine with SSAO. So I feel that a UI warning would not be helpful. If you disagree, then it might be worth scheduling the work of restructuring the pipeline so transparency is added after the SSAO compositing. Other than some additional code, the main cost of that approach is one more RGB FBO.

Sounds good, thanks. On rereading what I wrote, honestly I wasn't very clear, but really I just meant that if we're enabling volume rendering and SSAO - do we then benefit from replacing the warning here with something else. For example, if SSAO is on but the only mesh layer in the scene is transparent or there are no mesh layers in the scene, then that could also be a warning. But with volume rendering and SSAO not permitted to be on together, then I think we just keep the current warning as it is the most important one.

seankmartin · 2026-05-19T15:56:52Z

@@ -0,0 +1,292 @@
+/**


As a general comment here, I could be missing it but I don't see a test for a case where ao is between 0 and 1, or in other words a non 1x1 size texture example to check sampling from nearby neighbours and having ao < 1.0 if the centre pixel is being occluded by neighbour pixels. Would that be possible to add?

Thanks for checking. I committed two new tests for these situations.

Those look great, thank you very much for that, will be really useful for catching any clear regressions.

chrisj · 2026-05-19T21:02:53Z

+        SSAO_RADIUS_RANGE.max,
+        Math.max(SSAO_RADIUS_RANGE.min, this.viewer.ssaoRadius.value),
+      );
+      const radius = ssaoRadius * this.navigationState.zoomFactor.value;


removing this operation causes the radius slider to have an effect on my device. It does mean it now becomes softer as you zoom out.

The old code scaled the radius incorrectly, so the clamping at MAX_KERNEL_FRACTION was happening too often. I checked in new code that clarifies how the slider value affects the sampling kernel and the falloff factor. I think it looks reasonable when zooming out, and lets the slider add some variation. The new default of 2 for the slider works well in every case I tried. I would not mind removing the slider altogether if it is confusing, but it also seems okay to leave it in for completeness.

seankmartin · 2026-05-19T21:05:17Z

One other thing that might be nice is if we add some kind of DEBUG_SSAO const to the ssao/shaders.ts which when enabled sets the fragment color to be the non-color composited ao value in the composite step (at the moment you can set fixed color to white in the segmentation layer instead for a similar effect) defineSSAOCompositeShader. For example:
AO debug:

While the regular result with color contribution is:

Could help for any later fine tuning, seems like there might be a bit of noise showing through in certain cases for e.g.
https://github.com/user-attachments/assets/432f450f-4b31-4fe4-928c-3c84fcf88b34

chrisj · 2026-05-22T12:42:18Z

+
+import type { ShaderBuilder } from "#src/webgl/shader.js";
+
+const glsl_gtao = `


if it can be done, I would try defining gtao as a function which should allow not having to split up the shader code into fragment code and fragment main

and call gtao inside the fragment main

Good idea. I checked in code that I think uses this approach.

chrisj · 2026-05-22T12:44:02Z

+}
+
+const glsl_blur = `
+// Bilateral falloff sharpness; tuned for normalized [0,1] depth so that


I would put this and glsl_ssaoComposite inside their respective functions

The new code is using this idea, too.

stuarteberg · 2026-05-22T13:20:08Z

I don't know much about the implementation here, so sorry if the following is just noise. But I was wondering...

Could this PR be modified to allow SSAO to be enabled (and perhaps configured) on a per-layer basis? Admittedly, from a conceptual point of view, it should probably apply to the entire scene, not just particular layers. However, that goes a little against the grain of the current neuroglancer UI, which offers per-layer controls for mesh opacity and mesh silhouette settings.

The fact that the user can't combine SSAO with opacity/silhouette effects is further motivation for moving its settings into the layer controls. That way, if the user enables SSAO for a layer, then the opacity and silhouette controls would just disappear and the SSAO controls would appear. (That's similar to how the volume rendering controls appear/disappear in img datasources, for example.) It also would give the user the ability to select their own keybinding for the control, rather than hard-coding it to q.

If this is implementable, it has the advantage that users could specify different SSAO settings for each layer. I'm sure that violates the basic principles behind SSAO in some sense, but there could be practical reasons that a user might do this, for example, to highlight a particular population of cells without disabling SSAO on them entirely. (A disadvantage is that if users want all of their layers to use consistent SSAO settings anyway, they'd have to separately specify those settings in each layer.)

Again, I know little of the implementation, so maybe this isn't feasible. But the fact that the current implementation is able to exclude specific objects from SSAO (e.g. the highlighted cell) made me wonder if SSAO really needs to be a global setting.

hubbardp · 2026-05-22T18:56:19Z

I don't know much about the implementation here, so sorry if the following is just noise. But I was wondering...
Could this PR be modified to allow SSAO to be enabled (and perhaps configured) on a per-layer basis?

Thanks for the suggestion. The goal of having per-layer control for SSAO sounds appealing, but the details get tricky.

SSAO involves darkening a point on one mesh based on how much of the hemisphere above it is blocked by other meshes. What makes it fast is that the other meshes are found from the current state of the depth buffer. SSAO also involves a compositing step to do the actual darkening, once the depth buffer has been used to determine how much darkening is appropriate.

Say there are three layers---A, B, C---and A and C opt into SSAO. What does it mean that B has opted out?

Should the meshes from B be darkened by SSAO? No.
Should the meshes from B contribute to the darkening of meshes on A and C? Unclear.

If the answer to question 2 is "yes" then the current sentinel mechanism could be reused; layer B would be treated like an annotation or skeleton layer. If the answer is "no" then the solution would require more complicated depth-buffer management that would not be trivial.

I think it is important to focus on the most beneficial use cases for SSAO: using it on all the meshes to provide perceptual cues that help a user see and understand the data. Refinements to handle other cases make more sense as optional follow-on work.

hubbardp · 2026-05-25T20:01:30Z

One other thing that might be nice is if we add some kind of DEBUG_SSAO const to the ssao/shaders.ts which
when enabled sets the fragment color to be the non-color composited ao value in the composite step...

The debugging mode is a good idea, and I added a DEBUG_SSAO flag to ssao/shaders.ts, in the glsl_ssaoComposite routine.

The noise you notice is real, and is part of the GTAO algorithm. A hash function based on the fragment coordinates jitters the angle and step for the horizon-angle sampling, and it works well in general. But the rejection of blurring across depth boundaries can reject enough samples that the jitter becomes visible sometimes. Some sort of temporal adjustment would help but would require additional machinery and an FBO to maintain state. In my experience, the noise is noticeable mainly on broad, smooth surfaces, and since such surfaces are uncommon in neuroscience data sets I think the current noise is acceptable.

hubbardp · 2026-05-25T21:07:54Z

I think I have now addressed all the comments from reviewers.

seankmartin · 2026-06-08T15:17:19Z

I think I have now addressed all the comments from reviewers.

Thanks for all the changes! Sorry for the delay, I'll try to get around to taking a last look soon. Ideally I'd also like to see @jbms has any further thoughts on this one since it is a change that affects the neuroglancer state

jbms · 2026-06-08T16:30:37Z

A few comments:

It may be better to make all ssao options under a single top-level state key ssao --- both in json and in code, so that there is only one option to pass down instead of 3.
The README.md file contains a mix of implementation details and user-level documentation. It would be better to keep implementation details in the README but move user documentation to the sphinx documentation.
The screenshots add over 1MB to repo size and I'm not sure you can even see much in those particular screenshots other than that it becomes darker with SSAO. I think we could just skip them from the implementation detail documentation. Screenshots would be super useful in the user documentation but I think those need to be stored ouside the repo, either in a bucket or using git lfs, since comprehensive docs of everything would surely have hundreds of screenshots but we don't want to balloon the repo size to hundreds of MB of data.

feat: add screen-space ambient occlusion (SSAO)

5b03482

chrisj reviewed May 15, 2026

View reviewed changes

Fixes a test failure due to Python line length.

f37900d

Fixes SSAO for anisotropic volumes.

bfd71cc

Fixes the scale bar with SSAO.

ceb02d0

seankmartin reviewed May 19, 2026

View reviewed changes

chrisj reviewed May 19, 2026

View reviewed changes

hubbardp added 3 commits May 20, 2026 10:44

Fixes use of the ssao radius slider value.

2c539a1

Updates documentation on types omitted from SSAO.

514097b

Improves how highlighted segments are excluded from SSAO.

ad4e877

seankmartin mentioned this pull request May 21, 2026

feat(panels): allow a debug visualisation for offscreen textures #1002

Open

Factors out shared code.

e799d44

chrisj reviewed May 22, 2026

View reviewed changes

seankmartin mentioned this pull request May 22, 2026

refactor: volume rendering layer and perspective panel interactions #1008

Open

hubbardp added 2 commits May 25, 2026 10:41

Adds more SSAO tests.

b79840c

Adds DEBUG_SSAO for visualizing AO alone.

3548525

Improves the SSAO GLSL organization.

9469c88

seankmartin mentioned this pull request Jun 9, 2026

Annotation user guide #1006

Merged


		import type { ShaderBuilder } from "#src/webgl/shader.js";

		const glsl_gtao = `

Uh oh!

Conversation

hubbardp commented May 15, 2026

Uh oh!

chrisj commented May 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubbardp commented May 15, 2026

Uh oh!

chrisj commented May 15, 2026

Uh oh!

jbms commented May 15, 2026

Uh oh!

hubbardp commented May 15, 2026

Uh oh!

fcollman commented May 16, 2026

Uh oh!

hubbardp commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcollman commented May 16, 2026

Uh oh!

seankmartin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seankmartin May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seankmartin May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seankmartin commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

hubbardp commented May 16, 2026 •

edited

Loading

seankmartin May 22, 2026 •

edited

Loading

seankmartin May 19, 2026 •

edited

Loading

seankmartin commented May 19, 2026 •

edited

Loading

hubbardp May 25, 2026 •

edited

Loading