Skip to content

feat(canvas): draw NativeScript views into a canvas on Windows - #257

Open
triniwiz wants to merge 1 commit into
masterfrom
feat/html-in-canvas-windows
Open

triniwiz wants to merge 1 commit into
masterfrom
feat/html-in-canvas-windows

Conversation

@triniwiz

@triniwiz triniwiz commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

HTML-in-Canvas on NativeScript Windows (WinUI 3).

Change

NSCDrawableHost (canvas-napi, host/windows/drawable.rs)

  • Children live in a transparent XAML Canvas on top of the SwapChainPanel, each in its own Grid slot. Opacity hides them but keeps them hit-testable.
  • A slot carries the drawn geometry, so XAML's own hit testing finds the view where it was drawn:
    • a RenderTransform (the only transform XAML input follows: it ignores Projection and UIElement.TransformMatrix);
    • a RectangleGeometry clip for canvasClip;
    • IsHitTestVisible only while the child has geometry.
      Raising uses Canvas.ZIndex, so the element is never re-parented.
  • Matrices with perspective (picking.rs): the host subclasses the XAML island's windows and sees each pointer message before XAML. For every such view it maps the point back through the matrix and gives the slot the matrix's tangent there, which is exact at the pointer. XAML then hit-tests and delivers the event as usual, so mouse, pen and a touch's first contact all reach the view where it was drawn, as in Chrome. NSCDrawableHost.aim(x, y) does the same for a point without a pointer (the spec uses it).
  • Hosted views keep the size they were given: layout rounding is off for them, which otherwise makes 50 DIP 50.4 at 125%. Snapshots are resampled to the requested pixel size, since RenderToSizeAsync takes whole DIPs.
  • WinUI 3 has no per-element invalidation and only an async snapshot (RenderTargetBitmap). Views are re-rendered and compared: a quarter-size probe first, then the full snapshot if it differs. The schedule is hot for 0.5 s after a change or input, then backs off from 66 ms to 1 s. It runs on time, not frames. A paint event follows a change by a frame or more.
  • The async steps are polled from update() rather than given completion handlers, because windows-rs delegates must be Send and XAML objects are not. Input on the hosted views only sets an atomic flag.
  • JS makes one update() call a frame. Pixel comparison, BGRA loading, and the crop/scale for captureElementImage and cropped uploads (from the last snapshot) all happen in Rust.
  • host/windows/xaml.rs declares the WinUI 3 members the host calls (the windows crate covers only Windows.*): each interface's IID and its vtable up to the last slot used, taken from the Windows App SDK 1.6.250205002 metadata (the version the NativeScript Windows runtime ships), plus the pointer/key event delegate. Published WinRT interfaces never change, and each is reached by QueryInterface.

TS: DrawableController.canvasCssSize(): XAML lays the panel out, so the Windows controller reads its size from there. A press on a hosted view no longer takes pointer capture and focus for the canvas. The htmlincanvas suite is registered on Windows; its hit-testing checks use VisualTreeHelper.FindElementsInHostCoordinates, which hit-tests as input does.

Demo: follows the system theme (dark .ns-dark colors, DayNight on Android); the canvas-html card uses classes for it.

Verified

On Windows 11 (Intel Iris Xe, 125% scale):

  • htmlincanvas: 25/25.
  • The full canvas spec suite: all passing, together with fix(canvas): Windows font fallback and WebGPU canvases as WebGL sources #258.
  • The canvas-html demo renders the rotated form. Real mouse clicks on the drawn button and typing into the drawn text field reach those views.
  • With the form's hit geometry given a perspective rotation (rotateY 40°, offset from where it is drawn): real mouse clicks and injected touch taps at the button's projected position tap it, a click and typing at the text field's projected position edit it, and a click at the old position does nothing.
  • htmlincanvas.hittest has a perspective case at points 2 px inside and outside the button's edges, where an approximation taken at the button's center is wrong on both; it fails when aim is skipped.

The hit-testing checks call VisualTreeHelper.FindElementsInHostCoordinates(Point, UIElement), which needs NativeScript/windows#55 to reach the Point overload. Loading canvas on Windows needs NativeScript/windows#54.

Canvases that present through a SwapChainPanel are not in the snapshots (RenderTargetBitmap does not capture them). A view whose hit geometry changes (an animation, or the pointer moving over a view with perspective) re-renders on every check, since XAML rasterizes through the transform and each one gives different pixels.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 196cbf32-6ae9-47d3-af6c-af11af4c2ab0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@triniwiz
triniwiz force-pushed the feat/html-in-canvas-windows branch 9 times, most recently from 2cf499a to aa81a81 Compare November 20, 2026 00:13
@triniwiz
triniwiz marked this pull request as ready for review November 20, 2026 00:17
@triniwiz
triniwiz force-pushed the feat/html-in-canvas-windows branch from aa81a81 to f7d20b8 Compare November 20, 2026 01:15
HTML-in-Canvas on WinUI 3. The children of a content="drawable"
canvas live in a transparent XAML Canvas on top of the SwapChainPanel,
each in a Grid slot that carries the drawn geometry (a RenderTransform,
a clip, and hit-test visibility), so XAML's own hit testing finds the
view where it was drawn. XAML input follows neither a Projection nor a
4x4 TransformMatrix, so for a matrix with perspective the host sees each
pointer message before XAML (it subclasses the island's windows): the
view's slot gets the matrix's tangent at the point under the pointer,
exact there, for mouse, pen and a touch's first contact alike. Hosted
views keep the size they were given (no layout rounding), and snapshots
are resampled to the requested pixel size, since RenderTargetBitmap
takes whole DIPs.

The host is native (NSCDrawableHost in canvas-napi): JS makes one
update() call a frame. host/windows/xaml.rs declares the WinUI 3
members it calls, from the Windows App SDK 1.6 metadata (the windows
crate covers only Windows.*). WinUI 3 has no per-element invalidation
and only an async snapshot, so views are re-rendered with
RenderTargetBitmap on a schedule that is hot for half a second after a
change or input and backs off to once a second when idle, and compared
(a quarter-size probe first). The async steps are polled from update():
XAML objects are not Send. captureElementImage and cropped uploads come
from the last snapshot.

A press on a hosted view no longer takes pointer capture and focus for
the canvas. The htmlincanvas spec suite is registered on Windows. The
demo follows the system theme.
@triniwiz
triniwiz force-pushed the feat/html-in-canvas-windows branch from f7d20b8 to 92e8540 Compare November 20, 2026 02:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant