Skip to content

feat: add VirtualFileSystem module - #9960

Draft
lloydrichards wants to merge 43 commits into
Effect-TS:v4/next-minorfrom
lloydrichards:feat/vfs
Draft

lloydrichards wants to merge 43 commits into
Effect-TS:v4/next-minorfrom
lloydrichards:feat/vfs

Conversation

@lloydrichards

@lloydrichards lloydrichards commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Type

  • Feature

Description

For the attention of @fubhy, this is the mvp version of the VirtualFileSystem that will be needed for the follow ups with the MemoryFileSystem and the NodeVirtualProvider. Have reimplemented a lot of internals from the @effect-vfs/core with a sync based engine based a lot on Result. This has been to make the NodeVirtualProvider more of a first-class feature, but means there is a naive runtime implementation needed.

WIP

Since its not possible to stack PRs from forked repos I've needed to back fill the changes for #9730 and #9870 into this branch to make sure everything works. Once these are sorted then I'll rebase and remove the commits before making this ready.

Adds an experimental, in-memory VirtualFileSystem for tests and tools that need file operations without touching the host filesystem.

flowchart TB
subgraph VirtualFileSystem
    Owner(["Owner"]) -->|authorizes| Caller(["Scoped callers"])
    Caller -->|reads and writes| Files(["Files, dir and links"])
    Files -->|opens| Handles(["Scoped handles"])
    Guards(["Revision conditions"]) -->|guards| Files
    Owner <-->|captures and restores| Capture(["Capture and restore"])
end
subgraph core
  Owner -->|powers| Volume["Shared volumes and callers"]
  Handles -->|supplies sync APIs| Sync["Synchronous adapter binding"]
  Capture -->|supplies state| Snapshot["Portable snapshots"]
end

  classDef preview fill:#ede9fe,stroke:#7c3aed,color:#2e1065,stroke-width:3px
  classDef core fill:#dbeafe,stroke:#2563eb,color:#172554,stroke-width:3px

  class Owner,Caller,Files,Handles,Guards,Capture preview
  class Volume,Watches,Fixtures,Testing,Snapshot,Overlay,Delta,Search,Sync,Live core
Loading

The module provides:

  • Files, directories, symbolic links, hard links, and scoped handles.
  • Explicit caller credentials, roots, and working directories.
  • Configurable limits for storage, handles, and operation work.
  • Atomic changes that publish no partial state after an expected failure.
  • Capture and restore that preserve contents, metadata, and links.

Effect APIs and synchronous Result APIs share the same checks.

import { Effect } from "effect"
import { VirtualFileSystem } from "effect/virtual-file-system"

const program = Effect.scoped(Effect.gen(function*() {
	const owner = yield* VirtualFileSystem.make()
	const files = yield* owner.authorize({
		credentials: { uid: 0, gid: 0 }
	})

	yield* files.writeFile("/hello.txt", new TextEncoder().encode("Hello"))
	return new TextDecoder().decode(yield* files.readFile("/hello.txt"))
}))

await Effect.runPromise(program) // "Hello"

Benchmarks

Median of three round means, in µs per sample; lower is better. Node 26.7.0. Namespace operations use 4,096 background files.

Operation VirtualFileSystem @effect-vfs/core @effect-vfs/memory FileSystem on Node node:fs/promises
Stat 9.52 3.36 4.31 10.56 9.13
Open + close 16.22 21.36 22.35 35.82 34.05
Create + unlink 36.01 25.98 26.37 111.13 114.42
Rename roundtrip 58.61 32.57 40.21 118.73 116.36
Grow 4 → 8 KiB 15.37 11.86 13.48 86.96 103.42
List 4,096 entries 363.31 1,121.58 2,257.09 1,450.64 1,438.87
Recursive mkdir, depth 32 102.15 213.81 265.26 1,679.35 1,596.89
Read 4 KiB 10.20 4.65 4.97 48.65 50.49
Overwrite 1 MiB 123.39 75.85 116.74 239.85 284.62
Write 4 KiB through an open handle 5.46 9.32 11.71 8.59 9.29
Scan 32 files: stat + read 619.02 253.96 297.52 1,810.79 2,032.70
Read 32 files, concurrency 32 326.28 155.25 156.11 373.07 528.89

Setup, cleanup and cursor resets are excluded. Roundtrips and batches include all named operations. Host I/O uses warm caches without fsync.

These are exploratory cross-implementation results; API semantics and Effect versions differ. Vitest also reported export-getter overhead in the shared benchmark harness.

Related

@changeset-bot

changeset-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 7f91d41

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Minor
@effect/opentelemetry Minor
@effect/vitest Minor
@effect/ai-amazon-bedrock Minor
@effect/ai-anthropic Minor
@effect/ai-cloudflare Minor
@effect/ai-openai-compat Minor
@effect/ai-openai Minor
@effect/ai-openrouter Minor
@effect/ai-typesafe Minor
@effect/atom-react Minor
@effect/atom-solid Minor
@effect/atom-vue Minor
@effect/platform-browser Minor
@effect/platform-bun Minor
@effect/platform-cloudflare Minor
@effect/platform-deno Minor
@effect/platform-node-shared Minor
@effect/platform-node Minor
@effect/sql-clickhouse Minor
@effect/sql-d1 Minor
@effect/sql-libsql Minor
@effect/sql-mssql Minor
@effect/sql-mysql2 Minor
@effect/sql-pg Minor
@effect/sql-pglite Minor
@effect/sql-sqlite-bun Minor
@effect/sql-sqlite-do Minor
@effect/sql-sqlite-node Minor
@effect/sql-sqlite-react-native Minor
@effect/sql-sqlite-wasm Minor
@effect/docgen Minor
@effect/doctest Minor
@effect/graphql-generator Minor
@effect/openapi-generator Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 40.18 KB 40.18 KB 0.00 KB (0.00%)
basic.ts 7.35 KB 7.35 KB 0.00 KB (0.00%)
batching.ts 11.30 KB 11.30 KB 0.00 KB (0.00%)
brand.ts 7.00 KB 7.00 KB 0.00 KB (0.00%)
cache.ts 11.83 KB 11.83 KB 0.00 KB (0.00%)
config.ts 23.18 KB 23.18 KB 0.00 KB (0.00%)
differ.ts 22.30 KB 22.30 KB 0.00 KB (0.00%)
http-client.ts 27.70 KB 27.70 KB 0.00 KB (0.00%)
http-router.ts 37.72 KB 37.70 KB +0.02 KB (+0.05%)
logger.ts 12.28 KB 12.28 KB 0.00 KB (0.00%)
metric.ts 9.74 KB 9.74 KB 0.00 KB (0.00%)
optic.ts 7.21 KB 7.21 KB 0.00 KB (0.00%)
pubsub.ts 16.34 KB 16.34 KB 0.00 KB (0.00%)
queue.ts 13.20 KB 13.20 KB 0.00 KB (0.00%)
schedule.ts 12.07 KB 12.07 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.33 KB 14.33 KB -0.00 KB (-0.01%)
schema-binary.ts 42.58 KB 42.58 KB +0.00 KB (+0.01%)
schema-class.ts 22.06 KB 22.06 KB 0.00 KB (0.00%)
schema-codegen.ts 30.80 KB 30.80 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.67 KB 30.67 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.36 KB 28.35 KB +0.00 KB (+0.01%)
schema-string-transformation.ts 15.64 KB 15.64 KB 0.00 KB (0.00%)
schema-string.ts 12.53 KB 12.53 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.51 KB 16.51 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.75 KB 39.75 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 21.21 KB 21.21 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.38 KB 21.38 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.47 KB 21.47 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.51 KB 26.51 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.55 KB 21.55 KB 0.00 KB (0.00%)
schema.ts 21.14 KB 21.14 KB 0.00 KB (0.00%)
stm.ts 14.56 KB 14.56 KB 0.00 KB (0.00%)
stream.ts 10.66 KB 10.66 KB 0.00 KB (0.00%)
supervisor.ts 11.61 KB 11.61 KB 0.00 KB (0.00%)

@effect-bot
effect-bot force-pushed the v4/next-minor branch 2 times, most recently from 0dfd080 to e75d84d Compare October 26, 2026 20:48
@effect-bot
effect-bot force-pushed the v4/next-minor branch 5 times, most recently from aca2532 to 06bee48 Compare October 27, 2026 00:51
@effect-bot
effect-bot force-pushed the v4/next-minor branch 2 times, most recently from 87236f9 to ade8259 Compare October 27, 2026 20:13
@effect-bot
effect-bot force-pushed the v4/next-minor branch 2 times, most recently from 97fda01 to 04046f5 Compare October 27, 2026 22:05
@lloydrichards
lloydrichards force-pushed the feat/vfs branch 3 times, most recently from 8e643c6 to c7adb80 Compare October 27, 2026 22:19
@effect-bot
effect-bot force-pushed the v4/next-minor branch 5 times, most recently from a6e263b to 170ef52 Compare October 27, 2026 22:57
@lloydrichards

lloydrichards commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

I paused my development of effect-vfs for almost a month waiting on this and still there is no movement on the BytePath or UnixNano 🤷 have gone and ported this code over into an effect-preview package which I'll use at the foundations for some more ideas I want to explore.

I might port back some patches into this branch from time to time if needed but will hold off on pursuing the effect port until I hear some updates.

This branch is waiting to be deployed

1 waiting (outdated) deployment
fork — be990e7a Waiting Oct 28, 2026 by lloydrichards via approval-gate #23811
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants