feat(orch): collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults

[kalyazin]

Why

A cold (GCS-backed) sandbox resume demand-faults guest RAM through userfaultfd at 2 MiB hugepage granularity, and each distinct 2 MiB frame is a separate serial fetch from remote storage (~tens of ms each). On the resume critical path the user cgroup is still frozen (freeze-user-cgroup), so the only pages faulted before envd answers /init are envd's own. envd is a long-lived Go process whose heap fragments over time across many distinct guest-physical 2 MiB frames at low fill — so restoring envd touches far more frames than its live byte size implies, and that dominates time-to-envd on a cold resume.

MADV_COLLAPSE migrates those scattered 4 KiB live pages into contiguous 2 MiB hugepages, collocating them into fewer distinct frames — so the post-pause snapshot faults fewer frames on the next resume. This PR adds an envd endpoint to do that and wires the orchestrator to call it just before pause, behind a flag.

What

envd: POST /collapse (packages/envd/, version bumped to 0.6.4)

• Walks /proc/self/maps, and for each anonymous read-write region marks it MADV_HUGEPAGE then issues MADV_COLLAPSE per 2 MiB chunk (skipping empty chunks).
• Returns 200 + JSON CollapseResult{regions,chunks,collapsed,skipped,elapsedMs} and logs the same stats in-guest.
• Notes:
  • Chunk-wise, not whole-VMA: a single MADV_COLLAPSE over a whole region EINVALs on the first empty chunk; iterating 2 MiB windows and skipping empties is what makes it work on a sparse heap.
  • MADV_HUGEPAGE first sets VM_HUGEPAGE so the collapse is eligible regardless of the system THP mode (no global transparent_hugepage=always needed).
  • Best-effort: per-chunk failures are counted, never fatal.

orchestrator: flag-gated pre-pause collapse (packages/orchestrator/pkg/sandbox/)

• New LD flag collapse-envd-heap (default off). When set, bestEffortReclaim calls envd's /collapse just before pause, alongside the existing freeze step.
• Gated on envd version ≥ 0.6.4 (MinEnvdVersionForHeapCollapse) so older orchestrators/envds never call an endpoint that isn't there; dedicated 10 s timeout; best-effort so a failure never blocks pause.
• callEnvdCollapse decodes the CollapseResult (envd's client stub regenerated for the new 200+body shape).

Observability (packages/shared/pkg/telemetry, orchestrator)

• Histogram orchestrator.sandbox.envd.collapse.duration (round-trip ms, success attribute) — the pause-path cost.
• Counter orchestrator.sandbox.envd.collapse.chunks split by result=collapsed|skipped — chunk attempts (total) vs successes (collapsed).
• Dedicated envd-collapse span carrying collapse.{regions,chunks,collapsed,skipped,duration_ms,success}, plus a success log line ("envd heap collapsed") so efficacy is visible in Loki even when per-process metric counters reset on an orchestrator restart.

Validation

• Unit tests (internal/services/memory): /proc/self/maps anon-rw parser, collapseRange against a sparsely-populated mmap'd region (one touched page per 2 MiB window), and a CollapseSelf smoke test. go build/vet clean; golangci-lint 0 issues across the changed packages.
• resume-build A/B (GCS-cold, envd heap physically scattered): cold-resume working set CTRL 1586 → COLL 465 distinct 2 MiB frames (−71 %), latency 32.2 s → 8.2 s (−74 %); collapse consolidated ~250 of ~290 chunks.
• Full managed-orchestrator A/B (e2b API → orchestrator → Firecracker, use-nfs-for-snapshots=false to force GCS-cold): CTRL 1005 frames / 2010 MB / 22.6 s → COLL 392 frames / 784 MB / 9.5 s — −61 % faults, −61 % bytes, −58 % latency, with 385 chunks consolidated in 428 ms at pause. Metrics + span confirmed in Mimir/Tempo end-to-end (service_version 0.2.0-…).

[cursor]

PR Summary

Medium Risk
Touches the pause/snapshot path and envd's live heap via madvise; rollout is flag- and version-gated and best-effort, but enabled runs add pause latency and alter captured memory layout.

Overview
Adds a best-effort pre-pause step so envd compacts its own scattered Go heap into 2 MiB transparent hugepages, reducing distinct guest frames that cold resume must fault from remote storage. envd 0.6.4 exposes POST /collapse, which walks anonymous read-write mappings and applies per-2 MiB MADV_HUGEPAGE / MADV_COLLAPSE, returning CollapseResult stats. The orchestrator calls it from the existing pre-pause reclaim path when LaunchDarkly collapse-envd-heap is on (default off), gated on envd ≥ 0.6.4, with a 10s timeout and failures never blocking pause; metrics and spans record collapse duration and chunk outcomes. resume-build gains a -collapse flag to force the feature in dev benches.

^{Reviewed by Cursor Bugbot for commit d58d318. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 27.72277% with 146 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/orchestrator/pkg/sandbox/reclaim.go	0.00%	58 Missing ⚠️
packages/envd/internal/api/collapse.go	0.00%	36 Missing ⚠️
packages/orchestrator/pkg/sandbox/envd.go	0.00%	34 Missing ⚠️
...es/envd/internal/services/memory/collapse_linux.go	80.00%	8 Missing and 6 partials ⚠️
packages/orchestrator/cmd/resume-build/main.go	0.00%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[gemini-code-assist]

Code Review

The fields of envd.CollapseResult (such as Collapsed, Skipped, Regions, and Chunks) are generated as pointers because they are optional in the OpenAPI schema. Accessing them directly in bestEffortCollapse will cause compile-time errors or potential nil-pointer dereference panics at runtime, so they should be safely dereferenced using a helper like utils.DerefOrDefault before use.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The fields of envd.CollapseResult are generated as pointers because they are optional in the OpenAPI schema, which causes compile-time errors when converting them directly to integers or passing them to logging and tracing functions. Safely dereference these fields using utils.DerefOrDefault to prevent compilation failures and potential nil-pointer dereference panics.

	collapsed := utils.DerefOrDefault(stats.Collapsed, 0)
	skipped := utils.DerefOrDefault(stats.Skipped, 0)
	regions := utils.DerefOrDefault(stats.Regions, 0)
	chunks := utils.DerefOrDefault(stats.Chunks, 0)

	envdCollapseChunks.Add(ctx, int64(collapsed), metric.WithAttributes(attribute.String("result", "collapsed")))
	envdCollapseChunks.Add(ctx, int64(skipped), metric.WithAttributes(attribute.String("result", "skipped")))
	span.SetAttributes(
		attribute.Int("collapse.regions", regions),
		attribute.Int("collapse.chunks", chunks),
		attribute.Int("collapse.collapsed", collapsed),
		attribute.Int("collapse.skipped", skipped),
	)

	logger.L().Info(ctx, "envd heap collapsed",
		logger.WithSandboxID(s.Runtime.SandboxID),
		zap.Int("regions", regions),
		zap.Int("chunks", chunks),
		zap.Int("collapsed", collapsed),
		zap.Int("skipped", skipped),
		zap.Int64("duration_ms", elapsedMs),
	)

[kalyazin]

This is a false positive here. reclaim.go consumes the orchestrator's generated client (packages/orchestrator/pkg/sandbox/envd/envd.gen.go), whose cfg.yaml sets prefer-skip-optional-pointer: true — so CollapseResult fields are plain values (Chunks int, Collapsed int, ElapsedMs int64, …), not pointers. int64(stats.Collapsed) compiles, and utils.DerefOrDefault(stats.Collapsed, 0) would actually fail to compile (can't deref a non-pointer). Confirmed by the passing orchestrator build/lint/ARM64 checks. (The pointer-typed CollapseResult is the envd-side generated type, which is not used on this path.)

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d58d318. Configure here.}

[cursor]

Collapse ignores request cancellation

Medium Severity

PostCollapse runs memory.CollapseSelf() without honoring r.Context(). When the orchestrator’s 10s POST /collapse times out or the client disconnects, envd can keep issuing MADV_COLLAPSE while pause continues, unlike /freeze which uses the request context for its lock.

Additional Locations (1)

• packages/envd/internal/services/memory/collapse_linux.go#L29-L52

 

^{Reviewed by Cursor Bugbot for commit d58d318. Configure here.}