Performance: add opt-in profiler and reproducible benchmarks with an explicit data collection contract

[Eric-Song-Nop]

Goal

Establish an opt-in profiler and reproducible benchmark baseline before optimizing quickjs-oxide. The collected data should tell us where memory is retained, where allocations occur, how much initialization costs, and which operations are slow relative to the pinned QuickJS 2026-06-04 reference and previous Oxide revisions.

This issue tracks the observability foundation and its data contract. The interfaces below are proposed, not implemented functionality.

Data we will collect

Area	Planned data	Measurement boundary
Runtime memory snapshots (-d / --dump)	Counts for objects, shapes/properties, atoms, strings, arrays/elements, functions/bytecode and backing stores; owned bytes and buffer capacity where measurable; runtime allocator live allocation count/bytes where supported.	Record the snapshot phase, runtime identity, resource coverage and accounting basis. Deduplicate shared storage. Keep logical counts, requested bytes, capacity and usable bytes separate.
Allocation events (-T / --trace)	Allocation / reallocation / free events, event sequence, allocation identity, requested size, result and failures where observable; previous/new identity and size for reallocations where available.	Trace from runtime creation through teardown. State exactly which allocation paths are covered. Mark partial coverage and dropped events/output failures; object creation counts are not allocator events.
Runtime / Context lifecycle timing (-q -d)	Separate durations for Runtime creation, Context creation, Context destruction and Runtime destruction, with iteration count and raw repeated samples.	Identify timer and units explicitly (CPU time vs monotonic wall time). Report summaries with their aggregation rule; keep process startup separate from API lifecycle cost.
Operation benchmarks	Per-case elapsed time, iteration/operation count, ns/op, repeated independent run results, minimum and median with variation, and per-case ratios against a recorded baseline. Start with loop, property access, array, call and numeric workloads from the pinned tests/microbench.js.	Preserve benchmark bodies; adapt only host facilities as needed. Measure cold start, parse/compile and steady-state execution separately where the harness supports that separation. Record successful execution/result checks, failures and unsupported cases.
Reproduction metadata	Report schema version, engine version/commit, reference release, workload/harness hashes, CPU/architecture, OS, pointer width, Rust/C compiler versions, build profile/optimization flags, enabled features, profiler mode, timer, warm-up and repetition policy.	Save alongside every result so comparisons use matching workloads, environments and measurement definitions.

Accounting rules

• Start with reliable existing HeapCounts and GcStats capabilities. GC node statistics describe a collection; they do not establish complete byte accounting.
• Represent unavailable metrics as null / unavailable, with a reason. Never substitute zero or present node_count * size_of as total memory usage.
• Label estimates and resource ownership rules. Current live memory, peak memory, cumulative allocation and process RSS are separate quantities; peak/RSS collection can be added later with an explicit source.
• Preserve the project's unsafe_code = "forbid" policy. Assess safe allocation instrumentation coverage before committing to full allocator interception; use coverage=partial when appropriate.
• Snapshots must not execute getters/Proxy traps, advance jobs, add persistent roots or implicitly force GC. Explicit post-GC experiments must be identified separately.
• This first scope covers memory/allocation observation and timing benchmarks. Integrated CPU/call-stack sampling and JavaScript function execution tracing are future extensions; allocation trace must not be described as either.

Collection and benchmark workflow

• Add an optional profiling feature, forwarded by the CLI, with collection disabled by default. A build without support should explain unsupported profiling flags.
• Keep report formatting and I/O outside core resource accounting. Send diagnostics to stderr or a dedicated file, keeping script stdout usable.
• Produce a human-readable summary and versioned machine-readable results; retain raw benchmark samples and reference output.
• Use the same pinned workload and equivalent harness for both engines. Save any host-adaptation patch and its hash; minimal harness support does not imply complete QuickJS std/os compatibility.
• Run timing benchmarks with dump/trace disabled, on a fixed machine, serially with alternating engine order. Do not mix CPU-time and wall-time results into speedup ratios.
• Record unsupported/failed workloads explicitly and exclude them from performance aggregates. Keep Test262 correctness results separate from performance scores.
• Measure instrumentation overhead separately for four modes: feature absent, feature compiled but inactive, snapshot enabled, and trace enabled. Publish the measurements rather than assuming zero overhead.

Deliverables / acceptance

• Define the report schema, units, snapshot/exit phases, coverage labels and unavailable-field behavior.
• Implement opt-in memory snapshots and four-stage lifecycle timing using reliable counts first, then documented byte accounting.
• Document allocation-path coverage and implement trace within that boundary, including realloc failure and runtime teardown handling.
• Add a reproducible dual-engine benchmark runner with fixed inputs, raw samples, metadata and per-case comparison reports.
• Capture an initial baseline and a short report identifying observed bottlenecks and measurement limitations, without assuming a target speedup.
• Verify shared-resource deduplication and relevant GC/jobs/Context lifecycle boundaries; reporting failures must preserve JavaScript results and original errors.
• Verify inactive collection produces no diagnostic files/output or event-buffer growth, and report overhead for each mode.
• Run relevant Rust and QuickJS comparison tests plus the full Test262 correctness gate with no new result differences.

[Eric-Song-Nop]

本 issue 的结果总结，实现和报告位于 PR #5。 / Results for this issue; implementation and reports are in PR #5.

Benchmark 与 profiling 结果 / Benchmark and profiling results — 2026-09-09

已完成 benchmark 基线、工作负载隔离、CPU 采样和定向计数。主要优化候选是属性名重复 intern、普通对象提前执行数组键处理、通用数值转换、引用清理检查和 Context 布局初始化。本轮尚未实施或验证算法优化。

The benchmark baseline, workload isolation, CPU sampling and targeted counters are complete. Main optimization candidates are repeated property-key interning, array-key work on ordinary objects, generic numeric conversion, reference-cleanup checks and Context layout initialization. No algorithm optimization has been implemented or measured yet.

1. 测量环境 / Measurement environment

所有构建与测量在同一 PocketLab 机器串行执行：Ryzen 7 7840HS、Linux 6.18.44-1-lts、Rust 1.94.1 release 构建，对照 QuickJS 2026-06-04；外部 benchmark 固定为 ahaoboy/js-engine-benchmark@2034d98。双引擎 benchmark 每项各运行三次，交替引擎顺序，关闭诊断。不同实验的准确源码版本、构建参数与二进制哈希见报告；没有声称锁定 CPU 频率或完全隔离系统。

Builds and measurements ran serially on the same PocketLab machine: Ryzen 7 7840HS, Linux 6.18.44-1-lts, Rust 1.94.1 release builds, against QuickJS 2026-06-04, using ahaoboy/js-engine-benchmark@2034d98. Dual-engine benchmarks used three repetitions per case/engine, alternating engine order, with diagnostics disabled. Reports identify the distinct source revisions, build flags and binary hashes for each experiment. CPU frequency and system isolation were not controlled.

2. Benchmark 基线 / Benchmark baseline

同计时器微基准 / Matched-clock microbench

项目 / Case	Oxide ns/op	QuickJS ns/op	耗时比 / Time ratio
empty_loop	200	10	20×
prop_read	1,000	10	100×
array_read	500	10	50×
func_call	1,000	25	40×
int_arith	400	10	40×

30 次运行全部成功。表格是三次 harness 最小 ns/op 估计的中位数；两引擎均验证使用 Date.now，保留原函数体。毫秒时钟和自校准使结果量化，不能当作精确单次延迟；三次相同不代表零波动。早期混用时钟的实验已排除。

All 30 runs passed. Values are medians of three harness-reported minimum ns/op estimates; both engines verified the Date.now fallback with unchanged function bodies. Millisecond timing/calibration quantizes these estimates: they are not precise individual-operation latency, and identical triplicates do not imply zero variance. An earlier mixed-clock experiment was excluded.

固定工作量对照 / Fixed-workload comparison

工作量 / Workload	Oxide ms	QuickJS ms	耗时比 / Time ratio
空循环 1,000 万次 / 10 million empty-loop iterations	1,893.948	42.181	44.9×
属性读取 2,000 万次 / 20 million property reads	10,999.540	122.887	89.5×

这是后续实验，去掉自校准 harness，执行相同固定函数体。数字为三次完整进程墙钟时间的中位数，包含启动、编译和收尾，不能与上表混为同一次 ns/op 测量。

This follow-up removes the adaptive harness and executes identical fixed function bodies. Values are median whole-process wall times over three runs, including startup, compilation and teardown; they are a separate experiment from the ns/op table.

V8 v7 独立子测试 / Isolated V8 v7 suites

Suite	Oxide 中位分数 / Median score	QuickJS 中位分数 / Median score
Richards	15	1,352
DeltaBlue	20.9	1,236
RayTrace	34.8	2,829
Splay	126	4,958
Crypto	3/3 超时 / timeouts (90 s)	1,549
Earley-Boyer	3/3 超时 / timeouts (90 s)	3,439
RegExp	3/3 超时 / timeouts (90 s)	627
Navier-Stokes	3/3 超时 / timeouts (90 s)	3,108

分数越高越好。Oxide 的四项成功子测试、QuickJS 的所有子测试均成功 3/3 次。不生成缺项总分，不从超时推断精确倍率、死循环或语法不支持。本轮没有对四个超时子测试采样。

Higher scores are better. The four successful Oxide suites and every QuickJS suite completed 3/3 runs. No incomplete aggregate score is reported; timeouts do not establish a precise ratio, infinite loop or unsupported syntax. The four timed-out suites were not CPU-profiled in this follow-up.

3. 内存、生命周期和诊断开销 / Memory, lifecycle and diagnostic overhead

指标 / Metric	结果 / Result
诊断模式完整进程中位时间，每模式 11 次 / Whole-process median, 11 runs per mode	plain 14.580 ms; compiled-off 14.573 ms; dump 15.688 ms; trace 14.604 ms
快照 / Snapshot	1,171 objects; 65 shapes; 383 table-backed atoms; 1,293 live arena slots
Arena 内联存储 / Arena inline storage	used 1,171,584 B; capacity 1,851,392 B
Arena trace	11 events: 1 allocation, 9 capacity growths, 1 release; 0 dropped
生命周期中位时间，100 次 / Lifecycle medians, 100 iterations	Runtime create 5.711 µs; Context create 2.394 ms; Context drop 61 ns; Runtime drop 184.648 µs

Dump 的观测中位时间高 7.6%，包括输出 I/O；其他模式的范围重叠，不能承诺零开销。内存数字只覆盖部分拥有的存储，排除嵌套分配与 allocator 开销；总内存、峰值和 RSS 不可用。Trace 是 Vec 存储变化，不是物理 malloc/realloc 追踪；Runtime 销毁后 arena 容量归零。Context drop 仅测量该 API 边界，保留的 realm 环在 Runtime drop 清理。QuickJS 生命周期使用不同类型时钟，未计算跨时钟倍率。

Dump's observed median was 7.6% higher, including output I/O; overlapping ranges for other modes do not establish zero overhead. Memory accounting covers only partial owned storage, excluding nested allocations and allocator overhead; total memory, peak and RSS are unavailable. Trace records Vec storage transitions, not physical malloc/realloc calls; arena capacity reached zero after Runtime destruction. Context drop measures only that API boundary; retained realm cycles are cleaned up during Runtime drop. No ratio is computed against QuickJS lifecycle measurements using a different clock type.

4. CPU 热点与计数 / CPU hotspots and counters

使用未加诊断探针的源码构建，release + frame pointers，perf record -e cycles:u -F 199 --call-graph fp，每工作负载三次；另用独立诊断构建采集计数和阶段时间。以下 inclusive 为包含子调用的周期权重占比，self 为函数自身占比；inclusive 会重叠，不可相加，也不是收益承诺。

CPU samples used release builds from source without diagnostic probes, with frame pointers and perf record -e cycles:u -F 199 --call-graph fp, three runs per workload. Separate diagnostic builds supplied counters/timers. Below, inclusive includes callees and self covers the leaf function, weighted by cycle period. Inclusive shares overlap, must not be added, and are not promised speedups.

1. 属性名重复处理 / Repeated property-key processing

   constant_property_key inclusive：固定属性读取 33.45%，Richards 24.44%。2,000 万次读取产生 20,000,006 次 constant-key 调用和 20,001,385 次 Atom intern。建议复用 Runtime 持有的固定键 Atom，保持生命周期和所有权正确。

   constant_property_key inclusive: 33.45% in fixed property reads, 24.44% in Richards. Twenty million reads caused 20,000,006 constant-key calls and 20,001,385 Atom-intern calls. Candidate: reuse runtime-owned constant-key Atoms with correct lifetime/ownership.

2. 非数组提前处理数组键 / Array-key work on non-arrays

   array_own_key 在 Richards 中 inclusive 13.10%。固定 Richards ×10 中，**1,013,477 / 1,032,270 次（98.18%）**接收非数组，但在类型检查之前已 intern "length"。建议先检查数组类型。

   array_own_key is 13.10% inclusive in Richards. In fixed Richards ×10, 1,013,477 / 1,032,270 calls (98.18%) receive non-arrays, but "length" is interned before checking the class. Candidate: move the array-class check earlier.

3. 引用与清理 / References and cleanup

   Richards 中 release_object_handle self 8.26%，deferred drain self 4.22%。固定属性读取产生 140,008,893 次 deferred-drain 入口调用，实际处理 24 个 deferred 操作。应检查临时句柄和无效清理入口，保留析构和重入语义；这不是对象分配数，也不是完整 GC 次数。

   Richards self time: release_object_handle 8.26%, deferred drain 4.22%. Fixed property reads caused 140,008,893 deferred-drain entries but only 24 processed deferred operations. Investigate temporary handles and unnecessary cleanup-entry work while preserving destruction/reentrancy semantics. These are not object-allocation or full-GC counts.

4. 数值循环 / Numeric loop

   固定空循环 self：binding read 17.69%、numeric update 16.14%、通用 to_primitive bridge 11.16%。1,000 万次迭代触发 30,000,002 次转换，全部输入已为 Number；30,000,004 次绑定读取全部是 Direct。建议数值与直接绑定快速路径，保持溢出、BigInt、对象转换及异常行为。

   Fixed empty-loop self time: binding reads 17.69%, numeric updates 16.14%, generic to_primitive bridge 11.16%. Ten million iterations caused 30,000,002 conversions, all already Number, and 30,000,004 binding reads, all Direct. Candidates: numeric/direct-binding fast paths preserving overflow, BigInt, object conversion and exceptions.

5. Context 初始化 / Context initialization

   生命周期采样中布局替换 inclusive 55.88%。每 Context 778 次布局替换、943 次 shape 查询、其中 158 次命中。建议评估内建属性布局批量构建和重复 shape/hash/边保留工作。

   Layout replacement is 55.88% inclusive in lifecycle samples. Each Context performs 778 layout replacements and 943 shape queries, with 158 hits. Candidates: bulk intrinsic-layout construction and reduced repeated shape/hash/edge-retention work.

补充：固定属性读取约构造 8,000 万个 UTF-16 迭代器，输入全部为平坦字符串，rope 为零；迭代器析构 self 7.97%。先减少重复 intern，再重新判断字符串迭代/哈希特化的必要性。

Additionally, fixed property reads constructed about 80 million UTF-16 iterators, all for flat strings, with zero rope inputs; iterator destruction was 7.97% self time. Re-measure after reducing redundant interning before deciding on further string-iteration/hash specialization.

原始 Richards harness 的粗阶段中位时间：Context 2.842 ms、编译 13.426 ms、执行 9,675.615 ms，主要成本在执行。执行边界仍包含 warmup、harness 和输出，不是纯 steady-state。原自校准空循环 harness 还带入了循环体本身没有的属性访问，因此增加固定工作量实验来减少干扰。

Original Richards harness coarse phase medians: Context 2.842 ms, compilation 13.426 ms, execution 9,675.615 ms: execution dominates. That boundary includes warmup, harness and output, not pure steady state. The adaptive empty-loop harness introduced property accesses absent from the loop body, motivating fixed-workload measurements.

5. 限制、验证和后续 / Limits, validation and next steps

• 最终采样 lost samples 为零，未知叶函数权重很低；但 Richards 40.99% 的周期权重达到 127 帧上限，近端热点可用，外层调用关系不完整。无效 DWARF 栈未用于归因。
  Final samples had zero lost samples and very low unresolved leaf weight; 40.99% of Richards cycle weight reached the 127-frame limit. Near-leaf hotspots are useful; outer ancestry is incomplete. Unreliable DWARF captures were excluded from attribution.
• 采样、计数与 benchmark 使用不同且可追溯的二进制；关闭探针仍可能改变代码生成，不能把构建间差异解释为优化收益。
  Sampling, counters and benchmarks use separately identified binaries. Even disabled probes can change code generation; cross-build differences are not optimization wins.
• 基线验证：2,993 Rust tests pass，1 项原有忽略；固定 Rust 1.88 strict Clippy、CLI/source gates、6 项 runner tests 通过。102,037 行 Test262 outcome 与 PR refactor: establish Cargo workspace and clarify project ownership #2 完全一致，零变化/零新增失败。冻结的完整文件 checksum gate 仍失败，原因是源码 fingerprint 改变报告头；没有静默更新基线。完整 Test262 在 e0b3ef3 执行，未在后续报告提交重跑。
  Baseline validation: 2,993 Rust tests passed, one existing ignored; pinned Rust 1.88 strict Clippy, CLI/source gates and six runner tests passed. All 102,037 Test262 outcome rows matched PR refactor: establish Cargo workspace and clarify project ownership #2, with zero changes/new failures. The frozen full-file checksum gate still failed because the source fingerprint changed the header; its baseline was not silently updated. Full Test262 ran at e0b3ef3, not at the later report commit.
• 建议顺序：数组类型提前检查 → 固定属性 Atom 复用 → 数值快速路径 → 清理检查 → Context 布局。每种机制独立修改，验证相关语义，关闭诊断重测，再采样。
  Suggested order: early array-class check → constant-key Atom reuse → numeric fast paths → cleanup checks → Context layouts. Change one mechanism at a time, validate affected semantics, benchmark without diagnostics, then re-profile.

6. 报告和证据 / Reports and evidence

• Benchmark 报告 / report · 全部测量 / measurements JSON
• CPU 热点报告 / CPU report · 采样与计数 / samples and counters JSON
• 诊断补丁与四张火焰图 / Diagnostic patches and four flame graphs

以上链接固定到报告提交 1654067。CPU 实验产品源码为 9ed39e2，baseline 报告分别记录较早实验版本；诊断补丁仅应用于独立源码副本。原始包保存在本地与远端工作区 target/，未作为本评论附件上传；公开跟踪的 JSON 可直接查看全部数值证据。

Links are pinned to report commit 1654067. CPU measurements used product source 9ed39e2; the baseline report identifies its earlier measured revisions. Diagnostic patches were applied only to a separate source copy. Raw archives remain in local and remote workspace target/ directories and are not attached here; tracked JSON provides the numerical evidence directly.

SHA-256:

• Baseline: 19120bc1f848dc1f7c17347de4efab5aba7c3d56e0246c864a48a4a302444e06
• CPU: 93ef6154096799dfeb8bd88159b73fd4e90fd3b9333d4d091ce411ff1d8e638a