Skip to content

feat(profiling): add runtime diagnostics and external benchmark baseline - #5

Draft
Eric-Song-Nop wants to merge 21 commits into
pocket-stack:mainfrom
Eric-Song-Nop:perf/profilor
Draft

feat(profiling): add runtime diagnostics and external benchmark baseline#5
Eric-Song-Nop wants to merge 21 commits into
pocket-stack:mainfrom
Eric-Song-Nop:perf/profilor

Conversation

@Eric-Song-Nop

Copy link
Copy Markdown

Adds opt-in memory snapshots, bounded arena backing-storage tracing, lifecycle timing, and reproducible external benchmark runners. CLI supports -d, -T, -q -d and JSONL output, with explicit partial accounting and unavailable values. External benchmark sources and generated bundles stay outside this repository.

Depends on #2 (refact/clean-up-structure, 9fa0e9d). This draft targets upstream main, so its current diff also includes the changes from #2. Keep this PR unmerged until #2 lands, then update this branch to leave only the profiler changes. Implements the profiling plan introduced in #2. See docs/profiling.md and docs/reports/profiler-baseline.md / .json for usage, exact revisions, hashes, all measurements and limitations.

Validation ran on eric-83am / PocketLab: 2,993 Rust tests pass, one existing ignored; pinned Rust 1.88 strict Clippy, CLI feature tests, source gates and six Python runner tests pass. All 102,037 Test262 outcome rows match PR #2; the frozen full-file checksum gate still fails because the source fingerprint changes the header. No baseline was silently updated.

All five selected matched-clock microbench cases pass. Four of eight isolated V8 v7 suites complete all three repetitions in Oxide; Crypto, Earley-Boyer, RegExp and Navier-Stokes time out at 90 seconds in every repetition. Native QuickJS completes all suites. No incomplete aggregate score is reported. The first mixed-clock microbench experiment is explicitly excluded. Profiler coverage is partial arena storage, not a full allocator or CPU sampler.

Raw evidence archive SHA-256: 19120bc1f848dc1f7c17347de4efab5aba7c3d56e0246c864a48a4a302444e06. Numerical evidence is tracked in the report JSON; the raw archive is retained in the local and remote workspace target directories.

@Eric-Song-Nop

Copy link
Copy Markdown
Author

本 PR 及关联 issue #4 的结果总结,补充后续 CPU 调查。 / Results for this PR and tracking issue #4, including the follow-up CPU investigation.

Benchmark 与 profiling 结果 / Benchmark and profiling results — 2026-09-09

已完成 benchmark 基线、工作负载隔离、CPU 采样和定向计数。主要优化候选是属性名重复 intern、普通对象提前执行数组键处理、通用数值转换、引用清理检查和 Context 布局初始化。本轮尚未实施或验证算法优化。

The benchmark baseline, workload isolation, CPU sampling and targeted counters are complete. Main optimization candidates are repeated property-key interning, array-key work on ordinary objects, generic numeric conversion, reference-cleanup checks and Context layout initialization. No algorithm optimization has been implemented or measured yet.

1. 测量环境 / Measurement environment

所有构建与测量在同一 PocketLab 机器串行执行:Ryzen 7 7840HS、Linux 6.18.44-1-lts、Rust 1.94.1 release 构建,对照 QuickJS 2026-06-04;外部 benchmark 固定为 ahaoboy/js-engine-benchmark@2034d98。双引擎 benchmark 每项各运行三次,交替引擎顺序,关闭诊断。不同实验的准确源码版本、构建参数与二进制哈希见报告;没有声称锁定 CPU 频率或完全隔离系统。

Builds and measurements ran serially on the same PocketLab machine: Ryzen 7 7840HS, Linux 6.18.44-1-lts, Rust 1.94.1 release builds, against QuickJS 2026-06-04, using ahaoboy/js-engine-benchmark@2034d98. Dual-engine benchmarks used three repetitions per case/engine, alternating engine order, with diagnostics disabled. Reports identify the distinct source revisions, build flags and binary hashes for each experiment. CPU frequency and system isolation were not controlled.

2. Benchmark 基线 / Benchmark baseline

同计时器微基准 / Matched-clock microbench

项目 / Case Oxide ns/op QuickJS ns/op 耗时比 / Time ratio
empty_loop 200 10 20×
prop_read 1,000 10 100×
array_read 500 10 50×
func_call 1,000 25 40×
int_arith 400 10 40×

30 次运行全部成功。表格是三次 harness 最小 ns/op 估计的中位数;两引擎均验证使用 Date.now,保留原函数体。毫秒时钟和自校准使结果量化,不能当作精确单次延迟;三次相同不代表零波动。早期混用时钟的实验已排除。

All 30 runs passed. Values are medians of three harness-reported minimum ns/op estimates; both engines verified the Date.now fallback with unchanged function bodies. Millisecond timing/calibration quantizes these estimates: they are not precise individual-operation latency, and identical triplicates do not imply zero variance. An earlier mixed-clock experiment was excluded.

固定工作量对照 / Fixed-workload comparison

工作量 / Workload Oxide ms QuickJS ms 耗时比 / Time ratio
空循环 1,000 万次 / 10 million empty-loop iterations 1,893.948 42.181 44.9×
属性读取 2,000 万次 / 20 million property reads 10,999.540 122.887 89.5×

这是后续实验,去掉自校准 harness,执行相同固定函数体。数字为三次完整进程墙钟时间的中位数,包含启动、编译和收尾,不能与上表混为同一次 ns/op 测量。

This follow-up removes the adaptive harness and executes identical fixed function bodies. Values are median whole-process wall times over three runs, including startup, compilation and teardown; they are a separate experiment from the ns/op table.

V8 v7 独立子测试 / Isolated V8 v7 suites

Suite Oxide 中位分数 / Median score QuickJS 中位分数 / Median score
Richards 15 1,352
DeltaBlue 20.9 1,236
RayTrace 34.8 2,829
Splay 126 4,958
Crypto 3/3 超时 / timeouts (90 s) 1,549
Earley-Boyer 3/3 超时 / timeouts (90 s) 3,439
RegExp 3/3 超时 / timeouts (90 s) 627
Navier-Stokes 3/3 超时 / timeouts (90 s) 3,108

分数越高越好。Oxide 的四项成功子测试、QuickJS 的所有子测试均成功 3/3 次。不生成缺项总分,不从超时推断精确倍率、死循环或语法不支持。本轮没有对四个超时子测试采样。

Higher scores are better. The four successful Oxide suites and every QuickJS suite completed 3/3 runs. No incomplete aggregate score is reported; timeouts do not establish a precise ratio, infinite loop or unsupported syntax. The four timed-out suites were not CPU-profiled in this follow-up.

3. 内存、生命周期和诊断开销 / Memory, lifecycle and diagnostic overhead

指标 / Metric 结果 / Result
诊断模式完整进程中位时间,每模式 11 次 / Whole-process median, 11 runs per mode plain 14.580 ms; compiled-off 14.573 ms; dump 15.688 ms; trace 14.604 ms
快照 / Snapshot 1,171 objects; 65 shapes; 383 table-backed atoms; 1,293 live arena slots
Arena 内联存储 / Arena inline storage used 1,171,584 B; capacity 1,851,392 B
Arena trace 11 events: 1 allocation, 9 capacity growths, 1 release; 0 dropped
生命周期中位时间,100 次 / Lifecycle medians, 100 iterations Runtime create 5.711 µs; Context create 2.394 ms; Context drop 61 ns; Runtime drop 184.648 µs

Dump 的观测中位时间高 7.6%,包括输出 I/O;其他模式的范围重叠,不能承诺零开销。内存数字只覆盖部分拥有的存储,排除嵌套分配与 allocator 开销;总内存、峰值和 RSS 不可用。Trace 是 Vec 存储变化,不是物理 malloc/realloc 追踪;Runtime 销毁后 arena 容量归零。Context drop 仅测量该 API 边界,保留的 realm 环在 Runtime drop 清理。QuickJS 生命周期使用不同类型时钟,未计算跨时钟倍率。

Dump's observed median was 7.6% higher, including output I/O; overlapping ranges for other modes do not establish zero overhead. Memory accounting covers only partial owned storage, excluding nested allocations and allocator overhead; total memory, peak and RSS are unavailable. Trace records Vec storage transitions, not physical malloc/realloc calls; arena capacity reached zero after Runtime destruction. Context drop measures only that API boundary; retained realm cycles are cleaned up during Runtime drop. No ratio is computed against QuickJS lifecycle measurements using a different clock type.

4. CPU 热点与计数 / CPU hotspots and counters

使用未加诊断探针的源码构建,release + frame pointers,perf record -e cycles:u -F 199 --call-graph fp,每工作负载三次;另用独立诊断构建采集计数和阶段时间。以下 inclusive 为包含子调用的周期权重占比,self 为函数自身占比;inclusive 会重叠,不可相加,也不是收益承诺。

CPU samples used release builds from source without diagnostic probes, with frame pointers and perf record -e cycles:u -F 199 --call-graph fp, three runs per workload. Separate diagnostic builds supplied counters/timers. Below, inclusive includes callees and self covers the leaf function, weighted by cycle period. Inclusive shares overlap, must not be added, and are not promised speedups.

  1. 属性名重复处理 / Repeated property-key processing

    constant_property_key inclusive:固定属性读取 33.45%,Richards 24.44%。2,000 万次读取产生 20,000,006 次 constant-key 调用和 20,001,385 次 Atom intern。建议复用 Runtime 持有的固定键 Atom,保持生命周期和所有权正确。

    constant_property_key inclusive: 33.45% in fixed property reads, 24.44% in Richards. Twenty million reads caused 20,000,006 constant-key calls and 20,001,385 Atom-intern calls. Candidate: reuse runtime-owned constant-key Atoms with correct lifetime/ownership.

  2. 非数组提前处理数组键 / Array-key work on non-arrays

    array_own_key 在 Richards 中 inclusive 13.10%。固定 Richards ×10 中,**1,013,477 / 1,032,270 次(98.18%)**接收非数组,但在类型检查之前已 intern "length"。建议先检查数组类型。

    array_own_key is 13.10% inclusive in Richards. In fixed Richards ×10, 1,013,477 / 1,032,270 calls (98.18%) receive non-arrays, but "length" is interned before checking the class. Candidate: move the array-class check earlier.

  3. 引用与清理 / References and cleanup

    Richards 中 release_object_handle self 8.26%,deferred drain self 4.22%。固定属性读取产生 140,008,893 次 deferred-drain 入口调用,实际处理 24 个 deferred 操作。应检查临时句柄和无效清理入口,保留析构和重入语义;这不是对象分配数,也不是完整 GC 次数。

    Richards self time: release_object_handle 8.26%, deferred drain 4.22%. Fixed property reads caused 140,008,893 deferred-drain entries but only 24 processed deferred operations. Investigate temporary handles and unnecessary cleanup-entry work while preserving destruction/reentrancy semantics. These are not object-allocation or full-GC counts.

  4. 数值循环 / Numeric loop

    固定空循环 self:binding read 17.69%、numeric update 16.14%、通用 to_primitive bridge 11.16%。1,000 万次迭代触发 30,000,002 次转换,全部输入已为 Number;30,000,004 次绑定读取全部是 Direct。建议数值与直接绑定快速路径,保持溢出、BigInt、对象转换及异常行为。

    Fixed empty-loop self time: binding reads 17.69%, numeric updates 16.14%, generic to_primitive bridge 11.16%. Ten million iterations caused 30,000,002 conversions, all already Number, and 30,000,004 binding reads, all Direct. Candidates: numeric/direct-binding fast paths preserving overflow, BigInt, object conversion and exceptions.

  5. Context 初始化 / Context initialization

    生命周期采样中布局替换 inclusive 55.88%。每 Context 778 次布局替换、943 次 shape 查询、其中 158 次命中。建议评估内建属性布局批量构建和重复 shape/hash/边保留工作。

    Layout replacement is 55.88% inclusive in lifecycle samples. Each Context performs 778 layout replacements and 943 shape queries, with 158 hits. Candidates: bulk intrinsic-layout construction and reduced repeated shape/hash/edge-retention work.

补充:固定属性读取约构造 8,000 万个 UTF-16 迭代器,输入全部为平坦字符串,rope 为零;迭代器析构 self 7.97%。先减少重复 intern,再重新判断字符串迭代/哈希特化的必要性。

Additionally, fixed property reads constructed about 80 million UTF-16 iterators, all for flat strings, with zero rope inputs; iterator destruction was 7.97% self time. Re-measure after reducing redundant interning before deciding on further string-iteration/hash specialization.

原始 Richards harness 的粗阶段中位时间:Context 2.842 ms、编译 13.426 ms、执行 9,675.615 ms,主要成本在执行。执行边界仍包含 warmup、harness 和输出,不是纯 steady-state。原自校准空循环 harness 还带入了循环体本身没有的属性访问,因此增加固定工作量实验来减少干扰。

Original Richards harness coarse phase medians: Context 2.842 ms, compilation 13.426 ms, execution 9,675.615 ms: execution dominates. That boundary includes warmup, harness and output, not pure steady state. The adaptive empty-loop harness introduced property accesses absent from the loop body, motivating fixed-workload measurements.

5. 限制、验证和后续 / Limits, validation and next steps

  • 最终采样 lost samples 为零,未知叶函数权重很低;但 Richards 40.99% 的周期权重达到 127 帧上限,近端热点可用,外层调用关系不完整。无效 DWARF 栈未用于归因。
    Final samples had zero lost samples and very low unresolved leaf weight; 40.99% of Richards cycle weight reached the 127-frame limit. Near-leaf hotspots are useful; outer ancestry is incomplete. Unreliable DWARF captures were excluded from attribution.
  • 采样、计数与 benchmark 使用不同且可追溯的二进制;关闭探针仍可能改变代码生成,不能把构建间差异解释为优化收益。
    Sampling, counters and benchmarks use separately identified binaries. Even disabled probes can change code generation; cross-build differences are not optimization wins.
  • 基线验证:2,993 Rust tests pass,1 项原有忽略;固定 Rust 1.88 strict Clippy、CLI/source gates、6 项 runner tests 通过。102,037 行 Test262 outcome 与 PR refactor: establish Cargo workspace and clarify project ownership #2 完全一致,零变化/零新增失败。冻结的完整文件 checksum gate 仍失败,原因是源码 fingerprint 改变报告头;没有静默更新基线。完整 Test262 在 e0b3ef3 执行,未在后续报告提交重跑。
    Baseline validation: 2,993 Rust tests passed, one existing ignored; pinned Rust 1.88 strict Clippy, CLI/source gates and six runner tests passed. All 102,037 Test262 outcome rows matched PR refactor: establish Cargo workspace and clarify project ownership #2, with zero changes/new failures. The frozen full-file checksum gate still failed because the source fingerprint changed the header; its baseline was not silently updated. Full Test262 ran at e0b3ef3, not at the later report commit.
  • 建议顺序:数组类型提前检查 → 固定属性 Atom 复用 → 数值快速路径 → 清理检查 → Context 布局。每种机制独立修改,验证相关语义,关闭诊断重测,再采样。
    Suggested order: early array-class check → constant-key Atom reuse → numeric fast paths → cleanup checks → Context layouts. Change one mechanism at a time, validate affected semantics, benchmark without diagnostics, then re-profile.

6. 报告和证据 / Reports and evidence

以上链接固定到报告提交 1654067。CPU 实验产品源码为 9ed39e2,baseline 报告分别记录较早实验版本;诊断补丁仅应用于独立源码副本。原始包保存在本地与远端工作区 target/,未作为本评论附件上传;公开跟踪的 JSON 可直接查看全部数值证据。

Links are pinned to report commit 1654067. CPU measurements used product source 9ed39e2; the baseline report identifies its earlier measured revisions. Diagnostic patches were applied only to a separate source copy. Raw archives remain in local and remote workspace target/ directories and are not attached here; tracked JSON provides the numerical evidence directly.

SHA-256:

  • Baseline: 19120bc1f848dc1f7c17347de4efab5aba7c3d56e0246c864a48a4a302444e06
  • CPU: 93ef6154096799dfeb8bd88159b73fd4e90fd3b9333d4d091ce411ff1d8e638a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant