td: golden differential testing (Phase 2 framework)#134
Closed
jcelerier wants to merge 11 commits into
Closed
Conversation
TouchDesigner has no headless mode, but `TouchDesigner.exe <file>.toe` runs the project's Execute-DAT callbacks. Adds: - td_sweep_driver.py: paste into an Execute DAT (Frame Start on), Save As a driver .toe once. Waits a few frames (so Custom OPs have registered), runs the generated sweep runner, writes the report, quits. Runner/report paths come from env vars. - run_td_sweep.py: the unattended harness -- stages *_td.dll into the .toe's Plugins/, generates td_run_all.py from the dump JSON (instantiate + cook every operator), launches the driver .toe, auto-dismisses the modal "Plugin Load Error" dialog (BM_CLICK the OK button) that otherwise blocks startup, waits for td_test_report.json, and prints a pass/crash summary. An operator that crashes TD on instantiation yields no report / an early exit, which is flagged. - README.md: the one-time GUI setup + usage. The .toe itself must be created once in the GUI: TouchDesigner's binary project format can't be reliably synthesized by hand (a toecollapse'd Execute DAT's callbacks don't fire). After that the sweep is fully unattended. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…sume) Getting the sweep to actually instantiate operators surfaced several TD facts: - opType is sanitize_td_name(C_NAME) (the binding passes @AVND_C_NAME@), not the display name -- fixed the type derivation in gen_tester_patches. - TD's create() needs the OP's Python *class* (e.g. AvndadditionCHOP = opType + family), not the opType string; and a real COMP parent (root '/' won't hold leaf ops). The runner now resolves the class (trying families) via eval and creates inside a baseCOMP. It's fully inlined -- it's exec()'d in an Execute DAT callback frame, where nested defs / comprehensions can't see its locals. - TD loads Custom OP DLLs only from the top level of Documents/Derivative/ Plugins (not a .toe-adjacent Plugins/, not subfolders). The harness stages there, deduped by opType (a build emits Foo_CHOP_AUDIO + Foo_CHOP_MESSAGE, both the same opType), and removes exactly what it staged (retrying, as TD holds the handles briefly after kill). - Crash resilience: the runner flushes the report after each op and drops a breadcrumb (td_current.txt); if an op hard-crashes TD, the harness records it and relaunches, and the runner resumes past everyone already tested. First full run (110 objects): 61 ok, 45 "Not enough sources" (filters needing an input -- benign), 3 class-not-found, and 1 that crashes TouchDesigner (avnd_vb_fourses_tilde). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…rtifacts Commit the driver .toe (a single Execute DAT that runs AVND_TD_RUNNER on frame start and quits) so the one-time GUI step is optional -- run_td_sweep.py works against it directly. Ignore the per-run outputs (generated runner, report, breadcrumb, staged Plugins/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
… (Phase 2) Extends the TD sweep into a differential test against the golden oracle (PR #133). The runner now, per object: loads golden/<file>.json (from AVND_TD_GOLDEN_DIR), applies the recorded input control values to n.par, cooks, and captures the output CHOP channels into the report. run_td_sweep.py then diffs each captured output against the golden's output over their overlapping prefix (within --atol/--rtol) and prints match / MISMATCH / skip counts. Framework proven: audio generators are compared and 2 match the golden bit-for- bit on the overlap; the diff also correctly flags avnd_sines (TD cooks 1 sample vs the golden's 64, and the first sample differs -- a time-origin/rate question). Known coverage gaps (next slices, each a distinct TD mechanism): - control-only objects: their output is an Info-CHOP channel, not a CHOP output channel, so td_audio is empty -- need to read info channels to compare golden.outputs.controls (78 objects currently "no-golden-audio"). - audio filters: need the exact golden input signal fed in (Table DAT -> DAT to CHOP) before the comparison is meaningful (currently "skip-needs-input"). - generators: TD cooks 1 sample per frame; capturing the golden's N frames + aligning sample-rate/time-origin is needed for full-length comparison. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…o+controls
Big coverage jump for the Phase 2 differential (matches 3 -> 21):
- Input feeding so value-IO objects and filters actually cook:
* value inputs (golden controls that aren't TD params -> input channels):
a Constant CHOP, wired to the op's input.
* audio inputs (time-varying): a Script CHOP whose callback replays the golden
samples from a Table DAT (TD has no datToCHOP class).
- Output capture now keeps channel NAMES (td_out = [{name, samples}]): avnd
output *controls* surface as named output channels (e.g. 'Out'), not a separate
Info CHOP -- so control objects are comparable via compare_controls (match by
name), audio via compare_audio (positional), routed by which the golden has.
Result vs the golden oracle: 21 match, 12 MISMATCH flagged (real signals: e.g.
avnd_sines time-origin, avnd_modular/per_bus channel-count, sidechain 2ch vs 3ch,
all_ports_types uninitialized). Remaining tail: ~45 control objects still produce
no TD output (need input we don't yet supply) + unnamed-port name matching.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…fo CHOP Control outputs (avnd val_port outputs) surface as NAMED output channels once the object cooks (e.g. n['Out']); the runner now also captures the op's Info-CHOP channels (getInfoCHOPChan) as a fallback, and the harness matches golden output controls by name against either source. This dropped "no-td-controls" from 45 to 3 for control objects. Confirmed via standalone diagnostic: TestFloatSlider (param input + val_port output, 0 input connectors) produces channel 'Out'=<In> with no input wired. Remaining puzzle: in the full batch sweep those same objects yield 0 output channels (and no custom Info-CHOP channel), while value-port-input objects like val_float work in both -- a TD cook-context difference between standalone and the 100-object container run, still to debug (currently shows as no-name-match). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…l tail Diagnostic: param-input + val_port-output objects (TestFloatSlider, bargraph, enum, ...) report numChannels=0 in the full 100-plugin batch sweep -- so their output channel isn't produced -- even though output_channel_count_visitor is a structural constant 1, they work standalone (numChannels=1), and value-port-input objects fed a CHOP (val_float) get numChannels=1 in both. Root cause is a batch-vs-standalone cook-context difference in getOutputInfo, not opType collision (confirmed: no cross-object opType clashes) -- needs C++-level logging in the TD binding to resolve. Recorded dbg_nc/dbg_ns in the report to track it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
Attempt to make unfed value-IO objects cook in the batch: connect a downstream Null CHOP and cook it to pull the op's output, and record dbg_nc/dbg_ns. Finding (not yet resolved): param-input + val_port-output objects (float_slider, enum, bargraph) report numChannels=0 in the 100-plugin batch but 1 standalone; value-port-input objects fed a Constant CHOP (val_float) work in both. The Null CHOP pull did NOT force them to cook. C++ logging added to the binding's getOutputInfo was inconclusive -- the plugin's fopen writes standalone but not in the batch process (val_float, which works, also logged nothing), so it can't distinguish "getOutputInfo not called" from "called but returns 0". The batch cook scheduling for an unfed, output-less CHOP_MESSAGE differs from standalone; needs interactive/textport debugging in TD, not headless plugin-side logs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…oses tail) The control tail was NOT a cook-context mystery -- the harness was staging the wrong plugin. Objects built for both CHOP_AUDIO and CHOP_MESSAGE share one opType (sanitize(c_name)), so only one can register; the dedup's sorted() kept AUDIO, so control objects (val_port I/O, no audio ports) ran through audio_processor, whose getOutputInfo yields 0 output channels for them -- hence "0 channels in batch, 1 standalone" (standalone I'd hand-staged the MESSAGE variant). Fix: choose the variant from the dump -- audio objects (any audio port) get CHOP_AUDIO, everything else CHOP_MESSAGE. Found via a numChannels marker in getOutputInfo that never showed in the batch, proving a different dll was loaded (thanks to the "fopen can't differ in batch" hunch). Result: match 21 -> 47, no-name-match 49 -> 17. Control objects now produce their output channel (float_slider Out=In, bargraph, enum Index, val_float Out). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
The runner now records entry['applied'] (golden controls successfully set on
n.par) and entry['par_names'] (the op's parameters), which is what let me
classify the mismatches: CHOP_AUDIO objects expose NO custom params (applied={},
only built-in Common-page pars) while CHOP_MESSAGE objects do -- so audio
effects' controls (Grunge, Preamp, Gain) can't be set, hence their outputs
differ from the golden. parameter_setup iterates control_input_introspection<T>
identically for both processors, so the gap is in what that introspection yields
for audio objects -- a real binding issue to chase at C++ level.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
…puts Two runner fixes: - Single cook: the runner used to force-cook the target three times (n.cook, a Null-CHOP output pull, and the Info-CHOP read all with force=True), which advances any STATEFUL object (counters, oscillators, feedback) N times vs the golden's single process() pass. Now: one n.cook(force=True), read n's channels directly, and cook the Info CHOP WITHOUT force so it reuses that single cook. Verified via the op's total_cooks info channel: 3 -> 1. (The Null-CHOP pull was a leftover from the mis-diagnosed control tail, since fixed by staging the right CHOP variant, so it's removed.) - Skip 'unrecorded' golden values: pulse/aggregate inputs (impulse_button, colour) have no serialisable value; setting a TD param to the string 'unrecorded' made pulses fire spuriously. Skip them. test_impulse still mismatches (Count 3 vs 0) but for a different, now-isolated reason: with total_cooks=1 confirmed, its operator() still increments 3x per cook / the bang reads truthy while unset -- a pulse-representation issue, not the triple-cook. Overall differential unchanged at match 50. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011s7huWR2wFsLFiMJPjx1z2
This was referenced Jul 2, 2026
Member
Author
|
Superseded by #137, which combines all outstanding work into a single PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of output verification (
OUTPUT_VERIFICATION_PLAN.md): turns the TD sweep into a differential test against the golden oracle (#133). Depends on #131 (TD sweep harness) and #133 (golden generator, for thegolden/<name>.jsonfiles).What
The TD runner now, per object: loads
golden/<file>.json(fromAVND_TD_GOLDEN_DIR), applies the recorded input control values ton.par, cooks, and captures the output CHOP channels into the report.run_td_sweep.py --goldensthen diffs each captured output against the golden's output over their overlapping prefix (within--atol/--rtol) and prints match / MISMATCH / skip counts.Proven
Audio generators are compared and 2 match the golden on the overlap; the diff also correctly flags
avnd_sines(TD cooks 1 sample vs the golden's 64, and the first sample differs — a time-origin/rate question). The whole machinery — golden load, input apply, output capture, tolerance diff, reporting — is in place and working.Documented coverage gaps (next slices, each a distinct TD mechanism)
getInfoCHOPChan), not output CHOP channels — reading them from Python needs work (a plain Info CHOP returns only generic op info: cook stats, sample rate).The framework is the foundation; these expand coverage.
🤖 Generated with Claude Code