test: surfpool mainnet-fork integration for built-in IDL fuzz path by shahan-khatchadourian-anchorage · Pull Request #202 · anchorageoss/visualsign-parser

shahan-khatchadourian-anchorage · 2026-03-12T22:43:55Z

Summary

Adds a solana_test_utils workspace crate exposing SurfpoolManager/SurfpoolConfig, which spawns a surfpool subprocess (--no-tui), polls its RPC until ready, and kills it on Drop.
Adds two integration tests in chain_parsers/visualsign-solana/tests/surfpool_fuzz.rs (both #[ignore], network-bound):
- surfpool_lifecycle — starts surfpool against a mainnet fork and verifies the RPC responds.
- surfpool_jupiter_swap_roundtrip — loads an Anchor IDL from IDL_FILE, builds a synthetic transaction whose data starts with the first instruction's 8-byte discriminator, and asserts the parser returns Ok with a non-empty payload.
Adds an idl-meta Rust workspace tool (src/tools/idl-meta/) that replaces the previous inline python3 snippets in scripts/fuzz_all_idls.sh (locate-idls, counts <file>).
Adds scripts/surfpool_fuzz_all_idls.sh — runs the surfpool_jupiter_swap_roundtrip test once per embedded IDL via IDL_FILE=… and prints a per-IDL pass/fail table; on failure echoes the captured cargo output so CI logs show the diagnostic.
Adds .github/workflows/surfpool-solana.yml — label-gated CI (surfpool), skips fork PRs (secrets unavailable), downloads the SHA256-pinned upstream prebuilt surfpool binary (~5s vs ~10min compile), runs surfpool_fuzz_all_idls.sh against all embedded IDLs, and tags the PR with surfpool-failure on a red run.

Helius RPC

SurfpoolConfig::default() resolves the upstream fork URL from env in this order:

HELIUS_API_KEY — wrapped into https://mainnet.helius-rpc.com/?api-key=$KEY
SOLANA_RPC_URL — used as-is
https://api.mainnet-beta.solana.com — fallback (rate-limited)

The repo's HELIUS_API_KEY secret is plumbed into the CI job env automatically; local runs need the env var set.

Test plan

Add the surfpool label to this PR to trigger CI; expect 13 passed, 0 failed in the per-IDL table.
Local: HELIUS_API_KEY=<key> PROPTEST_CASES=32 ./scripts/surfpool_fuzz_all_idls.sh.
Local requires the surfpool binary on PATH. Easiest is to grab the prebuilt:
```
curl -fsSL https://github.com/txtx/surfpool/releases/download/v1.1.1/surfpool-linux-x64.tar.gz \
  | tar xz -C ~/.local/bin
```
Or compile from source: cargo install --git https://github.com/txtx/surfpool --rev d58df4cd7c3188561e671d8da0401487e3a60762 --locked.

🤖 Generated with Claude Code

Copilot

Pull request overview

Adds a new solana_test_utils workspace crate and a surfpool-backed property test to exercise the built-in IDL lookup path end-to-end against real mainnet program IDs (via simulateTransaction on a surfpool mainnet fork).

Changes:

Introduces solana_test_utils crate with shared proptest IDL byte strategies (idl_strategies) and a SurfpoolManager/SurfpoolConfig for managing a surfpool subprocess in tests.
Adds surfpool_real_idl_roundtrip (ignored, networked) integration property test and a helper script to run it across all embedded IDLs.
Adds/commits a set of Anchor IDL JSON files used by the surfpool fuzz script, and updates workspace/test dependencies accordingly.

Reviewed changes

Copilot reviewed 25 out of 31 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/Cargo.toml	Adds `solana_test_utils` to the workspace members.
src/solana_test_utils/Cargo.toml	New crate manifest (surfpool + proptest + solana deps).
src/solana_test_utils/src/lib.rs	Exposes `idl_strategies` and `surfpool` modules.
src/solana_test_utils/src/idl_strategies/mod.rs	Re-exports shared IDL byte strategies.
src/solana_test_utils/src/idl_strategies/bytes.rs	Implements borsh-correct proptest byte generation for IDL types/instructions.
src/solana_test_utils/src/surfpool/mod.rs	Wires up `SurfpoolConfig`/`SurfpoolManager`.
src/solana_test_utils/src/surfpool/config.rs	Default config for surfpool fork URL + RPC port selection.
src/solana_test_utils/src/surfpool/manager.rs	Spawns/kills surfpool and provides readiness polling + `RpcClient`.
src/solana_test_utils/tests/surfpool_fuzz.rs	New ignored Tokio property test that simulates + parses real-program IDL instructions.
src/solana_test_utils/.gitignore	Attempts to ignore surfpool runtime state output.
src/solana_test_utils/idls/ape_pro.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/cndy.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/drift.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/jupiter.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/jupiter_agg_v6.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/jupiter_limit.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/kamino.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/lifinity.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/meteora.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/openbook.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/orca.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/raydium.json	Committed IDL fixture used by surfpool fuzz runner.
src/solana_test_utils/idls/stabble.json	Committed IDL fixture used by surfpool fuzz runner.
src/chain_parsers/visualsign-solana/Cargo.toml	Adds `proptest` under dev-dependencies for the fuzz/integration tests.
src/chain_parsers/visualsign-solana/tests/fuzz_idl_parsing.rs	Adds/contains property-based crash-safety + valid-data IDL parsing tests.
src/chain_parsers/visualsign-solana/tests/pipeline_integration.rs	Adds/contains end-to-end pipeline tests and proptests for IDL visualization.
src/chain_parsers/visualsign-solana/tests/fuzz_idl_parsing.proptest-regressions	Stores proptest regression seed(s).
scripts/fuzz_all_idls.sh	Runner script for running real-IDL fuzz tests across IDLs.
scripts/surfpool_fuzz_all_idls.sh	New runner script for surfpool-backed real-program IDL fuzzing across IDLs.
PLAN.md	Adds a planning document for surfpool + IDL-fuzz integration approach.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

shahan-khatchadourian-anchorage · 2026-05-07T17:38:45Z

Security Review

No security vulnerabilities found above the reporting threshold.

Summary

Reviewed the PR diff covering: a new Solana surfpool integration test, a solana_test_utils workspace crate, an idl-meta Rust tool, two helper bash scripts, and a new GitHub Actions workflow that consumes the existing HELIUS_API_KEY repo secret.

Areas examined and dismissed:

Command injection (manager.rs:57) — Command::new("surfpool").args(&args) does not invoke a shell; args are typed/validated config from auto-selected ports and developer-controlled URL/log level. No untrusted input source.
Bash script injection — All variable expansions properly quoted; IDL_FILES is populated by glob over a trusted directory. Neither script echoes HELIUS_API_KEY nor uses set -x.
Path traversal (idl-meta) — Tool reads paths but is invoked only from a script that globs a trusted IDL directory; not exposed to untrusted input.
Secret exposure in CI logs — manager.rs:27 does info!("Starting Surfpool with config: {:?}", config) and SurfpoolConfig.rpc_url carries the full Helius URL including API key. However, no tracing-subscriber is initialized in the test binary, the workflow, or the script, so the macro is a no-op at runtime. GitHub Actions also auto-masks secrets.HELIUS_API_KEY in workflow logs. Below the HIGH-CONFIDENCE threshold; flagged informationally as a Debug-prints-secret footgun if a subscriber is later added.
GitHub Actions expression injection — Workflow if: and run: blocks don't interpolate any user-controllable string (PR title, body, branch name, head ref) into shell context. PR_NUMBER is passed via env: correctly.
pull_request_target — Workflow correctly uses plain pull_request and gates on github.event.pull_request.head.repo.full_name == github.repository, so HELIUS_API_KEY never reaches fork-PR code.
Cache poisoning — Fork PRs gated out cannot populate the cache; GitHub scopes cache keys by ref.
Untrusted code execution — Fork-PR gate prevents fork PRs from running with secrets. Residual internal-contributor abuse is the standard CI threat model and not PR-introduced.
Insecure deserialization — idl-meta uses only serde_json (safe), not serde_yaml/bincode on untrusted input.
Path injection in test — surfpool_fuzz.rs reads IDL_FILE from env (operator-controlled, test file).

Informational (not findings)

src/solana_test_utils/src/surfpool/manager.rs:27,55 — info!/debug! print SurfpoolConfig/args containing the Helius URL with API key via {:?}. Currently inert (no tracing subscriber, GitHub secret masking) but worth a custom Debug impl on SurfpoolConfig (or redact wrapper on rpc_url) as defense in depth.

🤖 Generated with Claude Code

shahan-khatchadourian-anchorage · 2026-05-07T19:48:13Z

@prasanna-anchorage ready for re-review

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Use `surfpool start` subcommand with correct flags (--port, --ws-port, --rpc-url, --no-tui, --ci) instead of obsolete top-level flags - Remove ledger_path/reset_ledger config (not surfpool concepts) - Use multi_thread tokio runtime in tests (required by solana-rpc-client) - All 15 IDLs pass surfpool integration tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Wires the existing `HELIUS_API_KEY` repo secret into a label-gated CI job that runs `scripts/surfpool_fuzz_all_idls.sh` against all 13 embedded IDLs. The Solana parser's `SurfpoolConfig::default()` already prefers `HELIUS_API_KEY` over the public mainnet endpoint, so passing the secret through the job env is sufficient -- no code changes required. Modelled on `fuzz-solana.yml` and `proptest-solana.yml`: label-gated to control RPC quota usage, caches `~/.cargo/bin/surfpool` to skip the ~minute-long install on subsequent runs, and tags the PR with `surfpool-failure` on a red run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Addresses code review on the prior commit: 1. Pin surfpool to tag v1.1.1 (matches the version verified locally against the Helius mainnet fork) and key the cached binary on the pinned version. The previous cache used `src/Cargo.lock` hash, which is decoupled from surfpool's upstream and would silently pin CI to the first-cached binary indefinitely. Splitting the binary cache from the workspace cache also lets each invalidate independently. 2. Skip the job on fork PRs (head repo != base repo). Secrets are not passed to fork PRs, so HELIUS_API_KEY would be empty and the labeling step would lack `gh pr edit` permissions; both would silently fall back to flaky public mainnet and a no-op label step. 3. Use `surfpool --version` instead of `command -v surfpool` to gate reinstall. A corrupted cached binary (interrupted previous install, ABI mismatch after toolchain bump) is now detected and rebuilt. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… install Three fixes addressing outstanding review comments and the failing CI: 1. CI install command -- the `surfpool` package was renamed to `surfpool-cli` upstream, so `cargo install surfpool --tag v1.1.1` fails with "could not find `surfpool` ... with version `*`". Drop the package name argument so cargo uses the workspace's `default-members` (`crates/cli`), which produces the `surfpool` binary regardless of future package renames within the workspace. 2. `wait_ready` no longer blocks a Tokio worker. The synchronous `RpcClient::get_version()` is now dispatched via `spawn_blocking` and the inter-attempt delay uses `tokio::time::sleep` instead of `thread::sleep`. Without this, a stalled RPC could starve other tasks on the same multi-thread runtime. 3. `surfpool_fuzz_all_idls.sh` now prints the captured cargo output on failure (both the no-summary and `N failed` paths). Previously a red IDL row printed only `FAIL (...)` and the diagnostics were silently dropped, making CI failures hard to debug. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tags are mutable -- a force-push could silently change what we install, and `--locked` only protects transitive deps (it uses the source's Cargo.lock; it does not lock the source commit). Pin to the SHA `v1.1.1` resolved to (d58df4c) so the install target is bit-for-bit reproducible. Cache key now keys on the rev. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The toolchain action exports `RUSTFLAGS: -D warnings` by default, which made the `cargo install` step fail on a benign `unused_parens` lint in upstream `surfpool-gql` v1.1.1. Deny-warnings is the right policy for our own code but wrong for installing a third-party binary at a pinned revision -- whether it compiles cleanly under future rustc is not something we want CI to gate on. Set `RUSTFLAGS=--cap-lints=warn` for just the install step so upstream warnings stay warnings without affecting any subsequent step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replaces the ~10-minute `cargo install` compile with a ~5-second curl of the upstream release asset. Pins the tarball's SHA256 in the workflow so a force-pushed release asset cannot silently change what we install. The binary is dropped into `~/.local/bin/` (added to `GITHUB_PATH` for subsequent steps). Cache is keyed on (version, sha256) so a checksum bump invalidates automatically. Drops `RUSTFLAGS=--cap-lints=warn` from the install step; that was only needed to work around upstream lint regressions during compile. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The workflow toggles a `surfpool-failure` label on PRs. With the prior `if: contains(labels, 'surfpool')` guard, that toggle re-fires the workflow on the same PR (since the `surfpool` label is still present when the `surfpool-failure` label is added). Any other `labeled` event on a PR that happens to carry the `surfpool` label would also re-trigger unnecessarily. Match the pattern used in `.github/workflows/stagex.yml`: on `labeled` events only run when the label being added is `surfpool`; on other events (`opened`, `synchronize`, `reopened`) require the label to already be present. Addresses copilot review comment on #253 (which proposed the same guard); folded here so #253 can be closed with the workflow's review comments resolved on this PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

prasanna-anchorage

Approving. Rebased onto current main (1 conflict in visualsign-solana/Cargo.toml — resolved by keeping main's solana_parser rev 2146929 and adding the new solana_test_utils dev-dep). Workspace cargo check, clippy --all-targets -D warnings, cargo fmt, and cargo test --no-run all green locally. Substantive feedback from earlier review rounds has been addressed (python3 → Rust idl-meta tool, spawn_blocking for blocking RPC, SHA256-pinned tarball install, fork-PR skip, etc.).

Follow-ups worth filing as separate issues (not blocking this merge):

surfpool_jupiter_swap_roundtrip starts surfpool but never queries it — surfpool is a side effect, not a participant. Either tighten the test to actually use the fork, or rename to reflect that this is IDL-fuzz with a surfpool smoke-start.
SurfpoolManager::airdrop returns Ok(signature) on Ok(Some(_status)) regardless of whether the inner Result is success or TransactionError. Currently unused in-tree but latent bug.
.surfpool/ should be added to the workspace-root .gitignore — Copilot's original concern is partly unaddressed: removing the per-crate .gitignore doesn't help when surfpool runs from the repo root.

Minor: Command::spawn inherits stdio (noisy in CI); scripts/fuzz_all_idls.sh invokes cargo run -p idl-meta per IDL inside the loop; idl-meta doesn't accept the --manifest-path=value form. None blocking.

Copilot AI review requested due to automatic review settings March 12, 2026 22:43

Copilot started reviewing on behalf of shahan-khatchadourian-anchorage March 12, 2026 22:44 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

shahan-khatchadourian-anchorage marked this pull request as draft March 13, 2026 01:13

prasanna-anchorage reviewed Mar 13, 2026

View reviewed changes

Comment thread scripts/fuzz_all_idls.sh Outdated

Comment thread scripts/fuzz_all_idls.sh Outdated

Comment thread scripts/fuzz_all_idls.sh

shahan-khatchadourian-anchorage force-pushed the shahankhatch/surfpool_testing branch from ea00e0a to 613c3b7 Compare April 14, 2026 18:37

shahan-khatchadourian-anchorage mentioned this pull request Apr 15, 2026

ci: add surfpool integration workflow #253

Closed

3 tasks

shahan-khatchadourian-anchorage added surfpool fuzz proptest labels May 7, 2026

shahan-khatchadourian-anchorage marked this pull request as ready for review May 7, 2026 19:47

shahan-khatchadourian-anchorage mentioned this pull request May 7, 2026

test(surfpool): native cargo runner via build.rs + per-IDL macro #285

Draft

2 tasks

shahan-khatchadourian-anchorage and others added 13 commits May 12, 2026 21:41

feat: add solana_test_utils crate with SurfpoolManager

2bf41d0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: add surfpool mainnet-fork integration test for Solana parser

0ba1bd3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: add idl-meta tool for Python-free IDL metadata extraction

a94ca57

refactor: replace python3 with idl-meta in fuzz_all_idls.sh

0cf5c93

test: add surfpool_fuzz_all_idls.sh for mainnet-fork IDL testing

6bff5ee

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

prasanna-anchorage force-pushed the shahankhatch/surfpool_testing branch from 3daa361 to b713cc7 Compare May 12, 2026 23:53

prasanna-anchorage approved these changes May 12, 2026

View reviewed changes

prasanna-anchorage merged commit 450dbb6 into main May 12, 2026
9 checks passed

prasanna-anchorage deleted the shahankhatch/surfpool_testing branch May 12, 2026 23:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: surfpool mainnet-fork integration for built-in IDL fuzz path#202

test: surfpool mainnet-fork integration for built-in IDL fuzz path#202
prasanna-anchorage merged 13 commits into
mainfrom
shahankhatch/surfpool_testing

shahan-khatchadourian-anchorage commented Mar 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shahan-khatchadourian-anchorage commented May 7, 2026

Uh oh!

shahan-khatchadourian-anchorage commented May 7, 2026

Uh oh!

prasanna-anchorage left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shahan-khatchadourian-anchorage commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Helius RPC

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shahan-khatchadourian-anchorage commented May 7, 2026

Security Review

Summary

Informational (not findings)

Uh oh!

shahan-khatchadourian-anchorage commented May 7, 2026

Uh oh!

prasanna-anchorage left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shahan-khatchadourian-anchorage commented Mar 12, 2026 •

edited

Loading