test: add a hosted-runner e2e path alongside the self-hosted one - #689
test: add a hosted-runner e2e path alongside the self-hosted one#689mdozhdev wants to merge 10 commits into
Conversation
e2e-tests-local needs Docker and the iOS Simulator on one machine, which GitHub-hosted macOS runners cannot provide, so it runs on a self-hosted Mac as a single shard at ~2h per run. Add the same suite on macos-latest with the regtest stack on an ubuntu runner, reached over Tailscale. Both jobs reuse build-local, so the app is built once. e2e-tests-local is untouched and e2e-status still gates only on it, so the new job reports without blocking until it has earned replacing it. regtest-stack and e2e-tests-remote must not depend on each other: the stack job only finishes once the tests are done, so a dependency either way deadlocks. Requires secrets.TS_AUTHKEY and the suite fixes in synonymdev/bitkit-e2e-tests#207. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lets the new jobs be exercised on this draft without marking it ready and without occupying the self-hosted Mac, which e2e-tests-local would. Revert before merging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ated Reverts the draft-guard change, which did not work: detect-changes, build-local and e2e-branch are all draft-guarded too, so the new jobs skipped along with their dependencies. Instead park e2e-tests-local, so this PR can be marked ready and exercise e2e-tests-remote without tying up the shared self-hosted Mac for hours. e2e-status treats a skipped shard as a failure, so it will report red until this is restored. Revert before merging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Greptile SummaryThe PR adds a hosted macOS E2E path that connects over Tailscale to a regtest stack on an Ubuntu runner, while temporarily parking the self-hosted path. The stack lifetime can end before the remote test job’s configured timeout.
Confidence Score: 4/5The workflow should not merge until the regtest stack is guaranteed to remain available for the full remote-test lifetime; the mutable Tailscale action reference is also worth hardening. The stack job can exit at 150 minutes while the remote job remains active for up to 180 minutes, destroying the backend runner and breaking remaining tests. Files Needing Attention: .github/workflows/e2e-tests.yml
|
| Filename | Overview |
|---|---|
| .github/workflows/e2e-tests.yml | Adds the cross-runner E2E workflow, but its 150-minute stack hold can terminate before the 180-minute remote test job and its Tailscale action is not immutably pinned. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Build[build-local] --> Stack[regtest-stack on Ubuntu]
Build --> Tests[e2e-tests-remote on macOS]
Branch[e2e-branch] --> Stack
Branch --> Tests
Stack -->|Tailscale services and LND credentials| Tests
Hold[150-minute hold loop] --> Stack
Tests -->|up to 180 minutes| Result[Remote E2E result]
Reviews (1): Last reviewed commit: "test: TEMPORARY - park e2e-tests-local w..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| deadline=$(( SECONDS + 9000 )) | ||
| while (( SECONDS < deadline )); do |
There was a problem hiding this comment.
When the remote E2E job runs longer than 150 minutes, this deadline ends the hold loop while the remote job can continue for up to 180 minutes. The Ubuntu runner is then destroyed along with the regtest services and Tailscale connection, causing the remaining tests to lose access to LND, Electrum, and bitcoind.
| - uses: tailscale/github-action@v3 | ||
| with: | ||
| authkey: ${{ secrets.TS_AUTHKEY }} | ||
| hostname: regtest-${{ github.run_id }} |
There was a problem hiding this comment.
The new Tailscale step passes TS_AUTHKEY to an action referenced by the mutable v3 tag, so repointing or compromising that tag would expose the tailnet credential to unreviewed action code. Pinning the action to a reviewed commit would make the executed code immutable. How this was verified: Both new jobs provide secrets.TS_AUTHKEY directly to tailscale/github-action@v3.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
The hold loop was sized for a single-spec run and gave up after 150 minutes. The full shard with three attempts takes about three hours, so the stack was torn down 32 seconds before the first of seven "connect ETIMEDOUT ...:43782" errors in attempt 3, and the job still reported success. Raise the deadline and both job timeouts past the worst realistic case, and fail loudly if the stack is ever torn down while tests are running. Also drop @hardware_wallet from the remote shard: ensureTrezorEmulator() shells out to docker compose on the machine running the tests, which has no Docker here. Tracked with the trezor-emu job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Un-parks e2e-tests-local so both paths run on the same commit, and keeps artifacts from passing runs, so the self-hosted screenshot can be compared against the remote one. @send_2 fails on the remote path with 8998 available where the test needs more than 10001, and reproduces exactly across attempts. The reserve on a 100k channel should be identical on both paths, so the self-hosted figure is needed to explain the difference rather than guess at it. Revert both before merging: e2e-tests-local should stay as it is, but the artifact steps go back to failure(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adopts the approach from #692: rather than teaching every call site the stack address, relay its ports onto 127.0.0.1 on the Mac. The suite, the specs and the app all keep reaching 127.0.0.1, so no address plumbing is needed at all — every LND_HOST/ELECTRUM_HOST/E2E_LOCAL_HOST override is gone, and the credentials are written where lndConfig looks by default. Readiness is checked end to end rather than with a port probe: the relay accepts before it has dialled anything, so nc would pass with the stack unreachable. The self-hosted Mac is unresponsive, so e2e-tests-local is parked and e2e-status now reads the remote path. Both revert before merging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stack runner starts the emulator alongside regtest and the relay carries its bridge and controller ports, so the Mac drives it with TREZOR_REMOTE=1 instead of needing a Docker daemon of its own. Also restores e2e-tests-local and the e2e-status gate, which were parked while the remote path was validated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The trezor step was inserted between `docker compose up` and the waits that follow it, so those waits ran without the docker working directory and never found the lnd macaroon. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The controller imports trezorlib lazily in get-address, so seeding the emulator succeeded and deriving its address then failed on the missing module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Matches the self-hosted job; always() was for comparing the two paths during validation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jvsena42
left a comment
jvsena42
left a comment
There was a problem hiding this comment.
Reviewed the workflow as a secrets/supply-chain surface as well as for correctness. The trigger is a plain pull_request with no untrusted-ref checkout, and e2e-status still gates only the self-hosted path, so the blast radius is contained.
Five low items. I dropped a sixth (pinning tailscale/github-action) as a duplicate of greptile's comment at :528; the only unsaid part is that the second call site at :678 has the same issue.
For context, Android has no equivalent: bitkit-android/.github/workflows/e2e.yml runs the docker stack on the same runner as the emulator, so there is no cross-platform shape to conform to here — this split is forced by hosted macOS lacking Docker.
| cp docker/lnd/tls.cert /tmp/creds/ | ||
| cp docker/lnd/data/chain/bitcoin/regtest/admin.macaroon /tmp/creds/ | ||
| chmod -R a+r /tmp/creds | ||
| nohup python3 -m http.server 8081 --bind 0.0.0.0 --directory /tmp/creds \ |
There was a problem hiding this comment.
🔵 LND admin.macaroon served unauthenticated on 0.0.0.0, and the access log sits in the served directory
Bind the credential server to the tailnet address rather than every interface (--bind "$(tailscale ip -4)"), and write access.log somewhere outside --directory so it is not itself served. Regtest-only credentials on a single-tenant runner, so nothing is exposed today — but this step is the template anyone copies for the next stack.
| fail-fast: false | ||
| matrix: | ||
| shard: | ||
| - { name: e2e, grep: '@transfer|@send|@lnurl|@lightning|@backup|@onboarding|@onchain_1|@onchain_2|@numberpad|@widgets|@boost|@receive|@settings|@security|@multi_address_1|@multi_address_3|@multi_address_4|@hardware_wallet' } |
There was a problem hiding this comment.
🔵 18-tag shard grep is copy-pasted from e2e-tests-local and will drift
This is byte-identical to line 244 in e2e-tests-local. Adding or renaming a tag in one place silently changes what the two paths cover, and since e2e-status does not gate on the remote path the divergence would go unnoticed. Hoist the grep into a single top-level env: value and reference it from both matrices, so the 'same suite' claim is enforced rather than asserted.
| if: github.event.pull_request.draft == false && needs.detect-changes.outputs.code == 'true' | ||
| runs-on: macos-latest | ||
| needs: [detect-changes, build-local, e2e-branch] | ||
| timeout-minutes: 360 |
There was a problem hiding this comment.
🔵 Gate the trial path behind a label or dispatch instead of running it on every PR
timeout-minutes: 360 here and 420 on regtest-stack mean one wedged trial run burns ~6 hours of GitHub-hosted macOS (billed 10x) plus 7 hours of ubuntu on a PR that e2e-status does not gate on — the failure is silent and expensive. Drop both to something near the observed run time. If the parallel trial is meant to be long-lived, a e2e-remote label gate would also let you turn it off without a revert.
|
|
||
| - name: Hold the stack up until the tests finish | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} |
There was a problem hiding this comment.
🔵 Add permissions: — the jobs API needs actions: read, and the error is swallowed into a misleading failure
2>/dev/null || echo 1 pins pending=1 on any gh api failure, and the finished guard below then reports stack expired while e2e-tests-remote was still running — the wrong diagnosis when the tests actually passed 6 hours earlier. Let the API failure surface, or distinguish it in the error message. Also worth adding permissions: { contents: read, actions: read } to this job so it does not ride on the repo-wide default token scope.
| - uses: tailscale/github-action@v3 | ||
| with: | ||
| authkey: ${{ secrets.TS_AUTHKEY }} | ||
| hostname: regtest-${{ github.run_id }} |
There was a problem hiding this comment.
🔵 Tailnet hostname omits run_attempt, so a re-run can resolve to the previous attempt's node
github.run_id is stable across re-runs. The peer lookup matches on HostName == and takes first(...), and jq's iteration order over the Peer map is arbitrary, so if the previous attempt's node has not been reaped yet the tester can pick a dead peer's IP and then spend 30 minutes failing port checks. Include ${{ github.run_attempt }} in both hostnames (here and tester- at line 681, and the same two in #692), and make sure TS_AUTHKEY is an ephemeral key so stale nodes actually go away.
2 participants
Description
e2e-tests-localneeds Docker and the iOS Simulator on one machine, which GitHub-hosted macOS runners cannot provide. It therefore runs on the self-hosted Mac as a single shard, ~2h per run — against ~20m for the same suite on Android's hosted Linux runners.Adds the same suite on
macos-latest, with the regtest stack on anubuntu-latestrunner reached over Tailscale:regtest-stacke2e-tests-remoteBoth reuse
build-local, so the app is still built once.Nothing existing changes.
e2e-tests-localis untouched, ande2e-statusstill gates only on it — the new job reports but does not block, until it has earned replacing it.The two new jobs must not depend on each other:
regtest-stackonly finishes once the tests are done, so a dependency either way deadlocks.Pairs with
synonymdev/bitkit-e2e-tests#208 — same branch name, so
determine-e2e-branchselects it automatically.Verified
Full
@lightningspec green on this arrangement, 39 min end to end. This PR runs the full 18-tag shard against a remote stack for the first time — that result is the thing to watch.Linked Issues/Tasks