Skip to content

fix: Generalize LTSM abort handling and reset WASM - #245

Merged
highesttt merged 6 commits into
mainfrom
fix/generalize-ltsm-abort
Sep 29, 2026
Merged

highesttt merged 6 commits into
mainfrom
fix/generalize-ltsm-abort

Conversation

@hifi

@hifi hifi commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Qwens attempt at fixing the following panic:

1790232769079	2026-09-24T06:52:49.079Z	panic: ltsm: WASM abort called
1790232769079	2026-09-24T06:52:49.079Z	
1790232769079	2026-09-24T06:52:49.079Z	goroutine 52529 [running]:
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Imports).Import_m(0x4b111c79b90?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/embind.go:447 +0x2a
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f48(0x4b111c79b90, 0x18530000?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:8012 +0xd6
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f595(0x4b111c79b90, 0x801980, 0x0, 0x0)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:408018 +0x51
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).callIndirectT0(0x4b113109630?, 0x1b6abc7?, 0x0?, 0x11b355c0?, 0x4b1?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:277 +0x488
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f589(0x4b111c79b90, 0x13109600?, 0xc?, 0x35?, 0x800684)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:407454 +0x12a
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).callIndirectT9(0x80067000800650?, 0xb?, 0x80066c?, 0x0?, 0x40574000?, 0x800668?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:979 +0x13a
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f230(0x4b111c79b90, 0x800668?, 0x17a09f48, 0x800678, 0x800684)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:69013 +0x172
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f440(0x4b111c79b90, 0x800698, 0x8017b0, 0xd1442abe?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:216539 +0x25c
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).callIndirectT3(0x18?, 0x0?, 0x0?, 0x469e1e?, 0x0?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:597 +0x1b7
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).f285(0x4b111c79b90, 0x11b35500?, 0x4b1?, 0x1300?)
1790232769079	2026-09-24T06:52:49.079Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:111453 +0x14e
1790232769079	2026-09-24T06:52:49.079Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Module).callIndirectT0(0x0?, 0x20?, 0x0?, 0x20?, 0x0?)
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/wbc_generated.go:225 +0x156
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Imports).callIndirect(0x4b111b355c0, 0x5a, {0x4b111bcea80, 0x3, 0x48550a?}, {0x4b113109828, 0x3, 0x0?})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/embind.go:1001 +0x51f
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Imports).CallMethod(0x4b111b355c0, {0x4d57378, 0x4}, {0x4d5cc42, 0x6}, 0x8017b0, {0x4b1131098ec, 0x1, 0x40?})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/embind.go:864 +0x228
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/ltsm.(*Runtime).HmacDigest(0x4b111bb0a20, 0x8017b0, {0x4b1139c5a80?, 0x4b113109a60?, 0x1?})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/ltsm/runtime.go:126 +0xca
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg.(*Runner).GetSignature(0x4b11140ca20, {0x4b122b70084?, 0x2?}, {0x4b113109a60, 0x2}, {0x4b1111f1d40, 0x21f})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/runner.go:295 +0x198
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/line.(*Client).callRPCWithBaseURLContext(0x4b11ab0f950, {0x8eed5a0, 0x4b111bd4050}, {0x4f021d4?, 0x4b11b2746b0?}, {0x4d6e4b2, 0xb}, {0x4d93515, 0x11}, {0x0, ...})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/line/client.go:383 +0xbf8
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/line.(*Client).callRPCContext(...)
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/line/client.go:338
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/line.(*Client).GetLastOpRevisionContext(0x217691f?, {0x8eed5a0?, 0x4b111bd4050?})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/line/methods.go:648 +0x65
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/connector.init.func9({0x8eed5a0?, 0x4b111bd4050?}, 0x8afb138?)
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/connector/sync.go:103 +0x2c
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/connector.(*LineClient).handleReceiveAuthProbe(0x4b110d28900, {0x8eed5a0, 0x4b111bd4050})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/connector/sync.go:1689 +0x89
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/connector.(*LineClient).pollLoop.func3()
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/connector/sync.go:1563 +0x7a
1790232769080	2026-09-24T06:52:49.080Z	github.com/highesttt/matrix-line-messenger/pkg/connector.(*LineClient).pollLoop(0x4b110d28900, {0x8eed5a0, 0x4b111bd4050})
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/connector/sync.go:1641 +0x4df
1790232769080	2026-09-24T06:52:49.080Z	created by github.com/highesttt/matrix-line-messenger/pkg/connector.(*LineClient).Connect in goroutine 210
1790232769080	2026-09-24T06:52:49.080Z		/go/pkg/mod/github.com/beeper/line@v0.0.0-20260918150712-ab397713a013/pkg/connector/client.go:647 +0x10b9

I don't know if it's safe to recover WASM state like that, I presume we should probably unwind the whole bridge and relogin?

@indent

indent Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
PR Summary

Targets the production panic: ltsm: WASM abort called crash. Recovered LTSM exceptions were leaking WASM stack and heap until malloc aborted. This PR turns ordinary LTSM exceptions into errors and restores the WASM stack pointer after them. It also runs channel encrypt/decrypt on a disposable copy of the module, so their leaks can't exhaust the main runtime.

  • pkg/ltsm/embind.go: EM_JS and C++ throws panic with a new wasmException type. recoverError (deferred in callIndirect and Destroy) turns them into errors and restores g0; aborts and unexpected panics are re-panicked. noteChannelState marks the crypto copy stale when the main runtime touches Curve25519Key, E2EEKey, E2EEChannel or E2EEKeychain.
  • pkg/ltsm/runtime.go: channelCrypto runs E2EEChannelEncryptV1/V2 and DecryptV1/V2 on a shadow Runtime. The shadow is re-copied from the main module (memory, g0, emval table, fd state) at the start of the next channel call when it is stale, after any error or panic, or every 128 calls.
  • pkg/e2ee/manager.go: decrypts pick V1 or V2 from the first chunk's length (16 bytes = V2, anything else = V1). If that fails with a non-abort error, they fall back to the other version.
  • Tests: channel crypto leaves the main module untouched even when a call aborts, and version routing and fallback are covered.

Issues

All clear! No issues remaining. 🎉

7 issues already resolved
  • LTSM reports ordinary crypto failures (MAC/authentication failure, bad AES params) through the EM_JS "throw error" import, which panics, and checkDead now marks the runtime dead on any panic. One bad inbound ciphertext makes every later WASM call (including the V2→V1 decrypt fallback and request signing) fail until a reset. Only mark the runtime dead for ErrAbort, and turn EM_JS/C++ throws into plain errors as main did. (fixed by commit 2eb7ded)
  • Runner.reset clears keyStore, channelStore, goChannels and storageKey on the process-wide Runner. Every login's e2ee.Manager keeps those runner IDs (myKeyID, keyByRawID, channelByPair, groupKeys, groupSessionChannels) and never reloads them, so after one reset all E2EE sends and decrypts fail with unknown key/unknown channel for every user until restart or re-login. Rebuild the managers' state after a reset, or re-create WASM objects under the same IDs. (fixed by commit 2eb7ded)
  • E2EEChannelEncryptV2/DecryptV2 call writeStdString (which runs the module's malloc export directly) before CallMethod's dead check and outside callIndirect's checkDead. With encryptV2PanicSafe/decryptV2PanicSafe gone, an abort or panic there crashes the bridge instead of returning an error, and malloc still runs on an already-dead runtime. (fixed by commit 2eb7ded)
  • If reset() fails once, runnerErr keeps that error. A later successful reset then falls through to return globalRunner, runnerErr, so every caller gets the old error for good. Clear runnerErr when a reset succeeds, or return globalRunner, nil on that path. (fixed by commit 2eb7ded)
  • GetRunner reads rt.imp.dead holding only runnerMu, while WASM calls write it under Runner.mu, which go test -race will report. The runtimeDead hook is labeled a test seam, but no test uses it, so GetRunner/reset have no test coverage. (fixed by commit 2eb7ded)
  • Recovering an EM_JS/C++ exception unwinds WASM frames without restoring the stack pointer (Module.g0) or running C++ destructors, so each one leaks about 160–320 B of stack and up to ~5 KB of heap in the fixed 16 MB heap. After about 1.6k V2-decrypt exceptions (the V2→V1 fallback path), the next allocation calls _abort, for example in HmacDigest during GetSignature, which is the crash in this PR's description, so the PR does not fix it. Restore g0 on recovery, and avoid the throwing path or recycle the runtime before the heap runs out. (fixed by commit 72a3e0e)
  • Runtime.channelCrypto restores memory, g0, emval and fd state only after call() returns normally. If ErrAbort is re-panicked inside a channel encrypt/decrypt, the half-finished call's state stays in the module, and the runner's *PanicSafe wrappers then recover and keep using it. Run the restore in a defer so aborts are rolled back too. (fixed by commit 5152743)

CI Checks

All CI checks passed on a364fb1.

Comment thread pkg/ltsm/embind.go Outdated
Comment thread pkg/runner.go Outdated
Comment thread pkg/runner.go Outdated
Comment thread pkg/runner.go Outdated
@highesttt

Copy link
Copy Markdown
Collaborator

I don't know if it's safe to recover WASM state like that, I presume we should probably unwind the whole bridge and relogin?

I don't think it's safe to reset the runtime here. Runner.reset() drops the encryption keys and channels but the e2ee managers still hold their IDs, so an abort like that would make messages unable to send or decrypt.
checkDead also treats ordinary decrypt errors as fatal, which would break the LINE v2 (android & desktop) -> v1 (ios) decryption fallback

Comment thread pkg/ltsm/embind.go Outdated
Comment thread pkg/ltsm/runtime.go Outdated
@highesttt
highesttt merged commit 9316913 into main Sep 29, 2026
9 checks passed
@highesttt
highesttt deleted the fix/generalize-ltsm-abort branch September 29, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants