spanner: sporadic "Failed to initialize transaction due to invalid mutation key" on mutation-only transactions after upgrading 8.6.0 -> 8.10.0

[diego-dakszewicz]

Environment

• @google-cloud/spanner: 8.10.0 (google-gax 5.0.6, grpc-gcp as wired by the library)
• Node.js 24 on GKE (Linux)
• Multiplexed sessions: library defaults (enabled, including read/write)

Workload
High-volume mutation-only read-write transactions, always the same shape:

await database.runTransactionAsync(async transaction => {
    transaction.upsert('Events', mutations);
    return await transaction.commit();
});

Keys are INT64 (Spanner.int over stringified values), well-formed. Thousands of these commits per day against the same table.

Behavior

• On 8.6.0 (in production from March through August, multiplexed sessions enabled, mutation-key selection already present): zero errors.
• Within hours of deploying 8.10.0: sporadic 3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key at a rate of roughly 6 per day out of thousands of identical transactions. The exact same mutation shape succeeds thousands of times a day; the failing mutations carry valid keys (verified from logs).

Analysis
We diffed build/src between 8.6.0 and 8.10.0:

• The mutation-key selection heuristic (_mutationKey and its selection rules in transaction.js) is identical between both versions.
• The new code in the transaction path is the per-transaction gRPC channel affinity for multiplexed sessions: _affinityKey = mux-affinity-... in the Snapshot constructor, affinity injected per request, unbind: true injected on commit/rollback/end.

Given the sporadic, race-like profile under concurrency, we suspect the BeginTransaction/Commit pair is occasionally affected by the new affinity routing.

Workaround
Making the affinity branch unreachable (transaction.js:181, if (session.metadata && session.metadata.multiplexed)) restores the 8.6.0 transaction behavior while keeping multiplexed sessions on. We are running this as a pnpm patch and validating that the error rate drops to zero.

Happy to provide timestamps, request IDs, or run an instrumented build if that helps.