Skip to content

g-cloudflare: large single statement (~124KB) reliably kills the connection on Cloudflare Workers - "Connection terminated unexpectedly" #3757

Description

@tenhsor1

Environment

  • pg: 8.16.3 (bundled pg-cloudflare 1.3.0; the socket write path is unchanged in 1.4.0)
  • Runtime: Cloudflare Workers, production (connect() TCP sockets, TLS)
  • Consumer: Prisma @prisma/adapter-pg 6.x → pg.Pool
  • Server: Supabase Postgres behind Supavisor transaction pooler (port 6543)

Summary

A single parameterized statement whose wire message is large (~124KB of bind parameters in our case) deterministically fails from a deployed Worker with:

Error: Connection terminated unexpectedly
    at <bundle>
    at async PgTransaction.performIO
    at async PgTransaction.executeRaw

Small statements on the same connection, milliseconds earlier, succeed. The failure reproduces 100% of the time, including on a freshly opened connection (new pool, single connection, TLS handshake and auth complete, BEGIN and a small SELECT set_config(...) succeed first).

What the server sees

This is the part that localizes the bug to the client side:

  • Postgres logs: nothing at all at the failure timestamps (no error, no crash, no statement logged).

  • Supavisor (pooler) logs:

    ClientHandler: (ECLIENTSOCKETCLOSED) Client socket closed while state was busy (transaction)
    

    i.e. from the server's perspective, the client closed the socket mid-statement. The connection is not being dropped by the pooler or the database — pg (or the Workers socket underneath it) is aborting its own connection while flushing the large message, and then reporting the resulting close as "Connection terminated unexpectedly". The underlying write error, if any, is never surfaced to the application.

Reproduction

Observed shape (via Prisma's pg adapter, but the adapter is a thin passthrough to pg):

// Deployed Worker (not wrangler dev), pg.Pool({ max: 5 }), TLS to Supabase pooler
await client.query('BEGIN')
await client.query("SELECT set_config('request.jwt.claims', $1, TRUE)", [claimsJson]) // ~600B — OK
await client.query(bigInsertSql, bigParams) // ~124KB of array params — connection dies in ~100-200ms

The failing statement is a CTE INSERT ... SELECT FROM unnest($1::uuid[], $2::text[], ...) with ~30 array parameters totaling ~124KB (about 650 rows across 5 tables). Nothing exotic about the SQL itself:

  • The identical statement pattern with smaller arrays succeeds.
  • The identical data split into chunked statements of ~25KB each (200 rows per unnest), inside the same interactive transaction, succeeds 100% of the time. This is our current workaround.
  • Retrying on a brand-new connection fails identically (~100-200ms after the statement is sent), so it is not a stale/idle-socket problem.

I don't have an exact byte threshold; payloads that historically stayed well under ~64KB never failed, ~124KB fails every time.

Suspected cause

pg-cloudflare's CloudflareSocket.write() forwards the entire buffer to a WritableStreamDefaultWriter.write() and returns true immediately — there is no backpressure handling:

write(data, encoding, callback) {
  ...
  this._cfWriter!.write(data).then(
    () => callback(),
    (err) => callback(err)
  )
  return true
}

and destroy() tears down via end()_cfSocket.close(), which produces exactly the abrupt client-side close the pooler logs. If the runtime's writer rejects or aborts on a large write, the error is only visible to the app as the generic post-close "Connection terminated unexpectedly", which made this quite painful to diagnose.

Expected behavior

Either the large write succeeds (with backpressure/chunking as needed for the Workers streams API), or the actual write error is propagated so the application can see why the connection died.

Notes

  • Reproduced only in production Workers; I have not verified whether local wrangler dev behaves the same.
  • Happy to provide more logs/timings or test patches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions