rtmp/rtsp: unblock a sender stuck in a socket write on disconnect

[RomanHerbstmann]

rtmp/rtsp: unblock a sender stuck in a socket write on disconnect

Problem

With TCP backpressure (server ACKs no longer arriving, send window full), RtmpSender blocks in a plain java.io socket write. A reConnect() (or a user disconnect()) then hangs until the network recovers or TCP gives up, which can take minutes.

Thread dump taken during the hang (two dumps, 12 s apart, identical):

"DefaultDispatcher-worker-2" Native
  at java.net.SocketOutputStream.socketWrite0(Native method)
  at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
  - locked (a java.io.BufferedOutputStream)
  at com.pedro.common.socket.java.TcpStreamSocketJavaBase.flush(TcpStreamSocketJavaBase.kt:48)
  at com.pedro.rtmp.utils.socket.TcpSocket.flush(TcpSocket.kt:40)
  at com.pedro.rtmp.rtmp.CommandsManager.sendVideoPacket(CommandsManager.kt:218)
  at com.pedro.rtmp.rtmp.RtmpSender.onRun(RtmpSender.kt:83)

Why it deadlocks:

• RtmpClient.disconnect() calls rtmpSender.stop() first, and BaseSender.stop() ends with job?.cancelAndJoin().
• Coroutine cancellation is cooperative, and a blocking java.net.Socket write never checks it. Thread.interrupt() would not help either, and java.net.Socket has no send timeout.
• The only thing that unblocks the write is closing or shutting down the socket. But closeConnection() runs after stop(), so the join waits forever.

Fix

• Try a cooperative stop first. This is the normal case: it finishes in milliseconds and keeps the graceful close (FCUnpublish/deleteStream, RTSP TEARDOWN).
• If the sender does not stop within 1 s, close the socket to unblock the write and stop again. TcpStreamSocketJavaBase.close() only does shutdownOutput/shutdownInput/close, so it does not take the stream lock the writer holds.
• A cancelled first stop() has already set running = false and reset the packets. The second stop() completes the join and clears the queue. sendClose afterwards fails fast and is already wrapped in runCatching.
• No duplicate onConnectionFailed: the woken writer's onMainThread runs in an already cancelled job, so it throws CancellationException without running the callback.
• RTSP over TCP (interleaved, RtpSocketTcp) writes to the same socket and gets the same change.
• SRT and UDP are not affected, because UDP sends do not block on missing ACKs.

Test

Android emulator (root image), RTMP to MediaMTX, backpressure produced with iptables -A INPUT -p tcp --sport 1935 -j DROP. The app requests a reConnect after detecting the stall.

	before	after
reConnect → onConnectionStarted	hung 30–65 s, until the network came back or the app gave up	1 s after the failure the sender stop times out and logs sender blocked in socket write, closing socket to unblock it; onConnectionStarted follows 3.8 s after the failure (1 s stop timeout + 1.5 s retry delay). Retries then continue every ~7 s (connect timeout) and reconnect about 3 s after the backpressure is lifted.

Normal stop/start and reconnect without backpressure: the new path is not taken (the warning appears exactly once in the run, only during the backpressure). The user stop at the end still closes gracefully (server log: closed: EOF).

Limitations

• rtmps: SSLSocket.close() sends close_notify and could itself block under the same backpressure. It is not covered here and was not verified. Plain rtmp:// and RTSP over TCP are covered.
• No unit test: the blocked write needs a real socket with a full send buffer. The client creates its socket internally, so a unit test would need a socket-injection refactor. The fix was verified end-to-end with the thread dumps and the emulator runs above instead.

[pedroSG94]

Hello,

Thank you for the fix.
Merged but I did a refactor over it based in your fix adding the timeout into stop method of the sender:
#2197

This is equivalent but avoid repeat code in the future if other protocol need control it.