DeepgramTTSService WebSocket not fully torn down after pipeline ends

[ValleZ]

pipecat version

0.0.108

Python version

3.13.5

Operating System

Linux (Debian 13, x86_64)

Issue description

File: pipecat/services/deepgram/tts.py — stop() / _disconnect() teardown path
Related: pipecat/services/websocket_service.py — _maybe_try_reconnect

Severity: Low — no functional impact on completed calls, but leaks an open WebSocket connection and generates hourly error noise.

What happens

After a pipeline runs to completion (EndFrame processed, runner.run() returns), the DeepgramTTSService's underlying WebSocket connection remains open. Deepgram times out idle connections with a 1011 internal error (NET-0003) after ~60 minutes. Because _disconnecting is False at that point (suggesting _disconnect() was either not called or its effect didn't persist to the receive task), _maybe_try_reconnect treats this as a recoverable error, reconnects, and the cycle repeats every hour until the process restarts.

Suspected cause

Possible race between the receive task being cancelled and Deepgram sending its 1011 — or _disconnect() completing but _disconnecting being reset before the 1-hour timeout fires. Needs reproduction with a controlled sleep/disconnect sequence.

Reproduction steps

1. Create a pipeline with DeepgramTTSService and RTVIProcessor
2. Run a call to completion (pipeline processes EndFrame naturally — e.g. agent says goodbye and hangs up)
3. Confirm runner.run() returns and the "Finally block" is reached in logs
4. Wait ~60 minutes
5. Observe logs

Expected behavior

stop(EndFrame) → _disconnect() sets _disconnecting = True and cancels _receive_task. After that, any close event from Deepgram should hit the _disconnecting guard in _maybe_try_reconnect and exit the receive loop without reconnecting.

Actual behavior

After runner.run() returns, DeepgramTTSService's WebSocket connection remains open.
Deepgram times out the idle connection after ~60 minutes with a 1011 internal error (NET-0003).
Because _disconnecting is False at that point, _maybe_try_reconnect treats it as a recoverable
error, reconnects successfully, and the cycle repeats every hour indefinitely until the process
restarts.

Logs

● 2026-05-03 07:32:54 | INFO  | main:_post_call_cleanup - Call record saved — completed, summary: ready                                                                                                                                                                                                            
  2026-05-03 07:32:54 | INFO  | main:websocket_endpoint - Finally block reached                                                                                                                                                                                                                                    
  # ~60 minutes pass, no activity                                                                                                                                                                                                                                                                                  
  2026-05-03 08:30:34 | WARNING | pipecat.services.websocket_service:_maybe_try_reconnect:149 - DeepgramTTSService#2 connection closed, but with an error: received 1011 (internal error) NET-0003; then sent 1011 (internal error) NET-0003                                                                       
  2026-05-03 08:30:34 | WARNING | pipecat.services.websocket_service:_try_reconnect:86 - DeepgramTTSService#2 reconnecting, attempt 1                                                                                                                                                                              
  2026-05-03 08:30:34 | WARNING | pipecat.services.websocket_service:_reconnect_websocket:66 - DeepgramTTSService#2 reconnecting (attempt: 1)                                                                                                                                                                      
  2026-05-03 08:30:34 | DEBUG  | pipecat.services.deepgram.tts:_connect_websocket:233 - DeepgramTTSService#2: Websocket connection initialized
  2026-05-03 08:30:34 | INFO   | pipecat.services.websocket_service:_try_reconnect:88 - DeepgramTTSService#2 reconnected successfully on attempt 1                                                                                                                                                                 
  # repeats every 60 minutes:
  2026-05-03 09:30:34 | WARNING | pipecat.services.websocket_service:_maybe_try_reconnect:149 - DeepgramTTSService#2 connection closed, but with an error: received 1011 (internal error) NET-0003; then sent 1011 (internal error) NET-0003
  2026-05-03 10:30:34 | WARNING | pipecat.services.websocket_service:_maybe_try_reconnect:149 - DeepgramTTSService#2 connection closed, but with an error: received 1011 (internal error) NET-0003; then sent 1011 (internal error) NET-0003

[markbackman]

I haven’t been able to reproduce this so far.

From the teardown path, if an EndFrame reaches DeepgramTTSService, it should call stop() → _disconnect(), set _disconnecting = True, cancel the receive task, close the WebSocket, and clear self._websocket.

Can you confirm exactly how the pipeline is being ended in your app? Are you explicitly pushing an EndFrame, calling task.stop_when_done(), or ending it some other way?

[ValleZ]

Hello! Thanks for checking it out!

I end the pipeline by calling task.queue_frames([EndFrame()]) explicitly. There are two paths in my app that trigger it:

1. My own REST endpoint sets an end_reason flag and calls task.queue_frames([EndFrame()])
2. The on_client_disconnected event handler calls task.queue_frames([EndFrame()]) if end_reason hasn't already been set

The stale connection happens on path 1. Immediately after queuing the EndFrame, still in the same coroutine, I make a synchronous Twilio REST call to hang up the phone:

client.calls(call_sid).update(status="completed") # blocks event loop

This blocks the event loop while the EndFrame is still queued. The Twilio hang-up causes the WebSocket to close, and by the time the event loop resumes the transport is already torn down — so stop() / _disconnect() gets interrupted before it can cancel _receive_task.

On my side I plan to wrap the Twilio call in asyncio.to_thread() and explicitly call await tts.stop(EndFrame()) in the finally block of my WebSocket handler as a fallback.

On pipecat's side, wrapping _receive_task cancellation in a finally in _disconnect() would make teardown resilient to this race regardless of what app code does:

async def _disconnect(self):
self._disconnecting = True
try:
if self._websocket:
await self._websocket.close()
self._websocket = None
finally:
if self._receive_task:
self._receive_task.cancel()
self._receive_task = None

This may also explain why you can't reproduce it — without a synchronous blocking call between queue_frames([EndFrame()]) and the transport closing, the race window never opens.