Speech interruption brakes client with 'Missing required parameter' error

[siwk]

Hi,

while testing examples/streaming_cli.py and trying to interrupt generated audio response, OpenAI Realtime Client stops responding and display errors:

• Missing required parameter error
• Error streaming: received 1000 (OK); then sent 1000 (OK)

Error may be related to wrong client implementation of 'conversation.item.truncate' message. According to API Reference message requires following fields: type, item_id, content_index and audio_end_ms but client sends only type and item_id (content_index and audio_end_ms are missing).

Below part of log with websockets.client debug enabled:

--- begin of streaming_cli.py log ---
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...CAQL2AeMB1wHdAecB5wE="}' [2782 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": .../6v/y/wAABQD///3/BAA="}' [2782 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...//v/9//f/+P8AAAMA/P8="}' [2782 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.audio.delta","event_id":"even...RABQAGwAPABUAEAARABgA"}' [16200 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.audio.delta","event_id":"even...f/x//M/8//1r/Zv98/4D/"}' [16200 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.audio.delta","event_id":"even...X9MX1y/ZP9Zz2RvdQ+Vf7"}' [16200 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...//f/+/////f/+/////P8="}' [2782 bytes]
DEBUG:websockets.client:< TEXT '{"type":"input_audio_buffer.speech_started","ev...AtBHKXqYTimXrHbB4R80M"}' [146 bytes]

[Speech detected]

[Handling interruption]
DEBUG:websockets.client:> TEXT '{"type": "response.cancel"}' [27 bytes]
DEBUG:websockets.client:> TEXT '{"type": "conversation.item.truncate", "item_id...AtBHK3VsJ17qbpRYv0cUo"}' [79 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.audio.done","event_id":"event...x":0,"content_index":0}' [188 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...AAAD//wEAAwADAAIAAgA="}' [2782 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.audio_transcript.done","event...o puedo ayudarte hoy?"}' [250 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.content_part.done","event_id"... puedo ayudarte hoy?"}}' [270 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.output_item.done","event_id":...uedo ayudarte hoy?"}]}}' [343 bytes]
DEBUG:websockets.client:< TEXT '{"type":"response.done","event_id":"event_AtBHK...:34}},"metadata":null}}' [902 bytes]
DEBUG:websockets.client:< TEXT '{"type":"conversation.item.input_audio_transcri..."transcript":"Hey.\n"}' [184 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...//f///wAAAAAAAAEAAgA="}' [2782 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": ...//P/8//3//f/9//3//P8="}' [2782 bytes]
DEBUG:websockets.client:> TEXT '{"type": "input_audio_buffer.append", "audio": .../8f/s/+j/3v/N/73/sP8="}' [2782 bytes]
DEBUG:websockets.client:< TEXT '{"type":"error","event_id":"event_AtBHK2jaBjb7V...ndex","event_id":null}}' [231 bytes]
DEBUG:websockets.client:< TEXT '{"type":"error","event_id":"event_AtBHK9MFo0K8u...:null,"event_id":null}}' [197 bytes]
DEBUG:websockets.client:< CLOSE 1000 (OK) [2 bytes]
DEBUG:websockets.client:= connection is CLOSING
DEBUG:websockets.client:> CLOSE 1000 (OK) [2 bytes]
Error: {'type': 'invalid_request_error', 'code': 'missing_required_parameter', 'message': "Missing required parameter: 'content_index'.", 'param': 'content_index', 'event_id': None}
Error: {'type': 'invalid_request_error', 'code': None, 'message': 'Cancellation failed: no active response found', 'param': None, 'event_id': None}
DEBUG:websockets.client:= connection is CLOSED
Error streaming: received 1000 (OK); then sent 1000 (OK)
--- end of streaming_cli.py log ---_

[ASAD-BE18]

Did you solve this issue?

[parkers0405]

I have something out side of this but it works in a custom async flow, ping me if you want to see how

[ASAD-BE18]

Yes please

[parkers0405]

Its a little janky, its not a direct communication with the RAG its more of a stream text, and pass that to the voice model, so not sure if that is what you are looking for