Best way to stream multiple messages for useChat to consume?

[edave]

Based off the Next.js AI Chatbot I'm using several streamText calls (across multiple models/prompts/tools) within a single Next.js route.ts call.

However, I cannot figure out how to chain the streamText responses in a way that useChat will render multiple messages. For various reasons, I can't use message.parts (unless there's something I don't understand about message.parts being sub-messages).

Background

My code looks something like:

// route.ts
return createDataStreamResponse({
    execute: async (dataStream) => {
      const result1 = streamText({ ... })
      result1.mergeIntoDataStream(dataStream, {
              sendReasoning: true,
       });
      const result2 = streamText({ ... })
      result1.mergeIntoDataStream(dataStream, {
              sendReasoning: true,
       });
    }

// Chat.tsx
useChat({
    ...
  });

 
  return (
    <>
       ...
        <Messages
          chatId={id}
          isLoading={isLoading}
          votes={votes}
          messages={messages}
          setMessages={setMessages}
          reload={reload}
          isReadonly={isReadonly}
          isArtifactVisible={isArtifactVisible}
        />
    ...
   );

useChat receives and renders the streams from each streamText, but into a single <Message>

What I've tried:

1. I have tried using the experimental_sendStart and experimental_sendFinish flags to mergeIntoDataStream() but it has not changed anything.
2. I have also tried to start a new <Message> by manually injecting a new message type into the data stream, but this just gets wrapped as a generic data object:

dataStream.writeData({
  f: {
     messageId: generateUUID()
    }
});
// Inspecting the response stream in the browser shows a chunk: 2:[{"f":{"messageId":"6f73cd0d-e81b-46a7-9631-218ad074daa9"}}]

My Question
What should I be doing to have useChat correctly render several Messages from multiple streamText responses?

Thanks in advance!

[marbemac]

Also interested in ideas here - @edave did you end up landing on a solution?

[rychlis]

Also hit this issue and I don't see any clear solution. I have similar code and useChat hook returns just one message while streaming multiple messages and merges all content/parts into that one message.
We store multiple messages on backend side so when you reload messages from backend, it is inconsistent with what useChat returns

[rychlis]

we're going to workaround it by adding custom annotation mapping messageIds to message index on backend side and than split the merged message into multiple on FE side

[edave]

@rychlis I did not find a good existing answer to this. There is PR #2929 which adds support for branching, but it mostly stores the tree state client-side and still interacts with the Vercel AI SDK via a single-message mode.

I've started a similar workaround to what you described, although I'm using experimental_sendStart (currently a noop) and experimental_sendFinish in mergeIntoDataStream to send message IDs. This has a number of shortcomings similar to the approach you suggested of using custom annotations:

1. Message Annotations within the call chain of useChat, specifically, processChatResponse are assumed to be a global bucket of annotations, so they are automatically appended to and not cleared at any point.
2. There is not a way to pass a message ID to streamText as a variable (you could provide a custom method for experimental_generateMessageId that is deterministic in some way though) so it hard to know in advance of the streaming what the message ID will be if you need to provide unique annotations for each message.
3. Splitting out the merged messages on the FE is very difficult to do as a layer on top of useChat. I ended up forking useChat and processChatResponse to implement custom splitting. I am still working on an issue that the <Chat> component does not recognize updates to the messages array and appropriately re-render as new messages are added.

Hope that helps!