AI-Powered Self-Hosted Blog

Designing Streaming Responses for AI Agents: Implementation Patterns for Handling Interruptions, Reconnections, and Partial Display During Generation

Published

A streaming response looks simple until a user closes their laptop halfway through an answer. They reopen it 30 seconds later: the agent may still be working, but the page shows nothing. If the application treated the network connection as the response itself, it has no reliable way to recover what happened.
This problem grows with AI agents. A single model call may produce text in a steady sequence. An agent may search, call tools, revise its plan, and generate an answer over several minutes. Users need to see useful progress, stop work they no longer want, and recover their place after a connection drops. Those requirements make streaming part of the response architecture.

Separate the generation from the connection

A network stream is temporary; an agent run has its own lifecycle. Give every run a stable response_id, and assign each emitted event a monotonically increasing sequence number. The client should be able to identify what it has seen without relying on an open socket.
A compact event might look like this:

{
  "response_id": "resp_8f2",
  "sequence": 17,
  "type": "message.delta",
  "message_id": "msg_3",
  "text": "The second source confirms"
}

The response_id identifies the run. The message_id identifies the piece of output being updated. The sequence number gives the client a position from which to resume. Keep these roles distinct: an agent may emit several messages and tool updates within one response.
Define an explicit lifecycle rather than asking the UI to infer completion from a closed connection. A useful event vocabulary includes:

Event Meaning
response.started The run has been accepted.
message.delta New displayable text has arrived.
tool.started and tool.completed A named tool step has begun or ended.
message.completed One message is final.
response.completed The whole run is final.
response.failed The run ended with an error.
response.cancelled The run stopped after a cancellation request.
A socket closing is not a completion event. It may indicate a lost network connection while generation continues on the server.

Make replay a deliberate part of the protocol

Suppose the client receives events through sequence 42, then loses connectivity for 20 seconds. On reconnect, it requests events after 42 for the same response_id. The server returns the missed events and then resumes live delivery. If the same event arrives twice during the handoff, the client ignores the duplicate sequence number.
Server-Sent Events can support this pattern with event IDs and Last-Event-ID. WebSockets can support it too, but require an application-level resume request. The transport choice matters less than the replay contract: specify how long events remain available, whether replay is ordered, and what happens when the requested position has expired.
For a long-running agent, storing every token forever may be unnecessary. One practical approach is to retain recent events for a limited window, such as 10 minutes, and periodically save a snapshot of the current response state. If replay is no longer available, return the snapshot plus its latest sequence number. The client replaces its local view with that authoritative state before subscribing to newer events.
Snapshots must include more than visible text. They should identify completed messages, the active tool step, terminal status, and the last sequence number. Otherwise, a reconnected UI might display an answer while missing the fact that its citation lookup failed.

Treat interruption as a state transition

Users press Stop because they expect work to stop promptly. Sending a cancellation signal is only the beginning. The server must associate it with a specific response_id, propagate it to the model or tool where possible, and emit a terminal response.cancelled event when the run has actually ended.
There is a race to handle: the user may press Stop just as the agent finishes. Make terminal states exclusive. Once a response is completed, failed, or cancelled, later deltas should not change it. The client can show “Stopping…” while cancellation is pending, then preserve the text already received when cancellation is confirmed.
Tool calls need their own rules. A database read may be easy to abandon. An external action that has already committed may not be reversible. Cancellation should stop further agent work, while the UI reports the outcome of any action already taken. A “Stop” button should not imply that a completed side effect was undone.
User steering is related but distinct. If a user sends a new instruction during generation, decide whether it cancels the current run or starts a new one. In either case, give the new run a new response_id. This prevents a delayed delta from the old run from appearing inside the new answer.

Render partial output honestly

Partial text is useful, but it is not necessarily a finished message. Markdown may arrive halfway through a code fence or link. Structured output may temporarily be invalid JSON. Tool results may be pending even while the agent has begun explaining its approach.
The UI should maintain a working buffer for each message and render it as provisional until message.completed arrives. For ordinary prose, append deltas and update the display at a controlled rate, such as once per animation frame, rather than forcing a full render for every token. For Markdown, use a renderer that tolerates incomplete syntax or display unfinished segments as plain text until they form a stable block. Do not parse a half-received JSON object as if it were final data.
Show tool activity separately from answer text. A status such as “Searching documents” can help during a 15-second pause; dumping raw tool arguments into the answer usually cannot. When a tool completes, replace the pending status with a concise result or proceed to the next step. If it fails, show that failure in the run state rather than leaving a spinner indefinitely.
The distinction between provisional and final content also matters for citations and actions. A sentence may change as the agent gathers evidence. Avoid presenting an unverified claim with the same visual certainty as a completed answer. If the product supports copying or sharing, make clear whether the selected text is still being generated.

Design the failure paths before shipping

A resilient stream needs defined behavior for at least four cases: the client disconnects, the server restarts, a tool fails, and cancellation arrives near completion. Each case should lead either to replayable progress or to a clear terminal state. Timeouts and heartbeat events can help detect a dead connection, but a missed heartbeat should trigger reconnection, not mark the agent run as failed.
Test the contract with realistic interruptions. Start a two-minute tool-assisted response, disconnect after event 42, and verify that reconnecting produces the same final UI state as an uninterrupted run. Deliver one event twice and confirm it appears once. Press Stop during a tool call and check both the terminal event and the tool’s actual outcome. Cut the stream halfway through a code fence and confirm the page remains readable.
These tests expose a simple principle: the client is a view of an agent run, not the owner of that run. Once generation has an identity, ordered events, replay or snapshots, and explicit terminal states, interruptions become recoverable. Streaming can then show useful work in progress without pretending that every partial fragment is a finished answer.