Translated from the Korean original. Some screenshots show the original Korean interface.
Designing AI Chat Streaming with an SSE Event Protocol
Hi, I'm Sumi Byeon, a frontend developer on the LLM Platform Team.
Our team builds eXemble, an LLMOps platform. eXemble is a service that lets companies build AI agents on their own data, connect them into workflows, and run them in production.
In this service, I own the chat frontend as a whole, where users send questions to AI agents and get answers back in real time.
When a user sends a question in chat, the agent runs the nodes defined in the workflow one after another to generate an answer. This usually takes 10 to 30 seconds. We can't show users a blank screen for that long, so streaming was a given.
But for us, streaming isn't just about showing text one character at a time.
A single stream has to deliver three things at once. > Answer text + Reference document list + Workflow execution flow
While the user reads the answer in chat, the workflow canvas next to it has to show in real time which node is running.
This complexity shaped every technical decision that followed, and the most decisive one was how to design the message event protocol between the server and the frontend.
This post covers how we designed the event protocol and how that design held up when we actually added a feature.
1. Streaming in AI chat: why it's needed and how it works
1-1. Why streaming is needed
AI chat is everywhere now, so watching an answer flow out one character at a time in ChatGPT or Claude is a familiar sight.
Text that appears as if it's being typed, as in the image, is called streaming.
LLMs generate tokens one at a time, so if you wait for the full response, the user stares at a blank screen for tens of seconds.
When LLM responses take 10-30 seconds, streaming isn't just a UX improvement. It's a core technique for keeping users from leaving.
According to research by Nielsen Norman Group, a delay of more than 1 second interrupts the user's flow of thought, and beyond 10 seconds the user loses attention entirely.
1-2. Technologies for implementing streaming
There are three main ways to implement streaming in AI chat.
Polling
- The client repeatedly asks "Any new data?" at fixed intervals.
- It's simple to implement, but it generates a lot of unnecessary requests and is less real-time.
- To keep up with the speed at which LLMs generate tokens (on the order of tens of ms), the request interval has to be extremely short, which sharply increases server load.
- Because the client has to check every time whether the response is complete, requests keep firing even when no tokens are being generated.
WebSocket
- The client and server open a bidirectional connection and exchange data freely.
- However, AI chat is mostly a one-way flow: "the user asks → the server streams back the answer."
- Using WebSocket when you don't need bidirectional communication adds the burden of managing connection state on the server.
- It suits real-time chat and games, but the server has to manage connection state, and compatibility with existing HTTP infrastructure requires extra configuration.
SSE (Server-Sent Events)
- The server sends data one way to the client.
- It runs over HTTP, so existing infrastructure can be used as is, and it supports automatic reconnection when the connection drops.
Because it matches the AI chat flow exactly, where "the server sends data to the client in one direction,"
most major AI chat services, including ChatGPT, Claude, and Gemini, use SSE.
1-3. Our choice: SSE + delta
We chose SSE too, for three reasons.
- Our infrastructure uses an Nginx (OpenResty)-based gateway, so SSE, which runs over HTTP, could be applied right away with no extra configuration.
- AI chat is a one-way flow, "the user asks → the server streams back the answer," so WebSocket, which is built for bidirectional communication, was overkill.
- We wanted to keep the server stateless.
We adopted the delta approach, which sends only the new fragment generated since the previous event. This keeps the server stateless and reduces transmission overhead.
The delta approach has a tradeoff: if an intermediate event is lost, the text can break. But when the stream ends, a final correction is possible with the
final_answer message, which contains the full answer, so it was a tradeoff we could accept.There was, however, one constraint in using SSE.
The browser-standard 
MDN Web DocsUsing server-sent events - Web APIs | MDN
MDN Web Docs
EventSource API supports only GET requests and doesn't let you set custom headers.
See MDN - Using server-sent events - Web APIs | MDN
With our JWT authentication setup, every request needed a POST body and an Authorization header,
so we chose the
@microsoft/fetch-event-source library.This library implements SSE parsing logic on top of the fetch API,
so it supports POST requests and custom headers while integrating naturally with our existing fetch-based code.
That covers a typical AI chat streaming implementation.
You receive text fragments (delta), accumulate them on screen, and end the stream when the completion (done) signal arrives.
But for our service, that wasn't enough.
2. Why we need an event protocol
2-1. Plain delta isn't enough
A typical AI chat needs only two messages: text fragments (delta) and a completion (done) signal.
But in our service, the workflow engine runs multiple nodes in sequence.
The frontend has to handle three things at once.
- Accumulate answer text token by token in the chat area
- Highlight the currently running node on the canvas
- Show reference documents together with the final answer
The problem is that SSE provides only one stream over a single HTTP connection.
Text fragments, node state changes, and reference document info are completely different kinds of data, yet they all arrive mixed together through the same pipeline.
If data is sent without a separate convention, the frontend has to parse each incoming message and guess what kind it is.
What's needed is a structure in which each message declares for itself what it is
in other words, a contract between the server and the frontend on "which message will carry which data." We call this contract the event protocol.
3. Designing the message event protocol: a contract with the server
The frontend and the server have to agree clearly on which messages arrive, in what order, and with what data.
We had two design principles.
- For Speed: a simple structure we could build quickly. Delta is at the core, and the server manages no state.
- For Future: every message is distinguished by a
typefield. When a new message is needed, we add a handler without modifying existing code.
3-1. Message sequence: the flow a single question creates
When a user sends a question, the server sends messages in the following event order.
Every message in the SSE stream has a
type field.The frontend looks at this value to decide what to do.
When
plan arrives, it shows the execution plan on screen; when node_start arrives, it highlights that node; and when delta arrives, it appends the text to the screen.Here's what an actual SSE response looks like.
3-2. Design decisions: why we split it this way
The core principle was that each message carries exactly one clear meaning.
If one type carries two meanings, the handler has to branch again on "which meaning is it this time?", which defeats the purpose of splitting types in the first place.
Here are a few concrete decisions where this principle applied.
node_start and node_done: why we split them- We also considered a single
node_statusmessage with a field distinguishing start and end. But the two events happen at completely different times, and the frontend has to do different things for each.
- When
node_startarrives, the frontend has to highlight the node and show a spinner; whennode_donearrives, it has to remove the spinner and show the result.
- Bundling them into one type would mean branching again on "start or end?" inside the handler, so splitting the type itself fit the principle.
final_answer: why it's a separate message type- Our service provides a list of reference documents (source, page, relevance) along with the final answer.
- Mixing this metadata into
deltawould pollute its meaning as a "text fragment," and putting it indonewould dilute its meaning as an "end signal."
- The final answer has its own independent meaning, so we split it into a separate type.
heartbeat: why it's separate from business logic- SSE is a long-lived connection, so the network can drop silently.
- The heartbeat the server sends periodically to detect this is an infrastructure-level signal, unrelated to "what happened in the workflow."
- We filter it out in the layer that handles the SSE connection so it never reaches the business logic (Reducer), and thanks to this separation we could improve the connection-detection logic independently of the business logic. (This pays off in 5-2.)
When each message carries only one meaning,
we can simply add a new type whenever a new meaning is needed, instead of changing an existing one.
Section 4 puts this structure to a real test.
3-3. The event protocol determines the state management structure
Sometimes a single message has to change several UI states at once.
For example, when a
node_start message arrives,- the canvas has to change the node state to
RUNNING
- and at the same time the chat state has to change to
STREAMING.
"Chat is WAITING but the node is RUNNING"
A state combination like this must never be shown on screen.
Thanks to React's state update batching , changing multiple states in the same handler doesn't actually expose intermediate states on screen.
But this implicitly relies on React's internal runtime behavior; the code structure itself doesn't guarantee it. If node state and chat state are spread across separate hooks or stores, figuring out which state combinations are possible and which aren't means tracing all the scattered code.
What we wanted was to define every state change for an event in one place, so that we could structurally block impossible state combinations from ever occurring.
In 3-2 we set "one type, one meaning" as a principle, and we applied the same principle to state management. If you gather the states an event must change in one place, the Reducer, then the code structure guarantees consistency, not the runtime behavior.
We implemented this structure with
useReducer.useSSEStream.ts: SSE handler
chatReducer.ts — Reducer
As a result, we ended up with a clean structure where SSE message types map 1:1 to Reducer actions.
When a new message type is added, you add one case to the SSE handler and one action to the Reducer.
4. Testing extensibility: adding an if/else branch node
Whether the "For Future" principle actually held up was put to the test sooner than we expected.
4-1. The problem: the execution plan changes at runtime
An if/else branch node was added to our workflows.
A branch node is a node where the next path to run depends on a condition during workflow execution.
Previously, the execution plan was fixed at the moment the question was asked.
Once we received the "A → B → C" path in a
plan message,
we just displayed it on the canvas and highlighted the nodes in order.But with branch nodes, there are now cases where the execution path is decided at runtime.
The canvas shows the path "A → B → C or D,"
but depending on the branch condition, "A → B → D" may be what actually runs.
For the user, this can cause confusion when actual progress diverges from what they expected based on the path shown on screen.
The frontend has to know that the plan changed midway so it can update the execution path on the canvas correctly and keep users from getting confused.
So what message should the server use to announce this change?
4-2. Design choice: reuse an existing type vs. add a new one
Within the event protocol structure from section 3, there were two ways to solve this.
Option A: Send the existing
plan message againIt's simple to implement, but
plan would then mean both "initial plan" and "plan change."
This goes directly against the "one type, one meaning" principle we set in 3-2.It's also a problem from an operations perspective. When
plan shows up twice in the logs,
telling whether it's the initial plan or a path change means cross-checking timestamps and the surrounding messages.The handling logic would also have to branch within one event type on "is this the first plan, or an update?"
Option B:
plan_updated, a new message typeEach message clearly expresses "what happened on the server."
plan is always the initial plan, and plan_updated is always a plan change.Of the two, we chose Option B.
It follows the "For Future" principle and makes both debugging and extension easier.
plan_updated contains the entire changed path, and the frontend simply replaces the old path with the new one.
We also considered a diff approach, but execution plans are small, so full replacement was simpler to both implement and debug.4-3. What actually changed
The entire change was one new row in the SSE → Reducer mapping table from 3-3.
- SSE handler:
plan_updated→dispatch({ type: 'UPDATE_PLAN' }), 1 case added
- Reducer: 1 action added that updates only the path while keeping existing node states
- Canvas component: path re-rendering
The existing handling logic for
plan, delta, node_start, node_done, final_answer, and done didn't change by a single line.What makes this meaningful is that the same pattern can be repeated going forward.
Each time a new message type is needed, we add one case to the SSE handler, one action to the Reducer, and one piece of UI to the component.
Existing code stays untouched.
The Open-Closed Principle from software design (open for extension, closed for modification) applied naturally at the event protocol level.
5. Troubleshooting: lessons from production
Once the design is done and you deploy, problems that weren't visible at design time show up.
Here are three problems we ran into in production and how we solved them.
5-1. Answers cut off in inactive tabs
This was the first problem we hit after deployment.
If a user did something else while waiting for an answer, or switched to another browser tab, the answer would be cut off midway when they came back.
The cause was the default behavior of
@microsoft/fetch-event-source, the library we used to implement SSE.By default, this library uses the Page Visibility API to close the SSE connection when the tab becomes inactive and tries to reconnect when it becomes active again. But in AI chat, tokens that have already been generated can't be received again, so even after reconnecting, the part of the answer lost in between isn't recovered.
It's a sensible default to avoid wasting resources, but in a service like AI chat, where generating an answer takes 10-30 seconds, it leads to a critical UX problem: "I looked at another tab for a moment and came back to a cut-off answer."
We fixed it by setting the
openWhenHidden: true option.Keeping the connection open in the background might raise resource concerns, but the SSE connection closes once the response is complete, so it never stays open indefinitely. The connection stays open only while the user is away from the tab and is cleaned up naturally when the answer finishes, so the real added cost was negligible.
5-2. Connections that drop silently
Because SSE is a long-lived connection, when it drops due to a network problem the client sometimes doesn't notice right away.
The heartbeat we designed in section 3 became the solution here.
The server sends a heartbeat every N seconds.
The client's connection manager records when it last received a heartbeat, and if no heartbeat arrives for M seconds or more, it treats the connection as dropped.
When it detects a dropped connection, it tries to reconnect using an exponential backoff strategy.
The first retry happens after 1 second, the second after 2 seconds, and the third after 4 seconds. We use this so we don't overload the server while the network is temporarily unstable.
Separating the heartbeat from business logic at design time paid off here.
The entire reconnection logic is self-contained in the connection manager, and we didn't touch the chat or canvas business logic at all.
5-3. Infrastructure collaboration: optimizing TTFT
Even with streaming in place, it felt slow for the first character to appear after the user sent a question.
When we checked the Network tab, the server was sending tokens immediately, but they were arriving late in the browser.
No data came in for a few seconds, then several tokens arrived in a burst.
The cause was Nginx's default buffering.
Nginx collects upstream responses in a buffer up to a certain size before passing them on to the client.
For typical API responses this is sensible behavior that improves network efficiency, but in SSE, where each token (a few bytes) has to be delivered as soon as it's generated, this buffering causes delays.
Working with a backend engineer on our team, we applied streaming-specific settings (
proxy_buffering off, among others)
and cut TTFT (Time To First Token) to under 0.5 seconds.My teammate Jeongsu Sim covers this process in detail in How Far Have You Taken Nginx? (feat. OpenResty).
The frontend alone can't deliver a complete streaming experience. The server, infrastructure, and frontend each have to optimize at their own layer, and collaboration where they meet is what matters.
Wrapping up
The most impactful decision in this project wasn't the choice of a particular technology but how we designed the message event protocol.
A single simple rule, distinguishing every message by a
type field, combined with the Reducer's switch/case to create an extension pattern where adding a new feature doesn't touch existing code.Adding the
if/else branch node was the first test of this pattern, and we expect the same pattern to repeat as we extend features in the future.Of course, this design isn't optimal in every situation.
Right now we send all data over a single stream, but as workflows get more complex and more agents run concurrently, we may need to separate channels or handle ordering guarantees between messages separately.
What mattered was not a universally "good" technology but technology that works in our environment.
When that design was validated against real requirements, I saw our initial principles turn into a strategy.
I hope this post is a useful reference for anyone designing an AI chat service.
Related articles
