Streaming LLM output to mobile with SSE, and surviving malformed JSON

Most LLM features feel slow for one reason: the user waits for the whole response before seeing anything. On mobile networks that wait is even more noticeable. Streaming fixes the perception problem, but it also exposes a second one: models don’t always return valid JSON, and you now have to parse it while it’s still arriving.

Here’s the approach I use when building LLM-backed apps, and the reliability patterns that make it production-ready.

Why SSE rather than WebSockets

For LLM output, traffic flows one way: server to client. That makes Server-Sent Events (SSE) a natural fit:

• It’s plain HTTP, so it works with your existing auth, proxies, load balancers and CDN rules.
• The browser (and most mobile HTTP clients) handle framing and reconnection for you.
• There’s no connection-upgrade handshake or bidirectional protocol to secure and operate.

Reach for WebSockets when you genuinely need low-latency two-way messaging. For “stream this answer to the screen,” SSE is simpler and easier to run.

A minimal streaming endpoint

The shape of the server is the same in any framework. This sketch uses Fastify’s raw response:

app.post('/api/generate', async (req, reply) => {
  reply.raw.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache, no-transform',
    Connection: 'keep-alive',
  });

  const controller = new AbortController();
  req.raw.on('close', () => controller.abort()); // stop paying for tokens nobody reads

  const heartbeat = setInterval(() => reply.raw.write(': ping\n\n'), 15_000);

  try {
    for await (const chunk of llm.stream(req.body, { signal: controller.signal })) {
      reply.raw.write(`data: ${JSON.stringify({ delta: chunk })}\n\n`);
    }
    reply.raw.write('event: done\ndata: {}\n\n');
  } catch (err) {
    reply.raw.write(`event: error\ndata: ${JSON.stringify({ message: 'generation_failed' })}\n\n`);
  } finally {
    clearInterval(heartbeat);
    reply.raw.end();
  }
});

Three details matter more than they look: abort on disconnect (so a closed app stops costing money), a heartbeat (so idle proxies don’t drop the connection) and an explicit done/error event (so the client never has to guess how a stream ended).

The harder problem: broken JSON

If you ask a model for structured output (a quiz question, a list of topics, a form definition), you’ll eventually receive something like this mid-stream:

{"questions":[{"prompt":"What does CAP stand for?","options":["Consistency","Avail

A strict JSON.parse throws, and a truncated or slightly malformed final response throws just as hard. A resilient pipeline treats model output as untrusted, partial input:

1. Accumulate chunks into a buffer.
2. Try a strict parse first. Most responses are fine.
3. Fall back to repair: close any open strings, arrays and objects, drop a dangling comma or colon, and parse again.
4. Emit completed items early. If the array has two finished objects and a third in progress, show the two.
5. Keep the last good state so a bad tail never wipes what the user already sees.

A small repair function covers most real-world cases:

export function repairPartialJson(input: string): unknown | undefined {
  const closers: string[] = [];
  let inString = false;
  let escaped = false;

  for (const ch of input) {
    if (inString) {
      if (escaped) escaped = false;
      else if (ch === '\\') escaped = true;
      else if (ch === '"') inString = false;
      continue;
    }
    if (ch === '"') inString = true;
    else if (ch === '{') closers.push('}');
    else if (ch === '[') closers.push(']');
    else if (ch === '}' || ch === ']') closers.pop();
  }

  let repaired = input;
  if (inString) repaired += '"';
  repaired = repaired.replace(/[,:]\s*$/, (m) => (m.startsWith(':') ? ': null' : '')); // dangling comma / colon
  repaired += closers.reverse().join('');

  try {
    return JSON.parse(repaired);
  } catch {
    return undefined; // caller keeps the last good value
  }
}

It’s deliberately conservative: when repair can’t produce valid JSON, it returns undefined and the caller keeps the previous good state.

Reliability checklist

• Provider-agnostic interface. Hide the model vendor behind one stream() contract so you can swap models, add fallbacks or A/B test without touching product code.
• Timeouts and retries with exponential backoff on the model call, but only before the first byte is sent to the user.
• Log raw model output (with appropriate redaction) so you can reproduce parsing failures instead of guessing.
• Validate after repairing. Repaired JSON is still only syntactically valid; check it against a schema before trusting it.
• Protect the endpoint: authentication, rate limiting and per-user quotas, because streaming endpoints are expensive to abuse.

None of this is exotic, but together it’s the difference between a demo that wows once and a feature people trust every day. If you’re taking an LLM prototype to production and want a second pair of eyes on the architecture, get in touch.