A streaming AI agent that shows its reasoning live — every token, tool call, latency and cost rendered in real time as the model thinks. Here's how it works under the hood.
It's a chat app split into two panes: a normal chat on the left, and a live trace panel on the right. While the assistant answers, the trace panel reveals what's normally hidden — when the model is reasoning vs. responding, which tools it calls, how long each step takes, how many tokens it burns, and the running cost in USD.
Every internal step of the ReAct loop is surfaced as a typed event and drawn as a card with status, duration and reasoning.
Server-Sent Events push tokens the instant the LLM produces them. No polling, no waiting for the full answer.
AI, protocol, API and UI live in separate folders with a one-way import rule — no cross-layer shortcuts.
Code is organized by responsibility, and dependencies flow in exactly one direction. The UI knows about the API; the API knows nothing about the UI; the AI layer knows about neither.
StreamEvent discriminated union plus encode/decode helpers — generatorToStream() on the server, parseSSE() on the client.streamChat()) and the route handler that runs session, validation, rate-limit and moderation before streaming.useAgentStream hook that turns the event stream into reducer state. A single "use client" boundary at ChatContainer.STREAM_EVENT, TRACE_STATUS, GRAPH_EVENTS) live in as const objects so logic never depends on raw string literals.From keystroke to rendered token, here's the full round trip. Colors mark the segment: client, server pipeline, model loop, stream back.
SEND_START: adds the user message and an empty streaming assistant bubble, flips isStreaming on.{ message, history } to /api/chat, attaching API-key headers from localStorage and an AbortSignal for the stop button.createAgent() builds a ReAct graph; graph.streamEvents() (v2) fires LangGraph events. Handlers translate each into a StreamEvent.web_search / get_current_datetime, sees results, then streams the answer (label "Responding"). Loops up to 4 iterations.data: <json>\n\n. A DONE frame carries total latency, TTFT, token counts and estimated cost.applyStreamEvent() switches on the event type: append a token, open a trace step, close one with its duration, or record the final metrics.sessionStorage so a refresh keeps the conversation.Every server→client message is a single SSE frame: data: <JSON>\n\n. The JSON is one variant of the StreamEvent discriminated union — nine event types, each with a typed payload. A switch on event.type stays exhaustive at compile time.
| Event | Fires when… | Key payload |
|---|---|---|
| model_start | A model call begins | modelCallId, label |
| token_delta | Each reply token (reasoning excluded) | content |
| model_end | A model call finishes | durationMs, reasoning? |
| tool_start | A tool is invoked | toolName, toolCallId, args |
| tool_result | A tool returns | result, durationMs |
| step_end | A ReAct step closes | stepIndex (marker) |
| moderation | Content check completes | blocked, reason?, durationMs |
| done | Run complete | latencyMs, ttftMs, inputTokens, outputTokens, estimatedCostUsd, truncated? |
| error | Recoverable failure | message |
The agent is an async generator yielding SSE strings. generatorToStream() wraps it in a Web ReadableStream, TextEncoder-ing each chunk and emitting an error frame if the generator throws.
parseSSE() reads the body reader, buffers partial UTF-8, splits on the \n\n frame boundary, strips data: , and JSON.parses each into a typed event — dropping malformed frames safely.
A LangGraph ReAct agent built with createAgent() from the langchain package (the prebuilt createReactAgent is deprecated). The LLM is a swappable adapter — pick it right in Settings → Agent provider (sent per request as x-llm-provider, with LLM_PROVIDER as the server default): Fireworks via the OpenAI-compatible ChatOpenAI, or Anthropic Claude via ChatAnthropic. Both plug into the same graph and stream identically; the agent is given two tools.
tool_calls (graph routes to the tools node and goes around again) or doesn't (graph routes to END and the run finishes). The for await in runAgentStream() is only a consumer of the resulting event stream — it narrates the loop, it doesn't control it. The hard stop is the graph's recursionLimit, not the model's discretion.Tavily search, Zod-typed { query }. Returns Tavily's synthesized answer plus up to 3 snippets.
No arguments. Returns a UTC timestamp so the model can reason about "today" without hallucinating dates.
A model call is split into two visible phases — the agent relabels the step from "Reasoning" to "Responding" on the first content token. Reasoning tokens are accumulated and shown in the trace only, never mixed into the reply.
All client state lives in useAgentStream, backed by a single useReducer. The reducer is the only place state changes — the stream just feeds it actions.
applyStreamEvent()patchById() let memoized cards skip re-renders.sessionStorage (per-tab, survives refresh); API keys go to localStorage (shared across tabs). Nothing sensitive touches the server beyond the request itself.Two independent defense layers run before the model ever sees a request — and both degrade gracefully if their backing service is missing.
1. Fast regex jailbreak patterns ("ignore previous instructions", DAN, etc.) run synchronously. 2. If they pass and an OpenAI key exists, the text goes to the Moderation API. Fails open — unavailability never blocks a legit user.
Production: Upstash Redis sliding window keyed by session. Dev: in-memory bucket map with an unref'd sweep timer. Demo keys get a tighter budget than your own keys.
A core rule of the codebase: no magic numbers or strings. Every constant lives in one config file or an as const object.
Beyond the live trace, every run is measured, optionally traced to LangSmith, and the golden path is guarded by an end-to-end test.
The done event carries TTFT, total latency, token usage and estimated USD cost — computed server-side in graph.ts from per-provider pricing tables. Per-step durations ride each model_end / tool_result.
Because it's a LangChain app, setting the LANGSMITH_* env vars auto-traces every run — no code. Runs are tagged with provider + model; in dev the trace panel shows a "View in LangSmith" deep link.
A CI-gated test drives the real UI against a production build and mocks /api/chat with canned SSE frames — covering send → tool-call → stream → replay with no LLM or network calls.