7 · Streaming with heartbeat

"stream":true gets normal SSE chunks plus : ping keep-alives on long turns; SDKs ignore the pings.

What it is

Standard OpenAI-shaped SSE streaming, plus periodic SSE comment lines (: ping, not a data: event) injected while an upstream — especially a slow CLI agent or a pull-worker — is still thinking and hasn't produced its next chunk yet.

Why

Idle proxies, load balancers, and some HTTP clients drop a connection that goes quiet for too long. A CLI agent turn can easily take longer than that idle timeout between real content chunks. The heartbeat keeps the connection alive without changing the response shape — SSE comment lines are invisible to any spec-compliant SSE/OpenAI client parser.

Configure

models.yaml

stream_heartbeat: 15s   # default ~15s; 0 disables heartbeat comments entirely

stream a slow agent turn

curl -N localhost:8080/v1/chat/completions \
  -d '{"model":"devin/claude-opus-4.8","stream":true,
       "messages":[{"role":"user","content":"long task"}]}'
# → data: {...} chunks, with ": ping" comment lines every ~15s while it thinks

The heartbeat interval is configurable via stream_heartbeat in models.yaml; the OpenAI SDK's SSE parser silently discards comment lines (anything starting with :), so no client code needs to change to tolerate it.

How to adapt

Tune stream_heartbeat to sit comfortably under whatever idle timeout sits in front of routsi (a corporate proxy, an ALB, nginx) — shorter if that timeout is aggressive, 0 to disable entirely if nothing in the path times out. This applies uniformly whether the upstream is a raw forward passthrough, a CLI agent (streaming is faked off its one-shot Complete call), or a pull-worker.

Next: Dynamic group routing →