Streaming

Set stream: true. Frames are the OpenAI format, forwarded as the GPU produces them.

stream = client.chat.completions.create(
    model="qwen/qwen3-8b",
    messages=[{"role": "user", "content": "Count to ten."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage:
        print(f"\n{chunk.usage.total_tokens} tokens")

Usage in a stream

With stream_options.include_usage set, the last frame before [DONE] carries an empty choices array and a populated usage object. That is the OpenAI convention, and the SDKs handle it — but your own parsing has to expect a frame with no choices.

Keep-alive comments

During a long time-to-first-token — a cold model, or a very long prompt — we emit SSE comment lines every 15 seconds:

: ping

These keep intermediaries from treating a slow start as a dead connection. Every compliant SSE client ignores comment lines automatically; if you’re parsing the stream by hand, skip lines beginning with :.

Frames split across packets

A single SSE frame is routinely split across TCP reads. If you’re writing your own parser rather than using an SDK, buffer until you see a blank line (\n\n) before parsing — a parser that assumes one read is one frame works in testing and silently drops tokens under load.

Cancellation

Closing the connection aborts generation immediately and frees the GPU. You are billed only for what was produced. In Python, break out of the loop and call stream.close(); in TypeScript, use an AbortController.

const controller = new AbortController();

const stream = await client.chat.completions.create(
  { model: 'qwen/qwen3-8b', messages, stream: true },
  { signal: controller.signal },
);

// Stop generating — and stop being billed — immediately.
controller.abort();

Errors mid-stream

If an upstream fails after tokens have started flowing, we cannot re-route: splicing two generations together would produce output that is wrong in a way you can’t detect. You receive a frame containing an error object and the stream ends without [DONE].