Serverless GPU Streaming Chat Inference

Hostnot GPU's current public chat endpoint does not support streaming. It accepts stream: false or an omitted stream field and rejects stream: true. This article explains how to build a responsive completed-response client, and how a future streaming implementation would differ. The title is a topic, not an availability promise.

Confirm the wire contract

Chat generation uses POST https://serverless.hostnotgpu.ae/v1/inference/chat with bearer authentication, inference:write and an Idempotency-Key. On success it returns JSON with object: inference.result and output: {type: "text", text: "..."}. There are no documented incremental token events or final SSE usage event on this route.

Neither a chat-shaped messages array nor model discovery implies streaming support. The catalog publishes modality and context metadata, not a streaming capability flag. Use the current endpoint contract rather than enabling an SDK's streaming option and hoping the model will accept it.

Make completed responses usable in a browser

1. Authenticate the browser to your application; keep the platform key in your backend secret store.
2. Validate input and admit the request through a bounded queue. Store an operation ID before dispatch.
3. Show a truthful pending state such as “Waiting for response.” Do not invent percentages or present an animation as actual generation progress.
4. Apply an overall deadline covering queue wait and HTTP time. If the caller leaves, remove unsent work and mark sent work as canceled locally or outcome-unknown as appropriate.
5. Validate the completed JSON, then display output as text or safely rendered markup under your own output policy.

Artificially revealing a completed response character by character is a presentation effect. It does not reduce backend generation time or provide real token streaming. Preserve an accessible nonanimated presentation and a useful error state.

Disconnection is not remote cancellation

Aborting fetch can stop your client from waiting; it does not establish that the remote model stopped or that no charge occurred. The current public contract does not provide a generation cancellation endpoint. Keep the original operation key and request ID where available, and reconcile uncertain outcomes before allowing a fresh paid retry. See error and delivery handling.

If your product requires real token events

Evaluate a self-managed serving endpoint whose exact version documents streaming, using the vLLM deployment guide as an operational starting point. That introduces responsibility for TLS, authentication, admission, process health and GPU lifecycle. Do not infer that renting a GPU automatically installs a secure streaming service.

For such a separate service, read the SSE protocol guide: event framing is independent of network chunks, partial lines must be buffered, and a connection ending is not inherently a successful completion. Define final-result, error, usage and reconnection semantics explicitly. These are requirements for an implementation you control, not features currently offered by Hostnot GPU chat.