Serverless GPU Errors and Rate Limits

The same HTTP failure can mean “nothing was accepted,” “fix the input,” or “work completed but the response was lost.” Your serverless client should record those states separately. Treating every error as a retryable generation request can duplicate expensive work and make wallet reconciliation harder.

Decode the transport before the application error

Inspect status and Content-Type before parsing JSON. Platform errors normally use {"error":{"code":"...","message":"..."}}; field-validation failures may add error.fields. A proxy outage can return HTML or an empty body. Keep the HTTP status even when parsing fails, and capture X-Request-Id when present. An inference result's body ID and the HTTP request ID serve different purposes.

Choose the next action by failure class

Status	Investigation	Safe next action
400	Strict field validation or absent/oversized idempotency key	Correct the body and headers; no unchanged retry.
401 / 403	Inactive or unverified account, invalid key, scope or IP restriction	Repair authentication or authorization using the account controls.
402	Available wallet funds cannot cover authorization	Review balance and reservations before choosing smaller work or adding funds.
404	Unknown public model or unavailable operation	Refresh discovery and check modality; never silently substitute another model.
409	Missing pricing or wallet/model currency conflict	Recheck configuration and current catalog; repeated identical requests do not repair it.
429	Platform admission or upstream rate limiting	Reduce admission and honor Retry-After if supplied.
502 / 503 / 504 or transport timeout	Unusable output, availability failure or uncertain completion	Classify delivery, retain evidence and reconcile before a bounded retry.

Codes add context, but status remains important: platform limiting uses RATE_LIMITED, while upstream inference limiting is translated to a 429 REQUEST_REJECTED. A 503 may also reflect maintenance or disabled serverless availability. The error reference and API contract are the starting points for implementation.

Preserve uncertainty after a timeout

Store a logical operation ID and its idempotency key before sending. Do not rotate that key merely because the connection closed. The handler uses the key in reservation, settlement and upstream request references, but does not expose a general cached-result lookup or guarantee identical response replay. Concurrent duplicate submissions should be prevented in your own application.

Before delivery, the failure path attempts to release the hold. After delivery, a settlement error retains reconciliation state rather than pretending the work was free. Thus a non-200 response is not sufficient evidence for a refund or for immediate resubmission. Use the wallet investigation timeline for unresolved funds, and do not promise a fixed release time.

Build a bounded recovery policy

For transient reads, cap attempts, use exponential backoff with jitter, and stop at the overall deadline. For generation, first resolve whether a retry is appropriate for that operation. Remove canceled queued jobs and distinguish a local timeout from a confirmed server rejection. Redact keys, full prompts and outputs from logs; retain UTC time, route, status, safe code and request ID for support.

Test malformed JSON responses, missing request IDs, rate-limit headers, disconnects after delivery and settlement uncertainty with fixtures before enabling paid traffic. These cases test the recovery state machine without generating real requests or charges.