Serverless GPU Rate Limits and Quotas

A rate limit governs admission, while latency and concurrency govern how many requests remain in flight. Keeping request rate under a nominal allowance is insufficient if long generations accumulate faster than they finish. A reliable Hostnot GPU client needs a paced queue, bounded active work and a retry budget.

Understand the current limiter contract

The platform resolves requests-per-minute settings using a key-specific override, then a customer override, then the configured category default. Serverless generation uses the serverless category; model discovery uses the default API category. No single published RPM number is a permanent allowance for every account. Bearer-key admission is tracked by key identity, while session requests use customer-session identity; do not create more keys to evade limits.

A platform limiter rejection returns HTTP 429, error.code: RATE_LIMITED and Retry-After. An upstream inference rate-limit rejection can also be HTTP 429 but use REQUEST_REJECTED without that header. Therefore branch first on HTTP status and treat the header as optional. The API reference describes the public envelope.

Pace arrivals and bound concurrent work separately

Put all workers sharing a credential behind coordinated admission. A per-process limiter does not control the aggregate rate of several replicas. Cap active requests independently of the arrival rate, set a finite queue length, and reject or defer jobs whose queue wait would exhaust their end-to-end deadline. Include retries in the same queue rather than letting a second retry loop bypass admission.

Measure queue wait and HTTP duration separately. Growing queue wait with stable server latency points to excess local arrivals; rising server latency and 429s suggest reducing active work. Track completed useful responses rather than treating every accepted attempt as successful throughput. Do not infer token-per-minute quotas from a requests-per-minute setting: application token and spending controls are additional safeguards you implement.

Honor delay guidance without retry storms

Retry-After can be delay-seconds or an HTTP date. Parse either form, reject malformed values, and wait at least the valid server delay. When absent, use capped exponential backoff with random jitter. If the requested delay exceeds the job deadline, end the attempt rather than clipping the delay and retrying early.

Retrying a billable POST requires more than a timer. Preserve the same logical operation and idempotency key, and reconcile ambiguous delivery before resubmitting. Authentication, schema, balance and model-configuration errors require correction, not automatic retries. Cancellation should remove queued jobs immediately; aborting an active HTTP request does not guarantee cancellation of remote compute.

Prove behavior with local fixtures

• Simulate 429 with and without Retry-After, including a future HTTP date.
• Simulate a queue that outlives the caller, and verify that expired work is never dispatched.
• Run several local worker fixtures against one shared limiter and assert aggregate admission.
• Simulate timeout after delivery and ensure the client records an uncertain outcome instead of generating a fresh operation key.

For a capacity review, provide safe request IDs, times, endpoint, arrival rate, peak active count and error distribution through support. See backend limiter design for application-side fairness and spending controls.