LLM API Rate Limits, Timeouts, and Failover

Updated 9 November 2026 · Standard formula, shown below

Handling rate limits, timeouts, and provider failures well keeps an LLM-powered feature responsive without sending duplicate requests or hiding persistent problems. You’ll need an API client, a way to configure timeouts and retries, and—if you want failover—a second compatible provider or deployment with its own credentials.

What You’ll Need

Your provider’s documentation for rate limits, retry guidance, timeout behavior, and error responses.
An application-level HTTP client that exposes status codes, response headers, and timeout settings.
Securely stored credentials for each provider or deployment you plan to use.
Logging and metrics for request duration, outcome, retry count, and provider selection. Avoid logging secrets or sensitive prompts.

Step-by-Step

Classify failures before retrying. Treat HTTP 429 as rate limiting; use the provider’s retry-after information when available. Transient network errors, 502, 503, and 504 may be retryable. Authentication errors, invalid requests, unsupported models, and most other 4xx responses usually need a configuration or input fix, not another attempt. Provider-specific guidance takes precedence.
Set a total request deadline. Decide how long the user-facing operation may take, then configure connection and response timeouts within that budget. Include time spent waiting between attempts: three 10-second attempts with delays can exceed a 30-second deadline. Stop when the total deadline is reached, even if retry opportunities remain.
Use bounded retries with exponential backoff. Retry only failures likely to clear on their own. Increase the delay after each failure—for example, roughly 0.5, 1, then 2 seconds—and add random jitter so many clients do not retry together. Set a small maximum number of attempts and cap each delay. If the server supplies a valid Retry-After value, respect it only when it fits your deadline; otherwise fail or move to an allowed fallback.
Check whether the request is safe to repeat. A timeout does not prove the server failed to process the request; it may have completed after the client stopped waiting. Repeating a generation can incur extra usage or produce a different answer. For tool calls or other actions with side effects, use an idempotency mechanism if supported, or make the action itself safe to repeat. Do not automatically replay a partially streamed response as if it were a fresh continuation.
Prevent retry storms. Apply a concurrency limit or queue so a traffic surge does not create unlimited simultaneous calls. Track 429 responses and temporarily slow or pause traffic for the affected provider, model, or account. A circuit breaker can stop calls to a repeatedly failing route for a short interval, then allow a limited test request before reopening it.
Add a controlled fallback route. Choose a secondary model or provider that can handle the same task, and verify its credentials, request format, limits, and output handling in advance. Fail over on defined transient failures, such as repeated 5xx errors or an open circuit—not on every error. A 429 may mean the account or shared quota is exhausted, so switching routes helps only if the fallback has independent available capacity.
Preserve application behavior during failover. Make fallback differences explicit in your design: models can vary in latency, context capacity, tool support, output format, and quality. Validate structured outputs and tool-call arguments regardless of route. Do not silently claim that a fallback response has the same capabilities or guarantees as the primary response.
Return a clear outcome when recovery fails. If the deadline expires or all routes fail, return a controlled, retryable error to the caller rather than hanging or returning a misleading partial answer. For streaming, clearly signal interruption according to your application protocol. Include a request or trace ID so the failure can be investigated.
Measure and test the policy. Record latency, status, timeout type, attempts, retry delays, and selected route. Use dashboards or alerts to spot rising 429s, exhausted deadlines, and fallback spikes. Test simulated 429, slow responses, connection failures, and provider outages; confirm that retries stop at the intended limit and that failover does not duplicate side effects.

Common Mistakes

Retrying every error, including invalid input or bad credentials. These failures generally persist until something changes.
Retrying indefinitely or using fixed, synchronized delays. This can worsen an outage and overload a recovering service.
Setting a per-attempt timeout but forgetting an overall deadline. The user may wait far longer than the intended limit.
Assuming a timeout means the request was never processed. That can lead to duplicate generation, charges, or side effects.
Failing over on every 429 without checking quota scope. The secondary route may share the same limit or may be unavailable too.
Logging full prompts, API keys, or sensitive outputs while trying to diagnose failures. Keep telemetry limited and protect any retained data.

Tips

Keep retry policy configurable by provider and operation; different endpoints may document different limits and retry behavior.
Use jittered backoff and a bounded queue. If your application can defer work, queueing may be better than asking users to wait through retries.
Keep primary and fallback credentials separate, restrict access, and rotate them using your normal secret-management process.
Track fallback frequency as a reliability signal. A fallback that is constantly active may conceal a primary-route problem rather than solve it.
For streaming, set limits for time to first token as well as total duration, and decide in advance how interrupted streams appear to users.

Troubleshooting

429 responses continue after backoff: Check the response headers and provider guidance, then confirm whether the limit is per model, project, account, or shared across workers. Reduce concurrency or request volume; retries cannot create capacity.

Requests time out despite a generous timeout: Separate connection time from time to first response and total generation time. Check network connectivity, queue wait, provider latency, and whether the chosen model or request size is appropriate. Ensure an outer proxy or job runner is not imposing a shorter deadline.

Failover does not activate: Verify that the failure is classified as retryable, the circuit-breaker state permits routing, and the secondary credentials and request format work. Check whether the fallback shares the same quota or network dependency.

Users receive duplicate actions or inconsistent results: Stop blindly replaying timed-out operations. Add idempotency protection for side effects, preserve request IDs, and distinguish a complete response from a partial stream.

Latency rises after adding retries: Inspect attempts per request and total deadline consumption. Reduce the retry count, increase backoff where appropriate, add concurrency controls, or fail quickly to a tested fallback instead of stacking long waits.

Frequently Asked Questions

How many times should an LLM API request be retried?
There is no universal number. Use a small bounded limit that fits your latency budget and follow the provider’s guidance. Retry only transient failures.

Should I retry a 429 response?
Usually, after waiting as directed by Retry-After or provider documentation. If the wait exceeds your deadline, return a controlled failure or use a fallback with genuinely separate capacity.

What timeout should I set?
Set it from the user-facing deadline and expected operation, including connection setup, queueing, response time, and any retries. Streaming requests may need separate first-token and total-duration limits.

Can I fail over to a different model?
Yes, if it supports the required inputs and output behavior and has available capacity. Validate its responses because model capabilities and results can differ.

Is it safe to retry after a timeout?
Not always. The provider may have completed the request even if your client did not receive the response. Protect side effects with idempotency and account for possible duplicate usage.

Should a fallback happen after every provider error?
No. Correct non-retryable errors such as invalid requests or authentication failures first. Reserve failover for defined transient failures and outages.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.