LLM Failover: Keep RAG Apps Answering

Retrieval-augmented generation, or RAG, applications depend on language model APIs to turn retrieved documents into answers. Those APIs are external services with rate limits, occasional slowdowns and outages. When the model call fails, a support assistant stops answering, an internal knowledge tool returns errors and users lose trust quickly. LLM failover is the set of techniques that keep a RAG application useful when its primary model is unavailable or overloaded: respecting rate limits, retrying sensibly, switching to backup models, serving cached answers and degrading gracefully with honest messages. This guide explains how to design these layers and how to test them before an outage tests them for you.

Where RAG applications fail

A RAG request usually involves several steps, each of which can fail:

• Embedding the user's question
• Searching the vector or keyword index
• Calling the language model with the question and retrieved passages
• Post-processing and returning the answer

Model calls are often the slowest and most failure-prone step, because they depend on an external provider and can be limited by request rates, token rates or overall quota. Failures also happen in embedding calls and search services, so failover planning should cover the whole chain.

Respect rate limits first

Many failures are self-inflicted: the application sends more requests than its limit allows. When this happens, APIs commonly return HTTP 429 Too Many Requests, which MDN describes as indicating the client has sent too many requests in a given amount of time (MDN 429). A Retry-After header may indicate how long to wait (MDN Retry-After).

Good practice includes:

• Tracking your own request and token usage against provider limits
• Queueing or smoothing bursts rather than sending them all at once
• Honouring Retry-After when present
• Separating limits for interactive and batch work, so background jobs do not starve users

Retry with backoff and jitter

Temporary failures, such as timeouts or brief overloads, often succeed on a second attempt. Retry with exponential backoff, which waits longer after each failure, and jitter, which randomises waits so many clients do not retry at once. Limit the number of retries and the total time, because users will not wait indefinitely. Do not retry errors that will not change, such as invalid requests.

Set timeouts deliberately

Without timeouts, a slow model call can hold a request open until the user gives up. Set a timeout for each model call based on how long users will reasonably wait and how long the model normally takes for your typical prompt size. Streaming responses can help users see progress, but they still need an overall limit. When a call times out, treat it as a failure for the purposes of retries and circuit breakers, and move on to the next option rather than waiting again on the same slow service.

Use circuit breakers

If a provider is failing consistently, continuing to call it wastes time and adds load. A circuit breaker monitors failures and, once they reach a threshold, trips so further calls return an error immediately without making the call, as Martin Fowler describes (Martin Fowler: CircuitBreaker). After a pause, it allows a test call through to check whether the service has recovered. In a RAG application, a tripped breaker is the signal to move to a fallback.

Add fallback models

A fallback model is a second model, from the same provider or a different one, used when the primary is unavailable. Before relying on one:

• Test answer quality on your own questions, since models differ
• Adjust prompts if the fallback needs different formatting
• Check context window size, so retrieved passages still fit
• Confirm data handling terms meet your requirements for each provider
• Consider cost differences at your volumes

A smaller or cheaper fallback may give shorter or less nuanced answers, which can be acceptable during an outage if users are told.

Serve cached answers

Many RAG questions repeat. Caching answers to common questions lets the application respond even when models are unavailable, and reduces cost the rest of the time. Exact-match caching is simple; semantic caching matches similar questions but needs care to avoid returning wrong answers. Our guide to LLM semantic caching explains how to set thresholds and keep cached answers fresh.

Degrade gracefully

When no model is available, the application can still help:

• Show the most relevant retrieved passages with links, so users can read the source
• Offer cached answers to similar questions, clearly labelled
• Explain that answers are temporarily limited and suggest trying again later
• Route urgent requests to a person

A clear, honest message is far better than an error page or a confident answer from a model that should not have been used.

Worked example: LLM failover sequence

The Python sketch below shows the order of fallbacks for one request. It is a simplified, untested outline; functions such as call_model, retrieve and cache_lookup stand in for your own code.

# Simplified, untested sketch of a failover sequence
def answer(question):
    passages = retrieve(question)  # search your index first

    for model in ["primary-model", "fallback-model"]:
        if breaker_open(model):
            continue
        try:
            reply = call_model(model, question, passages, timeout=20)
            record_success(model)
            return {"answer": reply, "mode": model}
        except RateLimited as err:
            wait_or_queue(err.retry_after)  # honour Retry-After where given
        except (Timeout, ServiceError):
            record_failure(model)  # may trip the circuit breaker

    cached = cache_lookup(question)
    if cached:
        return {"answer": cached, "mode": "cached"}

    return {"answer": None, "mode": "sources_only", "sources": passages}

The response includes a mode, so the interface can tell users when they are seeing a fallback, a cached answer or sources only.

Handle retrieval failures too

If the search index is unavailable, the model cannot ground its answers. Avoid letting the model answer from general knowledge in that case, as answers may be wrong or out of date. Return a clear message instead, or a cached answer if one exists. Our guide to private RAG covers how retrieval and access controls are designed in private deployments.

Monitor and alert

Track for each model and step:

• Request counts, latency and error rates
• Rate limit responses and Retry-After values
• Circuit breaker state changes
• Share of answers from fallback, cache or sources-only modes
• User feedback during degraded periods

Alert when fallback use rises or breakers trip, so the team knows the application is degraded even if users are still being served.

Plan capacity for peaks

Some failures are predictable. Product launches, month-end and internal training days can all multiply usage. Check whether your provider limits cover expected peaks, ask about higher limits in advance where possible, and consider precomputing answers to likely questions before a known busy period.

Test failover regularly

Failover that has never run may not work. In a test environment, simulate rate limits, timeouts and outages for each model and service, and confirm the application behaves as designed. Repeat after changing models, prompts or providers.

Frequently Asked Questions

What is LLM failover?

It is the set of techniques that keep an AI application working when its primary model is unavailable, such as retries, circuit breakers, fallback models, cached answers and graceful degradation.

How should a RAG app handle rate limits?

Track usage against limits, smooth bursts with queues, honour Retry-After headers and keep interactive traffic separate from batch work.

Is a fallback model always a good idea?

Usually, but test its answer quality, prompts, context size, data handling terms and cost first, and tell users when it is in use.

What should users see when no model is available?

An honest message, plus relevant source passages or labelled cached answers where available, and a route to a person for urgent requests.

Keep your AI applications answering

We design RAG applications with rate limit handling, fallback models, caching, graceful degradation and monitoring. See our AI agent orchestration and RAG service, or talk to us about LLM failover.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.