Provider failures

Providers time out, reject keys, hit quotas and return 429s - and a service that does not handle it fails silently or hangs. The service below answers from a primary connection and falls back to a secondary one when the primary fails - the caller receives an answer or an error within the configured timeouts, and every failed call still lands in the audit log, where the built-in alert rules measure it.

Prerequisites. Two LLM connections, Primary LLM and Fallback LLM - two providers, or one provider and a self-hosted model as the fallback.

The exceptions

A failed call raises an exception - there are no error-shaped successes to check for. The exception states the provider's reason verbatim, and a provider that does not answer raises after the connection's timeout, as error handling describes.

The service

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class ResilientAsk(Service):
    """ Answers from the primary connection, falling back to the secondary one.
    """
    name = 'example.ai.resilient-ask'

    input = 'question'
    output = 'answer', 'served_by'

    def handle(self):

        question = self.request.input.question

        # The primary connection answers under normal circumstances ..
        try:
            response = self.llm['Primary LLM'].invoke(question)
            self.response.payload.served_by = 'primary'

        except Exception as e:

            # .. the exception's text is the provider's own reason - log it,
            # because "provider failed" without the reason is not actionable ..
            self.logger.warning('Primary LLM failed, using the fallback: %s', e)

            # .. and the fallback answers instead. If it fails too, the exception
            # propagates - a caller that gets HTTP 500 and retries later is better
            # off than one that gets a made-up answer.
            response = self.llm['Fallback LLM'].invoke(question)
            self.response.payload.served_by = 'fallback'

        self.response.payload.answer = response['text']

The configuration

Short timeouts on both connections - an interactive caller cannot use an answer that took a minute to fail:

llm:
  - name: Primary LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Primary_Key
    timeout: 15

  - name: Fallback LLM
    model: llama3.1
    address: http://llm-fallback.internal.example.com:11434/v1
    timeout: 15

channel_rest:
  - name: example.ai.resilient-ask
    service: example.ai.resilient-ask
    url_path: /example/resilient-ask

Run the service

curl localhost:17010/example/resilient-ask -d '{"question":"What is an integration platform?"}'

Expected output with the primary healthy - "served_by": "primary". To see the failure path without waiting for a real outage, deactivate Primary LLM in the Dashboard or break its key - the same call now answers with "served_by": "fallback" and the server log shows the provider's refusal:

Primary LLM failed, using the fallback: OpenAI request to `https://api.openai.com/v1/chat/completions` failed with HTTP 401 (Primary LLM)

The audit trail and alerts

The service's try and except covers the caller's side, and the platform's side needs no code. Every call through either connection, failed calls included, lands in the audit trail, and the built-in alert rules measure those events. A primary connection that fails on every call while the fallback serves all traffic is exactly the pattern the rules catch.

See also