Self-hosted models

One service that runs against a hosted provider in production, an Ollama model on a developer's laptop and a corporate LLM gateway in a regulated environment - with zero changes to the service between the three. The switch lives entirely in the connection's configuration, because the service names the connection and nothing else.

Prerequisites. For the self-hosted case, any OpenAI-compatible endpoint - the example uses Ollama with ollama pull llama3.1 already run.

The service

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class Classify(Service):
    """ Classifies a piece of text as positive, negative or neutral.
    """
    name = 'example.ai.classify'

    input = 'text'
    output = 'label'

    def handle(self):

        conn = self.llm['Classifier LLM']

        prompt = 'Answer with one word - positive, negative or neutral: ' + self.request.input.text
        response = conn.invoke(prompt)

        self.response.payload.label = response['text'].strip().lower()

There is no address, no key, no model and no protocol anywhere in the code - Classifier LLM is the only coupling.

The three connections

The same enmasse entry, three ways. Only one is imported per environment - all three share the name the service looks up:

# Production - a hosted provider
llm:
  - name: Classifier LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Classifier_Key

# A laptop - Ollama, without a key
llm:
  - name: Classifier LLM
    model: llama3.1
    address: http://localhost:11434/v1

# A regulated environment - the corporate LLM gateway proxies every call
llm:
  - name: Classifier LLM
    model: company-approved-model
    address: https://llm-gateway.internal.example.com/v1
    secret: Zato_Enmasse_Env.Gateway_Token

No provider field is needed - the model's name selects the protocol, as described under models and providers, and Ollama, vLLM, LiteLLM and corporate gateways all expose the OpenAI protocol.

Run the service

curl localhost:17010/example/classify -d '{"text":"The delivery was fast and the box was intact"}'

Expected output, from any of the three backends:

{"label": "positive"}

The REST channel for the curl call:

channel_rest:
  - name: example.ai.classify
    service: example.ai.classify
    url_path: /example/classify

Verify a backend

The Ping link in the connection's row lists the endpoint's models without generating anything - it confirms the address is reachable and the key is accepted, and a ping costs no tokens, against Ollama exactly as against a hosted provider.

Failure behavior

Pointing the address at an endpoint that is down makes invoke raise - with an OpenAI-compatible endpoint that answered with an error, the exception contains the endpoint's own response body, so model "llama3.1" not found from Ollama arrives verbatim instead of as a generic failure. The wrong-address case fails at ping time already, before any service runs. To make a service robust against both, see provider failures.

See also