Invoking LLMs from services

Services call models through self.llm, indexed by the name of an LLM connection. The invoke method is a one-shot call - one question, one answer, nothing remembered between calls. For conversations that keep their history, use chat instead.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class Summarize(Service):

    name = 'demo.llm.summarize'

    def handle(self):

        # Look up the connection by its name from the Dashboard ..
        conn = self.llm['My OpenAI']

        # .. and make a one-shot call.
        response = conn.invoke('Summarize this in one sentence: ' + self.request.raw_request)

        self.response.payload = response['text']

The connection's name is the only coupling between your code and the provider - which model answers, at what address and with what key is all configuration, changed in the Dashboard without touching the service.

The response

invoke returns a dictionary:

The usage key uses the same two names regardless of the provider, so the code that reads it does not change when the connection's model does:

response = conn.invoke('What is an integration platform?')

self.logger.info('Reply: %s', response['text'])
self.logger.info('Tokens: %s in, %s out',
    response['usage']['input_tokens'],
    response['usage']['output_tokens'])

Call with a skill

A skill is a reusable set of instructions sent as the system context of the call - name it with the skill parameter:

response = conn.invoke(text, skill='support-agent')

Error handling

A call that fails raises an exception - a timeout, an unreachable address or a provider rejecting the request. When it is the provider that answered with an error, the exception includes the provider's response body verbatim, so the reason - an invalid key, an unknown model, an exceeded quota - is stated in the exception itself.

Failed calls are recorded with their outcome and duration, like successful ones, in the audit trail.

See also