Creating LLM connections
Give every service access to Claude, OpenAI, Gemini or self-hosted models - credentials stay in the platform.
An LLM connection is how your Python services call a model - Claude, OpenAI, Gemini or anything self-hosted that speaks the OpenAI protocol, such as Ollama, vLLM or LiteLLM. The connection holds the provider's address and API key, and services refer to it by name alone, through self.llm['My Connection']. The key is never visible in service code, and swapping one provider for another is a change to the connection, not to any code.
Connections are managed in the Dashboard under AI > LLM connections.
Configuration
| Field | Meaning |
|---|---|
| Name | A unique name for this connection - services look it up by this name through self.llm[name] |
| Active | Whether this connection can be used - services cannot look up an inactive connection |
| Model | The model every call through this connection uses - pick one from the catalog or enter any name by hand |
| Address | The base URL of the API to call - it follows the model you pick and you can point it at any self-hosted or proxy endpoint |
| API key | The key sent with each call - required for hosted providers, while self-hosted endpoints may not need one |
| Pool size | How many HTTP clients the pool keeps ready - each concurrent call checks one out (default: 50) |
| Timeout | How many seconds to wait for the provider's response (default: 60) |
| Max tokens | The most tokens the model may generate per reply, sent to providers that require it (default: 1024) |
| Max history turns | How many past turns of a chat are sent to the provider - a turn is one user message plus the assistant's reply (default: 20) |
| Chat expiry | How many seconds a chat's history is kept after its last message (default: 86400, i.e. one day) |
Once a connection exists, its API key is updated through the Change API key link in its row.
Models and providers
The model you pick decides which protocol the connection speaks - there is no separate provider field.
- Names picked from the catalog, such as "Sonnet 5", resolve to their provider and to the id the provider's API expects on the wire.
- Names entered by hand select the protocol by prefix -
claude-speaks the Claude API,gemini-speaks the Gemini API, and everything else speaks the OpenAI protocol, which is what self-hosted and proxied endpoints use too.
The catalog itself lives in default-models.yaml in the server's user-conf directory and the Dashboard reads it afresh each time, so you can add or remove models without a restart.
Each hosted provider has its default address, filled in automatically when you pick its model:
| Provider | Address |
|---|---|
| Claude | https://api.anthropic.com |
| OpenAI | https://api.openai.com/v1 |
| Gemini | https://generativelanguage.googleapis.com/v1beta |
For a self-hosted model, enter its name by hand and point the address at your endpoint, e.g. http://localhost:11434/v1 for Ollama. The API key can stay empty when the endpoint does not require one.
Ping
The Ping link in a connection's row confirms that the address is reachable and that the key is accepted - it lists the provider's models without generating anything, so a ping costs no tokens.
The audit trail
Every call through an LLM connection is recorded in the audit log - the connection's name, the outcome, the duration and the endpoint, for successful and failed calls alike. This is what the built-in alerting measures error rates and latency over, so a connection that starts failing or slowing down raises an alert without any additional configuration.
See also
| Feature | What it does |
|---|---|
| Invoking LLMs | One-shot calls through a connection, with per-call token usage |
| Multi-turn conversations | Chats whose history the platform keeps and trims |
| Skills | Reusable instructions sent as the system context of a call |
| GitOps | Connections declared in YAML and kept in version control |