Structured output
Get JSON out of a model that downstream code can rely on - parsed, validated and retried.
A model asked for JSON returns JSON most of the time - and prose around JSON, markdown fences or a missing field often enough to break anything that assumes otherwise. The service below extracts order data from free-form emails and returns a dictionary that downstream code can rely on, because everything the model returns is parsed, validated and retried before any caller sees it.
Prerequisites. An LLM connection named Extractor LLM - any backend from self-hosted models works - and the skill below.
The skill
The skill states the format contract. Keeping the format rules in a skill means tightening them later is a file edit, not a redeployment - config/repo/skills/order-extractor/SKILL.md:
---
name: order-extractor
description: Extracts order data from free-form text as strict JSON
---
You extract order data from customer emails.
Respond with a single JSON object and absolutely nothing else - no explanations,
no markdown fences, no text before or after it.
The object has exactly these keys:
* "customer_name" - a string, the customer's name
* "product" - a string, what they are ordering
* "quantity" - an integer, how many they want
When a value is not stated in the text, use null for it.
The service
# -*- coding: utf-8 -*-
# stdlib
import json
# Zato
from zato.server.service import Service
# The keys the model's reply must contain - the same contract the skill states
_required_keys = ('customer_name', 'product', 'quantity')
# How many times a malformed reply is retried before the service gives up
_max_attempts = 3
class ExtractOrder(Service):
""" Extracts structured order data from free-form text.
"""
name = 'example.ai.extract-order'
input = 'text'
output = 'customer_name', 'product', 'quantity'
def _parse_reply(self, reply):
""" Returns the reply as a dict, or None when it is not the JSON we asked for.
"""
# Models sometimes wrap JSON in markdown fences or prose - taking the slice
# between the first { and the last } recovers the object from both cases.
start = reply.find('{')
end = reply.rfind('}')
if start == -1 or end == -1:
return None
try:
data = json.loads(reply[start:end + 1])
except ValueError:
return None
# A reply that parses but misses a key breaks callers too
for key in _required_keys:
if key not in data:
return None
return data
def handle(self):
conn = self.llm['Extractor LLM']
text = self.request.input.text
for attempt in range(_max_attempts):
response = conn.invoke(text, skill='order-extractor')
data = self._parse_reply(response['text'])
if data is not None:
self.response.payload = data
return
self.logger.info('Malformed extractor reply, attempt %d: %s', attempt + 1, response['text'])
raise Exception(f'No valid JSON after {_max_attempts} attempts')
Run the service
curl localhost:17010/example/extract-order -d '{"text":"Hi, this is Jane Smith, I would like to order two ergonomic keyboards."}'
Expected output:
The REST channel:
channel_rest:
- name: example.ai.extract-order
service: example.ai.extract-order
url_path: /example/extract-order
Failure behavior
The retry loop is the failure path - a reply wrapped in prose or missing a key is logged with its attempt number and asked again, and the caller receives either a valid object or the final exception. Each attempt is a separate call in the audit log, so an extraction that routinely needs two attempts is visible there as double traffic - a measured signal that the skill's wording needs tightening or the model needs changing. Each retry's cost is in the response's usage dictionary - see token usage.
See also
| Feature | What it does |
|---|---|
| Skills | The instruction files that carry the format contract |
| Token usage | What each attempt of the retry loop costs |
| Evaluating AI flows | Structural assertions over responses like this one |