Tracing one call

"The agent got a wrong answer at 14:32" is only debuggable if one identifier connects everything that request touched. In Zato that identifier is the CID - the correlation ID assigned when a request arrives and passed through every layer it crosses. The trace below follows one agent request end to end: the audit event, the server log, the tool's own logging, the response shaping trace and the LLM call the tool made along the way.

Prerequisites. An MCP gateway with its audit log on and a tool that itself invokes an LLM - a summarization tool is used here, but any tool traces the same way.

The report

An agent called the example.docs.summarize tool and the user reports the summary looked cut short. The five artifacts below share one CID.

The audit event

The Dashboard's MCP audit page, filtered to the gateway around 14:32, has the request:

The audit log already answers half the report - the call succeeded and took the time duration_ms in its data document records. The CID connects everything else below.

The shaping trace

The same event's data document holds the trace of what response shaping did to this one response - one key per finding, with nothing written for stages that did nothing:

{
    "method": "tools/call",
    "duration_ms": 2140.4,
    "request_size": 388,
    "pii_removed": 2,
    "tokens_before": 4680,
    "tokens_after": 1000,
    "was_truncated": true
}

This answers the report - the tool produced an estimated 4680 tokens, the gateway's size cap truncated the response to its 1000-token limit, and two pieces of PII were replaced on the way. Since untouched stages write nothing, what is listed is exactly what happened.

The server log

The same CID finds every line the request produced in the server's log:

grep 9f0da4306ffd7e6a341bce60 ~/env/qs-1/server1/logs/server.log

Every line a service logs through self.logger includes the request's CID automatically, so the tool's own logging is already part of the trace with no code written for it - and anything downstream the service invoked logged under the same CID too.

The LLM call

The summarization tool called a model to do its work, and that call is an event of its own in the LLM audit trail - the connection, the provider's endpoint, the outcome and how long the provider took, under the same CID. Comparing its duration with the MCP event's duration_ms splits the 2140 ms into the provider's share and the platform's share - whether the time was spent in the platform or in the model is answered by subtraction.

Other entry points

The trace also runs backwards. When the starting point is an alert - say the gateway's error rate fired a rule - the alert's evidence pack contains the recent audit events, each with its CID, and any one of them enters the same path at the audit event above. And when the starting point is the agent's user, the response the agent received is the only artifact without the CID on it - which is why the timestamp, the gateway and the tool name are enough: the audit page narrows those three to the event, and the event has the CID.

Failure behavior

A request that fails authentication has the shortest trace - an auth-failed audit event and a rejection line in the server log, both with the CID, and nothing else, because nothing else ran. A trace that stops early shows where the request stopped - the deepest layer with the CID is the layer that refused it.

See also