Monitoring with Datadog

Datadog reads Zato's Prometheus endpoint directly - the Datadog Agent ships with an OpenMetrics check that scrapes any Prometheus-format endpoint, so no Prometheus server of your own is needed in between.

Before starting, make sure the endpoint itself works - the Prometheus guide shows how to verify it with one curl command and where its password lives.

Configuring the Agent

On the host or in the cluster where the Datadog Agent runs, create the OpenMetrics check configuration, e.g. in /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml:

instances:
  - openmetrics_endpoint: http://zato-server:11223/metrics
    namespace: zato
    auth_type: basic
    username: metrics
    password: your-zato-metrics-password
    metrics:
      - zato_.*

Then restart the Agent and confirm the check is running:

sudo systemctl restart datadog-agent
sudo datadog-agent status

The output's checks section should list openmetrics with no errors. In a Kubernetes cluster, the same configuration goes into the check's autodiscovery annotations or a ConfigMap, per your cluster's Datadog setup.

Two details worth knowing:

• namespace: zato combined with the metric names already starting with zato_ means the metrics appear in Datadog as zato.zato_rest_channel_requests_total and so on. If you prefer shorter names, remap them with the metrics mapping syntax, e.g. - zato_rest_channel_requests_total: rest.requests
• Counters arrive in Datadog as .count metrics and histograms as .bucket, .sum and .count series, which is Datadog's standard Prometheus translation

Finding the metrics

After a minute or two, the metrics appear in Metrics -> Summary - type zato into the search box:

Every label from the Prometheus endpoint becomes a Datadog tag, so channel_name, service_name, status_code, error_source, topic_name and job_name are all available for filtering and grouping.

Building a dashboard

Create a new dashboard and add widgets grouped by the tags:

• Request rate - zato.zato_rest_channel_requests_total.count, summed by channel_name
• Error ratio - the same metric filtered to status_code IN (4xx, 5xx), divided by the total, as a query value widget with thresholds
• Error source - the same metric filtered to error_source NOT IN (none), grouped by error_source, which separates failures of your services from failures of the external systems they call
• Latency distribution - zato.zato_rest_channel_request_duration_seconds.bucket, grouped by le, or as a percentile if you remap the histogram to a Datadog distribution

The finished screen gives the whole team one place to watch API traffic:

Monitors

A metric monitor on the error ratio is the natural first alert - trigger when errors exceed 1% of traffic over 5 minutes, grouped by channel_name so each channel alerts independently. The error_source tag belongs in the alert message, because it answers the first question anyone asks when paged - was it us or the system we call.

Alerts can also stay inside the platform entirely, with the built-in alerting engine and its delivery by email, Slack, Microsoft Teams or webhooks.

Logs

Metrics are half of the Datadog picture - the logging chapter shows how to ship Zato's server and HTTP access logs to Datadog as well, so dashboards and log search meet in one place.

Learn more