Alerting

Alerting tells you that something in your environment has gone wrong without anyone having to watch a screen. A connection that stopped answering, an error rate that climbed, responses that became slow, a certificate that is about to expire - each of these is measured continuously and, when it crosses a threshold, sent to you by email, Slack, Microsoft Teams, a webhook, one of your own services or a pub/sub topic.

Zato ships with its rules already written and switched on, so alerting works out of the box. What you do have to do is say where the alerts should go, which is the subject of the notifications page.

Alerting reads the audit log

Alerting measures the audit log, not the Prometheus metrics. Two things follow from that:

• For a channel or a connection that has an Audit log checkbox, switching it off also stops its alerts - the checkbox is that object's alerting switch too. Sources without a checkbox, such as SQL connections, the scheduler or certificate checks, always record and are always measured.
• Every alert links back to the events that caused it.

What is measured

Each rule measures one thing about one kind of object and compares it against a threshold. Four shapes account for most of the rules - a run of consecutive failures, an error rate, a response time and an error rate high enough to be worth an AI diagnosis - and the rest are single-subject rules, such as a certificate nearing expiry or a feed that has gone quiet.

The rules page lists every threshold and how to change it.

When an alert repeats

A condition that stays broken would otherwise alert on every measurement. Instead:

• The first time a condition trips, the alert is raised and delivered.
• If the same condition is still tripping later, the alert is not delivered again for the length of the rule's own deduplication window - an hour, unless the rule says otherwise. The existing alert has its count raised instead, and the message you eventually see is prefixed with the count, for instance [3x] REST outgoing crm.api - 5 consecutive failure(s).
• The window is measured from the most recent occurrence, not the first, so a condition that keeps tripping keeps pushing the window forward and stays quiet until it clears.
• Critical alerts are never suppressed. A connection that is down is delivered every time it is measured.

Each object and each kind of measure is its own alert, so a REST connection's error rate and the same connection being down are suppressed independently of each other.

After delivery

Zato raises alerts and sends them out. There is no screen listing them and nothing to acknowledge, close or assign - that belongs in whatever receives the alert, reached through the webhook action if it is an incident tool.

Learn more