Pub/sub topics and message queues

Event-Driven Architecture (EDA) and publish/subscribe are built-in parts of Zato that that let you use it as a message broker in addition to SOA/API services.

In EDA, business data can be exchanged between applications using a notion of topics and delivery queues. Publishers produce messages that are routed by Zato to subscribers, potentially applying transformations or security rules while flowing through the platform.

Each application exchanging information using publish/subscribe may use a different connection protocol and different security mechanism without influencing each other.

A key difference between publish/subscribe and point-to-point or peer-to-peer integrations is that producers of data don't necessarily need to know who the recipients will be. Conversely, the recipients don't need to know who's producing the information that they are receiving. Moreover, with publish/subscribe, neither producers nor consumers have to be up and running at the same time.

Conceptually, a topic is a named address backed by persistent storage, to which data is sent (published). If you're coming from an SQL background, think of a topic as if it were a table to which rows are written, with each row representing a single message published to the topic.

A counterpart of a topic is a queue. Each consumer of messages, also known as a subscriber, will receive messages not from the topic directly, but from a queue of messages - an ordered collection of all the messages from all the topics it subscribed to.

Each queue is independent of each other - each consumer has its own - and each queue will retain messages for as long as it's needed for the consumer to read them.

Consumers can either pull messages from queues, which means that they will periodically connect to Zato to read their messages, or Zato can push messages to their own API endpoints.

Pub/sub topics

Topic attributes

Each topic in EDA has a name and description. The description is for your convenience, but the name is more nuanced - it can be an exact name that your events will be published to, or it can be a pattern.

• Name of a topic must be unique and can be up to 200 characters long. Only ASCII characters are permitted, and the "#" character is not allowed. Topic names are case-insensitive and serve as the routing key for message delivery.

• Patterns allow flexible topic matching using wildcards.

  • Single wildcard * matches any characters within one topic segment and stops at dots. For example, orders.* matches orders.processed and orders.cancelled, but not orders.processed.daily.

  • Multi-level wildcard ** matches across multiple segments and crosses dots. For example, orders.** matches orders.processed, orders.cancelled, and deeply nested topics like orders.processed.daily.summary.

  • Wildcard placement is flexible - wildcards can appear anywhere in a pattern, including the beginning, like *.orders, or middle - like orders.**.new. The ** wildcard matches zero or more segments from its position.

  • Pattern evaluation follows alphabetical order with exact patterns evaluated before wildcards, and the first match wins.

  • Evaluation order means that if you have patterns transaction.priority and transaction.*, a message to transaction.priority will match the exact pattern first, and evaluation stops there.

  • Patterns are used by both publishers and subscribers, e.g. you can say that you will have a pattern such as hvac.*.alert, but it only means that in principle, any string will fit the * charater, and then your publisher may, for instance, only ever publish to hvac.123.alert, and your subscribers will be still able to subscribe to a subset of this pattern, e.g. only to hvac.456.alert or hvac.789.alert. In other words, that a pattern is used in the topic's name, does not mean that your subscribers are not allowed to subscribe to only a subset of it.

• Publications to topics require the publisher to have matching permissions based on pattern rules. Messages published to a topic are routed to all active subscriptions for that topic.

• Message routing ensures that subscribers receive messages from all topics they are subscribed to through a single message queue per user.

Pub/sub message queues

• Message queues are created automatically for each subscriber when they subscribe to their first topic. Each user has a unique subscription key that identifies their personal message queue, and all messages from all subscribed topics are delivered to that single queue.

• Message delivery follows a priority-based ordering where messages with higher priority (0-9, with 9 being highest) are delivered first. Within the same priority level, messages are delivered in chronological order based on publication time, with oldest messages first.

• Automatic acknowledgment happens when messages are retrieved through the /messages/get endpoint. Once a message is pulled from the queue, it is immediately acknowledged and removed, preventing duplicate delivery.

• Queue persistence ensures that messages remain in the queue until explicitly retrieved by the subscriber. Messages are not lost if the subscriber is temporarily offline, and they accumulate in the queue until the subscriber reconnects and pulls them.

• Pull-based delivery is used for REST API subscriptions where clients call the /messages/get endpoint to retrieve messages on demand. This approach gives clients full control over when and how many messages to retrieve, with configurable limits on message count and total data size.

• Push-based delivery can be configured for subscribers to have messages automatically delivered to their REST endpoints as soon as they arrive, without requiring the subscriber to poll for new messages.

Getting started with EDA

Using EDA