Delivery retries and dead-letter queues

Every outgoing REST, SOAP, HL7 FHIR and HL7 MLLP connection has a Delivery tab in the Dashboard. The settings on that tab determine what happens to a request that could not be delivered:

• It's retried right away
• It waits in the connection's queue until it goes through
• It moves to the dead-letter queue (DLQ), where you can look at it, retry it or discard it

REST and SOAP channels have the same tab, and with the queue on, a channel accepts each request at once and runs its service later, from the queue, see the queue on a channel.

What happens to a failed request

You can configure retries in 3 ways:

1. Retries - the call is retried a few times, with a growing wait in between, and your service waits for the outcome
2. The queue - the call is made once and, if it failed and retries are set, the request is put in the connection's queue and your service moves on, the queue then retries it in rounds until it goes through
3. The dead-letter queue - a request that a round could not deliver is set aside in the DLQ, from where it's retried later, forwarded to a topic, discarded or kept for you

By default, nothing is turned on, a failed call is reported to your service and nothing is retried.

Retries

A request is retried when:

• The remote system is unreachable
• It did not answer in time
• It answered with HTTP 429
• No HL7 MLLP acknowledgment came back

Everything else, e.g. an HTTP 500 or a negative HL7 acknowledgment, is a response and reaches your service as it arrived.

In the Delivery tab, click the On failure line to set the retries:

Each next wait is the previous one times the multiplier, 1 second at least, and no wait goes past 8 seconds or past what is left of the total.

This is how long a request that keeps failing waits between its attempts:

• With the defaults, the request is sent once and is not retried
• 5 retries, a 2-second first wait, a multiplier of 2 and 60 seconds in total - the waits are 2, 4, 8, 8 and 8 seconds, so the request is sent six times
• The same with 10 retries - the waits are 2, 4, 8, 8, 8, 8, 8, 8 and 6 seconds, 60 seconds together, so the tenth retry does not take place
• 3 retries and a 20-second first wait - the waits are 8, 8 and 8 seconds
• 3 retries, a 10-second first wait and 5 seconds in total - the one wait is 5 seconds, so the request is sent twice
• 4 retries, a 3-second first wait and a multiplier of 1 - the waits are 3, 3, 3 and 3 seconds
• 4 retries, a 3-second first wait and a multiplier of 0 - the waits are 3, 1, 1 and 1 seconds

Rate-limited responses

An endpoint that answers with HTTP 429 may say how long to wait in a Retry-After header, as a number of seconds or as an HTTP date, and the connection waits that long instead of following its own schedule, the 8-second cap does not apply then.

A wait that does not fit in what is left of the total cap ends the retries and the 429 response is returned to your service.

Retry-After is honoured by the retries above, not by the rounds of the queue below.

The queue

Turn on Use queue and a request that could not be delivered is put in the connection's queue and your service moves on.

• The call is made once, without retries, and any failure puts the request in the queue, an error response such as an HTTP 500 included
• With Max. retries at 0, a request whose call failed is not queued, it moves to the DLQ, or with the DLQ off, the error is returned to your service
• If the queue already has requests in it, a new one is put in the queue behind them without being sent first
• Requests that send data are queued - a GET or another request that only reads gets its answer or its error at once
• The queue survives server restarts
• Each connection has a queue of its own and requests are delivered in the order they arrived
• Each delivery is a round of attempts under the retry settings above, and a request stays in the queue until a round succeeds or the request expires

A request that a round could not deliver moves to the dead-letter queue. With the DLQ off, it stays at the head of the queue and is tried again, round after round, while everything behind it waits.

The dead-letter queue

Use DLQ is on once the queue is on. A request that a round could not deliver moves to the connection's dead-letter queue with a note of why, when, and how many attempts it took, and the rest of the queue moves on.

Action says what happens to each request in the DLQ after Every has passed since it arrived:

Every defaults to a minute and Max. retries to 3.

Watching the queue and the DLQ

An outgoing connection's row in the Dashboard has a Delivery queue link, which opens a page with two tabs, Delivery queue and Dead-letter queue. Both list the requests in them and when they arrived, and the DLQ tab also shows how many rounds there were and what the last error was. For each request you can:

• See its body and headers
• Download it
• Edit the body before it's retried
• Discard it, or retry it if it's in the DLQ

From your services

Nothing changes in how a service calls a REST, SOAP, HL7 MLLP or HL7 FHIR connection, but with the queue on, a call returns a result instead of a response. is_ok means the request was delivered and the endpoint's answer is in response, is_in_queue means it's in the queue under msg_id, is_in_dlq means it moved to the DLQ under msg_id, and error says why if it was not delivered.

# Get a REST connection ..
conn = self.rest['Billing API']

# .. send the invoice ..
result = conn.post(invoice)

# .. and check what happened to it.
if result.is_ok:
    self.logger.info(f'Delivered -> {result.response.data}')

elif result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')

# Get a SOAP connection ..
conn = self.soap['Immunization Registry']

# .. invoke an operation ..
result = conn.invoke('submitSingleMessage', request)

# .. and check if it's in the queue.
if result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')

# Get a REST connection ..
conn = self.rest['Billing API']

# .. and queue the invoice.
result = conn.publish(invoice)

# Get an MLLP connection ..
conn = self.mllp['EHR Main']

# .. send the message ..
result = conn.send(message)

# .. and check what happened to it.
if result.is_ok:
    self.logger.info(f'Acknowledged -> {result.response.ack_code}')

elif result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')

# Get a FHIR connection ..
client = self.fhir['FHIR.Sample']

# .. build a patient ..
patient = client.resource('Patient')
patient.birthDate = '1974-12-25'

# .. save it ..
result = patient.save()

# .. and check if it's in the queue.
if result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')

# Get a FHIR connection ..
client = self.fhir['FHIR.Sample']

# .. the resource to queue ..
patient = {
    'resourceType': 'Patient',
    'birthDate': '1974-12-25',
}

# .. and queue it.
result = client.publish(patient)

The queue on a channel

A REST or SOAP channel is the address under which a service of yours accepts requests. Turn on Use queue in the channel's Delivery tab and the channel no longer runs the service while the caller waits:

1. The request is stored in the channel's queue, as it arrived - the body, the headers, the path and query parameters
2. The caller gets an acknowledgment at once
3. The service runs from the queue, with the request as if it had just arrived, and a run that raises an exception is retried, round after round, under the channel's retry and DLQ settings above

Accepting a request never waits on anything but storing it, so a channel takes requests in as fast as they come, whatever the service does with them.

The acknowledgment is an HTTP 200 whose body says the request was accepted and under which correlation id, the same id the service's run then logs under. A REST channel answers in JSON:

{"is_ok": true, "cid": "20260312-120534-7301-7f2c0e51a"}

A SOAP channel answers with an envelope whose body wraps the operation's response element with the same two children, is_ok and cid.

The body can be replaced in two ways. The enmasse key queue_response on a channel_rest or channel_soap definition gives a fixed body, returned as it is, as JSON, XML or plain text by what it opens with, and always as XML on a SOAP channel. Or the service builds the body itself by overriding get_queue_response - the method runs once the request is stored, in parallel with anything else the channel is doing, and can read the id of the stored message from self.request.queue.msg_id:

from zato.server.service import Service

class OrderIntake(Service):

    def handle(self):
        # Runs from the queue, once per request, retried if it raises
        order = self.request.payload
        self.rest['Fulfilment'].post(order)

    def get_queue_response(self):
        # Runs as the request is accepted, what it sets is the caller's response
        self.response.payload = {
            'accepted': True,
            'order_id': self.request.payload['order_id'],
            'msg_id': self.request.queue.msg_id,
        }

A service with the method wins over a fixed body, and a channel with neither returns the default acknowledgment. The method is run only on channels whose queue is on - with the switch off, the channel invokes the service and returns the service's own response, as it always has.

The channel's row in the Dashboard has the same Delivery queue link an outgoing connection has, with the queued requests on one tab and the dead-lettered ones on the other, and each row names the method, the path and the service a request is for.

See also

Learn more