# Delivery retries and dead-letter queues

Turn on the queue and a request that cannot be delivered waits, is retried and, if it never gets through, is set aside for inspection.

Every outgoing REST, SOAP, HL7 FHIR and HL7 MLLP connection has a **Delivery** tab in the Dashboard. The settings on that tab determine what happens to a request that could not be delivered:

- It's retried right away
- It waits in the connection's queue until it goes through
- It moves to the dead-letter queue (DLQ), where you can look at it, retry it or discard it

REST and SOAP channels have the same tab, and with the queue on, a channel accepts each request at once and runs its service later, from the queue, see [the queue on a channel](https://zato.io/docs/dev/delivery.html#the-queue-on-a-channel).

## What happens to a failed request {#what-happens-to-a-failed-request}

You can configure retries in 3 ways:

1. **Retries** - the call is retried a few times, with a growing wait in between, and your service waits for the outcome
2. **The queue** - the call is made once and, if it failed and retries are set, the request is put in the connection's queue and your service moves on, the queue then retries it in rounds until it goes through
3. **The dead-letter queue** - a request that a round could not deliver is set aside in the DLQ, from where it's retried later, forwarded to a topic, discarded or kept for you

By default, nothing is turned on, a failed call is reported to your service and nothing is retried.

## Retries {#retries}

A request is retried when:

- The remote system is unreachable
- It did not answer in time
- It answered with HTTP 429
- No HL7 MLLP acknowledgment came back

Everything else, e.g. an HTTP 500 or a negative HL7 acknowledgment, is a response and reaches your service as it arrived.

In the **Delivery** tab, click the **On failure** line to set the retries:

| Setting | Default | What it does |
| --- | --- | --- |
| **Max. retries** | 0 | How many times a failed request is retried |
| **Wait before the first retry** | 2 seconds | How long to wait before the first retry, 8 seconds at most |
| **Wait multiplier** | 2 | How much longer each next wait is |
| **Wait in total at most** | 60 seconds | How long all the waits may take together, once that is reached, no more retries take place |

Each next wait is the previous one times the multiplier, 1 second at least, and no wait goes past 8 seconds or past what is left of the total.

![Retry options on a connection](https://zatosource-production.b-cdn.net/docs/gfx/rest/outconn-retries.webp?v=1792089118)

Dashboard menu: Connections > Outgoing > REST

This is how long a request that keeps failing waits between its attempts:

- With the defaults, the request is sent once and is not retried
- 5 retries, a 2-second first wait, a multiplier of 2 and 60 seconds in total - the waits are 2, 4, 8, 8 and 8 seconds, so the request is sent six times
- The same with 10 retries - the waits are 2, 4, 8, 8, 8, 8, 8, 8 and 6 seconds, 60 seconds together, so the tenth retry does not take place
- 3 retries and a 20-second first wait - the waits are 8, 8 and 8 seconds
- 3 retries, a 10-second first wait and 5 seconds in total - the one wait is 5 seconds, so the request is sent twice
- 4 retries, a 3-second first wait and a multiplier of 1 - the waits are 3, 3, 3 and 3 seconds
- 4 retries, a 3-second first wait and a multiplier of 0 - the waits are 3, 1, 1 and 1 seconds

### Rate-limited responses {#rate-limited-responses}

An endpoint that answers with HTTP 429 may say how long to wait in a `Retry-After` header, as a number of seconds or as an HTTP date, and the connection waits that long instead of following its own schedule, the 8-second cap does not apply then.

A wait that does not fit in what is left of the total cap ends the retries and the 429 response is returned to your service.

`Retry-After` is honoured by the retries above, not by the rounds of the queue below.

## The queue {#the-queue}

Turn on **Use queue** and a request that could not be delivered is put in the connection's queue and your service moves on.

- The call is made once, without retries, and any failure puts the request in the queue, an error response such as an HTTP 500 included
- With **Max. retries** at 0, a request whose call failed is not queued, it moves to the DLQ, or with the DLQ off, the error is returned to your service
- If the queue already has requests in it, a new one is put in the queue behind them without being sent first
- Requests that send data are queued - a GET or another request that only reads gets its answer or its error at once
- The queue survives server restarts
- Each connection has a queue of its own and requests are delivered in the order they arrived
- Each delivery is a round of attempts under the retry settings above, and a request stays in the queue until a round succeeds or the request expires

A request that a round could not deliver moves to the dead-letter queue. With the DLQ off, it stays at the head of the queue and is tried again, round after round, while everything behind it waits.

## The dead-letter queue {#the-dead-letter-queue}

**Use DLQ** is on once the queue is on. A request that a round could not deliver moves to the connection's dead-letter queue with a note of why, when, and how many attempts it took, and the rest of the queue moves on.

**Action** says what happens to each request in the DLQ after **Every** has passed since it arrived:

| Action | What it does |
| --- | --- |
| **Keep in DLQ** | The default, the request waits for you to retry, forward or discard it in the Dashboard |
| **Retry** | The request goes back to the queue for another round, at most **Max. retries** times, then stays in the DLQ |
| **Forward to topic** | The request is published to a pub/sub topic of your choice, **Keep the DLQ header** says whether the note goes with it |
| **Discard** | The request is deleted |

**Every** defaults to a minute and **Max. retries** to 3.

## Watching the queue and the DLQ {#watching-the-queue-and-the-dlq}

An outgoing connection's row in the Dashboard has a **Delivery queue** link, which opens a page with two tabs, **Delivery queue** and **Dead-letter queue**. Both list the requests in them and when they arrived, and the DLQ tab also shows how many rounds there were and what the last error was. For each request you can:

- See its body and headers
- Download it
- Edit the body before it's retried
- Discard it, or retry it if it's in the DLQ

## From your services {#from-your-services}

Nothing changes in how a service calls a [REST](https://zato.io/docs/dev/rest/calling-apis.html), [SOAP](https://zato.io/docs/dev/soap/index.html), [HL7 MLLP](https://zato.io/docs/dev/healthcare/hl7v2/mllp/outgoing.html#send-from-a-service) or [HL7 FHIR](https://zato.io/docs/dev/healthcare/hl7/fhir/connections/index.html#use-the-connection-in-a-service) connection, but with the queue on, a call returns a result instead of a response. `is_ok` means the request was delivered and the endpoint's answer is in `response`, `is_in_queue` means it's in the queue under `msg_id`, `is_in_dlq` means it moved to the DLQ under `msg_id`, and `error` says why if it was not delivered.

**REST and SOAP**

```python
# Get a REST connection ..
conn = self.rest['Billing API']

# .. send the invoice ..
result = conn.post(invoice)

# .. and check what happened to it.
if result.is_ok:
    self.logger.info(f'Delivered -> {result.response.data}')

elif result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')
```

```python
# Get a SOAP connection ..
conn = self.soap['Immunization Registry']

# .. invoke an operation ..
result = conn.invoke('submitSingleMessage', request)

# .. and check if it's in the queue.
if result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')
```

A [REST outgoing connection](https://zato.io/docs/dev/rest/outconns.html) can also queue a request without attempting a call first, and `publish` returns as soon as the request is stored:

```python
# Get a REST connection ..
conn = self.rest['Billing API']

# .. and queue the invoice.
result = conn.publish(invoice)
```

**HL7 MLLP**

With the queue on, a send answered with an `AR`, a [negative acknowledgment](https://zato.io/docs/dev/healthcare/hl7v2/mllp/acknowledgments.html#when-you-send-messages) that requests a retry, is queued rather than returned when retries are set. A send answered with an `AE`, which says the message itself is at fault, is returned to your service, since sending it again would get the same answer. `response` is the acknowledgment of a delivered message:

```python
# Get an MLLP connection ..
conn = self.mllp['EHR Main']

# .. send the message ..
result = conn.send(message)

# .. and check what happened to it.
if result.is_ok:
    self.logger.info(f'Acknowledged -> {result.response.ack_code}')

elif result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')
```

**HL7 FHIR**

With the queue on, [saving a resource](https://zato.io/docs/dev/healthcare/hl7/fhir/resources/index.html#create-a-resource) returns the result instead of the saved resource:

```python
# Get a FHIR connection ..
client = self.fhir['FHIR.Sample']

# .. build a patient ..
patient = client.resource('Patient')
patient.birthDate = '1974-12-25'

# .. save it ..
result = patient.save()

# .. and check if it's in the queue.
if result.is_in_queue:
    self.logger.info(f'Queued -> {result.msg_id}')
```

A [FHIR connection](https://zato.io/docs/dev/healthcare/hl7/fhir/connections/index.html#queue-documents-for-guaranteed-delivery) can also put a resource directly in the queue, the call returns as soon as the resource is stored:

```python
# Get a FHIR connection ..
client = self.fhir['FHIR.Sample']

# .. the resource to queue ..
patient = {
    'resourceType': 'Patient',
    'birthDate': '1974-12-25',
}

# .. and queue it.
result = client.publish(patient)
```

## The queue on a channel {#the-queue-on-a-channel}

A REST or SOAP channel is the address under which a service of yours accepts requests. Turn on **Use queue** in the channel's **Delivery** tab and the channel no longer runs the service while the caller waits:

1. The request is stored in the channel's queue, as it arrived - the body, the headers, the path and query parameters
2. The caller gets an acknowledgment at once
3. The service runs from the queue, with the request as if it had just arrived, and a run that raises an exception is retried, round after round, under the channel's retry and DLQ settings above

Accepting a request never waits on anything but storing it, so a channel takes requests in as fast as they come, whatever the service does with them.

The acknowledgment is an HTTP 200 whose body says the request was accepted and under which correlation id, the same id the service's run then logs under. A REST channel answers in JSON:

```json
{"is_ok": true, "cid": "20260312-120534-7301-7f2c0e51a"}
```

A SOAP channel answers with an envelope whose body wraps the operation's response element with the same two children, `is_ok` and `cid`.

The body can be replaced in two ways. The enmasse key `queue_response` on a `channel_rest` or `channel_soap` definition gives a fixed body, returned as it is, as JSON, XML or plain text by what it opens with, and always as XML on a SOAP channel. Or the service builds the body itself by overriding `get_queue_response` - the method runs once the request is stored, in parallel with anything else the channel is doing, and can read the id of the stored message from `self.request.queue.msg_id`:

```python
from zato.server.service import Service

class OrderIntake(Service):

    def handle(self):
        # Runs from the queue, once per request, retried if it raises
        order = self.request.payload
        self.rest['Fulfilment'].post(order)

    def get_queue_response(self):
        # Runs as the request is accepted, what it sets is the caller's response
        self.response.payload = {
            'accepted': True,
            'order_id': self.request.payload['order_id'],
            'msg_id': self.request.queue.msg_id,
        }
```

A service with the method wins over a fixed body, and a channel with neither returns the default acknowledgment. The method is run only on channels whose queue is on - with the switch off, the channel invokes the service and returns the service's own response, as it always has.

The channel's row in the Dashboard has the same **Delivery queue** link an outgoing connection has, with the queued requests on one tab and the dead-lettered ones on the other, and each row names the method, the path and the service a request is for.

## See also {#see-also}

- [REST outgoing connections](https://zato.io/docs/dev/rest/outconns.html) - Creating connections, pools, timeouts and OpenAPI import
- [REST channels](https://zato.io/docs/dev/rest/channels.html) - The channels whose queue accepts requests for a later run
- [Error handling](https://zato.io/docs/dev/rest/error-handling.html) - What your service receives when an endpoint fails
- [Enmasse reference](https://zato.io/docs/admin/enmasse-reference.html) - The same settings as YAML keys, under each connection type
- [Pub/sub topics and message queues](https://zato.io/docs/eda/flow/index.html) - The topics a dead-lettered request can be forwarded to

## Learn more {#learn-more}

- [Development documentation](https://zato.io/docs/dev/) - The complete reference for writing services
- [Requests and responses](https://zato.io/docs/dev/request-response/) - What a service receives, what it returns and how both are structured
- [Integration examples](https://zato.io/docs/dev/examples/) - Ready-made code for the systems you are likely to connect to
- [IDE and debugging](https://zato.io/docs/dev/ide/) - Write services in the Dashboard or in your own editor
- [Data models](https://zato.io/docs/dev/model/) - Declare inputs and outputs and have them validated for you
- [In-depth API tutorial](https://zato.io/tutorials/main/01.html) - The full platform tutorial, from installation to production patterns