Production best practices

Zato provides three built-in user accounts. Administrators sign in to the Dashboard as admin, connect to the container over SSH as zato, and deploy code from an IDE as ide_publisher.

Zato reads the password of each account from its environment variable when the container starts. If the variable is not set, Zato generates a random password for the account at every start, and the password changes with each restart.

Before you start a production container, set each of the three variables to its own password, taken from the secrets store of your organization. Do not use Zato_Password in production - it assigns one password to all three accounts and is intended for development environments.

docker run \
  -e Zato_Dashboard_Password="$ZATO_DASHBOARD_PASSWORD" \
  -e Zato_SSH_Password="$ZATO_SSH_PASSWORD" \
  -e Zato_IDE_Password="$ZATO_IDE_PASSWORD" \
  zatosource/zato-4.1@sha256:<digest>

To change a password later, set the new value in the variable and restart the container, as described in changing passwords.

The Dashboard gives full control over the environment, so make it reachable from your internal network only. Publish the Dashboard port on an address of the internal network, and administrators connect to it from the office or through a VPN.

docker run \
  -p 10.0.0.5:8184:8184 \
  zatosource/zato-4.1@sha256:<digest>

A Dashboard session lasts two hours by default. To change it, set Zato_Dashboard_Session_Timeout to the number of seconds a session lasts.

Leave the following variables unset in production. They are meant for development environments.

If you start the production environment from a Compose file or a script copied from a development environment, remove these variables from it.

Zato fetches the latest Zato code each time the container starts, so two starts of the same image can run different code. In production, run the code that you tested - refer to the image by its digest instead of latest, and set Zato_Install_Updates to False. With this setting, Zato also stops the automatic updates scheduled in the Dashboard.

docker run \
  -e Zato_Install_Updates=False \
  zatosource/zato-4.1@sha256:<digest>

docker images --digests zatosource/zato-4.1 shows the digest of each image that you pulled. To move to a new version, test the new image and then change the digest, as described in updating environments.

Zato rotates server.log and http_access.log when a file reaches 1 GB, and keeps two rotated copies of each. To change it, set Zato_Server_Log_Max_Size to the size in bytes and Zato_Server_Log_Backup_Count to the number of copies, and restart the container.

Docker keeps the container output in a file of its own, which has no size limit by default. Limit it with the --log-opt options of docker run:

docker run \
  -e Zato_Server_Log_Max_Size=500000000 \
  -e Zato_Server_Log_Backup_Count=4 \
  --log-opt max-size=100m \
  --log-opt max-file=3 \
  zatosource/zato-4.1@sha256:<digest>

Set Zato_Env_Name to the name of the environment, for instance Production. Zato prints the name in the container output when the environment starts, so docker logs shows which environment the output comes from.

docker run \
  -e Zato_Env_Name=Production \
  zatosource/zato-4.1@sha256:<digest>

Configure SSL/TLS for every environment that API clients or administrators reach over a network, as described in SSL/TLS configuration.

Use a certificate issued by the certificate authority of your organization instead of the self-signed certificate that Zato generates, as described in use your own SSL certificate.

Each Zato interface has a plain-text port and a TLS port. Docker makes a port reachable from outside the host only if you publish it with the -p option of docker run.

Publish only the TLS ports of the interfaces you use. Do not publish the plain-text ports, and do not publish port 22, which is used for SSH.

docker run \
  -p 8184:8184 \
  -p 11224:11224 \
  zatosource/zato-4.1@sha256:<digest>

If a reverse proxy or a load balancer in front of Zato sets X-Forwarded-For, set Zato_Trust_Forwarded_Headers to True. Zato then takes the address of the caller from that header, so IP filtering, rate limits and the access log use the address of the caller instead of the address of the proxy.

docker run \
  -e Zato_Trust_Forwarded_Headers=True \
  zatosource/zato-4.1@sha256:<digest>

Leave the variable unset when callers connect to Zato directly. Otherwise, a caller can send X-Forwarded-For itself and choose the address that IP filtering and rate limits apply to. The details are in client IP resolution.

If administrators open the Dashboard through a proxy, set Zato_Dashboard_CSRF_Trusted_Origins to the address that they open in the browser, with the scheme and with the port if it is not the default one, for instance https://dashboard.example.com. Separate several addresses with commas. Otherwise, Dashboard forms fail with the error "CSRF verification failed". The details are in set CSRF trusted origins for the Dashboard.

The /metrics endpoint requires HTTP Basic Auth with the user metrics. Set Zato_Metrics_Password to a password from the secrets store of your organization, and give the same password to the system that collects the metrics. Without the variable, Zato generates a new password at each start, and the collector stops receiving metrics after a restart.

docker run \
  -e Zato_Metrics_Password="$ZATO_METRICS_PASSWORD" \
  zatosource/zato-4.1@sha256:<digest>

The details are in monitoring.

Zato writes the server log to the container output as well as to server.log. The container empties its logs directory at each start, so send the container output to your log platform through a Docker logging driver:

docker run \
  --log-driver=fluentd \
  --log-opt fluentd-address=logs.example.com:24224 \
  zatosource/zato-4.1@sha256:<digest>

The details are in logging.

A log agent, such as the Datadog Agent, can read the container output, which carries the server log. Each entry in the server log begins with a date, so configure the agent to start a new record at each date - otherwise, every line of a traceback becomes a record of its own.

With Datadog, set DD_LOGS_ENABLED=true on the Datadog Agent and label the Zato container:

docker run \
  -l com.datadoghq.ad.logs='[{"source": "zato", "service": "zato", "log_processing_rules": [{"type": "multi_line", "name": "new_entry", "pattern": "\\d{4}-\\d{2}-\\d{2}"}]}]' \
  zatosource/zato-4.1@sha256:<digest>

The metrics side of Datadog is in Datadog.