Real-world file transfer feeds

Marker files, slow uploads, competing consumers, several patterns over one directory and daily batches.

The schedules chapter lists the fields. This one puts them together into complete feeds.

A partner drop with marker files

A partner uploads a file to their SFTP server on their own timetable and signals that the upload is finished by putting down a second, empty file named after the first.

Field	Value
Directory	`/incoming/orders`
Pattern	`orders_*.csv`
Ready when	Marker
Marker suffix	`.done`
On success	move
Move directory	`processed`
Run every	10 minutes

A run that finds orders_20260801.csv on its own does nothing - the upload is not finished. Once orders_20260801.csv.done appears, the next run takes the file, invokes the service, moves the file to processed/ and deletes the marker.

Ask for this arrangement whenever the partner can provide it. It is exact, it costs nothing to check, and it works no matter how slow or how interrupted the upload was.

An upload with no marker

Not every sender will put down a marker, and many of them are scripts nobody wants to touch. Stability mode covers this - the run looks at the file twice and takes it only if it has stopped changing.

Field	Value
Directory	`MyShare/incoming/stock`
Pattern	`*.xml`
Ready when	Stability
Stability delay	5
On success	move
Run every	5 minutes

A file still being written has either a different size or a different modification time on the second look, so it is left for a later run and taken once the writing stops.

The delay is the whole judgement here. Too short and a sender pausing mid-upload looks finished, too long and every run costs that wait. Five seconds suits most links, and a slow or congested one deserves more.

More than one Zato environment over one directory

Two environments watching one directory will both see the same files. Claiming is what keeps them from both processing them.

Turn Claim the file first on in both schedules. Each file is renamed with a .processing suffix before anything reads it, and a rename that fails means somebody else got there first, so that environment skips the file without treating it as an error. A file that already has the suffix is skipped by everybody.

Field	Value
Directory	`/incoming/settlements`
Claim first	yes
On success	move
Run every	1 minute

Between the two of them every file goes through exactly once. If a service refuses a file, the claim is undone and the file goes back to its own name so either environment can try again.

Several feeds in one directory

A partner who puts everything in one directory does not force you into one service. Give the connection one schedule per kind of file, each with its own pattern and its own service.

Schedule	Pattern	Service
`feed.orders`	`orders_*.csv`	`partner.process-order`
`feed.invoices`	`invoice_*.csv`	`partner.process-invoice`
`feed.catalog`	`catalog_*.xml`	`partner.process-catalog`

Each schedule ignores what the others are for, and each has its own interval - a catalog that arrives weekly does not need to be looked for every minute.

The same file name every day

Plenty of feeds send orders.csv and nothing else, every single day. There is nothing to configure for this, but there is something to know about it - the destination directory holds one file of any given name, so today's arrival takes the place of yesterday's.

If the destination is meant to be a record of what came in, either ask the sender to date the names, or have your service copy the contents somewhere of its own before returning:

from zato.server.service import Service

class ProcessOrders(Service):

    def handle(self):

        item = self.request.raw_request

        # Keep a dated copy of our own before the schedule moves the file away
        conn = self.sftp[item.conn_name]
        stamp = item.last_modified.replace(':', '').replace('-', '')

        conn.write(item.data, f'{item.directory}/archive/{stamp}-{item.file_name}')

A feed where some files are bad

A partner sending a hundred files a day will eventually send one that cannot be parsed. One bad file does not stop the rest - the run logs it, leaves it where it is, and continues with the next file. The healthy files of the same run go through as usual.

The bad file comes back on every run and keeps reporting the same error until somebody deals with it. See refusing a file for the two ways to handle that.

A feed you want to stop for a while

Set the schedule to inactive. The definition and its scheduler job stay as they were, nothing runs, and files pile up in the directory until you switch it back on, at which point the next run takes everything that accumulated. Use this for a maintenance window instead of deleting the schedule.