File transfer schedules

Watching a directory, deciding when a file has finished uploading and invoking a service once per file.

A schedule is one recurring pickup task belonging to one connection. It looks into a directory as often as you tell it to, works out which files are ready, and invokes a service of yours once per ready file. When the service is done with a file, the file is moved out of the way or deleted, so the next run never sees it again.

A connection may have as many schedules as you need, each watching its own directory, or several watching one directory with different patterns.

Creating a schedule

In Dashboard, open the list of outgoing SFTP or SMB connections and click the Schedules link of the connection the schedule belongs to. From the list that opens, click Create a new schedule.

The wizard has three steps - what to pick up, what to do with it, and a review of both before anything is saved.

What to pick up

Field	Notes
Name	A label, unique among the schedules of this connection. It also names the scheduler job that runs the schedule, e.g. a schedule called `invoices.hourly` on an SFTP connection called `Partner SFTP` is run by a job called `sftp.Partner SFTP.invoices.hourly`.
Active	An inactive schedule keeps its definition and never runs.
Directory	The directory to look into. An SFTP directory is an ordinary path, e.g. `/incoming/invoices`. An SMB directory starts with the share, e.g. `MyShare/incoming/invoices`. A trailing slash is accepted and ignored.
Pattern	Which names are of interest, as an fnmatch pattern, e.g. `orders_.csv`. The default is ``, meaning everything.

A run only looks at the directory it was given. Subdirectories are listed but never descended into, and directories and symlinks are never picked up, no matter what the pattern says.

When a file is ready

A file that is still being uploaded must not be picked up, and there are two ways to know that an upload has finished. This is the single most important choice in the schedule, because getting it wrong means delivering half a file.

Stability

The run takes a first look at the file, waits, and takes a second look. If the size and the modification time are both unchanged, the upload is over and the file is taken. If either has moved, the file is left for a later run.

Stability delay is how many seconds pass between the two looks, with a default of 2. One wait covers every candidate of the run, so a directory with fifty growing files costs one delay, not fifty.

This mode needs nothing from the sender, which is what makes it the default. Its limit is that a sender pausing for longer than the delay looks finished, so give the delay some room on a slow link.

Marker

The run takes a file only once a second file has appeared next to it, named after it plus a suffix. The sender uploads orders.csv at whatever pace it likes and then puts down an empty orders.csv.done, at which point the run knows the upload is complete.

Marker suffix is that suffix, with a default of .done. Markers are never picked up themselves, and a marker is deleted along with the file it belongs to.

This mode is exact, and it is what to ask a partner for when you can. Nothing is ever taken early, however slow or however interrupted the upload was.

Claiming

Claim the file first renames a file before anything reads it, adding a .processing suffix. Two Zato environments watching the same directory therefore never take the same file - whichever one renames it first has it, and the other one moves on, since a file that already has the suffix is skipped.

The rename is also what makes the file invisible to the sender's own tooling while it is being read.

If the service refuses the file, the claim is undone and the file goes back to its own name, ready for another run.

Leave this off when only one environment watches the directory. See real-world feeds for the arrangement in full.

What happens next

Field	Notes
Service	The service to invoke, once per file. See receiving files for what it is given.
On success	`move` or `delete`.
Move directory	Where a file goes once the service has taken it, always relative to the directory being watched, with a default of `processed`. It is created on first use. A path of its own, one pointing at the watched directory itself, and one pointing above it are all refused. A file whose name the destination already holds keeps both - the newcomer is given the moment it arrived as a suffix.

Success means the service returned without raising. A file the service refuses is left exactly where it was and comes back on the next run, so nothing is ever lost to a failure.

How often to look

Field	Notes
Run every	A positive number.
Unit	`seconds`, `minutes`, `hours`, `days` or `weeks`.
Start date	When the first run happens. A date in the future means the schedule sits idle until then.

Every schedule is run by an interval-based job of its own in the scheduler, created, renamed and deleted along with the schedule. The job appears under Scheduler in Dashboard like any other, so its history and its next run time are visible in the usual place, and a change made to its interval there is written back to the schedule.

A worked example

A partner drops invoices as CSV files on their SFTP server, finishing each upload with a marker file, and they should be picked up every ten minutes and kept afterwards:

Field	Value
Name	`invoices.hourly`
Directory	`/incoming/invoices`
Pattern	`invoice_*.csv`
Ready when	Marker
Marker suffix	`.done`
Service	`partner.process-invoice`
On success	move
Move directory	`processed`
Run every	10 minutes

A run that finds invoice_20260801.csv and invoice_20260801.csv.done invokes partner.process-invoice with the file, then moves the file to /incoming/invoices/processed/ and deletes the marker.