File transfer in enmasse
SFTP and SMB connections and their schedules in YAML - every field, its default and how re-imports work.
Outgoing SFTP and SMB connections, and the file transfer schedules that belong to them, are all declared in enmasse YAML, so a whole feed can live in version control and be imported into every environment.
The connection fields themselves are in the enmasse reference. This chapter covers the schedules list a file transfer connection holds.
A connection with its schedules
sftp:
- name: Partner SFTP
address: sftp.example.com:22
username: zato
password: Zato_Enmasse_Env.Partner_SFTP_Password
private_key: /keys/id_ed25519
strict_host_key_checking: true
schedules:
- name: invoices.hourly
directory: /incoming/invoices
pattern: 'invoice_*.csv'
ready_how: marker
marker_suffix: .done
service: partner.process-invoice
on_success: move
move_directory: processed
run_every: 10
run_unit: minutes
- name: catalog.weekly
directory: /incoming/catalog
pattern: '*.xml'
service: partner.process-catalog
run_every: 1
run_unit: weeks
The same works under smb, the only difference being that each directory starts with the share:
smb:
- name: Finance Share
host: smb.example.com
port: 445
username: zato
password: Zato_Enmasse_Env.Finance_SMB_Password
schedules:
- name: stock.levels
directory: MyShare/incoming/stock
pattern: '*.xml'
stability_delay: 5
service: retail.process-stock
run_every: 5
run_unit: minutes
The fields of one schedule
| Name | Required | Description |
|---|---|---|
| name | Yes | A label, unique among the schedules of this connection. The scheduler job is named after it. |
| directory | Yes | The directory to watch. An SMB directory starts with the share. |
| service | Yes | The service invoked once per file |
| run_every | Yes | A positive number |
| run_unit | Yes | seconds, minutes, hours, days or weeks |
| start_date | --- | When the first run happens, in ISO-8601. If it is not given, the schedule starts as soon as it is imported. |
| is_active | --- | Whether the schedule runs (default is true) |
| pattern | --- | An fnmatch pattern of the names to pick up (default is *) |
| ready_how | --- | stability or marker (default is stability) |
| stability_delay | --- | Seconds between the two looks stability mode takes (default is 2) |
| marker_suffix | --- | The suffix of the file that says an upload is finished (default is .done) |
| should_claim | --- | Whether to rename a file before reading it, so that no two environments take the same one (default is false) |
| on_success | --- | move or delete (default is move) |
| move_directory | --- | Where a taken file goes, relative to the watched directory (default is processed) |
Anything left out means its default, and anything left at its default is not written out on export.
The YAML is the source of truth
An explicit schedules key, even an empty list, means the YAML decides what schedules this connection has.
Importing the file again brings each schedule up to date and, just as importantly, deletes the ones the YAML no longer contains, along with their scheduler jobs. Dropping a schedule from the file and re-importing is how a feed is retired.
Leaving the key out altogether means enmasse says nothing about schedules and whatever the connection already has is left alone.
Exporting
An export of an environment that has file transfer connections writes their schedules out with them, so the round trip is complete - export from one environment, import into another, and both watch the same directories with the same services.
Related
- File transfer schedules - what each of these fields does
- Real-world feeds - the arrangements worth putting in YAML
- enmasse reference - every other object type