File transfer & remote systems engineer

Collect the bank statements over SFTP

Every morning the treasury collects the CSV statements left by the bank in its SFTP outbox. The script stores the connection once, downloads every new statement, archives it on the server, uploads an acknowledgement and cleans the old folders.

#1. The connection is stored once in the connection manager;
cm set "rx_cm_sftp_bank" {execute "file.remote.sftp.config.get"
	"[hostname]" "sftp.bank.example.com"
	"[port]" "22"
	"[user]" "acme_treasury"
	"[password]" "change_me"
	"[connectTimeout]" 10000
	"[dataTimeout]" 30000
;};
#2. Session, remote outbox and local folder;
file mkdir "tmp/rx_in";
sftp connect "rx_sftp" {cm get "rx_cm_sftp_bank";};
-> "[home]" (sftp home "rx_sftp");
sftp cd "rx_sftp" "outbox/statements";
sftp lcd "rx_sftp" "tmp/rx_in";
sftp mkdir "rx_sftp" "archive_2026-11";
#3. Download every statement, then move it to the archive;
json load "rx_ls" (sftp ls "rx_sftp" "*.csv");
json parse_array "rx_ls" "/" "[f]" {
	json load "rx_f" [f];
	-> "[name]" (json select "rx_f" "/name");
	sftp get "rx_sftp" [name] (concat "tmp/rx_in/" [name]);
	sftp rename "rx_sftp" [name] (concat "archive_2026-11/" [name]);
};
#4. Acknowledge and clean the old archive;
file create "tmp/rx_in/ack.txt" (concat "received " (json count "rx_ls" "/") " files");
sftp put "rx_sftp" "tmp/rx_in/ack.txt" "../../inbox/ack.txt" "OVERWRITE";
sftp rm "rx_sftp" "../../inbox/ack_previous.txt";
sftp rm dir "rx_sftp" "archive_2026-09";
-> "[info]" (concat "SFTP v" (sftp version "rx_sftp") ", home " [home] ", remote " (sftp pwd "rx_sftp") ", local " (sftp lpwd "rx_sftp"));
sftp disconnect "rx_sftp";
concat (json count "rx_ls" "/") " statements downloaded. " [info];

Send orders and fetch delivery reports over FTP

A logistics partner still works with plain FTP. The script opens a session in passive or active mode depending on the firewall, sends the orders of the day in binary with compression, downloads the delivery reports, moves them to a done folder and removes the old files.

#1. Connection of the partner and the FTP mode allowed by its firewall;
cm set "rx_cm_ftp_logi" {execute "file.remote.ftp.config.get"
	"[hostname]" "ftp.logistics.example.com"
	"[port]" "21"
	"[user]" "acme"
	"[password]" "change_me"
;};
-> "[mode]" "passive";
#2. Session: timeout, mode, binary type with compression;
ftp set timeout 10000;
ftp connect "rx_ftp" {cm get "rx_cm_ftp_logi";};
if (equal [mode] "active") {ftp active "rx_ftp";} {ftp passive "rx_ftp";};
ftp set type "rx_ftp" "BINARY";
ftp active compression "rx_ftp" true;
#3. Send the orders of the day;
ftp mkdir "rx_ftp" "/in/2026-11-01";
ftp cd "rx_ftp" "/in/2026-11-01";
ftp put "rx_ftp" "tmp/rx_orders_20261101.csv" "RESUME";
#4. Fetch the delivery reports and move them to done;
ftp cd "rx_ftp" "/out";
json load "rx_rep" (ftp ls "rx_ftp" "*.xml");
json parse_array "rx_rep" "/" "[f]" {
	json load "rx_f" [f];
	-> "[name]" (json select "rx_f" "/name");
	ftp get "rx_ftp" [name] (concat "tmp/rx_reports/" [name]);
	ftp rename "rx_ftp" [name] (concat "/done/" [name]);
};
#5. Remove the heartbeat file and last month folder;
ftp rm file "rx_ftp" "/out/heartbeat.tmp";
ftp rm dir "rx_ftp" "/in/2026-10-01";
-> "[where]" (ftp pwd "rx_ftp");
ftp disconnect "rx_ftp";
concat (json count "rx_rep" "/") " reports downloaded, last folder " [where];

Bank returns over FTPS, published on the accounting share

The bank publishes payment return files on an explicit FTPS server. The script secures the data channel, downloads the new returns, marks them as processed, uploads an acknowledgement, then copies the files to the Windows share of the accounting team and lists what they will see.

#1. FTPS of the bank and Windows share of accounting;
cm set "rx_cm_ftps_bank" {execute "file.remote.ftps.config.get"
	"[hostname]" "ftps.bank.example.com"
	"[port]" "21"
	"[user]" "acme"
	"[password]" "change_me"
	"[protocol]" "TLS"
	"[isImplicit]" false
	"[connectTimeout]" 10000
	"[dataTimeout]" 30000
	"[keepAliveTimeout]" 300
;};
cm set "rx_cm_cifs_compta" {execute "file.remote.cifs.config.get"
	"[hostname]" "192.168.10.20"
	"[port]" "445"
	"[domain]" "ACME"
	"[user]" "svc_mentdb"
	"[password]" "change_me"
	"[connectTimeout]" "10000"
	"[readTimeout]" "30000"
;};
#2. Secure the data channel, passive mode, binary;
ftps connect "rx_ftps" {cm get "rx_cm_ftps_bank";};
ftps parse pbsz "rx_ftps" 0;
ftps exec prot "rx_ftps" P;
-> "[mode]" "passive";
if (equal [mode] "active") {ftps active "rx_ftps";} {ftps passive "rx_ftps";};
ftps set type "rx_ftps" "BINARY";
#3. Download the returns and mark them as processed;
ftps cd "rx_ftps" "/returns";
json load "rx_dirs" (ftps ls dirs "rx_ftps");
json load "rx_files" (ftps ls "rx_ftps" "PAY_.*\.xml");
if (not (string contains (json doc "rx_dirs") "processed")) {ftps mkdir "rx_ftps" "/returns/processed";};
json parse_array "rx_files" "/" "[f]" {
	json load "rx_f" [f];
	-> "[name]" (json select "rx_f" "/name");
	ftps get "rx_ftps" [name] (concat "tmp/rx_returns/" [name]);
	ftps rename "rx_ftps" [name] (concat "/returns/processed/" [name]);
};
ftps put "rx_ftps" "tmp/rx_ack_20261101.xml" "/acks/ack_20261101.xml" "RESUME";
ftps rm "rx_ftps" "/returns/heartbeat.tmp";
ftps rm dir "rx_ftps" "/returns/2026-09";
-> "[pwd]" (ftps pwd "rx_ftps");
ftps disconnect "rx_ftps";
#4. Publish on the share: new folder, files, rotate the latest marker;
cifs mkdir "Treasury/2026-11" {cm get "rx_cm_cifs_compta";};
json parse_array "rx_files" "/" "[f]" {
	json load "rx_f" [f];
	cifs put (concat "tmp/rx_returns/" (json select "rx_f" "/name")) (concat "Treasury/2026-11/" (json select "rx_f" "/name")) {cm get "rx_cm_cifs_compta";};
};
cifs rm "Treasury/latest_old.txt" {cm get "rx_cm_cifs_compta";};
cifs rename "Treasury/latest.txt" "latest_old.txt" {cm get "rx_cm_cifs_compta";};
cifs get "Treasury/rules.ini" "tmp/rx_rules.ini" {cm get "rx_cm_cifs_compta";};
json load "rx_share" (cifs ls "Treasury/2026-11" ".*\.xml" {cm get "rx_cm_cifs_compta";});
concat (json count "rx_share" "/data") " files on the share, FTPS folder " [pwd];

Watch a drop folder, push files over SSH and notify the head office

Each sales export dropped in a folder must reach the reporting server and be imported there, and the head office MentDB must be told. The script copies the file with SCP, runs the import and a check over SSH, gets the import log back, sends the file name safely through a MentDB tunnel, then (re)starts the folder watcher that runs this for every new file.

#1. The reporting server by SSH, the head office MentDB by tunnel;
cm set "rx_cm_ssh_report" {execute "file.remote.ssh.config.get"
	"[hostname]" "report.example.com"
	"[port]" "22"
	"[user]" "deploy"
	"[password]" "change_me"
	"[connectTimeout]" 10000
;};
cm set "rx_cm_mentdb_hq" {execute "mentdb.remote.config.get"
	"[hostname]" "hq.example.com"
	"[port]" "9998"
	"[key]" "change_me"
	"[user]" "admin"
	"[password]" "change_me"
	"[connectTimeout]" "10000"
	"[readTimeout]" "30000"
	"[subTunnels]" (mql {[MQL_TO_REPLACE]})
;};
#2. Push the export, run the import, check the disk, get the log;
-> "[file]" "sales_20261101.csv";
ssh connect "rx_ssh" {cm get "rx_cm_ssh_report";};
ssh scp to "rx_ssh" (concat "tmp/rx_drop/" [file]) (concat "/data/import/" [file]);
-> "[out]" (ssh execute_1_cmd "rx_ssh" (concat "/opt/report/bin/import.sh /data/import/" [file]));
-> "[check]" (ssh execute_n_cmd "rx_ssh" "cd /data/import;ls -l;df -h /data");
ssh scp from "rx_ssh" "/opt/report/logs/import.log" "tmp/rx_drop/import.log";
ssh disconnect "rx_ssh";
#3. Tell the head office: local values are encoded into the remote MQL;
tunnel connect "rx_hq" {cm get "rx_cm_mentdb_hq";};
-> "[ack]" (tunnel execute "rx_hq" (concat "-> \"[f]\" \"" (mql encode [file]) "\";" (mql {concat "received " [f]})));
tunnel disconnect "rx_hq";
-> "[hq_time]" (tunnel execute_hot "rx_hq_once" {cm get "rx_cm_mentdb_hq";} (mql {date systimestamp}));
#4. Restart the watcher of the drop folder with the current script;
if (file_watcher exist "rx_fw_drop") {file_watcher kill "rx_fw_drop";};
file_watcher start "rx_fw_drop" "admin" "tmp/rx_drop" "rx.sales.on_drop";
concat [ack] " at " [hq_time] " | import: " [out] " | watchers: " (file_watcher show);

Send a file to the least loaded node of a cluster

The price list of the day must reach the shop servers. The script sends the file in base64 to the node with the lowest signal of prod_cluster, checks the size received, runs the import through a hot cluster tunnel with the nodes taken in turn, then archives the local file.

#1. The price list of the day, read in base64 without the line breaks;
-> "[file]" "prices_20261101.csv";
-> "[b64]" (string regexp_replace (file b64_read (concat "tmp/rx_out/" [file])) "\s" "");
-> "[local_size]" (file size (concat "tmp/rx_out/" [file]));
#2. Write it on the least loaded node of the cluster, chosen by its signal;
-> "[remote_size]" (execute "cluster.1n.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "SIGNAL"
	"[request]" (concat "file b64_write \"" [b64] "\" \"data/import/" [file] "\"; file size \"data/import/" [file] "\";")
);
#3. Run the import on the fly through a cluster tunnel, nodes taken in turn;
-> "[imported]" (include "cluster.1n.hot.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "LB_50_50"
	"[tunnel_id]" "rx_t_prices"
	"[request]" (concat "-> \"[f]\" \"" (mql encode [file]) "\";" (mql {execute "shop.prices.import" "[file]" (concat "data/import/" [f]);}))
);
#4. Same size on both sides, then archive the local file;
if (== [local_size] [remote_size]) {file copy_file (concat "tmp/rx_out/" [file]) (concat "tmp/rx_out/sent_" [file]); file delete (concat "tmp/rx_out/" [file]);};
concat [file] ": " [local_size] " bytes sent, " [remote_size] " received, import: " [imported]

Declare the partner endpoints, secrets kept in private parameters

Onboard four partners (bank over SFTP, logistics over FTPS, accounting share over CIFS, a Linux relay over SSH). The passwords live in private parameters, the connections are declared but not opened, and the registry report shows hosts and ports only, never a secret.

#1. The secrets are private parameters, never written in the scripts;
parameter add "tuto_p4i_bank_pwd" "Bank-Sftp-2026!" "SFTP password of the bank" true;
parameter add "tuto_p4i_logi_pwd" "Logi-Ftps-2026!" "FTPS password of the carrier" true;
#2. The endpoints, declared only: nothing is opened here;
cm set "tuto_p4i_bank" {execute "file.remote.sftp.config.get"
	"[hostname]" "sftp.bank.example.com"
	"[port]" "22"
	"[user]" "acme_treasury"
	"[password]" (parameter get value "tuto_p4i_bank_pwd")
	"[connectTimeout]" 10000
	"[dataTimeout]" 30000
;};
cm set "tuto_p4i_logi" {execute "file.remote.ftps.config.get"
	"[hostname]" "ftps.carrier.example.com"
	"[port]" "990"
	"[user]" "acme_edi"
	"[password]" (parameter get value "tuto_p4i_logi_pwd")
	"[protocol]" "TLS"
	"[isImplicit]" true
	"[connectTimeout]" 10000
	"[dataTimeout]" 30000
	"[keepAliveTimeout]" 300
;};
cm set "tuto_p4i_share" {execute "file.remote.cifs.config.get"
	"[hostname]" "fs01.acme.example.com"
	"[port]" "445"
	"[domain]" "ACME"
	"[user]" "svc_mentdb"
	"[password]" (parameter get value "tuto_p4i_bank_pwd")
	"[connectTimeout]" "10000"
	"[readTimeout]" "30000"
;};
cm set "tuto_p4i_relay" {execute "file.remote.ssh.config.get"
	"[hostname]" "relay.acme.example.com"
	"[port]" "2222"
	"[user]" "transfer"
	"[password]" (parameter get value "tuto_p4i_logi_pwd")
	"[connectTimeout]" 10000
;};
#3. Registry: type, host and port of each endpoint;
-> "[rep]" "";
json load "rx_p4i_ids" "[\"tuto_p4i_bank\",\"tuto_p4i_logi\",\"tuto_p4i_share\",\"tuto_p4i_relay\"]";
json parse_array "rx_p4i_ids" "/" "[id]" {
	json load "rx_p4i_cm" (cm get [id]);
	-> "[rep]" (concat [rep] [id] ": " (json select "rx_p4i_cm" "/type") " " (json select "rx_p4i_cm" "/hostname") ":" (json select "rx_p4i_cm" "/port") [_n_]);
};
json unload "rx_p4i_cm";
-> "[rep]" (concat [rep] "sftp declared: " (cm exist "tuto_p4i_bank" "sftp") ", bank as ftp: " (cm exist "tuto_p4i_bank" "ftp") ", relay listed in ssh: " (string contains (cm show "ssh") "tuto_p4i_relay") [_n_] "secrets private: " (parameter is_private "tuto_p4i_bank_pwd") " and " (parameter is_private "tuto_p4i_logi_pwd") ", secret in the report: " (string contains [rep] "2026!"));
#4. Demo cleanup;
json parse_array "rx_p4i_ids" "/" "[id]" {cm remove [id];};
json unload "rx_p4i_ids";
parameter remove "tuto_p4i_bank_pwd";
parameter remove "tuto_p4i_logi_pwd";
concat [rep] [_n_] "after cleanup: connection " (cm exist "tuto_p4i_bank") ", parameter " (parameter exist "tuto_p4i_bank_pwd");

tuto_p4i_bank: sftp sftp.bank.example.com:22
tuto_p4i_logi: ftps ftps.carrier.example.com:990
tuto_p4i_share: cifs fs01.acme.example.com:445
tuto_p4i_relay: ssh relay.acme.example.com:2222
sftp declared: 1, bank as ftp: 0, relay listed in ssh: 1
secrets private: 1 and 1, secret in the report: 0
after cleanup: connection 0, parameter 0

SFTP pickup with an atomic rename and a remote archive

Collect the payment returns from a bank SFTP without ever reading a file still being uploaded: rename each file into a .processing name first, download it, check it is not empty, then move it to an archive folder of the bank server.

#1. Connect with the declared connection;
sftp connect "rx_p4i_bank" {cm get "tuto_p4i_bank";};
-> "[info]" (concat "home=" (sftp home "rx_p4i_bank") " protocol=" (sftp version "rx_p4i_bank"));
sftp cd "rx_p4i_bank" "/out/returns";
sftp lcd "rx_p4i_bank" "tmp/rx_p4i_bank_inbox";
-> "[archive]" (concat "/out/archive/" (date sysdate));
sftp mkdir "rx_p4i_bank" [archive];
#2. Claim each file with a rename: the bank does not write into .processing names;
json load "rx_p4i_ls" (sftp ls "rx_p4i_bank" "*.xml");
-> "[n]" 0;
json parse_array "rx_p4i_ls" "/" "[f]" {
	sftp rename "rx_p4i_bank" [f] (concat [f] ".processing");
	#3. Download, keep it only if it is not empty;
	sftp get "rx_p4i_bank" (concat [f] ".processing") (concat "tmp/rx_p4i_bank_inbox/" [f]);
	if (> (file size (concat "tmp/rx_p4i_bank_inbox/" [f])) 0) {
		#4. Archive on the server;
		sftp rename "rx_p4i_bank" (concat [f] ".processing") (concat [archive] "/" [f]);
		++ "[n]";
	} {
		sftp rename "rx_p4i_bank" (concat [f] ".processing") [f];
		log write "rx_p4i_bank" (concat "Empty file left on the server: " [f]);
	};
};
-> "[local]" (sftp lpwd "rx_p4i_bank");
sftp disconnect "rx_p4i_bank";
concat [info] [_n_] [n] " file(s) collected into " [local] " and archived in " [archive];

Incremental FTP mirror of a carrier's tracking files

A carrier publishes its tracking files on a legacy FTP server. Download only the files not yet present locally, in binary and passive mode, then move them to a done folder on the server so they are never downloaded twice.

#1. A session that works through the firewall;
cm set "tuto_p4i_carrier" {execute "file.remote.ftp.config.get"
	"[hostname]" "ftp.carrier.example.com"
	"[port]" "21"
	"[user]" "acme"
	"[password]" (parameter get value "tuto_p4i_carrier_pwd")
;};
ftp set timeout 15000;
ftp connect "rx_p4i_ftp" {cm get "tuto_p4i_carrier";};
ftp passive "rx_p4i_ftp";
ftp set type "rx_p4i_ftp" "BINARY";
ftp active compression "rx_p4i_ftp" true;
ftp cd "rx_p4i_ftp" "/tracking";
#2. What is new compared with the local mirror;
json load "rx_p4i_remote" (ftp ls "rx_p4i_ftp" "TRK_*.csv");
-> "[local]" (file dir_list "tmp/rx_p4i_mirror");
-> "[new]" 0;
json parse_array "rx_p4i_remote" "/" "[f]" {
	if (not (string contains [local] (concat "\"" [f] "\""))) {
		#3. Download, then 4. move to done on the server;
		ftp get "rx_p4i_ftp" [f] (concat "tmp/rx_p4i_mirror/" [f]);
		ftp rename "rx_p4i_ftp" [f] (concat "/tracking/done/" [f]);
		++ "[new]";
	};
};
-> "[where]" (ftp pwd "rx_p4i_ftp");
ftp disconnect "rx_p4i_ftp";
concat [new] " new tracking file(s) from " [where] ", " (json count "rx_p4i_remote" "/") " on the server";

Remote housekeeping of a Linux relay over SSH

Every night, check the disk of the transfer relay, compress the old transfer logs on the remote side, copy the daily report back to MentDB and alert when the disk is above 85 %.

#1. Connect to the relay;
ssh connect "rx_p4i_relay" {cm get "tuto_p4i_relay";};
#2. Disk usage of /data in percent;
-> "[used]" (string trim (ssh execute_1_cmd "rx_p4i_relay" "df -P /data | tail -1 | awk '{print $5}' | tr -d '%'"));
#3. Compress the logs older than 7 days and write a small report;
-> "[out]" (ssh execute_n_cmd "rx_p4i_relay" "find /data/transfer/logs -name '*.log' -mtime +7 -exec gzip {} \\;;du -sh /data/transfer > /tmp/relay_report.txt;date >> /tmp/relay_report.txt");
#4. Bring the report back;
ssh scp from "rx_p4i_relay" "/tmp/relay_report.txt" "tmp/rx_p4i_relay_report.txt";
ssh disconnect "rx_p4i_relay";
if (> [used] 85) {
	log write "rx_p4i_relay" (concat "Relay disk at " [used] " %");
};
concat "Disk /data: " [used] " %" [_n_] (file load "tmp/rx_p4i_relay_report.txt");

Ask a cluster of remote MentDB nodes for their transfer queue

Two remote MentDB engines relay files to the plants. Declare the remote engine, ask one node of the cluster on the fly how many files wait in its outbox, and run the stored dispatch script through the cluster with a 50/50 load balancing.

#1. The remote engine of the plant;
cm set "tuto_p4i_plant1" {execute "mentdb.remote.config.get"
	"[hostname]" "mentdb-plant1.acme.example.com"
	"[port]" "9998"
	"[key]" (parameter get value "tuto_p4i_plant_key")
	"[user]" "relay"
	"[password]" (parameter get value "tuto_p4i_plant_pwd")
	"[connectTimeout]" "10000"
	"[readTimeout]" "30000"
	"[subTunnels]" "[MQL]"
;};
#2. On the fly: a node is picked in the cluster, the MQL text runs there through a hot tunnel;
-> "[queue]" (include "cluster.1n.hot.exe"
	"[cluster_id]" "tuto_p4i_plants"
	"[method]" "LB_50_50"
	"[tunnel_id]" "rx_p4i_plant_hot"
	"[request]" "execute \"plant.outbox.count.get\";"
;);
#3. Same through a classic tunnel session: connect, execute, disconnect;
-> "[sent]" (execute "cluster.1n.exe"
	"[cluster_id]" "tuto_p4i_plants"
	"[method]" "LB_50_50"
	"[request]" "execute \"plant.outbox.dispatch.exe\";"
;);
concat "Files waiting on the chosen node: " [queue] [_n_] "Dispatch answer: " [sent];