Cluster & high-availability engineer

Is this server ready for a cluster?

Before declaring prod_cluster, the engineer lists the clusters and the signals already received (keys masked), reads the node name and the expulsion delay in the CLUSTER section of the config, and checks the heartbeat job.

#1. Clusters declared here and the target one;
json load "rx_cl" (cluster show);
-> "[target]" "prod_cluster";
#2. Signals received from the agents, keys and passwords masked;
json load "rx_sig" (cluster signal show_obj);
#3. Node name and expulsion delay in the server config;
json load "rx_cfg" (config show);
for (-> "[i]" 0) (< [i] (json count "rx_cfg" /)) (++ "[i]") {
	if (equal (json select "rx_cfg" (concat "/[" [i] "]/section")) "CLUSTER") {
		concat_var "[conf]" (json select "rx_cfg" (concat "/[" [i] "]/key")) "=" (json select "rx_cfg" (concat "/[" [i] "]/value")) " ";
	};
};
#4. The heartbeat job that sends the signals;
json load "rx_jobs" (job show);
concat "{\"clusters\": " (json count "rx_cl" /) ", \"target_exists\": " (cluster exist [target]) ", \"signal_clusters\": " (json count "rx_sig" /) ", \"config\": \"" (string trim [conf]) "\", \"heartbeat_job\": " (job exist "server.cluster.signal") ", \"heartbeat\": \"" (json select "rx_jobs" "/server.cluster.signal/pattern") "\", \"active\": " (json select "rx_jobs" "/server.cluster.signal/activate") "}"

{"clusters": 0, "target_exists": 0, "signal_clusters": 0, "config": "nodename=DEV_N1 life_before_expulsion=30000", "heartbeat_job": 1, "heartbeat": "0/30 * * * * ?", "active": 1}

Build a 2-node cluster and replicate the scripts

Two servers share the load. The engineer creates the cluster and its nodes with credentials taken from private parameters, deploys the CPU signal, checks the nodes, replicates the shop scripts to node_b (check then deploy) and asks which node would take the next request.

#1. Credentials of the cluster account come from private parameters;
-> "[key]" (parameter get value "CLUSTER_USER_KEY");
-> "[pwd]" (parameter get value "CLUSTER_USER_PWD");
#2. A cluster of two nodes, both answering the signal;
cluster create "prod_cluster";
cluster node set "prod_cluster" "node_a" "10.0.1.11" "9998" "cluster" [key] [pwd] 10000 30000 true;
cluster node set "prod_cluster" "node_b" "10.0.1.12" "9998" "cluster" [key] [pwd] 10000 30000 true;
#3. The signal is the system CPU of each node, node_b also reports its JVM CPU;
signal deploy "prod_cluster" "10.0.1.10" "9998" "cluster" [key] [pwd] 10000 30000 "metric current cpu system;";
cluster signal set "prod_cluster" "node_b" "10.0.1.10" "9998" "cluster" [key] [pwd] 10000 30000 "metric current cpu jvm;";
signal process;
cluster nodes show_txt "prod_cluster";
#4. Replicate the shop scripts to node_b: check first, then deploy;
cm set "node_b_mentdb" {execute "mentdb.remote.config.get"
	"[hostname]" "10.0.1.12"
	"[port]" "9998"
	"[key]" [key]
	"[user]" "cluster"
	"[password]" [pwd]
	"[connectTimeout]" "10000"
	"[readTimeout]" "30000"
	"[subTunnels]" "[MQL_TO_REPLACE]"
;};
execute "script.deploy.exe" "[startsWith]" "shop." "[mentdbCmId]" "node_b_mentdb" "[groups]" "shop_dev" "[check]" true;
execute "script.deploy.exe" "[startsWith]" "shop." "[mentdbCmId]" "node_b_mentdb" "[groups]" "shop_dev" "[check]" false;
#5. Which node would receive the next request;
json load "best" (cluster node "prod_cluster" "SIGNAL");
json select "best" /node_id

Failover of a node

Node A stops answering. The engineer expels it, reads the signals, runs an order request on the best remaining node with retries and through a tunnel, routes the nightly job through the cluster, then reinstates node A and removes an old node.

#1. Node A stops answering: take it out of the cluster;
cluster node expels "prod_cluster" "node_a" "Disk full on node_a";
cluster signal show "prod_cluster";
signal remote_show "prod_cluster";
#2. Run a request on the best remaining node, with retries;
-> "[order]" "A-1234";
execute "cluster.2n.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "SIGNAL"
	"[request]" (concat
		"-> \"[order]\" " (mql encode [order]) ";"
		(mql {
			execute "shop.orders.total.post" "[order]" [order];
		})
	)
;
#3. Same request on the fly through a tunnel, and the nightly job through the cluster;
include "cluster.1n.hot.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "LB_50_50"
	"[tunnel_id]" "tunnel_orders"
	"[request]" (concat "-> \"[order]\" " (mql encode [order]) ";" (mql {concat "checked " [order]}))
;
job set cluster_script "etl_orders_nightly" "cluster.1n.job_execution.exe";
#4. Node A is repaired: give it a low signal and put it back;
cluster signal give "prod_cluster" "node_a" 5 (date now);
cluster node reinstate "prod_cluster" "node_a";
#5. Decommission an old node;
cluster signal delete "prod_cluster" "node_old";
cluster node delete "prod_cluster" "node_old";
cluster nodes show_txt "prod_cluster"

Compute the load signal of this node

Before choosing the signal MQL of a cluster, the engineer reads the metrics of this node (system and JVM CPU, available memory, JVM heap, root disk, processors) and combines them into one load score: the worst of CPU, memory and heap, with a verdict for the load balancer. Values change at every run.

#1. Raw metrics of this node, as a signal would read them;
-> "[cpu]" (metric current cpu system);
-> "[cpu_jvm]" (metric current cpu jvm);
-> "[avail]" (metric current available mem);
-> "[total]" (metric total mem);
-> "[procs]" (metric system nb processor);
-> "[jvm]" (metric current mem jvm);
#2. JVM heap used / allocated / max in MB;
-> "[heap_pct]" (math round (* 100 (/ (atom get [jvm] 1 "/") (atom get [jvm] 3 "/"))) 1);
-> "[mem_pct]" (math round (* 100 (- 1 (/ [avail] [total]))) 1);
#3. Root disk usage;
json load "rx_fs" (metric file system roots);
-> "[disk_pct]" (math round (* 100 (/ (json select "rx_fs" "/[0]/usedSpace") (json select "rx_fs" "/[0]/totalSpace"))) 1);
json unload "rx_fs";
#4. One load score: the worst of CPU, memory and heap, and the verdict for the balancer;
-> "[score]" (math max (math max [cpu] [mem_pct]) [heap_pct]);
concat "{\"os\": \"" (metric system name) " " (metric system architecture) "\", \"java\": \"" (metric java version) "\", \"processors\": " [procs] ", \"cpu_sys\": " [cpu] ", \"cpu_jvm\": " [cpu_jvm] ", \"mem_used_pct\": " [mem_pct] ", \"heap_used_pct\": " [heap_pct] ", \"disk_used_pct\": " [disk_pct] ", \"signal\": " [score] ", \"accept_traffic\": " (and (< [score] 85) (< [disk_pct] 90)) "}"

{"os": "Mac OS X aarch64", "java": "21.0.13", "processors": 11, "cpu_sys": 7.5, "cpu_jvm": 0.2, "mem_used_pct": 72.9, "heap_used_pct": 5.1, "disk_used_pct": 68.5, "signal": 72.9, "accept_traffic": 1}

Retire a staging cluster cleanly

The staging cluster is no longer used. The engineer lists its nodes, keeps the code that rebuilds them and the remote signals, removes the signal and the node of each member, deletes the cluster and checks it is gone.

#1. Inventory of the staging cluster before it is retired;
-> "[cl]" "staging_cluster";
json load "rx_nodes" (cluster nodes show_obj [cl]);
#2. Keep the code that rebuilds the nodes, and the signals sent to the other servers;
-> "[backup]" (cluster node generate_set [cl]);
-> "[remote]" (signal remote_show [cl]);
#3. Remove the signal of each node, then the node itself;
json parse_obj "rx_nodes" "/" "[node_id]" "[node]" {
	cluster signal delete [cl] [node_id];
	cluster node delete [cl] [node_id];
};
#4. Delete the cluster and check;
cluster delete [cl];
-> "[left]" (cluster exist [cl]);
json unload "rx_nodes";
concat "cluster " [cl] " exists: " [left] " | rebuild code:" (string char 10) [backup] (string char 10) "remote signals:" (string char 10) [remote]

Expel and reinstate drill on a lab cluster

Before touching production, the failover procedure is rehearsed on a lab cluster: two nodes are declared (auto signal off), node_b is expelled with the error of its health check, then reinstated. The script shows the node table at each step and counts the backup lines before deleting the lab cluster.

#1. A lab cluster with two nodes declared, auto signal off;
cluster create "tuto_p4h_lab";
cluster node set "tuto_p4h_lab" "node_a" "node-a.lab.local" "9998" "svc_cluster" "change_me" "change_me" 10000 30000 false;
cluster node set "tuto_p4h_lab" "node_b" "node-b.lab.local" "9998" "svc_cluster" "change_me" "change_me" 10000 30000 false;
-> "[t0]" (cluster nodes show_txt "tuto_p4h_lab");
#2. Drill: node_b fails its health check and is expelled;
cluster node expels "tuto_p4h_lab" "node_b" "health check timeout after 30000 ms";
-> "[t1]" (cluster nodes show_txt "tuto_p4h_lab");
#3. node_b is repaired and reinstated;
cluster node reinstate "tuto_p4h_lab" "node_b";
-> "[t2]" (cluster nodes show_txt "tuto_p4h_lab");
-> "[backup]" (string count (cluster node generate_set "tuto_p4h_lab") "cluster node set");
#4. Remove the lab cluster;
cluster node delete "tuto_p4h_lab" "node_a";
cluster node delete "tuto_p4h_lab" "node_b";
cluster delete "tuto_p4h_lab";
concat "declared:" [_n_] [t0] [_n_] "after expel:" [_n_] [t1] [_n_] "after reinstate:" [_n_] [t2] [_n_] "backup lines: " [backup] " | lab cluster left: " (cluster exist "tuto_p4h_lab")

declared:
IN_THE_CLUSTER / ACTIVE_SIGNAL / CLUSTER_ID / NODE_ID / USER@HOSTNAME:PORT / CONNECT_TIMEOUT / READ_TIMEOUT / SIGNAL / LAST_TIME / NB_CMD : ERROR
1 / 0 / tuto_p4h_lab / node_a / svc_cluster@node-a.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 :
1 / 0 / tuto_p4h_lab / node_b / svc_cluster@node-b.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 :

after expel:
IN_THE_CLUSTER / ACTIVE_SIGNAL / CLUSTER_ID / NODE_ID / USER@HOSTNAME:PORT / CONNECT_TIMEOUT / READ_TIMEOUT / SIGNAL / LAST_TIME / NB_CMD : ERROR
1 / 0 / tuto_p4h_lab / node_a / svc_cluster@node-a.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 :
0 / 0 / tuto_p4h_lab / node_b / svc_cluster@node-b.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 : health check timeout after 30000 ms

after reinstate:
IN_THE_CLUSTER / ACTIVE_SIGNAL / CLUSTER_ID / NODE_ID / USER@HOSTNAME:PORT / CONNECT_TIMEOUT / READ_TIMEOUT / SIGNAL / LAST_TIME / NB_CMD : ERROR
1 / 0 / tuto_p4h_lab / node_a / svc_cluster@node-a.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 :
1 / 0 / tuto_p4h_lab / node_b / svc_cluster@node-b.lab.local:9998 / 10000 / 30000 / 0 / 2026-11-01 18:32:00 / 0 :

backup lines: 2 | lab cluster left: 0

Round robin or signal: which node gets the request?

Three lab nodes are declared. The script shows which node the cluster picks for 6 requests in LB_50_50 (round robin), then gives a load signal to each node and shows the node chosen in SIGNAL mode (the lowest load).

#1. A lab cluster of three nodes, auto signal off;
cluster create "tuto_p4h_lb";
cluster node set "tuto_p4h_lb" "node_a" "node-a.lab.local" "9998" "svc_cluster" "change_me" "change_me" 10000 30000 false;
cluster node set "tuto_p4h_lb" "node_b" "node-b.lab.local" "9998" "svc_cluster" "change_me" "change_me" 10000 30000 false;
cluster node set "tuto_p4h_lb" "node_c" "node-c.lab.local" "9998" "svc_cluster" "change_me" "change_me" 10000 30000 false;
#2. Round robin: 6 requests, only the chosen node id is kept;
-> "[rr]" "";
for (-> "[i]" 0) (< [i] 6) (++ "[i]") {
	json load "rx_p4h_n" (cluster node "tuto_p4h_lb" "LB_50_50");
	-> "[rr]" (concat [rr] " " (json select "rx_p4h_n" "/node_id"));
};
#3. Signal mode: each node reports its load, the lowest one is chosen;
cluster signal give "tuto_p4h_lb" "node_a" 82 (date systimestamp);
cluster signal give "tuto_p4h_lb" "node_b" 35 (date systimestamp);
cluster signal give "tuto_p4h_lb" "node_c" 57 (date systimestamp);
json load "rx_p4h_s" (cluster node "tuto_p4h_lb" "SIGNAL");
-> "[sig]" (json select "rx_p4h_s" "/node_id");
#4. Remove the lab cluster;
cluster node delete "tuto_p4h_lb" "node_a";
cluster node delete "tuto_p4h_lb" "node_b";
cluster node delete "tuto_p4h_lb" "node_c";
cluster delete "tuto_p4h_lb";
concat "LB_50_50 over 6 requests:" [rr] " | SIGNAL with loads a=82 b=35 c=57 -> " [sig] " | cluster left: " (cluster exist "tuto_p4h_lb")

LB_50_50 over 6 requests: node_a node_b node_c node_a node_b node_c | SIGNAL with loads a=82 b=35 c=57 -> node_b | cluster left: 0

Run MQL on the least loaded node of the cluster

A heavy monthly report is sent to the least loaded node of the production cluster (SIGNAL method), with a local variable encoded into the remote request. Many small stock calls go through the hot variant, which keeps a tunnel open, in round robin.

#1. Run a heavy report on the least loaded node of the production cluster;
-> "[month]" "2026-10";
-> "[report]" (execute "cluster.1n.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "SIGNAL"
	"[request]" (concat
		"-> \"[m]\" " (mql encode [month]) ";"
		(mql {
			concat (name) " built the report of " [m] ": " (execute "report.monthly.sales.exe" "[month]" [m])
		})
	)
);
#2. Many small calls: keep the tunnel open with the hot variant, round robin;
-> "[stock]" (include "cluster.1n.hot.exe"
	"[cluster_id]" "prod_cluster"
	"[method]" "LB_50_50"
	"[tunnel_id]" "tunnel_stock"
	"[request]" (mql {
		concat (name) ": " (execute "stock.level.get" "[sku]" "BK-0042")
	})
);
concat [report] [_n_] [stock]

How many nodes for Black Friday?

Traffic is expected to be multiplied by 6. From the live CPU and memory of this node, the script computes how many identical nodes keep each node below 70% CPU and 80% RAM, then adds one node for N+1 redundancy. The figures change with the load of the server.

#1. Load of this node now;
-> "[cpus]" (metric system nb processor);
-> "[cpu]" (math max 1 (metric current cpu system));
-> "[ram_used]" (- (metric total mem) (metric current available mem));
-> "[ram]" (metric total mem);
#2. Black Friday: traffic x6, keep each node below 70 percent CPU and 80 percent RAM;
-> "[cpu_need]" (/ (* [cpu] 6) 70);
-> "[ram_need]" (/ (* [ram_used] 6) (* 0.8 [ram]));
-> "[nodes]" (math ceil (math max [cpu_need] [ram_need]));
#3. N+1 redundancy: one more node to survive a failure;
concat "now: cpu " [cpu] "% on " [cpus] " CPUs, RAM " (math round (/ (* 100 [ram_used]) [ram]) 1) "% | at x6: cpu needs " (math round [cpu_need] 2) " nodes, RAM needs " (math round [ram_need] 2) " nodes | plan: " [nodes] " + 1 = " (+ [nodes] 1) " identical nodes"

now: cpu 14.6% on 11 CPUs, RAM 72.8% | at x6: cpu needs 1.25 nodes, RAM needs 5.46 nodes | plan: 6 + 1 = 7 identical nodes

Availability, MTTR, MTBF and SLA of a 2-node cluster

From the outages of one node over 90 days, the script computes its availability, MTTR and MTBF, then the availability of two nodes in active-active (1 - (1 - A)^2), the number of nines and the downtime budget left against a 99.95% SLA.

#1. Last 90 days of one node: outages in minutes;
-> "[outages]" "[12, 45, 7, 30, 18]";
-> "[period]" (math calc "90*24*60");
#2. Availability, MTTR and MTBF of one node;
-> "[down]" (stat sum [outages]);
-> "[A]" (/ (- [period] [down]) [period]);
-> "[mttr]" (stat mean [outages]);
json load "rx_p4h_o" [outages];
-> "[mtbf]" (/ (- [period] [down]) (json count "rx_p4h_o" /));
#3. Two nodes in active-active: down only if both are down;
-> "[A2]" (math calc "1-(1-[A])^2");
-> "[nines]" (math floor (math calc "-log(1-[A2])"));
concat "1 node: " (math round (* 100 [A]) 4) "% MTTR " [mttr] " min MTBF " (math round (/ [mtbf] 1440) 1) " days | 2 nodes: " (math round (* 100 [A2]) 6) "% (" [nines] " nines) | SLA 99.95%: " (if (>= [A2] 0.9995) {"met"} {"missed"}) ", downtime budget left " (math round (- (* 0.0005 [period]) (* (- 1 [A2]) [period])) 1) " min"

1 node: 99.9136% MTTR 22.4 min MTBF 18 days | 2 nodes: 99.999925% (6 nines) | SLA 99.95%: met, downtime budget left 64.7 min