Profiles / Engineering & data
IS

IoT & sensor engineer

Sensor calibration, smoothing, drift detection and measurement uncertainty.

17 recipes on this page · open in the interactive view

Recipes
  1. Linear calibration of a pressure sensor
  2. Drift detection of a humidity sensor
  3. Temperature uncertainty of a PT100 probe
  4. Remove mains hum from a sensor by FFT
  5. Battery life of a LoRaWAN sensor node
  6. Health report of an IoT gateway
  7. Decode and acknowledge a Modbus status register
  8. LoRaWAN link quality report of a sensor
  9. A/B test of a firmware update on the radio latency
  10. 10-minute wind report of a weather station
  11. Smooth a noisy temperature with a Kalman filter
  12. Detect abnormal machines with an isolation forest
  13. Quantiles and distinct devices of a telemetry stream
  14. Alarm sequences that precede a failure
  15. Route the maintenance tickets
  16. Event combinations that trip a pump
  17. Detect a change in a sensor stream
executed on MentDB

Linear calibration of a pressure sensor

Raw ADC counts of a pressure sensor are recorded against a reference gauge. The script fits the gain and offset, checks the linearity and converts a new reading into bar.

Pipeline
Script
admin
#1. Raw ADC counts of a pressure sensor against a reference gauge in bar;
-> "[adc]" "[412, 1235, 2050, 2871, 3690]";
-> "[ref]" "[0.00, 2.51, 5.00, 7.52, 10.01]";
#2. Linear calibration p = a + b*x;
json load "rx_iot_cal" (stat fit [adc] [ref] "linear");
-> "[expr]" (json select "rx_iot_cal" /expression);
#3. Convert a new reading of 3000 counts;
-> "[p]" (math eval [expr] "{\"x\": 3000}");
concat "offset a = " (math round_sig (json select "rx_iot_cal" /a) 4) " bar, gain b = " (math round_sig (json select "rx_iot_cal" /b) 4) " bar/count, r2 = " (math round (json select "rx_iot_cal" /r2) 6) ", 3000 counts = " (math round [p] 3) " bar"
Result
mentdb
offset a = -1.260 bar, gain b = 0.003055 bar/count, r2 = 0.999998, 3000 counts = 7.906 bar
Go further
  • Use stat fit_model to get the standard errors of the gain and the offset
  • Fit a second order polynomial with pa polynomial_curve_fit_get_coeff for non-linear sensors
  • Compute the non-linearity as the largest residual in percent of the full scale
executed on MentDB

Drift detection of a humidity sensor

A humidity sensor is compared every day with a reference. The script smooths the offsets, tests for a monotonic trend and estimates the drift per month to decide a recalibration.

Pipeline
Script
admin
#1. Daily offset of a humidity sensor against a reference, in percent RH;
-> "[d]" "[0.1, 0.3, 0.2, 0.4, 0.3, 0.6, 0.5, 0.7, 0.9, 0.8, 1.0, 1.1, 1.0, 1.3]";
#2. Smooth the daily noise with a 3-day moving average;
-> "[m]" (dsp moving_average [d] 3);
#3. Monotonic trend test and slope of the drift;
json load "rx_iot_k" (stat kendall "[1,2,3,4,5,6,7,8,9,10,11,12,13,14]" [d]);
json load "rx_iot_f" (stat fit "[1,2,3,4,5,6,7,8,9,10,11,12,13,14]" [d] "linear");
-> "[drift]" (math calc (concat "30*" (json select "rx_iot_f" /b)));
-> "[p]" (json select "rx_iot_k" /p_value);
concat "smoothed offset from " (math round (stat min [m]) 2) " to " (math round (stat max [m]) 2) " %RH, Kendall p-value = " (math round_sig [p] 3) ", drift = " (math round [drift] 2) " %RH per 30 days -> " (if (and (< [p] 0.05) (> [drift] 1)) {"recalibrate"} {"OK"})
Result
mentdb
smoothed offset from 0.2 to 1.13 %RH, Kendall p-value = 0.0000144, drift = 2.58 %RH per 30 days -> recalibrate
Go further
  • Use stat spearman as another trend test
  • Remove the trend with dsp detrend and check the remaining noise with stat stddev
  • Run the check every night with a job on the stored readings
executed on MentDB

Temperature uncertainty of a PT100 probe

A PT100 probe gives 109.73 ohm. The script converts it into a temperature, propagates the uncertainties of R and R0 and writes the result with an expanded uncertainty k = 2.

Pipeline
Script
admin
#1. PT100 probe: R measured with its uncertainty, R0 and alpha of the class;
-> "[vars]" "{\"R\": [109.73, 0.02], \"R0\": [100, 0.01], \"alpha\": 0.00385}";
#2. Propagate the uncertainty through T = R/R0 - 1 / alpha;
json load "rx_iot_u" (num uncertainty "(R/R0-1)/alpha" [vars]);
#3. Expanded uncertainty k = 2, written with the usual rounding;
-> "[T]" (json select "rx_iot_u" /value);
-> "[U]" (math calc (concat "2*" (json select "rx_iot_u" /uncertainty)));
concat "T = " (math round_uncertainty [T] [U]) " degC (k = 2), share of R0 in the variance: " (math round (math calc (concat "100*" (json select "rx_iot_u" /variance_share/R0))) 0) " %"
Result
mentdb
T = 25.27 ± 0.12 degC (k = 2), share of R0 in the variance: 23 %
Go further
  • Use the Callendar-Van Dusen equation with num solve above 0 degC
  • Add the self-heating error as another variable
  • Read variance_share to find the dominant source of uncertainty
executed on MentDB

Remove mains hum from a sensor by FFT

A vibration sensor picks up 50 Hz mains hum and its 150 Hz harmonic. With 1024 samples at 1024 Hz, the hum falls exactly into FFT bins: the script zeroes them, goes back with the inverse FFT, smooths the noise with Savitzky-Golay and draws the spectrum of the cleaned signal.

Pipeline
Script
admin
#1. Vibration sensor sampled at 1024 Hz for 1 s: slow 1.5 Hz motion, 50 Hz mains hum and its 150 Hz harmonic, a little noise;
-> "[x]" (vector add (matrix col (num table "2+0.5*sin(2*pi*1.5*t)+0.3*sin(2*pi*50*t)+0.1*sin(2*pi*150*t)" "t" 0 0.9990234375 0.0009765625) 2) (stat dist_random "normal" "[0, 0.05]" 1024));
#2. FFT: 1 Hz per bin, so the hum is exactly in bins 50 and 150 and their mirrors 974 and 874;
json load "rx_X" (concat "{\"X\": " (num fft [x]) "}");
json uarray "rx_X" /X 50 "0" STR; json uarray "rx_X" /X 974 "0" STR;
json uarray "rx_X" /X 150 "0" STR; json uarray "rx_X" /X 874 "0" STR;
#3. Back to the time domain, then Savitzky-Golay smoothing of the noise;
-> "[clean]" (dsp savgol (num ifft (json select "rx_X" /X)) 21 3);
#4. Spectrum of the cleaned signal: no more line at 50 and 150 Hz;
plot spectrum [clean] 1024
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Do the same with dsp filter bandstop around 50 Hz when the sampling does not match the bins.
  • Draw raw and cleaned signals together with plot chart.
executed on MentDB

Battery life of a LoRaWAN sensor node

A LoRaWAN node sleeps at 4 uA, measures for 50 ms every 10 s and transmits at SF12 for 400 ms every 5 minutes. Compute the average current, the battery life on two derated AA lithium cells and the date of the replacement visit, moved to a working day.

Pipeline
Script
admin
#1. LoRaWAN node: sleep 4 uA, measure 12 mA for 50 ms every 10 s, radio SF12 45 mA for 400 ms every 5 min;
-> "[sleep]" (unit calc "4 uA" "mA");
-> "[meas]" (math calc "12*0.05/10");
-> "[tx]" (math calc "45*0.4/300");
#2. Average current of the duty cycle;
-> "[iavg]" (math calc "[sleep]+[meas]+[tx]");
#3. Two AA lithium cells of 3000 mAh in parallel, 80 pct usable after derating;
-> "[hours]" (math calc "2*3000*0.8/[iavg]");
-> "[days]" (math floor (/ [hours] 24));
#4. Replacement date from the installation day, moved to a working day;
-> "[install]" "2026-11-02";
-> "[change]" (date add [install] "DAY" [days]);
if (== (date is_weekend [change]) 1) {-> "[change]" (date next_day [change] "monday");};
concat "average current " (math round (* 1000 [iavg]) 1) " uA | battery life " [days] " days = " (math round (/ [days] 365.25) 2) " years | replacement visit on " [change] ", ISO week " (date week_of_year [change]) ", " (date nb_business_days [install] [change]) " business days after installation"
Result
mentdb
average current 124 uA | battery life 1612 days = 4.41 years | replacement visit on 2031-04-02, ISO week 14, 1152 business days after installation
Go further
  • Compare SF7 and SF12 airtimes: the radio term drops by about 20.
  • Use date add_business_days to plan a check visit every 60 working days.
executed on MentDB

Health report of an IoT gateway

An edge gateway running MentDB must publish a small health message every few minutes. The script reads the live CPU and available memory, extracts the number of cores and the disk usage from the system report, derives an anonymous gateway id from the hostname and builds the JSON status message.

Pipeline
Script
admin
#1. Live resources;
-> "[cpu]" (metric current cpu jvm);
-> "[mem]" (metric current available mem);
#2. Cores and disk usage of the root file system, from the system report;
-> "[sys]" (metric system);
-> "[cores]" (string regexp_extract [sys] "nb_proc: ([0-9]+)" 1);
-> "[disk]" (string regexp_extract [sys] "\[/\] ([0-9]+)%" 1);
#3. Anonymous gateway id: first 8 hex digits of the SHA of the hostname;
-> "[gid]" (string left (string sha (os hostname)) 8);
#4. Status message;
json load "rx_p4f_gw" "{}";
json iobject "rx_p4f_gw" / "gateway" (concat "gw-" [gid]) STR;
json iobject "rx_p4f_gw" / "cores" [cores] NUM;
json iobject "rx_p4f_gw" / "history_days" (string regexp_count (metric date) "[0-9]{4}-[0-9]{2}-[0-9]{2}") NUM;
json iobject "rx_p4f_gw" / "status" (if (or (> [cpu] 80) (or (< [mem] 500) (> [disk] 90))) {"degraded"} {"ok"}) STR;
json iobject "rx_p4f_gw" / "checks" (concat "cpu " (if (> [cpu] 80) {"high"} {"normal"}) ", memory " (if (< [mem] 500) {"low"} {"normal"}) ", disk " (if (> [disk] 90) {"full"} {"normal"})) STR;
json doc "rx_p4f_gw"
Result
mentdb
{
  "cores": 11.0,
  "checks": "cpu normal, memory normal, disk normal",
  "history_days": 31.0,
  "gateway": "gw-eedfe8d8",
  "status": "ok"
}
Go further
  • Add the free swap and the JVM heap with metric current used mem jvm
  • Publish the message every 5 minutes with a job and keep the last 31 days with metric date
executed on MentDB

Decode and acknowledge a Modbus status register

A field sensor exposes a 16-bit Modbus status register (read as hex 02C5). The script decodes the active flags with bit masks, acknowledges the two alarm bits with an XOR, sets the maintenance-mode bit with an OR and returns the register to write back in hex and binary.

Pipeline
Script
admin
#1. Register read on the bus and the meaning of its bits;
-> "[reg]" (string hex_to_int "02C5");
json load "rx_p4f_bits" "[[\"running\", 0], [\"low_battery\", 2], [\"over_temp\", 3], [\"sensor_fault\", 6], [\"comm_warning\", 7], [\"door_open\", 9]]";
-> "[active]" "";
#2. Active flags;
for (-> "[i]" 0) (< [i] (json count "rx_p4f_bits" /)) (++ "[i]") {
  if (> (math bit_and [reg] (math power 2 (json select "rx_p4f_bits" (concat "/[" [i] "]/[1]")))) 0) {
    -> "[active]" (concat [active] (json select "rx_p4f_bits" (concat "/[" [i] "]/[0]")) " ");
  };
};
json unload "rx_p4f_bits";
#3. Acknowledge the alarms of bits 6 and 7, then set maintenance mode bit 12;
-> "[ack]" (math bit_xor [reg] (math bit_and [reg] 192));
-> "[new]" (math bit_or [ack] 4096);
concat "0x02C5 = " (string lpad (string bin [reg]) "0" 16) " | active: " (string trim [active]) " | write back 0x" (string upper (string lpad (string int_to_hex [new]) "0" 4)) " = " (string lpad (string bin [new]) "0" 16)
Result
mentdb
0x02C5 = 0000001011000101 | active: running low_battery sensor_fault comm_warning door_open | write back 0x1205 = 0001001000000101
Go further
  • Compare two successive reads with math bit_xor to log only the flags that changed
  • Write the register back with a Modbus gateway REST call
executed on MentDB

A/B test of a firmware update on the radio latency

A new firmware is deployed on part of the fleet. The script compares the end-to-end latencies of the two groups with the non-parametric Mann-Whitney test, then checks with a z-test (known jitter 4 ms from the radio datasheet) whether the new firmware meets the 125 ms target.

Pipeline
Script
admin
#1. Latencies in ms of the old and new firmware groups;
-> "[old]" "[128, 135, 122, 140, 131, 126, 138, 133]";
-> "[new]" "[118, 121, 115, 129, 119, 123, 117, 120, 124]";
#2. Non-parametric comparison of the two groups;
json load "rx_p4f_mw" (stat mann_whitney [old] [new]);
-> "[p]" (json select "rx_p4f_mw" "/p_value"); -> "[method]" (json select "rx_p4f_mw" "/method");
#3. z-test of the new group against the 125 ms target, sigma known = 4 ms;
json load "rx_p4f_z" (stat ztest [new] 125 4);
-> "[z]" (json select "rx_p4f_z" "/z"); -> "[pz]" (json select "rx_p4f_z" "/p_value");
json unload "rx_p4f_mw"; json unload "rx_p4f_z";
concat "median old " (stat median [old]) " ms, new " (stat median [new]) " ms | Mann-Whitney " [method] " p = " (math round_sig [p] 3) (if (< [p] 0.05) {" -> significant change"} {" -> no proof of change"}) " | z = " (math round [z] 3) ", p = " (math round_sig [pz] 3) (if (and (< [z] 0) (< [pz] 0.05)) {" -> mean below 125 ms: roll out"} {" -> target not proven"})
Result
mentdb
median old 132 ms, new 120 ms | Mann-Whitney exact p = 0.00156 -> significant change | z = -3.25, p = 0.00115 -> mean below 125 ms: roll out
Go further
  • Use stat bootstrap style resampling of the medians to give a confidence interval
  • Check the latency distribution of each group with stat frequency
executed on MentDB

10-minute wind report of a weather station

A weather station measures the wind direction every minute around north, where a simple average fails (350 and 10 deg average to 180). The script computes the vector mean direction with atan2, the Yamartino standard deviation of the direction and the resultant wind speed, as in WMO reports.

Pipeline
Script
admin
#1. Directions in deg and speeds in m/s, one per minute;
json load "rx_p4f_dir" "[350, 355, 5, 10, 358, 2, 15, 345, 8, 0]";
json load "rx_p4f_v" "[6.1, 5.8, 6.4, 7.0, 6.2, 5.9, 7.3, 5.5, 6.6, 6.0]";
json load "rx_p4f_s" "[]"; json load "rx_p4f_c" "[]"; json load "rx_p4f_u" "[]"; json load "rx_p4f_w" "[]";
#2. Unit vector components and wind vector components;
for (-> "[i]" 0) (< [i] 10) (++ "[i]") {
  -> "[a]" (math calc (concat (json select "rx_p4f_dir" (concat "/[" [i] "]")) "*pi/180"));
  -> "[v]" (json select "rx_p4f_v" (concat "/[" [i] "]"));
  json iarray "rx_p4f_s" / (math sin [a]) NUM; json iarray "rx_p4f_c" / (math cos [a]) NUM;
  json iarray "rx_p4f_u" / (* [v] (math sin [a])) NUM; json iarray "rx_p4f_w" / (* [v] (math cos [a])) NUM;
};
#3. Vector mean direction and Yamartino sigma;
-> "[sa]" (math avg (json doc "rx_p4f_s")); -> "[ca]" (math avg (json doc "rx_p4f_c"));
-> "[dir]" (math calc (concat (math atan2 [sa] [ca]) "*180/pi"));
-> "[e]" (math sqrt (- 1 (+ (* [sa] [sa]) (* [ca] [ca]))));
-> "[sig]" (math calc (concat (math asin [e]) "*(1+0.1547*" [e] "^3)*180/pi"));
#4. Resultant wind speed;
-> "[vr]" (math hypot (math avg (json doc "rx_p4f_u")) (math avg (json doc "rx_p4f_w")));
-> "[naive]" (math avg (json doc "rx_p4f_dir"));
json unload "rx_p4f_dir"; json unload "rx_p4f_v"; json unload "rx_p4f_s"; json unload "rx_p4f_c"; json unload "rx_p4f_u"; json unload "rx_p4f_w";
concat "vector mean direction " (math round (if (< [dir] 0) {+ [dir] 360} {[dir]}) 1) " deg (simple average " [naive] " deg) | Yamartino sigma " (math round [sig] 2) " deg | resultant speed " (math round [vr] 2) " m/s"
Result
mentdb
vector mean direction 0.8 deg (simple average 144.8 deg) | Yamartino sigma 8.74 deg | resultant speed 6.21 m/s
Go further
  • Fit a Weibull distribution of the speeds and get the mean with math gamma of 1 + 1/k
  • Flag a stuck vane when sigma stays at 0 for an hour
executed on MentDB

Smooth a noisy temperature with a Kalman filter

Filter 12 noisy readings of a slowly drifting temperature probe with a local level Kalman filter, then give the last filtered value, its standard deviation and the largest correction of the smoother.

Pipeline
Script
admin
#1. Readings in C, sensor noise variance 0.09, drift variance 0.005;
-> "[t]" "[20.1, 20.6, 19.8, 20.4, 20.9, 20.2, 20.7, 21.1, 20.5, 21.0, 21.4, 20.9]";
json load "rx_k" (ml ts kalman [t] 0.005 0.09);
#2. Largest gap between a reading and the smoothed value;
-> "[gap]" 0;
json load "rx_t" [t];
for (-> "[i]" 0) (< [i] 12) (++ "[i]") {
  -> "[d]" (math abs (- (json select "rx_t" (concat "/[" [i] "]")) (json select "rx_k" (concat "/smoothed/[" [i] "]"))));
  if (> [d] [gap]) {-> "[gap]" [d];};
};
-> "[res]" (concat "last filtered " (math round (json select "rx_k" "/filtered/[11]") 3) " C +/- " (math round (math sqrt (json select "rx_k" "/variance/[11]")) 3) " | largest correction " (math round [gap] 3) " C");
json unload "rx_k"; json unload "rx_t";
[res]
Result
mentdb
last filtered 20.824 C +/- 0.138 | largest correction 0.628 C
Go further
  • A larger q follows faster changes, a larger r trusts the sensor less.
  • ml ts ema is a simpler smoothing with a fixed factor.
executed on MentDB

Detect abnormal machines with an isolation forest

Learn the normal behavior of a motor from vibration RMS, temperature and current, then score new measurements and flag the anomalies; confirm with a one-class SVM.

Pipeline
Script
admin
#1. Normal operation: RMS mm/s, temperature C, current A;
-> "[normal]" "[[2.1, 55, 10.2], [2.3, 56, 10.4], [2.0, 54, 10.1], [2.2, 55, 10.3], [2.4, 57, 10.6], [2.1, 56, 10.2], [2.3, 55, 10.5], [2.2, 54, 10.3], [2.0, 55, 10.0], [2.5, 57, 10.7], [2.2, 56, 10.4], [2.1, 54, 10.2]]";
ml fit "rx_iso" "isolation_forest" [normal] "{\"contamination\": 0.05, \"seed\": 3}";
#2. New measurements;
-> "[new]" "[[2.2, 55, 10.3], [4.8, 63, 12.9], [2.3, 56, 10.5]]";
-> "[flags]" (ml model predict "rx_iso" [new]);
json load "rx_s" (ml model transform "rx_iso" [new]);
#3. Second opinion: one-class SVM learned on the same normal data;
ml fit "rx_oc" "one_class_svm" [normal] "{\"nu\": 0.05}";
-> "[res]" (concat "isolation forest flags " [flags] " | scores " (math round (json select "rx_s" "/[0]/[0]") 3) ", " (math round (json select "rx_s" "/[1]/[0]") 3) ", " (math round (json select "rx_s" "/[2]/[0]") 3) " | one-class SVM flags " (ml model predict "rx_oc" [new]));
json unload "rx_s"; ml model delete "rx_iso"; ml model delete "rx_oc";
[res]
Result
mentdb
isolation forest flags [0, 1, 0] | scores 0.428, 0.636, 0.488 | one-class SVM flags [0, 1, 0]
Go further
  • ml anomaly mahalanobis works on a large history where one outlier cannot inflate the covariance.
  • Save the detector with ml model save and score each new frame in a file watcher.
executed on MentDB

Quantiles and distinct devices of a telemetry stream

Summarize a stream of latency measures and device ids without keeping it: 95th and 99th percentiles with a T-Digest, number of distinct devices with HyperLogLog and the 3 noisiest devices with Top-K.

Pipeline
Script
admin
#1. The sketches;
ml sketch create "rx_lat" "tdigest";
ml sketch create "rx_dev" "hll";
ml sketch create "rx_top" "topk" "{\"capacity\": 100}";
#2. Two batches of telemetry: latency in ms and device id;
ml sketch add "rx_lat" "[12, 15, 11, 19, 14, 120, 13, 16, 12, 18, 14, 13, 17, 15, 250, 12, 16, 14, 13, 15]";
ml sketch add "rx_dev" "[\"s01\", \"s02\", \"s03\", \"s01\", \"s04\", \"s07\", \"s02\", \"s01\", \"s05\", \"s07\"]";
ml sketch add "rx_top" "[\"s01\", \"s02\", \"s03\", \"s01\", \"s04\", \"s07\", \"s02\", \"s01\", \"s05\", \"s07\"]";
ml sketch add "rx_dev" "[\"s06\", \"s01\", \"s08\", \"s07\"]";
ml sketch add "rx_top" "[\"s06\", \"s01\", \"s08\", \"s07\"]";
#3. Read the summaries;
json load "rx_t" (ml sketch query "rx_top" 3);
json load "rx_d" (ml sketch result "rx_dev");
-> "[res]" (concat "p50 " (ml sketch query "rx_lat" 0.5) " ms | p95 " (math round (ml sketch query "rx_lat" 0.95) 1) " ms | p99 " (math round (ml sketch query "rx_lat" 0.99) 1) " ms | share <= 20 ms: " (ml sketch cdf "rx_lat" 20) " | distinct devices " (json select "rx_d" "/distinct") " | top: " (json select "rx_t" "/[0]/value") " x" (json select "rx_t" "/[0]/count") ", " (json select "rx_t" "/[1]/value") " x" (json select "rx_t" "/[1]/count"));
json unload "rx_t"; json unload "rx_d";
ml sketch delete "rx_lat"; ml sketch delete "rx_dev"; ml sketch delete "rx_top";
[res]
Result
mentdb
p50 14.5 ms | p95 126.5 ms | p99 225.3 ms | share <= 20 ms: 0.9 | distinct devices 8 | top: s01 x4, s07 x3
Go further
  • The T-Digest keeps the exact values up to 10 000 measures, then a compressed summary of constant size.
  • A ml sketch create bloom filter tells in constant memory if a message id was already processed.
executed on MentDB

Alarm sequences that precede a failure

From the alarm histories of 6 machines, find the sequences of alarms that come before a stop, and the sequential rules with their confidence to trigger a preventive maintenance.

Pipeline
Script
admin
#1. Alarm history of each machine, in time order;
-> "[h]" "[[\"T_HIGH\", \"VIB\", \"P_LOW\", \"STOP\"], [\"VIB\", \"T_HIGH\", \"STOP\"], [\"T_HIGH\", \"P_LOW\", \"VIB\", \"STOP\"], [\"P_LOW\", \"OK\"], [\"T_HIGH\", \"VIB\", \"STOP\"], [\"VIB\", \"OK\"]]";
json load "rx_sp" (ml assoc sequences [h] 0.5);
json load "rx_sr" (ml assoc sequence_rules [h] 0.5 0.8);
#2. The rules that end with a STOP;
-> "[res]" (concat (json select "rx_sp" "/count") " frequent sequences, " (json select "rx_sr" "/count") " rules | before STOP: ");
for (-> "[i]" 0) (< [i] (json select "rx_sr" "/count")) (++ "[i]") {
  -> "[r]" (concat "/rules/[" [i] "]");
  if (equal (json select "rx_sr" (concat [r] "/then/[0]")) "STOP") {
    -> "[lhs]" "";
    for (-> "[k]" 0) (< [k] (json count "rx_sr" (concat [r] "/if"))) (++ "[k]") {-> "[lhs]" (concat [lhs] (json select "rx_sr" (concat [r] "/if/[" [k] "]")) " ");};
    -> "[res]" (concat [res] [lhs] "conf " (math round (json select "rx_sr" (concat [r] "/confidence")) 2) "; ");
  };
};
json unload "rx_sp"; json unload "rx_sr";
[res]
Result
mentdb
8 frequent sequences, 3 rules | before STOP: T_HIGH conf 1; T_HIGH VIB conf 1; VIB conf 0.8;
Go further
  • Give several alarms at the same time as one itemset: [["T_HIGH", "VIB"], "STOP"].
  • Count the alarms in real time with ml sketch countmin to watch the frequent codes.
executed on MentDB

Route the maintenance tickets

Turn the text of past maintenance tickets into TF-IDF features, train a logistic regression, then vectorize the new tickets with the same vocabulary to send them to the right team.

Pipeline
Script
admin
#1. Past tickets and their team;
-> "[t]" "[\"oil leak under the pump\", \"pump seal leaking\", \"leak of hydraulic oil on the floor\", \"motor too hot\", \"overheating of the motor after one hour\", \"high temperature alarm on the motor\", \"strange noise from the bearing\", \"vibration and noise on the bearing\", \"bearing worn, strong vibration\", \"oil drops near the pump\", \"motor temperature above the limit\", \"loud noise during rotation\"]";
-> "[y]" "[\"leak\", \"leak\", \"leak\", \"heat\", \"heat\", \"heat\", \"bearing\", \"bearing\", \"bearing\", \"leak\", \"heat\", \"bearing\"]";
json load "rx_v" (ml prep tfidf [t] "{\"stop_words\": \"english\"}");
ml train "rx_txt" "liblinear_logistic" (json select "rx_v" "/X") [y] "{\"C\": 10}";
#2. The new tickets get the columns of the training tickets;
-> "[voc]" (concat "{\"stop_words\": \"english\", \"vocabulary\": " (json select "rx_v" "/vocabulary") ", \"idf\": " (json select "rx_v" "/idf") "}");
-> "[new]" "[\"hydraulic leak near the pump\", \"the motor is overheating\", \"noise in the bearing\"]";
json load "rx_n" (ml prep tfidf [new] [voc]);
-> "[res]" (concat (json count "rx_v" "/vocabulary") " words | teams: " (ml model predict "rx_txt" (json select "rx_n" "/X")));
ml model delete "rx_txt"; json unload "rx_v"; json unload "rx_n";
[res]
Result
mentdb
29 words | teams: ["leak", "heat", "bearing"]
Go further
  • {"ngram": 2} adds the pairs of words, ex: "oil leak".
  • Save the vocabulary and the idf with the model (ml model save) to classify the tickets of tomorrow.
executed on MentDB

Event combinations that trip a pump

In the event log of a pumping station, find the partially ordered episode rules: a pressure drop and a temperature rise, in any order, followed by a pump trip within a few minutes.

Pipeline
Script
admin
#1. Event log: minute, event;
-> "[log]" "[[9, \"T_RISE\"], [10, \"P_DROP\"], [11, \"PUMP_TRIP\"], [21, \"T_RISE\"], [22, \"P_DROP\"], [24, \"PUMP_TRIP\"], [34, \"P_DROP\"], [36, \"DOOR\"], [43, \"P_DROP\"], [44, \"T_RISE\"], [45, \"PUMP_TRIP\"], [53, \"T_RISE\"], [54, \"P_DROP\"], [56, \"PUMP_TRIP\"], [66, \"T_RISE\"], [68, \"DOOR\"], [74, \"P_DROP\"], [75, \"T_RISE\"], [77, \"PUMP_TRIP\"], [85, \"P_DROP\"], [86, \"T_RISE\"], [88, \"PUMP_TRIP\"], [97, \"P_DROP\"], [99, \"DOOR\"], [109, \"P_DROP\"], [110, \"T_RISE\"], [111, \"PUMP_TRIP\"], [119, \"P_DROP\"], [120, \"T_RISE\"], [122, \"PUMP_TRIP\"], [130, \"P_DROP\"], [132, \"DOOR\"]]";
json load "rx_r" (ml assoc episode_rules [log] "{\"min_support\": 3, \"x_span\": 2, \"y_span\": 1, \"window\": 4, \"min_confidence\": 0.7}");
-> "[res]" "";
for (-> "[i]" 0) (< [i] (json select "rx_r" "/count")) (++ "[i]") {
  -> "[p]" (concat "/rules/[" [i] "]");
  -> "[a]" ""; for (-> "[k]" 0) (< [k] (json count "rx_r" (concat [p] "/if"))) (++ "[k]") {-> "[a]" (concat [a] (if (> [k] 0) {" + "} {""}) (json select "rx_r" (concat [p] "/if/[" [k] "]")));};
  -> "[b]" ""; for (-> "[k]" 0) (< [k] (json count "rx_r" (concat [p] "/then"))) (++ "[k]") {-> "[b]" (concat [b] (if (> [k] 0) {" + "} {""}) (json select "rx_r" (concat [p] "/then/[" [k] "]")));};
  -> "[res]" (concat [res] [a] " => " [b] " conf " (math round (json select "rx_r" (concat [p] "/confidence")) 2) " | ");
};
json load "rx_t" (ml assoc topk_episodes [log] 3 4);
-> "[res]" (concat [res] "most frequent: " (json select "rx_t" "/episodes/[0]/episode/[0]/[0]") " x" (json select "rx_t" "/episodes/[0]/support"));
json unload "rx_r"; json unload "rx_t";
[res]
Result
mentdb
T_RISE + P_DROP => PUMP_TRIP conf 1 | T_RISE => PUMP_TRIP conf 0.89 | P_DROP => PUMP_TRIP conf 0.73 | most frequent: P_DROP x11
Go further
  • The antecedent is a set: P_DROP then T_RISE and T_RISE then P_DROP count for the same rule.
  • Watch the antecedents in real time and raise a warning before the trip.
executed on MentDB

Detect a change in a sensor stream

The vibration of a motor arrives in batches: an ADWIN detector keeps a window of the recent values, cuts it when its mean changes, and gives the position of the change, without storing the whole history.

Pipeline
Script
admin
#1. Two batches of vibration RMS in mm/s;
-> "[b1]" "[1.897, 2.236, 2.105, 2.064, 1.754, 2.203, 1.884, 2.164, 2.074, 2.054, 1.94, 1.819, 1.97, 1.841, 1.9, 2.007, 2.008, 2.048, 2.081, 2.143, 1.794, 1.866, 1.828, 1.784, 1.998, 1.98, 2.134, 1.886, 1.941, 2.025, 2.032, 2.046, 2.221, 2.144, 1.818, 2.186, 2.208, 1.98, 2.024, 2.085, 2.17, 1.962, 1.807, 2.32, 2.149, 2.269, 2.16, 2.01, 1.792, 2.346, 1.814, 2.055, 2.074, 1.803, 2.135, 1.759, 2.075, 2.034, 1.875, 2.083, 2.034, 1.924, 2.273, 1.977, 1.932, 1.906, 2.046, 1.77, 2.083, 1.903, 2.024, 1.902, 1.922, 2.024, 1.972, 2.362, 2.087, 1.999, 1.991, 1.988]";
-> "[b2]" "[2.497, 2.724, 2.605, 2.645, 2.446, 2.318, 2.644, 2.738, 2.592, 2.362, 2.561, 2.475, 2.853, 2.514, 2.673, 2.402, 2.77, 2.92, 2.453, 2.503, 2.488, 2.367, 2.651, 2.68, 2.551, 2.665, 2.575, 2.519, 2.448, 2.458, 2.598, 2.335, 2.506, 2.584, 2.77, 2.475, 2.298, 2.597, 2.679, 2.416]";
ml sketch create "rx_vib" "adwin" "{\"delta\": 0.002}";
ml sketch add "rx_vib" [b1];
-> "[d1]" (ml sketch query "rx_vib" "drift");
ml sketch add "rx_vib" [b2];
json load "rx_r" (ml sketch result "rx_vib");
-> "[res]" (concat "after batch 1: drift " [d1] " | after batch 2: " (json select "rx_r" "/drifts") " change at value " (json select "rx_r" "/drift_positions/[0]") " (true change at 81: ADWIN waits for enough evidence), current mean " (math round (json select "rx_r" "/mean") 3) " mm/s on " (json select "rx_r" "/width") " values");
ml sketch delete "rx_vib"; json unload "rx_r";
[res]
Result
mentdb
after batch 1: drift 0 | after batch 2: 1 change at value 105 (true change at 81: ADWIN waits for enough evidence), current mean 2.559 mm/s on 40 values
Go further
  • delta is the false alarm risk: smaller = fewer alarms but later detection.
  • A Hoeffding tree sketch (type hoeffding) learns a classifier on the same stream, row by row.

Also in Engineering & data

TR
Telecom & radar engineer
Link budgets in dB, radar equation, detection thresholds and spectra of modulated signals.
11 recipes
CE
Control engineer
Transfer functions, margins, PID and state feedback: complex numbers, CAS, ODE and matrices.
10 recipes
RE
Robotics engineer
Kinematics, Jacobians and trajectories: matrices, vectors, non-linear systems and splines.
10 recipes
EE
Electrical engineer
Impedances, three-phase power and cable sizing with complex numbers and units.
10 recipes
ES
Electronics & signal engineer
Filter design, spectra, peak detection and SNR on sampled signals.
12 recipes
MC
Mechanical & civil engineer
Deflection, stresses, safety factors and concrete strength statistics.
12 recipes
IQ
Industrial quality engineer
SPC control limits, process capability and tests on machines.
20 recipes
DS
Data scientist
Regression, correlation, feature scaling and drift tests.
22 recipes
SB
Statistician & biostatistician
Confidence intervals, sample size, ANOVA, chi-square and non-parametric tests.
27 recipes
DQ
Data quality analyst
Outliers, format checks, normalization and duplicate detection.
12 recipes