ML Clustering and anomalies
ml dbscanml hdbscanml opticsml hclustml elbowml anomaly zscoreml anomaly madml anomaly iqrml anomaly mahalanobisml anomaly lofml distanceml distance_matrixml umapml umapml tsneml tsneml mdsml ca
ml dbscan <json_X> <eps> <min_points> <metric>
Description
DBSCAN density clustering in N dimensions; the label -1 is the noise.
Metrics: euclidean, sqeuclidean, manhattan, chebyshev, canberra, cosine.
DBSCAN density clustering in N dimensions; the label -1 is the noise.
Metrics: euclidean, sqeuclidean, manhattan, chebyshev, canberra, cosine.
Parameters
json_X: The rows - String - required
eps: The radius of the neighborhood - Number - required
min_points: The minimum number of points in the neighborhood (the point itself included) - Number - required
metric: The distance - String - required
json_X: The rows - String - required
eps: The radius of the neighborhood - Number - required
min_points: The minimum number of points in the neighborhood (the point itself included) - Number - required
metric: The distance - String - required
admin
ml dbscan "[[1, 1], [1.2, 1.1], [0.9, 1], [5, 5], [5.1, 5.2], [4.9, 5], [9, 0]]" 0.5 2 "euclidean"
mentdb
{"labels": [0, 0, 0, 1, 1, 1, -1], "n_clusters": 2, "n_noise": 1, "sizes": [3, 3], "core_points": [0, 1, 2, 3, 4, 5]}
Used in use cases (1 recipe)
Machine learning engineer › Clusters of different densities with HDBSCAN
Machine learning engineer › Clusters of different densities with HDBSCAN
ml hdbscan <json_X> <min_cluster_size> <min_samples> <metric>
Description
HDBSCAN* density clustering (Campello et al. 2013): no radius to choose, finds clusters of different densities; returns the labels (-1 = noise) and the membership strength of each point
HDBSCAN* density clustering (Campello et al. 2013): no radius to choose, finds clusters of different densities; returns the labels (-1 = noise) and the membership strength of each point
Parameters
json_X: The rows - String - required
min_cluster_size: The smallest group considered as a cluster - Number - required
min_samples: The number of neighbors of a core point (often = min_cluster_size) - Number - required
metric: The distance (as ml dbscan) - String - required
json_X: The rows - String - required
min_cluster_size: The smallest group considered as a cluster - Number - required
min_samples: The number of neighbors of a core point (often = min_cluster_size) - Number - required
metric: The distance (as ml dbscan) - String - required
admin
ml hdbscan "[[0, 0], [0.2, 0.1], [-0.1, 0.2], [0.1, -0.2], [0.2, 0.2], [5, 5], [6.5, 4.8], [5.8, 6.4], [4.6, 6.1], [6.1, 5.6], [5.2, 4.1], [12, -3]]" 4 4 "euclidean"
mentdb
{"labels": [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, -1], "probabilities": [1, 1, 1, 0.7669649888473705, 1, 1, 1, 0.98058067569092, 1, 1, 0.9038769075777342, 0], "n_clusters": 2, "n_noise": 1, "sizes": [5, 6]}
Used in use cases (1 recipe)
Machine learning engineer › Clusters of different densities with HDBSCAN
Machine learning engineer › Clusters of different densities with HDBSCAN
ml optics <json_X> <min_points> <eps_cluster> <metric>
Description
OPTICS (Ankerst et al. 1999): ordering of the points and reachability distance (the reachability plot shows the density structure); the clusters are extracted at the radius eps_cluster (-1 = noise)
OPTICS (Ankerst et al. 1999): ordering of the points and reachability distance (the reachability plot shows the density structure); the clusters are extracted at the radius eps_cluster (-1 = noise)
Parameters
json_X: The rows - String - required
min_points: The number of neighbors of a core point (the point itself counts) - Number - required
eps_cluster: The radius used to extract the clusters - Number - required
metric: The distance (as ml dbscan) - String - required
json_X: The rows - String - required
min_points: The number of neighbors of a core point (the point itself counts) - Number - required
eps_cluster: The radius used to extract the clusters - Number - required
metric: The distance (as ml dbscan) - String - required
admin
ml optics "[[1, 1], [1.2, 1.1], [0.9, 1], [1.1, 0.9], [5, 5], [5.1, 5.2], [4.9, 5], [5.2, 4.8], [9, 0]]" 3 0.6 "euclidean"
mentdb
{"ordering": [0, 2, 3, 1, 6, 4, 5, 7, 8], "reachability": [null, 0.14142135623730953, 0.14142135623730953, 0.22360679774997896, 5.375872022286245, 0.2828427124746186, 0.22360679774997896, 0.2828427124746193, 6.1220911460055865], "core_distance": [0.14142135623730953, 0.22360679774997902, 0.22360679774997896, 0.22360679774997896, 0.2828427124746186, 0.22360679774997896, 0.2828427124746186, 0.3605551275463989, 6.4031242374328485], "labels": [0, 0, 0, 0, 1, 1, 1, 1, -1], "n_clusters": 2, "n_noise": 1}
ml hclust <json_X> <linkage> <k>
Description
Agglomerative hierarchical clustering: labels for k clusters and the merges of the dendrogram [cluster a, cluster b, height, size] (the same heights as scipy)
Agglomerative hierarchical clustering: labels for k clusters and the merges of the dendrogram [cluster a, cluster b, height, size] (the same heights as scipy)
Parameters
json_X: The rows - String - required
linkage: single, complete, average or ward - String - required
k: The number of clusters - Number - required
json_X: The rows - String - required
linkage: single, complete, average or ward - String - required
k: The number of clusters - Number - required
admin
ml hclust "[[1, 1], [1.2, 1.1], [5, 5], [5.1, 5.2], [9, 0]]" "ward" 2
mentdb
{"labels": [0, 0, 1, 1, 1], "merges": [[0, 1, 0.22360679774997896, 2], [2, 3, 0.22360679774997896, 2], [6, 4, 7.448713535459215, 3], [5, 7, 8.934465102437116, 5]]}
ml elbow <json_X> <k_max> <seed>
Description
Inertia and silhouette of a k-means for k = 1 to k_max (to choose k)
Inertia and silhouette of a k-means for k = 1 to k_max (to choose k)
Parameters
json_X: The rows - String - required
k_max: The maximum k - Number - required
seed: The seed - Number - required
json_X: The rows - String - required
k_max: The maximum k - Number - required
seed: The seed - Number - required
admin
ml elbow "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" 4 42
mentdb
[{"k": 1, "inertia": 236.85166666666666}, {"k": 2, "inertia": 116.875, "silhouette": 0.5760306294804248}, {"k": 3, "inertia": 2.265, "silhouette": 0.914346911876748}, {"k": 4, "inertia": 1.2399999999999995, "silhouette": 0.7774923004215486}]
ml anomaly zscore <json_values> <threshold>
Description
Anomalies of one variable: |z-score| greater than the threshold
Anomalies of one variable: |z-score| greater than the threshold
Parameters
json_values: The values - String - required
threshold: The threshold, ex: 3 - Number - required
json_values: The values - String - required
threshold: The threshold, ex: 3 - Number - required
admin
ml anomaly zscore "[10, 11, 9, 10, 12, 10, 11, 35]" 2
mentdb
{"scores": [-0.4007249984023573, -0.2862321417159695, -0.5152178550887451, -0.4007249984023573, -0.1717392850295817, -0.4007249984023573, -0.2862321417159695, 2.461596418757338], "anomalies": [7], "values": [35]}
ml anomaly mad <json_values> <threshold>
Description
Anomalies of one variable with the modified z-score 0.6745 * (x - median) / MAD (robust)
Anomalies of one variable with the modified z-score 0.6745 * (x - median) / MAD (robust)
Parameters
json_values: The values - String - required
threshold: The threshold, ex: 3.5 - Number - required
json_values: The values - String - required
threshold: The threshold, ex: 3.5 - Number - required
admin
ml anomaly mad "[10, 11, 9, 10, 12, 10, 11, 35]" 3.5
mentdb
{"median": 10.5, "mad": 0.5, "scores": [-0.6745, 0.6745, -2.0235, -0.6745, 2.0235, -0.6745, 0.6745, 33.0505], "anomalies": [7], "values": [35]}
ml anomaly iqr <json_values> <k>
Description
Anomalies of one variable outside [Q1 - k * IQR, Q3 + k * IQR] (Tukey fences)
Anomalies of one variable outside [Q1 - k * IQR, Q3 + k * IQR] (Tukey fences)
Parameters
json_values: The values - String - required
k: The factor, ex: 1.5 - Number - required
json_values: The values - String - required
k: The factor, ex: 1.5 - Number - required
admin
ml anomaly iqr "[10, 11, 9, 10, 12, 10, 11, 35]" 1.5
mentdb
{"q1": 10, "q3": 11.25, "lower": 8.125, "upper": 13.125, "anomalies": [7], "values": [35]}
ml anomaly mahalanobis <json_X> <alpha>
Description
Multivariate anomalies: squared Mahalanobis distance of each row and chi2 p-value; anomaly if p-value < alpha
Multivariate anomalies: squared Mahalanobis distance of each row and chi2 p-value; anomaly if p-value < alpha
Parameters
json_X: The rows - String - required
alpha: The significance level, ex: 0.01 - Number - required
json_X: The rows - String - required
alpha: The significance level, ex: 0.01 - Number - required
admin
ml anomaly mahalanobis "[[1, 2], [2, 4.1], [3, 6], [4, 8.2], [5, 9.9], [6, 12.1], [3, 1]]" 0.1
mentdb
{"distances": [2.2613007052208705, 0.9415042049692907, 0.20796588104688948, 0.28806874905158547, 0.875400330647159, 2.295783849403799, 5.1299762796604025], "p_values": [0.3228232392007079, 0.6245323792483213, 0.9012406719841778, 0.8658579840900398, 0.6455193029151456, 0.31730496758966753, 0.07692009372179709], "anomalies": [6]}
ml anomaly lof <json_X> <k> <threshold>
Description
Local outlier factor (k neighbors); anomaly if the score is greater than the threshold (about 1 = normal)
Local outlier factor (k neighbors); anomaly if the score is greater than the threshold (about 1 = normal)
Parameters
json_X: The rows - String - required
k: The number of neighbors - Number - required
threshold: The threshold, ex: 1.5 - Number - required
json_X: The rows - String - required
k: The number of neighbors - Number - required
threshold: The threshold, ex: 1.5 - Number - required
admin
ml anomaly lof "[[0, 0], [0, 1], [1, 0], [1, 1], [10, 10]]" 2 1.5
mentdb
{"scores": [1, 1, 1, 1, 13.211723385168426], "anomalies": [4]}
ml distance <metric> <json_a> <json_b>
Description
Distance or similarity between two vectors.
Metrics: euclidean, sqeuclidean, manhattan, chebyshev, minkowski (p = 3), canberra, cosine (distance), cosine_similarity, pearson, hamming, jaccard (sets of labels), dtw (dynamic time warping, sizes may differ).
Distance or similarity between two vectors.
Metrics: euclidean, sqeuclidean, manhattan, chebyshev, minkowski (p = 3), canberra, cosine (distance), cosine_similarity, pearson, hamming, jaccard (sets of labels), dtw (dynamic time warping, sizes may differ).
Parameters
metric: The metric - String - required
json_a: The first vector - String - required
json_b: The second vector - String - required
metric: The metric - String - required
json_a: The first vector - String - required
json_b: The second vector - String - required
admin
ml distance "euclidean" "[0, 0]" "[3, 4]"
mentdb
5
admin
ml distance "cosine_similarity" "[1, 2, 3]" "[2, 4, 6.5]"
mentdb
0.9992825883286348
Dynamic time warping
admin
ml distance "dtw" "[1, 2, 3, 4, 3]" "[1, 1, 2, 3, 4, 3]"
mentdb
0
Jaccard similarity of two sets
admin
ml distance "jaccard" "[\"a\", \"b\", \"c\"]" "[\"b\", \"c\", \"d\"]"
mentdb
0.5
ml distance_matrix <json_X> <metric>
Description
Distance matrix of the rows (same metrics as ml distance, except jaccard, dtw, pearson and hamming)
Distance matrix of the rows (same metrics as ml distance, except jaccard, dtw, pearson and hamming)
Parameters
json_X: The rows - String - required
metric: The metric - String - required
json_X: The rows - String - required
metric: The metric - String - required
admin
ml distance_matrix "[[0, 0], [3, 4], [6, 8]]" "euclidean"
mentdb
[[0, 5, 10], [5, 0, 5], [10, 5, 0]]
ml umap <json_X>
Description
UMAP (McInnes 2018): coordinates in 2 dimensions that keep the local neighborhoods and much of the global structure, faster than t-SNE (up to 5000 rows); share of the nearest neighbors preserved
UMAP (McInnes 2018): coordinates in 2 dimensions that keep the local neighborhoods and much of the global structure, faster than t-SNE (up to 5000 rows); share of the nearest neighbors preserved
Parameters
json_X: The rows - String - required
json_X: The rows - String - required
admin
ml umap "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]"
mentdb
{"n": 12, "n_neighbors": 11, "min_dist": 0.1, "a": 1.5800000000000012, "b": 0.8950000000000005, "edges": 66, "neighbor_preservation": 1, "embedding": [[12.115993206293298, -2.7056732489822757], [11.973139264301736, -3.8854274149731562], [10.78051945274033, -3.5622233788340836], [11.415060666416615, -3.3708369232975217], [10.559549627855567, -1.7047103702875137], [9.038073969755388, -1.5597974486794444], [9.54613676132389, -2.1814018007854266], [9.74341869448137, -1.650260485667652], [9.863558409602987, 0.2575773541194988], [8.948395955733803, 0.501712236545915], [9.055409370337495, -0.22295528697542205], [9.249911586294047, -0.00712606873091402]]}
ml umap <json_X> <json_params>
Description
UMAP with parameters: n_neighbors (15), min_dist (0.1), n_components (2), epochs, scale (standardize the columns), seed
UMAP with parameters: n_neighbors (15), min_dist (0.1), n_components (2), epochs, scale (standardize the columns), seed
Parameters
json_X: The rows - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_params: The parameters - String - required
admin
ml umap "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" "{\"n_neighbors\": 4, \"min_dist\": 0.3, \"seed\": 3}"
mentdb
{"n": 12, "n_neighbors": 4, "min_dist": 0.3, "a": 0.9900000000000008, "b": 1.114999999999998, "edges": 29, "neighbor_preservation": 0.75, "embedding": [[-22.879125136015578, -9.199478592070305], [-24.354138159299364, -10.678306570292596], [-23.83703097177956, -10.159357382873457], [-23.458196607182867, -9.77979717123262], [-8.778836334394242, 1.3161210431659651], [-7.329357054820781, -0.13754087755321412], [-8.25668474310402, 0.7902973278887538], [-7.91287564965638, 0.44649689373234186], [16.131833028609904, -0.4909612252383978], [17.55990860958745, -1.6017213251757076], [16.27346757951913, -1.4008143483579394], [16.76401796213982, -1.327443680945905]]}
ml tsne <json_X>
Description
Exact t-SNE (van der Maaten): coordinates in 2 dimensions that keep the neighborhoods, to see the clusters of data with many columns (up to 2000 rows); KL divergence
Exact t-SNE (van der Maaten): coordinates in 2 dimensions that keep the neighborhoods, to see the clusters of data with many columns (up to 2000 rows); KL divergence
Parameters
json_X: The rows - String - required
json_X: The rows - String - required
admin
ml tsne "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]"
mentdb
{"embedding": [[-14.741173957957502, -46.018375510532614], [-8.603367979878762, -37.19427255480157], [-7.312341976961227, -44.65548849052038], [-16.094223273382692, -38.599250846045905], [-20.78229321906685, 18.904238081038446], [-27.29322755216707, 27.694397652069974], [-28.16668689664174, 20.077663254534205], [-19.921005802732864, 26.555191523411214], [37.170679825485344, 10.76511294935944], [43.646891600811394, 14.670150183335748], [47.5111031527042, 8.062146848300326], [40.97384331612568, 4.287901146798551]], "kl_divergence": 0.048039908326738906, "perplexity": 3.6666666666666665}
ml tsne <json_X> <json_params>
Description
t-SNE with parameters: n_components (2), perplexity (30, at most (n - 1) / 3), max_iter (1000), learning_rate, seed
t-SNE with parameters: n_components (2), perplexity (30, at most (n - 1) / 3), max_iter (1000), learning_rate, seed
Parameters
json_X: The rows - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_params: The parameters - String - required
admin
ml tsne "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" "{\"perplexity\": 3, \"max_iter\": 500, \"seed\": 7}"
mentdb
{"embedding": [[143.81846843526804, -79.55212863167866], [140.05274585545834, -94.09838532459521], [158.42561910657668, -81.97394519322165], [155.08320624003608, -96.54112838972748], [-51.93547987830321, -69.79335701050267], [-36.96036648614424, -68.58991555627914], [-36.97324683975851, -53.524415948344114], [-51.656830198024345, -54.623992555658155], [-15.497259290097643, 113.06060087185064], [-28.89635956474452, 117.77350415745826], [-13.442595650831546, 99.11862277738345], [-30.33748454814782, 100.26803992604943]], "kl_divergence": 0.08133576577453568, "perplexity": 3}
ml mds <json> <dims> <input>
Description
Classical multidimensional scaling (Torgerson): coordinates in a few dimensions that keep the distances, eigenvalues, explained ratio and Kruskal stress-1
Classical multidimensional scaling (Torgerson): coordinates in a few dimensions that keep the distances, eigenvalues, explained ratio and Kruskal stress-1
Parameters
json: The rows (data) or a square distance matrix (distances) - String - required
dims: The number of dimensions - Number - required
input: data or distances - String - required
json: The rows (data) or a square distance matrix (distances) - String - required
dims: The number of dimensions - Number - required
input: data or distances - String - required
admin
ml mds "[[0, 300, 400], [300, 0, 500], [400, 500, 0]]" 2 "distances"
mentdb
{"coordinates": [[-65.81288103026219, 153.12231211771308], [-215.23109896708587, -107.02033365299452], [281.043979997348, -46.10197846471837]], "eigenvalues": [129641.4799648327, 37025.186701833976], "explained_ratio": [0.7778488797889961, 0.22215112021100383], "stress": 1.392373714442771E-16}
admin
ml mds "[[0, 0], [3, 4], [6, 8], [0, 5]]" 1 "data"
mentdb
{"coordinates": [[-4.70563099497423], [0.28787831379579415], [5.281387622565816], [-0.8636349413873835]], "eigenvalues": [50.86475751619266], "explained_ratio": [0.884604478542481], "stress": 0.15744664604346512}
ml ca <json_table> <dims>
Description
Correspondence analysis (AFC) of a contingency table: coordinates of the rows and of the columns, inertia and explained ratio of each dimension, chi2 test of independence
Correspondence analysis (AFC) of a contingency table: coordinates of the rows and of the columns, inertia and explained ratio of each dimension, chi2 test of independence
Parameters
json_table: The table of counts (rows x columns) - String - required
dims: The number of dimensions - Number - required
json_table: The table of counts (rows x columns) - String - required
dims: The number of dimensions - Number - required
admin
ml ca "[[68, 15, 5], [119, 54, 29], [26, 14, 14], [7, 10, 16]]" 2
mentdb
{"row_coordinates": [[-0.3891644281915679, 0.09259534052841673], [-0.047273588023180164, -0.05990015263127676], [0.2547227943258173, 0.02819100868839848], [0.9103243505438257, 0.07360989078314283]], "column_coordinates": [[-0.2522474606001559, 0.02819126471370606], [0.13373088654964677, -0.1144413443539129], [0.6727729512955801, 0.06939010606091466]], "inertia": [0.11838047658016371, 0.004511957470690889], "explained_ratio": [0.9632853112110727, 0.03671468878892722], "total_inertia": 0.1228924340508546, "chi2": 46.330447637172185, "p_value": 2.5442855045021417E-8}
Used in use cases (1 recipe)
Statistician & biostatistician › Correspondence analysis of a survey table
Statistician & biostatistician › Correspondence analysis of a survey table