Model Evaluation Functions#
Regression#
All regression evaluation metrics will follow the below request body structure.
Query Request:
{
"select": [
{
"function": "[rmse|mae|rSquared]",
"alias": "<alias_name> [optional string]",
"parameters": {
"ground_truth_property": "<attribute_name> [string]",
"predicted_property": "<attribute_name> [string]"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": "<evaluation_value> [float]"
}
]
}
RMSE#
Get the RMSE between a prediction attribute and a ground truth attribute.
Sample Request:
{
"select": [
{
"function": "rmse",
"alias": "error",
"parameters": {
"ground_truth_property": "FICO_actual",
"predicted_property": "FICO_predicted"
}
}
]
}
Sample Response:
{
"query_result": [
{
"error": 0.76
}
]
}
MAE#
Get the Mean Absolute Error between a prediction attribute and a ground truth attribute.
This function takes an optional parameter aggregation that allows swapping the aggregation from "avg"
to either "min" or "max". This can be helpful if you’re looking for extremes, as in the lowest or highest absolute error, respectively.
Additionally, this functions supports optional params normalizationMax and normalizationMin
that accept numbers and will perform min/max normalization on the values before aggregation if both params are provided.
Query Request:
{
"select": [
{
"function": "mae",
"alias": "<alias_name> [optional string]",
"parameters": {
"predicted_property": "<predicted_property_name> [string]",
"ground_truth_property": "<ground_truth_property_name> [string]",
"aggregation": "[avg|min|max] (default avg, optional)",
"normalizationMin": "<value> [optional number]",
"normalizationMax": "<value> [optional number]"
}
}
]
}
Sample Request:
{
"select": [
{
"function": "mae",
"alias": "error",
"parameters": {
"ground_truth_property": "FICO_actual",
"predicted_property": "FICO_predicted"
}
}
]
}
Sample Response:
{
"query_result": [
{
"error": 0.76
}
]
}
R Squared#
Get the R Squared value between a prediction attribute and a ground truth attribute.
Sample Request:
{
"select": [
{
"function": "rSquared",
"alias": "rsq",
"parameters": {
"ground_truth_property": "FICO_actual",
"predicted_property": "FICO_predicted"
}
}
]
}
Sample Response:
{
"query_result": [
{
"rsq": 0.94
}
]
}
Binary Classification#
When using any binary classification evaluation functions with a multiclass model, outputs will be calculated assuming a one vs. all approach.
Confusion Matrix#
Calculates the confusion matrix for a classification model. For binary classifiers, users must specify a probability threshold to count a prediction as a positive class.
Query Request:
{
"select": [
{
"function": "confusionMatrix",
"alias": "<alias_name> [optional string]",
"parameters": {
"threshold": "<value [float]> [required only for binary classifiers]"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": {
"true_positive": "<count> [int]",
"false_positive": "<count> [int]",
"true_negative": "<count> [int]",
"false_negative": "<count> [int]"
}
}
]
}
Sample Request: Calculate the confusion matrix for a binary classifier with a threshold of 0.5 (standard threshold for confusion matrix).
{
"select": [
{
"function": "confusionMatrix",
"parameters": {
"threshold": 0.5
}
}
]
}
Sample Response:
{
"query_result": [
{
"confusionMatrix": {
"true_positive": 100480,
"false_positive": 100076,
"true_negative": 100302,
"false_negative": 99142
}
}
]
}
Confusion Matrix Rate#
Calculates the confusion matrix rates for a classification model. For binary classifiers, users must specify a probability threshold to count a prediction as a positive class.
Query Request:
{
"select": [
{
"function": "confusionMatrixRate",
"alias": "<alias_name> [optional string]",
"parameters": {
"threshold": "<value [float]> [required only for binary classifiers]"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": {
"true_positive_rate": "<rate> [float]",
"false_positive_rate": "<rate> [float]",
"true_negative_rate": "<rate> [float]",
"false_negative_rate": "<rate> [float]",
"accuracy_rate": "<rate> [float]"
}
}
]
}
Sample Request: Calculate the confusion matrix for a binary classifier with a threshold of 0.5 (standard threshold for confusion matrix).
{
"select": [
{
"function": "confusionMatrixRate",
"parameters": {
"threshold": 0.5
}
}
]
}
Response:
{
"query_result": [
{
"confusionMatrixRate": {
"true_positive_rate": 0.5033513340213003,
"false_positive_rate": 0.49943606583557076,
"true_negative_rate": 0.5005639341644292,
"false_negative_rate": 0.4966486659786997
}
}
]
}
Confusion Matrix Variants#
If you only want a specific metric derived from a confusion matrix, you can use one of the following functions:
truePositiveRatefalsePositiveRatetrueNegativeRatefalseNegativeRateaccuracyRatebalancedAccuracyRatef1sensitivityspecificityprecisionrecall
For example, to return the truePositiveRate:
{
"select": [
{
"function": "truePositiveRate",
"parameters": {
"threshold": 0.5,
"ground_truth_property":"class_a",
"predicted_property":"ground_truth_a"
}
}
]
}
Response:
{
"query_result": [
{
"truePositiveRate": 0.5033513340213003
}
]
}
AUC#
The Area Under the ROC Curve can also be computed for binary classifiers.
Sample Query:
{
"select": [
{
"function": "auc",
"parameters": {
"ground_truth_property":"class_a",
"predicted_property":"ground_truth_a"
}
}
]
}
Response:
{
"query_result": [
{
"auc": 0.9192331426352897
}
]
}
Multiclass Classification#
Multiclass Accuracy Rate#
Calculates the global accuracy rate.
Query Request:
{
"select": [
{
"function": "accuracyRateMulticlass",
"alias": "<alias_name> [optional string]"
}
]
}
Query Response:
{
"query_result": [
{
"accuracyRateMulticlass": "<rate> [float]"
}
]
}
Example:
{
"select": [
{
"function": "accuracyRateMulticlass"
}
]
}
Response:
{
"query_result": [
{
"accuracyRateMulticlass": 0.785
}
]
}
Multiclass Confusion Matrix#
Calculates the confusion matrix for a multiclass model with regard to a single class. The predicted attribute and ground truth attribute must be passed as parameters.
Query Request:
{
"select": [
{
"function": "confusionMatrixMulticlass",
"alias": "<alias_name> [optional string]",
"parameters": {
"predicted_property": "<predicted_property_name>",
"ground_truth_property": "<ground_truth_property_name>"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": {
"true_positive": "<count> [int]",
"false_positive": "<count> [int]",
"true_negative": "<count> [int]",
"false_negative": "<count> [int]"
}
}
]
}
Example:
{
"select": [
{
"function": "confusionMatrixMulticlass",
"parameters": {
"predicted_property": "predicted_class_A",
"ground_truth_property": "gt_predicted_class_A"
}
}
]
}
Response:
{
"query_result": [
{
"confusionMatrix": {
"true_positive": 100480,
"false_positive": 100076,
"true_negative": 100302,
"false_negative": 99142
}
}
]
}
Multiclass Confusion Matrix Rate#
Calculates the confusion matrix rates for a multiclass classification model in regards to a single predicted class.
Query Request:
{
"select": [
{
"function": "confusionMatrixRateMulticlass",
"alias": "<alias_name> [optional string]",
"parameters": {
"predicted_property": "predicted_class_A",
"ground_truth_property": "gt_predicted_class_A"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": {
"true_positive_rate": "<rate> [float]",
"false_positive_rate": "<rate> [float]",
"true_negative_rate": "<rate> [float]",
"false_negative_rate": "<rate> [float]",
"accuracy_rate": "<rate> [float]",
"balanced_accuracy_rate": "<rate> [float]",
"precision": "<rate> [float]",
"f1": "<rate> [float]"
}
}
]
}
Example calculating the confusion matrix rates:
{
"select": [
{
"function": "confusionMatrixRateMulticlass",
"parameters": {
"predicted_property": "predicted_class_A",
"ground_truth_property": "gt_predicted_class_A"
}
}
]
}
Response:
{
"query_result": [
{
"confusionMatrixRateMulticlass": {
"true_positive_rate": 0.6831683168316832,
"false_positive_rate": 0.015653220951234198,
"true_negative_rate": 0.9843467790487658,
"false_negative_rate": 0.31683168316831684,
"accuracy_rate": 0.9378818737270875,
"balanced_accuracy_rate": 0.8337575479402245,
"precision": 0.8884120171673819,
"f1": 0.7723880597014925
}
}
]
}
If you only want a specific value from the confusion matrix rate function, you can use one of the following functions:
truePositiveRateMulticlassfalsePositiveRateMulticlasstrueNegativeRateMulticlassfalseNegativeRateMulticlass
For example, to return the truePositiveRate:
{
"select": [
{
"function": "truePositiveRateMulticlass",
"parameters": {
"predicted_property": "predicted_class_A",
"ground_truth_property": "gt_predicted_class_A"
}
}
]
}
Response:
{
"query_result": [
{
"truePositiveRate": 0.5033513340213003
}
]
}
Multiclass F1#
Calculates the components needed to compute a F1 score for a multiclass model.
In this example, the model has 3 classes: class-1, class-2, class-3 and
the corresponding ground truth labels class-1-gt, class-2-gt, class-3-gt.
Query Request:
{
"select": [
{
"function": "count",
"alias": "count"
},
{
"function": "confusionMatrixRateMulticlass",
"alias": "class-1",
"parameters": {
"predicted_property": "class-1",
"ground_truth_property": "class-1-gt"
}
},
{
"function": "countIf",
"alias": "class-1-gt",
"parameters": {
"property": "multiclass_model_ground_truth_class",
"comparator": "eq",
"value": "class-1-gt"
},
"stage": "GROUND_TRUTH"
},
{
"function": "confusionMatrixRateMulticlass",
"alias": "class-2",
"parameters": {
"predicted_property": "class-2",
"ground_truth_property": "class-2-gt"
}
},
{
"function": "countIf",
"alias": "class-2-gt",
"parameters": {
"property": "multiclass_model_ground_truth_class",
"comparator": "eq",
"value": "class-2-gt"
},
"stage": "GROUND_TRUTH"
},
{
"function": "confusionMatrixRateMulticlass",
"alias": "class-3",
"parameters": {
"predicted_property": "class-3",
"ground_truth_property": "class-3-gt"
}
},
{
"function": "countIf",
"alias": "class-3-gt",
"parameters": {
"property": "multiclass_model_ground_truth_class",
"comparator": "eq",
"value": "class-3-gt"
},
"stage": "GROUND_TRUTH"
}
]
}
Query Response:
{
"query_result": [
{
"count": 7044794,
"class-1-gt": 2540963,
"class-2-gt": 2263918,
"class-3-gt": 2239913,
"class-1": {
"true_positive_rate": 0.4318807475748368,
"false_positive_rate": 0.3060401245073361,
"true_negative_rate": 0.6939598754926639,
"false_negative_rate": 0.5681192524251633,
"accuracy_rate": 0.5994314383074935,
"balanced_accuracy_rate": 0.5629203115337503,
"precision": 0.4432575070302042,
"f1": 0.437495178612114
},
"class-2": {
"true_positive_rate": 0.42177322676881407,
"false_positive_rate": 0.3514795196528837,
"true_negative_rate": 0.6485204803471163,
"false_negative_rate": 0.578226773231186,
"accuracy_rate": 0.5756528863725469,
"balanced_accuracy_rate": 0.5351468535579652,
"precision": 0.3623427088234848,
"f1": 0.38980575845890253
},
"class-3": {
"true_positive_rate": 0.26144274353512836,
"false_positive_rate": 0.2805894672521546,
"true_negative_rate": 0.7194105327478454,
"false_negative_rate": 0.7385572564648716,
"accuracy_rate": 0.5737983254017079,
"balanced_accuracy_rate": 0.4904266381414869,
"precision": 0.3028268576818381,
"f1": 0.2806172238153916
}
}
]
}
With this result, you can calculate the weighted F1 score by multiplying each classes’s F1 score by the count of the ground truth and dividing by the total count. In this example, that would be
(class-1.f1 * class-1-gt + class-2.f1 * class-2-gt + class-3.f1 * class-3-gt) / count
and with numbers:
(0.437495178612114 * 2540963 +
0.38980575845890253 * 2263918 +
0.2806172238153916 * 2239913) / 7044794
= 0.3722898785
Object Detection#
Objects Detected#
For multiclass, multilabel, and regression models, querying model performance works the same for Arthur computer vision models as more tabular and NLP models. But Object Detection computer vision have some special fields you can use when querying.
Example query fetching all bounding box fields:
{
"select": [
{
"property": "inference_id"
},
{
"property": "objects_detected"
}
]
}
The reponse will have 1 object per bounding box.
{
"query_result": [
{
"inference_id": "1",
"objects_detected.class_id": 0,
"objects_detected.confidence": 0.6,
"objects_detected.top_left_x": 23,
"objects_detected.top_left_y": 45,
"objects_detected.width": 20,
"objects_detected.height": 30
},
{
"inference_id": "1",
"objects_detected.class_id": 1,
"objects_detected.confidence": 0.6,
"objects_detected.top_left_x": 23,
"objects_detected.top_left_y": 45,
"objects_detected.width": 20,
"objects_detected.height": 30
},
{"inference_id": 2,
"...": "..."}
]
}
You can also specify only a single nested field:
{
"select": [
{
"property": "inference_id"
},
{
"property": "objects_detected.class_id"
},
{
"property": "objects_detected.confidence"
}
]
}
The reponse will have 1 object per bounding box.
{
"query_result": [
{
"inference_id": "1",
"objects_detected.class_id": 0,
"objects_detected.confidence": 0.6
},
{
"inference_id": "1",
"objects_detected.class_id": 1,
"objects_detected.confidence": 0.6
},
{"inference_id": 2,
"...": "..."}
]
}
Note
When supplying the bounding box specific fields in filters, group bys, or order bys the columns must also be supplied in the select clause in order for the query to succeed.
Mean Average Precision#
Calculates Mean Average Precision for an object detection model. This is used as measure of accuracy for object detection models.
threshold determines minimum IoU value to be considered a match for a label. predicted_property and ground_truth_property are optional parameters and should be the names of the predicted and ground truth attributes for the model. They default to "objects_detected" and "label" respectively if nothing is specified for these parameters.
Query Request:
{
"select": [
{
"function": "meanAveragePrecision",
"alias": "<alias_name> [Optional]",
"parameters": {
"threshold": "<threshold> [float]",
"predicted_property": "<predicted_property> [str]",
"ground_truth_property": "<ground_truth_property> [str]"
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": "<result> [float]"
}
]
}
Example:
{
"select": [
{
"function": "meanAveragePrecision",
"parameters": {
"threshold": 0.5,
"predicted_property": "objects_detected",
"ground_truth_property": "label"
}
}
]
}
Query Response:
{
"query_result": [
{
"meanAveragePrecision": 0.78
}
]
}
Bias#
Bias Mitigation#
Calculates mitigated predictions based on conditional thresholds, returning 0/1 for each inference.
Note
This function returns null for inferences that don’t match any of the provided conditions.
Query Request:
{
"select":
[
{
"function": "biasMitigatedPredictions",
"alias": "<alias_name> [Optional]",
"parameters":
{
"predicted_property": "<predicted_property> [str]",
"thresholds":
[
{
"conditions":
{
"property": "<attribute_name> [string or nested]",
"comparator": "<comparator> [string] Optional: default 'eq'",
"value": "<string or number to compare with property>"
},
"threshold": "<threshold> [float]"
}
]
}
}
]
}
Query Response:
{
"query_result": [
{
"<function_name/alias_name>": "<result> [int]"
}
]
}
Example:
{
"select":
[
{
"function": "biasMitigatedPredictions",
"parameters":
{
"predicted_property": "prediction_1",
"thresholds":
[
{
"conditions":
[
{
"property": "SEX",
"value": 1
}
],
"threshold": 0.4
},
{
"conditions":
[
{
"property": "SEX",
"value": 2
}
],
"threshold": 0.6
}
]
}
}
]
}
Response:
{
"query_result":
[
{
"SEX": 1,
"biasMitigatedPredictions": 1
},
{
"SEX": 2,
"biasMitigatedPredictions": 0
},
{
"SEX": 1,
"biasMitigatedPredictions": 0
}
]
}