arthurai.core.bias.bias_metrics.BiasMetrics#

class arthurai.core.bias.bias_metrics.BiasMetrics(arthur_model)#

Bases: object

Methods

demographic_parity

Get group-conditional positivity rates for all inferences, with the option to filter for a batch_id or a particular chunk of time.

group_confusion_matrices

Get group-conditional confusion matrices for all inferences, with the option to filter for a batch_id or a particular chunk of time.

group_positivity_rates

Get group-conditional positivity rates for all inferences, with the option to filter for a batch_id or a particular chunk of time.
demographic_parity(attr_name, batch_id=None, start_time=None, end_time=None)#

Get group-conditional positivity rates for all inferences, with the option to filter for a batch_id or a particular chunk of time. Currently only supports calculating bias with respect to a single sensitive attribute at a time. This method currently only handles binary models.

Parameters

  • attr_name (str) – The name of the sensitive attribute to get metrics for.

  • batch_id (Optional[str]) – Optional filter to limit calculations to a specific batch ID.

  • start_time (Optional[datetime]) – Optional filter to limit calculations to inferences after a specific timestamp.

  • end_time (Optional[datetime]) – Optional filter to limit calculations to inferences before a specific timestamp.

Returns

A dict of the form {sens_1: pos_1, sens_2: pos_2, …}

group_confusion_matrices(attr_name, pred_property=None, batch_id=None, start_time=None, end_time=None, return_by_metric=True)#

Get group-conditional confusion matrices for all inferences, with the option to filter for a batch_id or a particular chunk of time. Currently only supports calculating bias with respect to a single sensitive attribute at a time. This method handles both binary and multiclass models.

Parameters

  • attr_name (str) – The name of the sensitive attribute to get metrics for.

  • pred_property (Optional[str]) – For multiclass models, the predicted label to get the confusion matrix for.

  • batch_id (Optional[str]) – Optional filter to limit calculations to a specific batch ID.

  • start_time (Optional[datetime]) – Optional filter to limit calculations to inferences after a specific timestamp.

  • end_time (Optional[datetime]) – Optional filter to limit calculations to inferences before a specific timestamp.

  • return_by_metric – Whether the returned dictionary of results should be keyed by metric or by sensitive attribute value.

Returns

Either a dict of the form { metric: { sens_1: val1, sens_2: val2 }, …} (default) or its reverse.

group_positivity_rates(attr_name, batch_id=None, start_time=None, end_time=None)#

Get group-conditional positivity rates for all inferences, with the option to filter for a batch_id or a particular chunk of time. Currently only supports calculating bias with respect to a single sensitive attribute at a time. This method currently only handles binary models.

Parameters

  • attr_name (str) – The name of the sensitive attribute to get metrics for.

  • batch_id (Optional[str]) – Optional filter to limit calculations to a specific batch ID.

  • start_time (Optional[datetime]) – Optional filter to limit calculations to inferences after a specific timestamp.

  • end_time (Optional[datetime]) – Optional filter to limit calculations to inferences before a specific timestamp.

Returns

A dict of the form {sens_1: pos_1, sens_2: pos_2, …}