peal.training.trainers.calculate_test_accuracy

peal.training.trainers.calculate_test_accuracy(model, test_dataloader, device, calculate_group_accuracies=False, max_test_batches=None, tracking_level=2)[source]

Compute the accuracy of model on a dataloader, optionally per group.

Groups are y + output_size * has_confounder, i.e. label x single binary confounder; datasets without has_confounder fall back to per-class groups and empty groups are skipped. The dataset is switched to dict output with groups enabled for the duration of the call and restored afterwards.

Parameters:

  • model (torch.nn.Module) – Classifier returning logits.

  • test_dataloader (torch.utils.data.DataLoader) – Loader over a PEAL dataset (needs enable_groups etc. for groups).

  • device (str or torch.device) – Device the inputs are moved to.

  • calculate_group_accuracies (bool, optional) – Also compute per-group accuracies. Default False.

  • max_test_batches (int, optional) – Stop after this many batches (accuracy is still divided by the full dataset length).

  • tracking_level (int, optional) – Progress bar is updated when >= 1.

Returns:

Accuracy alone, or (accuracy, group_accuracies, group_distribution, group_counts, worst_group_accuracy) when calculate_group_accuracies is set.

Return type:

float or tuple

Raises:

TypeError – When the data config names more than one confounding factor, so has_confounder is a list rather than a tensor.