peal.training.training_utils.calculate_validation_statistics

peal.training.training_utils.calculate_validation_statistics(model, dataloaders, tracked_keys, base_path, output_size, device, logits_to_prediction, use_confusion_matrix, explainer, max_validation_samples)[source]

Compute accuracy / error statistics and explainer outputs on validation data.

For every dataloader the model is evaluated with temperature-scaled softmax (explainer.explainer_config.temperature), the prediction is mapped to a class with logits_to_prediction and the explainer is asked to explain the batch towards the target (y_pred + 1) % output_size. Keys of the explainer result that appear in tracked_keys are accumulated.

Parameters:

  • model (nn.Module) – Classifier producing logits of shape (B, output_size).

  • dataloaders (list of torch.utils.data.DataLoader) – Validation loaders; the sample budget is split evenly between them and a trailing batch smaller than batch_size stops the loop.

  • tracked_keys (list of str) – Explainer result keys to collect, e.g. "x_counterfactual_list". "hint_list" / "idx_list" additionally unpack a list-valued y into label, hints and indices.

  • base_path (str) – Directory handed to explainer.explain_batch for its artefacts.

  • output_size (int) – Number of classes.

  • device (str or torch.device) – Device the inputs are moved to.

  • logits_to_prediction (callable) – Maps a confidence tensor to integer predictions.

  • use_confusion_matrix (bool) – If True and the accuracy is below 1.0, the normalised off-diagonal confusion matrix is returned as error_matrix; otherwise a uniform off-diagonal matrix is used.

  • explainer (ExplainerInterface) – Explainer whose explain_batch runs in mode="validation".

  • max_validation_samples (int) – Upper bound on the number of samples over all dataloaders.

Returns:

  • tracked_values (dict) – {key: list} for every key in tracked_keys.

  • validation_stats (dict) – accuracy (float), confidence_score_stats (zero tensor of shape (output_size, output_size), currently a placeholder) and error_matrix (flattened tensor of length output_size ** 2 that sums to one). Only the last dataloader’s values are returned.