peal.adaptors.counterfactual_knowledge_distillation.CFKDConfig

class peal.adaptors.counterfactual_knowledge_distillation.CFKDConfig

Bases: AdaptorConfig

The config template for an running the CFKD adaptor.

Loaded with load_yaml_config(..., CFKDConfig) from a yaml whose adaptor_type is "CFKD". Each field is described by the string literal that follows it in the class body, which is what Sphinx renders as that field’s attribute documentation. The few fields without such a string are:

Parameters:

• sparse_dictionary (SparseDictionaryConfig, dict or None) – If set, this dictionary is loaded and attached to the generator, overriding whatever the generator loaded itself; component_indices index into it.

• correct_clusters (list of int) – Cluster indices treated as correct by teachers that judge clusters.

• use_true_counterfactuals (bool) – Whether “true” counterfactuals (labelled with the target class) are added to the finetune dataset besides the “false” ones.

• seed (int or None) – Random seed of the run.

• component_indices (list of int or None) – Dictionary components the explainer edits along; when given, the explainer’s component_indices and num_attempts are set from it.

Notes

Fields such as current_iteration, feedback_accuracies, group_accuracies and model_path are progress logs that CFKD writes back into base_dir/config.yaml after every iteration, which is what makes a run resumable.

Methods

Attributes

adaptor_type	The adaptor_type for CFKDConfig has to be CFKD to find the CFKDConfig class when loading from a yaml file.
attribution_threshold	The attribution threshold when using the SegmentationMask teacher.
avg_group_accuracies	The average group accuracies of the model after every finetuning iteration.
base_dir	The base directory where the run of CFKD is stored.
batch_size	What batch_size is used for creating the counterfactuals?
best_feedback_accuracy	Logging of the Feedback Accuracy.
calculate_group_accuracies	Whether to calculate group accuracies or not.
clustering_strategy	The clustering strategy used by the counterfactual explainer
component_indices
continuous_learning	Whether to continue training from the current student model or start training from scratch again.
correct_clusters
counterfactual_type	What type of counterfactuals are valid.
current_iteration	Logging of the current finetune iteration
data	The config of the data used to create the counterfactuals from.
explainer	The config of the counterfactual explainer that is used.
explainer_stats_clusters	The cluster indices used for calculating explainer stats (sparsity, diversity, feedback metrics).
feedback_accuracies
finetune_iterations	The number of finetune iterations when executing the adaptor.
generator	The config of the generator used.
generator_performance	The performance of the generative model measured e.g. in FID score.
group_accuracies	The group accuracies of the model after every finetuning iteration.
is_loaded	Dummy field to be able to use it as a model config!
lazy_feedback	Whether to always give feedback directly after creating validation counterfactuals or whether to wait until the next train feedback shall be given as well (which means less interruptions!)
max_test_batches	The maximum number of test batches.
max_validation_samples	The maximum number of validation samples that are used for tracking stats every iteration.
min_train_samples	The minimum number of samples used for finetuning in every iteration.
mixing_ratio	How aggressively to change the model based on the counterfactual samples.
model_path	The path of the last finetuned model
overwrite	Whether to overwrite the logs and cache intermediate results.
reproducible_sampling
seed	Optional.
sparse_dictionary
student	The path of the student used.
task	The config of the task the student model shall solve.
teacher	The type of teacher used.
test_data	The config of the test data used evaluate the real progress on.
training	The config of the training used for finetuning the student model.
transition_restrictions	The restriction to interesting counterfactual transitions.
use_confusion_matrix	Whether to draw samples for counterfactual creation according to the error matrix or not.
use_true_counterfactuals
validation_runs	The number validation runs used for evaluating CFKD.
visualize_latent_diversity	Whether to visualize the individual latent diversity scores as collages.
visualize_latent_sparsity	Whether to visualize the individual latent sparsity scores as collages.

adaptor_type: str = 'CFKD'

The adaptor_type for CFKDConfig has to be CFKD to find the CFKDConfig class when loading from a yaml file.

min_train_samples: pydantic.PositiveInt = 800

The minimum number of samples used for finetuning in every iteration. The actual number could be higher since not for every sample a counterfactual can be found and processing is done in batches.

max_validation_samples: pydantic.PositiveInt = 200

The maximum number of validation samples that are used for tracking stats every iteration.

max_test_batches: None | pydantic.PositiveInt = None

The maximum number of test batches. If set to None the test will be done on the full test set.

finetune_iterations: int = 1

The number of finetune iterations when executing the adaptor. If set to 0 only the explanation and no adaption is done.

task: TaskConfig | None = None

The config of the task the student model shall solve.

explainer: dict | ExplainerConfig

The config of the counterfactual explainer that is used. All parameters regarding paths, where the generator is from etc in there are overwritten by CFKD and only used if the information is not available for CFKD

training: TrainingConfig | None

The config of the training used for finetuning the student model. If not set student config can be used.

data: DataConfig = None

The config of the data used to create the counterfactuals from.

test_data: DataConfig = None

The config of the test data used evaluate the real progress on. Often this data has a distribution shift compared to the training data or comes from a totally different data source. Hence the option to give its own config. If set to None the normal data config is taken.

student: PredictorConfig | str | None = None

The path of the student used. Can be either the path to a PyTorch or an onnx model directly or the path to a predictor config or a PredictorConfig object.

teacher: str | dict = 'cluster@8000'

The type of teacher used.

generator: GeneratorConfig | None = None

The config of the generator used. This value will be overwritten if Generator is given via constructor directly. If the Generator is not given via constructor and this value is set to None explainer config is searched for generator config.

base_dir: str = 'peal_runs/cfkd'

The base directory where the run of CFKD is stored. All the visualizations and the caching is stored here.

current_iteration: int = 0

Logging of the current finetune iteration

continuous_learning: str = 'finetune'

Whether to continue training from the current student model or start training from scratch again. Can e.g. be “retrain”, which retrains model on original data and counterfactuals from scratch, “finetune”, which starts of at the weights of the uncorrected student or “deep_feature_reweighting”, which only finetunes the last layer of the the uncorrected student.

use_confusion_matrix: bool = False

Whether to draw samples for counterfactual creation according to the error matrix or not. Makes particular sense in the multiclass setting where some classes might be in very different modes and one only wants to restrict to connected modes.

best_feedback_accuracy: float = 0.0

Logging of the Feedback Accuracy.

attribution_threshold: float = 0.0

The attribution threshold when using the SegmentationMask teacher. Setting it to 0.0 means that every counterfactual that did on average bigger changes inside than outside the mask is considered a True counterfactual and every counterfactual that does not is considered a false counterfactual. If it is set higher the bar for True Counterfactual is set higher and if is set lower the bar is set lower as well.

batch_size: pydantic.PositiveInt = 1

What batch_size is used for creating the counterfactuals?

validation_runs: pydantic.PositiveInt = 1

The number validation runs used for evaluating CFKD.

calculate_group_accuracies: bool = False

Whether to calculate group accuracies or not. This can only be done if confounding factors are known.

overwrite: bool = True

Whether to overwrite the logs and cache intermediate results. If overwrite is set to False cached results are loaded. If CFKDConfig is stored as yaml on disk overwrite is automatically set to False so that CFKD can be continued at the last cached result. Using this feature dramatically improves ability to debug!

mixing_ratio: float = 0.5

How aggressively to change the model based on the counterfactual samples. 0 -> No change, 1 -> Full change

feedback_accuracies: list = []

” A list of the feedback accuracies.

group_accuracies: list = []

The group accuracies of the model after every finetuning iteration.

avg_group_accuracies: list = []

The average group accuracies of the model after every finetuning iteration.

counterfactual_type: str = '1sided'

What type of counterfactuals are valid. 1sided means that we can only start from samples with correct prediction, 2sided also allows that we start from samples with wrong original prediction.

lazy_feedback: bool = True

Whether to always give feedback directly after creating validation counterfactuals or whether to wait until the next train feedback shall be given as well (which means less interruptions!)

model_path: str = ''

The path of the last finetuned model

is_loaded: bool = False

Dummy field to be able to use it as a model config!

generator_performance: dict = {}

The performance of the generative model measured e.g. in FID score.

transition_restrictions: list | None = None

The restriction to interesting counterfactual transitions. Helpful in the case of datasets with a lot of classes and heavy modes like ImageNet.

clustering_strategy: str | None = 'attempt_nr'

The clustering strategy used by the counterfactual explainer

sparse_dictionary: SparseDictionaryConfig | dict | None = None

correct_clusters: list = [0]

use_true_counterfactuals: bool = False

seed: int | None = 0

Optional. If True (and seed is set), get_batch reseeds from seed * 1000003 + a per-call counter instead of OS entropy, so the choice of samples is reproducible per seed and each call still draws a different shuffle. Used by the Camelyon17 seed-variability runs. False (the default) keeps the original behaviour: torch.manual_seed(torch.seed()).

reproducible_sampling: bool = False

visualize_latent_sparsity: bool = True

Whether to visualize the individual latent sparsity scores as collages. Each collage contains: factual, counterfactual, target end confidence, sparsity score.

visualize_latent_diversity: bool = True

Whether to visualize the individual latent diversity scores as collages. Each collage contains: factual, cf1, cf2, target end confidences, diversity score.

explainer_stats_clusters: list = [0, 1]

The cluster indices used for calculating explainer stats (sparsity, diversity, feedback metrics). Only these clusters are considered when computing latent difference statistics and feedback stats.

component_indices: list | None = None