peal.data.interfaces.DataConfig

class peal.data.interfaces.DataConfig

Bases: BaseModel

Pydantic description of a dataset, loadable from yaml.

The individual fields are documented by the string literals that precede them in the source. The fields without such a literal configure diffusion based data augmentation and are consumed by dataset_factory and peal.data.transformations.DiffusionAugmentation.

Parameters:

• generator (str or object or None) – Generator (config path, config or instance) used for augmentation.

• diffusion_augmented (bool) – If True the training transform noises every sample with generator and denoises it again (a stochastic augmentation).

• sampling_time_fraction (float) – Fraction of the diffusion schedule to noise to (default 0.3).

• num_discretization_steps (int) – Number of denoising steps used by the augmentation (default 20).

• batch_wise_augmentation (bool) – If True (default) the per-sample DiffusionAugmentation transform is not added to the dataset; augmentation is expected to be applied to whole batches elsewhere.

Methods

Attributes

background	The background contexts to use for the experiment.
batch_wise_augmentation
class_ratios	The ratio of the classes in the dataset.
config_name	The type of config.
confounder_probability	The correlation strength of the target and the confounding variable.
confounding_factors	A pair of known confounding factors, one usually being the target.
crop_size	The number of classes in the dataset.
dataset_class	The name of the dataset.
dataset_origin_path	The path of the original dataset.
dataset_path	The path to the dataset.
delimiter	The delimiter used for the csv file.
diffusion_augmented
downsize	The way how to downsize an sample if required.
font_path	The path to the font file.
foreground	The foreground categories to use for the experiment.
full_confounder_config	alternative to confounder_probability, specify individual group sizes, e.g. [0.25, 0.25, 0.25, 0.25].
generator
has_hints	Whether the dataset contains spatial annotations where the true feature is.
in_memory	Whether to load all datasets into the RAM or not.
input_size	The input size of data.
input_type	The input type of the data: Options: ['image', 'sequence', 'symbolic']
invariances	A list of the invariances exploited for data augmentation: Options: ['hflipping', 'vflipping', 'rotation', 'circlecut']
inverse	The path of the original dataset.
label_noise	The label noise of a generated dataset.
label_rel_path	Where the label path is relative to the dataset path.
normalization	The applied normalization.
num_discretization_steps
num_samples	The number of samples in the dataset.
output_size	The output size of the model.
output_split	The number of binary multiclass variables in the mixed setting.
output_type	The output type of the data.
sampling_time_fraction
seed	The seed the dataset was generated with.
set_negative_to_zero	Whether to set negative values to zero.
split	The split between train, validation and test set.
spray_groups_balanced	Whether to re-balance group sizes after dropping samples without a spray label
spray_label_file	Path to spray label csv file.
tabular_preprocessing	List of preprocessing methods to apply to tabular datasets.
x_selection	The name of the folder where the images are stored.

config_name: str = 'DataConfig'

The type of config. This is necessary to find config class from yaml config

input_type: str = 'image'

The input type of the data: Options: [‘image’, ‘sequence’, ‘symbolic’]

output_type: str = 'singleclass'

The output type of the data. Options: [‘singleclass’, ‘multiclass’, ‘continuous’, ‘mixed’] ‘mixed’ is a hybrid between binary multiclass classification and continuous and requires ‘output_split’ to be set

input_size: list[pydantic.PositiveInt] = [3, 128, 128]

The input size of data. For images: [Channels, Height, Width] For sequences: [MaxLength, NumTokens] For symbolic: [NumVariables]

output_size: list[pydantic.PositiveInt] = [2]

The output size of the model. For singleclass: [NumClasses] For multiclass: [NumBinaryClasses] For continuous: [NumVariables] For mixed: [NumBinaryClasses + NumVariables]

dataset_path: None | str = None

The path to the dataset.

num_samples: None | int = None

The number of samples in the dataset. Sometimes important when executing specific experiments.

dataset_class: None | str = None

The name of the dataset. Only necessary to tell dataset factory which customized dataset class to use

split: list[float] = [0.8, 0.9]

The split between train, validation and test set.

has_hints: None | bool = False

Whether the dataset contains spatial annotations where the true feature is.

normalization: None | list = None

The applied normalization. Options: [‘mean0std1’]

invariances: list[str] = []

A list of the invariances exploited for data augmentation: Options: [‘hflipping’, ‘vflipping’, ‘rotation’, ‘circlecut’]

output_split: None | int = None

The number of binary multiclass variables in the mixed setting. Has to be smaller than output_size.

downsize: None | str = None

The way how to downsize an sample if required. Options: [‘Downsample’, ‘RandomCrop’, ‘CenterCrop’]

confounding_factors: None | list[str] = []

A pair of known confounding factors, one usually being the target. This knowledge helps for controlled sampling of confounders for experiments.

foreground: None | list[str] = None

The foreground categories to use for the experiment.

background: None | list[str] = None

The background contexts to use for the experiment.

confounder_probability: None | float = None

The correlation strength of the target and the confounding variable.

full_confounder_config: None | list[float] = None

alternative to confounder_probability, specify individual group sizes, e.g. [0.25, 0.25, 0.25, 0.25]

class_ratios: None | list = None

The ratio of the classes in the dataset.

seed: None | int = 0

The seed the dataset was generated with. Only relevant for generated datasets!

label_noise: None | float = None

The label noise of a generated dataset. Necessary to mimic real dataset behauviour and avoid trivial non-robust solutions.

set_negative_to_zero: bool = True

Whether to set negative values to zero.

font_path: None | str = None

The path to the font file.

delimiter: None | str = ','

The delimiter used for the csv file.

crop_size: None | int = None

The number of classes in the dataset.

dataset_origin_path: None | str = None

The path of the original dataset.

inverse: None | str = None

The path of the original dataset.

label_rel_path: str = 'data.csv'

Where the label path is relative to the dataset path.

x_selection: str = 'imgs'

The name of the folder where the images are stored. The header of the column in the underlying csv either has to be called like this as well or the path to the images has to be in the first column.

in_memory: bool = False

Whether to load all datasets into the RAM or not. Careful with big datasets!

spray_label_file: str | None = None

Path to spray label csv file. When set, use spray labels instead of true confounder labels. Samples without a spray label will be dropped

spray_groups_balanced: bool = False

Whether to re-balance group sizes after dropping samples without a spray label

generator: None | str | object = None

diffusion_augmented: bool = False

sampling_time_fraction: float = 0.3

num_discretization_steps: int = 20

batch_wise_augmentation: bool = True

tabular_preprocessing: None | list[str] = None

List of preprocessing methods to apply to tabular datasets. Options: ['minmax_-1_1']