peal.sparse_dictionaries.activation_cache.load_or_extract

peal.sparse_dictionaries.activation_cache.load_or_extract(generator, datasets, data_config, feature_extractor, device, **kw)[source]

The cached form of extract(), when PEAL_ACT_CACHE says so.

With the cache disabled this is exactly extract(). Otherwise the file from cache_path() is reused when its stored dataset sizes match [len(d) for d in datasets]; on a miss or size mismatch the activations are extracted, written to that file and returned.

Parameters:

  • generator (InvertibleGenerator) – Used only to derive the cache path.

  • datasets (list of torch.utils.data.Dataset) – Datasets to encode.

  • data_config (DataConfig) – Used only to derive the cache path.

  • feature_extractor (callable) – Encoder applied to each batch.

  • device (str or torch.device) – Device for the encoder inputs.

  • **kw – Forwarded to extract() (batch_size, num_workers).

Returns:

Same layout as extract().

Return type:

list of tuple