peal.sparse_dictionaries.sae_evaluation.run_sae_eval

peal.sparse_dictionaries.sae_evaluation.run_sae_eval(sae, x, y, base_path=None, label_names=None, matching_method='correlation', verbose=True, encode_batch_size=8192, store_latents=False, n_binary=None, x_holdout=None, y_holdout=None, f1_matching=True, subset_names=None, subset_key='subset')[source]

Full SAE evaluation for superposition datasets.

Parameters:

  • sae – SAE model with a .encode() method

  • x (torch.Tensor) – Input tensors, shape (N, 512)

  • y (torch.Tensor) – Labels, shape (N, num_labels). May mix binary (sparse) columns with continuous (dense) columns, as in SparseNumbersDenseZipf: 1000 binary “Num” columns followed by 8 continuous “Red” intensities.

  • label_names (list) – Optional names for the labels

  • matching_method (str) – “correlation” (optimal) or “greedy” (fast)

  • verbose (bool) – Print and save a report

  • n_binary (int) – Number of leading binary columns. If None, columns are classified automatically (a column is binary iff all its values are 0 or 1).

  • x_holdout (torch.Tensor) – Optional second input tensor, never used for matching or threshold selection. Since both the latent->label assignment and the per-label threshold are chosen to maximise agreement on x, the metrics on x are in-sample. Passing a disjoint split here gives the honest number under the same protocol.

  • y_holdout (torch.Tensor) – Labels for x_holdout.

  • subset_names (list) – Optional subset of label_names to additionally aggregate over (e.g. CELEBA_WELL_DEFINED_20). Reported as <subset_key>_f1_macro etc.

  • subset_key (str) – Metric-name prefix for that subset.

  • base_path (str)

  • encode_batch_size (int)

  • store_latents (bool)

  • f1_matching (bool)

Returns:

dict with all results

Return type:

dict