peal.web.analysis

Per-direction analysis shown on the feedback page of a web-demo job:

python -m peal.web.analysis <job_dir> python -m peal.web.analysis –accuracy-only <job_dir> # accuracy.json only

Writes <job>/analysis.json with

• the headline accuracy of the uploaded classifier on all uploaded images;

• one entry per ranked direction (run/sweep_results.pt) with its latent, ambient and verified flip counts, the verified factual/counterfactual pairs written to run/successful_flips (see its index.json) and four group accuracies: class x whether the direction’s SAE concept is active on the real image, plus their mean (average group accuracy) and minimum (worst group accuracy).

The concept split follows reproduction_scripts/overnight_group_eval.py: an image has the concept when the MSAE code of the direction’s atom (any of its atoms for a concept-replacement pair) on the image’s OpenAI CLIP ViT-L/14 embedding is positive. Unlike that script, which evaluates a trained probe on its held-out val+test split, this runs on every uploaded image: the uploaded classifier was not trained on them by PEAL, and the upload is small.

Module attributes

CLIP_MEAN

CLIP's input normalization (openai/CLIP, ViT-L/14).

Functions

analyse	Write <job_dir>/analysis.json (see the module docstring) and return it.
direction_atoms	SAE atom indices in a direction name, e.g. "squirting (SAE #5717)".
group_accuracies	Accuracy of the four (class, concept present) groups.
quick_accuracy	Write <job_dir>/accuracy.json: the classifier's accuracy on all uploaded images, overall and per class, before DiDAE starts.