peal.sparse_dictionaries.batch_topk_network.SparseAutoencoderBase

class peal.sparse_dictionaries.batch_topk_network.SparseAutoencoderBase

Bases: Module

Shared parameters and housekeeping of a tied-init sparse autoencoder.

Parameters:

cfg (dict) – Needs seed, act_size, dict_size, device, dtype and input_unit_norm.

Variables:

• W_enc (torch.nn.Parameter) – (act_size, dict_size) encoder weight.

• W_dec (torch.nn.Parameter) – (dict_size, act_size) decoder weight; rows are the dictionary atoms and are kept unit-norm during training.

• b_dec (b_enc,) – Encoder and decoder biases.

• num_batches_not_active (torch.Tensor) – Per-atom count of consecutive batches with no activation, used to find dead atoms.

Methods

decode
encode
make_decoder_weights_and_grad_unit_norm
postprocess_output	Undo preprocess_input() on a reconstruction.
preprocess_input	Optionally standardise each row; returns (x, mean, std).
update_inactive_features

__init__(cfg)

preprocess_input(x)

Optionally standardise each row; returns (x, mean, std).

postprocess_output(x_reconstruct, x_mean, x_std)

Undo preprocess_input() on a reconstruction.

make_decoder_weights_and_grad_unit_norm()

Project the decoder rows back to the unit sphere, and remove the radial component of their gradient so the optimiser step stays tangential.

update_inactive_features(acts)

Increment the dead-atom counters for atoms unused in this batch.

encode(x)

decode(acts)