peal.architectures.basic_modules.SelfAttentionLayer

class peal.architectures.basic_modules.SelfAttentionLayer[source]

Bases: Module

Multi-head self-attention over a (B, L, C) sequence.

A sinusoidal positional encoding is computed from the sequence length and added to the input on every forward pass before attention is applied.

Parameters:

  • inplanes (int) – Feature size C (embedding dimension of the attention).

  • num_heads (int, default 1) – Number of attention heads.

  • use_masking (bool, default False) – If True a causal (upper-triangular) mask is applied so that position i only attends to positions <= i.

Methods

forward

Add sin/cos positional encodings and apply self-attention.
__init__(inplanes, num_heads=1, use_masking=False)[source]
Parameters:

  • inplanes (int)

  • num_heads (int)

  • use_masking (bool)

Return type:

None

forward(x)[source]

Add sin/cos positional encodings and apply self-attention.

Parameters:

x (torch.Tensor) – Shape (B, L, C); C must be even for the encoding.

Returns:

Attention output of shape (B, L, C).

Return type:

torch.Tensor