peal.global_utils.LeakySoftplus

class peal.global_utils.LeakySoftplus

Bases: Module

Smooth, everywhere non-zero gradient replacement for ReLU.

Computes 0.5 * (LeakyReLU(x, 0.1) + Softplus(x, beta=10)). Swapping the ReLUs of a predictor for this activation (see replace_relu_with_leakysoftplus()) keeps the function close to the original one while removing the flat region that stalls the gradient-based counterfactual search.

Methods

forward

Return the mean of the leaky ReLU and the softplus of x.

__init__()

Build the leaky ReLU and softplus halves of the activation.

forward(x)

Return the mean of the leaky ReLU and the softplus of x.