spikingjelly.activation_based.surrogate package

Module contents

spikingjelly.activation_based.surrogate.heaviside(x: Tensor)

• API in English

参数:

x – 输入tensor

返回:

输出tensor

heaviside阶跃函数，定义为

\[\begin{split}g(x) = \begin{cases} 1, & x \geq 0 \\ 0, & x < 0 \\ \end{cases}\end{split}\]

阅读 HeavisideStepFunction 以获得更多信息。

• 中文API

参数:

x – the input tensor

返回:

the output tensor

The heaviside function, which is defined by

\[\begin{split}g(x) = \begin{cases} 1, & x \geq 0 \\ 0, & x < 0 \\ \end{cases}\end{split}\]

For more information, see HeavisideStepFunction.

spikingjelly.activation_based.surrogate.check_manual_grad(primitive_function, spiking_function, *args, **kwargs)

参数:

• primitive_function (callable) – 梯度替代函数的原函数

• spiking_function (callable) – 梯度替代函数

梯度替代函数的反向传播一般是手写的，可以用此函数去检查手写梯度是否正确。

此函数检查梯度替代函数spiking_function的反向传播，与原函数primitive_function的反向传播结果是否一致。“一致”被定义为，两者的误差不超过eps。

示例代码：

def s2nn_apply(x, alpha, beta):
    return surrogate.s2nn.apply(x, alpha, beta)

surrogate.check_manual_grad(surrogate.S2NN.primitive_function, s2nn_apply, alpha=4., beta=1.)

spikingjelly.activation_based.surrogate.check_cuda_grad(neu, surrogate_function, device, *args, **kwargs)

class spikingjelly.activation_based.surrogate.SurrogateFunctionBase(alpha, spiking=True)

基类：Module

set_spiking_mode(spiking: bool)

extra_repr()

static spiking_function(x, alpha)

static primitive_function(x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_code_start_comments()

cuda_code_end_comments()

forward(x: Tensor)

cuda_codes(y: str, x: str, dtype: str)

training: bool

class spikingjelly.activation_based.surrogate.MultiArgsSurrogateFunctionBase(spiking: bool, *args, **kwargs)

基类：Module

set_spiking_mode(spiking: bool)

cuda_code(x: str, y: str, dtype='fp32')

cuda_code_start_comments()

cuda_code_end_comments()

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.piecewise_quadratic_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.piecewise_quadratic(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.PiecewiseQuadratic(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

反向传播时使用分段二次函数的梯度（三角形函数）的脉冲发放函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} 0, & |x| > \frac{1}{\alpha} \\ -\alpha^2|x|+\alpha, & |x| \leq \frac{1}{\alpha} \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} 0, & x < -\frac{1}{\alpha} \\ -\frac{1}{2}\alpha^2|x|x + \alpha x + \frac{1}{2}, & |x| \leq \frac{1}{\alpha} \\ 1, & x > \frac{1}{\alpha} \\ \end{cases}\end{split}\]

该函数在文章 [2] [4] [7] [11] [13] 中使用。

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The piecewise quadratic surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} 0, & |x| > \frac{1}{\alpha} \\ -\alpha^2|x|+\alpha, & |x| \leq \frac{1}{\alpha} \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} 0, & x < -\frac{1}{\alpha} \\ -\frac{1}{2}\alpha^2|x|x + \alpha x + \frac{1}{2}, & |x| \leq \frac{1}{\alpha} \\ 1, & x > \frac{1}{\alpha} \\ \end{cases}\end{split}\]

The function is used in [2] [4] [7] [11] [13].

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

spikingjelly.activation_based.surrogate.piecewise_exp_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.piecewise_exp(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.PiecewiseExp(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

反向传播时使用分段指数函数的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2}e^{-\alpha |x|}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}e^{\alpha x}, & x < 0 \\ 1 - \frac{1}{2}e^{-\alpha x}, & x \geq 0 \end{cases}\end{split}\]

该函数在文章 [6] [11] 中使用。

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The piecewise exponential surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2}e^{-\alpha |x|}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}e^{\alpha x}, & x < 0 \\ 1 - \frac{1}{2}e^{-\alpha x}, & x \geq 0 \end{cases}\end{split}\]

The function is used in [6] [11] .

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

spikingjelly.activation_based.surrogate.sigmoid_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.sigmoid(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.Sigmoid(alpha=4.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

反向传播时使用sigmoid的梯度的脉冲发放函数。反向传播为

\[g'(x) = \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x)\]

对应的原函数为

\[g(x) = \mathrm{sigmoid}(\alpha x) = \frac{1}{1+e^{-\alpha x}}\]

该函数在文章 [4] [12] [14] [15] 中使用。

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The sigmoid surrogate spiking function. The gradient is defined by

\[g'(x) = \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x)\]

The primitive function is defined by

\[g(x) = \mathrm{sigmoid}(\alpha x) = \frac{1}{1+e^{-\alpha x}}\]

The function is used in [4] [12] [14] [15] .

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.soft_sign_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.soft_sign(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.SoftSign(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

反向传播时使用soft sign的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2(1 + |\alpha x|)^{2}} = \frac{1}{2\alpha(\frac{1}{\alpha} + |x|)^{2}}\]

对应的原函数为

\[g(x) = \frac{1}{2} (\frac{\alpha x}{1 + |\alpha x|} + 1) = \frac{1}{2} (\frac{x}{\frac{1}{\alpha} + |x|} + 1)\]

该函数在文章 [8] [11] 中使用。

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The soft sign surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2(1 + |\alpha x|)^{2}}\]

The primitive function is defined by

\[g(x) = \frac{1}{2} (\frac{\alpha x}{1 + |\alpha x|} + 1)\]

The function is used in [8] [11] .

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

spikingjelly.activation_based.surrogate.super_spike_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.super_spike(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.SuperSpike(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

SuperSpike: Supervised learning in multi-layer spiking neural networks 提出的反向传播时使用SuperSpike的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{(1 + (|x|))^2}\]

• 中文API

The SuperSpike surrogate spiking function proposed by SuperSpike: Supervised learning in multi-layer spiking neural networks. The gradient is defined by

\[g'(x) = \frac{\alpha}{(1 + (|x|))^2}\]

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.atan_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.atan(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.ATan(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

反向传播时使用反正切函数arc tangent的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2(1 + (\frac{\pi}{2}\alpha x)^2)}\]

对应的原函数为

\[g(x) = \frac{1}{\pi} \arctan(\frac{\pi}{2}\alpha x) + \frac{1}{2}\]

• 中文API

The arc tangent surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2(1 + (\frac{\pi}{2}\alpha x)^2)}\]

The primitive function is defined by

\[g(x) = \frac{1}{\pi} \arctan(\frac{\pi}{2}\alpha x) + \frac{1}{2}\]

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.nonzero_sign_log_abs_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.nonzero_sign_log_abs(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.NonzeroSignLogAbs(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

警告

原函数的输出范围并不是(0, 1)。它的优势是反向传播的计算量特别小。

反向传播时使用NonzeroSignLogAbs的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{1 + |\alpha x|} = \frac{1}{\frac{1}{\alpha} + |x|}\]

对应的原函数为

\[g(x) = \mathrm{NonzeroSign}(x) \log (|\alpha x| + 1)\]

其中

\[\begin{split}\mathrm{NonzeroSign}(x) = \begin{cases} 1, & x \geq 0 \\ -1, & x < 0 \\ \end{cases}\end{split}\]

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

Warning

The output range the primitive function is not (0, 1). The advantage of this function is that computation cost is small when backward.

The NonzeroSignLogAbs surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{1 + |\alpha x|} = \frac{1}{\frac{1}{\alpha} + |x|}\]

The primitive function is defined by

\[g(x) = \mathrm{NonzeroSign}(x) \log (|\alpha x| + 1)\]

where

\[\begin{split}\mathrm{NonzeroSign}(x) = \begin{cases} 1, & x \geq 0 \\ -1, & x < 0 \\ \end{cases}\end{split}\]

static spiking_function(x, alpha)

static backward(grad_output, x, alpha)

static primitive_function(x: Tensor, alpha: float)

training: bool

spikingjelly.activation_based.surrogate.erf_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.erf(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.Erf(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的平滑程度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

反向传播时使用高斯误差函数(erf)的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{\sqrt{\pi}}e^{-\alpha^2x^2}\]

对应的原函数为

\begin{split} g(x) &= \frac{1}{2}(1-\text{erf}(-\alpha x)) \\ &= \frac{1}{2} \text{erfc}(-\alpha x) \\ &= \frac{1}{\sqrt{\pi}}\int_{-\infty}^{\alpha x}e^{-t^2}dt \end{split}

该函数在文章 [1] [4] [18] 中使用。

• 中文API

参数:

• alpha – parameter to control smoothness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The Gaussian error (erf) surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{\sqrt{\pi}}e^{-\alpha^2x^2}\]

The primitive function is defined by

\begin{split} g(x) &= \frac{1}{2}(1-\text{erf}(-\alpha x)) \\ &= \frac{1}{2} \text{erfc}(-\alpha x) \\ &= \frac{1}{\sqrt{\pi}}\int_{-\infty}^{\alpha x}e^{-t^2}dt \end{split}

The function is used in [1] [4] [18].

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

spikingjelly.activation_based.surrogate.piecewise_leaky_relu_backward(grad_output: Tensor, x: Tensor, w: float, c: float)

class spikingjelly.activation_based.surrogate.piecewise_leaky_relu(*args, **kwargs)

基类：Function

static forward(ctx, x: Tensor, w=1, c=0.01)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.PiecewiseLeakyReLU(w=1.0, c=0.01, spiking=True)

基类：MultiArgsSurrogateFunctionBase

• API in English

参数:

• w – -w <= x <= w 时反向传播的梯度为 1 / 2w

• c – x > w 或 x < -w 时反向传播的梯度为 c

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

分段线性的近似脉冲发放函数。梯度为

\[\begin{split}g'(x) = \begin{cases} \frac{1}{2w}, & -w \leq x \leq w \\ c, & x < -w ~or~ x > w \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} cx + cw, & x < -w \\ \frac{1}{2w}x + \frac{1}{2}, & -w \leq x \leq w \\ cx - cw + 1, & x > w \\ \end{cases}\end{split}\]

该函数在文章 [3] [4] [5] [9] [10] [12] [16] [17] 中使用。

• 中文API

参数:

• w – when -w <= x <= w the gradient is 1 / 2w

• c – when x > w or x < -w the gradient is c

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The piecewise surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \frac{1}{2w}, & -w \leq x \leq w \\ c, & x < -w ~or~ x > w \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} cx + cw, & x < -w \\ \frac{1}{2w}x + \frac{1}{2}, & -w \leq x \leq w \\ cx - cw + 1, & x > w \end{cases}\end{split}\]

The function is used in [3] [4] [5] [9] [10] [12] [16] [17].

forward(x)

static spiking_function(x: Tensor, w, c)

static backward(grad_output, x, w, c)

static primitive_function(x: Tensor, w: float, c: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

class spikingjelly.activation_based.surrogate.squarewave_fourier_series(*args, **kwargs)

基类：Function

static forward(ctx, x: Tensor, n: int, T_period: float)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.SquarewaveFourierSeries(n: int = 2, T_period: float = 8, spiking=True)

基类：MultiArgsSurrogateFunctionBase

forward(x)

static spiking_function(x: Tensor, w, c)

static primitive_function(x: Tensor, n: int, T_period: float)

cuda_code(x: str, y: str, dtype='fp32')

training: bool

class spikingjelly.activation_based.surrogate.s2nn(*args, **kwargs)

基类：Function

static forward(ctx, x: Tensor, alpha: float, beta: float)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.S2NN(alpha=4.0, beta=1.0, spiking=True)

基类：MultiArgsSurrogateFunctionBase

• API in English

参数:

• alpha – 控制 x < 0 时梯度的参数

• beta – 控制 x >= 0 时梯度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

S2NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Neural Networks 提出的S2NN替代函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x), x < 0 \\ \\frac{beta}{(x + 1)}, x \ge 0 \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \mathrm{sigmoid} (\alpha x), x < 0 \\ \beta \mathrm{ln}(x + 1) + 1, x \ge 0 \end{cases}\end{split}\]

• 中文API

参数:

• alpha – the param that controls the gradient when x < 0

• beta – the param that controls the gradient when x >= 0

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The S2NN surrogate spiking function, which is proposed by S2NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Neural Networks. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x), x < 0 \\ \beta (x + 1), x \ge 0 \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \mathrm{sigmoid} (\alpha x), x < 0 \\ \beta \mathrm{ln}(x + 1) + 1, x \ge 0 \end{cases}\end{split}\]

forward(x)

static spiking_function(x: Tensor, alpha, beta)

static primitive_function(x: Tensor, alpha: float, beta: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

class spikingjelly.activation_based.surrogate.q_pseudo_spike(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.QPseudoSpike(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度函数尾部厚度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

Surrogate Gradients Design 提出的 \(q\)-PseudoSpike替代函数。反向传播为

\[g'(x) = (1+\frac{2|x|}{\alpha-1})^{-\alpha}\]

其中 \(\alpha>1\) 对应原文中的 \(q\)。

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}(1-\frac{2x}{\alpha-1})^{1-\alpha}, & x < 0 \\ 1 - \frac{1}{2}(1+\frac{2x}{\alpha-1})^{1-\alpha}, & x \geq 0. \end{cases}\end{split}\]

• 中文API

参数:

• alpha – parameter to control tail fatness of gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The \(q\)-PseudoSpike surrogate spiking function, which is first proposed in Surrogate Gradients Design. The gradient is defined by

\[g'(x) = (1+\frac{2|x|}{\alpha-1})^{-\alpha}\]

where \(\alpha>1\) corresponds to \(q\) in paper.

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}(1-\frac{2x}{\alpha-1})^{1-\alpha}, & x < 0 \\ 1 - \frac{1}{2}(1+\frac{2x}{\alpha-1})^{1-\alpha}, & x \geq 0. \end{cases}\end{split}\]

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.leaky_k_relu_backward(grad_output: Tensor, x: Tensor, leak: float, k: float)

class spikingjelly.activation_based.surrogate.leaky_k_relu(*args, **kwargs)

基类：Function

static forward(ctx, x, leak, k)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.LeakyKReLU(spiking=True, leak: float = 0.0, k: float = 1.0)

基类：MultiArgsSurrogateFunctionBase

• API in English

参数:

• spiking (bool) – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

• leak (float) – gradient when x < 0

• k (float) – gradient when ``x >= 0 ``

反向传播时使用LeakyKReLU的梯度的脉冲发放函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} k, & x \geq 0 \\ leak, & x < 0 \\ \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} k \cdot x, & x \geq 0 \\ leak \cdot x, & x < 0 \\ \end{cases}\end{split}\]

• 中文API

参数:

• spiking (bool) – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

• leak (float) – x < 0 时的梯度值

• k (float) – ``x >= 0 `` 时的梯度值

The LeakyKReLU surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} k, & x \geq 0 \\ leak, & x < 0 \\ \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} k \cdot x, & x \geq 0 \\ leak \cdot x, & x < 0 \\ \end{cases}\end{split}\]

static spiking_function(x, leak, k)

static primitive_function(x: Tensor, leak: float, k: float)

static backward(grad_output, x, leak, k)

forward(x)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.fake_numerical_gradient_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.fake_numerical_gradient(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.FakeNumericalGradient(alpha=0.3)

基类：SurrogateFunctionBase

static spiking_function(x, alpha)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.log_tailed_relu_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.log_tailed_relu(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.LogTailedReLU(alpha=0.0, spiking=True)

基类：SurrogateFunctionBase

• API in English

参数:

• alpha – 控制反向传播时梯度的参数

• spiking – 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

Deep Learning with Low Precision by Half-wave Gaussian Quantization 提出的 Log-tailed ReLU替代函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} \alpha, & x \leq 0 \\ 1, & 0 < x \leq 0 \\ \frac{1}{x}, x > 1 \\ \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \alpha x, & x \leq 0 \\ x, & 0 < x \leq 0 \\ log(x), x > 1 \\ \end{cases}\end{split}\]

• 中文API

参数:

• alpha – parameter to control gradient

• spiking – whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

The Log-tailed ReLU surrogate spiking function, which is first proposed in Deep Learning with Low Precision by `Half-wave Gaussian Quantization. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \alpha, & x \leq 0 \\ 1, & 0 < x \leq 0 \\ \frac{1}{x}, x > 1 \\ \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \alpha x, & x \leq 0 \\ x, & 0 < x \leq 0 \\ log(x), x > 1 \\ \end{cases}\end{split}\]

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

training: bool

spikingjelly.activation_based.surrogate.deterministic_pass_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.deterministic_pass(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.DeterministicPass(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

spikingjelly.activation_based.surrogate.rect_backward(grad_output: Tensor, x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.rect(*args, **kwargs)

基类：Function

static forward(ctx, x, alpha)

static backward(ctx, grad_output)

class spikingjelly.activation_based.surrogate.Rect(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

training: bool

class spikingjelly.activation_based.surrogate.poisson_pass(*args, **kwargs)

基类：Function

static forward(ctx, x)

static backward(ctx, grad_output)

References

[1] (1,2)

Esser S K, Appuswamy R, Merolla P, et al. Backpropagation for energy-efficient neuromorphic computing[J]. Advances in neural information processing systems, 2015, 28: 1117-1125.

[2] (1,2)

Esser S K, Merolla P A, Arthur J V, et al. Convolutional networks for fast, energy-efficient neuromorphic computing[J]. Proceedings of the national academy of sciences, 2016, 113(41): 11441-11446.

[3] (1,2)

Yin S, Venkataramanaiah S K, Chen G K, et al. Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations[C]//2017 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, 2017: 1-5.

[4] (1,2,3,4,5,6,7,8)

Wu Y, Deng L, Li G, et al. Spatio-temporal backpropagation for training high-performance spiking neural networks[J]. Frontiers in neuroscience, 2018, 12: 331.

[5] (1,2)

Huh D, Sejnowski T J. Gradient descent for spiking neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 1440-1450.

[6] (1,2)

Shrestha S B, Orchard G. SLAYER: spike layer error reassignment in time[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 1419-1428.

[7] (1,2)

Bellec G, Salaj D, Subramoney A, et al. Long short-term memory and learning-to-learn in networks of spiking neurons[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 795-805.

[8] (1,2)

Zenke F, Ganguli S. Superspike: Supervised learning in multilayer spiking neural networks[J]. Neural computation, 2018, 30(6): 1514-1541.

[9] (1,2)

Wu Y, Deng L, Li G, et al. Direct training for spiking neural networks: Faster, larger, better[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33(01): 1311-1318.

[10] (1,2)

Gu P, Xiao R, Pan G, et al. STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks[C]//IJCAI. 2019: 1366-1372.

[11] (1,2,3,4,5,6)

Neftci E O, Mostafa H, Zenke F. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[J]. IEEE Signal Processing Magazine, 2019, 36(6): 51-63.

[12] (1,2,3,4)

Roy D, Chakraborty I, Roy K. Scaling deep spiking neural networks with binary stochastic activations[C]//2019 IEEE International Conference on Cognitive Computing (ICCC). IEEE, 2019: 50-58.

[13] (1,2)

Panda P, Aketi S A, Roy K. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization[J]. Frontiers in Neuroscience, 2020, 14.

[14] (1,2)

Lotfi Rezaabad A, Vishwanath S. Long Short-Term Memory Spiking Networks and Their Applications[C]//International Conference on Neuromorphic Systems 2020. 2020: 1-9.

[15] (1,2)

Woźniak S, Pantazi A, Bohnstingl T, et al. Deep learning incorporating biologically inspired neural dynamics and in-memory computing[J]. Nature Machine Intelligence, 2020, 2(6): 325-336.

[16] (1,2)

Cheng X, Hao Y, Xu J, et al. LISNN: Improving Spiking Neural Networks with Lateral Interactions for Robust Object Recognition[C]//IJCAI. 1519-1525.

[17] (1,2)

Kaiser J, Mostafa H, Neftci E. Synaptic plasticity dynamics for deep continuous local learning (DECOLLE)[J]. Frontiers in Neuroscience, 2020, 14: 424.

[18] (1,2)

Yin B, Corradi F, Bohté S M. Effective and efficient computation with multiple-timescale spiking recurrent neural networks[C]//International Conference on Neuromorphic Systems 2020. 2020: 1-8.