spikingjelly.activation_based.surrogate module

spikingjelly.activation_based.surrogate.heaviside(x: Tensor)

API Language: 中文 | English

---

• 中文

heaviside阶跃函数，定义为

\[\begin{split}g(x) = \begin{cases} 1, & x \geq 0 \\ 0, & x < 0 \\ \end{cases}\end{split}\]

阅读 HeavisideStepFunction 以获得更多信息。

参数:

x (torch.Tensor) -- 输入tensor

返回:

输出tensor

返回类型:

torch.Tensor

---

• English

The heaviside function, which is defined by

\[\begin{split}g(x) = \begin{cases} 1, & x \geq 0 \\ 0, & x < 0 \\ \end{cases}\end{split}\]

For more information, see HeavisideStepFunction.

参数:

x (torch.Tensor) -- the input tensor

返回:

the output tensor

返回类型:

torch.Tensor

spikingjelly.activation_based.surrogate.check_manual_grad(primitive_function, spiking_function, *args, **kwargs)

API Language: 中文 | English

---

• 中文

梯度替代函数的反向传播一般是手写的，可以用此函数去检查手写梯度是否正确。

此函数检查梯度替代函数spiking_function的反向传播，与原函数primitive_function的反向传播结果是否一致。 "一致"被定义为，两者的误差不超过eps。

参数:

• primitive_function (callable) -- 梯度替代函数的原函数

• spiking_function (callable) -- 梯度替代函数

---

• English

The manual gradient of surrogate gradient functions is usually written by hand, and this function can be used to check if the manual gradient is correct.

This function checks if the backward pass of surrogate function spiking_function is consistent with the backward pass of the primitive function primitive_function. "Consistency" is defined as the error between the two not exceeding eps.

参数:

• primitive_function (callable) -- the primitive function of surrogate gradient

• spiking_function (callable) -- the surrogate function

---

• 代码示例 | Example

def s2nn_apply(x, alpha, beta):
    return surrogate.s2nn.apply(x, alpha, beta)


surrogate.check_manual_grad(
    surrogate.S2NN.primitive_function, s2nn_apply, alpha=4.0, beta=1.0
)

返回:

无返回值，直接打印对比结果

返回类型:

None

spikingjelly.activation_based.surrogate.check_cuda_grad(neu, surrogate_function, device, *args, **kwargs)

API Language: 中文 | English

---

• 中文

检查CUDA（CuPy）后端的梯度是否正确。将CuPy后端的梯度与PyTorch后端的梯度进行比较。

参数:

• neu (type) -- 神经元类

• surrogate_function (type) -- 替代函数类（未实例化）

• device (str or torch.device) -- 设备

• args (tuple) -- 传递给替代函数的位置参数

• kwargs (dict) -- 传递给替代函数的关键字参数

返回:

无返回值，直接打印对比结果

返回类型:

None

Example

check_cuda_grad(
    neuron.IFNode, surrogate.S2NN, device="cuda:1", alpha=4.0, beta=1.0
)

---

• English

Check whether the CUDA (CuPy) backend gradient is correct, by comparing the gradient of CuPy backend and the PyTorch backend.

参数:

• neu (type) -- neuron class

• surrogate_function (type) -- surrogate function class (not instantiated)

• device (str or torch.device) -- device

• args (tuple) -- positional arguments to pass to the surrogate function

• kwargs (dict) -- keyword arguments to pass to the surrogate function

返回:

no return value, directly prints comparison results

返回类型:

None

Example

check_cuda_grad(
    neuron.IFNode, surrogate.S2NN, device="cuda:1", alpha=4.0, beta=1.0
)

spikingjelly.activation_based.surrogate.plot_surrogate_function(surrogate_function)

API Language: 中文 | English

---

• 中文

绘制替代函数的原函数和梯度图像。同时绘制Heaviside阶跃函数作为参考。

参数:

surrogate_function (SurrogateFunctionBase) -- 替代函数模块的实例

返回:

无返回值，直接保存并显示图像

返回类型:

None

---

• English

Plot the primitive function and gradient of the surrogate function. Also plots the Heaviside step function as a reference.

参数:

surrogate_function (SurrogateFunctionBase) -- an instance of surrogate function module

返回:

no return value, directly saves and shows the figure

返回类型:

None

class spikingjelly.activation_based.surrogate.SurrogateFunctionBase(spiking=True, **kwargs)

基类：Module

API Language: 中文 | English

---

• 中文

所有替代函数模块的基类。提供脉冲发放模式（spiking mode）和常规模式之间的切换， 以及CUDA代码生成的基础框架。

参数:

spiking (bool) -- 是否输出脉冲。默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。 若为 False 则不使用替代梯度，前向传播时使用原函数。

返回:

SurrogateFunctionBase 实例

返回类型:

SurrogateFunctionBase

---

• English

Base class for all surrogate function modules. Provides switching between spiking mode and regular mode, as well as the basic framework for CUDA code generation.

参数:

spiking (bool) -- whether to output spikes. Default is True which means using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, no surrogate gradient is used, and the primitive function is used in forward propagation.

返回:

a SurrogateFunctionBase instance

返回类型:

SurrogateFunctionBase

set_spiking_mode(spiking: bool)

extra_repr()

static spiking_function(x, **kwargs)

static primitive_function(x, **kwargs)

cuda_code(x: str, y: str, dtype='fp32')

cuda_code_start_comments()

cuda_code_end_comments()

forward(x: Tensor)

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.PiecewiseQuadratic(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用分段二次函数的梯度（三角形函数）的脉冲发放函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} 0, & |x| > \frac{1}{\alpha} \\ -\alpha^2|x|+\alpha, & |x| \leq \frac{1}{\alpha} \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} 0, & x < -\frac{1}{\alpha} \\ -\frac{1}{2}\alpha^2|x|x + \alpha x + \frac{1}{2}, & |x| \leq \frac{1}{\alpha} \\ 1, & x > \frac{1}{\alpha} \\ \end{cases}\end{split}\]

该函数在文章 [2] [4] [7] [11] [13] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。 若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The piecewise quadratic surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} 0, & |x| > \frac{1}{\alpha} \\ -\alpha^2|x|+\alpha, & |x| \leq \frac{1}{\alpha} \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} 0, & x < -\frac{1}{\alpha} \\ -\frac{1}{2}\alpha^2|x|x + \alpha x + \frac{1}{2}, & |x| \leq \frac{1}{\alpha} \\ 1, & x > \frac{1}{\alpha} \\ \end{cases}\end{split}\]

The function is used in [2] [4] [7] [11] [13].

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.PiecewiseExp(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用分段指数函数的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2}e^{-\alpha |x|}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}e^{\alpha x}, & x < 0 \\ 1 - \frac{1}{2}e^{-\alpha x}, & x \geq 0 \end{cases}\end{split}\]

该函数在文章 [6] [11] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The piecewise exponential surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2}e^{-\alpha |x|}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}e^{\alpha x}, & x < 0 \\ 1 - \frac{1}{2}e^{-\alpha x}, & x \geq 0 \end{cases}\end{split}\]

The function is used in [6] [11] .

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.Sigmoid(alpha=4.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用sigmoid的梯度的脉冲发放函数。反向传播为

\[g'(x) = \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x)\]

对应的原函数为

\[g(x) = \mathrm{sigmoid}(\alpha x) = \frac{1}{1+e^{-\alpha x}}\]

该函数在文章 [4] [12] [14] [15] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The sigmoid surrogate spiking function. The gradient is defined by

\[g'(x) = \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x)\]

The primitive function is defined by

\[g(x) = \mathrm{sigmoid}(\alpha x) = \frac{1}{1+e^{-\alpha x}}\]

The function is used in [4] [12] [14] [15] .

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x: Tensor, alpha: float)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32') → str

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.SoftSign(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用soft sign的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2(1 + |\alpha x|)^{2}} = \frac{1}{2\alpha(\frac{1}{\alpha} + |x|)^{2}}\]

对应的原函数为

\[g(x) = \frac{1}{2} (\frac{\alpha x}{1 + |\alpha x|} + 1) = \frac{1}{2} (\frac{x}{\frac{1}{\alpha} + |x|} + 1)\]

该函数在文章 [8] [11] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The soft sign surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2(1 + |\alpha x|)^{2}}\]

The primitive function is defined by

\[g(x) = \frac{1}{2} (\frac{\alpha x}{1 + |\alpha x|} + 1)\]

The function is used in [8] [11].

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.SuperSpike(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

SuperSpike: Supervised learning in multi-layer spiking neural networks 提出的反向传播时使用SuperSpike的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{(1 + (|x|))^{2}}\]

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The SuperSpike surrogate spiking function proposed by SuperSpike: Supervised learning in multi-layer spiking neural networks. The gradient is defined by

\[g'(x) = \frac{\alpha}{(1 + (|x|))^{2}}\]

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.ATan(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用反正切函数arc tangent的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{2(1 + (\frac{\pi}{2}\alpha x)^{2})}\]

对应的原函数为

\[g(x) = \frac{1}{\pi} \arctan(\frac{\pi}{2}\alpha x) + \frac{1}{2}\]

该函数在文章 [#Huh2018]_ [5] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The arc tangent surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{2(1 + (\frac{\pi}{2}\alpha x)^{2})}\]

The primitive function is defined by

\[g(x) = \frac{1}{\pi} \arctan(\frac{\pi}{2}\alpha x) + \frac{1}{2}\]

The function is used in [#Huh2018]_ [5].

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

heaviside(x) 或原函数的结果，形状与 x 相同

返回类型:

torch.Tensor

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.NonzeroSignLogAbs(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

警告

原函数的输出范围并不是(0, 1)。它的优势是反向传播的计算量特别小。

反向传播时使用NonzeroSignLogAbs的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{1 + |\alpha x|} = \frac{1}{\frac{1}{\alpha} + |x|}\]

对应的原函数为

\[g(x) = \mathrm{NonzeroSign}(x) \log (|\alpha x| + 1)\]

其中

\[\begin{split}\mathrm{NonzeroSign}(x) = \begin{cases} 1, & x \geq 0 \\ -1, & x < 0 \\ \end{cases}\end{split}\]

该函数在文章 [3] [4] [8] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

Warning

The output range the primitive function is not (0, 1). The advantage of this function is that computation cost is small when backward.

The NonzeroSignLogAbs surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{1 + |\alpha x|} = \frac{1}{\frac{1}{\alpha} + |x|}\]

The primitive function is defined by

\[g(x) = \mathrm{NonzeroSign}(x) \log (|\alpha x| + 1)\]

where

\[\begin{split}\mathrm{NonzeroSign}(x) = \begin{cases} 1, & x \geq 0 \\ -1, & x < 0 \\ \end{cases}\end{split}\]

The function is used in [3] [4] [8].

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x, alpha)

static backward(grad_output, x, alpha)

static primitive_function(x: Tensor, alpha: float)

class spikingjelly.activation_based.surrogate.Erf(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用高斯误差函数(erf)的梯度的脉冲发放函数。反向传播为

\[g'(x) = \frac{\alpha}{\sqrt{\pi}}e^{-\alpha^{2}x^{2}}\]

对应的原函数为

\begin{split} g(x) &= \frac{1}{2}(1-\text{erf}(-\alpha x)) \\ &= \frac{1}{2} \text{erfc}(-\alpha x) \\ &= \frac{1}{\sqrt{\pi}}\int_{-\infty}^{\alpha x}e^{-t^{2}}dt \end{split}

该函数在文章 [1] [4] [18] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的平滑程度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The Gaussian error (erf) surrogate spiking function. The gradient is defined by

\[g'(x) = \frac{\alpha}{\sqrt{\pi}}e^{-\alpha^{2}x^{2}}\]

The primitive function is defined by

\begin{split} g(x) &= \frac{1}{2}(1-\text{erf}(-\alpha x)) \\ &= \frac{1}{2} \text{erfc}(-\alpha x) \\ &= \frac{1}{\sqrt{\pi}}\int_{-\infty}^{\alpha x}e^{-t^{2}}dt \end{split}

The function is used in [1] [4] [18].

参数:

• alpha (float) -- parameter to control smoothness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.PiecewiseLeakyReLU(w=1.0, c=0.01, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

分段线性的近似脉冲发放函数。梯度为

\[\begin{split}g'(x) = \begin{cases} \frac{1}{2w}, & -w \leq x \leq w \\ c, & x < -w ~or~ x > w \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} cx + cw, & x < -w \\ \frac{1}{2w}x + \frac{1}{2}, & -w \leq x \leq w \\ cx - cw + 1, & x > w \\ \end{cases}\end{split}\]

该函数在文章 [3] [4] [5] [9] [10] [12] [16] [17] 中使用。

参数:

• w (float) -- -w <= x <= w 时反向传播的梯度为 1 / 2w

• c (float) -- x > w 或 x < -w 时反向传播的梯度为 c

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The piecewise surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \frac{1}{2w}, & -w \leq x \leq w \\ c, & x < -w ~or~ x > w \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} cx + cw, & x < -w \\ \frac{1}{2w}x + \frac{1}{2}, & -w \leq x \leq w \\ cx - cw + 1, & x > w \end{cases}\end{split}\]

The function is used in [3] [4] [5] [9] [10] [12] [16] [17].

参数:

• w (float) -- when -w <= x <= w the gradient is 1 / 2w

• c (float) -- when x > w or x < -w the gradient is c

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x: Tensor, w, c)

static backward(grad_output, x, w, c)

static primitive_function(x: Tensor, w: float, c: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.SquarewaveFourierSeries(n: int = 2, T_period: float = 8, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

使用傅里叶级数近似方波的脉冲发放函数。反向传播使用傅里叶级数展开计算梯度。

对应的原函数为

\[g(x) = \frac{1}{2} + \frac{2}{\pi} \sum_{i=1}^{n-1} \frac{\sin((2i-1)\omega x)}{2i-1}\]

其中 \(\omega = \frac{2\pi}{T}\)。

参数:

• n (int) -- 傅里叶级数的项数

• T_period (float) -- 方波的周期

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The square-wave Fourier series surrogate spiking function. The gradient is computed via Fourier series expansion.

The primitive function is defined by

\[g(x) = \frac{1}{2} + \frac{2}{\pi} \sum_{i=1}^{n-1} \frac{\sin((2i-1)\omega x)}{2i-1}\]

where \(\omega = \frac{2\pi}{T}\).

参数:

• n (int) -- number of terms in Fourier series

• T_period (float) -- period of the square wave

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x: Tensor, n, T_period)

static primitive_function(x: Tensor, n: int, T_period: float)

class spikingjelly.activation_based.surrogate.S2NN(alpha=4.0, beta=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

S2NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Neural Networks 提出的S2NN替代函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x), x < 0 \\ \frac{\beta}{(x + 1)}, x \ge 0 \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \mathrm{sigmoid} (\alpha x), x < 0 \\ \beta \mathrm{ln}(x + 1) + 1, x \ge 0 \end{cases}\end{split}\]

参数:

• alpha (float) -- 控制 x < 0 时梯度的参数

• beta (float) -- 控制 x >= 0 时梯度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The S2NN surrogate spiking function, which is proposed by S2NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Neural Networks. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \alpha * (1 - \mathrm{sigmoid} (\alpha x)) \mathrm{sigmoid} (\alpha x), x < 0 \\ \frac{\beta}{x + 1}, x \ge 0 \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \mathrm{sigmoid} (\alpha x), x < 0 \\ \beta \mathrm{ln}(x + 1) + 1, x \ge 0 \end{cases}\end{split}\]

参数:

• alpha (float) -- the param that controls the gradient when x < 0

• beta (float) -- the param that controls the gradient when x >= 0

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

forward(x)

static spiking_function(x: Tensor, alpha, beta)

static primitive_function(x: Tensor, alpha: float, beta: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.QPseudoSpike(alpha=2.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

Surrogate Gradients Design 提出的 \(q\)-PseudoSpike替代函数。反向传播为

\[g'(x) = (1+\frac{2|x|}{\alpha-1})^{-\alpha}\]

其中 \(\alpha>1\) 对应原文中的 \(q\)。

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}(1-\frac{2x}{\alpha-1})^{1-\alpha}, & x < 0 \\ 1 - \frac{1}{2}(1+\frac{2x}{\alpha-1})^{1-\alpha}, & x \geq 0. \end{cases}\end{split}\]

参数:

• alpha (float) -- 控制反向传播时梯度函数尾部厚度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The \(q\)-PseudoSpike surrogate spiking function, which is first proposed in Surrogate Gradients Design. The gradient is defined by

\[g'(x) = (1+\frac{2|x|}{\alpha-1})^{-\alpha}\]

where \(\alpha>1\) corresponds to \(q\) in paper.

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \frac{1}{2}(1-\frac{2x}{\alpha-1})^{1-\alpha}, & x < 0 \\ 1 - \frac{1}{2}(1+\frac{2x}{\alpha-1})^{1-\alpha}, & x \geq 0. \end{cases}\end{split}\]

参数:

• alpha (float) -- parameter to control tail fatness of gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.LeakyKReLU(leak: float = 0.0, k: float = 1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

反向传播时使用LeakyKReLU的梯度的脉冲发放函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} k, & x \geq 0 \\ leak, & x < 0 \\ \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} k \cdot x, & x \geq 0 \\ leak \cdot x, & x < 0 \\ \end{cases}\end{split}\]

该函数在文章 [3] [4] [8] 中使用。

参数:

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

• leak (float) -- x < 0 时的梯度值

• k (float) -- ``x >= 0 `` 时的梯度值

返回:

无返回值

返回类型:

None

---

• English

The LeakyKReLU surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} k, & x \geq 0 \\ leak, & x < 0 \\ \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} k \cdot x, & x \geq 0 \\ leak \cdot x, & x < 0 \\ \end{cases}\end{split}\]

The function is used in [3] [4] [8].

参数:

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

• leak (float) -- gradient when x < 0

• k (float) -- gradient when ``x >= 0 ``

返回:

No return value.

返回类型:

None

static spiking_function(x, leak, k)

static primitive_function(x: Tensor, leak: float, k: float)

static backward(grad_output, x, leak, k)

forward(x)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.FakeNumericalGradient(alpha=0.3)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

模拟数值梯度的脉冲发放函数，反向传播为

\[g'(x) = \mathrm{clip}(\frac{\mathrm{sign}(x)}{x}, \alpha)\]

参数:

alpha (float) -- 梯度裁剪阈值

返回:

无返回值

返回类型:

None

---

• English

The fake numerical gradient surrogate spiking function. The gradient is defined by

\[g'(x) = \mathrm{clip}(\frac{\mathrm{sign}(x)}{x}, \alpha)\]

参数:

alpha (float) -- gradient clip threshold

返回:

No return value.

返回类型:

None

static spiking_function(x, alpha)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.LogTailedReLU(alpha=0.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

Deep Learning with Low Precision by Half-wave Gaussian Quantization 提出的 Log-tailed ReLU替代函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} \alpha, & x \leq 0 \\ 1, & 0 < x \leq 0 \\ \frac{1}{x}, x > 1 \\ \end{cases}\end{split}\]

对应的原函数为

\[\begin{split}g(x) = \begin{cases} \alpha x, & x \leq 0 \\ x, & 0 < x \leq 0 \\ log(x), x > 1 \\ \end{cases}\end{split}\]

该函数在文章 [4] [5] [11] 中使用。

参数:

• alpha (float) -- 控制反向传播时梯度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

---

• English

The Log-tailed ReLU surrogate spiking function, which is first proposed in Deep Learning with Low Precision by Half-wave Gaussian Quantization. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \alpha, & x \leq 0 \\ 1, & 0 < x \leq 0 \\ \frac{1}{x}, x > 1 \\ \end{cases}\end{split}\]

The primitive function is defined by

\[\begin{split}g(x) = \begin{cases} \alpha x, & x \leq 0 \\ x, & 0 < x \leq 0 \\ log(x), x > 1 \\ \end{cases}\end{split}\]

The function is used in [4] [5] [11].

参数:

• alpha (float) -- parameter to control gradient

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

无返回值

返回类型:

None

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

cuda_code(x: str, y: str, dtype='fp32')

cuda_codes(y: str, x: str, dtype: str)

class spikingjelly.activation_based.surrogate.DeterministicPass(spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

直通估计器（Straight-Through Estimator, STE）替代函数。前向传播使用 heaviside 阶跃函数，反向传播直接传递梯度。

参数:

spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。 若为 False 则直接返回输入。

返回:

无返回值

返回类型:

None

---

• English

The straight-through estimator (STE) surrogate function. Forward uses heaviside step function, backward directly passes the gradient through.

参数:

spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, returns the input directly.

返回:

No return value.

返回类型:

None

forward(x: Tensor)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.PoissonPass(spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

泊松发放替代函数。前向传播使用伯努利采样生成随机脉冲（泊松过程），反向传播直接传递梯度。

参数:

spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用伯努利采样生成脉冲而在反向传播使用替代梯度。 若为 False 则直接返回输入。

返回:

无返回值

返回类型:

None

---

• English

The Poisson firing surrogate function. Forward uses Bernoulli sampling to generate stochastic spikes (Poisson process), backward directly passes the gradient through.

参数:

spiking (bool) -- whether output spikes. The default is True which means that using Bernoulli sampling in forward propagation and using surrogate gradient in backward propagation. If False, returns the input directly.

返回:

No return value.

返回类型:

None

forward(x: Tensor)

static backward(grad_output, x, alpha)

class spikingjelly.activation_based.surrogate.Rect(alpha=1.0, spiking=True)

基类：SurrogateFunctionBase

API Language: 中文 | English

---

• 中文

矩形窗替代函数。反向传播为

\[\begin{split}g'(x) = \begin{cases} \alpha, & |x| < \frac{0.5}{\alpha} \\\\ 0, & |x| \geq \frac{0.5}{\alpha} \\end{cases}\end{split}\]

对应的原函数为

\[g(x) = \mathrm{clip}(\alpha x + 0.5, 0, 1)\]

参数:

• alpha (float) -- 控制矩形窗口宽度的参数

• spiking (bool) -- 是否输出脉冲，默认为 True，在前向传播时使用 heaviside 而在反向传播使用替代梯度。若为 False 则不使用替代梯度，前向传播时，使用反向传播时的梯度替代函数对应的原函数

返回:

无返回值

返回类型:

None

---

• English

The rectangular window surrogate spiking function. The gradient is defined by

\[\begin{split}g'(x) = \begin{cases} \alpha, & |x| < \frac{0.5}{\alpha} \\\\ 0, & |x| \geq \frac{0.5}{\alpha} \\end{cases}\end{split}\]

The primitive function is defined by

\[g(x) = \mathrm{clip}(\alpha x + 0.5, 0, 1)\]

参数:

• alpha (float) -- parameter to control the width of the rectangular window

• spiking (bool) -- whether output spikes. The default is True which means that using heaviside in forward propagation and using surrogate gradient in backward propagation. If False, in forward propagation, using the primitive function of the surrogate gradient function used in backward propagation

返回:

No return value.

返回类型:

None

static spiking_function(x, alpha)

static primitive_function(x: Tensor, alpha: float)

static backward(grad_output, x, alpha)

References

[1] (1,2)

Esser S K, Appuswamy R, Merolla P, et al. Backpropagation for energy-efficient neuromorphic computing[J]. Advances in neural information processing systems, 2015, 28: 1117-1125.

[2] (1,2)

Esser S K, Merolla P A, Arthur J V, et al. Convolutional networks for fast, energy-efficient neuromorphic computing[J]. Proceedings of the national academy of sciences, 2016, 113(41): 11441-11446.

[3] (1,2,3,4,5,6)

Yin S, Venkataramanaiah S K, Chen G K, et al. Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations[C]//2017 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, 2017: 1-5.

[4] (1,2,3,4,5,6,7,8,9,10,11,12,13,14)

Wu Y, Deng L, Li G, et al. Spatio-temporal backpropagation for training high-performance spiking neural networks[J]. Frontiers in neuroscience, 2018, 12: 331.

[5] (1,2,3,4,5,6)

Huh D, Sejnowski T J. Gradient descent for spiking neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 1440-1450.

[6] (1,2)

Shrestha S B, Orchard G. SLAYER: spike layer error reassignment in time[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 1419-1428.

[7] (1,2)

Bellec G, Salaj D, Subramoney A, et al. Long short-term memory and learning-to-learn in networks of spiking neurons[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018: 795-805.

[8] (1,2,3,4,5,6)

Zenke F, Ganguli S. Superspike: Supervised learning in multilayer spiking neural networks[J]. Neural computation, 2018, 30(6): 1514-1541.

[9] (1,2)

Wu Y, Deng L, Li G, et al. Direct training for spiking neural networks: Faster, larger, better[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33(01): 1311-1318.

[10] (1,2)

Gu P, Xiao R, Pan G, et al. STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks[C]//IJCAI. 2019: 1366-1372.

[11] (1,2,3,4,5,6,7,8)

Neftci E O, Mostafa H, Zenke F. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[J]. IEEE Signal Processing Magazine, 2019, 36(6): 51-63.

[12] (1,2,3,4)

Roy D, Chakraborty I, Roy K. Scaling deep spiking neural networks with binary stochastic activations[C]//2019 IEEE International Conference on Cognitive Computing (ICCC). IEEE, 2019: 50-58.

[13] (1,2)

Panda P, Aketi S A, Roy K. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization[J]. Frontiers in Neuroscience, 2020, 14.

[14] (1,2)

Lotfi Rezaabad A, Vishwanath S. Long Short-Term Memory Spiking Networks and Their Applications[C]//International Conference on Neuromorphic Systems 2020. 2020: 1-9.

[15] (1,2)

Woźniak S, Pantazi A, Bohnstingl T, et al. Deep learning incorporating biologically inspired neural dynamics and in-memory computing[J]. Nature Machine Intelligence, 2020, 2(6): 325-336.

[16] (1,2)

Cheng X, Hao Y, Xu J, et al. LISNN: Improving Spiking Neural Networks with Lateral Interactions for Robust Object Recognition[C]//IJCAI. 1519-1525.

[17] (1,2)

Kaiser J, Mostafa H, Neftci E. Synaptic plasticity dynamics for deep continuous local learning (DECOLLE)[J]. Frontiers in Neuroscience, 2020, 14: 424.

[18] (1,2)

Yin B, Corradi F, Bohté S M. Effective and efficient computation with multiple-timescale spiking recurrent neural networks[C]//International Conference on Neuromorphic Systems 2020. 2020: 1-8.