Loss Functions#

适用于 SNN 的 损失函数 实现。

Loss functions suitable for SNNs.

spikingjelly.activation_based.functional.loss.kernel_dot_product(x: Tensor, y: Tensor, kernel='linear', *args)[源代码]#

API Language: 中文 | English

中文

计算批量数据 x 和 y 在核空间的内积。记2个M维tensor分别为 \(\boldsymbol{x_{i}}\) 和 \(\boldsymbol{y_{j}}\) ， kernel 定义了不同形式的内积：

linear ，线性内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}}\)。
polynomial ，多项式内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = (\boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})^{d}\)，其中 \(d = args[0]\)。
sigmoid ，Sigmoid内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{sigmoid}(\alpha \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})\)，其中 \(\alpha = args[0]\)。
gaussian ，高斯内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{exp}(- \frac{||\boldsymbol{x_{i}} - \boldsymbol{y_{j}}||^{2}}{2\sigma^{2}})\) ，其中 \(\sigma = args[0]\)。

参数:

x (torch.Tensor) -- shape=[N, M]的tensor，看作是N个M维向量
y (torch.Tensor) -- shape=[N, M]的tensor，看作是N个M维向量
kernel (str) -- 计算内积时所使用的核函数
args -- 用于计算内积的额外的参数

返回:

ret, shape=[N, N]的tensor， ret[i][j] 表示 x[i] 和 y[j] 的内积

返回类型:

torch.Tensor

English

Calculate inner product of x and y in kernel space. These 2 M-dim tensors are denoted by \(\boldsymbol{x_{i}}\) and \(\boldsymbol{y_{j}}\). kernel determine the kind of inner product:

linear -- Linear kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}}\).
polynomial -- Polynomial kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = (\boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})^{d}\), where \(d = args[0]\).
sigmoid -- Sigmoid kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{sigmoid}(\alpha \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})\), where \(\alpha = args[0]\).
gaussian -- Gaussian kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{exp}(- \frac{||\boldsymbol{x_{i}} - \boldsymbol{y_{j}}||^{2}}{2\sigma^{2}})\), where \(\sigma = args[0]\).

参数:

x (torch.Tensor) -- Tensor of shape=[N, M]
y (torch.Tensor) -- Tensor of shape=[N, M]
kernel (str) -- Type of kernel function used when calculating inner products.
args -- Extra parameters for inner product

返回:

ret, Tensor of shape=[N, N], ret[i][j] is inner product of x[i] and y[j].

返回类型:

torch.Tensor

spikingjelly.activation_based.functional.loss.spike_similar_loss(spikes: Tensor, labels: Tensor, kernel_type='linear', loss_type='mse', *args)[源代码]#

API Language: 中文 | English

中文

将N个数据输入到输出层有M个神经元的SNN，运行T步，得到shape=[N, M, T]的脉冲。这N个数据的标签为shape=[N, C]的 labels。

用shape=[N, N]的矩阵 sim 表示 实际相似度矩阵， sim[i][j] == 1 表示数据i与数据j相似，反之亦然。若 labels[i] 与 labels[j] 共享至少同一个标签，则认为他们相似，否则不相似。

用shape=[N, N]的矩阵 sim_p 表示 输出相似度矩阵 ， sim_p[i][j] 的取值为0到1，值越大表示数据i与数据j的脉冲越相似。

使用内积来衡量两个脉冲之间的相似性， kernel_type 是计算内积时，所使用的核函数种类：

linear ，线性内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}}\)。
sigmoid ，Sigmoid内积， \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{sigmoid}(\alpha \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})\)，其中 \(\alpha = args[0]\)。
gaussian ，高斯内积，\(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{exp}(- \frac{||\boldsymbol{x_{i}} - \boldsymbol{y_{j}}||^{2}}{2\sigma^{2}})\)，其中 \(\sigma = args[0]\)。

当使用Sigmoid或高斯内积时，内积的取值范围均在[0, 1]之间；而使用线性内积时，为了保证内积取值仍然在[0, 1]之间，会进行归一化，按照 \(\text{sim}_p[i][j]=\frac{\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}})}{||\boldsymbol{x_{i}}|| · ||\boldsymbol{y_{j}}||}\)。

对于相似的数据，根据输入的 loss_type ，返回度量 sim 与 sim_p 差异的损失：

mse -- 返回sim与sim_p的均方误差（也就是l2误差）。
l1 -- 返回sim与sim_p的l1误差。
bce -- 返回sim与sim_p的二值交叉熵误差。

备注

脉冲向量稀疏、离散，最好先使用高斯核进行平滑，然后再计算相似度。

参数:

spikes (torch.Tensor) -- shape=[N, M, T]，N个数据生成的脉冲
labels (torch.Tensor) -- shape=[N, C]，N个数据的标签， labels[i][k] == 1 表示数据i属于第k类，反之亦然，允许多标签
kernel_type (str) -- 使用内积来衡量两个脉冲之间的相似性， kernel_type 是计算内积时，所使用的核函数种类
loss_type (str) -- 返回哪种损失，可以为'mse', 'l1', 'bce'
args -- 用于计算内积的额外参数

返回:

shape=[1]的tensor，相似损失

返回类型:

torch.Tensor

English

A SNN consisting M neurons will receive a batch of N input data in each timestep (from 0 to T-1) and output a spike tensor of shape=[N, M, T]. The label is a tensor of shape=[N, C].

The groundtruth similarity matrix sim has a shape of [N, N]. sim[i][j] == 1 indicates that input i is similar to input j and vice versa. If and only if labels[i] and labels[j] have at least one common label, they are viewed as similar.

The output similarity matrix sim_p has a shape of [N, N]. The value of sim_p[i][j] ranges from 0 to 1, represents the similarity between output spike from both input i and input j.

The similarity is measured by inner product of two spikes. kernel_type is the type of kernel function when calculating inner product:

linear, Linear kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}}\).
sigmoid, Sigmoid kernel, \(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{sigmoid}(\alpha \boldsymbol{x_{i}}^{T}\boldsymbol{y_{j}})\), where \(\alpha = args[0]\).
gaussian, Gaussian kernel，\(\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}}) = \mathrm{exp}(- \frac{||\boldsymbol{x_{i}} - \boldsymbol{y_{j}}||^{2}}{2\sigma^{2}})\), where \(\sigma = args[0]\).

When Sigmoid or Gaussian kernel is applied, the inner product naturally lies in \([0, 1]\). To make the value consistent when using linear kernel, the result will be normalized as: \(\text{sim}_p[i][j]=\frac{\kappa(\boldsymbol{x_{i}}, \boldsymbol{y_{j}})}{||\boldsymbol{x_{i}}|| · ||\boldsymbol{y_{j}}||}\).

For similar data, return the specified discrepancy loss between sim and sim_p according to loss_type.

mse -- Return the Mean-Square Error (squared L2 norm) between sim and sim_p.
l1 -- Return the L1 error between sim and sim_p.
bce -- Return the Binary Cross Entropy between sim and sim_p.

Note

Since spike vectors are usually discrete and sparse, it would be better to apply Gaussian filter first to smooth the vectors before calculating similarities.

参数:

spikes (torch.Tensor) -- shape=[N, M, T], output spikes corresponding to a batch of N inputs
labels (torch.Tensor) -- shape=[N, C], labels of inputs, labels[i][k] == 1 means the i-th input belongs to the k-th category and vice versa. Multi-label input is allowed.
kernel_type (str) -- Type of kernel function used when calculating inner products. The inner product is the similarity measure of two spikes.
loss_type (str) -- Type of loss returned. Can be: 'mse', 'l1', 'bce'
args -- Extra parameters for inner product

返回:

shape=[1], similarity loss

返回类型:

torch.Tensor

spikingjelly.activation_based.functional.loss.temporal_efficient_training_cross_entropy(x_seq: Tensor, target: Tensor)[源代码]#

API Language: 中文 | English

中文

Temporal efficient training (TET) 交叉熵损失, 是每个时间步的交叉熵损失的平均。

备注

TET交叉熵是 Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting 一文提出的。

参数:

x_seq (torch.Tensor) -- shape=[T, N, C, *] 的预测值，其中 C 是类别总数
target (torch.Tensor) -- shape=[N] 的真实值，其中 target[i] 是真实类别

返回:

损失值

返回类型:

torch.Tensor

English

The temporal efficient training (TET) cross entropy, which is the mean of cross entropy of each time-step.

Note

The TET cross entropy is proposed by Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting.

参数:

x_seq (torch.Tensor) -- the predicted value with shape=[T, N, C, *], where C is the number of classes
target (torch.Tensor) -- the ground truth tensor with shape=[N], where target[i] is the label

返回:

the temporal efficient training cross entropy

返回类型:

torch.Tensor

示例代码 | Example

def tet_ce_for_loop_version(x_seq: torch.Tensor, target: torch.LongTensor):
    loss = 0.0
    for t in range(x_seq.shape[0]):
        loss += F.cross_entropy(x_seq[t], target)
    return loss / x_seq.shape[0]


T = 8
N = 4
C = 10
x_seq = torch.rand([T, N, C])
target = torch.randint(low=0, high=C - 1, size=[N])
print(
    f"max error = {(tet_ce_for_loop_version(x_seq, target) - temporal_efficient_training_cross_entropy(x_seq, target)).abs().max()}"
)
# max error < 1e-6