介紹

神經(jīng)網(wǎng)絡(luò)當(dāng)中的激活函數(shù)用來提升網(wǎng)絡(luò)的非線性，以增強網(wǎng)絡(luò)的表征能力。它有這樣幾個特點：有界，必須為非常數(shù)，單調(diào)遞增且連續(xù)可求導(dǎo)。我們常用的有sigmoid或者tanh，但我們都知道這兩個都存在一定的缺點，有的甚至是無腦用Relu。所以今天就來學(xué)習(xí)并實現(xiàn)一些其他的激活函數(shù)。

下面激活函數(shù)使用的圖像都是可以通過這個腳本就行修改：

import torch
import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10, 10, 60)
y = F.silu(x)plt.plot(x.numpy(), y.numpy())
plt.title('Silu Activation Function')
plt.xlabel('x')
plt.ylabel('silu(x)')
plt.grid()
plt.tight_layout()
plt.show()

SiLU

import torch
import torch.nn as nn
import torch.nn.functional as Fclass SiLU(nn.Module):@staticmethoddef forward(x):return x * torch.sigmoid(x)if __name__=="__main__":m = nn.SiLU()input = torch.randn(2)output = m(input)print("官方實現(xiàn):",output)n = SiLU()output = n(input)print("自定義:",output)

官方實現(xiàn): tensor([ 0.2838, -0.2578])
自定義: tensor([ 0.2838, -0.2578])

Mish

import torch
import torch.nn as nn
import torch.nn.functional as Fclass Mish(nn.Module):@staticmethoddef forward(x):return x * F.softplus(x).tanh()if __name__=="__main__":m = nn.Mish()input = torch.randn(2)output = m(input)print("官方實現(xiàn):",output)n = Mish()output = n(input)print("自定義:",output)

官方實現(xiàn): tensor([2.8559, 0.2204])
自定義: tensor([2.8559, 0.2204])

Hard-SiLU

import torch
import torch.nn as nn
import torch.nn.functional as Fclass Hardswish(nn.Module):# Hard-SiLU activation https://arxiv.org/abs/1905.02244@staticmethoddef forward(x):return x * F.hardtanh(x + 3, 0.0, 6.0) / 6.0if __name__=="__main__":m = nn.Hardswish()input = torch.randn(2)output = m(input)print("官方實現(xiàn):",output)n = Hardswish()output = n(input)print("自定義:",output)

官方實現(xiàn): tensor([-0.1857, -0.0061])
自定義: tensor([-0.1857, -0.0061])