目錄

1.Transformer宏觀結(jié)構(gòu)

2.Transformer結(jié)構(gòu)細(xì)節(jié)

2.1輸入

2.2編碼部分

2.3解碼部分

2.4多頭注意力機(jī)制

2.5線性層和softmax

2.6?損失函數(shù)

3.參考代碼

---

🍨 本文為[🔗365天深度學(xué)習(xí)訓(xùn)練營(yíng)]內(nèi)部限免文章（版權(quán)歸 *K同學(xué)啊* 所有）
🍖 作者：[K同學(xué)啊]

Transformer整體結(jié)構(gòu)圖，與seq2seq模型類似，Transformer模型結(jié)構(gòu)中的左半部分為編碼器（encoder），右半部分為解碼器（decoder），接下來(lái)拆解Transformer。

1.Transformer宏觀結(jié)構(gòu)

Transformer模型類似于seq2seq結(jié)構(gòu)，包含編碼部分和解碼部分。不同之處在于它能夠并行計(jì)算整個(gè)序列輸入，無(wú)需按時(shí)間步進(jìn)行逐步處理。

其宏觀結(jié)構(gòu)如下：

其中，每層encoder由兩部分組成：

• Self-Attention Layer
• Feed Forward Neural Network（前饋神經(jīng)網(wǎng)絡(luò)，FFNN）

decoder在encoder的Self-Attention和FFNN中間多加了一個(gè)Encoder-Decoder Attention層。該層的作用是幫助解碼器集中注意力于輸入序列中最相關(guān)的部分。

2.Transformer結(jié)構(gòu)細(xì)節(jié)

2.1輸入

Transformer的數(shù)據(jù)輸入與seq2seq不同。除了詞向量，Transformer還需要輸入位置向量，用于確定每個(gè)單詞的位置特征和句子中不同單詞之間的距離特征。

2.2編碼部分

編碼部分的輸入文本序列經(jīng)過(guò)處理后得到向量序列，送入第一層編碼器。每層編碼器輸出一個(gè)向量序列，作為下一層編碼器的輸入。第一層編碼器的輸入是融合位置向量的詞向量，后續(xù)每層編碼器的輸入則是前一層編碼器的輸出。

2.3解碼部分

最后一個(gè)編碼器輸出一組序列向量，作為解碼器的K、V輸入。

解碼階段的每個(gè)時(shí)間步輸出一個(gè)翻譯后的單詞。當(dāng)前時(shí)間步的解碼器輸出作為下一個(gè)時(shí)間步解碼器的輸入Q，與編碼器的輸出K、V共同組成下一步的輸入。重復(fù)此過(guò)程直到輸出一個(gè)結(jié)束符。

解碼器中的 Self-Attention 層，和編碼器中的 Self-Attention 層的區(qū)別：

• 在解碼器里，Self-Attention 層只允許關(guān)注到輸出序列中早于當(dāng)前位置之前的單詞。具體做法是：在 Self-Attention 分?jǐn)?shù)經(jīng)過(guò) Softmax 層之前，屏蔽當(dāng)前位置之后的那些位置（將Attention Score設(shè)置成-inf）。
• 解碼器 Attention層是使用前一層的輸出來(lái)構(gòu)造Query 矩陣，而Key矩陣和Value矩陣來(lái)自于編碼器最終的輸出。

2.4多頭注意力機(jī)制

Transformer論文引入了多頭注意力機(jī)制（多個(gè)注意力頭組成），以進(jìn)一步完善Self-Attention。

• 它擴(kuò)展了模型關(guān)注不同位置的能力
• 多頭注意力機(jī)制賦予Attention層多個(gè)“子表示空間”。

殘差鏈接&Normalize：?編碼器和解碼器的每個(gè)子層（Self-Attention 層和 FFNN）都有一個(gè)殘差連接和層標(biāo)準(zhǔn)化（layer-normalization），細(xì)節(jié)如下圖

2.5線性層和softmax

Decoder最終輸出一個(gè)浮點(diǎn)數(shù)向量。通過(guò)線性層和Softmax，將該向量轉(zhuǎn)換為一個(gè)包含模型輸出詞匯表中每個(gè)單詞分?jǐn)?shù)的logits向量（假設(shè)有10000個(gè)英語(yǔ)單詞）。Softmax將這些分?jǐn)?shù)轉(zhuǎn)換為概率，使其總和為1。然后選擇具有最高概率的數(shù)字對(duì)應(yīng)的詞作為該時(shí)間步的輸出單詞。

2.6?損失函數(shù)

在Transformer訓(xùn)練過(guò)程中，解碼器的輸出和標(biāo)簽一起輸入損失函數(shù)，以計(jì)算損失（loss）。最終，模型通過(guò)方向傳播（backpropagation）來(lái)優(yōu)化損失。

3.參考代碼

class SelfAttention(nn.Module):def __init__(self, embed_size, heads):super(SelfAttention, self).__init__()self.embed_size = embed_sizeself.heads = headsself.head_dim = embed_size // headsassert (self.head_dim * heads == embed_size), "Embed size needs to be div by heads"self.values = nn.Linear(self.head_dim, self.head_dim, bias=False)self.keys = nn.Linear(self.head_dim, self.head_dim, bias=False)self.queries = nn.Linear(self.head_dim, self.head_dim, bias=False)self.fc_out = nn.Linear(heads * self.head_dim, embed_size)def forward(self, values, keys, query, mask):N =query.shape[0]value_len , key_len , query_len = values.shape[1], keys.shape[1], query.shape[1]# split embedding into self.heads piecesvalues = values.reshape(N, value_len, self.heads, self.head_dim)keys = keys.reshape(N, key_len, self.heads, self.head_dim)queries = query.reshape(N, query_len, self.heads, self.head_dim)values = self.values(values)keys = self.keys(keys)queries = self.queries(queries)energy = torch.einsum("nqhd,nkhd->nhqk", queries, keys)# queries shape: (N, query_len, heads, heads_dim)# keys shape : (N, key_len, heads, heads_dim)# energy shape: (N, heads, query_len, key_len)if mask is not None:energy = energy.masked_fill(mask == 0, float("-1e20"))attention = torch.softmax(energy/ (self.embed_size ** (1/2)), dim=3)out = torch.einsum("nhql, nlhd->nqhd", [attention, values]).reshape(N, query_len, self.heads*self.head_dim)# attention shape: (N, heads, query_len, key_len)# values shape: (N, value_len, heads, heads_dim)# (N, query_len, heads, head_dim)out = self.fc_out(out)return outclass TransformerBlock(nn.Module):def __init__(self, embed_size, heads, dropout, forward_expansion):super(TransformerBlock, self).__init__()self.attention = SelfAttention(embed_size, heads)self.norm1 = nn.LayerNorm(embed_size)self.norm2 = nn.LayerNorm(embed_size)self.feed_forward = nn.Sequential(nn.Linear(embed_size, forward_expansion*embed_size),nn.ReLU(),nn.Linear(forward_expansion*embed_size, embed_size))self.dropout = nn.Dropout(dropout)def forward(self, value, key, query, mask):attention = self.attention(value, key, query, mask)x = self.dropout(self.norm1(attention + query))forward = self.feed_forward(x)out = self.dropout(self.norm2(forward + x))return outclass Encoder(nn.Module):def __init__(self,src_vocab_size,embed_size,num_layers,heads,device,forward_expansion,dropout,max_length,):super(Encoder, self).__init__()self.embed_size = embed_sizeself.device = deviceself.word_embedding = nn.Embedding(src_vocab_size, embed_size)self.position_embedding = nn.Embedding(max_length, embed_size)self.layers = nn.ModuleList([TransformerBlock(embed_size,heads,dropout=dropout,forward_expansion=forward_expansion,)for _ in range(num_layers)])self.dropout = nn.Dropout(dropout)def forward(self, x, mask):N, seq_length = x.shapepositions = torch.arange(0, seq_length).expand(N, seq_length).to(self.device)out = self.dropout(self.word_embedding(x) + self.position_embedding(positions))for layer in self.layers:out = layer(out, out, out, mask)return outclass DecoderBlock(nn.Module):def __init__(self, embed_size, heads, forward_expansion, dropout, device):super(DecoderBlock, self).__init__()self.attention = SelfAttention(embed_size, heads)self.norm = nn.LayerNorm(embed_size)self.transformer_block = TransformerBlock(embed_size, heads, dropout, forward_expansion)self.dropout = nn.Dropout(dropout)def forward(self, x, value, key, src_mask, trg_mask):attention = self.attention(x, x, x, trg_mask)query = self.dropout(self.norm(attention + x))out = self.transformer_block(value, key, query, src_mask)return outclass Decoder(nn.Module):def __init__(self,trg_vocab_size,embed_size,num_layers,heads,forward_expansion,dropout,device,max_length,):super(Decoder, self).__init__()self.device = deviceself.word_embedding = nn.Embedding(trg_vocab_size, embed_size)self.position_embedding = nn.Embedding(max_length, embed_size)self.layers = nn.ModuleList([DecoderBlock(embed_size, heads, forward_expansion, dropout, device)for _ in range(num_layers)])self.fc_out = nn.Linear(embed_size, trg_vocab_size)self.dropout = nn.Dropout(dropout)def forward(self, x ,enc_out , src_mask, trg_mask):N, seq_length = x.shapepositions = torch.arange(0, seq_length).expand(N, seq_length).to(self.device)x = self.dropout((self.word_embedding(x) + self.position_embedding(positions)))for layer in self.layers:x = layer(x, enc_out, enc_out, src_mask, trg_mask)out =self.fc_out(x)return outclass Transformer(nn.Module):def __init__(self,src_vocab_size,trg_vocab_size,src_pad_idx,trg_pad_idx,embed_size = 256,num_layers = 6,forward_expansion = 4,heads = 8,dropout = 0,device="cuda",max_length=100):super(Transformer, self).__init__()self.encoder = Encoder(src_vocab_size,embed_size,num_layers,heads,device,forward_expansion,dropout,max_length)self.decoder = Decoder(trg_vocab_size,embed_size,num_layers,heads,forward_expansion,dropout,device,max_length)self.src_pad_idx = src_pad_idxself.trg_pad_idx = trg_pad_idxself.device = devicedef make_src_mask(self, src):src_mask = (src != self.src_pad_idx).unsqueeze(1).unsqueeze(2)# (N, 1, 1, src_len)return src_mask.to(self.device)def make_trg_mask(self, trg):N, trg_len = trg.shapetrg_mask = torch.tril(torch.ones((trg_len, trg_len))).expand(N, 1, trg_len, trg_len)return trg_mask.to(self.device)def forward(self, src, trg):src_mask = self.make_src_mask(src)trg_mask = self.make_trg_mask(trg)enc_src = self.encoder(src, src_mask)out = self.decoder(trg, enc_src, src_mask, trg_mask)return out