transformer是一种不同于RNN的架构，模型同样包含 encoder 和 decoder ，但是encoder 和 decoder 抛弃 了RNN，而使用各种前馈层堆叠在一起。
Encoder：
编码器是由N个完全一样的层堆叠起来的，每层又包括两个子层(sub-layer)，第一个子层是multi-head self-attention mechanism层，第二个子层是一个简单的多层全连接层(fully connected feed-forward network)
Decoder：
解码器也是由N 个相同层的堆叠起来的。 但每层包括三个子层(sub-layer)，第一个子层是multi-head self-attention层，第二个子层是multi-head context-attention 层，第三个子层是一个简单的多层全连接层(fully connected feed-forward network)
模型的架构如下
一 module
（1）multi-head self-attention
multi-head self-attention是key=value=query=隐层的注意力机制
Encoder的multi-head self-attention是key=value=query=编码层隐层的注意力机制
Decoder的multi-head self-attention是key=value=query=解码层隐层的注意力机制
这里介绍自注意力机制（self-attention）也就是key=value=query=H的情况下的输出
隐层所有时间序列的状态H，$h_{i}$代表第i个词对应的隐藏层状态
[H = left[ egin{array}{l}
{h_1}\
{h_2}\
...\
{h_n}
end{array}
ight] in {R^{n imes dim }}{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {h_i} in {R^{1 imes dim }}]
H的转置为
[{H^T} = [h_1^T,h_2^T,...,h_n^T] in {R^{dim imes n}}{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {h_i} in {R^{1 imes dim }}]
如果只计算一个单词对应的隐层状态$h_{i}$的self-attention
[egin{array}{l}
weigh{t_{_{{h_i}}}}{
m{ = softmax}}(({h_i}W_{query}^i)*(W_{key}^i*left[ {egin{array}{*{20}{c}}
{h_1^T}&{h_2^T}&{...}&{h_n^T}
end{array}}
ight])) = left[ {egin{array}{*{20}{c}}
{weigh{t_{i1}},}&{weigh{t_{i2}},}&{...}&{weigh{t_{in}}}
end{array}}
ight]\
{
m{value = }}left[ {egin{array}{*{20}{l}}
{{h_1}}\
{{h_2}}\
{...}\
{{h_n}}
end{array}}
ight]*W_{value}^i = left[ {egin{array}{*{20}{l}}
{{h_1}W_{value}^i}\
{{h_2}W_{value}^i}\
{...}\
{{h_n}W_{value}^i}
end{array}}
ight]\
Attentio{n_{{h_i}}} = weigh{t_{_{{h_i}}}}*value = sumlimits_{k = 1}^n {({h_k}W_{value}^i} )(weigh{t_{ik}})
end{array}]
同理，一次性计算所有单词隐层状态$h_{i}(1<=i<=n)$的self-attention
[egin{array}{l}
{
m{weight}}{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{ = softmax}}(left[ {egin{array}{*{20}{l}}
{{h_1}}\
{{h_2}}\
{...}\
{{h_n}}
end{array}}
ight]W_{query}^i*(W_{key}^i*left[ {egin{array}{*{20}{c}}
{h_1^T}&{h_2^T}&{...}&{h_n^T}
end{array}}
ight])\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{ = softmax}}(left[ {egin{array}{*{20}{l}}
{{h_1}W_{query}^i}\
{{h_2}W_{query}^i}\
{...}\
{{h_n}W_{query}^i}
end{array}}
ight]*left[ {egin{array}{*{20}{c}}
{W_{key}^ih_1^T}&{W_{key}^ih_2^T}&{...}&{W_{key}^ih_n^T}
end{array}}
ight]\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{ = softmax}}(left[ {egin{array}{*{20}{c}}
{({h_1}W_{query}^i)(W_{key}^ih_1^T)}&{({h_1}W_{query}^i)(W_{key}^ih_2^T)}&{...}&{({h_1}W_{query}^i)(W_{key}^ih_n^T)}\
{({h_2}W_{query}^i)(W_{key}^ih_1^T)}&{({h_2}W_{query}^i)(W_{key}^ih_2^T)}&{...}&{({h_2}W_{query}^i)(W_{key}^ih_n^T)}\
{...}&{...}&{...}&{...}\
{({h_n}W_{query}^i)(W_{key}^ih_1^T)}&{({h_n}W_{query}^i)(W_{key}^ih_2^T)}&{...}&{({h_n}W_{query}^i)(W_{key}^ih_n^T)}
end{array}}
ight])\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{ = }}left[ {egin{array}{*{20}{c}}
{{
m{softmax}}(({h_1}W_{query}^iW_{key}^ih_1^T)}&{({h_1}W_{query}^iW_{key}^ih_2^T)}&{...}&{({h_1}W_{query}^iW_{key}^ih_n^T))}\
{{
m{softmax}}(({h_2}W_{query}^iW_{key}^ih_1^T)}&{({h_2}W_{query}^iW_{key}^ih_2^T)}&{...}&{({h_2}W_{query}^iW_{key}^ih_n^T))}\
{...}&{...}&{...}&{...}\
{{
m{softmax}}(({h_n}W_{query}^iW_{key}^ih_1^T)}&{({h_n}W_{query}^iW_{key}^ih_2^T)}&{...}&{({h_n}W_{query}^iW_{key}^ih_n^T))}
end{array}}
ight]
end{array}]
[egin{array}{l}
{
m{sum}}(weight*value) = left[ egin{array}{l}
{
m{Weigh}}{{
m{t}}_{11}}({h_1}W_{value}^i) + {
m{Weigh}}{{
m{t}}_{12}}({h_2}W_{value}^i) + ...{kern 1pt} {kern 1pt} + {
m{Weigh}}{{
m{t}}_{1n}}({h_n}W_{value}^i)\
{
m{Weigh}}{{
m{t}}_{21}}({h_1}W_{value}^i) + {
m{Weigh}}{{
m{t}}_{22}}({h_2}W_{value}^i) + ... + {
m{Weigh}}{{
m{t}}_{2n}}({h_n}W_{value}^i)\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} ......\
{
m{Weigh}}{{
m{t}}_{n1}}({h_1}W_{value}^i) + {
m{Weigh}}{{
m{t}}_{n2}}({h_2}W_{value}^i) + ... + {
m{Weigh}}{{
m{t}}_{nn}}({h_n}W_{value}^i)
end{array}
ight]\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} = {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} left[ egin{array}{l}
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{1k}}({h_k}W_{value}^i)} \
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{2k}}({h_k}W_{value}^i)} \
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} .......\
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{nk}}({h_k}W_{value}^i)}
end{array}
ight]
end{array}]
所以最后的注意力向量为$head_{i}$
[egin{array}{l}
hea{d_i} = Attention(QW_{query}^i,KW_{query}^i,VW_{query}^i)\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{ = }}{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{sum}}(weight*value)\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} = {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} left[ egin{array}{l}
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{1k}}({h_k}W_{value}^i)} \
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{2k}}({h_k}W_{value}^i)} \
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} .......\
sumlimits_{k = 1}^n {{
m{Weigh}}{{
m{t}}_{nk}}({h_k}W_{value}^i)}
end{array}
ight]
end{array}]
softmax函数需要加一个平滑系数$ sqrt {{{ m{d}}_k}} $
[egin{array}{l}
hea{d_i} = Attention(QW_{query}^i,KW_{key}^i,VW_{value}^i)\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} = {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {
m{softmax}}(frac{{(QW_{query}^i){{(KW_{key}^i)}^T}}}{{sqrt {{d_k}} }})VW_{value}^i\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} = {
m{softmax}}(left[ {egin{array}{*{20}{c}}
{frac{{({h_1}W_{query}^i)(W_{key}^ih_1^T)}}{{sqrt {{d_k}} }}}&{frac{{({h_1}W_{query}^i)(W_{key}^ih_2^T)}}{{sqrt {{d_k}} }}}&{...}&{frac{{({h_1}W_{query}^i)(W_{key}^ih_n^T)}}{{sqrt {{d_k}} }}}\
{frac{{({h_2}W_{query}^i)(W_{key}^ih_1^T)}}{{sqrt {{d_k}} }}}&{frac{{({h_2}W_{query}^i)(W_{key}^ih_2^T)}}{{sqrt {{d_k}} }}}&{...}&{frac{{({h_2}W_{query}^i)(W_{key}^ih_n^T)}}{{sqrt {{d_k}} }}}\
{...}&{...}&{...}&{...}\
{frac{{({h_n}W_{query}^i)(W_{key}^ih_1^T)}}{{sqrt {{d_k}} }}}&{frac{{({h_n}W_{query}^i)(W_{key}^ih_2^T)}}{{sqrt {{d_k}} }}}&{...}&{frac{{({h_n}W_{query}^i)(W_{key}^ih_n^T)}}{{sqrt {{d_k}} }}}
end{array}}
ight])VW_{value}^i\
{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} = s{
m{um}}({
m{weigh}}{{
m{t}}_{sqrt {{d_k}} }}*value)
end{array}]
注意$sqrt d_k$ 是softmax中的temperature参数：
[{p_i} = frac{{{e^{frac{{{ m{logit}}{{ m{s}}_i}}}{ au }}}}}{{sumlimits_i {e_i^{frac{{{ m{logit}}{{ m{s}}_i}}}{ au }}} }}]
t越大，则经过softmax的得到的概率值之间越接近。t越小，则经过softmax得到的概率值之间越差异越大。当t趋近于0的时候，只有最大的一项是1，其他均几乎为0：
[mathop {lim }limits_{ au o 0} {p_i} o 1{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} if{kern 1pt} {kern 1pt} {kern 1pt} {p_i} = max ({p_k}){kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} 1 le k le N{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} else{kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} {kern 1pt} 0]
MultiHead注意力向量由多个$head_{i}$拼接后过一个线性层得到最终的MultiHead Attention
[egin{array}{l}
MulitiHead = Concat(hea{d_1},hea{d_2},...,hea{d_n}){W^o}{kern 1pt} {kern 1pt} {kern 1pt} \
where{kern 1pt} {kern 1pt} hea{d_i} = Attention(QW_{query}^i,KW_{key}^i,VW_{value}^i) = {
m{softmax}}(frac{{(QW_{query}^i){{(KW_{key}^i)}^T}}}{{sqrt {{d_k}} }})VW_{value}^i
end{array}]
（2）LayerNorm+Position-wise Feed-Forward Networks
[FFN(x) = max (0,x{W_1} + {b_1}){W_2} + {b_2}]
注意这里实现上和论文中有点区别，具体实现是先LayerNorm然后再FFN
class PositionwiseFeedForward(nn.Module): """ A two-layer Feed-Forward-Network with residual layer norm. Args: d_model (int): the size of input for the first-layer of the FFN. d_ff (int): the hidden layer size of the second-layer of the FNN. dropout (float): dropout probability(0-1.0). """ def __init__(self, d_model, d_ff, dropout=0.1): super(PositionwiseFeedForward, self).__init__() self.w_1 = nn.Linear(d_model, d_ff) self.w_2 = nn.Linear(d_ff, d_model) self.layer_norm = onmt.modules.LayerNorm(d_model) self.dropout_1 = nn.Dropout(dropout) self.relu = nn.ReLU() self.dropout_2 = nn.Dropout(dropout) def forward(self, x): """ Layer definition. Args: input: [ batch_size, input_len, model_dim ] Returns: output: [ batch_size, input_len, model_dim ] """ inter = self.dropout_1(self.relu(self.w_1(self.layer_norm(x)))) output = self.dropout_2(self.w_2(inter)) return output + x
（3）Layer Normalization
[{
m{x = }}left[ {egin{array}{*{20}{c}}
{{x_1}}&{{x_2}}&{...}&{{x_n}}
end{array}}
ight]] $x_{1}, x_{2}, x_{3},... ,x_{n}$为样本$x$的不同特征
[{{hat x}_i} = frac{{{x_i} - E(x)}}{{sqrt {Var(x)} }}]
[{
m{hat x = }}left[ {egin{array}{*{20}{c}}
{{{hat x}_1}}&{{{hat x}_2}}&{...}&{{{hat x}_n}}
end{array}}
ight]]
最终$hat x$为layer normalization的输出，并且$hat x$均值为0，方差为1：
[egin{array}{l}
E({
m{hat x}}) = frac{1}{n}sumlimits_{i = 1}^n {{{hat x}_i}} = frac{1}{n}sumlimits_{i = 1}^n {frac{{{x_i} - E(x)}}{{sqrt {Var(x)} }} = } frac{1}{n}frac{{[({x_1} + {x_2} + ... + {x_n}) - nE(x)]}}{{sqrt {Var(x)} }} = 0\
Var({
m{hat x}}) = frac{1}{{n - 1}}sumlimits_{i = 1}^n {{{({
m{hat x}} - E({
m{hat x}}))}^2}} = frac{1}{{n - 1}}sumlimits_{i = 1}^n {{{{
m{hat x}}}^2}} = frac{1}{{n - 1}}sumlimits_{i = 1}^n {frac{{{{({x_i} - E(x))}^2}}}{{Var(x)}} = } frac{{frac{1}{{n - 1}}sumlimits_{i = 1}^n {{{({x_i} - E(x))}^2}} }}{{Var(x)}} = frac{{Var(x)}}{{Var(x)}} = 1
end{array}]
但是通常引入两个超参数w和bias， w和bias通过反向传递更新，但是初始值$w_{initial}=1, bias_{bias}=0$，$varepsilon$防止分母为0:
[{{hat x}_i} = w*frac{{{x_i} - E(x)}}{{sqrt {Var(x) + varepsilon } }} + bias]
伪代码如下:
class LayerNorm(nn.Module): """ Layer Normalization class """ def __init__(self, features, eps=1e-6): super(LayerNorm, self).__init__() self.a_2 = nn.Parameter(torch.ones(features)) self.b_2 = nn.Parameter(torch.zeros(features)) self.eps = eps def forward(self, x): """ x=[-0.0101, 1.4038, -0.0116, 1.4277], [ 1.2195, 0.7676, 0.0129, 1.4265] """ mean = x.mean(-1, keepdim=True) """ mean=[[ 0.7025], [ 0.8566]] """ std = x.std(-1, keepdim=True) """ std=[[0.8237], [0.6262]] """ return self.a_2 * (x - mean) / (std + self.eps) + self.b_2 """ self.a_2=[1,1,1,1] self.b_2=[0,0,0,0] return [[-0.8651, 0.8515, -0.8668, 0.8804], [ 0.5795, -0.1422, -1.3475, 0.9101]] """
（4）Embedding
位置向量 Position Embedding
[egin{array}{l}
P{E_{pos,2i}} = sin (frac{{pos}}{{{
m{1000}}{{
m{0}}^{frac{{{
m{2i}}}}{{{{
m{d}}_{mod el}}}}}}}}) = sin (pos*div\_term)\
P{E_{pos,2i + 1}} = cos (frac{{pos}}{{{
m{1000}}{{
m{0}}^{frac{{{
m{2i}}}}{{{{
m{d}}_{mod el}}}}}}}}) = cos (pos*div\_term)\
div\_term = {e^{log (frac{1}{{{
m{1000}}{{
m{0}}^{frac{{{
m{2i}}}}{{{{
m{d}}_{mod el}}}}}}}}{
m{)}}}}{
m{ = }}{{
m{e}}^{ - frac{{2i}}{{{{
m{d}}_{mod el}}}}log (10000)}} = {{
m{e}}^{2i*( - frac{{log (10000)}}{{{{
m{d}}_{mod el}}}})}}
end{array}]
计算Position Embedding举例：
输入句子$S=[w_1,w_2,...,w_{max\_len}]$, m为句子长度 ，假设max_len=3，且$d_{model}=4$:
pe = torch.zeros(max_len, dim)
position = torch.arange(0, max_len).unsqueeze(1) #position=[0,1,2] position.shape=(3,1) div_term = torch.exp((torch.arange(0, dim, 2, dtype=torch.float) *-(math.log(10000.0) / dim))) """ torch.arange(0, dim, 2, dtype=torch.float)=[0,2,4] shape=(3) -(math.log(10000.0) / dim)=-1.5350567286626973 (torch.arange(0, dim, 2, dtype=torch.float) *-(math.log(10000.0) / dim))=[0,2,4]*-1.5350567286626973=[-0.0000, -3.0701, -6.1402] div_term=exp([-0.0000, -3.0701, -6.1402])=[1.0000, 0.0464, 0.0022] """ pe[:, 0::2] = torch.sin(position.float() * div_term) """ pe=[[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.8415, 0.0000, 0.0464, 0.0000, 0.0022, 0.0000], [ 0.9093, 0.0000, 0.0927, 0.0000, 0.0043, 0.0000]] """ pe[:, 1::2] = torch.cos(position.float() * div_term) """ pe=[[ 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000], [ 0.8415, 0.5403, 0.0464, 0.9989, 0.0022, 1.0000], [ 0.9093, -0.4161, 0.0927, 0.9957, 0.0043, 1.0000]] """ pe = pe.unsqueeze(1) #pe.shape=[3,1,6]
max_len=20，$d_{model}=4$Position Embedding,可以观察到同一个时间序列内t位置内大约只有前半部分起到区分位置的作用：
语义向量normal Embedding:
$x=[x_1,x_2,x_3,...,x_n]$,$x_i$为one-hot行向量
那么，代表语义的embedding是$emb=[emb_{1},emb_{2},emb_{3},...,emb_{n}$ $emb_{i}=x_iW$，transformer中的词向量表示为语义向量emb_{i}和位置向量pe_{i}之和
$emb^{final}_{i}=emb_{i}+pe_{i}$
二 Encoder
(1)Encoder是由多个相同的层堆叠在一起的：$[input ightarrow embedding ightarrow self-attention ightarrow Add Norm ightarrow FFN ightarrow Add Norm]$:
(2)Encoder的self-attention是既考虑前面的词也考虑后面的词的，而Decoder的self-attention只考虑前面的词，因此mask矩阵是全1。因此encoder的self-attention如下图：
伪代码如下：
class TransformerEncoderLayer(nn.Module): """ A single layer of the transformer encoder. Args: d_model (int): the dimension of keys/values/queries in MultiHeadedAttention, also the input size of the first-layer of the PositionwiseFeedForward. heads (int): the number of head for MultiHeadedAttention. d_ff (int): the second-layer of the PositionwiseFeedForward. dropout (float): dropout probability(0-1.0). """ def __init__(self, d_model, heads, d_ff, dropout): super(TransformerEncoderLayer, self).__init__() self.self_attn = onmt.modules.MultiHeadedAttention( heads, d_model, dropout=dropout) self.feed_forward = PositionwiseFeedForward(d_model, d_ff, dropout) self.layer_norm = onmt.modules.LayerNorm(d_model) self.dropout = nn.Dropout(dropout) def forward(self, inputs, mask): """ Transformer Encoder Layer definition. Args: inputs (`FloatTensor`): `[batch_size x src_len x model_dim]` mask (`LongTensor`): `[batch_size x src_len x src_len]` Returns: (`FloatTensor`): * outputs `[batch_size x src_len x model_dim]` """ input_norm = self.layer_norm(inputs) context, _ = self.self_attn(input_norm, input_norm, input_norm, mask=mask) out = self.dropout(context) + inputs return self.feed_forward(out)
二 Decoder
(1)decoder中的self attention层在计算self attention的时候，因为实际预测中只能知道前面的词，因此在训练过程中只需要计算当前位置和前面位置的self attention,通过掩码来计算Masked Multi-head Attention层。
例如"I have an app",中翻译出第一个词后"I",
"I"的self attention只计算与"I"与自己的self attention: Attention("I","I")，来预测下一个词
翻译出"I have"后，计算"have"与"have","have"与"I"的self attention： Attention("have","I"), Attention("have","have"),来预测下一个词
翻译出"I have an"后，计算"an"与"an","an"与"have","an"与"I"的self attention： Attention("an","an"), Attention("an","have"),Attention("an","I")来预测下一个词
可以用下图来表示：
self-attention的伪代码如下:
class MultiHeadedAttention(nn.Module): """ Args: head_count (int): number of parallel heads model_dim (int): the dimension of keys/values/queries, must be divisible by head_count dropout (float): dropout parameter """ def __init__(self, head_count, model_dim, dropout=0.1): assert model_dim % head_count == 0 self.dim_per_head = model_dim // head_count self.model_dim = model_dim super(MultiHeadedAttention, self).__init__() self.head_count = head_count self.linear_keys = nn.Linear(model_dim,model_dim) self.linear_values = nn.Linear(model_dim,model_dim) self.linear_query = nn.Linear(model_dim,model_dim) self.softmax = nn.Softmax(dim=-1) self.dropout = nn.Dropout(dropout) self.final_linear = nn.Linear(model_dim, model_dim) def forward(self, key, value, query, mask=None, layer_cache=None, type=None): """ Compute the context vector and the attention vectors. Args: key (`FloatTensor`): set of `key_len` key vectors `[batch, key_len, dim]` value (`FloatTensor`): set of `key_len` value vectors `[batch, key_len, dim]` query (`FloatTensor`): set of `query_len` query vectors `[batch, query_len, dim]` mask: binary mask indicating which keys have non-zero attention `[batch, query_len, key_len]` Returns: (`FloatTensor`, `FloatTensor`) : * output context vectors `[batch, query_len, dim]` * one of the attention vectors `[batch, query_len, key_len]` """ batch_size = key.size(0) dim_per_head = self.dim_per_head head_count = self.head_count key_len = key.size(1) query_len = query.size(1) def shape(x): """ projection """ return x.view(batch_size, -1, head_count, dim_per_head) .transpose(1, 2) def unshape(x): """ compute context """ return x.transpose(1, 2).contiguous() .view(batch_size, -1, head_count * dim_per_head) # 1) Project key, value, and query. if layer_cache is not None: key = self.linear_keys(key) #key.shape=[batch_size,key_len,dim] => key.shape=[batch_size,key_len,dim] value = self.linear_values(value) #value.shape=[batch_size,key_len,dim] => key.shape=[batch_size,key_len,dim] query = self.linear_query(query) #query.shape=[batch_size,key_len,dim] => key.shape=[batch_size,key_len,dim] key = shape(key) #key.shape=[batch_size,head_count,key_len,dim_per_head] value = shape(value) #value.shape=[batch_size,head_count,value_len,dim_per_head] query = shape(query) #query.shape=[batch_size,head_count,query_len,dim_per_head] key_len = key.size(2) query_len = query.size(2) # 2) Calculate and scale scores. query = query / math.sqrt(dim_per_head) scores = torch.matmul(query, key.transpose(2, 3)) #query.shape=[batch_size,head_count,query_len,dim_per_head] #key.transpose(2, 3).shape=[batch_size,head_count,dim_per_head,key_len] #scores.shape=[batch_size,head_count,query_len,key_len] if mask is not None: mask = mask.unsqueeze(1).expand_as(scores) scores = scores.masked_fill(mask, -1e18) # 3) Apply attention dropout and compute context vectors. attn = self.softmax(scores) #scores.shape=[batch_size,head_count,query_len,key_len] drop_attn = self.dropout(attn) context = unshape(torch.matmul(drop_attn, value)) #drop_attn.shape=[batch_size,head_count,query_len,key_len] #value.shape=[batch_size,head_count,value_len,dim_per_head] #torch.matmul(drop_attn, value).shape=[batch_size,head_count,query_len,dim_per_head] #context.shape=[batch_size,query_len,head_count*dim_per_head] output = self.final_linear(context) #context.shape=[batch_size,query_len,head_count*dim_per_head] return output
(2)Decoder的结构为$[input ightarrow embedding ightarrow self-attention ightarrow Add Norm ightarrow context-attention ightarrow FFN ightarrow Add Norm]$:
class TransformerDecoderLayer(nn.Module): """ Args: d_model (int): the dimension of keys/values/queries in MultiHeadedAttention, also the input size of the first-layer of the PositionwiseFeedForward. heads (int): the number of heads for MultiHeadedAttention. d_ff (int): the second-layer of the PositionwiseFeedForward. dropout (float): dropout probability(0-1.0). self_attn_type (string): type of self-attention scaled-dot, average """ def __init__(self, d_model, heads, d_ff, dropout, self_attn_type="scaled-dot"): super(TransformerDecoderLayer, self).__init__() self.self_attn_type = self_attn_type if self_attn_type == "scaled-dot": self.self_attn = onmt.modules.MultiHeadedAttention( heads, d_model, dropout=dropout) elif self_attn_type == "average": self.self_attn = onmt.modules.AverageAttention( d_model, dropout=dropout) self.context_attn = onmt.modules.MultiHeadedAttention( heads, d_model, dropout=dropout) self.feed_forward = PositionwiseFeedForward(d_model, d_ff, dropout) self.layer_norm_1 = onmt.modules.LayerNorm(d_model) self.layer_norm_2 = onmt.modules.LayerNorm(d_model) self.dropout = dropout self.drop = nn.Dropout(dropout) mask = self._get_attn_subsequent_mask(MAX_SIZE) # Register self.mask as a buffer in TransformerDecoderLayer, so # it gets TransformerDecoderLayer's cuda behavior automatically. self.register_buffer('mask', mask) def forward(self, inputs, memory_bank, src_pad_mask, tgt_pad_mask, previous_input=None, layer_cache=None, step=None): """ Args: inputs (`FloatTensor`): `[batch_size x 1 x model_dim]` memory_bank (`FloatTensor`): `[batch_size x src_len x model_dim]` src_pad_mask (`LongTensor`): `[batch_size x 1 x src_len]` tgt_pad_mask (`LongTensor`): `[batch_size x 1 x 1]` Returns: (`FloatTensor`, `FloatTensor`, `FloatTensor`): * output `[batch_size x 1 x model_dim]` * attn `[batch_size x 1 x src_len]` * all_input `[batch_size x current_step x model_dim]` """ dec_mask = torch.gt(tgt_pad_mask + self.mask[:, :tgt_pad_mask.size(1), :tgt_pad_mask.size(1)], 0) input_norm = self.layer_norm_1(inputs) all_input = input_norm if previous_input is not None: all_input = torch.cat((previous_input, input_norm), dim=1) dec_mask = None if self.self_attn_type == "scaled-dot": query, attn = self.self_attn(all_input, all_input, input_norm, mask=dec_mask, layer_cache=layer_cache, type="self") elif self.self_attn_type == "average": query, attn = self.self_attn(input_norm, mask=dec_mask, layer_cache=layer_cache, step=step) query = self.drop(query) + inputs query_norm = self.layer_norm_2(query) mid, attn = self.context_attn(memory_bank, memory_bank, query_norm, mask=src_pad_mask, layer_cache=layer_cache, type="context") output = self.feed_forward(self.drop(mid) + query) return output, attn, all_input
五 label smoothing (标签平滑)
普通的交叉熵损失函数：
[egin{array}{l}
{
m{loss}} = - sumlimits_{k = 1}^K {tru{e_k}log (p(k|x))} \
p(k|x) = softmax (log it{s_k})\
log it{s_k} = sumlimits_i {{w_{ik}}{z_i}}
end{array}]
梯度为：
[egin{array}{l}
Delta {w_{ik}} = frac{{partial loss}}{{partial {w_{ik}}}} = frac{{partial loss}}{{partial logit{s_{ik}}}}frac{{partial logits}}{{partial {w_{ik}}}} = ({y_k} - labe{l_k}){z_k}\
label = [egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{frac{alpha }{4}}&{frac{alpha }{4}}
end{array}}&{1 - alpha }&{frac{alpha }{4}}&{frac{alpha }{4}}
end{array}]
end{array}]
有一个问题
只有正确的那一个类别有贡献，其他标注数据中不正确的类别概率是0，无贡献，朝一个方向优化，容易导致过拟合
因此提出label smoothing 让标注数据中正确的类别概率小于1，其他不正确类别的概率大于0：
也就是之前$label=[0,0,0,1,0]$,通过标签平滑,给定一个固定参数$alpha$, 概率为1地方减去这个小概率，标签为0的地方平分这个小概率$alpha$变成:
[labe{l^{new}} = [egin{array}{*{20}{c}}
{egin{array}{*{20}{c}}
{frac{alpha }{4}}&{frac{alpha }{4}}
end{array}}&{1 - alpha }&{frac{alpha }{4}}&{frac{alpha }{4}}
end{array}]]
损失函数为
[egin{array}{l}
loss = - sumlimits_{k = 1}^K {label_k^{new}log p(k|x)} \
label_k^{new} = (1{
m{ - }}alpha ){delta _{k,y}} + frac{alpha }{K}({delta _{k,y}} = 1 quad if quad k==y quad else quad 0)\
loss = - (1{
m{ - }}alpha )sumlimits_{k = 1}^K {{
m{label}}log p(k|x)} - frac{alpha }{K}sumlimits_{k = 1}^K {(frac{alpha }{K})log p(k|x)} \
loss = (1{
m{ - }}alpha )CrossEntropy({
m{label}},p(k|x)) + frac{alpha }{K}CrossEntropy(frac{alpha }{K},p(k|x))
end{array}]
引入相对熵函数：
[{D_{KL}}(Y||X) = sumlimits_i {Y(i)log (frac{{Y(i)}}{{X(i)}})} = sumlimits_i {Y(i)log Y(i)} - Y(i)log X(i)]
pytorch中的torch.nn.function.kl_div用来计算相对熵：
torch.nn.function.kl_div(y,x):$x=[x_1,x_2,...,x_N] y=[y_1,y_2,...,y_N]$:
$L=l_1+l_2+...+l_N其中 l_i=x_i*(log(x_i)-y_i)$
举例:x=[3] y=[2] torch.nn.function.kl_div(y,x)=3(log3-2)=-2.7042
class LabelSmoothingLoss(nn.Module): """ With label smoothing, KL-divergence between q_{smoothed ground truth prob.}(w) and p_{prob. computed by model}(w) is minimized. """ def __init__(self, label_smoothing, tgt_vocab_size, ignore_index=-100): assert 0.0 < label_smoothing <= 1.0 self.padding_idx = ignore_index super(LabelSmoothingLoss, self).__init__() smoothing_value = label_smoothing / (tgt_vocab_size - 2) one_hot = torch.full((tgt_vocab_size,), smoothing_value) one_hot[self.padding_idx] = 0 self.register_buffer('one_hot', one_hot.unsqueeze(0)) self.confidence = 1.0 - label_smoothing def forward(self, output, target): """ output (FloatTensor): batch_size x n_classes target (LongTensor): batch_size """ model_prob = self.one_hot.repeat(target.size(0), 1) model_prob.scatter_(1, target.unsqueeze(1), self.confidence) model_prob.masked_fill_((target == self.padding_idx).unsqueeze(1), 0) return F.kl_div(output, model_prob, size_average=False)
附： Transformer与RNN的结合RNMT+(The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation)
（1）RNN：难以训练并且表达能力较弱 trainability versus expressivity
（2）Transformer：有很强的特征提取能力（a strong feature extractor），但是没有memory机制，因此需要额外引入位置向量。
