day-18 滑动平均模型测试样例

为了使训练模型在测试数据上有更好的效果，可以引入一种新的方法：滑动平均模型。通过维护一个影子变量，来代替最终训练参数，进行训练模型的验证。

在tensorflow中提供了ExponentialMovingAverage来实行滑动平均模型，模型会维护一个影子变量，其计算公式为：

shadow_variable = decay * shadow_variable + (1 - decay) * variable

当训练模型时，维护训练参数的滑动平均值是有好处的。相比较最终训练值，验证时使用滑动平均值有时能产生更好的结果。
apply()函数方法会添加一个影子拷贝到训练变量中，然后在他们影子副本上维护训练参数的滑动平均值计算操作。这个操作通常在一轮训练完之后进行。
average()和average_name()函数提供了访问影子变量和他们名字的方法。这在构建一个评估模型或者从checkpoint文件中重载模型时非常有用。在验证时，可以帮助使用滑动平均值替换最后训练值。要使用这个模型，需要有3个步骤：

1、创建一个滑动平均模型对象

step = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
ema = tf.train.ExponentialMovingAverage(decay=0.99,num_updates=step)

decay就是前面公式里面的衰减因此，合理的decay值可以是接近1.0，例如0.999,0.9999等多个9中变换。num_updates为一个可选的参数，decay值由如下公式决定：

min(decay, (1 + num_updates) / (10 + num_updates))。目的是使影子变量在刚开始训练的时候，更新的更快。因此num_updates通常可以传入一个递增的训练步数变量。

2、加入训练参数列表到模型中进行维护

新建两个训练参数，并将其加入滑动平均模型对象中维护，apply()函数接受一个参数列表。

var0 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
var1 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
maintain_averages_op = ema.apply([var0,var1])

3、训练完成以后，更新滑动平均模型中各个影子变量的值

sess.run(maintain_averages_op)
print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[10,4.555,10,9.01]

完整的滑动平均模型测试样例如下：

# 导入tensorflow库
import tensorflow as tf

# 创建一个滑动平均模型对象
step = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
ema = tf.train.ExponentialMovingAverage(decay=0.99,num_updates=step)

# 创建两个训练参数，并将其加入滑动平均模型对象中，对象会为这两个训练参数创建两个影子变量
# 影子变量shadow_variable = decay * shadow_variable + (1 - decay) * variable
# 如果滑动平均模型对象创建时，指定了num_updates，则decay = min{decay,(1 + num_updates)/(10 + num_updates)}
var0 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
var1 = tf.Variable(initial_value=0,dtype=tf.float32,trainable=False)
maintain_averages_op = ema.apply([var0,var1])

# 测试更新影子变量值
with tf.Session() as sess:
    init = tf.global_variables_initializer()
    sess.run(init)

    # 第一次初始滑动平均
    sess.run(maintain_averages_op)
    # decay = min(0.99,0.1) = 0.1
    # 初始时：
    # shadow_variable_var0 = var0 = 0
    # shadow_variable_var1 = var1 = 0
    print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))

    # 第二次更新滑动平均
    sess.run(tf.assign(var0,5.0))
    sess.run(tf.assign(var1, 10.0))
    # decay = min(0.99,(1+0)/(10+0)) = 0.1
    # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.1*0 + (1-0.1)*5 = 4.5
    # shadow_variable_var1 = 9.0
    sess.run(maintain_averages_op)
    print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[5.0,4.5,10,9.0]

    # 第三次更新滑动平均
    sess.run(tf.assign(step,10000))
    sess.run(tf.assign(var0,10))
    # decay = min(0.99,(1+10000)/(10+10000)) = 0.99
    # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.99*4.5 + (1-0.99)*10 = 4.555
    # shadow_variable_var1 = 0.99*9.0+(1-0.99)*10 = 9.01
    sess.run(maintain_averages_op)
    print(sess.run([var0,ema.average(var0),var1,ema.average(var1)]))  # 输出[10,4.555,10,9.01]

    # 第四次更新滑动平均
    # decay = min(0.99,(1+10000)/(10+10000)) = 0.99
    # shadow_variable_var0 = decay * shadow_variable + (1 - decay) * variable = 0.99*4.555 + (1-0.99)*10 = 4.60945
    # shadow_variable_var1 = 0.99*9.01+(1-0.99)*10 = 9.0199
    sess.run(maintain_averages_op)
    print(sess.run([var0, ema.average(var0), var1, ema.average(var1)]))  # 输出[10,4.60945,10,9.0199]

下面是tensorflow官方给出的两种滑动模型使用场景：

  Example usage when creating a training model:

  ```python
  # Create variables.
  var0 = tf.Variable(...)
  var1 = tf.Variable(...)
  # ... use the variables to build a training model...
  ...
  # Create an op that applies the optimizer.  This is what we usually
  # would use as a training op.
  opt_op = opt.minimize(my_loss, [var0, var1])

  # Create an ExponentialMovingAverage object
  ema = tf.train.ExponentialMovingAverage(decay=0.9999)

  with tf.control_dependencies([opt_op]):
      # Create the shadow variables, and add ops to maintain moving averages
      # of var0 and var1. This also creates an op that will update the moving
      # averages after each training step.  This is what we will use in place
      # of the usual training op.
      training_op = ema.apply([var0, var1])

  ...train the model by running training_op...
  ```


  There are two ways to use the moving averages for evaluations:

  *  Build a model that uses the shadow variables instead of the variables.
     For this, use the `average()` method which returns the shadow variable
     for a given variable.
  *  Build a model normally but load the checkpoint files to evaluate by using
     the shadow variable names.  For this use the `average_name()` method.  See
     the @{tf.train.Saver} for more
     information on restoring saved variables.

  Example of restoring the shadow variable values:

  ```python
  # Create a Saver that loads variables from their saved shadow values.
  shadow_var0_name = ema.average_name(var0)
  shadow_var1_name = ema.average_name(var1)
  saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})
  saver.restore(...checkpoint filename...)
  # var0 and var1 now hold the moving average values
  ```
  """