cs224n

Vanishing Gradients and Fancy RNNs

Vanishing gradient problem

Why is vanishing gradient a problem?

Another explanation: Gradient can be viewed as a measure of
the effect of the past on the future. If the gradient becomes vanishingly small over longer distances (step t to step t+n), then we can’t tell whether:

There’s no dependency between step t and t+n in the data
We have wrong parameters to capture the true dependency between t and t+n

Effect of vanishing gradient on RNN-LM

Why is exploding gradient a problem?

If the gradient becomes too big, then the SGD update step
becomes too big:

[ heta^{ ext {new}}= heta^{ ext { old }}-alpha abla_{ heta} J( heta) ]

This can cause bad updates: we take too large a step and reach
a bad parameter configuration (with large loss). In the worst case, this will result in Inf or NaN in your network (then you have to restart training from an earlier checkpoint).

Gradient clipping: solution for exploding gradient

Gradient clipping: if the norm of the gradient is greater than
some threshold, scale it down before applying SGD update

Intuition: take a step in the same direction, but a smaller step

How to fix vanishing gradient problem?

The main problem is that it’s too difficult for the RNN to learn to preserve information over many timesteps. In a vanilla RNN, the hidden state is constantly being rewritten:

[oldsymbol{h}^{(t)}=sigmaleft(oldsymbol{W}_{h} oldsymbol{h}^{(t-1)}+oldsymbol{W}_{x} oldsymbol{x}^{(t)}+oldsymbol{b} ight) ]

Long Short-Term Memory (LSTM)

How does LSTM solve vanishing gradients?

The LSTM architecture makes it easier for the RNN to preserve information over many timesteps
LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies

Gated Recurrent Units (GRU)

LSTM vs GRU

The biggest difference is that GRU is quicker to compute and has fewer parameters
There is no conclusive evidence that one consistently performs better than the other
LSTM is a good default choice (especially if your data has particularly long dependencies, or you have lots of training data)
Rule of thumb: start with LSTM, but switch to GRU if you want something more efficient

Conclusion: Though vanishing/exploding gradients are a general
problem, RNNs are particularly unstable due to the repeated
multiplication by the same weight matrix