本章博文主要收集 deep learning 方面看的好的文章及其链接
(1), 一位南大Ph.D candidate 总结的 the tricks/tips in deep learning
http://lamda.nju.edu.cn/weixs/project/CNNTricks/CNNTricks.html
(2), GAN 系列的paper 和 code 列表
https://github.com/zhangqianhui/AdversarialNetsPapers
(3), caffe 添加新层的教程
http://blog.csdn.net/shuzfan/article/details/51322976
(4), Christopher Olah的博客
http://colah.github.io/
(5), Deep Learning Tips from AndrewNg
From: https://twitter.com/Tbeltramelli/status/846300705635926017
(6), A visulization website for learning statistic theory
http://students.brown.edu/seeing-theory/
(7), 通俗详解 softmax 及其求导过程
http://mp.weixin.qq.com/s/MS8h8BUv1BC3Ql9w2oxmJg
(8), A Website to ploted the structure of caffe prototxt
http://ethereon.github.io/netscope/quickstart.html
(9), Training Tips for Deep Network
在很多deep network中,都会有这样的情况发生,在没有 pretrained_model 的情况下, startup_training 是一个skillful work。在ResNet中提过,对于resnet-101这样深网络,直接采用0.1 的lr去train很难让初始的网络趋向converge。采用的training 方式是: 先用小的lr, 如0.01去warm up 整个网络的W, 大约数百次(400 iterations)之后 training error 就下降 80%, 然后再回去0.1的lr, 开始training。
这种方法在很多paper上被应用来train deep network。
(10), ConvNet (cifar-10) training demo
可以从简单的training demo可以观察各种参数的变化,feature map的变化,gradients 的activation。 (很好的network visualization 例子)
http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html
(11), 论文“ Understanding the difficulty of training deep feedforward neural networks ” 给我们的tips & tricks:
--1, Monitoring activations and gradients across layers and training iterations is a powerful investigative tool for understanding training difficulties in deep nets
在训练深度网络的时候,检测激活函数的值,梯度,layer的值,帮助你更好的train
--2, 对于 random initialization 的weights,应避免使用 非0对称的sigmoid函数,否则会导致top hidden layer 的 saturation。
(12), multi-GPU about Caffe
Caffe tutorial: https://github.com/BVLC/caffe/blob/master/docs/multigpu.md
Reconcile Python_layer with Multi_GPU: https://github.com/BVLC/caffe/issues/2936
Python Multi-GPU (for releasing GIL): https://github.com/BVLC/caffe/pull/4360
(13),