训练集(train set) 验证集(validation set) 测试集(test set)。
http://blog.sina.com.cn/s/blog_4d2f6cf201000cjx.html
一般需要将样本分成独立的三部分训练集(train set)，验证集(validation set)和测试集(test set)。其中训练集用来估计模型，验证集用来确定网络结构或者控制模型复杂程度的参数，而测试集则检验最终选择最优的模型的性能如何。一个典型的划分是训练集占总样本的50％，而其它各占25％，三部分都是从样本中随机抽取。
样本少的时候，上面的划分就不合适了。常用的是留少部分做测试集。然后对其余N个样本采用K折交叉验证法。就是将样本打乱，然后均匀分成K份，轮流选择其中K－1份训练，剩余的一份做验证，计算预测误差平方和，最后把K次的预测误差平方和再做平均作为选择最优模型结构的依据。特别的K取N，就是留一法（leave one out）。
http://www.cppblog.com/guijie/archive/2008/07/29/57407.html
这三个名词在机器学习领域的文章中极其常见，但很多人对他们的概念并不是特别清楚，尤其是后两个经常被人混用。Ripley, B.D（1996）在他的经典专著Pattern Recognition and Neural Networks中给出了这三个词的定义。
Training set: A set of examples used for learning, which is to fit the parameters [i.e., weights] of the classifier.
Validation set: A set of examples used to tune the parameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.
Test set: A set of examples used only to assess the performance [generalization] of a fully specified classifier.
显然，training set是用来训练模型或确定模型参数的，如ANN中权值等； validation set是用来做模型选择（model selection），即做模型的最终优化及确定的，如ANN的结构；而 test set则纯粹是为了测试已经训练好的模型的推广能力。当然，test set这并不能保证模型的正确性，他只是说相似的数据用此模型会得出相似的结果。但实际应用中，一般只将数据集分成两类，即training set 和test set，大多数文章并不涉及validation set。
Ripley还谈到了Why separate test and validation sets?
1. The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model.
2. After assessing the final model with the test set, YOU MUST NOT tune the model any further.
Step 1) Training: Each type of algorithm has its own parameter options (the number of layers in a Neural Network, the number of trees in a Random Forest, etc). For each of your algorithms, you must pick one option. That’s why you have a validation set.
Step 2) Validating: You now have a collection of algorithms. You must pick one algorithm. That’s why you have a test set. Most people pick the algorithm that performs best on the validation set (and that's ok). But, if you do not measure your top-performing algorithm’s error rate on the test set, and just Go with its error rate on the validation set, then you have blindly mistaken the “best possible scenario” for the “most likely scenario.” That's a recipe for disaster.
Step 3) Testing: I suppose that if your algorithms did not have any parameters then you would not need a third step. In that case, your validation step would be your test step. Perhaps Matlab does not ask you for parameters or you have chosen not to use them and that is the source of your confusion.
My Idea is that those option in neural network toolbox is for avoiding overfitting. In this situation the weights are specified for the training data only and don't show the global trend. By having a validation set, the iterations are adaptable to where decreases in the training data error cause decreases in validation data and increases in validation data error; along with decreases in training data error, this demonstrates the overfitting phenomenon.
http://blog.sciencenet.cn/blog-397960-666113.html
for each epoch
for each training data instance
propagate error through the network
adjust the weights
calculate the accuracy over training data
for each validation data instance
calculate the accuracy over the validation data
if the threshold validation accuracy is met
exit training
else
continue training
Once you're finished training, then you run against your testing set and verify that the accuracy is sufficient.
Training Set: this data set is used to adjust the weights on the neural network.
Validation Set: this data set is used to minimize overfitting. You're not adjusting the weights of the network with this data set, you're just verifying that any increase in accuracy over the training data set actually yields an increase in accuracy over a data set that has not been shown to the network before, or at least the network hasn't trained on it (i.e. validation data set). If the accuracy over the training data set increases, but the accuracy over then validation data set stays the same or decreases, then you're overfitting your neural network and you should stop training.
Testing Set: this data set is used only for testing the final solution in order to confirm the actual predictive power of the network.
Validating set is used in the process of training. Testing set is not. The Testing set allows
1)to see if the training set was enough and
2)whether the validation set did the job of preventing overfitting. If you use the testing set in the process of training then it will be just another validation set and it won't show what happens when new data is feeded in the network.
Training set: A set of examples used for learning, that is to fit the parameters [i.e., weights] of the classifier.
Validation set: A set of examples used to tune the parameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.
Test set: A set of examples used only to assess the performance [generalization] of a fully specified classifier.
The error surface will be different for different sets of data from your data set (batch learning). Therefore if you find a very good local minima for your test set data, that may not be a very good point, and may be a very bad point in the surface generated by some other set of data for the same problem. Therefore you need to compute such a model which not only finds a good weight configuration for the training set but also should be able to predict new data (which is not in the training set) with good error. In other words the network should be able to generalize the examples so that it learns the data and does not simply remembers or loads the training set by overfitting the training data.
The validation data set is a set of data for the function you want to learn, which you are not directly using to train the network. You are training the network with a set of data which you call the training data set. If you are using gradient based algorithm to train the network then the error surface and the gradient at some point will completely depend on the training data set thus the training data set is being directly used to adjust the weights. To make sure you don't overfit the network you need to input the validation dataset to the network and check if the error is within some range. Because the validation set is not being using directly to adjust the weights of the netowork, therefore a good error for the validation and also the test set indicates that the network predicts well for the train set examples, also it is expected to perform well when new example are presented to the network which was not used in the training process.
Early stopping is a way to stop training. There are different variations available, the main outline is, both the train and the validation set errors are monitored, the train error decreases at each iteration (backprop and brothers) and at first the validation error decreases. The training is stopped at the moment the validation error starts to rise. The weight configuration at this point indicates a model, which predicts the training data well, as well as the data which is not seen by the network . But because the validation data actually affects the weight configuration indirectly to select the weight configuration. This is where the Test set comes in. This set of data is never used in the training process. Once a model is selected based on the validation set, the test set data is applied on the network model and the error for this set is found. This error is a representative of the error which we can expect from absolutely new data for the same problem.
显然，training set是用来训练模型或确定模型参数的，如ANN中权值等； validation set是用来做模型选择（model selection），即做模型的最终优化及确定的，如ANN的结构；而 test set则纯粹是为了测试已经训练好的模型的推广能力。当然，test set这并不能保证模型的正确性，他只是说相似的数据用此模型会得出相似的结果。
但实际应用中，一般只将数据集分成两类，即training set 和test set，大多数文章并不涉及validation set。 Ripley还谈到了Why separate test and validation sets?
1. The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model.
2. After assessing the final model with the test set, YOU MUST NOT tune the model any further.
It is rarely useful to have a NN simply memorize a set of data, since memorization can be done much more efficiently by numerous algorithms for table look-up. Typically, you want the NN to be able to perform accurately on new data, that is, to generalize. There seems to be no term in the NN literature for the set of all cases that you want to be able to generalize to. Statisticians call this set the "population". Tsypkin (1971) called it the "grand truth distribution," but this term has never caught on. Neither is there a consistent term in the NN literature for the set of cases that are available for training and evaluating an NN. Statisticians call this set the "sample". The sample is usually a subset of the population. (Neurobiologists mean something entirely different by "population," apparently some collection of neurons, but I have never found out the exact meaning. I am going to continue to use "population" in the statistical sense until NN researchers reach a consensus on some other terms for "population" and "sample"; I suspect this will never happen.) In NN methodology, the sample is often subdivided into "training", "validation", and "test" sets.
The distinctions among these subsets are crucial, but the terms "validation" and "test" sets are often confused. Bishop (1995), an indispensable reference on neural networks, provides the following explanation (p. 372): Since our goal is to find the network having the best performance on new data, the simplest approach to the comparison of different networks is to evaluate the error function using data which is independent of that used for training. Various networks are trained by minimization of an appropriate error function defined with respect to a training data set. The performance of the networks is then compared by evaluating the error function using an independent validation set, and the network having the smallest error with respect to the validation set is selected. This approach is called the hold out method. Since this procedure can itself lead to some overfitting to the validation set, the performance of the selected network should be confirmed by measuring its performance on a third independent set of data called a test set.
And there is no book in the NN literature more authoritative than Ripley (1996), from which the following definitions are taken (p.354):
Training set: A set of examples used for learning that is to fit the parameters [i.e., weights] of the classifier.
Validation set: A set of examples used to tune the parameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.
Test set: A set of examples used only to assess the performance [generalization] of a fully-specified classifier.
The literature on machine learning often reverses the meaning of "validation" and "test" sets. This is the most blatant example of the terminological confusion that pervades artificial intelligence research. The crucial point is that a test set, by the standard definition in the NN literature, is never used to choose among two or more networks, so that the error on the test set provides an unbiased estimate of the generalization error (assuming that the test set is representative of the population, etc.). Any data set that is used to choose the best of two or more networks is, by definition, a validation set, and the error of the chosen network on the validation set is optimistically biased. There is a problem with the usual distinction between training and validation sets. Some training approaches, such as early stopping, require a validation set, so in a sense, the validation set is used for training. Other approaches, such as maximum likelihood, do not inherently require a validation set. So the "training" set for maximum likelihood might encompass both the "training" and "validation" sets for early stopping. Greg Heath has suggested the term "design" set be used for cases that are used solely to adjust the weights in a network, while "training" set be used to encompass both design and validation sets. There is considerable merit to this suggestion, but it has not yet been widely adopted.
http://www.cppblog.com/guijie/archive/2008/07/29/57407.html
http://www.faqs.org/faqs/ai-faq/neural-nets/part1/
在R2009的NN工具箱中，数据被自动分成training set、validation set 及test set 三部分，看了半天help没看懂validation set 是干嘛的，请指教，谢谢。
我的数据本来就不多，只有10组左右，可不可以全部用来训练？
validation set 好像应该是用来用新的数据来评测所建立的模型是否过拟合的
问题在于validation 和 test的算法有什么区别呢。从训练完成后的performance曲线看，通用15%的数据，validation error曲线和test error曲线往往有比较明显的差别。
另外从response图看，似乎GUI默认的15%validation和test数据并不是截取连续的一段数据，而是从整个样本中抽样出来的（从曲线上看是分散在整个样本长度上，和train数据夹杂在一起）。这样的话对于连续的时间序列不会产生问题么，毕竟时间序列样本之间的时序性连续性很重要啊。
不知道这个老话题现在翻出来还有没有人能够解释一下。
其实是这样的 ，在R2009的NN工具箱中，数据被自动分成training set、validation set 及test set 三部分，training set是训练样本数据，validation set是验证样本数据，test set是测试样本数据，这样这三个数据集是没有重叠的。在训练时，用training训练，每训练一次，系统自动会将validation set中的样本数据输入神经网络进行验证，在validation set输入后会得出一个误差（不是网络的训练误差，而是验证样本数据输入后得到的输出误差，可能是均方误差），而此前对validation set会设置一个步数，比如默认是6echo，则系统判断这个误差是否在连续6次检验后不下降，如果不下降或者甚至上升，说明training set训练的误差已经不再减小，没有更好的效果了，这时再训练就没必要了，就停止训练，不然可能陷入过学习。所以validation set有个设置步数，作用就在这里。在你的10组样本中，不可能全部作为训练样本的，还要有测试样本和验证样本。根据matlab版本的不同，具体怎么分配样本也不一样，像R2009应该是自动分配的。
The test set is not involved in training the classifier.
So if you have a large test set, you will probably get a better prediction of the quality of your classifier, that's all.
The training set can be selected by applying a random filter to the data, e.g., select 20% of the points at random to generate the model and test against the remaining 80%. If you want to be especially careful, then do this multiple times, i.e., select different random training sets and compare the models. If you get similar models then your model has probably captured the essential chemistry and physics of the problem. If the models are very different, then you are just fitting equations without a good physical basis. Whats the difference between training set and test set?. Available from: https://www.researchgate.net/post/Whats_the_difference_between_training_set_and_test_set [accessed Jul 27, 2017].
k-折交叉验证(k-fold CrossValidation)
在matlab中，可以利用：
indices=crossvalind('Kfold',x,k);
来实现随机分包的操作，其中x为一个N维列向量（N为数据集A的元素个数，与x具体内容无关，只需要能够表示数据集的规模），k为要分成的包的总个数，输出的结果indices是一个N维列向量，每个元素对应的值为该单元所属的包的编号（即该列向量中元素是1~k的整随机数），利用这个向量即可通过循环控制来对数据集进行划分。例：
[M,N]=size(data);//数据集为一个M*N的矩阵，其中每一行代表一个样本
indices=crossvalind('Kfold',data(1:M,N),10);//进行随机分包
for k=1:10//交叉验证k=10，10个包轮流作为测试集
test = (indices == k); //获得test集元素在数据集中对应的单元编号
train = ~test;//train集元素的编号为非test元素的编号
train_data=data(train,:);//从数据集中划分出train样本的数据
train_target=target(:,train);//获得样本集的测试目标，在本例中是实际分类情况
test_data=data(test,:);//test样本集
test_target=target(:,test);
[HammingLoss(1,k),RankingLoss(1,k),OneError(1,k),Coverage(1,k),Average_Precision(1,k),Outputs,Pre_Labels.MLKNN]=MLKNN_algorithm(train_data,train_target,test_data,test_target);//要验证的算法
end
//上述结果为输出算法MLKNN的几个验证指标及最后一轮验证的输出和结果矩阵，每个指标都是一个k元素的行向量