Training, Validation, and Test Sets¶

You should test a model against a different set of examples than those used to train the model. As you'll learn a little later, testing on different examples is stronger proof of your model's fitness than testing on the same set of examples. Where do you get those different examples? Traditionally in machine learning, you get those different examples by splitting the original dataset. You might assume, therefore, that you should split the original dataset into two subsets:

A training set that the model trains on.
A test set for evaluation of the trained model.

Figure 8 diagram showing a horizontal bar representing a dataset split into a large training set and a smaller test set.

Figure 8. Not an optimal split.

Dividing the dataset into two sets is a decent idea, but a better approach is to divide the dataset into three subsets. In addition to the training set and the test set, the third subset is:

A validation set performs the initial testing on the model as it is being trained.

Figure 9 horizontal bar chart showing a dataset split into a large training set, a smaller validation set, and an equally small test set.

Figure 9. A much better split.

Use the validation set to evaluate results from the training set. After repeated use of the validation set suggests that your model is making good predictions, use the test set to double-check your model.

The following figure suggests this workflow. In the figure, "Tweak model" means adjusting anything about the model — from changing the learning rate, to adding or removing features, to designing a completely new model from scratch.

Figure 10 flowchart illustrating the iterative cycle of training on a training set, evaluating on a validation set, and tweaking the model, followed by picking the best model and confirming results on a test set.

Figure 10. A good workflow for development and testing.

The workflow shown in Figure 10 is optimal, but even with that workflow, test sets and validation sets still "wear out" with repeated use. That is, the more you use the same data to make decisions about hyperparameter settings or other model improvements, the less confidence that the model will make good predictions on new data. For this reason, it's a good idea to collect more data to "refresh" the test set and validation set. Starting anew is a great reset.

Additional problems with test sets¶

After splitting a dataset into training, validation, and test sets, delete any examples in the validation set or test set that are duplicates of examples in the training set. The only fair test of a model is against new examples, not duplicates.

For example, consider a model that predicts whether an email is spam, using the subject line, email body, and sender's email address as features. Suppose you divide the data into training and test sets, with an 80-20 split. After training, the model achieves 99% precision on both the training set and the test set. You'd probably expect a lower precision on the test set, so you take another look at the data and discover that many of the examples in the test set are duplicates of examples in the training set. The problem is that you neglected to scrub duplicate entries for the same spam email from your input database before splitting the data. You've inadvertently trained on some of your test data.

In summary, a good test set or validation set meets all of the following criteria:

Large enough to yield statistically significant testing results.
Representative of the dataset as a whole. In other words, don't pick a test set with different characteristics than the training set.
Representative of the real-world data that the model will encounter as part of its business purpose.
Zero examples duplicated in the training set.