The Art of Statistics: Learning from Data

by David Spiegelhalter

survivors that are correctly predicted is known as the sensitivity of the algorithm, while the percentage of true non-survivors that are correctly predicted is known as the specificity. These terms arise from medical diagnostic testing. Although the overall accuracy is simple to express, it is a very crude measure of performance and takes no account of the confidence with which a prediction is made. If we look at the tips of the branches of the classification tree, we can see that the discrimination of the training data is not perfect, and at all branches there are some who survive and some not. The crude allocation rule simply chooses the outcome in the majority, but instead we could assign to new cases a probability of surviving corresponding to the proportion in the training set. For example, someone with the title ‘Mr’ could be given a probability of 16% of surviving, rather than a simple categorical prediction that they will not survive.

Sensitivity & specificity of algorithm