Errors and residuals in statistics

Statistical errors and residuals occur because measurement is never exact.

It is not possible to do an exact measurement, but it is possible to say how accurate a measurement is. One can measure the same thing again and again, and collect all the data together. This allows us to do statistics on the data. What is meant by errors and residuals is the difference between the observed or measured value and the real value, which is unknown.

If there is only one random variable, the difference between statistical errors and residuals is the difference between the mean of the population against the mean of the (observed) sample. In that case the residual is the difference between what the probability distribution says, and what was actually measured.

Suppose there is an experiment to measure the height of 21-year-old men from a certain area. The mean of the distribution is 1.75 m. If one man chosen at random is 1.80 m tall, the "(statistical) error" is 0.05 m (5 cm); if he is 1.70 tall, the error is −5 cm.

A residual (or fitting error), on the other hand, is an observable estimate of the unobservable statistical error. The simplest case involves a random sample of n men whose heights are measured. The sample mean is used as an estimate of the population mean. Then we have:

  • The difference between the height of each man in the sample and the unobservable population mean is a statistical error, and
  • The difference between the height of each man in the sample and the observable sample mean is a residual.

The sum of the residuals within a random sample must be zero. The residuals are therefore not independent. The sum of the statistical errors within a random sample need not be zero; the statistical errors are independent random variables if the individuals are chosen from the population independently.

In sum:

  • Residuals are observable; statistical errors are not.
  • Statistical errors are often independent of each other; residuals are not (at least in the simple situation described above, and in most others).