Bias of an Estimator

Let be an estimator of a parameter that depends on random variables . The bias of the estimator ^[1] is defined as

Bias of an estiamtor is something like accuracy, though we've got to be careful with that word since it seems overused^[2] at this point.

I like to think of an estimator aiming for a bullseye, in say darts. The bullseye is the goal. Bias of an estimator is like consistently being off target. Some estimators are unbiased, where they consistently hit the target perfectly. In this case, the mathematical definition of bias is equal to zero.

The plot below depicts two bulleyes with six darts each. The bullseye on the left showcases non-zero bias, depicted as low accuracy (and high variation), while the bullseye on the right showcases small bias depicted as low accuracy (and low variation).

Examples

The mean is an unbiased estimator. This is easy to see, by taking its expectation and using the linearity of expectation.

On the other hand, the variance estimator is a biased estimator. The following math adds just the right zero, in just the right spot, to show that this estimator's expectation is not equal to its bullseye, which we'll call . Let's denote the expectation of each as .

Becuase the expectation of the estimator of with denominator is not equal to , this estimator is biased.