It is possible to infer the mass and spin of the remnant black hole from binary black hole mergers by comparing the ringdown gravitational wave signal to results from studies of perturbed Kerr spacetimes. Typically these studies are based on the fundamental quasinormal mode of the dominant l=m=2 harmonic. By modeling the ringdown of accurate numerical relativity simulations, we find that the fundamental mode alone is insufficient to recover the true underlying mass and spin, unless the analysis is started very late in the ringdown. Including higher overtones associated with this l=m=2 harmonic resolves this issue, and provides an unbiased estimate of the true remnant parameters. Further, including overtones allows for the modeling of the ringdown signal for all times beyond the peak strain amplitude, indicating that the linear quasinormal regime starts much sooner than previously expected. A model for the ringdown beginning at the peak strain amplitude can exploit the higher signal-to-noise ratio in detectors, reducing uncertainties in the extracted remnant quantities. Tests of the no-hair theorem should consider incorporating overtones in the analysis.