Knowledge of the nitrogen-vacancy (NV) center formation kinetics in diamond is critical to engineering sensors and quantum information devices based on this defect. Here we utilize the longitudinal tracking of single NV centers to elucidate NV defect kinetics during high-temperature annealing from 800--1100 ${}^{\ensuremath{\circ}}\mathrm{C}$ in high-purity chemical-vapor-deposition diamond. We observe three phenomena which can coexist: NV formation, NV quenching, and NV orientation changes. Of relevance to NV-based applications, a 6- to 24-fold enhancement in the NV density, in the absence of sample irradiation, is observed by annealing at 980 ${}^{\ensuremath{\circ}}\mathrm{C}$, and NV orientation changes are observed at 1050 ${}^{\ensuremath{\circ}}\mathrm{C}$. With respect to the fundamental understanding of defect kinetics in ultrapure diamond, our results indicate a significant vacancy source can be activated for NV creation between 950--980 ${}^{\ensuremath{\circ}}\mathrm{C}$ and suggest that native hydrogen from ${\mathrm{NVH}}_{y}$ complexes plays a dominant role in NV quenching, supported by recent ab initio calculations. Finally, the direct observation of orientation changes allows us to estimate an NV diffusion barrier of 4.7 $\ifmmode\pm\else\textpm\fi{}$ 0.9 eV.