We propose a minimalistic representational model for the head direction (HD) system, a crucial component of spatial navigation in mammals. Our model leverages the symmetry of the rotation group U(1) and the inherent circular geometry of the head direction. We develop fully connected and convolutional versions of the model, both aiming to learn a high-dimensional representation of head direction that captures essential properties of HD cells. Our learning method results in representations that form a perfect 2D circle when projected onto their principal components, reflecting the underlying geometry of the problem. We also demonstrate that individual dimensions of our learned representation exhibit Gaussian-like tuning profiles akin to biological HD cells. Our model achieves accurate multi-step path integration, effectively updating its internal heading estimate over time. These results demonstrate that simple symmetry-based constraints can yield biologically realistic HD representations, offering new insights into the computational principles underlying spatial navigation in mammals.