Spatial memory is a core cognitive function of many mobile animals. The study of spatial cognition is highly interdisciplinary and different approaches have led to discordant hypotheses about spatial memory. The neuroscientific discovery of place cells and grid cells in the rodent hippocampus and entorhinal cortex led to the highly influential “cognitive map” hypothesis. Conversely, behavioral evidence led to the “cognitive graph” hypothesis that suggests memory is distortion-prone. We argue that between-species differences in sensory and perceptual systems cause vision to play a predominant role in primates. We build on previous modeling work by developing a visual version of the SR model to show that it can provide a unified framework to account for seemingly disparate findings from the brain and behavior, thus providing evidence for our hypothesis that primate (e.g., human) spatial cognition is driven by the hippocampal system, which instantiates a visually-guided cognitive graph.