Infants' object play is closely tied to language learning and 3D object understanding. How does this kind of embodied visual experience support infants' abilities to categorize objects in the wild?'' We address this question using domain adaptation, a machine learning framework for studying distribution shifts, i.e., how to transfer knowledge learned from one data distribution to a related but different distribution. We formalize a specific distribution shift problem inspired by infant visual learning---the VI-Shift problem---which mimics the tradeoff between object instances and viewpoints in these two regimes of visual experience. We study the VI-Shift problem through the lens of domain adaptation in deep learning architectures, in particular using novel metrics to demonstrate how the clusterability of learned features contributes to robust generalization. We show that two classic domain adaptation methods do not perform well on the VI-Shift problem, and we demonstrate a novel loss function that improves performance by leveraging some of the distinctive visual characteristics of embodied object play experiences. Our results illustrate one potential learning route through which the distinctive visual properties of embodied object experience can boost robust generalization.