By understanding the anatomical and functional organization of the visual system at birth we can gain critical insights into the mechanisms that support early developing perceptual and cognitive abilities. In the current study, we used resting-state fMRI and diffusion tensor imaging (DTI) to understand the cortical and subcortical organization of the visual system in newborn human infants (37-42 weeks gestation). We used an adult probabilistic atlas of retinotopic maps (Wang et al. 2015) to identify putative visual areas in neonates. To achieve this, each neonate’s cortical surface was registered to an adult cortical surface template and the adult probabilistic atlas was projected onto each neonate’s cortical surface. Functional correlations between cortical areas revealed that the infant visual system exhibits an adult-like hierarchical organization, with distinct clusters for regions of the occipital cortex, as well as ventral, lateral, and dorsal visual pathways. Direct comparisons between neonates and adults revealed that correlation patterns within occipital and dorsal areas were more adult-like than ventral and lateral areas. We then examined the degree to which developmental motifs discovered in the connectivity between cortical visual areas were mirrored in the organization of structural connections between each cortical area and the pulvinar – a subcortical structure that is extensively interconnected with the entire visual cortex in adults and plays a crucial role in visual processing (Arcaro et al., 2018). Probabilistic tractography analyses reliably identified white matter pathways between the pulvinar and each cortical visual area. These connections showed region-level specificity and overlapped with homologous pathways of adults. However, we found developmental differences within the fine-grained connectivity pattern of the pulvinar. Although the coarse connectivity organization for all neonate networks were qualitatively similar to those of adults, the connectivity maps for ventral visual areas were immature and did not show strong specificity within the pulvinar. Altogether, our findings indicate that the large-scale anatomical and functional organization of the visual system is established by birth. However, they also revealed developmental differences in the maturity of different pathways with the dorsal pathway maturing earlier than the ventral pathway. Thus, by understanding the initial organization of the human visual system, we may shed light on the biological constraints that shape visual development.