A limitation of genomic prediction models is that they do not readily incorporate or take into account genotype by environment effects, which decrease the prediction accuracy of genomic selection (GS) across different growing conditions. In a large collaborative project with commercial breeding companies, we tested whether GS prediction accuracy could be improved by taking into account the environmental influence on phenotype by incorporating a large number of phenomic markers using high-throughput field phenotyping methods. To assess the effectiveness of the approach on a scale approximating that of a practical breeding programme, 44 winter wheat elite populations comprising 2,994 lines were grown on two sites contrasting in soil type over two years. Remote sensing data were collected from multi- and hyperspectral cameras mounted on drones and piloted aircraft, and were combined with ground-based visual crop assessment scores and ultra-local spatial variations in soil physical properties. The predictive power for grain yield was tested with or without incorporating genome-wide marker data sets. Models using phenomic traits alone had a greater predictive value (R2 = 0.39–0.47) than genomic data (approximately R2 = 0.1). The average improvement in predictive power by combining trait and marker data was 6%–12% over the best phenomic-only model, and performed best when data from one full location was used to predict the yield on an entire second location. The results suggest that genetic gain can be increased by utilising phenomic data, perhaps most efficiently at early stages of the breeding cycle.