Recent research in urban visual analytics has focused on crowdsourcing judgements of qualities such as beauty and safety, for large-scale city-wide predictive evaluation. This study examines the extent and nature of inter-subject variability in such judgements and argues that these subjective qualities often have low generalizability across individuals. We conducted an online study involving 94 participants across 19 countries, where subjects arranged streetscape scenes on visual scales of beauty and safety. An analysis of the arrangements revealed very low inter-subject consistency, including within demographic groups based on age, sex, race and nationality. K-means clustering also revealed large clusters with contradictory judgements with respect to visual features. There was however higher intra-subject consistency when rating the same scenes twice. Based on these findings, we recommend a cautious approach to the use of “average” crowdsourced judgements of urban qualities, and encourage the adoption of subject-specific prediction when evaluating such qualities at scale.