Judging similarity between pieces of music is critical for interacting with it in everyday life. But how do musical and emotional features drive our subjective judgments of similarity? Much of the previous work has focused on low-level features and has largely ignored the impact of lyrics and emotion on perceived similarity. Here, we tested the influence of a comprehensive set of musical and emotional features on similarity, using original popular songs and cover versions to match clips on lyrics and melody. We found that tempo most strongly predicts lower similarity ratings, but key, voice type, and timbre differences predict similarity in an interactive manner. While emotional arousal did not predict similarity above and beyond tempo, emotional valence did. Together, these results suggest that both musical and emotional factors influence judgments of similarity, shedding light on the fine-grained explanatory mechanisms of listeners’ everyday impressions of popular music.