We introduce an approach and method that helps explain how humans compare images. We produce Alignment Importance Score (AIS) heatmaps from deep-vision models, focusing on feature maps in the deepest convolutional layer. The AIS reflects a feature-map's unique contribution to the alignment of Deep Neural Network's (DNN) representational geometry and that of humans. We first validate the AIS by showing that prediction of out-of-sample human similarity judgments is improved when constructing representations using only higher-AIS feature maps identified by a training set. We then compute image-specific heatmaps that visually indicate the areas that correspond to feature-maps with higher AIS scores. These maps provide an intuitive explanation of which image areas are more important when it is compared to other images in a cohort. We find that these heatmaps have good correspondence with saliency maps produced by models trained to predict gaze location. However, in some exceptions, meaningful differences emerge, as the relevant dimensions for comparison are not necessarily the most visually salient. To conclude, by using AIS it is possible to improve prediction of human similarity judgments from DNN embeddings, and to depict the relevant information in image space.

Authors:

Nhut Truong: University of Trento; Dario Pesenti: University of Trento; Uri Hasson: University of Trento