When we communicate with others, we often repeat aspects of each other’s communicative behavior such as sentence structures and words. Such behavioral alignment has been mostly studied for speech or text. Yet, language use is mostly multimodal, flexibly using speech and gestures to convey messages. Here, we explore the use of alignment in speech (words) and co-speech gestures (iconic gestures) in a referential communication task aimed at finding labels for novel objects in interaction. In particular, we investigate how people flexibly use lexical and gestural alignment to create shared labels for novel objects and whether alignment in speech and gesture are related over time. The present study shows that interlocutors not only establish shared labels multimodally but also keep aligning in words and iconic gestures over the interaction. We also show that the amount of lexical alignment positively correlates with the amount of gestural alignment over time, suggesting a close relationship between alignment in the vocal and manual modalities.

Authors:

Sho Akamine: Max Planck Institute for Psycholinguistics; Esam Ghaleb: University of Amsterdam; Marlou Rasenberg: Meertens Institute; Raquel Fernandez: University of Amsterdam; Antje Meyer: Max Planck Institute for Psycholinguistics; Asli Özyürek: Max Planck Institute