Canada

Online data collection has become a prominent option due to the COVID-19 pandemic. It is crucial to understand to what extent online studies can be compared with face-to-face studies, particularly in multimodal language research on which the modes of communication have a crucial effect. This study investigated multimodal communication across face-to-face and videoconferencing settings, focusing on gesture production and speech disfluency in a daily routine description task (N=64). Results suggested that overall disfluency rate was higher for those who communicated via videoconferencing than those who communicated face-to-face. The use of specific disfluency types also differed across the two settings, signaling an interplay between cognitive and communicative strategies. Overall gesture frequency and iconic gesture use were comparable across the two settings. Iconic gesture use negatively predicted the overall disfluency rate, regardless of the setting. Using different contexts is required to understand whether multimodal language differs between face-to-face and online communication.

CogSci 2022

Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency

poster

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





Gestures facilitate speech production by helping speakers reduce cognitive load. Studies on gesture-speech interaction mostly examined the effect of representational gestures on spatial contexts. However, abstract deictics (e.g., pointing at objects that are not visually present) might also have a role in facilitating cognitive processes. The present study investigated the effect of gestures on disfluency rates by presenting a narrative task in three conditions: spontaneous, restricted, and encouraged gesture use. We found that disfluency rates across three conditions did not significantly differ. The use of abstract deictics in the spontaneous gesture use condition was a significant predictor of disfluency rates in the gesture restricted condition. Results indicate that gestures’ facilitative roles might be manifested differently depending on the context. Abstract deictics might also benefit speakers, especially in a narrative context. Studying abstract deictics can provide new insights on gesture and speech production interaction.

Gesture and Speech Disfluency in Narrative Context: Disfluency Rates in Spontaneous, Restricted, and Encouraged Gesture Conditions

Myriad linguistic cues, subtle and otherwise, have been shown to shape thinking and reasoning. Such linguistic framing effects are attributed variously to irrational biases, affective processes, and analogy. Across two studies, we explored pragmatic reasoning (PR) as a potential unifying mechanism underlying disparate framing effects. In Study 1, we adapted a measure of PR used in attribute framing research, assessing the ability to infer information subtly communicated by a speaker’s choice of frame. Our measure captured individual differences in PR distinct from cognitive reflection and mentalizing ability. In Study 2, PR predicted the effect of identity labels on attitudes towards illegal immigration, but not other framing effects. Two such effects were driven, however, by participants’ explicit recognition of the influence of the framing language. These results suggest that framing effects may indeed be mediated by PR, but that our measure of PR does not fully capture its mediating role.

Exploring the Role of Pragmatic Reasoning in Linguistic Framing Effects

Accessibility plays a major role in speech production. Here we investigate and measure four factors that influence speakers to produce one word over another more optimal word form. Three experiments asked participants to label images of insects and instruments. Participants were incentivized to produce an accurate specific label (e.g., bee), over a more general label (e.g., insect), so that specific labels were more optimal. Each of three experiments manipulated a different factor that could influence accessibility – word frequency, priming, and interference – and all experiments additionally varied whether labels had to be produced under time pressure or not. Results showed that each variable significantly influenced the accessibility of labels: participants produced more specific labels when those labels were higher frequency, when they were primed, when a visually-similar label had not been primed, and when participants were unconstrained by time pressure. These findings demonstrate that multiple factors influence the accessibility of familiar words during production, regularly leading participants to rely on “good-enough” rather than optimal options to convey their message.

Accessibility factors that lead to good-enough language production

Sentence comprehension involves simultaneous processes such as maintaining and integrating different types of verbal representations. As such, it has been argued that sentence comprehension relies on working memory (WM). Some findings suggest that semantic (word meaning) WM rather than phonological (speech sound) WM is critical for comprehension. This study took a case-series multiple regression approach to examine the relationship between sentence comprehension and WM for 56 individuals with aphasia. We examined the independent contribution of phonological and semantic WM in predicting comprehension for higher WM target sentences relative to matched lesser WM sentences, while also controlling for single word processing. We found that only semantic WM had a significant contribution to comprehension for three contrasts. However, for the fourth contrast of trials requiring syntactic processing with those requiring only lexical processing, both WM contributions were significant. The possible backup role of phonological WM for comprehension of role reversals is discussed.

Semantic Working Memory Predicts Relative Clause Sentence Comprehension: A Case Series Approach

Neural networks struggle in continual learning settings from catastrophic forgetting: when trials are blocked, new learning can overwrite the learning from previous blocks. Humans learn effectively in these settings, in some cases even showing an advantage of blocking, suggesting the brain contains mechanisms to overcome this problem. Here, we build on previous work and show that neural networks equipped with a mechanism for cognitive control do not exhibit catastrophic forgetting when trials are blocked. We further show an advantage of blocking over interleaving when there is a bias for active maintenance in the control signal, implying a tradeoff between maintenance and the strength of control. Analyses of map-like representations learned by the networks provided additional insights into these mechanisms. Our work highlights the potential of cognitive control to aid continual learning in neural networks, and offers an explanation for the advantage of blocking that has been observed in humans.

A Neural Network Model of Continual Learning with Cognitive Control

Recent work has shown that the brain organizes abstract, non-spatial relationships between entities into map-like representations. However, an animal’s objectives often depend on only a subset of the features of the environment. Under these circumstances, cognitive control – the capacity to flexibly select the features most relevant in the current context – becomes paramount. Here, we explore the relationship between cognitive control and the geometry of map-like representations by combining fMRI with neural network modeling. We find that brain areas including hippocampus and entorhinal cortex spontaneously organize pairwise relationships into 2D map-like representations, and that this 2D structure was controlled by compressing task-irrelevant dimensions in areas of prefrontal and parietal cortex. Our neural network model reproduced these findings and additionally predicted warping in the geometry along a context-invariant axis. This prediction was confirmed with fMRI, which showed that the degree of warping was correlated with individual differences in cognitive control.

The Geometry of Map-Like Representations under Dynamic Cognitive Control

Distributed vector representations are a key bridging point between connectionist and symbolic representations of cognition. It is unclear how uncertainty should be modelled in systems using such representations. One may place vector-valued distributions over vector representations, although that may assign non-zero probabilities to vector symbols that cannot occur. In this paper we discuss how bundles of symbols in Vector Symbolic Architectures (VSAs) can be understood as defining an object that has a relationship to a probability distribution, and how statements in VSAs can be understood as being analogous to probabilistic statements. We sketch novel designs for networks that compute entropy and mutual information. In this paper we restrict ourselves to operators proposed for Holographic Reduced Representations, and representing real-valued data. However, we suggest that the methods presented in this paper should translate to any VSA where the dot product between fractionally bound symbols induces a valid kernel.

Fractional Binding in Vector Symbolic Architectures as Quasi-Probability Statements

Models in cognitive science are often restricted for the sake of interpretability, and as a result may miss patterns in the data that are instead classified as noise. In contrast, deep neural networks can detect almost any pattern given sufficient data, but have only recently been applied to large-scale data sets and tasks for which there already exist process-level models to compare against. Here, we train deep neural networks to predict human play in 4-in-a-row, a combinatorial game of intermediate complexity, using a data set of 10,874,547 games. We compare these networks to a planning model based on a heuristic function and tree search, and make suggestions for model improvements based on this analysis. This work provides the foundation for estimating a noise ceiling on massive data sets as well as systematically investigating the processes underlying human sequential decision-making.

Improving a model of human planning via large-scale data and deep neural networks

“Slavery ended in the United States because slavery is morally wrong.” This explanation does not seem to fit the typical criteria for explaining an event, since it appeals to ethics rather than causal factors as the reason for this social change. But do people perceive these ethical claims as explanatory, and if so, why? In Study 1, we find that people accept ethical explanations for social change and that this is predicted by their meta-ethical beliefs in moral progress and moral objectivism, suggesting that they treat morality somewhat akin to a causal force. In Study 2, we find that people recognize this relationship between ethical explanations and meta-ethical commitments, using the former to make inferences about individuals’ beliefs in moral progress and objectivism. Together these studies demonstrate that our moral commitments shape our judgments of explanations and that explanations shape our moral inferences about others.

Ethical Explanations

To defend or burnish our moral reputation, we often tell moral narratives. Moral narratives describe morally relevant actions and explanations of those actions, detailing how people acted and why they did so. When and why does communication about moral events include descriptions of peoples’ actions and explanations? We hypothesize that informational, reputational, and presentational goals of narrators shape whether their communication contains clear actions and explanations. We asked a group of “narrators” to communicate with other people following a moral decision. Another group of “audience” members judged them based on their chosen statement. We find that the informational and reputational goals of narrators can explain what information they decide to reveal. Narrators choose what to say based on inferring the audience’s likely perceptions but underestimate how much audiences in fact expect answers to “what” and “why”. Audiences, however, do not always perceive the lack of expected information as indicative of deceit.

Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency

Downloads

Next from CogSci 2022

Gesture and Speech Disfluency in Narrative Context: Disfluency Rates in Spontaneous, Restricted, and Encouraged Gesture Conditions

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES