United States

We investigated how verbal communication with a robot differs from talking to a human in terms of brain activity by analysing an open-source fMRI dataset. We focused on modeling conversational dynamics rather than conversation as a whole, by analysing fine-grained events, in particular turn initiation. The results indicate that turn initiation in a conversation with a human involves higher activation in auditory and visual cortex than turn initiation with a robot. Conversely, listening to the robot showed higher engagement of auditory cortex than listening to a human. We suggest that verbal and non-verbal turn-taking cues provided by the human agent engage more cognitive processing for picking up the turn. On the other hand, listening to a robot agent requires more processing than listening to a human. Both findings suggest that the accurate simulation of appropriate turn-taking cues and behaviors will help robots to establish more natural conversation dynamics and that the use of brain imaging can provide valuable objective measurements for assessing user states in human-robot interaction.

CogSci 2023

Investigating Conversational Dynamics in Human-Robot Interaction with fMRI

poster

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

It has been argued that patterns of cross-linguistic variation in the semantic categories labelled by individual words are a result of a trade-off between cognitive pressures (so as to be simple to learn and use) and communicative pressures (so as to be efficient in communication). However, the question of what exact mechanisms control this trade-off has been left largely unanswered. We argue that one factor could be the extent to which referential contexts at the level of local interactions are similar or different across users of a category system. To test this hypothesis we propose a hierarchical Bayesian model for communication in a multidimensional meaning space, in which agents actively consider spatial similarity relations during interaction. Our models predict that less variability in contexts across interactions induces categories with lower communicative cost, while more variable contexts across partners are more strongly associated with category systems with lower cognitive cost.

Degree of heterogeneity in the contexts of language users mediates the cognitive-communicative trade-off in semantic categorization

Category learning is a core competence for minimizing cognitive load and optimizing decision-making. An identical problem can be solved by employing a rule-based or a similarity-based strategy. This research examined whether the use of the two strategies was supported by common or distinct neural substrates. We conducted a category learning experiment with rule-plus-similarity stimuli using EEG-fNIRS fusion methodology. Participants learned two artificial categories using either a rule- or similarity-based strategy. The results showed a common visual-perceptual-analysis process and distinct decision-making processes between the uses of the two strategies. Larger P300 and N400 amplitudes and Wernicke's area activation indicated that hypotheses testing and verbal rule abstraction processes were critical for rule-based categorization. In contrast, increased frontopolar cortex activity indicated that integration of multiple dimensions was critical for similarity-based categorization. These results were consistent with COVIS theory, implying an explicit system in rule-based category learning whereas an implicit system in similarity-based learning.

Common and distinct neural bases for rule- and similarity-based category learning

When making category decisions, humans sample features following their dynamic informativeness. Attention optimization models successfully predict these categorization behaviors, but optimization is not the only solution. Alternatively, categorization can be viewed as a Reinforcement Learning (RL) task in which learners sample information based on its expected utility. However, RL models of information sampling have high computational load, even though human learners solve this problem on the millisecond timescale. Therefore, we propose ATHENA-RSS, a model that implements reward-based information search in a more computationally efficient way, via the rapid sequential storage of memories and recurrent retrieval. To test the model, we conducted an experiment where participants (N = 99) learned hierarchically structured categories by uncovering stimulus features. We then conducted a simulation study, where ATHENA-RSS successfully reproduced all search patterns exhibited by participants. We conclude that rapid sequential storage and recurrent memory retrieval were sufficient to achieve human-like information sampling in this task.

Dynamic Information Sampling via Rapid Sequential Storage and Recurrence

The ability to selectively attend to stimuli increases the efficiency of learning. However, learning traps can develop when attention prematurely narrows to a subset of the features that predict outcomes, resulting in suboptimal decisions. The current work investigated the potential for learning traps to be particularly damaging in dynamic environments, where the features that predict rewards and losses change during learning. Two experiments (N=316) found that when learners received choice-contingent feedback, they frequently fell into a learning trap, using a suboptimal categorisation rule. Critically, these learners were unlikely to detect a subsequent rule change nor learn the new optimal rule. This change blindness was not attenuated by priming participants to expect change. These results show that the pernicious effects of learning traps are amplified in dynamic environments.

Learning traps lead to change blindness in dynamic environments

Data visualizations are powerful tools for communicating quantitative information. While prior work has focused on how experts design informative graphs, little is known about the intuitions non-experts have about what makes a graph effective for communicating a specific message. In the current study, we asked participants (N=398) which of 8 graphs would be most useful for answering a particular question, where all graphs were generated from the same dataset but varied in how data were arranged. We tested the degree to which participants based their decisions on sensitivity to how easily other participants (N=542) would actually be able to answer that question with that graph. Our results suggest that while people were biased towards graphs that were minimally informative (i.e., contained the relevant variables), their decisions did not necessarily reflect sensitivity to more graded, but systematic variation in actual graph comprehension performance.

How do communicative goals guide which data visualizations people think are effective?

The risk–reward correlation refers to a tendency to infer the inverse proportionality between probability of an outcome and the outcome's magnitude and is considered to reflect the statistical structure of gambling environments. Existing studies have demonstrated this inverse proportionality using numerical probabilities. However, verbal probability phrases can also convey contextual information about uncertainty without numbers, and express contextual information that is not contained by numerical probabilities. Specifically, verbal probabilities have positive and negative directionalities that can attune the listener to the occurrence or non-occurrence of an outcome and consequently, the listener’s decision making. Therefore, this study aimed to examine whether the risk–reward correlation also hold in verbal probability phrases, by conducting two empirical studies in which the participants were asked to estimate the reward values and winning probability of gambles expressed by verbal probability phrases such as “certain” or “impossible.” The results indicate that risk–reward correlation also hold in verbal probability phrases, and that negative probability phrases were associated with the larger rewards than the positive probability phrases.

Does “never” implicate a larger reward than “possible”? : Risk–reward correlation in verbal probability phrases

Theories in cognitive science are primarily aimed at explaining human behavior in general, appealing to universal constructs such as perception or attention. When it is considered, modeling of individual differences is typically performed by adapting model parameters. The implicit assumption of this standard approach is that people are relatively similar, employing the same basic cognitive processes in a given problem domain. In this work, we consider a broader evaluation of the way in which people may differ. We evaluate 23 models of risky choice on around 300 individuals, and find that most models---spanning various constructs from heuristic rules and attention to regret and subjective perception---explain the behavior of different subpopulations of individuals. These results may account for part of the difficulty in obtaining a single elegant explanation of behavior in some long-studied domains, and suggest a more serious consideration of individual variability in theory comparisons going forward.

To each their own theory: Exploring the limits of individual differences in decisions under risk

Probability judgments appear to violate basic axioms of probability theory, which seems to contradict with the recent successes of Bayesian models of cognition. To explain these violations, we propose the Quantum Sequential Sampler model, which combines quantum probability for explaining conjunction and disjunction fallacies, and a sequential sampling model that maps subjective quantum probabilities into responses. Our model explains probability judgments by a dynamical process, and achieves state-of-the-art performance in the biggest dataset for probability judgments to-date. Comparing with existing Bayesian models, our model predicts both probability judgments and violations of probability identities better.

Quantum Sequential Sampler: a dynamical model for human probability reasoning and judgments

Social cognition is shaped by cultural and social-psychological contexts. Depending on the social cues displayed, the cognitive processes that emerge during embodied social interactions can vary. This study explored the influence of gender diverse social cues on interpersonal coordination and decision making during dyadic interaction in virtual reality. Participants embodied either a transgender or cisgender avatar and interacted with a transgender confederate (human-controlled virtual agent). Our results suggest that transgender embodied women and men displayed more interpersonal affiliation via their coordination dynamics with the confederate after an explicit gender identity disclosure. Additionally, we found that cognitive flexibility for ambiguous decisions was influenced by attributions of responsibility made toward the confederate.

Embodied Transgender Interactions: Exploring Dyadic Interpersonal Coordination and Decision Making in Virtual Reality

Through training using distorted vision, the perception of the trained limb position is shifted based on visual information. We investigated whether and how such proprioceptive recalibration transfers to untrained limbs. The results of experiments using human-like virtual limbs confirmed that transfer to some limbs occurred. The manner in which the transfer occurred varied according to the participants' body posture and the type of trained limb. When the participants sat, the recalibration transferred from one arm to another arm symmetrically around the body midline. Conversely, in the case of a sitting leg and standing arm, it was directly copied to another leg or arm.

Premium content

Investigating Conversational Dynamics in Human-Robot Interaction with fMRI

Next from CogSci 2023

Degree of heterogeneity in the contexts of language users mediates the cognitive-communicative trade-off in semantic categorization

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES