We investigated how verbal communication with a robot differs from talking to a human in terms of brain activity by analysing an open-source fMRI dataset. We focused on modeling conversational dynamics rather than conversation as a whole, by analysing fine-grained events, in particular turn initiation. The results indicate that turn initiation in a conversation with a human involves higher activation in auditory and visual cortex than turn initiation with a robot. Conversely, listening to the robot showed higher engagement of auditory cortex than listening to a human. We suggest that verbal and non-verbal turn-taking cues provided by the human agent engage more cognitive processing for picking up the turn. On the other hand, listening to a robot agent requires more processing than listening to a human. Both findings suggest that the accurate simulation of appropriate turn-taking cues and behaviors will help robots to establish more natural conversation dynamics and that the use of brain imaging can provide valuable objective measurements for assessing user states in human-robot interaction.

Investigating Conversational Dynamics in Human-Robot Interaction with fMRI

How responsible someone is for an outcome depends on both the causal role of their actions, and what those actions reveal about their moral character. Prior work has successfully modeled people's causal attributions and mental state inferences using planning algorithms assumed to approximate people's intuitive theory of mind about others' behavior. In this paper, we develop a unified computational framework for responsibility judgments in which the same generative planner can model both of these processes. We test our framework on a variety of animated social scenarios in two experiments. Experiment 1 features simple cases of helping and hindering. Experiment 2 features more complex interactions that require recursive reasoning, including cases where one agent affects another by merely signaling their intentions without physically acting on the world. Our model accurately captures participants' counterfactual simulations and intention inferences. Together, these two factors explain responsibility judgments.

A computational framework for responsibility judgments from counterfactual simulations and intention inferences

With AIs becoming prevalent in our daily lives, it is still controversial whether they are a reliable conversational assistant. We, therefore, developed a chatbot, a Friendly-Bot, where users can share interpersonal experiences (e.g., friendship and romantic relationships). We then manipulated chatbot appearances into two types: robot and human-looking. We found that chatbot appearance predicted users' trust. Participants preferred the robot-looking chatbot over the human-looking one. Participants also showed higher trust in robot-looking chatbots when conversing about positive interpersonal relationships. Participants showed a different pattern of trust depending on the appearance condition: positive experience led to higher trust in robot-looking condition, whereas the opposite was observed in human-looking condition. Our findings show how chatbot appearance influences rapport-building in AI-assisted interactions (e.g., counseling).

Friendly-Bot: The Impact of Chatbot Appearance and Relationship Style on User Trust

Modeling human navigation and route selection in complex environments has used path planning in the past and more recently, dynamical models. The current study modeled the navigational movement trajectories of human herder agents tasked with corralling an autonomous target agent into a specified containment zone. Analyses of trajectories of participants completing a first-person herding game revealed that their movement entailed sequential (i) approach and then (ii) corral phases that were invariant to the initial location of the herder and target agents. More importantly, the trajectories of human herders could be captured using a low-dimensional environmentally coupled dynamical model. The novelty of this work is in extending the previous research in modeling navigational behaviors of humans to include moving targets that need to be manipulated into a desired position. Implications of the findings for the development of artificial agents for human-machine herding, as well as more general human-machine teaming are discussed.

Modeling Human Navigation in First-Person Herding Tasks

Abstract reasoning in early childhood is often described as following a "relational shift," over which children become increasingly sensitive to relations. However, recent work has challenged the generality of this account, showing that children in the US and China follow distinct trajectories in a relational match-to-sample task (Carstensen et al., 2019). This difference aligns with multiple cultural and linguistic factors implicated in relational reasoning, in which English speakers in the US and Mandarin speakers in China appear at opposite ends of a continuum spanning from a focus on objects (US) to relations (CN). We explore early relational reasoning in a context that represents a cultural middle ground with a key linguistic similarity (noun spurts) to the US: Korean-learning children in South Korea. In two experiments with 262 Korean children, we document relational reasoning in this novel cultural context, revealing similarities and differences to developmental trajectories in the US and China.

Relational abstraction in early childhood: Three cultures and three trajectories

We investigated the effect of implicit theories (belief about malleability of ability) on help-seeking behavior in peer learning situation. We predicted that entity theorists (those who believe that ability is fixed) are less likely to seek help because they anticipate that peer members would lower their evaluation of competence. We conducted a scenario experiment and required participants to indicate to what extent they would ask questions to the peer member when they encounter an incomprehensive term which was already explained, and others seemed to understand. The results did not support our hypotheses. The results revealed that the interaction between one’s own and perceived others implicit theories predicted intention to ask questions. Specifically, while perceived others’ implicit theory did not affect incremental theorists’ (those who believe that ability is malleable) intention to ask a question, entity theorists were more likely to ask a question when they perceive others as entity theorists.

The effect of implicit theories on help-seeking behavior: Focusing on anticipated evaluation and perceived implicit theories of the peer member.

Intention prediction often plays a crucial role in successful social interaction. Previous studies have attempted to understand this skill by focusing on the role of movement kinematics in isolation. However, this approach is limited as the same kinematics typically map to multiple action possibilities (affordances) and as a result, individual's also employ contextual information to predict others' intentions. In this study we present preliminary findings from a qualitative study aimed at investigating intention prediction in naturalistic contexts. Participants viewed an individual reaching for a cup with one of two object-directed intentions: to drink OR to clear the table. A third non-object-directed intention was also included where the observed individual placed their hand on the table next to the cup. For each intention the contextual information was varied by changing the environmental scene between (1) cups full of juice, (2) almost empty cups, and (3) half-empty cups. The findings reveal that participants perceived the cup's functional (most salient) affordance - drink - for the intention so far as the movement kinematics specified an object-directed intention (drink or clear) with a context that clearly afforded it (full and half-full cups). However, participants were also sensitive to the kinematic differences between the object-directed intentions when the context made the functional affordance seem improbable (almost empty cups). The methodological and theoretical implications of the results with regards to studying intention prediction are discussed.

Predicting intentions: How do we predict other's action intentions?

Recent years have seen remarkable advances in the development and use of Artificial Intelligence (AI) in image classification, driving cars, and writing scientific articles. Although AI can outperform humans in many tasks, there remain domains where humans and AI working together can outperform either working alone. For humans and AI to work together effectively, the human must trust the AI bot to the right degree (calibrated). If the human does not trust the bot sufficiently, or conversely trusts the bot more than is warranted, the human-bot team will not perform as well as they could. We report three experiments examining trust in human-AI teaming. While existing studies typically collect binary responses (to trust, or not to trust), we present a novel paradigm that quantifies trust in a bot-recommendation in a continuous fashion. These data allow better precision, and in the future the development of more refined models of human-bot trust.

Trust in Human-bot Teaming: Applications of the Judge Advisor System

People can understand images that vary in visual abstraction---from detailed illustrations to schematic icons. 
To what degree are current vision algorithms sensitive to such variation when encoding their meaning? 
We first obtained >80K human-generated sketches produced under different time limits (4s, 8s, 16s, 32s; N=5,563 participants) and AI-generated sketches (Vinker et al., 2022) produced under different ink limits (8, 16, 32 strokes) of 2,048 real-world objects spanning 128 categories from the THINGS dataset (Hebart et al.,2019).
We then evaluated how 13 state-of-the-art vision algorithms varying in architecture and training represented the meaning of these sketches, both at the category and exemplar levels, and without additional training.
We found that these models were generally sensitive to variation in visual abstraction, although some models were substantially more sensitive than others.
Together, these findings suggest promising avenues for building better models of human visual abstraction.

Evaluating machine comprehension of sketch meaning at different levels of abstraction

Nudges have been shown to be successful behavioral interventions. However, nudges have also been criticized to be paternalistic and infringe upon decision makers’ (DMs) autonomy. One solution that has been proposed is self-nudges—let DMs nudge themselves. Using default nudges, we tested whether self-nudges are as effective as traditional nudges. Participants were presented with a choice between two videos. We either nudged the participants by preselecting one video as default on their behalf, or asked participants to self-nudge by preselecting a video for themselves prior to their choice. We found that a much larger percentage of participants stayed with the default option and watched the preselected video when they preselected the video themselves (82.5% on average between the two videos) than when we preselected the video for them (59.5% on average between the two videos). Our findings provide preliminary evidence for the efficacy of self-nudges.

Are self-nudges effective?

How do essentialist beliefs about categories arise? We hypothesize that such beliefs are transmitted via language. We subject large language models (LLMs) to vignettes from the literature on essentialist categorization and find that they align well with people when the studies manipulated teleological information – information about what something is for. We examine whether in a classic test of essentialist categorization – the transformation task – LLMs prioritize teleological properties over information about what something looks like, or is made of. Experiments 1 and 2 find that telos and what something is made of matter more than appearance. Experiment 3, manipulates all three factors and finds that what something is for matters more than what it’s made of. Overall, these studies suggest that language alone may be sufficient to give rise to essentialist beliefs, and that information about what something is for matters more.

You are what you’re for: Essentialist categorization in large language models

Interpersonal physiological connection is thought to play a crucial role in group activities. A previous study with participants who had little musical experience has shown that the groups with higher heart rate synchronization were better able to perform simple drumming tasks in a coordinated manner. In the present study, we investigated whether heart-rate synchronization is associated with improvement in a complex harmonic performance of an orchestra. The subject of this study was an orchestra composed of several semi-professional and amateur musicians rehearsing a new original piece. The heart rates of sixteen musicians were simultaneously measured during a joint rehearsal. The results indicate that while the average heart rate of the musicians decreased, heart-rate synchronization among the musicians increased over the course of repeated group rehearsals. For a harmonic performance, the musicians may need to have a physiological connection to each other without their physiological arousal being too high.

Underline digital video library

Investigating Conversational Dynamics in Human-Robot Interaction with fMRI

A computational framework for responsibility judgments from counterfactual simulations and intention inferences

Friendly-Bot: The Impact of Chatbot Appearance and Relationship Style on User Trust

Modeling Human Navigation in First-Person Herding Tasks

Relational abstraction in early childhood: Three cultures and three trajectories

The effect of implicit theories on help-seeking behavior: Focusing on anticipated evaluation and perceived implicit theories of the peer member.

Predicting intentions: How do we predict other's action intentions?

Trust in Human-bot Teaming: Applications of the Judge Advisor System

Evaluating machine comprehension of sketch meaning at different levels of abstraction

Are self-nudges effective?

You are what you’re for: Essentialist categorization in large language models

Interpersonal physiological connection is associated with improved orchestral performance

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES