United States

Predictability is a prerequisite for effective human control of artificial intelligence (AI). The inability to predict malfunctioning of AI, for example, impedes timely human intervention. In this paper, we empirically investigate how AI’s predictability compares to the predictability of humans in a real-effort task. We show that humans are worse at predicting AI performance than at predicting human performance. Importantly, participants are not aware of the differences in relative predictability of AI and overestimate their prediction skills. These results raise doubts about the human ability to effectively exercise control of AI — at least in certain contexts.

CogSci 2023

Human control redressed: Comparing AI-to-human vs. human-to-human predictability in a real-effort task

poster

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Large language models have been shown useful in multiple
domains including conversational agents, education, and explainable AI. ChatGPT is a large language model developed
by OpenAI as a conversational agent. ChatGPT was trained on
data generated by humans and by receiving human feedback.
This training process results in a bias toward humans’ traits
and preferences. In this paper, we stress multiple biases of
ChatGPT, and show that its responses are closer to human responses than to fully rational responses. We Begin by showing
a very high correlation between the frequency of digits generated by ChatGPT and humans’ favorite numbers, with the
most frequent digit generated by ChatGPT, matching humans'
most favorable number, 7. We continue by showing that ChatGPT’s responses in several social experiments are much closer
to those of humans’ than to those of fully rational agents. Finally, we show that several cognitive biases, known in humans,
are also present in ChatGPT’s responses.



ChatGPT: More Human-Like Than Computer-Like, but Not Necessarily in a Good Way

Lexical Decision Task is one of the most widely used experimental paradigms for studying word recognition that allows researchers to make inferences about lexical processing and lexical representations of words and non-words. Evidence Accumulation Models (EAM) have been successfully used to model this task. Despite the accurate prediction of participants’ reaction time and accuracy, these models lack a mechanism for representing the lexical features of words and non-words. Incorporating lexical features directly into an EAM can open up new and better ways to study lexical processing and lexical representations. For this purpose, we developed two models by combining FastText and BERT models, with the race-diffusion model. In this framework representations of words and non-words are generated by FastText or BERT models and transformed into the race-diffusion model’s drift rate. Results show that a combination of FastText and race diffusion model is a promising approach for modeling the lexical decision task.


Modeling Human Lexical Decision-Making with Artificial Neural Networks and Evidence Accumulation Models

Recent advances in large language models (LLMs) have shown that it is possible to derive expressive meaning representations by leveraging linguistic associations in large-scale natural language data. These potentially Gestalt representations have enabled state-of-the-art performance for many practical applications. It would appear that we are on a pathway to empirically deriving a robust and expressive computable semantics. A key question that arises is how far can language data alone enable computers to understand the necessary truth about the physical world? Attention to this question is warranted because our future interactions with intelligent machines depends on how well our techniques correctly represent and process the concepts (objects, properties, and processes) that humans commonly observe to be true. After reviewing existing protocols, we explore this question using a novel and tightly controlled reasoning test and compare human norms against the LLMs. Our findings highlight the categories of common sense relations models could learn directly from data and areas of weakness which require moving beyond associative learning.


On Reality and the Limits of Language Data: Aligning LLMs with Human Norms

Widely considered a cornerstone of human morality, trust shapes many aspects of human social interactions. In this work, we show that artificial intelligence (AI) agents learn to trust ― and rationally adapt the extent to which they trust ― by interacting with their counterparts in the trust game (Berg, Dickhaut, & McCabe, 1995), the canonical task for studying trust in behavioral and brain sciences. Leveraging reinforcement learning (RL) to train our AI agents, we systematically investigate learning trust under various parameterizations of this task and show that trust can arise from pure self-interest. Our work sheds new light on the computational underpinnings of trust, taking us a step closer to developing AI agents whose values are aligned with human values.


AI Agents Learn to Trust

Conversational AI devices are increasingly present in our daily lives and used by both adults and children to ask questions, play and learn. However, their appearance oddly blurs the line between agents and objects—they both look and behave like inanimate objects yet engage in polite helpful conversations—raising questions about how humans conceptualize such novel entities. In this work we explore adult and children's attributions of the AI's "mental states" through a variant of a false belief task. We establish that, while adults successfully differentiate the individual minds of people from the shared "minds" of conversational AI agents, children through age 8 reason about AI agent minds much as they do human minds: as separate entities that do not share information. These findings suggest conversational AI agents are rather unintuitive to young children, (i.e. understanding develops sometime after age 8) and we discuss implications of this for childhood AI education.


Theory of AI Minds: Children's reasoning about the "mental states" of conversational AI agents

Explicit similarity judgments tend to emphasize relational information more than do difference judgments. We propose and test the hypothesis that this asymmetry arises because human reasoners represent the relation different as the negation of the relation same, so that processing difference is more cognitively demanding than processing similarity. For both verbal comparisons between word pairs, and visual comparisons between sets of geometric shapes, we asked participants to select which of two options was either more similar to or more different from a standard. On unambiguous trials, one option was unambiguously more similar to the standard; on ambiguous trials, one option was more featurally similar to the standard, whereas the other was more relationally similar. Given the higher cognitive complexity of assessing relational similarity, we predicted that detecting relational difference would be particularly demanding. We found that participants (1) had more difficulty accurately detecting relational difference than they did relational similarity on unambiguous trials, and (2) tended to emphasize relational information more when judging similarity than when judging difference on ambiguous trials. The latter finding was captured by a computational model of comparison that weights relational information more heavily for similarity than for difference judgments. Our results provide convergent evidence for a representational asymmetry between the relations same and different.


Asymmetry in similarity and difference judgments results from asymmetry in the complexity of the relations same and different

Preschoolers have difficulties in relational reasoning tasks, which are usually attributed to their object focus. However, studies directly testing children’s object focus yielded inconsistent results. Indirect investigations supported the role of object focus, yet the differences in study procedures might hinder a clear conclusion. To address these issues, in two experiments, we manipulated task characteristics that accommodated the previous studies and tested whether children’s use of object property words (e.g., color, shape) was directly associated with their performance in the relational match-to-sample task (RMTS). Our results showed that stimuli with basic shapes enhanced preschoolers’ performance in RMTS, and children’s use of object property words dampened their performance. These findings underline considering task- and child-related factors when investigating children’s relational reasoning in particular and cognitive performance in general.


Object Focus and Preschoolers’ Relational Reasoning

Past work has found that dogs perceive and respond to certain characteristics of items, specifically the object's familiarity and value when making choices. However, in the real world, these characteristics don't exist in isolation, and understanding the interaction between familiarity and valuation as they relate to object choice can provide insight into how dogs make decisions. We aimed to explore how item familiarity and value intersect to form dogs’ preferences for one food over another in a two-alternative forced choice task. We found that dogs choices were driven only by the value of the food item, and the familiarity of the item did not impact choice behavior. Determining what motivates dogs and contributes to their preferences has implications for understanding decision-making at large, as well as for advancing canine science.


The Impact of Value and Familiarity on Dogs' Food Preferences

In humans, simple 2D visual displays of launching events (“Michottean launches”) can evoke the impression of causality. Direct launching events are regarded as causal, which are contrasted with similar non-causal events where a temporal and/or spatial gap is added between the movements of the two objects. This ability to distinguish between causal and non-causal events is perceptual in nature and develops early and preverbally in infancy. In the present study we investigated the evolutionary origins of this phenomenon and tested whether Guinea baboons (Papio papio) perceive causality in launching events. We used a discrimination and categorisation task of causal and non-causal events. Our results indicate that Guinea baboons discriminate between different Michottean events, but we did not find a learning advantage for a categorisation based on causality. This implies that they focused on the spatio- and temporal gap to achieve accurate categorisation, but not on causality per se. Currently we cannot rule out that Guinea baboons have causal representations of Michottean events, but our findings point to a feature-based discrimination strategy in a sorting task.


Causal perception in Guinea Baboons (Papio papio)

Philosophy drawing on studies of animal behavior has paved a middle-way path regarding natural and non-natural meaning (Bar-On, 2021; Bar-On & Moore, 2017; Green, 2019; Moore, 2018). Elaborating the question of communicative intention goes hand-in-hand with growing appreciation of the reaches of non-human animal mindedness (Andrews & Beck, 2018; Merritt, 2021), pushing the plausible bounds of higher-order cognitive capacities beyond the singular province of homo sapiens. The question of relating human languaging to non-human languaging is enriched through a dialogue between enactive philosophy of social cognition and post-Gricean discourse. Speaker meaning presupposes coherent, stable, individual, internal, and prior intention as a cognitive or mental state. The enactive theory of social cognition, participatory sense-making (De Jaegher & Di Paolo, 2007), calls all of these premises into serious question. Prioritizing the role of interactions in cognition brings attention to the multitude of possible partners humans had and have in communication and sense-making engagements.


Premium content

Human control redressed: Comparing AI-to-human vs. human-to-human predictability in a real-effort task

Next from CogSci 2023

ChatGPT: More Human-Like Than Computer-Like, but Not Necessarily in a Good Way

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES