United States

Psycholinguistic analyses provide a means of evaluating large language model (LLM) output and making systematic comparisons to human-generated text. These methods can be used to characterize the psycholinguistic properties of LLM output and illustrate areas where LLMs fall short in comparison to human-generated text. In this work, we apply psycholinguistic methods to evaluate the psycholinguistic properties within sentences from student-generated long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT. We use 78 features extracted from Coh-metrix that analyze text cohesion, language, and readability (Graesser, McNamara, Louwerse, &amp; Cai, 2004). We perform an unsupervised cluster analysis. Results illustrate a high level of performance for clustering student-generated and chatGPT-generated analogies. To evaluate which features contribute most to model performance, we use a hierarchical cluster analysis. Results from the hierarchical cluster analysis illustrate several linguistic differences between the two groups.


CogSci 2023

Long-form analogies generated by chatGPT lack human-like psycholinguistic properties

Psycholinguistic analyses provide a means of evaluating large language model (LLM) output and making systematic comparisons to human-generated text. These methods can be used to characterize the psycholinguistic properties of LLM output and illustrate areas where LLMs fall short in comparison to human-generated text. In this work, we apply psycholinguistic methods to evaluate the psycholinguistic properties within sentences from student-generated long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT. We use 78 features extracted from Coh-metrix that analyze text cohesion, language, and readability (Graesser, McNamara, Louwerse, & Cai, 2004). We perform an unsupervised cluster analysis. Results illustrate a high level of performance for clustering student-generated and chatGPT-generated analogies. To evaluate which features contribute most to model performance, we use a hierarchical cluster analysis. Results from the hierarchical cluster analysis illustrate several linguistic differences between the two groups.


technical paper

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Technology to predict responses is a key element in human-to-human messaging that has increasingly been utilized to enable AI-mediated communication. When response suggestions from AI are incorporated into human messaging, it will have an impact on the flow and content of the dialogue. In this paper, we investigated the effect of AI response suggestion sentences used for chat messaging on dialogue flow from two aspects: dialogue act and initiative. Usage rates of response suggestions for different dialogue acts were measured with BLEU scores, and we found that the response suggestions contributed to the establishment of a proper dialogue flow, such as an answer in response to a question. The results of our case study indicated that users who take the initiative in the dialogue tend to utilize response suggestions less frequently. We also found that some written responses were based on the suggested sentence structure but conveyed different messages.

The Effect of Response Suggestion on Dialog Flow: Analysis Based on Dialogue Act and Initiative

This study used XAI, which shows its purposes and attention as explanations of its process, and investigated how these explanations affect human trust in and use of AI. In this study, we generated heat maps indicating AI attention, conducted Experiment 1 to confirm the validity of the interpretability of the heat maps, and conducted Experiment 2 to investigate the effects of the purpose and heat maps in terms of reliance (depending on AI) and compliance (accepting answers of AI). The results of structural equation modeling (SEM) analyses showed that (1) displaying the purpose of AI positively and negatively influenced trust depending on the types of AI usage, reliance or compliance, and task difficulty, (2) just displaying the heat maps negatively influenced trust in a more difficult task, and (3) the heat maps positively influenced trust according to their interpretability in a more difficult task.

Modeling Reliance on XAI Indicating Its Purpose and Attention

To inform Explainable AI (XAI) design for updating users’ beliefs about AI based on their mental models, we examined the similarities and differences between humans and AI in object detection in driving scenarios. In humans, individuals differed in adopting focused or explorative attention strategies, with better performance associated with the focused strategy. AI (Yolo-v5s) had higher similarity in attended features to the focused than the explorative strategy in humans, and achieved human-expert-level performance in vehicle detection even in difficult cases such as occlusion and degradation. In contrast, it performed much poorer than humans in detecting humans with low attended feature similarity due to humans’ attention bias for stimuli with evolutionary significance. Also, higher similarity to humans’ attended features was associated with better AI performance, suggesting that human attention may be used for guiding AI design. These findings have significant implications for both AI and XAI designs.

Humans vs. AI in Detecting Vehicles and Humans in Driving Scenarios

Regular polysemes are sets of ambiguous words that all
share the same relationship between their meanings, such
as CHICKEN and LOBSTER both referring to an an-
imal or its meat. To probe how a context embedding
model, here exemplified by BERT, represents regular pol-
ysemy, we analyzed whether its embeddings support an-
swering sense analogy questions similar to “is the map-
ping between CHICKEN (as an animal) to CHICKEN (as
a meat) the same as that which maps between LOBSTER
(as an animal) and LOBSTER (as a meat)?” We found
that (1) the model was sensitive to the shared structure
within a regularity type; (2) the shared structure varies
across regularity types, potentially reflective of a “reg-
ularity continuum;” (3) some high-order latent structure
may be shared across regularity types, suggestive of a
similar latent structure across types; and (4) the is equivo-
cal evidence that the aforementioned effects are explained
by meaning overlap.


Probing the Representational Structure of Regular Polysemy in a Contextual Word Embedding Model via Sense Analogy Questions

There is a growing interest in emotions in textual data. The studies on labeling dataset or detecting emotions from text are based on some psychological theories. However, there has been little research on the appropriateness of using these theories. In this study, we reviewed the commonly used basic emotion theories. We analyze 3 major theories: Ekman's Basic Emotions, Plutchik's Wheel of Emotions, and GoEmotions. We used an emotionally fine-tuned language model to analyze the emotional relationship between words that each theory designates as basic emotions and evaluated the adequacy of the emotional structure each theory presents. As a result of the fine-tuning, clear patterns of similarity emerged based on the emotional meaning of each word, and through this we analyzed the structure of basic emotions. Ekman's and GoEmotions were almost in line with our results, while Plutchik's had some differences. We thoroughly discussed these matches and mismatches.


A Structure of basic emotions: A review of basic emotion theories using an emotionally fine-tuned language model

Online experiments with participants from around the world speaking different languages can inform research about culture, cognition, and language. To achieve this, it is necessary to objectively assess participants' language proficiency. However, available proficiency tests are small and only cover a few languages, making them unsuitable for cross-cultural online studies. Here, we present an automated pipeline to generate vocabulary tests based on Wikipedia called ``WikiVocab". Our pipeline samples rare nouns from a word-frequency distribution and creates pseudowords with the same low-level statistics. Six behavioral experiments (N=236) show that (a) our test can distinguish between native speakers of closely related languages, (b) participants' scores are consistent across test items, and (c) performance strongly correlates with existing tests (LexTale) and self-reports. We benchmark our results with GPT-3 and show that test accuracy and language distance are negatively correlated. Our test, available in eight languages, can easily be extended to other languages.


Around the world in 60 words: A generative vocabulary test for online research

Is “cow” more closely related to “grass” or to “chicken”? Speakers of different languages judge similarity in this context differently, but why? One possibility is that cultures co-varying with these languages induce differences in conceptualizations of similarity. Specifically, East Asian cultures may promote reasoning about thematic similarity, by which cow and grass are more related, whereas Western cultures may bias judgments toward taxonomic relations, like cow and chicken. We measure similarity judgments across the US, China, and Vietnam and replicate US-China differences, but do not find that responding in Vietnam patterns with China. Instead, similarity judgments in Vietnam are intermediate between the US and China. We also show that word embedding models (fastText models for each language) are related to judgments within each country, suggesting a possible alternative interpretation of cross cultural differences. Perhaps notions of similarity are similar across contexts, but the statistics of the linguistic environment vary.


Re-examining cross-cultural similarity judgments using language statistics

People often prefer simpler explanations (that posit the presence of fewer causes), judging these more probable than more complex alternatives. However, simplicity preferences are only mathematically justified under certain conditions. We examine one case where, mathematically, complexity preferences are justified, and test whether this corresponds to a boundary condition on simplicity preferences. Specifically, we focus on cases where causes occur frequently (rather than rarely), and where explanations specify the absence of additional causes (e.g., “Cause A and not B or C”) rather than remaining agnostic about their presence or absence (e.g., “Cause A”). Study 1 showed that, in these cases, simplicity preferences were attenuated, but not reversed. Study 2 suggests that simplicity preferences partly stemmed from failures to explicitly represent absent causes. We suggest that biases towards oversimplification may arise due to over-application of a cognitively simple version of Ockham’s razor, that is insensitive to the probability of absent causes.


The Edge of Ockham’s Razor: Examining Boundary Conditions on Preferences for Simpler Explanations

What shape do people's mental models take? We hypothesize that people build causal models that are suited to the task at hand. These models abstract away information to represent what matters. To test this idea empirically, we presented participants with causal learning paradigms where some features were outcome-relevant and others weren't. In Experiment 1, participants had to learn what objects of different shape and color made a machine turn on. In Experiment 2, they had to predict whether blocks sliding down ramps would cross a finish line. In both experiments, participants made systematic errors in a surprise test that asked them to recall what they had seen earlier. The errors people made suggest that they had build mental models of the task that privileged causally relevant information. Our results contribute to recent efforts trying to characterize the important role that causal abstraction plays in human learning and inference.


Learning what matters: Causal abstraction in human inference

Deciding whether an event could happen, or should happen, is critical to learning and decision-making yet takes years to develop, as young children are prone to deny the possibility and permissibility of any event that violates their expectations. Here, we explore whether this form of reasoning—modal cognition—is aided by cognitive reflection, or the disposition to privilege analysis over intuition. Ninety-nine children between the ages of 4 and 12 judged the possibility and permissibility of several hypothetical events, and their judgments were compared to their score on a developmental version of the cognitive reflection test, the CRT-D. Children’s CRT-D scores predicted their ability to differentiate possible events from impossible ones and impermissible events from permissible ones, as well as their ability to differentiate possibility from permissibility in general. Such differentiations were predicted by children’s CRT-D scores independent of age and executive function. These findings suggest that mature modal cognition may require the ability to reflect on, and override, the intuition that unexpected events cannot happen.


Premium content

Long-form analogies generated by chatGPT lack human-like psycholinguistic properties

Downloads

Next from CogSci 2023

The Effect of Response Suggestion on Dialog Flow: Analysis Based on Dialogue Act and Initiative

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES