United States

How do people understand and evaluate claims about others&#39; beliefs, even though these beliefs cannot be directly observed? In this paper, we introduce a cognitive model of epistemic language interpretation, grounded in Bayesian inferences about other agents&#39; goals, beliefs, and intentions: a language-augmented Bayesian theory-of-mind (LaBToM). By translating natural language into an epistemic language-of-thought, then evaluating these translations against the inferences produced by inverting a probabilistic generative model of rational action and perception, LaBToM captures quantitative human judgments about initial and present belief claims. We validate our model in an experiment where participants evaluate belief sentences about an agent who has to find keys hidden in boxes to reach its goal. In contrast with multimodal LLMs (GPT-4o, Gemini Pro) and various ablations, our model produces a much stronger fit with human ratings for a wide range of epistemic expressions, including modal language, uncertainty expressions, knowledge claims, likelihood comparisons, and descriptions of false beliefs.

NAACL 2025

Understanding Epistemic Language with a Language-augmented Bayesian Theory of Mind

belief representation

epistemic language

social reasoning

theory of mind

cognitive modeling

How do people understand and evaluate claims about others' beliefs, even though these beliefs cannot be directly observed? In this paper, we introduce a cognitive model of epistemic language interpretation, grounded in Bayesian inferences about other agents' goals, beliefs, and intentions: a language-augmented Bayesian theory-of-mind (LaBToM). By translating natural language into an epistemic language-of-thought, then evaluating these translations against the inferences produced by inverting a probabilistic generative model of rational action and perception, LaBToM captures quantitative human judgments about initial and present belief claims. We validate our model in an experiment where participants evaluate belief sentences about an agent who has to find keys hidden in boxes to reach its goal. In contrast with multimodal LLMs (GPT-4o, Gemini Pro) and various ablations, our model produces a much stronger fit with human ratings for a wide range of epistemic expressions, including modal language, uncertainty expressions, knowledge claims, likelihood comparisons, and descriptions of false beliefs.

technical paper

### Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Welcome to the 2025 meeting of the Nations of the Americas Chapter of the Association for Computational Linguistics! I am proud to help organize the first NAACL conference to carry the new name of our organization, one that emphasizes inclusion for all of the Americas. I am also pleased to welcome you to Albuquerque, New Mexico, a state whose unique blend of cultural influences will make for an excellent backdrop for NAACL 2025, especially with this year’s special theme on NLP in a Multicultural World. 
**[Continue reading...](https://drive.google.com/file/d/1jX-qGhqVSZZCIrAnJaz798pflu5Irrdn/view?usp=sharing)**

*- Colin Cherry, Google, NAACL 2025 General Chair* 

[![](https://assets.underline.io/markdown_image/1/image/b087f8a4dc5816d6e1a6514e59c59ac3.png)](https://drive.google.com/file/d/1T96GzPqObXrMln2BMByCSXTSizjTg69P/view?usp=sharing)

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

To access NAACL 2025 event page you are required to register. Please follow [**this link**](https://2025.naacl.org/registration/registration/) to register. Access will depend on your registration type.

Please register!

Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Text-To-Image (TTI) models, such as DALL-E and StableDiffusion, have demonstrated remarkable prompt-based image generation capabilities. Multilingual encoders may have a substantial impact on the cultural agency of these models, as language is a conduit of culture. In this study, we explore the cultural perception embedded in TTI models by characterizing culture across three hierarchical tiers: cultural dimensions, cultural domains, and cultural concepts. Based on this ontology, we derive prompt templates to unlock the cultural knowledge in TTI models, and propose a comprehensive suite of evaluation techniques, including intrinsic evaluations using the CLIP space, extrinsic evaluations with a Visual-Question-Answer (VQA) model and human assessments, to evaluate the cultural content of TTI-generated images. To bolster our research, we introduce the CulText2I dataset, derived from six diverse TTI models and spanning ten languages. Our experiments provide insights regarding Do, What, Which and How research questions about the nature of cultural encoding in TTI models, paving the way for cross-cultural applications of these models.

Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models

Since the SCICAP dataset’s launch in 2021, the research community has made significant progress in generating captions for scientific figures in scholarly articles. In 2023, the first SCICAP Challenge took place, inviting global teams to use an expanded SCICAP dataset to develop models for captioning diverse figure types across various academic fields. At the same time, text generation models advanced quickly, with many powerful pre-trained large multimodal models (LMMs) emerging that showed impressive capabilities in various vision-and-language tasks. This paper presents an overview of the first SCICAP Challenge and details the performance of various models on its data, capturing a snapshot of the field’s state. We found that professional editors overwhelmingly preferred figure captions generated by GPT-4V over those from all other models and even the original captions written by authors. Following this key finding, we conducted detailed analyses to answer this question: Have advanced LMMs solved the task of generating captions for scientific figures?

Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023.

In Natural Language Processing (NLP), predicting linguistic structures, such as parsing and chunking, has mostly relied on manual annotations of syntactic structures. This paper introduces an unsupervised approach to chunking, a syntactic task that involves grouping words in a non-hierarchical manner. We present a Hierarchical Recurrent Neural Network (HRNN) designed to model word-to-chunk and chunk-to-sentence compositions. Our approach involves a two-stage training process: pretraining with an unsupervised parser and finetuning on downstream NLP tasks. Experiments on multiple datasets reveal a notable improvement of unsupervised chunking performance in both pretraining and finetuning stages. Interestingly, we observe that the emergence of the chunking structure is transient during the neural model's downstream-task training. This study contributes to the advancement of unsupervised syntactic structure discovery and opens avenues for further research in linguistic theory.

The Emergence of Chunking Structures with Hierarchical HRNN

Knowledge Graph Embedding (KGE) methods are widely used to map entities and relations from knowledge graphs (KGs) into continuous vector spaces, enabling non-classical reasoning over knowledge structures. Despite their effectiveness, the uncertainty of KGE methods has not been extensively studied in the literature. This gap poses significant challenges, particularly when deploying KGE models in high-stakes domains like medicine, where reliability and risk assessment are critical. This dissertation seeks to investigate various types of uncertainty in KGE methods and explore strategies to quantify, mitigate, and reason under uncertainty effectively. The outcomes of this research will contribute to enhancing the reliability of KGE methods, providing greater confidence in their use beyond benchmark datasets, and supporting their application in real-world, high-stakes domains.

Thesis Proposal: Uncertainty in Knowledge Graph Embeddings

This study investigates the ability of Large Language Models (LLMs) to understand and interpret Chinese Xiehouyu, a unique linguistic phenomenon in Chinese involving a riddle and sometimes an explicit answer that requires complex reasoning to arrive at the correct non-literal meaning. 
We first create a new evaluation benchmark, X-Riddles, with 345 annotated xiehouyu of three categories: polysemy, homophony, and direct, and 671 sentences containing these xiehouyu.
We then evaluate LLMs through a series of experiments focusing on their proficiency in matching the riddle with the answer and explaining the linguistic phenomena involved. 
Our results indicate that while LLMs perform well on high-frequency xiehouyu, their accuracy drops significantly on low-frequency ones, especially those involving homophonic puns, whereas for human subjects, the gap between high- and low-frequency ones is much smaller.
This suggests that unlike humans, LLMs cannot perform reasoning based on linguistic and world knowledge for low-frequency xiehouyu. 
Their high performance on high-frequency ones may result from memorizing relevant information in the training data rather than actual reasoning. 
Our findings expose LLMs' inability in understanding the non-literal meaning in xiehouyu and contributes to the growing literature on LLMs' comprehension of the nuances of figurative language.

X-Riddles: Evaluating the Reasoning Abilities of LLMs with Chinese Xiehouyu Riddles

This paper presents a research thesis proposal to develop a generalizable Native American language identification system, the first of its kind. Despite their cultural and historical significance, Native American languages remain entirely unsupported by major commercial language identification systems. This omission not only underscores the systemic neglect of endangered languages in technological development, but also highlights the urgent need for dedicated, community-driven solutions. We propose a two-pronged approach: (1) systematically curating linguistic resources across all Native American languages for robust training, and (2) tailored data augmentation to generate synthetic yet linguistically coherent training samples. As proof of concept, we extend an existing rudimentary Athabaskan language classifier by integrating Plains Apache, an extinct Southern Athabaskan language, as an additional language class. We also adapt a data generation framework for low-resource languages to create synthetic Plains Apache data, highlighting the potential of data augmentation. This proposal advocates for a community-driven, technological approach to supporting Native American languages.

What is it? Towards a Generalizable Native American Language Identification System

SFT (Supervised Fine-Tuning) and PO (Preference Optimization) are key methods for enhancing language models and aligning them with human preferences. However, scaling preference datasets for PO training is challenging, leading AI customer support systems to rely on SFT. To address this, we propose a new methodology to generate customer preference datasets without human intervention based on public dataset for SFT. Our approach includes sentiment analysis to classify responses as aligned or not aligned with customer preferences, fine-tuning models to generate aligned responses, and applying advanced sampling and evaluation techniques to ensure diversity and quality. Ultimately, we generated 1,174 customer preference datasets based on 357 test datasets, and through experiments, we confirmed that the AI customer support system trained on these datasets is capable of carefully considering customer emotions and generating professional and appropriate responses.

Sentimatic: Sentiment-guided Automatic Generation of Preference Datasets for Customer Support Dialogue System

This paper presents a federated learning system with differential privacy for hate speech detection, tailored to low-resource languages. By fine-tuning pre-trained language models, ALBERT emerged as the most effective option for balancing performance and privacy. Experiments demonstrated that federated learning with differential privacy performs adequately in low-resource settings, though datasets with fewer than 20 sentences per client struggled due to excessive noise. Balanced datasets and augmenting hateful data with non-hateful examples proved critical for improving model utility. These findings offer a scalable and privacy-conscious framework for integrating hate speech detection into social media platforms and browsers, safeguarding user privacy while addressing online harm.

Privacy-Preserving Federated Learning for Hate Speech Detection

CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.
However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.
In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.
The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code will be publicly available.

Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation

Past vocabulary learning techniques identify the relevant vocabulary prior to training based on statistical and entropy-based assumptions that largely neglect the role of model training. Empirically, we observe that the trained translation models are induced to use a byte-pair encoding (BPE) vocabulary subset distinct from the original BPE vocabulary while leading to performance improvements upon retraining with the induced vocabulary. In this paper, we first analyze this difference in the context of neural machine translation by examining the vocabulary and entropy shifts during self-training -- where each iteration creates a labeled dataset by pairing source sentences with the model's predictions to define a new vocabulary. Based on our findings, we propose *self-vocabularizing training*, which iteratively self-selects vocabulary that is both smaller in size and more optimal with up to $1.49$ BLEU improvement. Moreover, we found that using deeper model architectures leads to both an increase in the use of unique tokens and a significant reduction in the vocabulary size by 6-8\%.

Downloads

Next from NAACL 2025

Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAACL 2025

Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads