United States

Artificial intelligence (AI) is transforming healthcare by improving patient care and access, especially in remote and resource-limited areas, but it also introduces risks such as biased recommendations, misdiagnoses, and inequities that disproportionately affect underserved groups. These issues often arise when AI fails to consider diverse patient characteristics and social determinants of health. To address this, we propose a human-AI storytelling framework that uses speculative narratives, similar to Black Mirror, to simulate potential unintended harms in AI-driven healthcare systems before development or deployment in the real world. By crafting realistic user stories, our approach reveals hidden biases, flawed data assumptions, and over-reliance on automation, helping developers, clinicians, and stakeholders identify risks early. These narrative ``stress tests&#39;&#39; improve AI fairness, transparency, and responsiveness to real-world needs, ultimately enhancing AI safety, promoting health equity, and guiding better system design.

NAACL 2025

Towards Safer AI in Healthcare: Using Storytelling to Educate About Unintended Harms of AI-Powered Tools

ai in healthcare

responsible ai

storytelling

Artificial intelligence (AI) is transforming healthcare by improving patient care and access, especially in remote and resource-limited areas, but it also introduces risks such as biased recommendations, misdiagnoses, and inequities that disproportionately affect underserved groups. These issues often arise when AI fails to consider diverse patient characteristics and social determinants of health. To address this, we propose a human-AI storytelling framework that uses speculative narratives, similar to Black Mirror, to simulate potential unintended harms in AI-driven healthcare systems before development or deployment in the real world. By crafting realistic user stories, our approach reveals hidden biases, flawed data assumptions, and over-reliance on automation, helping developers, clinicians, and stakeholders identify risks early. These narrative ``stress tests'' improve AI fairness, transparency, and responsiveness to real-world needs, ultimately enhancing AI safety, promoting health equity, and guiding better system design.

poster

### Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Welcome to the 2025 meeting of the Nations of the Americas Chapter of the Association for Computational Linguistics! I am proud to help organize the first NAACL conference to carry the new name of our organization, one that emphasizes inclusion for all of the Americas. I am also pleased to welcome you to Albuquerque, New Mexico, a state whose unique blend of cultural influences will make for an excellent backdrop for NAACL 2025, especially with this year’s special theme on NLP in a Multicultural World. 
**[Continue reading...](https://drive.google.com/file/d/1jX-qGhqVSZZCIrAnJaz798pflu5Irrdn/view?usp=sharing)**

*- Colin Cherry, Google, NAACL 2025 General Chair* 

[![](https://assets.underline.io/markdown_image/1/image/b087f8a4dc5816d6e1a6514e59c59ac3.png)](https://drive.google.com/file/d/1T96GzPqObXrMln2BMByCSXTSizjTg69P/view?usp=sharing)

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

To access NAACL 2025 event page you are required to register. Please follow [**this link**](https://2025.naacl.org/registration/registration/) to register. Access will depend on your registration type.

Please register!

Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Text Style Transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TST outputs is a multidimensional challenge, requiring the assessment of style transfer accuracy, content preservation, and naturalness. Using human evaluation is ideal but costly, same as in other NLP tasks; however, automatic metrics for TST have not received as much attention as metrics for, e.g., machine translation or summarization.
In this paper, we examine both set of existing and novel metrics from broader NLP tasks for TST evaluation, focusing on two popular subtasks—sentiment transfer and detoxification—in a multilingual context comprising English, Hindi, and Bengali. By conducting meta-evaluation through correlation with human judgments, we demonstrate the effectiveness of these metrics when used individually and in ensembles. Additionally, we investigate the potential of Large Language Models (LLMs) as tools for TST evaluation.
Our findings highlight that certain advanced NLP metrics and experimental-hybrid-techniques provide better insights than existing TST metrics for delivering more accurate, consistent, and reproducible TST evaluations.

Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?

The rapid spread of misinformation on social media underscores the need for effective tools to analyze public discourse. In this thesis proposal, we focus on the concept of truthfulness stance and propose a truthfulness stance detection framework that leverages large language models (LLMs) with retrieval-augmented generation (RAG) to enhance the contextual understanding of tweets in relation to claims. The framework is evaluated on a newly developed dataset and established benchmark datasets.
Experimental results demonstrate that our approach outperforms state-of-the-art methods, significantly improving the Macro-F1 score. To highlight the practical impact of truthfulness stance detection in mitigating misinformation, we showcase several real-world applications and potential future directions.

Thesis Proposal: Understanding Misinformation on Social Media Through Truthfulness Stance

We introduce CLEAR (Contrasting Textual Feedback with Experts and Amateurs for Reasoning), a novel approach to language model reasoning that leverages the strengths of a larger (expert) model and smaller (amateur) model. The expert and amateur models each provide feedback on a model's initial output and are contrasted with each other into refined feedback. This feedback is subsequently applied to iteratively improve CLEAR's responses. Across several challenging tasks, CLEAR outperforms other state-of-the-art reasoning methods, notably achieving a state-of-the-art score on the GSM8K benchmark (97.2%).

CLEAR: Contrasting Textual Feedback with Experts and Amateurs for Reasoning

In multi-turn dialogues, large language models face a critical challenge of ensuring coherence while adapting to user-specific information.. This study introduces the persona knowledge gap, the discrepancy between a model’s internal understanding and the knowledge required for coherent, personalized conversations. While prior research has recognized these gaps, computational methods for their identification and resolution remain underexplored. We propose Conversation Preference Elicitation and Recommendation (CPER), a novel framework that dynamically detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement. CPER consists of three key modules: a Contextual Understanding Module for preference extraction, a Dynamic Feedback Module for measuring uncertainty and refining persona alignment, and a Persona-Driven Response Generation module for adapting responses based on accumulated user context. We evaluate CPER on two real-world datasets: CCPE-M for preferential movie recommendations and ESConv for mental health support. Using A/B testing, human evaluators preferred CPER's responses 42% more often than baseline models in CCPE-M and 27% more often in ESConv. A qualitative human evaluation confirms that CPER's responses are preferred for maintaining contextual relevance and coherence, particularly in longer (12+ turn) conversations.

(CPER) From Guessing to Asking: An Approach to Resolving Persona Knowledge Gap in LLMs during Multi-Turn Conversations

The widespread integration of large language models (LLMs) across diverse domains, from entertainment to healthcare and banking, has underscored the pressing need to address the risks associated with artificial intelligence (AI). Indeed, as LLMs have grown in scale, new challenges have emerged, including privacy breaches, dissemination of misinformation, issues related to robustness and alignment, and concerns regarding the societal biases ingrained in these models. In response, there has been a surge in research focused on responsible AI and AI safety. To address these challenges, recent literature has introduced novel techniques for mitigating the risks associated with LLMs and testing the efficacy of safety protocols through red-teaming. Despite these efforts, concerns persist regarding the perpetuation of biases in next-generation LLMs and the lack of robustness in existing safety measures. Additionally, recent studies have highlighted the tendency of safety safeguards to over-fit certain keywords or identity terms, raising questions about their effectiveness in real-world scenarios. In this thesis proposal, we look at common evaluation practices in LLMs safety and their potential pitfalls.

Rethinking Safety Evaluation in Large Language Models: A Research Proposal

Large language models (LLMs) like GPT-4 and LLaMA-3 offer transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles. However, deploying LLMs faces challenges such as limited training data, high computational costs, and issues with transparency and explainability. Our research focuses on distilling compact, parameter-efficient tailored language models (TLMs) from LLMs for domain-specific tasks with comparable performance. Current approaches like knowledge distillation, fine-tuning, and model parallelism address computational efficiency but lack hybrid strategies to balance efficiency, adaptability, and accuracy. We present ANON - an adaptive knowledge distillation framework integrating knowledge distillation with adapters to generate computationally efficient TLMs without relying on labeled datasets. ANON uses cross-entropy loss to transfer knowledge from the teacher's outputs and internal representations while employing adaptive prompt engineering and a progressive distillation strategy for phased knowledge transfer. We evaluated ANON's performance in the crisis domain, where accuracy is critical and labeled data is scarce. Experiments showed that ANON outperforms recent approaches of knowledge distillation, both in terms of the resulting TLM performance and in reducing the computational costs for training and maintaining accuracy compared to LLMs for domain-specific applications.

Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models

Prompt optimization is crucial for improving output quality of Large Language Models (LLMs), but many existing methods are inefficient, requiring extensive computation and manual tuning. We propose Momentum-Aided Prompt Optimization (MAPO), which builds on ProTeGi by incorporating positive natural language "gradients" and a momentum-based memory mechanism to refine prompts while avoiding local minima and oscillations. It also employs beam search and an Upper Confidence Bound (UCB) algorithm for balanced candidate expansion and selection. MAPO achieves faster convergence time with fewer API calls and higher performance than ProTeGi, demonstrating it as a robust and scalable solution for automated prompt optimization in LLMs.

Momentum-Aided Natural Language Gradient Descent for Prompt Optimization

Rapid improvements in large language models have unveiled a critical challenge in human-AI interaction: sycophancy. In this context, sycophancy refers to the tendency of models to excessively agree with or flatter users, often at the expense of factual accuracy. While previous studies have primarily analyzed this behavior in single-turn interactions, its persistence and evolution in multi-step conversations remain largely unexplored. We introduce \methodname, a benchmark specifically designed to evaluate sycophancy in extended dialogues, where language models must navigate iterative user feedback, challenges, and persuasion. We prompt models to elicit four types of sycophantic biases. We then propose and test sycophancy reduction strategies, evaluating their effectiveness beyond single-step interactions.

TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models

The recent large-scale emergence of LLMs has left an open space for dealing with their consequences, such as plagiarism or the spread of false information on the Internet. Coupling this with the rise of AI detector bypassing tools, reliable machine-generated text detection is in increasingly high demand. We investigate the paraphrasing attack resilience of various machine-generated text detection methods, evaluating three approaches: fine-tuned RoBERTa, Binoculars, and text feature analysis, along with their ensembles using Random Forest classifiers. We discovered that Binoculars-inclusive ensembles yield the strongest results, but they also suffer the most significant losses during attacks. In this paper, we present the dichotomy of performance versus resilience in the world of AI text detection, which complicates the current perception of reliability among state-of-the-art techniques.

Paraphrasing Attack Resilience of Various Machine-Generated Text Detection Methods

In recent years stylometry has been more investigated in many different fields. Hence, in this work, we are going to study detecting, generating, and evaluating the writing styles of different authors. In the first step, the sentences will be extracted from several different books, each belonging to a different author, to create a dataset. Then the selected models will be trained to detect the author of sentences in the dataset. After that generator models will be trained without paired dataset to generate sentences based on the authors' writing styles. Finally, in order to evaluate the performance of generators, the first trained models will be leveraged to evaluate the generated sentences.

Next from NAACL 2025

Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES