United States

The field of natural language generation has accumulated a rich ecosystem of automatic evaluation metrics, yet it lacks a coherent theory of what those metrics are actually measuring. Drawing on measurement theory from the quantitative social sciences, this paper argues that current NLG evaluation practices suffer from a fundamental construct validity problem: metrics are treated as proxies for output quality without explicit specification of the underlying constructs they are meant to operationalize. We examine four dominant evaluation paradigms (reference-based metrics, embedding-based metrics, LLM-as-judge, and human evaluation) and demonstrate that each conflates construct definition with operationalization. Building on a long psychometric tradition reaching back to Cronbach and Meehl (1955) and on recent NLP work that has begun to apply this tradition to bias measurement, dialogue evaluation, and benchmark design, we propose that the field adopt a measurement modeling perspective for NLG evaluation. We borrow the concepts of construct validity, reliability, and consequential validity as a foundation for more principled evaluation, and we outline a preliminary taxonomy of NLG quality constructs as a starting point for this work.

ACL 2026

Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation

atural language generation

evaluation

The OWSD General Assembly and international conference will be held entirely online for the first time in OWSD's 28-year history, allowing as many of OWSD's 7600+ members to participate as possible.

The conference theme is 'Women, science and development', exploring three areas of relevance: 1) the importance of basic sciences for development; 2) the impact of applied sciences on development and 3) including sex and gender as variables in scientific research.


**Keynote speakers include:** 
* Fabiola Gianotti, Director-General of CERN; 
* Bina Agarwal, an internationally celebrated economist specializing in the political economy of gender, poverty and inequality; and 
* Angela Saini, award-winning author of Inferior: How Science Got Women Wrong and the New Research That’s Rewriting the Story, Superior: The Return of Race Science, and other books. 

OWSD members will present their own research in the three themes during poster sessions throughout the conference. In addition, members will also present their own scientific and personal responses to the COVID-19 pandemic. All members will also have the opportunity to take part in a workshop on increasing their visibility and impact as a scientist, with a session on using podcasts for storytelling by Joy Owango and Adelle Onyango of the Training Centre in Communication (TCC Africa), followed by a session on filmmaking for impact by Nicole Leghissa, OWSD video consultant.

The conference will include a handover of the previous OWSD Executive Board to the new members, elected in June-July 2021. On the last day of the conference (19 November), the OWSD Italy Country Network will be launched. **This will be an in-person event in Trieste.** Italian government representatives and donors are invited to attend. The Trieste event will be coordinated with similar in person events being held in OWSD countries throughout the developing world and streamed online. At least 2,000 OWSD members, women scientists from developing countries around the world, are expected to participate online in the General Assembly and conference.

 

**ABOUT THE PROGRAMME**

In order to approach the complex and wide-ranging challenges of the 2030 Agenda for Sustainable Development, societies around the world must access and harness all the talent and experience available to them. The concerns and perceptions of the greatest number of all possible actors and recipients must be considered in order that the most relevant issues and effective solutions are found. Women are considerably underrepresented as researchers in STEM subjects around the globe; according to UNESCO Institute of Statistics (UIS) data, in 2019 the global average percentage of women researchers was still below 30%. Yet women typically have first hand specialist knowledge of the day-to-day challenges diverse individuals face within communities, and can be key informers in the design and development of their solutions. Moreover, since women are most often the providers of informal community support, if they are not fully informed—and if solutions are developed without their consultation—it may be the case that products, services and solutions are not relevant or appropriate. Technological advances may mean improved machines on large farms, for example, potentially increasing food production, but since the key points of food insecurity in the world are on small-scale farms managed by women throughout the Global South, such capital investments will have little impact in the areas where food production is most critical—and where women’s knowledgeable implementation of appropriate innovative technologies could make the most difference.

 
 


**Keynote Speakers:**
* Fabiola Gianotti, Director-General of CERN; 
* Bina Agarwal, an internationally celebrated economist specializing in the political economy of gender, poverty and inequality; and 
* Angela Saini, award-winning author of Inferior: How Science Got Women Wrong and the New Research That’s Rewriting the Story, Superior: The Return of Race Science, and other books. 

**Eminent Guests:**
* Elena Bonetti, Italian Minister for Family and Equal Opportunities
* Maria Cristina Messa,	Italian Minister of Universities and Research
* Shamila Nair-Bedouelle,	Assistant Director General for Natural Sciences, UNESCO
* Maria Chiara Carrozza,	President, National Research Council of Italy
* Kerstin Jonsson Cisse,	Head of Unit for Globally Sustainable Economic Development, Sida - Sweden
* Naser Faruqui,	Director of Education and Science at IDRC - Canada
* Ylann Schemm,	Director, Elsevier Foundation
* Alberto Quadrio Curzio,	President Emeritus of the Italian Academy of Sciences and Humanities 
* Mohamed Hassan,	President, TWAS the World Academy of Sciences

**Chairs, panelists, interviewers and other speakers:**
Jennifer Thomson, Tonya Blowers, Rana Dajani, Shobhana Narasimhan, Joy Owango, Elizabeth Pollitzer, Kausila Timsina, Elizabeth Bandason, Susana Arrechea, Reinabelle Reyes, Dorine Odongo, Elizabeth Pollitzer, Adelle Onyango, Nicole Leghissa, Jana Rodriguez Hertz, Maryse Nkoua Ngavouka, Kate Shaw, Chao Mbogo, Prativa Pandey, Edna Patricia Matta-Camacho, Amira Shaheen, Wezi Nyambili, Ana Lucía Martinez Abarca, Mashura Shammi, Therese Umuhoza, Dionica Gamboa, Marian Nkansah, Babalola Olubukola Oluranti, Esi Awuah, Fortunate Farirai, Atya Kapley, Hasin Anupama Azhari, Huda Basaleem, Nashwa Eassa, Shymaa Enany, Kleinsy Bonilla, Patricia Castillo-Briceno, Fang Xin

Please register for the conference to be able to enter the event page on Underline.io. If you have already registered, please login to Underline.io to be able to see full conference content. In case you do not have an Underline account, please create one. Thank you

Please login to your Underline account

OWSD 6th GENERAL ASSEMBLY AND INTERNATIONAL CONFERENCE

The conference theme is 'Women, science and development', exploring three areas of relevance: the importance of basic sciences for development; the impact of applied sciences on development, and including sex and gender as variables in scientific research.

workshop paper

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. 
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) 
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

Registration Required

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

Post-training methods such as supervised fine-tuning and preference optimization are widely used to align large language models, yet how their benefits distribute across architectural components and transfer across tasks and prompts remains unclear. In this work, we analyze component-level transfer in a Mixture-of-Experts language model by selectively replacing routers, attention modules, and expert networks between two post-trained Mixture of Experts models trained with different post-training recipes and dataset mixtures. Starting from a SFT+DPO checkpoint, we systematically replace its components (routers, attention, experts) with those from a Tulu3 checkpoint and evaluate the impact of each replacement and their combinations on mathematical and scientific reasoning and a general-purpose classification task under zero-shot, few-shot and Chain of Thought prompting. We find strong component-specific specialization: expert networks account for most gains on mathematical and scientific reasoning, while attention mechanisms consistently outweigh expert transfer on general tasks and router transfer alone provides minimal benefit or harms performance. Prompting strategy further modulates these effects, with expert transfer degrading zero-shot science performance but improving few-shot reasoning. Strategically combining components from different model versions can in some cases match or exceed the performance of the best available model, motivating principled techniques for composing post-trained models into task- and prompt-specific systems without additional training.

Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts

Large language models (LLMs) are increasingly evaluated using LLM-as-a-Judge frameworks, yet existing approaches often rely on fixed evaluation criteria that assume universal notions of answer quality. In expert domains, however, evaluation standards vary across communities and reflect domain-specific priorities. We introduce CAMEL, a framework for learning Community-Aligned Multi-metric Evaluator LLMs directly from large-scale human preference data. Using question–answer interactions from Stack Exchange communities, CAMEL discovers community-specific evaluation metrics, generates rubric-based scoring criteria, and learns metric importance weights that reproduce observed preference orderings. This process converts implicit community preferences into structured supervision for training evaluator models. The resulting evaluator produces interpretable reports that include metric-level scores, rubric-grounded explanations, and a calibrated overall score aligned with community norms. Experiments across seen and unseen communities show that CAMEL improves preference-order prediction compared to prompting-based judges and general-purpose evaluation models. In addition, a human-subject study with domain practitioners demonstrates that CAMEL’s metric-based evaluation reports are strongly preferred for their transparency, consistency, and diagnostic usefulness. Our results highlight the importance of community-aligned evaluation design and suggest that preference data from real-world expert communities can be leveraged to build more interpretable and human-aligned LLM evaluators.

CAMEL: Learning Community-Aligned Metrics and Weights for LLM Evaluation

Extractive QA tasks are commonly evaluated using Exact Match (EM) and F1-score, but these metrics often fail to reflect true model performance. Recent studies have proposed using large language models (LLMs) as judges (LLM-as-a-judge), yet they often lack comprehensive evaluation across datasets and overlook key factors such as sensitivity to answer types, prompt variations, and self-preference bias. In this work, we conduct a systematic study of LLM-as-a-judge across four extractive QA datasets and various prompt variations, assessing multiple LLM families in both answering and judging roles. Our results show that LLM-as-a-judge judgments correlate much more strongly with human evaluations than EM (0.22) and F1 (0.40), achieving correlations up to 0.85 with open-source models. Further analysis reveals that LLM-as-a-judge performs particularly well on number-related answers but faces challenges with more complex types, such as job titles. Contrary to findings in other NLP tasks, we observe no self-preference bias, even when the same model serves as both QA model and judge. Finally, we find that prompt phrasing has minimal impact, and zero-shot, context-free judging often yields the best evaluation performance.

Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses

Indic languages, spoken by over 1.5 billion people, pose unique challenges for NLP due to their cultural richness, linguistic diversity, and structural complexity. We present IndicMMLU-Pro, a comprehensive benchmark extending the MMLU-Pro framework to nine major Indic languages: Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu. Covering a wide range of tasks in comprehension, reasoning, and generation, IndicMMLU-Pro offers a standardized evaluation framework to advance AI model development in Indic contexts. This paper details the benchmark’s design, taxonomy, and data curation, and establishes baseline results using state-of-the-art multilingual models. As an open resource IndicMMLU-Pro aims to accelerate progress in Indic language technologies and support inclusive research in multilingual NLP.

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding

We design experiments to identify where Large Language Models (LLMs) struggle when answering complex questions. Our focus is on two key stages, mirroring the human QA process: 1) \textit{question decomposition}, where the model breaks down a complex question into sub-questions and 2) \textit{subproblem solving}, where it addresses each sub-question to obtain the final response. We preprocess and expand three multi-hop datasets to create experimental datasets featuring explicit and implicit multi-hop questions, crowdsourced and templated questions, and varying numbers of hops. Our results show that larger models (Llama 3.1 70B and o1) excel at decomposing explicit multi-hop questions but struggle with implicit ones, while smaller models (e.g., Llama 3.1 8B) have difficulty with both. In the sub-problem solving stage, all models perform well on simple questions with context. Furthermore, we found no correlation between accuracy in the question decomposition stage and final QA performance (direct response), highlighting a key difference between human and LLM reasoning.

Identifying Where Large Language Models Struggle in Answering Complex Questions

Tool-calling agents are evaluated on tool selection, parameter accuracy, and scope recognition, yet LLM trajectory assessments remain inherently *post-hoc*. Disconnected from the active execution loop, such assessments identify errors that are usually addressed through prompt-tuning or retraining, and fundamentally cannot course-correct the agent in real time. To close this gap, we move evaluation into the execution loop at *inference time*: a specialized reviewer agent evaluates provisional tool calls *prior to* execution, shifting the paradigm from post-hoc recovery to proactive evaluation and error mitigation. In practice, this architecture establishes a clear separation of concerns between the primary execution agent and a secondary review agent. As with any multi-agent system, the reviewer can introduce new errors while correcting others, yet no prior work to our knowledge has systematically measured this tradeoff. To quantify this tradeoff, we introduce *Helpfulness-Harmfulness metrics*: helpfulness measures the percentage of base agent errors that feedback corrects; harmfulness measures the percentage of correct responses that feedback degrades. These metrics directly inform reviewer design by revealing whether a given model or prompt provides net positive value. We evaluate our approach on BFCL (single-turn) and $\tau^2$-Bench (multi-turn stateful scenarios), achieving +5.5\% on irrelevance detection and +7.1\% on multi-turn tasks. Our metrics reveal that reviewer model choice is critical: the reasoning model o3-mini achieves a 3:1 benefit-to-risk ratio versus 2.1:1 for GPT-4o. Automated prompt optimization via GEPA provides an additional +1.5--2.8\%. Together, these results demonstrate a core advantage of separating execution and review: the reviewer can be systematically improved through model selection and prompt optimization, without retraining the base agent.

Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents

Automated grading of open-ended student answers is a natural setting for **LLM-as-evaluator** research, yet existing methods break down when students take diverse solution approaches to the same problem. We present **DynoGrader**, a human-in-the-loop (HITL) grading system that grounds every LLM judgment in explicit evidence quotes and applies approach-specific rubrics, along with **DynoBench**, a benchmark of 12,820 human-graded responses spanning two university courses. Spending under 9 minutes of TA time per question on lightweight HITL approach discovery (avg. 12 out of 120 unreviewed submissions), DynoGrader reduces MAE by 5.2% over the strongest LLM-evaluator baseline ($p < 0.001$)—recovering 90% of the gain from fully expert-annotated approaches, with no expert pre-annotation required. *This work is under review at ACL 2026.*

Reviewing Only 1/10 of Submissions: Efficient Human-in-the-Loop Discovery for Approach-Aware LLM Grading

When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing benchmarks for evaluating agent capabilities in such tip-of-the-tongue search processes are restricted to single-turn settings. To more realistically simulate tip-of-the-tongue search, we introduce Dual-agent based Evaluation Through Obscure Under-specified Retrieval (DETOUR), a dual-agent evaluation benchmark containing 1,011 prompts. The benchmark design involves a Primary Agent, which is the subject of evaluation, tasked with identifying the recollected entity through querying a Memory Agent that is held consistent across evaluations. Our results indicate that current state-of-the-art models still struggle with our benchmark, only achieving 36% accuracy when evaluated on all modalities (text, image, audio, and video), highlighting the importance of enhancing capabilities in underspecified scenarios.

DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--``write like me''--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: $\textit{domain discrimination}$, $\textit{authorship attribution}$, and $\textit{LLM-generated personalized vs non-personalized discrimination}$. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and provide guidance on how to reliably assess style-personalized text generation.

Evaluating Style-Personalized Text Generation: Challenges and Directions

Human-annotated data remains fundamental to training frontier Large Language Models (LLMs). However, crowd-sourced annotations often suffer from quality issues stemming from annotator misunderstanding or lack of engagement. To address this, we introduce a real-time requirement adherence (RE-AD) framework that leverages LLMs to proactively validate labeling quality. Our methodology involves decomposing Standard Operating Procedures (SOPs) into atomic rules via self-reflection, categorizing them by complexity, and applying tiered validation strategies. Evaluated on a synthetic benchmark, the system achieved an F1 score of 0.749. Furthermore, production deployment resulted in annotators accepting and fixing 82% of the errors flagged by the framework. We include ablation studies to demonstrate the impact of our core design decisions.

Downloads

Next from ACL 2026

Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts

Similar lecture

Rethinking the way we farm

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2026

Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts

Similar lecture

Rethinking the way we farm

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads