Singapore

Large Language Models are increasingly deployed as judges (LaaJ) in code generation pipelines. While attractive for scalability, LaaJs tend to overlook domain-specific issues raising concerns about their reliability in critical evaluation tasks. To better understand these limitations in practice, we examine LaaJ behavior in a concrete industrial use case: legacy code modernization via COBOL code generation. In this setting, we find that even production-deployed LaaJs can miss domain-critical errors, revealing consistent blind spots in their evaluation capabilities. To better understand these blind spots, we analyze generated COBOL programs and associated LaaJs judgments, drawing on expert knowledge to construct a preliminary taxonomy. Based on this taxonomy, we develop a lightweight analytic checker tool that flags over 30 domain-specific issues observed in practice. We use its outputs as {\it analytic hints}, dynamically injecting them into the judge’s prompt to encourage LaaJ to revisit aspects it may have overlooked. Experiments on a test set of 100 programs using four production-level LaaJs show that LaaJ alone detects only about 45% of the errors present in the code (in all judges we tested), while the analytic checker alone lacks explanatory depth. When combined, the LaaJ+Hints configuration achieves up to 94% coverage (for the best-performing judge and injection prompt) and produces qualitatively richer, more accurate explanations, demonstrating that analytic–LLM hybrids can substantially enhance evaluation reliability in deployed pipelines.

AAAI 2026

Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls

cobol

dynamic hints

analytic checker

blind spots

llm as a judge

evaluation

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. 
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) 
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

Registration Required

ACL 2026

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

workshop paper

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

Large Language Models (LLMs) produce strong results but are costly to serve. Static post-training quantization reduces memory and compute, yet uses a single bit width for all prompts, wasting resources on easy inputs and degrading accuracy on harder ones. We introduce Prompt-Adaptive Quantization (PAQ), a per-prompt precision framework that requires no retraining of the underlying model. PAQ trains a lightweight BERT-based router with perplexity-guided supervision to select the smallest adequate quantization level (2, 4, 8, or 16 bits) per input. At inference, prompts are automatically routed to the appropriate pre-quantized LLM variant. Overall, PAQ serves as a novel framework for adaptive per-prompt quantization, reducing latency while maintaining strong accuracy across tasks.

Prompt-Adaptive Quantization: Adaptive Per-Prompt Routing for Efficient LLM Inference

Large language models (LLMs) often generate fluent but factually incorrect statements despite having access to relevant evidence, a failure mode rooted in how they allocate attention between contextual and parametric knowledge. Understanding and steering this internal behavior is key both for trustworthy deployment and for scientific interpretability of model mechanisms. We introduce COMPASS (Context-Modulated PID Attention Steering System), a lightweight, interpretable control framework that embeds a model-based feedback loop directly within decoding. COMPASS quantifies context reliance via a transparent metric, the Context Reliance Score (CRS), which serves as an online probe of how attention heads ground generation in evidence. Using this interpretable signal, a PID controller dynamically modulates attention heads to maintain factual consistency without retraining or multi-pass decoding. Across benchmarks (HotpotQA, XSum, HaluEval, RAGTruth), COMPASS consistently reduces contextual hallucination rates (2.8–5.8% absolute) while revealing how distinct attention heads contribute to evidence alignment. These results highlight feedback-driven interpretability as a pathway toward scientific understanding of LLM behavior.

COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation

BSLM: A Bi-Level Speech-Language Model for the Joint Modeling of Discrete and Continuous Tokens

Granular Control of Nonverbal Expressions for Achieving Natural Emotional Text-to-Speech

AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio Embedding Sequences

Can You Hear Naples? Building and Benchmarking a Neapolitan Speech Corpus

AudioRAG: A Challenging Benchmark for Audio Reasoning and Information Retrieval

Generalizable speech deepfake detection via meta-learned LoRA

Optimization is a foundational pillar of artificial intelligence (AI), underpinning core techniques in planning, scheduling, decision-making, and machine learning. Yet despite decades of algorithmic advances, widespread adoption of state-of-the-art optimization solvers remains limited by the substantial expertise required for effective modeling and solving. This expertise barrier means that powerful optimization tools remain largely inaccessible to non-experts, with most users of leading solvers holding advanced degrees.

Recent advances in generative AI, particularly large language models (LLMs), offer a promising new path for democratizing optimization. By automating key steps in the optimization pipeline – from model formulation through solver configuration to model validation – LLMs promise to broaden access to powerful optimization tools. However, these models rarely work out of the box for complex reasoning tasks like optimization. 

This tutorial surveys emerging research at the intersection of LLMs and mathematical optimization, highlighting both practical systems and open research questions.  We will provide a comprehensive overview of how LLMs can support each stage of the optimization pipeline, including model formulation, solver configuration, and validation. The tutorial is designed to be accessible to attendees without prior experience in either field, offering both conceptual frameworks and practical insights for this rapidly evolving area of research.

LLMs for Optimization: Modeling, Solving, and Validating with Generative AI

There is a growing debate about the implications of multiplicity—conflicting behavior among a set of “good”’ models—for algorithmic decision-making. On one hand, there are concerns over unfair treatment due to conflicting predictions and explanations, further exacerbated in the generative AI ecosystem. Yet, on the other hand, multiplicity also offers the potential to find less discriminatory and more interpretable models. In this tutorial, we aim to increase awareness of different perspectives on multiplicity and position them among broader discussions in the community. Specifically, the main goals of our tutorial are:

* Highlighting the phenomenon of multiplicity in machine learning and calling attention to its growing literature.
* Linking multiplicity with uncertainty and churn, presenting a unified view of various measures of instability.
* Discussing the implications of multiplicity for fairness and explainability in algorithmic decision-making.
* Recognizing exacerbated concerns and new forms of multiplicity in the age of generative AI.
* Engaging the community on when and how to address multiplicity in various practical scenarios.
* Identifying open questions and motivating future research directions on multiplicity.

Next from AAAI 2026

Prompt-Adaptive Quantization: Adaptive Per-Prompt Routing for Efficient LLM Inference

Similar lecture

Evaluating LLM-as-a-Judge for Medical Term Simplification

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES