Singapore

Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78× reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy–cost trade-offs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175× ($0.45–$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.

AAAI 2026

LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost

software testing

cost analysis

large language models

software engineering

evaluation

reliability

workshop paper

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.<br><br>

To access this event page, you need to log in with the **email address you registered with**. <br>Access credentials will be sent to your email from Underline -  subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

We address the challenge of training reliable code-fixing agents in real-world repositories, where complex build systems and shifting dependencies make evaluation unstable. We developed a verifiable pipeline that defines success through post-fix build validation and improves reproducibility across roughly one thousand real issues by pinning dependencies and disabling automatic upgrades. Building on this foundation, we introduced a scalable, simplified pipeline for large-scale reinforcement learning (RL). Using this setup, we supervised fine-tuned Qwen3-32B in the full pipeline and applied RL on top of the SFT model in the simplified environment. The SFT model distilled from GPT-4.1 trajectories performed comparably while being 56× smaller, and RL added 7–20% absolute gains under matched train–test conditions. “Thinking mode” achieved similar or worse results in our experiments. Both SFT and RL models failed to generalize across environments, underscoring the importance of aligning training and evaluation settings when building reliable, real-world code-fixing agents.

Agentic Reinforcement Learning for Real-World Code Repair

Rapid growth in cloud services has generated substantial demand for cybersecurity tool enhancements. Upon receiving these requests, a security assessment process is required, including investigating documentation, examining code repositories, evaluating coverage, and assigning work to tooling teams with detailed metadata. This security assessment is performed manually, consuming valuable expert time.<br/>We introduce a multi-agent GenAI system that orchestrates specialized sub-agents to automate the security assessment workflow. Evaluation on historical requests shows the system achieves over 95\% accuracy compared to human experts. Early adoption shows the system saves approximately 0.5 to 1 hour of human expert time daily, demonstrating that agent collaboration can maintain quality while reducing manual workload.

Multi-Agent Framework for Automated Cloud Security Assessment

The proliferation of foundation models has given rise to ``vibe coding," a paradigm where end-users create software by describing desired outcomes in natural language. While platforms that enable this approach are democratizing software development, they often produce simplistic, fragile applications that lack the formal guarantees required for scalable, reliable systems. The core software engineering challenges of translating ambiguous user intent into formal specifications and managing the lifecycle of AI-generated code remain largely unaddressed. This demo presents a system built on the Software Engineering for Software Makers (SE4SM) framework, which introduces a disciplined, two-pillar approach to AI-assisted development. The first pillar, Intent Engineering, employs a collaborative human-AI dialogue to transform a user's high-level concept into a set of clear, unambiguous, and verifiable specifications while considering both explicitly stated needs and implicitly inferred requirements. The second, Realization Engineering, uses a spec-driven, multi-agent system to autonomously build, test, and verify the software. This methodology preserves the accessibility of vibe coding while embedding professional engineering practices, producing more trustworthy, evolvable systems.

From Vibe to Verifiable Spec-Driven Development: A Demo of Intent and Realization Engineering

Modern agentic frameworks operate under the fallacy of a Stationarity Trap: the dangerous assumption that meaning in underlying knowledge bases is static and user contexts are invariant. In this talk, evidence is presented from two distinct research streams to demonstrate that this assumption guarantees system failure. Applying our EchoCodes framework, we prove that “ground truth” content (e.g. data such as medical ontologies, intelligence signals, and software APIs) can undergo rate-independent semantic drift; and this can be detected before causing agents to execute employing meanings that have shifted. In the second dimension, as part of our work on the AiVisor system, we illustrate a “Personalization Paradox,” where agents optimizing for user utility incur statistically significant penalties in semantic consistency, effectively decoupling reasoning from lexically grounded baselines. Together, these findings define a non-stationary agent tradespace, where contextual drift (user alignment) and semantic dynamics (meaning discontinuity) converge to produce un-grounded agency. The convergence of these two phenomena, shifting ontological/ground-truth concepts and expanding contextual liabilities, creates a state of semantic insolvency. When such insolvent systems are granted the power to act, the result is un-grounded agency: high-confidence execution untethered from reality. Consequently, agentic deployments risk devolving into blind autonomy. We conclude by proposing a novel Semantic Guidance System to resolve the Stationarity Trap tradespace, moving the field beyond static evaluation toward dynamic semantic grounding for long-lifecycle agents.

Un-Grounded Agency and the Stationarity Trap in Agentic Workflows

We assume a category theoretical approach to reasoning that is separate from but integrates with conventional logical systems and linear algebraic techniques such as employed by large language models, and reinforcement learning. The agenda is motivated by the ability to reason topological-ly in an expanded vocabulary of types not motivated by conventional ontological needs. One high value application is reasoning over open world situations and causally dynamic systems where essential facts are missing, situated influence is substantial, and significant implicit influ-ence is present. We are particularly looking at dynamic, unstable, possible outcomes, and situa-tions where outcome engineering is desired. The program is motivated by collecting open influences as salients, and focusing on type systems to support the concept that can have topological induction. The focus of this paper is on the na-ture of a type system where salience is a first class citizen and category theoretical topology cen-tric induction is enabled. While the research is not motivated by a specific use case, we focus on central nervous system modelling where fear memory extinction is modelled, so that we can leverage prior work. But ap-plications are expected to be rather broad, characterised by open world insights and frangible possibilities. Our group builds systems in Haskell frameworks so our a type system needs to be consistent with what can be supported programmatically. Because our group is particularly interested in human/machine navigation of structures, a consistent visual grammar sensitive to these type def-initions is desirable. This paper will include a brief survey of the literature with an emphasis on situation theory as a framework for sale and influence. Potential sensor technologies are consid-ered.

Categorical Type Systems for Salient Influence

Maintaining human oversight and broad stakeholder participation remain key challenges for trust management in human-AI teams. With the increasing deployment of agentic AI and large language models, the risk of unpredictable yet highly consequential outcomes could, if left unaddressed, undermine public and expert confidence in such teams. This study revisits that debate to articulate a framework that secures fairness, legitimacy, and trust. Drawing on theoretical and empirical analyses of citizen participation in fields adjacent or parallel to AI, it shows that the weaknesses of black-box AI have analogous counterparts in the isomorphic processes of black-box value elicitation and implementation decisions. Specifically, this study focuses on circumstances in which experts operate outside their domain of expertise to develop ad-hoc theories of reality. Examples of such “inexpert expert judgments” include situations where judges must evaluate complex statistical evidence in court, and where algorithm specialists must select features to be included in models based not on relevance to the goal but on availability or intuition. In cases like that, even something as trivial as including or excluding an interaction term in a regression-based model could lead to highly problematic outcomes. Bypassing domain specialist expertise and substituting it with stand-in knowledge is unnecessary when such expertise could be incorporated with modest effort.

The Hidden Black-Box: Expert Labor and the Case for Participatory AI;

Large Language Models (LLMs) excel at single-turn tasks such as instruction following and summarization, yet real-world deployments require sustained multi-turn interactions where user goals and conversational context persist and evolve. A recurring challenge in this setting is context drift: the gradual divergence of a model’s outputs from goal-consistent behavior across turns. Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics. In this work, we present a study of context drift in multi-turn interactions and propose a simple dynamical framework to interpret its behavior. We formalize drift as the turn-wise KL divergence between the token-level predictive distributions of the test model and a goal-consistent reference model, and propose a recurrence model that interprets its evolution as a bounded stochastic process with restoring forces and controllable interventions. We instantiate this framework in both synthetic long-horizon rewriting tasks and realistic user–agent simulations, such as in tau-bench, measuring drift for several open-weight LLMs that are used as user simulators. Our experiments consistently reveal stable, noise-limited equilibria rather than runaway degradation, and demonstrate that simple reminder interventions reliably reduce divergence in line with theoretical predictions. Together, these results suggest that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay, providing a foundation for studying and mitigating context drift in extended interactions.

Drift No More? Context Equilibria in Multi-Turn LLM Interactions

Personalization of large language models (LLMs) has recently gained attention as a key approach to enhancing their usability and adaptability to individual users. However, when personalization relies on interaction histories, inconsistencies often arise between past and current user preferences, causing the model to fail to properly adapt to the user’s most recent intent. We propose a simple preference representation that encodes each preference as an intensity and polarity vector and aggregates past and recent preferences arithmetically to form a contradiction free, up to date personalized prompt. This enables the LLM to dynamically update user preferences at test time, linearly combining past and current preferences to produce a consistent, contradiction free personalized prompt. The proposed method does not require explicit ground truth data and can be applied to various downstream applications. Preliminary experiments using datasets demonstrate that our approach improves preference alignment compared to a non personalized model and achieves dynamic optimization over time.

Preference Descriptions for Dynamic Personalization of Large Language Models

As large language models (LLMs) become increasingly integrated into daily life, privacy concerns are on the rise. Driven by the appeal of convenient, universal solutions, current practices reflect a drift toward a one-size-fits-all approach to privacy. Unfortunately, we argue that a unified framework for prompt privacy is elusive and can instead mislead, creating even greater risk due to a false sense of safety. We identify five desirable properties of an effective privacy framework for prompts, namely protection guarantees, performance, efficiency, domain adaptability, and user accessibility, and highlight that existing frameworks, such as sanitization, differential privacy, cryptography, and contextual integrity, only satisfy a subset of these properties. Beyond individual frameworks, we find underlying tensions between these properties that preclude the development of a unified framework. We recommend two critical paths forward: emphasizing context-specific and application-specific evaluation of privacy frameworks, and fostering user awareness and privacy literacy.

A Unified Framework for Prompt Privacy is Elusive and Misleading

We propose a novel framework for persona-based language model system, motivated by the need for personalized AI agents that adapt to individual user preferences. In our approach, the agent embodies the user’s “persona” (e.g. user profile or taste) and is powered by a large language model (LLM). To enable the agent to leverage rich contextual information, we introduce a Knowledge-Graph-enhanced Retrieval Augmented Generation (Graph RAG) mechanism that constructs an LLM-derived graph index of relevant documents and summarizes communities of related information. Our framework generates personalized prompts by combining: (1) a summary of the user’s historical behaviors and preferences extracted from the knowledge graph, and (2) relevant global interaction patterns identified through graph-based community detection. This dynamic prompt engineering approach allows the agent to maintain consistent persona-aligned behaviors while benefiting from collective knowledge. On the LaMP benchmark, our method improves news categorization F1 by 11.1%, movie tagging F1 by 56.1%, and reduces product rating MAE by 10.4% over prior methods.

Content not yet available

Downloads

Next from AAAI 2026

Agentic Reinforcement Learning for Real-World Code Repair

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Content not yet available

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2026

Agentic Reinforcement Learning for Real-World Code Repair

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads