United States

Handwritten Mathematical Expression Recognition (HMER) has extensive applications in automated  grading and office automation. However, existing sequence-based decoding methods, which directly predict $\LaTeX$ sequences, struggle to understand and model the inherent tree structure of $\LaTeX$ and often fail to ensure syntactic correctness in the decoded results. To address these challenges, we propose a novel model named TAMER (Tree-Aware Transformer) for handwritten mathematical expression recognition. TAMER  introduces an innovative Tree-aware Module while maintaining the flexibility and efficient training of Transformer. TAMER combines the advantages of both sequence decoding and tree decoding models by jointly optimizing sequence prediction and tree structure prediction tasks, which enhances the model&#39;s understanding and generalization of complex mathematical expression structures. During inference, TAMER  employs a Tree Structure Prediction Scoring Mechanism to improve the structural validity of the generated $\LaTeX$ sequences. Experimental results on CROHME datasets demonstrate that TAMER outperforms traditional sequence decoding and tree decoding models, especially in handling complex mathematical structures, achieving state-of-the-art (SOTA) performance.

AAAI 2025

TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition

applications

Handwritten Mathematical Expression Recognition (HMER) has extensive applications in automated  grading and office automation. However, existing sequence-based decoding methods, which directly predict $\LaTeX$ sequences, struggle to understand and model the inherent tree structure of $\LaTeX$ and often fail to ensure syntactic correctness in the decoded results. To address these challenges, we propose a novel model named TAMER (Tree-Aware Transformer) for handwritten mathematical expression recognition. TAMER  introduces an innovative Tree-aware Module while maintaining the flexibility and efficient training of Transformer. TAMER combines the advantages of both sequence decoding and tree decoding models by jointly optimizing sequence prediction and tree structure prediction tasks, which enhances the model's understanding and generalization of complex mathematical expression structures. During inference, TAMER  employs a Tree Structure Prediction Scoring Mechanism to improve the structural validity of the generated $\LaTeX$ sequences. Experimental results on CROHME datasets demonstrate that TAMER outperforms traditional sequence decoding and tree decoding models, especially in handling complex mathematical structures, achieving state-of-the-art (SOTA) performance.

poster

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



Large language models (LLMs) have demonstrated significant progress in multilingual language understanding and generation. However, due to the imbalance in training data, their capabilities in non-English languages are limited. Recent studies revealed the English-pivot multilingual mechanism of LLMs, where LLMs implicitly convert non-English queries into English ones at the bottom layers and adopt English for thinking at the middle layers. However, due to the absence of explicit supervision for cross-lingual alignment in the intermediate layers of LLMs, the internal representations during these stages may become inaccurate. In this work, we introduce a deep supervision fine-tuning method (DFT) that incorporates additional supervision in the internal layers of the model to guide its workflow. 
Specifically, we introduce two training objectives on different layers of LLMs: one at the bottom layers to constrain the conversion of the target language into English, and another at the middle layers to constrain reasoning in English. To effectively achieve the guiding purpose, we designed two types of supervision signals: logits and feature, which represent a stricter constraint and a relatively more relaxed guidance. Our method guides the model to not only consider the final generated result when processing non-English inputs but also ensure the accuracy of internal representations. We conducted extensive experiments on typical English-centric large models, LLaMA-2 and Gemma-2, and the results on multiple multilingual datasets show that our method significantly outperforms traditional fine-tuning methods.

Enhancing Non-English Capabilities of English-Centric Large Language Models Through Deep Supervision Fine-Tuning

Neurosymbolic approaches blend the effectiveness of symbolic reasoning with the flexibility of neural networks. In this work, we propose an architecture for the generation of SQL queries that leverages neurosymbolic reasoning as an alternative to simply scaling the neural model size. To this end, we integrate a generic pretrained Language Model (LM) with a symbolic query checker, a neural query checker and a symbolic test and repair module that catch and correct errors made by the LM on SQL queries. We focus on improving the performance of smaller open-source LMs out-of-the-box, and we find that our tool, Xander, increases accuracy by an average of 10.9% and reduces runtime by an average of 28%  compared to the LM without Xander, enabling a smaller LM (with Xander) to outperform its four-times-larger counterpart (without Xander).

Enhancing SQL Query Generation with Neurosymbolic Reasoning

In this work, we introduce Modulated Flows (ModFlows), a novel approach for color transfer between images based on rectified flows. The primary goal of the color transfer is to adjust the colors of a target image to match the color distribution of a reference image. Our technique is based on optimal transport and executes color transfer as an invertible transformation within the RGB color space. The ModFlows utilizes the bijective property of flows, enabling us to introduce a common intermediate color distribution and build a dataset of rectified flows. We train an encoder on this dataset to predict the weights of a rectified model for new images. After training on a set of optimal transport plans, our approach can generate optimal plans for new pairs of distributions without additional fine-tuning. We additionally show that the trained encoder provides an image embedding, associated only with its color style. The presented method is capable of processing 4K images and achieves the state-of-the-art performance in terms of content and style similarity. Our code is avalable at https://anonymous.4open.science/r/modflows-FBB1/.

Color Transfer with Modulated Flows

The fault location task in power grids is crucial for maintaining social order and ensuring public safety. However, existing methods that rely on tabular state records often neglect the intrinsic topological influences of transmission lines, resulting in a segmented approach to fault location that consists of multiple stages. In this paper, we propose an \textbf{D}isentangled \textbf{T}able-\textbf{G}raph representation framework, termed DTG, which integrates fault location tasks at coarse-grained line levels and fine-grained point levels within an end-to-end learning paradigm. Our innovative disentanglement strategy produces interpretable attribution coefficients that connect tabular records and transmission line topology, thereby facilitating fault location at both line- and point-levels. The joint prediction tasks designed around our disentangled tabular graph representation promote mutual information exchange between features and topology of transmission lines in an interpretable manner.Experimental results on the 7-bus system, 36-bus system and a realistic 325-bus system in China demonstrate that the proposed method adapt to different topological structures and handle different types of faults. Compared to traditional methods, DTG4Power achieves high accuracy in both fault lines and fault points.Our code will be made publicly available.

Disentangled Table-Graph Representation for Interpretable Transmission Line Fault Location

Generative information retrieval, which encodes documents into a model and retrieves them by generating document identifiers, is a novel retrieval framework and has attracted much research interest. However, it is a topical problem to design efficient and accurate docid to index documents. Existing methods for designing docids tend to consider only the relevance of docids to the corresponding documents, while neglecting the ability of docids to distinguish the corresponding documents from similar ones, which is crucial for retrieval.
In this paper, we design learnable descriptive and discriminative document Identifiers (DocIDs) for generative retrieval and propose the paired retrieval model D3Gen. The DocIDs can be semantically similar to the documents (descriptive) and distinguish similar documents (discriminativate) in the corpus, thus enhancing retrieval performance.We use a comparative learning assisted generative retrieval task to enable the model to understand the document and complete the generative retrieval. We then design a DocID selection method called NR-IDR to select DocIDs based on the retrieval model's understanding of the document. On the input side of the retrieval model D3Gen, we also design filters to filter diverse, multi-granular pseudo query to help the model fully understand the semantics of the document. Our experiments on the MS MARCO and NQ320k dataset illustrate the effectiveness of the approach.

Descriptive and Discriminative Document Identifiers for Generative Retrieval

The Audio-Visual Question Answering (AVQA) task involves extracting question-related audio-visual clues from both temporal and spatial perspectives to answer questions accurately. Despite the promising performance of existing multi-modal AVQA models, thanks to large-scale pre-trained models, challenges remain in the field. Firstly, aligning audiovisual information across temporal and spatial dimensions is
difficult. Secondly, the fusion of audio-visual information is often weighted inadequately, limiting model performance. To address these issues, we propose the Audio-Visual Adaptive Fusion Network (AVAF-Net), which uses contrastive learning to align audio-visual information temporally and spatially and adaptively adjusts fusion weights based on the question. Specifically, we initially align visual and audio information temporally through a temporal-alignment contrastive loss. This is followed by an audio-visual clue-mining module
that highlights question-related cues, aligning them with the vocal region spatially using spatial alignment contrastive loss. Additionally, a question-oriented adaptive fusion module assigns different weights to audio and visual modalities based on the question content and then fuses them. The fused audio-visual cues are finally used to predict the answer. Extensive experiments on the MUSIC-AVQA dataset show that AVAF-Net surpasses all baseline models, with a maximum improvement of 15.90% in average accuracy and an average improvement of 9.80%.

Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning

Few-shot point cloud semantic segmentation aims to accurately segment "unseen" new categories in point cloud scenes using limited labeled data. However, pretraining-based methods not only introduce excessive time overhead but also overlook the local structure representation among irregular point clouds. To address these issues, we propose a pretraining-free local structure fitting network for few-shot point cloud semantic segmentation, named TaylorSeg. Specifically, inspired by Taylor series, we treat the local structure representation of irregular point clouds as a polynomial fitting problem and propose a novel local structure fitting convolution, called TaylorConv. This convolution learns the low-order basic information and high-order refined information of point clouds from explicit encoding of local geometric structures. Then, using TaylorConv as the basic component, we construct two variant of TaylorSeg: a non-parametric TaylorSeg-NN and a parametric TaylorSeg-PN. The former can achieve performance comparable to existing parametric models without pretraining. For the latter, we equip it with an Adaptive Push-Pull (APP) module to mitigate the feature distribution differences between the query set and the support set. Extensive experiments validate the effectiveness of the proposed method. Notably, under the 2-way 1-shot setting, TaylorSeg-PN achieves improvements of +2.28% and +4.37% mIoU on the S3DIS and ScanNet datasets respectively, compared to the previous state-of-the-art methods.

Taylor Series-Inspired Local Structure Fitting Network for Few-shot Point Cloud Semantic Segmentation

Adversarial robustness in the context of Unsupervised Domain Adaptation (UDA) is particularly challenging due to the lack of labels in the target domain. Pseudo labels are often used to make adversarially robust models but compromise robustness and accuracy, falling short of the performance seen in adversarial deep learning models. The main challenges in achieving robustness and accuracy include ensuring reli-
able pseudo labels and developing effective training methods that align clean and adversarial examples during model training. To address these challenges, we propose a novel train-
ing method, Dual Source Guided Adversarial Training (DS-GAT), which jointly trains on clean and adversarial target examples using scratch source and target models with pseudo labels generated from the pretrained source model. We next update the pseudo labels to ensure their reliability against the
inevitable misclassified examples in the target domain during training. We further integrate the Mart loss and minimize the Kullback-Leibler (KL) divergence between the softmax outputs of clean and adversarial examples to significantly enhance robustness and accuracy across multiple benchmark
datasets, including OfficeHome, PACS, and VisDA. Our proposed method achieves an average relative improvement in accuracy of 8.6% and in robustness of 10.2% compared to state-of-the-art methods on the PACS dataset.

Toward Improving Robustness and Accuracy in Unsupervised Domain Adaptation

Automatic Heuristic Design (AHD) is an active research area due to its utility in solving complex search and NP-hard combinatorial optimization problems in the real world. The recent advancements in Large Language Models (LLMs) introduce new possibilities by coupling LLMs with evolutionary computation to automatically generate heuristics, known as LLM-based Evolutionary Program Search (LLM-EPS). While previous LLM-EPS studies obtained great performance on various tasks, there is still a gap in understanding the properties of heuristic search spaces and achieving a balance between exploration and exploitation, which is a critical factor in large heuristic search spaces. In this study, we address this gap by proposing two diversity measurement metrics and perform an analysis on previous LLM-EPS approaches, including FunSearch, EoH, and ReEvo. Results on black-box AHD problems reveal that while EoH demonstrates higher diversity than FunSearch and ReEvo, its objective score is unstable. Conversely, ReEvo's reflection mechanism yields good objective scores but fails to optimize diversity effectively. With this finding in mind, we introduce HSEvo, an adaptive LLM-EPS framework that maintains a balance between diversity and convergence with a harmony search algorithm. Through experimentation, we find that HSEvo achieved high diversity indices and good objective scores while remaining cost-effective. These results underscore the importance of balancing exploration and exploitation and understanding heuristic search spaces in designing frameworks in LLM-EPS.

HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs

Prompt optimization automatically optimizes prompting expressions, unlocking the full potential of LLMs in downstream tasks. However, current prompt optimization methods are costly to train and lack sufficient interpretability. This paper proposes enhancing LLMs' reasoning performance by eliciting their causal inference ability from prompting instructions to correct answers. Specifically, we introduce the Self-Causal Instruction Enhancement (SCIE) method, which enables LLMs to generate high-quality, low-quantity observational data, then estimate the causal effect based on this data, and ultimately generate instructions with the optimized causal effect. In SCIE, the instructions are treated as the treatment, and proxy features are used to process natural language, establishing causal relationships through proxy treatments between instructions and downstream tasks. Additionally, we propose applying Object-Relational (OR) principles, where the uncovered causal relationships are treated as the inheritable class across task objects, ensuring low-cost reusability. Extensive experiments demonstrate that our method effectively generates instructions that enhance reasoning performance with reduced training cost, leveraging interpretable textual features to provide clear and actionable insights.

Premium content

Next from AAAI 2025

Enhancing Non-English Capabilities of English-Centric Large Language Models Through Deep Supervision Fine-Tuning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES