United States

The objective of Composed Image Retrieval (CIR) is to identify a target image that meets the requirement based on a multimodal query (including the reference image and the modification text) provided by the user. Despite the notable success of existing approaches, they fail to adequately address the modification relation between visual entities and modification actions. This limitation is non-trivial due to three challenges: 1) irrelevant factor perturbation, 2) vague semantic boundaries, and 3) implicit modification relations. To address the above challenges, we propose an Entity miNing and modifiCation relatiOn binDing nEtwoRk (ENCODER), which has been designed to mine visual entities and modification actions, and then bind modification relations. Among the various components of the proposed ENCODER, we have initially designed the Latent Factor Filter (LFF) module to filter visual and textual latent factors related to modification semantics based on a threshold gating mechanism. Secondly, we propose Entity-Action Binding (EAB), which comprises modality-shared Learnable Relation Queries (LRQ) that are capable of mining visual entities and modification actions, as well as learning implicit modification relations for entity-action binding. Finally, the Multi-scale Composition module is introduced to achieve multi-scale feature composition, with guidance provided by entity-action binding. Extensive experiments on four benchmark datasets demonstrate the superiority of our proposed method. Our codes and checkpoints are released in the supplement material.

AAAI 2025

ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval

poster

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



Dependency parsing is crucial in natural language processing for analyzing syntactic structures. Dependency parsers enhanced by pre-trained language models have achieved outstanding performance in high-resource languages. In contrast, cross-language dependency parsing is an effective strategy to learn useful knowledge from high-resource languages and compensate for the deficiency of low-resource languages. However, the key challenge for cross-language dependency parsing is to reduce distributional biases and excavate in-depth commonalities. To address this issue, we propose the novel dynamic syntactic feature filtering and injecting networks based on the traditional shared-private model which utilizes a shared and two private encoders to separate features from source or target languages. Concretely, a language-specific syntactic feature filtering network (LSFN) on private encoders emphasizes helpful information and ignores irrelevant or harmful information from the source language. Meanwhile, the language-invariant syntactic feature injection network (LIIN) on the shared encoder can combine the advantages of BiLSTM and improved transformer encoders to transcend language boundaries, thus amplifying syntactic commonalities across languages. We perform experiments on seven benchmark datasets to measure the efficacy of our proposed model and observe an average absolute gain of $1.84$ UAS and $3.43$ LAS compared with the shared-private model. Comparative experiments validate that both LSFN and LIIN components are complementary in transferring beneficial knowledge from source to target languages. Detailed analyses highlight that our model can effectively capture linguistic commonalities and minimize differences, showcasing its robustness and efficacy. Our code will be publicly available at https://flamelunar.github.io/.

Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing

Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding—accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.

Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-Tuning

Missing values in multivariate time series data can harm machine learning performance and introduce bias. These gaps arise from sensor malfunctions, blackouts, and human error. Previous work has addressed missing at random, complete blackouts, and forecasting scenarios. This paper addresses a more general missing pattern, termed $\textbf{partial blackout}$, where a subset of features is missing for consecutive time steps. This scenario is more common in real-world applications. We introduce a two-stage imputation process using self-attention and diffusion processes to model feature and temporal correlations. Our model outperforms state-of-the-art models in partial blackout scenarios and offers better scalability, promising practical data imputation solutions.

Self-attention-based Diffusion Model for Time-series Imputation in Partial Blackout Scenarios

With the prevalence of Multimodal Large Language Models(MLLMs), autonomous driving has encountered new opportunities and challenges. 
In particular, multi-modal video understanding is critical to interactively analyze what will happen in the procedure of autonomous driving.
However, videos in such a dynamical scene that often contains complex spatial-temporal movements,
which restricts the generalization capacity of the existing MLLMs in this field.
To bridge the gap, we propose a novel Hierarchical Mamba Adaptation (H-MBA) framework to fit the complicated motion changes in autonomous driving videos.
Specifically, our H-MBA consists of two distinct modules,
including Context Mamba (C-Mamba) and Query Mamba (Q-Mamba).
First, C-Mamba contains various types of structure state space models,
which can effectively capture multi-granularity video context for different temporal resolution.
Second, Q-Mamba flexibly transforms the current frame as the learnable query, 
and attentively select multi-granularity video context into query.
Consequently, it can adaptively integrate all the video contexts of multi-scale temporal resolutions to enhance video understanding.
Via a plug-and-play paradigm in MLLMs,
our H-MBA shows the remarkable performance on multi-modal video tasks in autonomous driving,
e.g., for risk object detection, it outperforms the previous SOTA method with 5.5\% mIoU improvement.

H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving

Face anti-spoofing (FAS) plays a pivotal role in ensuring the security and reliability of face recognition systems. With advancements in vision-language pretrained (VLP) models, recent two-class FAS techniques have leveraged the advantages of using VLP guidance, while this potential remains unexplored in one-class FAS methods. The one-class FAS focuses on learning intrinsic liveness features solely from live training images to differentiate between live and spoof faces. However, the lack of spoof training data can lead one-class FAS models to inadvertently incorporate domain information irrelevant to the live/spoof distinction (\eg, facial content), causing performance degradation when tested with a new application domain. To address this issue, we propose a novel framework called Spoof-aware one-class face anti-spoofing with Language Image Pretraining (SLIP). Given that live faces should ideally not be obscured by any spoof-attack-related objects (\eg, paper, or masks) and are assumed to yield zero spoof cue maps, we first propose an effective language-guided spoof cue map estimation to enhance one-class FAS models by simulating whether the underlying faces are covered by attack-related objects and generating corresponding nonzero spoof cue maps. Next, we introduce a novel prompt-driven liveness feature disentanglement to alleviate live/spoof-irrelative domain variations by disentangling live/spoof-relevant and domain-dependent information. Finally, we design an effective augmentation strategy by fusing latent features from live images and spoof prompts to generate spoof-like image features and thus diversify latent spoof features to facilitate the learning of one-class FAS. Our extensive experiments and ablation studies support that SLIP consistently outperforms previous one-class FAS methods.

SLIP: Spoof-Aware One-Class Face Anti-Spoofing with Language Image Pretraining

In this study, we consider an optimization problem with uncertainty dependent on decision variables, which has recently attracted attention due to its importance in applications. In this problem, the gradient of the objective function cannot be obtained explicitly because the decision-dependent distribution is unknown. Therefore, several zeroth-order methods have been proposed, which obtain noisy objective values by sampling and update the iterates.  Although these existing methods have theoretical convergence for optimization problems with decision-dependent uncertainty, they require strong assumptions about the function and distribution or exhibit large variances in their gradient estimators. To overcome these issues, we propose two zeroth-order methods under mild assumptions. First, we develop a zeroth-order method with a new one-point gradient estimator including a variance reduction parameter. The proposed method updates the decision variables while adjusting the variance reduction parameter. Second, we develop a zeroth-order method with a two-point gradient estimator. There are situations where only one-point estimators can be used, but if both one-point and two-point estimators are available, it is more practical to use the two-point estimator. As theoretical results, we show the convergence of our methods to stationary points and provide the worst-case iteration and sample complexity analysis. Our simulation experiments with real data on a retail service application show that our methods output solutions with lower objective values than the conventional zeroth-order methods.

Zeroth-Order Methods for Nonconvex Stochastic Problems with Decision-Dependent Distributions

Whole-body multimodal motion generation, controlled by text, speech, or music, has numerous applications including video generation and character animation. However, employing a unified model to achieve various generation tasks with different condition modalities presents two main challenges: motion distribution drifts across different tasks (e.g., co-speech gestures and text-driven daily actions) and the complex optimization of mixed conditions with varying granularities (e.g., text and audio). Additionally, inconsistent motion formats across different tasks and datasets hinder effective training toward multimodal motion generation. In this paper, we propose $\textbf{MotionCraft}$, a unified diffusion transformer that crafts whole-body motion with plug-and-play multimodal control. Our framework employs a coarse-to-fine training strategy, starting with the first stage of text-to-motion semantic pre-training, followed by the second stage of multimodal low-level control adaptation to handle conditions of varying granularities. To effectively learn and transfer motion knowledge across different distributions, we design $\textbf{MC-Attn}$ for parallel modeling of static and dynamic human topology graphs. To overcome the motion format inconsistency of existing benchmarks, we introduce $\textbf{MC-Bench}$, the first available multimodal whole-body motion generation benchmark based on the unified SMPL-X format. Extensive experiments show that $\textbf{MotionCraft}$ achieves state-of-the-art performance on various standard motion generation tasks.

MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls

This paper proposes a new principled multi-task representation learning framework (InfoMTL) to extract noise-invariant sufficient representations for all tasks. It promotes shared representations to preserve necessary information for all tasks and task-specific representations to eliminate redundant information for each task, which can enhance language understanding of pre-trained language models (PLMs) under the multi-task paradigm. Firstly, a shared information maximization principle is proposed to learn more sufficient shared representations for all target task. It adopts an opposite way with existing IB-based MTL to avoid the insufficiency issue. Secondly, a task-specific information minimization (TIMin) is proposed to mitigate the negative effect of potential redundant features in the shared representations for the target task. It alleviates the redundancy issue during the decoding process, which can compress task-irrelevant redundant information and preserves necessary information relevant to the target for multi-task prediction. Experiments on six classification benchmarks demonstrate that our method outperforms comparative multi-task learning approaches with different PLMs. Extensive experiments show that the shared and output representations are more sufficient, data-efficient, and robust.

An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understanding

The performance of Large Language Models (LLMs) is intrinsically linked to the quality of its training data. Although several studies have proposed methods for high-quality data selection, they do not consider the importance of knowledge richness in text corpora. In this paper, we propose a novel and gradient-free High-Knowledge Scorer (HKS) to select high-quality data from the dimension of knowledge, to alleviate the problem of knowledge scarcity in the pre-trained corpus. We propose a comprehensive multi-domain knowledge element pool and introduce knowledge density and coverage as metrics to assess the knowledge content of the text. Based on this, we propose a comprehensive knowledge scorer to select data with intensive knowledge, which can also be utilized for domain-specific high-knowledge data selection by restricting knowledge elements to the specific domain. We train a 1.1B model on a high-knowledge bilingual dataset, and experimental results demonstrate that our scorer improves the model's performance in knowledge-intensive and general comprehension tasks, and is effective in enhancing both the generic and domain-specific capabilities of the model.

Enhancing LLMs via High-Knowledge Data Selection

Data sharing is necessary for AI to be widely used, but sharing sensitive data with others with privacy is risky.
To solve these problems, it is necessary to synthesize realistic tabular data.
In many cases, tabular data contains a mixture of continuous, mixed, categorical columns.
Moreover, columns of the same type may have multimodal distribution or be highly imbalanced.
These issues make it challenging to synthesize tabular data.
The synthesized tabular data should reflect the relational meaning between columns of tabular data, so modeling the probability distribution of tabular data is a non-trivial task.
Traditional tabular data synthesizing models are based on GAN or diffusion models and are built using fully connected or convolutional layers.
However, fully connected layers have the disadvantage of low inductive bias, and convolutional layers are not invariant to the column order of tabular data.
Therefore, we assume that converting tabular data into graph-structured data and using a graph neural network would produce better synthetic data than using fully connected layers or convolutional layers.
Our study aims to show that GANs constructed with graph neural networks can outperform existing GAN models using fully connected layers or convolutional layers.
We propose CG-TGAN, a conditional GAN built using graph neural networks. To learn how to synthesize realistic data, the graph neural networks in the discriminator and generator learn graph-level tasks and node-level tasks together.
The discriminator of CG-TGAN learns a graph-level task to distinguish between real and synthetic data and node-level tasks to predict the value of the target node.
CG-TGAN’s generator learns a graph-level task to synthesize an overall graph similar to real data and node-level tasks to learn how to synthesize a fake graph with the proper relation between nodes.
In this paper, we show that CG-TGAN outperforms GAN-based models and is comparable to diffusion-based models.

ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval

Downloads

Next from AAAI 2025

Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2025

Dynamic Syntactic Feature Filtering and Injecting Networks for Cross-lingual Dependency Parsing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads