United States

Weakly-Supervised Dense Video Captioning (WSDVC) aims to localize and describe all events of interest in a video without requiring annotations of event boundaries. This setting poses a great challenge in accurately locating the temporal location of event, as the relevant supervision is unavailable. Existing methods rely on explicit alignment constraints between event locations and captions, which involve complex event proposal procedures during both training and inference. To tackle this problem, we propose a novel implicit location-caption alignment paradigm by complementary masking, which simplifies the complex event proposal and localization process while maintaining effectiveness. Specifically, our model comprises two components: a dual-mode video captioning module and a mask generation module. The dual-mode video captioning module captures global event information and generates descriptive captions,
while the mask generation module generates differentiable positive and negative masks for localizing the events. These masks enable the implicit alignment of event locations and captions by ensuring that captions generated from positively and negatively masked videos are complementary, thereby forming a complete video description. In this way, even under weak supervision, the event location and event caption can be aligned implicitly. Extensive experiments on the public datasets demonstrate that our method outperforms existing weakly-supervised methods and achieves competitive results compared to fully-supervised methods. Code will be released.

AAAI 2025

Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning

mining of visual multimedia

dmkm

multimodal data

technical paper

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



Tensor decomposition (TD) models are promising solutions for knowledge graph completion due to their simple structures but powerful representation capacities. The TD models typically adopt Tucker decomposition with a structured core tensor. Some models with a sparse core tensor, such as DistMult and ComplEx, are too simple and thus limit the interaction between embedding components, while other models with a dense core tensor are too complex and may lead to significant overfitting. To address these issues, we propose a new TD model called SPAC (Sparse Partitioning and Adaptive Core tensor pruning) model for knowledge graph completion. Specifically, SPAC captures coarse and fine-grained semantic information using a hybrid core tensor, where auxiliary cores are used to model sparse interactions and main cores for dense interactions. Moreover, SPAC introduces a gating mechanism to control the output of intermediate variables, enhancing the interaction between different partition groups. Furthermore, SPAC employs an adaptive pruning approach to dynamically adjust the shape of the core tensor. Due to the elaborate model design, the proposed TD model enhances expressive capacity and reduces the number of parameters in the core tensor. Experiments are conducted on datasets FB15k-237, WN18RR, and YAGO3-10. The results demonstrate that SPAC outperforms state-of-the-art tensor decomposition models, including MEIM and Tucker models. A series of ablation studies show that the gating mechanism and adaptive pruning strategy in SPAC are crucial for the performance improvement.

SPAC: Sparse Partitioning and Adaptive Core Tensor Pruning Model for Knowledge Graph Completion

The field of music generation has seen a surge of interest from both academia and industry, with innovative platforms such as Suno, Udio, and SkyMusic earning widespread recognition. However, the challenge of music infilling—modifying specific music segments without reconstructing the entire piece—remains a significant hurdle for both audio-based and symbolic-based models, limiting their adaptability and practicality. In this paper, we address symbolic music infilling by introducing the Collaborative Music Inpainter (CMI), an advanced human-in-the-loop (HITL) model for music infilling. The CMI features the Joint Embedding Predictive Autoregressive Generative Architecture (JEP-AGA), which learns the high-level predictive representations of the masked part that needs to be infilled during the autoregressive generative process, akin to how humans perceive and interpret music. The newly developed Dynamic Interaction Learner (DIL) achieves HITL by iteratively refining the infilled output based on user interactions alone, significantly reducing the interaction cost without requiring further input. Experimental results confirm CMI’s superior performance in music infilling, demonstrating its efficiency in producing high-quality music.

Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling

Knowledge compilation is a method of transforming knowledge into a compressed and tractable form, permitting more efficient operations. For Boolean functions, numerous representations have been proposed to enhance succinctness and tractability. In this paper, we introduce a new representation named structured Decomposable And-Sum Circuit (st-DASC), which employs $\wedge$ and $+$ nodes with signed edges, in place of the standard $\wedge$ and $\vee$ nodes with unsigned edges. Notably, incorporating negative signs permits polytime logical negation. By following the knowledge compilation map, we show that st-DASC is more succinct than the Sentential Decision Diagram (SDD) while still maintaining support for all of the operations on the knowledge compilation map that SDD supports. Furthermore, st-DASC is even more succinct than structured d-DNNF (st-d-DNNF), which itself is more succinct but supports fewer operations than SDD. Accordingly, st-DASC breaks the traditional trade-off between succinctness and tractability.

An and-Sum Circuit with Signed Edges That Is More Succinct than SDD

This paper presents VDAct, a dataset for a Video-grounded Dialogue on Event-driven Activities, alongside VDEval, a session-based context evaluation metric specially designed for the task. Unlike existing datasets, VDAct includes longer and more complex video sequences that depict a variety of event-driven activities that require advanced contextual understanding for accurate response generation. The dataset comprises 3,000 dialogues with over 30,000 question-and-answer pairs, derived from 1,000 videos with diverse activity scenarios. VDAct displays a notably challenging characteristic due to its broad spectrum of activity scenarios and wide range of question types. Empirical studies on state-of-the-art vision foundation models highlight their limitations in addressing certain question types on our dataset. Furthermore, VDEval, which integrates dialogue session history and video content summaries extracted from our supplementary KGs to evaluate individual responses, demonstrates a significantly higher correlation with human assessments on the VDAct dataset than existing evaluation metrics that rely solely on the context of single dialogue turns.

A Video-grounded Dialogue Dataset and Metric for Event-driven Activities

Deep reinforcement learning (DRL) has gained significant attention in autonomous systems, yet its black-box nature and lack of explainability hinder user trust in safety-critical domains such as autonomous driving. Existing experience replay approaches enhance sample efficiency but often fail to capture the internal causality of training data, leading to a convoluted training process that is difficult for humans to explain. In this work, we introduce Experience Replay with Causal Inference (ERCI), an explainable approach that integrates time series representation and causal inference to offer human-aligned explanations for DRL. Specifically, ERCI 1) introduces a novel multivariate time series representation to extract explainable Time Series Causal Factors (TSCF) from experimental data and 2) leverages internal causality in TSCFs with causal inference as a crucial standard for experience replay in DRL training. We evaluate ERCI using multiple baseline algorithms across diverse environments. Results show that ERCI provides human-aligned explanations and further improves sample efficiency through enhanced explainability. Notably, ERCI outperforms other state-of-the-art approaches by 15% in average performance, highlighting its effectiveness and generalizability.

ERCI: An Explainable Experience Replay Approach with Causal Inference for Deep Reinforcement Learning

This study evaluates the usage of OpenAI’s ChatGPT Large Language Model (LLM) as a tool for constructing multiple choice questions for assessing student academic performance through quizzes and exams. We randomly deploy questions constructed with and without use of the LLM tool and gauge the ability of the students to correctly answer, as well as their ability to correctly perceive the difference between human-authored and LLM-authored questions. In determining whether the questions written with the aid of ChatGPT were consistent with the instructor's questions and source text, we computed representative vectors of both the human and ChatGPT questions using SBERT and compared cosine similarity to the course textbook. A non-significant Mann-Whitney U test ($\textit{z}$ = 1.018, $\textit{p}$ = .309) suggests that students were unable to perceive whether questions were written with or without the aid of ChatGPT. However, student scores on LLM-authored questions were almost 9\% lower ($\textit{z}$ = 2.702, $\textit{p}$ $<$ .01). This result may indicate that either the AI questions were more difficult or that the students were more familiar with the instructor’s style of questions. Overall, the study suggests that while there is potential for using LLM tools to aid in the construction of assessments, care must be taken to ensure that the questions are fair, well-composed, and relevant to the course material.

LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI

This paper addresses the problem of preference learning, which aims to align robot behaviors through learning user-specific preferences (e.g., “good pull-over location”) from visual demonstrations. Despite its similarity to learning factual concepts (e.g., “red cube”), preference learning is a fundamentally harder problem due to its subjective nature and the paucity of person-specific training data. We address this problem using a novel framework called Synapse, which is a neuro-symbolic approach designed to efficiently learn preferential concepts from limited data. Synapse represents preferences as neuro-symbolic programs -- facilitating inspection of individual parts for alignment -- in a domain-specific language (DSL) that operates over images and leverages a novel combination of visual parsing, large language models, and program synthesis to learn programs representing individual preferences. We perform extensive evaluations on various preferential concepts as well as a small user case study demonstrating the ability of Synapse to align well with dissimilar user preferences. Our method significantly outperforms baselines, especially in out-of-distribution generalization. We show the importance of the design choices in the framework through multiple ablation studies.

SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine

Researchers, policymakers, and developers of artificial intelligence (AI) are actively collaborating to establish trustworthy AI standards that align with broader societal values, particularly in the context of large language models (LLMs). However, the critical discourse on bridging the vast knowledge gap between experts who shape and implement standards for LLMs and end-users whose values are at stake remains largely unaddressed. Taking a ‘bottom-up’ approach and using a mixed-method approach, we first conducted interviews (N = 12) to engage with end-users’ perceptions of normative standards in the context of LLMs. We thereby identified 68 criteria that end-users’ consider when evaluating value congruence. Second, we conducted an online survey (N = 379) to further investigate how end-users prioritise these standards and the identified criteria in conversational LLM-based applications. Findings reveal a clear preference for governance, monitoring, and transparent information provision over other criteria and highlight the necessity of non-technical complements to technical solutions for bridging the knowledge gap. We discuss end-user information requirements and outline actionable steps to effectively communicate trustworthy AI standards.

Bridging the Knowledge Gap: Understanding User Expectations for Trustworthy LLM Standards

Despite significant progress has been made in image deraining, most existing methods are limited to handling only a single type of rain degradation or a specific pattern of rain. However, real-world rain scenarios tend to contain diverse rainy patterns due to variations in the rainfall process and lighting conditions. To address this dilemma and advance this field, we introduce a new task: Universal Rainy Image Restoration (URIR), which aims to handle multiple types of rain degradation on a single model. To benchmark this task, we construct a high-quality dataset called URIR-8K, which contains four patterns: rain streak, raindrop, rain accumulation and nighttime rain. Building upon this dataset, we present a comprehensive study on existing approaches by evaluating their universal deraining capabilities and their effect on downstream object detection task. In addition, we design a multi-scale vision Mamba as a baseline model, leveraging the benefits of multi-scale learning for its robustness to diverse rain appearances. Unlike existing methods that use fixed-scale scanning for feature extraction, we employ a multi-scale 2D scanning technique to better help image restoration in the richer scale space. Extensive experimental analysis demonstrates the potential of our proposed task and the effectiveness of the baseline model. The code and dataset will be available.

Towards Universal Rainy Image Restoration: Benchmark and Baseline

Backdoor attacks have posed a serious threat in machine learning models, wherein adversaries can poison training samples with maliciously crafted triggers to compromise the victim model. Advanced backdoor attack methods have focused on selectively poisoning more vulnerable training samples, achieving a higher attack success rate (ASR). However, we found that when the manipulation strength of the trigger is constrained to a very small value for imperceptible attacks, they suffer from extremely uneven class-wise ASR due to the unequal selection of instances per class. To solve this issue, we propose a novel backdoor attack method based on \underline{I}nfluence-based \underline{F}air \underline{S}election (IFS), including two objectives: 1) selecting samples that significantly contribute to ASR and 2) ensuring class balance during the selection process. Specifically, we adapt Influence Functions, a classic technique in robust statistics, to evaluate the influence of trigger-embedded training samples on ASR. In this case, training samples contributing to reducing the backdoored test risk could possess higher influence scores. Further, a group-based pruning strategy is designed to avoid calculating the influence on ASR for all training samples, thereby significantly reducing the computational cost. Then, based on the influence score, we design an adaptive thresholding scheme to dynamically select samples with higher influence while maintaining class balance. Extensive experiments on four datasets verify the effectiveness of IFS compared with advanced methods.

Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning

Downloads

Next from AAAI 2025

SPAC: Sparse Partitioning and Adaptive Core Tensor Pruning Model for Knowledge Graph Completion

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2025

SPAC: Sparse Partitioning and Adaptive Core Tensor Pruning Model for Knowledge Graph Completion

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads