Live - Sustainability Working Group

Genetic Algorithms (GAs) have been studied across different fields such as engineering or medicine to optimize diverse problems such as network routing, or medical image segmentation. Moreover, they have been used to automatically find optimal architectures for deep neural networks. However, to our knowledge, they have not been applied as a weight optimizer for the Transformer model. While gradient descent has been the main paradigm for this task, we believe that GAs have advantages to bring to the table. In this paper, we will show that even though GAs are capable of fine-tuning Transformer encoders, their generalization ability is considerably poorer than that from Adam; however, on a closer look, GAs ability to exploit knowledge from 2 different pretraining datasets surpasses Adam's ability to do so.

How Well Can a Genetic Algorithm Fine-tune Transformer Encoders? A First Approach

The diverse range of naval vessels, ranging from aircraft carriers to unmanned surface vessels, and infrastructure, such as smart naval mines or buoys, presents a challenge for traditional machine learning (ML) techniques in naval defense. Traditional ML, which centralizes network data, struggles with the vast distances and unstable or insecure communication links typical in naval defense scenarios. 

Federated learning (FL) offers a solution by keeping the data and ML model training local to the naval vessels and only performing occasional ML model combinations, or "global aggregations" at a central server, or "mothership" for naval applications. FL does address some of the heterogeneity inherent to the naval setting, for example, it minimizes the communication burden during deployments through sporadic global ML model aggregations. "Statistical heterogeneity" here refers to data-related differences among network elements, such as various dataset sizes. "Systems heterogeneity" relates to physical differences between naval objects, such as varying reliability of communication links. 

However, existing literature in FL typically assumes the presence of fully labeled datasets across all naval elements, an unrealistic scenario in practical naval deployments where a diverse mixture of both \textbf{labeled} and \textbf{unlabeled} data is common. Current FL-based methodologies tend to produce a uniform global ML model for all naval elements, neglecting the information contained with the unlabeled data, potentially leading to suboptimal global ML models for diverse naval resources. 

To address this, we introduce a decentralized federated domain adaptation methodology which leverages both labeled and unlabeled data to identify the underlying statistical divergences among naval vessels. In doing so, we are able to optimize the transfer of ML models from naval elements with high quality labeled data (called sources) to naval resources with low quality or unlabeled data (called targets), thereby personalizing ML models to better fit the real-time needs of different naval vessels in a deployment. Our methodology, called Source-Target Determination and Link Formation (ST-LF), optimizes both (i) classification of naval resources into sources and targets and (ii) source-target link formation, in a manner that considers the trade-off between ML model accuracy and communication energy expenditures. To obtain a concrete objective function, we derive a measurable generalization error bound that accounts for estimates of source-target hypothesis deviations and divergences between data distributions. The resulting optimization problem is a mixed-integer signomial program, a class of NP-hard problems, for which we develop an algorithm based on successive convex approximations to solve it tractably. Subsequent numerical evaluations of ST-LF demonstrate that it improves classification accuracy and energy efficiency over state-of-the-art baselines.

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

Real-time semantic matching is vital to web and product search. Transformer-based models have shown to be highly effective at encoding queries into an embedding space where semantically similar entities (queries or results) are in close proximity. However, the computational complexity of large transformer models limits their utilization for real-time matching. In this paper, we propose KD-Boost, a novel knowledge distillation algorithm designed for real-time semantic matching. KD-Boost trains low latency accurate student models by leveraging soft labels from a teacher model as well as ground truth via pairwise query-product and query-query signal derived from direct audits, user behavior, and taxonomy-based data using custom loss functions. Experiments on internal and external e-commerce datasets demonstrate an improvement of 2-3% ROC-AUC compared to training student models directly, outperforming teacher and SOTA knowledge distillation benchmarks. Simulated online A/B tests using KD-Boost for automated Query Reformulation (QR) indicate a 6.31% increase in query-to-query matching, 2.76% increase in product coverage, and a 2.19% improvement in relevance.

KD-Boost: Boosting Real-Time Semantic Matching in E-commerce with Knowledge Distillation

This paper introduces a new IncidentAI dataset for safety prevention. Different from prior corpora that usually contain a single task, our dataset comprises three tasks: named entity recognition, cause-effect extraction, and information retrieval. The dataset is annotated by domain experts who have at least six years of practical experience as high-pressure gas conservation managers.
We validate the contribution of the dataset in the scenario of safety prevention. Preliminary results on the three tasks show that NLP techniques are beneficial for analyzing incident reports to prevent future failures. The dataset facilitates future research in NLP and incident management communities. The access to the dataset is also provided (The IncidentAI dataset is available at: https://github.com/Cinnamon/incident-ai-dataset).

Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures | VIDEO

Chemical reactions, as a core entity in the realm of chemistry, hold crucial implications in diverse areas ranging from hands-on laboratory research to advanced computational drug design. Despite a burgeoning interest in employing NLP techniques to extract these reactions, aligning this task with the real-world requirements of chemistry practitioners remains an ongoing challenge. In this paper, we present Reaction Miner, a system specifically designed to interact with raw scientific literature, delivering precise and more informative chemical reactions. Going beyond mere extraction, Reaction Miner integrates a holistic workflow: it accepts PDF files as input, bypassing the need for pre-processing and bolstering user accessibility. Subsequently, a text segmentation module ensures that the refined text encapsulates complete chemical reactions, augmenting the accuracy of extraction. Moreover, Reaction Miner broadens the scope of existing pre-defined reaction roles, including vital attributes previously neglected, thereby offering a more comprehensive depiction of chemical reactions. Evaluations conducted by chemistry domain users highlight the efficacy of each module in our system, demonstrating Reaction Miner as a powerful tool in this field.

Reaction Miner: An Integrated System for Chemical Reaction Extraction from Textual Data

Code search engine is an essential tool in software development. Many code search methods have sprung up, focusing on the overall ranking performance of code search. In this paper, we study code search from another perspective by analyzing the bias of code search models. Biased code search engines provide poor user experience, even though they show promising overall performance. Due to different development conventions (e.g., prefer long queries or abbreviations), some programmers will find the engine useful, while others may find it hard to get desirable search results. To mitigate biases, we develop a general debiasing framework that employs reranking to calibrate search results. It can be easily plugged into existing engines and handle new code search biases discovered in the future. Experiments show that our framework can effectively reduce biases. Meanwhile, the overall ranking performance of code search gets improved after debiasing. Our implementation is available at: https://github.com/KDEGroup/CodeSearchDebiasing.

Bias and Debias in Code Search: Improve Search Results beyond Overall Ranking Performance

Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.

Quantifying the Dialect Gap and its Correlates Across Languages

Online forums encourage the exchange and discussion of different stances on many topics. Not only do they provide an opportunity to present one's own arguments, but may also gather a broad cross-section of others' arguments. However, the resulting long discussions are difficult to overview. This paper presents a novel unsupervised approach using large language models (LLMs) to generating indicative summaries for long discussions that basically serve as tables of contents. Our approach first clusters argument sentences, generates cluster labels as abstractive summaries, and classifies the generated cluster labels into argumentation frames resulting in a two-level summary. Based on an extensively optimized prompt engineering approach, we evaluate 19 LLMs for generative cluster labeling and frame classification. To evaluate the usefulness of our indicative summaries, we conduct a purpose-driven user study via a new visual interface called **Discussion Explorer**: It shows that our proposed indicative summaries serve as a convenient navigation tool to explore long discussions.

Indicative Summarization of Long Discussions | VIDEO

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.
Here, we perform a systematic analysis of this \emph{intertraining} scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed \emph{independently} for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

Where to start? Analyzing the potential value of intermediate models

Factual probing is a method that uses prompts to test if a language model ``knows'' certain world knowledge facts.
A problem in factual probing is that small changes to the prompt can lead to large changes in model output.
Previous work aimed to alleviate this problem by optimizing prompts via text mining or fine-tuning.
However, such approaches are relation-specific and do not generalize to unseen relation types.
Here, we propose to use test-time augmentation (TTA) as a relation-agnostic method for reducing sensitivity to prompt variations by automatically augmenting and ensembling prompts at test time.
Experiments show improved model calibration, i.e., with TTA, model confidence better reflects prediction accuracy.
Improvements in prediction accuracy are observed for some models, but for other models, TTA leads to degradation.
Error analysis identifies the difficulty of producing high-quality prompt variations as the main challenge for TTA.

Underline digital video library

Live - Sustainability Working Group

How Well Can a Genetic Algorithm Fine-tune Transformer Encoders? A First Approach

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Computing Motion from Events-Based Cameras for UAV Surveillance

KD-Boost: Boosting Real-Time Semantic Matching in E-commerce with Knowledge Distillation

Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures | VIDEO

Reaction Miner: An Integrated System for Chemical Reaction Extraction from Textual Data

Bias and Debias in Code Search: Improve Search Results beyond Overall Ranking Performance

Quantifying the Dialect Gap and its Correlates Across Languages

Indicative Summarization of Long Discussions | VIDEO

Where to start? Analyzing the potential value of intermediate models

Test-time Augmentation for Factual Probing | VIDEO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES