Live - Sustainability Working Group

The diverse range of naval vessels, ranging from aircraft carriers to unmanned surface vessels, and infrastructure, such as smart naval mines or buoys, presents a challenge for traditional machine learning (ML) techniques in naval defense. Traditional ML, which centralizes network data, struggles with the vast distances and unstable or insecure communication links typical in naval defense scenarios. 

Federated learning (FL) offers a solution by keeping the data and ML model training local to the naval vessels and only performing occasional ML model combinations, or "global aggregations" at a central server, or "mothership" for naval applications. FL does address some of the heterogeneity inherent to the naval setting, for example, it minimizes the communication burden during deployments through sporadic global ML model aggregations. "Statistical heterogeneity" here refers to data-related differences among network elements, such as various dataset sizes. "Systems heterogeneity" relates to physical differences between naval objects, such as varying reliability of communication links. 

However, existing literature in FL typically assumes the presence of fully labeled datasets across all naval elements, an unrealistic scenario in practical naval deployments where a diverse mixture of both \textbf{labeled} and \textbf{unlabeled} data is common. Current FL-based methodologies tend to produce a uniform global ML model for all naval elements, neglecting the information contained with the unlabeled data, potentially leading to suboptimal global ML models for diverse naval resources. 

To address this, we introduce a decentralized federated domain adaptation methodology which leverages both labeled and unlabeled data to identify the underlying statistical divergences among naval vessels. In doing so, we are able to optimize the transfer of ML models from naval elements with high quality labeled data (called sources) to naval resources with low quality or unlabeled data (called targets), thereby personalizing ML models to better fit the real-time needs of different naval vessels in a deployment. Our methodology, called Source-Target Determination and Link Formation (ST-LF), optimizes both (i) classification of naval resources into sources and targets and (ii) source-target link formation, in a manner that considers the trade-off between ML model accuracy and communication energy expenditures. To obtain a concrete objective function, we derive a measurable generalization error bound that accounts for estimates of source-target hypothesis deviations and divergences between data distributions. The resulting optimization problem is a mixed-integer signomial program, a class of NP-hard problems, for which we develop an algorithm based on successive convex approximations to solve it tractably. Subsequent numerical evaluations of ST-LF demonstrate that it improves classification accuracy and energy efficiency over state-of-the-art baselines.

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

Recent studies of the computational power of recurrent neural networks (RNNs) reveal a hierarchy of RNN architectures, given real-time and finite-precision assumptions. Here we study auto-regressive Transformers with linearised attention, a.k.a. linear Transformers (LTs) or Fast Weight Programmers (FWPs). LTs are special in the sense that they are equivalent to RNN-like sequence processors with a fixed-size state, while they can also be expressed as the now-popular self-attention networks. We show that many well-known results for the standard Transformer directly transfer to LTs/FWPs. Our formal language recognition experiments demonstrate how recently proposed FWP extensions such as recurrent FWPs and self-referential weight matrices successfully overcome certain limitations of the LT, e.g., allowing for generalisation on the parity problem. Our code is public.

Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions

Despite the popularity of the pre-train then fine-tune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

Energy and Carbon Considerations of Fine-Tuning BERT

Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency—a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2$\times$ hardware efficiency improvement compared to 8-bit integer MAC unit.

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

Automated Essay Scoring (AES) aims to automatically assess the quality of essays. Automation enables large-scale assessment, improvements in consistency, reliability, and standardization. Those characteristics are of particular relevance in the context of language certification exams. However, a major bottleneck in the development of AES systems is the availability of corpora, which, unfortunately, are scarce, especially for languages other than English. In this paper, we aim to foster the development of AES for French by providing the TCFLE-8 corpus, a corpus of 6.5k essays collected in the context of the \textit{Test de Connaissance du Français} (TCF - French Knowledge Test) certification exam. We report the strict quality procedure that led to the scoring of each essay by at least two raters according to the CEFR levels and to the creation of a balanced corpus. In addition, we describe how linguistic properties of the essays relate to the learners' proficiency in TCFLE-8. We also advance the state-of-the-art performance for the AES task in French by experimenting with two strong baselines (i.e. RoBERTa and feature-based). Finally, we discuss the challenges of AES using TCFLE-8.

TCFLE-8: a Corpus of Learner Written Productions for French as a Foreign Language and its Application to Automated Essay Scoring | VIDEO

Knowledge graph completion (KGC) aims to predict unseen edges in knowledge graphs (KGs), resulting in the discovery of new facts. A new class of methods have been proposed to tackle this problem by aggregating path information. These methods have shown tremendous ability in the task of KGC. However they are plagued by efficiency issues. Though there are a few recent attempts to address this through learnable path pruning, they often sacrifice the performance to gain efficiency. In this work, we identify two intrinsic limitations of these methods that affect the efficiency and representation quality. To address the limitations, we introduce a new method, TAGNet, which is able to efficiently propagate information. This is achieved by only aggregating paths in a fixed window for each source-target pair. We demonstrate that the complexity of TAGNet is independent of the number of layers. Extensive experiments demonstrate that TAGNet can cut down on the number of propagated messages by as much as $90$% while achieving competitive performance on multiple KG datasets.

Distance-Based Propagation for Efficient Knowledge Graph Reasoning | VIDEO

The most recent approaches for clustering single-cell RNA-sequencing data rely on deep auto-encoders. However, three major challenges remain unaddressed. First, current models overlook the impact of the cumulative errors induced by the pseudo-supervised embedding clustering task (Feature Randomness). Second, existing methods neglect the effect of the strong competition between embedding clustering and reconstruction (Feature Drift). Third, the previous deep clustering models regularly fail to consider the topological information of the latent data, even though the local and global latent configurations can bring complementary views to the clustering task. To address these challenges, we propose a novel approach that explores the interaction between local and global latent configurations to progressively adjust the reconstruction and embedding clustering tasks. We elaborate a topological and probabilistic filter to mitigate Feature Randomness and a cell-cell graph structure and content correction mechanism to counteract Feature Drift. The Zero-Inflated Negative Binomial model is also integrated to capture the characteristics of gene expression profiles. We conduct detailed experiments on real-world datasets from multiple representative genome sequencing platforms. Our approach outperforms the state-of-the-art clustering methods in various evaluation metrics. 

Exploring the Interaction between Local and Global Latent Configurations for Clustering Single-cell RNA-seq: A Unified Perspective

Current generative transformer-based models have achieved state-of-the-art performance in long document summarization. However, this task witnessed a paradigm shift in developing ever-increasingly computationally-hungry models, focusing on effectiveness while ignoring the economic, environmental, and social costs of producing such results. Consequently, such extensive resources impact climate change and raise barriers to small and medium organizations characterized by low-resource regimes of hardware and data. As a result, this unsustainable trend has lifted many concerns in the community, proposing tools to monitor models' energy costs. Despite their importance, no evaluation measure considering models' eco-sustainability exists yet. In this work, we propose Carburacy, the first carbon-aware accuracy measure that captures both model effectiveness and eco-sustainability. We perform an extensive benchmark for long document summarization, comparing multiple state-of-the-art quadratic and linear transformers on several datasets under eco-sustainable regimes. Finally, thanks to Carburacy, we found optimal combinations of hyperparameters that let models be competitive in effectiveness with significantly lower costs.

Carburacy: Summarization Models Tuning and Comparison in Eco-Sustainable Regimes with a Novel Carbon-Aware Accuracy

Modern power systems will have to face difficult challenges in the years to come: frequent blackouts in urban areas caused by high peaks of electricity demand, grid instability exacerbated by the intermittency of renewable generation, and climate change on a global scale amplified by increasing carbon emissions. While current practices are growingly inadequate, the pathway of artificial intelligence (AI)-based methods to widespread adoption is hindered by missing aspects of trustworthiness. The CityLearn Challenge is an exemplary opportunity for researchers from multi-disciplinary fields to investigate the potential of AI to tackle these pressing issues within the energy domain, collectively modeled as a reinforcement learning (RL) task. Multiple real-world challenges faced by contemporary RL techniques are embodied in the problem formulation. In this paper, we present a novel method using the solution function of optimization as policies to compute the actions for sequential decision-making, while notably adapting the parameters of the optimization model from online observations. Algorithmically, this is achieved by an evolutionary algorithm under a novel trajectory-based guidance scheme. Formally, the global convergence property is established. **Our agent ranked first in the latest 2021 CityLearn Challenge**, being able to achieve superior performance in almost all metrics while maintaining some key aspects of interpretability.  

Underline digital video library

Live - Sustainability Working Group

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Computing Motion from Events-Based Cameras for UAV Surveillance

Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions

Energy and Carbon Considerations of Fine-Tuning BERT

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

TCFLE-8: a Corpus of Learner Written Productions for French as a Foreign Language and its Application to Automated Essay Scoring | VIDEO

Distance-Based Propagation for Efficient Knowledge Graph Reasoning | VIDEO

Exploring the Interaction between Local and Global Latent Configurations for Clustering Single-cell RNA-seq: A Unified Perspective

Carburacy: Summarization Models Tuning and Comparison in Eco-Sustainable Regimes with a Novel Carbon-Aware Accuracy

Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidance

Adaptive Temporal Planning for Multi-Robot Systems in Operations and Maintenance of Offshore Wind Farms

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES