United States

Federated learning (FL) is a distributed machine learning paradigm that enables clients to collaborate without accessing, infringing upon, or leaking user raw data by sharing only model parameters. Multimodal automatic target recognition (ATR) in FL aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this study, we propose multimodal fusion Federated learning with joint Modality and Client selection (mmFedMC), a new FL framework for multimodal ATR that can tackle the above-mentioned challenges. The joint selection algorithm incorporates two main components: (a) A modality selection criterion for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. This allows mmFedMC to flexibly balance performance against communication costs, depending on resource constraints and application requirements. (b) A client selection for the server is performed on the basis of the cross-entropy of prediction as a metric. This involves selecting a subset of clients for uploading and aggregating models, which further reduces communication overhead and optimizes the overall process. Experiments on a real-world dataset with six modalities demonstrate that mmFedMC achieves 98.19\% accuracy with a communication overhead of only 5MB per client. This represents an approximate 10\% increase in accuracy compared to random modality or client selection. Moreover, in comparison to other state-of-the-art methods that do not utilize a selection mechanism, mmFedMC enhances accuracy by a factor of 2x to 4x. Alternatively, to achieve comparable levels of accuracy, other methods require approximately 20x more communication overhead, i.e., about 100MB per client. Beyond the trade-off between performance and communication overhead, mmFedMC also offers an interpretable representation of the modality impact during the FL process. In the initial stages of FL, most modalities exhibit similar impacts. However, as the communication rounds progress, modalities with larger feature sets and complex models play a subordinate role in modality selection due to their higher communication costs. With the advancement of FL, more straightforward modalities that continue to convey substantial information emerge as primary contributors.

NAML 2024

Communication-Efficient Federated Learning for Multimodal Automatic Target Recognition

multimodal federated learning

communication efficiency

edge computing

data fusion

internet of things

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

The objective of this line of research is to use Convolutional Neural
Networks (CNNs) to automatically identify corrosion in photographs.
This approach provides a reliable alternative to traditional
approaches. Corrosion, a widespread issue in several sectors,
necessitates prompt detection to avoid structural harm and guarantee
operational security. The use of deep learning, particularly
Convolutional Neural Networks (CNNs), offers an advanced method for
detecting corrosion. This is achieved by using the neural networks'
capability to identify complex patterns in picture data. This research
investigates the creation and assessment of a Convolutional Neural
Network (CNN) model that was trained using a dataset consisting of
photos depicting both corrosion and non-corrosion. The model's
effectiveness is assessed by analyzing key performance indicators such
as True Positive Rate, False Positive Rate, accuracy, recall, and area
under the ROC curve. The study illuminates the capabilities and
constraints of the CNN-based technique, providing useful insights for
businesses in need of effective and precise corrosion detection
solutions.


Corrosion Detection Using Deep Learning

We consider the problem faced by an intelligent attacker that wants to discover new attack paths to compromise a network in a cyber-security network intrusion scenario. This problem is challenging because the number of possible actions for the attacker could increase exponentially with the number of nodes in the network and the number of services running on those nodes. We formulate the attacker's problem as a reinforcement learning task and use a decision transformer network to tackle the complexity of the attacker’s action space. We have evaluated our techniques within a gamified network intrusion detection scenario called Cyber AI Gym Environment (CAGE). Our results show that the proposed approach can enable the attacker to learn successful attack strategies against two difficulty levels of defender strategies.

Decision Transformer for Learning Intelligent Attacker Strategies in a Network Intrusion Detection Game

The diverse range of naval vessels, ranging from aircraft carriers to unmanned surface vessels, and infrastructure, such as smart naval mines or buoys, presents a challenge for traditional machine learning (ML) techniques in naval defense. Traditional ML, which centralizes network data, struggles with the vast distances and unstable or insecure communication links typical in naval defense scenarios. 

Federated learning (FL) offers a solution by keeping the data and ML model training local to the naval vessels and only performing occasional ML model combinations, or "global aggregations" at a central server, or "mothership" for naval applications. FL does address some of the heterogeneity inherent to the naval setting, for example, it minimizes the communication burden during deployments through sporadic global ML model aggregations. "Statistical heterogeneity" here refers to data-related differences among network elements, such as various dataset sizes. "Systems heterogeneity" relates to physical differences between naval objects, such as varying reliability of communication links. 

However, existing literature in FL typically assumes the presence of fully labeled datasets across all naval elements, an unrealistic scenario in practical naval deployments where a diverse mixture of both \textbf{labeled} and \textbf{unlabeled} data is common. Current FL-based methodologies tend to produce a uniform global ML model for all naval elements, neglecting the information contained with the unlabeled data, potentially leading to suboptimal global ML models for diverse naval resources. 

To address this, we introduce a decentralized federated domain adaptation methodology which leverages both labeled and unlabeled data to identify the underlying statistical divergences among naval vessels. In doing so, we are able to optimize the transfer of ML models from naval elements with high quality labeled data (called sources) to naval resources with low quality or unlabeled data (called targets), thereby personalizing ML models to better fit the real-time needs of different naval vessels in a deployment. Our methodology, called Source-Target Determination and Link Formation (ST-LF), optimizes both (i) classification of naval resources into sources and targets and (ii) source-target link formation, in a manner that considers the trade-off between ML model accuracy and communication energy expenditures. To obtain a concrete objective function, we derive a measurable generalization error bound that accounts for estimates of source-target hypothesis deviations and divergences between data distributions. The resulting optimization problem is a mixed-integer signomial program, a class of NP-hard problems, for which we develop an algorithm based on successive convex approximations to solve it tractably. Subsequent numerical evaluations of ST-LF demonstrate that it improves classification accuracy and energy efficiency over state-of-the-art baselines.

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Our task is to create an equivalent to “follow me” mode for ships at sea. Allowing a Sailor to launch a small quadcopter and forget about it while it follows and circles the ship of interest could be incredibly useful. Our particular application is autonomous inspection and battle damage assessment, say in the aftermath of a Naval casualty. 

We develop a computer vision pipeline that uses a drone’s camera feed to track the behavior of a ship. It uses camera frames to estimate the ship’s position and orientation to potentially correct the drone’s trajectory, ensuring that the ship is always in view, even in adverse conditions. Our proposed method can also be effective in broader applications in fields like Autonomous Driving and Robotics, where the main agent takes an action based on its surrounding environment. 

The first stage of our computer vision pipeline consists of calculating the orientation of the ship. Our method passes the images from the camera through a UNet image segmentation model to get the ship’s mask and boundaries. We then use an OpenCV algorithm to generate a rotated rectangle that minimizes its area while enclosing the entire mask of the ship. This rotated bounding box effectively represents the direction that the ship is facing based on its physical features, so we use its angle to estimate the ship’s orientation with respect to the drone. This technique is beneficial because it is less complex than 3D Pose Estimation models, allowing for faster real-time processing of image data. 

For our image segmentation model to perform effectively in various environmental conditions, it is crucial to have a variety of flight paths, elevation angles, and visibility conditions to prevent overfitting. Collecting real data with these criteria is an expensive and laborious process, so we have started with Unity-generated synthetic data with relevant variations as ground truth labels for training. Using this data to train our computer vision model resulted in highly accurate predicted masks from unseen flight test images.

Estimating Ship Position and Orientation using Deep Neural Networks and Synthetic Data

While Federated Learning (FL) on a distributed network is used extensively in commercial applications, the safety-critical nature of Naval applications presents a unique set of challenges. Specifically, the method must be robust against dynamic network conditions, where devices and communication channels within the network may experience unpredictable changes during both training and inference. Vertical Federated Learning (VFL) is a class of FL in which each client within a distributed network shares a common sample space but possesses only a subset of the overall features. This setup is particularly pertinent in practical scenarios such as sensor networks deployed for surveillance, where each sensor (client) gathers specific data (features) about the same environment (sample space). While VFL addresses several privacy concerns inherent in distributed learning, it typically operates under the assumption of ideal hardware and seamless communication capability. This assumption is a major limiting factor in the broader application of VFL, especially in the context of edge devices. Edge devices are inherently heterogeneous in their operational capabilities, and may frequently connect or disconnect from the network, leading to a dynamic and unpredictable network environment.

To address this gap, we study the test time performance of VFL under dynamic network conditions, which we call DN-VFL. We first formalize DN-VFL, including a message passing distributed inference algorithm, the corresponding risk, and a serverless setup. We develop a novel DN-VFL approach that synthesizes replication, gossiping, and selective feature omission that improves performance significantly over baselines. Furthermore, we implement new methods, propose metrics, and extensively analyze results based on a simulated sensor network. The results show that naively using VFL for DN-VFL is not the best approach. Rather, the proposed methods present a better alternative to handle changes in the network during
inference.


Fault-Tolerant Vertical Federated Learning on Dynamic Networks

SysML (Systems Modeling Language) is a widely-used formalism and international standard for designing and communicating complex system designs, and is especially relied upon in the development of safety-critical autonomous systems. SysML v2 is a major new version of SysML currently undergoing finalization, scheduled to be released mid-2024. Among many key enhancements, SysML v2 introduces a formal semantics for specifying the precise meaning of SysML models, which makes it possible to apply formal verification and automated reasoning to analyze safety and correctness properties of SysML designs automatically.

At Imandra, we are applying our Imandra Automated Reasoning System to the design and analysis of SysML v2 models. Key to this work is a formal semantics for SysML v2 models expressed in Imandra's logic, and corresponding verification automation for answering deep questions about the possible behaviors of SysML models. In addition to verification, we can then leverage Imandra's "proof-carrying prose" integration with LLMs to obtain conversational assistants (chatbots) which can rigorously answer complex questions about SysML v2 designs, backing every answer up with an independently verifiable logical audit trail produced by Imandra reasoning about the SysML v2 model.

During this talk, we will demonstrate how to use Automated Reasoning with SysML v2 models to:
- Mathematically prove properties of system correctness.
- Add a hallucination-free natural-language interface (chatbot) to empower more stakeholders to gain insights and leverage the SysML v2 model.

Imandra + SysML: Automated Reasoning for SysML (Systems Modeling Language) v2

The ever-growing size of deep learning models necessitates more extensive resources, making them unsuitable for tasks requiring lower latency and less computational power. This is particularly true for edge devices. Multiple compression techniques have been developed to bring the capabilities of these larger models to a smaller form factor, one of the most successful of which has been knowledge distillation. Knowledge distillation is employed to transfer knowledge from a large "teacher" model to a smaller "student" model. This technique facilitates learning a compressed representation of data, maintaining the performance benefits of large models while reducing computational and memory demands. It involves training the student model on both hard labels and soft labels provided by the teacher model, leading to better generalization and informed predictions.

Our work increases the effectiveness of knowledge distillation by augmenting the labels of both the teacher and student models with a vector quantized vector. Vector quantization is a mapping of high-dimensional data to a lower-dimensional codebook, effectively reducing data dimensionality. This technique has traditionally been applied to speech and image compression particularly in Variational Autoencoders (VQ-VAEs), and has proven effective at markedly increasing the effectiveness of student models to learn from teacher model representations.

The work presented here seeks to enhance the classification capabilities of the student model, carefully maintaining its compact size to balance efficiency and compactness. This approach improves retained knowledge distillation performance, while conferring model interpretability benefits. We show that while also increasing student model classification performance, the inclusion of the VQ layer also provides a measure for detecting classification quality and detect data which is out of distribution. This approach promises to increase the capabilities of edge deployed models both with respect to their mission performance as well as out of distribution detection.


Improving Knowledge Distillation with Discrete Feature Vector Representation

My poster presentation explains documented co-symbolic AI decision support and automated security. It includes contracted use and exercise support with US Special Operations Command; US Army Intelligence Command; US Army Program Executive Officer for Simulation, Training, and Instrumentation; and the US Marine Corps Tactical Operation Group. By Light combines a mature multi-domain warfighting knowledge graph symbolic AI (expert system), a budding self-supervising artificial general intelligence (AGI), and a proven faster than real-time wargaming tool. Each of the parts come together to support warfighting decisions while overcoming critical warfighting AI vulnerabilities, not achievable by most AI. These include 1) allowing risk manageability/trust across assurance, attribution, understanding, traceability, and bias; 2) preventing model collapse; 3) eliminating human data annotation bias and expense; 4) affording real-time data tag, label, aggregation, and neural networking with minimal if any latency regardless of schema, structure, or unstructured; 5) surmounting data latency - neural networking 10 petabytes of historical data in less than year; 6) acting as master middleware, applying all data and integrating all apps with low cost of entry - only web capable; 7) learning warfighter user behavior, optimizing and customizing to predominantly push answers instead of users always querying an answer; 8) learning operating environment behavior in real-time and recognizing asymmetry; 9) applying multi-domain warfighter expertise across all battle and warfighting functions; and 10) cyber-securing artificial intelligence automation across all information technology, operational technology, and internet of things infrastructure, because it takes AI to secure AI.
The presentation will include government results from theater and tactical level multi-domain peer-to-peer exercises and video demonstration of the parts. It will challenge the audience to consider the warfighting vulnerabilities present in other AI, such as ChatGPT and introduce a suitable, feasible, secure, trustworthy, and acceptable path for warfighter AI decision support.


Knowledge Enhanced Multi-domain Artificial Intelligence

Naval Asset Corrosion and Defect Segmentation Using UNet Architecture 

Abstract: 

Corrosion poses a significant challenge in the maintenance and upkeep of naval assets, leading to degradation and high cost. Timely and accurate detection of corrosion and other defects are crucial to ensure operational efficiency and system availability. Traditional methods for detection often involve manual inspections, which are time-consuming, prone to human error, and require inspectors with specialized training and expertise. 

There is a significant amount of research into automatic detection of corrosion via computer vision algorithms and semantic segmentation that we utilized to combat this problem. Our method makes use of hand-annotated high-resolution images of existing weathered structures (conex boxes) as inputs for a machine learning model to detect basic corrosion, as well as classify additional defects such as paint blistering and cracking. Due to the limited amount of data representing the extended classes, image augmentations were performed to expand our dataset. 

When classifying “stuff”, where the defects can be any shape or size, having an extremely high level of precision is necessary. To account for this, a UNet machine learning architecture, which consists of a contracting and symmetric expanding path, was chosen due to its capability of performing precise pixel-level image segmentation with fewer training data requirements. We have experimented with multiple variations of UNet and implemented additional modules like attention and deep supervision in order to increase the overall performance and generalization of the model.

Premium content

Communication-Efficient Federated Learning for Multimodal Automatic Target Recognition

Next from NAML 2024

Computing Motion from Events-Based Cameras for UAV Surveillance

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES