United States

Naval aviation assets on aircraft carriers such as aircraft, catapults and arresting gear are tracked through information systems that provide situational awareness to sailors. The current concept of operations requires highly skilled workstation operators to manage everything from aircraft status to flight schedules to keep information up to date. Operators receive information over radio communication channels from flight deck personnel. This process requires the operator to be dedicated to listening to updates and manually entering what they hear into workstations. This project developed an automated speech recognition prototype that can automatically listen into the audio communication pipeline and add information into computer systems.  
A speech recognition pipeline was created to recognize the standard phrases spoken by aircraft carrier flight deck personnel. Outputs can be used to correctly identify the actions that need to be performed to update computer systems automatically. The pipeline includes speech to text Automatic Speech Recognition (ASR) models and speech enhancement followed by post processing and similarity matching techniques to increase accuracy. Recursion was implemented to re-run the pipeline with a larger speech model if the result does not match a standard flight deck phrase. The team made use of Whisper, a transformer-based model with various model sizes ranging from 39 million to 1.56 billion trained parameters. The pipeline was tested in a reverberation chamber using flight deck phrases under various noise levels and model sizes. 
The team is currently working on fine-tuning the model and creating synthetic data. Data generation used by the team include the ability to fabricate semi-realistic speech in 108 varying voices and dialects. The team has tested fine-tuning efforts on a hold-out percentage of the dataset but has plans to continue fine-tuning the whisper model on both real-world and synthetic data to note results on increased model performance. While high-decibel noisy recordings were obtained from a laboratory, the researchers hope to obtain more real-world data and ultimately test the system either in a simulated environment or the flight deck environment. 


NAML 2024

Aircraft Carrier Flight Deck Information Updates using Speech Recognition 

automated speech recognition

speech to text

natural language processing

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


we discuss the development and deployment of a robust autonomous system capable of performing various tasks in the maritime domain under unknown dynamic conditions. We investigate a data-driven approach based on modular design for ease of transfer of autonomy across different maritime surface vessel platforms. The data-driven approach alleviates issues related to a priori identification of mechanis- tic system models that may become deficient under evolving system behaviors and/or shifting, unanticipated, environmental influences. Our proposed learning-based platform comprises a deep Koopman system model and a change point detector that provides guidance on domain shifts prompting relearning under severe exogenous/endogenous perturbations. Motion control of the autonomous system is achieved via an optimal controller design. The Koopman linearized model naturally lends itself to a linear–quadratic regulator (LQR) control design. We propose the C3DKL control architecture “Cascade Control with Change Point Detection and Deep Koopman Learning”. The framework is verified in both simulation and real experiments.

C3DKL: Cascade Control with Change Point Detection and Deep Koopman Learning for Time-varying Systems

Federated learning (FL) is a distributed machine learning paradigm that enables clients to collaborate without accessing, infringing upon, or leaking user raw data by sharing only model parameters. Multimodal automatic target recognition (ATR) in FL aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this study, we propose multimodal fusion Federated learning with joint Modality and Client selection (mmFedMC), a new FL framework for multimodal ATR that can tackle the above-mentioned challenges. The joint selection algorithm incorporates two main components: (a) A modality selection criterion for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. This allows mmFedMC to flexibly balance performance against communication costs, depending on resource constraints and application requirements. (b) A client selection for the server is performed on the basis of the cross-entropy of prediction as a metric. This involves selecting a subset of clients for uploading and aggregating models, which further reduces communication overhead and optimizes the overall process. Experiments on a real-world dataset with six modalities demonstrate that mmFedMC achieves 98.19\% accuracy with a communication overhead of only 5MB per client. This represents an approximate 10\% increase in accuracy compared to random modality or client selection. Moreover, in comparison to other state-of-the-art methods that do not utilize a selection mechanism, mmFedMC enhances accuracy by a factor of 2x to 4x. Alternatively, to achieve comparable levels of accuracy, other methods require approximately 20x more communication overhead, i.e., about 100MB per client. Beyond the trade-off between performance and communication overhead, mmFedMC also offers an interpretable representation of the modality impact during the FL process. In the initial stages of FL, most modalities exhibit similar impacts. However, as the communication rounds progress, modalities with larger feature sets and complex models play a subordinate role in modality selection due to their higher communication costs. With the advancement of FL, more straightforward modalities that continue to convey substantial information emerge as primary contributors.

Communication-Efficient Federated Learning for Multimodal Automatic Target Recognition

Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

The objective of this line of research is to use Convolutional Neural
Networks (CNNs) to automatically identify corrosion in photographs.
This approach provides a reliable alternative to traditional
approaches. Corrosion, a widespread issue in several sectors,
necessitates prompt detection to avoid structural harm and guarantee
operational security. The use of deep learning, particularly
Convolutional Neural Networks (CNNs), offers an advanced method for
detecting corrosion. This is achieved by using the neural networks'
capability to identify complex patterns in picture data. This research
investigates the creation and assessment of a Convolutional Neural
Network (CNN) model that was trained using a dataset consisting of
photos depicting both corrosion and non-corrosion. The model's
effectiveness is assessed by analyzing key performance indicators such
as True Positive Rate, False Positive Rate, accuracy, recall, and area
under the ROC curve. The study illuminates the capabilities and
constraints of the CNN-based technique, providing useful insights for
businesses in need of effective and precise corrosion detection
solutions.


Corrosion Detection Using Deep Learning

We consider the problem faced by an intelligent attacker that wants to discover new attack paths to compromise a network in a cyber-security network intrusion scenario. This problem is challenging because the number of possible actions for the attacker could increase exponentially with the number of nodes in the network and the number of services running on those nodes. We formulate the attacker's problem as a reinforcement learning task and use a decision transformer network to tackle the complexity of the attacker’s action space. We have evaluated our techniques within a gamified network intrusion detection scenario called Cyber AI Gym Environment (CAGE). Our results show that the proposed approach can enable the attacker to learn successful attack strategies against two difficulty levels of defender strategies.

Decision Transformer for Learning Intelligent Attacker Strategies in a Network Intrusion Detection Game

The diverse range of naval vessels, ranging from aircraft carriers to unmanned surface vessels, and infrastructure, such as smart naval mines or buoys, presents a challenge for traditional machine learning (ML) techniques in naval defense. Traditional ML, which centralizes network data, struggles with the vast distances and unstable or insecure communication links typical in naval defense scenarios. 

Federated learning (FL) offers a solution by keeping the data and ML model training local to the naval vessels and only performing occasional ML model combinations, or "global aggregations" at a central server, or "mothership" for naval applications. FL does address some of the heterogeneity inherent to the naval setting, for example, it minimizes the communication burden during deployments through sporadic global ML model aggregations. "Statistical heterogeneity" here refers to data-related differences among network elements, such as various dataset sizes. "Systems heterogeneity" relates to physical differences between naval objects, such as varying reliability of communication links. 

However, existing literature in FL typically assumes the presence of fully labeled datasets across all naval elements, an unrealistic scenario in practical naval deployments where a diverse mixture of both \textbf{labeled} and \textbf{unlabeled} data is common. Current FL-based methodologies tend to produce a uniform global ML model for all naval elements, neglecting the information contained with the unlabeled data, potentially leading to suboptimal global ML models for diverse naval resources. 

To address this, we introduce a decentralized federated domain adaptation methodology which leverages both labeled and unlabeled data to identify the underlying statistical divergences among naval vessels. In doing so, we are able to optimize the transfer of ML models from naval elements with high quality labeled data (called sources) to naval resources with low quality or unlabeled data (called targets), thereby personalizing ML models to better fit the real-time needs of different naval vessels in a deployment. Our methodology, called Source-Target Determination and Link Formation (ST-LF), optimizes both (i) classification of naval resources into sources and targets and (ii) source-target link formation, in a manner that considers the trade-off between ML model accuracy and communication energy expenditures. To obtain a concrete objective function, we derive a measurable generalization error bound that accounts for estimates of source-target hypothesis deviations and divergences between data distributions. The resulting optimization problem is a mixed-integer signomial program, a class of NP-hard problems, for which we develop an algorithm based on successive convex approximations to solve it tractably. Subsequent numerical evaluations of ST-LF demonstrate that it improves classification accuracy and energy efficiency over state-of-the-art baselines.

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

Our task is to create an equivalent to “follow me” mode for ships at sea. Allowing a Sailor to launch a small quadcopter and forget about it while it follows and circles the ship of interest could be incredibly useful. Our particular application is autonomous inspection and battle damage assessment, say in the aftermath of a Naval casualty. 

We develop a computer vision pipeline that uses a drone’s camera feed to track the behavior of a ship. It uses camera frames to estimate the ship’s position and orientation to potentially correct the drone’s trajectory, ensuring that the ship is always in view, even in adverse conditions. Our proposed method can also be effective in broader applications in fields like Autonomous Driving and Robotics, where the main agent takes an action based on its surrounding environment. 

The first stage of our computer vision pipeline consists of calculating the orientation of the ship. Our method passes the images from the camera through a UNet image segmentation model to get the ship’s mask and boundaries. We then use an OpenCV algorithm to generate a rotated rectangle that minimizes its area while enclosing the entire mask of the ship. This rotated bounding box effectively represents the direction that the ship is facing based on its physical features, so we use its angle to estimate the ship’s orientation with respect to the drone. This technique is beneficial because it is less complex than 3D Pose Estimation models, allowing for faster real-time processing of image data. 

For our image segmentation model to perform effectively in various environmental conditions, it is crucial to have a variety of flight paths, elevation angles, and visibility conditions to prevent overfitting. Collecting real data with these criteria is an expensive and laborious process, so we have started with Unity-generated synthetic data with relevant variations as ground truth labels for training. Using this data to train our computer vision model resulted in highly accurate predicted masks from unseen flight test images.

Estimating Ship Position and Orientation using Deep Neural Networks and Synthetic Data

While Federated Learning (FL) on a distributed network is used extensively in commercial applications, the safety-critical nature of Naval applications presents a unique set of challenges. Specifically, the method must be robust against dynamic network conditions, where devices and communication channels within the network may experience unpredictable changes during both training and inference. Vertical Federated Learning (VFL) is a class of FL in which each client within a distributed network shares a common sample space but possesses only a subset of the overall features. This setup is particularly pertinent in practical scenarios such as sensor networks deployed for surveillance, where each sensor (client) gathers specific data (features) about the same environment (sample space). While VFL addresses several privacy concerns inherent in distributed learning, it typically operates under the assumption of ideal hardware and seamless communication capability. This assumption is a major limiting factor in the broader application of VFL, especially in the context of edge devices. Edge devices are inherently heterogeneous in their operational capabilities, and may frequently connect or disconnect from the network, leading to a dynamic and unpredictable network environment.

To address this gap, we study the test time performance of VFL under dynamic network conditions, which we call DN-VFL. We first formalize DN-VFL, including a message passing distributed inference algorithm, the corresponding risk, and a serverless setup. We develop a novel DN-VFL approach that synthesizes replication, gossiping, and selective feature omission that improves performance significantly over baselines. Furthermore, we implement new methods, propose metrics, and extensively analyze results based on a simulated sensor network. The results show that naively using VFL for DN-VFL is not the best approach. Rather, the proposed methods present a better alternative to handle changes in the network during
inference.


Fault-Tolerant Vertical Federated Learning on Dynamic Networks

SysML (Systems Modeling Language) is a widely-used formalism and international standard for designing and communicating complex system designs, and is especially relied upon in the development of safety-critical autonomous systems. SysML v2 is a major new version of SysML currently undergoing finalization, scheduled to be released mid-2024. Among many key enhancements, SysML v2 introduces a formal semantics for specifying the precise meaning of SysML models, which makes it possible to apply formal verification and automated reasoning to analyze safety and correctness properties of SysML designs automatically.

At Imandra, we are applying our Imandra Automated Reasoning System to the design and analysis of SysML v2 models. Key to this work is a formal semantics for SysML v2 models expressed in Imandra's logic, and corresponding verification automation for answering deep questions about the possible behaviors of SysML models. In addition to verification, we can then leverage Imandra's "proof-carrying prose" integration with LLMs to obtain conversational assistants (chatbots) which can rigorously answer complex questions about SysML v2 designs, backing every answer up with an independently verifiable logical audit trail produced by Imandra reasoning about the SysML v2 model.

During this talk, we will demonstrate how to use Automated Reasoning with SysML v2 models to:
- Mathematically prove properties of system correctness.
- Add a hallucination-free natural-language interface (chatbot) to empower more stakeholders to gain insights and leverage the SysML v2 model.

Imandra + SysML: Automated Reasoning for SysML (Systems Modeling Language) v2

The ever-growing size of deep learning models necessitates more extensive resources, making them unsuitable for tasks requiring lower latency and less computational power. This is particularly true for edge devices. Multiple compression techniques have been developed to bring the capabilities of these larger models to a smaller form factor, one of the most successful of which has been knowledge distillation. Knowledge distillation is employed to transfer knowledge from a large "teacher" model to a smaller "student" model. This technique facilitates learning a compressed representation of data, maintaining the performance benefits of large models while reducing computational and memory demands. It involves training the student model on both hard labels and soft labels provided by the teacher model, leading to better generalization and informed predictions.

Our work increases the effectiveness of knowledge distillation by augmenting the labels of both the teacher and student models with a vector quantized vector. Vector quantization is a mapping of high-dimensional data to a lower-dimensional codebook, effectively reducing data dimensionality. This technique has traditionally been applied to speech and image compression particularly in Variational Autoencoders (VQ-VAEs), and has proven effective at markedly increasing the effectiveness of student models to learn from teacher model representations.

The work presented here seeks to enhance the classification capabilities of the student model, carefully maintaining its compact size to balance efficiency and compactness. This approach improves retained knowledge distillation performance, while conferring model interpretability benefits. We show that while also increasing student model classification performance, the inclusion of the VQ layer also provides a measure for detecting classification quality and detect data which is out of distribution. This approach promises to increase the capabilities of edge deployed models both with respect to their mission performance as well as out of distribution detection.


Premium content

Aircraft Carrier Flight Deck Information Updates using Speech Recognition

Next from NAML 2024

C3DKL: Cascade Control with Change Point Detection and Deep Koopman Learning for Time-varying Systems

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES