United States

Unsupervised Degrader Ranking

An algorithm that can accurately rank Naval Aviation Support Equipment (SE) top degraders is a desired product to allocate resources effectively. This project utilized Factor Analysis of Mixed Data (FAMD), Isometric Mapping (ISOMAP), and Magnetic Properties and Correlation (URMC) to calculate the ranking of top SE degraders for two different Program/Project Manager (PMA) programs. The current methodology for ranking top degraders relies on a z-scores and Subject Matter Expert (SME) provided weights used by the existing degrader ranking products. The goal of the project would be to optimize the degrader ranking solution to use a method that is less subjective in nature and more flexible, which can be automated and applied across a variety of equipment.

Research to date in this project tackled the main issue by looking into two ways to calculate degrader ranking in an unsupervised fashion: FAMD and ISOMAP. Factor Analysis of Mixed Data reduces the dimensionality of both categorical and numerical data to that of N-dimensions specified by the user before finding a best way to fit all data points on a line in each dimension. ISOMAP, like FAMD, reduces the dimensionality of the data, but in a non-linear methodology, and is only used for numerical variables that are statistically significant. For both algorithms, data obtained from the current degrader products is reduced to a varying range of two to four dimensions while being assigned a degrader percentile label (Rank/Total) to aid in the algorithms’ potencies.

At this point in time, only a FAMD implementation has been fully tested, and with reduction to three components, can explain 84% of the incoming data from the current degrader product’s rankings. Additional exploration and research is needed to evaluate the URMC algorithm’s performance and comparison against both FAMD and ISOMAP models.

NAML 2023

Unsupervised Degrader Ranking

unsupervised algorithms

human machine teaming

machine learning

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2022

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Longitudinal data, which is widely used in many disciplines to study cause and effect, poses significant computational challenges to both modeling and analysis. Longitudinal data is composed of readings on the same variable collected over time and is often high-dimensional with correlated features. The combinatorial search approach for identifying the optimal features is unrealistic for most applications. The alternative approaches, such as heuristics, greedy searches, and regularization techniques, including LASSO, can result in models that suffer from both low accuracy and unclear feature attribution. In this presentation, we propose a binary transformation on the data before applying LASSO for feature learning. As demonstrated in the paper, the binary transformation enhances signal in the data, resulting in highly accurate feature attribution, including associated time lags. It avoids the typical shortcomings of the LASSO algorithm, including saturation of the feature space and arbitrary or inconsistent sparse feature selection. Both synthetic data and real-world data sets were used to demonstrate the value of the proposed transformation and in every case substantial improvements in feature learning were seen. In addition, the scalability of the solution is superior to that of the standard LASSO since an integrated high performance computing (HPC) solution would utilize integer math for most of the computational effort.


LASSO Logic Engine: Harnessing the Logic Parsing Capabilities of the LASSO Algorithm for Longitudinal Feature Learning 

GPS has proven to be a valuable, highly accurate Positioning, Navigation, and Timing (PNT) solution in many military applications. Its susceptibility to jamming and interference, however, has sparked a great deal of research in PNT alternatives to address vulnerabilities in GPS-denied environments. In this study, we utilize micro-electro-mechanical system (MEMS) IMU sensors mounted on different parts of the body to estimate movement speed and position. MEMS IMUs are small and light enough to be wearable on a person and can accurately estimate orientation, but fails to accurately estimate position. To address this, we create a system that uses GNSS, motion capture, and inertial data to train a machine learning speed estimator. The estimated speed is paired with a main IMU and an extended Kalman filter to estimate position. We will then compare our results with two other IMU-based navigation techniques: step counting and zero-velocity update algorithm (ZUPT). Step counting works well for normal walking or running, but fails for more complex movements like crawling, strafing, or walking backwards. ZUPT requires high end IMUs mounted on the foot, which can be very cumbersome. We believe our proposed method addresses these problems and performs more accurately.

Machine Learning Aided Gait Recognition for Inertial Navigation and Orientation (MARIO)

Carrier Flight Deck Information Updates using Speech Recognition 

Naval aviation assets such as aircraft, catapults and arresting gear are tracked through information systems that provide situational awareness to sailors on aircraft carriers. The current concept of operations requires highly skilled workstation operators to manage everything from aircraft status to flight schedules in order to keep information up to date. Information is given to the operator through personnel on the flight deck by using an audio communication pipeline. This requires the operator to be dedicated to listening to updates and manually entering what they hear into workstations. Many times, there are loud noises from jets and other equipment, which will only make the operator’s job more difficult. This project proposes a way to read into the audio communication pipeline and completely automate the process of adding information into computer systems using automated speech recognition models. 

The research in this use case accomplished acquiring a model that can understand the standardized phrases said by carrier personnel and correctly identify the written equivalent of what was said. There were three Transformer-based models in total that were acquired/trained to test accuracy on pre-generated flight-deck phrases: Speechbrain, Whisper, and NemoASR. While Speechbrain was fine-tuned on our flight deck phrases, NemoASR and Whisper were used in the base forms to test relative accuracy. In order to bolster their accuracies, research was also conducted into noise cancelling and speech enhancement pipelines that almost completely erase all background noise and make the spoken instructions as loud as possible, so the model has a low fail rate. These pipelines are possible by analyzing the Short-time Fourier Transforms of the audio in question and running them through a Long Short-Term Memory (LSTM) or Generative Adversarial Network (GAN) to erase unwanted frequencies from the audio. 

The goal of this project was to obtain a high enough accuracy in providing a written equivalent to an audio flight deck phrase. At this moment in time, the team has a word error rate of less than 5% utilizing the state-of-the-art model, Whisper. While the team is content with the accuracy of the model, they are considering conducting additional fine-tuning on both NemoASR and Whisper to get as close to a perfect model as possible while working on syncing the model with carrier flight deck information systems.

Carrier Flight Deck Information Updates Using Speech Recognition

Video sensors are ubiquitous in the realm of security and defense. Successive image data from those sensors can serve as an integral part of early-warning systems by drawing attention to suspicious anomalies while leveraging object detection, computer vision, and machine learning. Automating some of those detection and classification tasks aids in maintaining a consistent level of situational awareness in environments with ever-present threats. Specifically, the ability to detect small objects in video feeds would help people and systems to protect themselves against far away or small hazards. This work proposes a way to accentuate smaller features in video stills by using pixel subtraction of surrounding frames to augment the current input frame. Our method allows us to highlight changing pixels and extract motion information from video data that can be used to boost detection capabilities for objects of interest.

Frame-Differencing Object Detection Enhancement in Videos

As the digitization of modern life continues full steam ahead, criminals
and state-sponsored actors continue to target government systems. The impact of these attacks on the Navy range from the exposure of highly sensitive data to the disruption of services and the exploitation of back doors for future attacks. Modern cybersecurity teams have to defend on-premises and multi-cloud footprints. Effective threat detection and response are challenged by the cost and complexity of siloed tools that neither scale nor provide capabilities for advanced threat detection. These advanced threats are causing federal organizations and agencies to reevaluate their security posture and architectures being used to enable the
analysis of petabytes of data in real time and apply advanced analytics to reduce their cyber risk and operational costs.

Databricks - Defending Hybrid Cloud Environments with Advanced Threat Detection through Machine Learning

Automatic target recognition (ATR) is a critical task in a wide range of naval applications such as surface vessel protection and port security via radar, sonar, and optical sensor data processing. Data-driven methods, such as deep learning (DL), have been successfully applied to a variety of ATR tasks. Despite their strong performance, current state-of-the-art DL ATR applications are reliant upon aggregating data from several devices to a central server for training. This process can be computationally costly and presents a potential security risk if the data itself is sensitive. Moreover, current state-of-the-art DL ATR models also produce point estimates at test time with no associated measure of uncertainty. Uncertainty quantification (UQ) on model predictions is critical for accurately characterizing and assessing the reliability of model outputs. In this research, under the Office of Naval Research (ONR) Threat and Situational Understanding with Networked-Online Machine Intelligence (TSUNOMI) program, we develop a novel ATR framework, which is capable of both producing UQ measurements and eliminating the need for data centralization. This novel approach begins by considering an ensemble of ATR models that are jointly utilized to produce model predictions and UQ estimates. The prediction given by the ensemble is characterized by the distribution of outputs from the individual models rather than a point estimate produced from a single DL ATR model. Characteristics of the ensemble, such as its prediction interval and coverage probability are used to express model uncertainty. Using Federated Learning, the ensemble is trained in a distributed manner, where each vessel with contributing data trains a local ensemble using its own computational resources. Local model parameters are periodically transmitted to a centralized location for aggregation with the other ensembles. These parameters are generally much smaller-sized communications payloads than the raw data itself by several orders of magnitude and are used to perform the same shared learning task as centralized DL. We evaluate the efficacy of this ATR framework on a prototype task and share preliminary results.

An Uncertainty Quantification-Based Automatic Target Recognition Framework Over Distributed Surface Vessels

With the proliferation and rapidly improving endurance of next-generation unmanned undersea systems, long-term deployment of human-on-the-loop subsea and seabed resident fleets are fast-approaching. More specifically, wave-generated power and other renewable power sources, autonomous docking capabilities, and increased edge processing capabilities enable subsea residence of both static and mobile underwater systems. In this research, we evaluate human-on-the-loop adaptive mission planning algorithms for managing a fleet of unmanned understeer systems in simulation. These systems could provide persistent monitoring, maintenance, and repair of critical undersea infrastructure vulnerable to grey zone operations; such a system could estimate and track infrastructure health, including detection and sensor data acquisition of significant changes. Moreover, subsea resident systems could perform continuous mine survey of a particular (even mobile) area of responsibility and conduct environmental surveys over extended periods of deployment. It is often impractical or impossible for these systems to maintain consistent contact with human operators, necessitating novel approaches to mission planning that bridges the gap between constant human oversight and fully autonomous operation of the platforms. 
These persistent, autonomous systems are expected to adapt to increasingly complex environments and tasking, collaborate with other autonomous systems, and to respond appropriately to the preferences of human operators without requiring unnecessary oversight. The mission planning algorithms used by these systems must decompose high level objectives into discrete tasks that can be described in terms of their cost to complete, and their efficacy can be evaluated by the computational time required to create a solution, as well as the efficiency of the solution itself. To study these algorithms, a multi-agent problem is posed in simulation, wherein a simulated group of systems (agents) complete a number of (potentially mobile) tasks in the shortest time possible without colliding with other agents, or with dynamic obstacles drifting through the environment. By implementing a multi-stage approach to the problem decomposition, this work proposes an algorithm that outperforms state-of-the-art algorithms, such as the Consensus-Based Bundle Algorithm, with respect to both the computational time to create the plan and the efficiency of the plan itself. 


Human On the Loop Adaptive Mission Planning 

The need for abundant, diverse, and relevant training data continues to increase as deep machine learning based computer vision models grow in complexity. We have seen that tasks such as image classification and object detection benefit from large and varied datasets. Unfortunately, the cost of obtaining accurate labeled data at scale is prohibitive especially for more complex tasks such as object detection which require both categorical labels and accurate bounding box annotations. Active learning (AL) seeks to alleviate the need for brute-force data annotation by analyzing and automatically sampling a small but informative subset of data for labeling. With an effective sampling strategy, the trained model can achieve comparable performance to fully supervised methods at a fraction of the labeling cost. Sampling strategies generally fall into three categories: uncertainty-based, diversity-based, and hybrids of the two. For satellite imagery applications of computer vision, AL is an attractive potential solution: vast volumes of unlabeled satellite imagery exist, but annotation is expensive and tedious. For object detection in synthetic aperture radar (SAR) imagery, labeling is particularly challenging. Labeling SAR images is more costly than electro-optical (EO) images because the average person lacks the domain expertise to interpret and label SAR imagery. Because of this, crowdsourcing data annotation is not a viable option. Thus, we must intelligently allocate scarce labeling resources to the data most likely to improve the performance and robustness of the model being developed. The goal of AL is to automate and optimize this dataset curation process. We propose to evaluate active learning methods applied to maritime object detection in SAR imagery. Recent work has shown the efficacy of active learning for common EO image datasets like PASCAL VOC and COCO, but how do these methods perform in SAR imagery with unique image properties and much smaller objects? We consider two active learning methods: CALD selects images based on model uncertainty cues, and VAAL selects diverse images from the full dataset independent of the task or model being trained. With uncertainty-based methods the uncertainty cues are gathered from the model under training, so there is an inherent coupling between the model and the selected data. When a labeled dataset acquired for one model (the acquisition model) is used to train another model of a different architecture (a successor model), what are the model performance impacts? Given the costs of labeling datasets, it is desirable for the labeled set to transfer well across a variety of model architectures, since datasets are likely to be reused throughout model development. Our evaluation comprises two parts:
• We measure AL algorithm performance across EO and SAR image datasets and three deep object detection models. 
• We examine transferability of AL-acquired data for training different successor models.

Evaluating Active Learning Methods for Synthetic Aperture Radar Maritime Object Detection

Model cards organize and portray information
about a model’s training data, hyper parameters, and behavior,
such as predictive accuracy and bias for machine learning (ML)
algorithms or models. Consumers use these cards to determine
whether a model is well suited for a particular use case.
However, the current design of model cards does not include
resource utilization metrics, which are important when models
are being optimized for specific hardware. The main objective
of this study was to determine which set of hardware-centric
performance metrics provided the most value to users when
comparing among different algorithms for Internet of Things
(IoT) edge devices. The Department of Defense (DoD) can utilize
these metrics to select the best model for key mission tasks
within the Command and Control (C2) missions that require
AI/ML deployment on edge devices. Our study focused on
finding correlations between resource availability and machine
learning models’ computational footprint using three key metrics:
energy consumption, execution time, and memory utilization.
The data was gathered from simulated environments hosted
by Docker containers (i.e., lightweight, stand-alone, executable
package of software that includes everything needed to run an
application) with various memory sizes. Two different series of
experiments were performed. The first series of experiments
simulated environments with 256 MB to 8GB (steps of the
powers of 2) of available memory. For each memory size, we
deployed 50 different containers, calculated the metrics from
each container, and then recorded the average result of each
metric. Two ML models from Microsoft’s EdgeML library that
are designed to run more efficiently on edge devices were used:
Bonsai, a decision tree based model, and ProtoNN, a KNN based
model. Using Spearman’s correlation, we found that none of
the metrics showed promising correlations for the Bonsai model,
while the ProtoNN model demonstrated a correlation of 0.5 for its
computational time. In order to investigate the lack of adequate
results, we ran a second series of experiments in a similar fashion
but with smaller memory sizes from 250MB to 950MB (in steps of
50MB). Using Spearman’s correlation for the Bonsai model with
smaller memory sizes, we found that RAM energy consumption
showed the most promising correlation value of 0.75. While for
ProtoNN model with smaller memory sizes, both the RAM energy
consumption and total energy consumption showed promising
correlation values of 0.63 and 0.71 respectively.

Incorporating Hardware Metrics to Model Cards to Evaluate the Utilization of Machine Learning Models on IoT devices

From 2021 to 2022 participation at Carderock in the Congressionally mandated Defense Organizational Climate Survey (DEOCS) increased 76%!  Great for us, right?!  The survey is mostly multiple choice, but there are ten free response questions on the survey as well.  Five questions are the same for everyone in the Navy, and five are specific to each Command.  The 2,974 responses are 365 pages long when printed out.  Historically, a team from the Commander’s office must read through all of those responses and assign classes to each one.  The categories for each comment are: Requires Redaction, Sentiment (Positive, Negative, Neutral), Requires Follow-up Action by Command, and Risk Factor.  Then there are also 20 Sub-Factors and 27 Sub-Categories.  Sub-Factors include, Cohesion, Inclusion, Stress, Morale, etc.  Sub-Categories include Compliance, Diversity, Workload, Food, Gym, Parking, etc.  Therefore, the Commander’s Office initiated a project to evaluate the feasibility of classifying the DEOCS responses using natural language processing (NLP) models.  We successfully used the 2021 survey results and classifications to implement transfer learning on a variety of BERT, RoBERTa, and Distillbert pretrained NLP models using the Hugginface API.  We found that a model pretrained on 5 million Tweets performed best for sentiment classification.  Sentiment and all the binary classifications performed well (>90% accuracy).  We identified that if we corrected errors in only the smallest classes of each category that we can further improve the final accuracy with at 70% reduction in labor.  We found that we do not yet have enough training data to achieve acceptable acuracy for the Sub-Factor (19%) and Sub-Category (66%) multiple classification tasks. We would like to work with other Warfare centers to combine datasets for the five common DEOCS questions in order to improve performance with more data. 

Premium content

Next from NAML 2023

LASSO Logic Engine: Harnessing the Logic Parsing Capabilities of the LASSO Algorithm for Longitudinal Feature Learning

Similar lecture

Naval Aviation Support Equipment Impact on Aircraft Readiness

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES