United States

This study investigates the ability of deep learning methods to identify salience of radio frequency (RF) data. We primarily study this through sensitivity and are Unfortunately, RF data exists in the presence of noise and nuisance signals. We define noise as any perturbation on the data which obscures signal, while nuisance is a signal within the data we are not interested in.

NAML 2023

Exploring Salience Metrics of Radio Frequency Data

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Current and future methods depend on more data and computational power to ensure mine counter measure(MCM) maritime dominance. Using Synthetic Aperture Sonar(SAS) Side-Scan data, we show how we are tackling the challenge of limited training data.

ATR for UUV Side-Scan and Synthetic Aperture Sonars

In this work, we leverage recent understanding to analyze the utility and practicality of cross-experiment transfer learning. We implement a practical approach where a pre-trained global model is applied on the fly to a new subject in an entirely new experiment using different EEG hardware. We are interested in quantifying the extent to which existing data from similar experiments may be useful to learn features of interest. Ultimately, this work aims to reduce the burden of required data per subject for learning common MI features to help design a system that is practically feasible and convenient. 

Cross-Experiment Transfer Learning for Real-time Brain Computer Interfaces 

Case-based reasoning (CBR) systems provide an interpretable reasoning process, supporting user trust; their decisions can be justified and explained in terms of the retrieved prior cases on which the decisions were based. A key factor that influences the trustworthiness of CBR systems is the quality of their retrieval, which depends on the features aggregated in cases to enable case comparison. Developing a feature vocabulary by traditional knowledge acquisition can be expensive, but the success of machine learning with deep neural networks suggests the potential to harness network learning to automatically generate features/indices for CBR retrieval. Initial studies have demonstrated that this approach can be effective, but related topics such as how such features may augment incomplete knowledge-engineered feature sets, how the highest-quality features may be extracted, and which network models may be most conducive to CBR feature extraction are poorly understood. This poster describes work addressing these questions, in which a CBR classifier is used as the primary system to support trust in model decisions through interpretability and a deep learning model is used to generate high-quality features to increase the accuracy of the CBR system.

On Extracting Network Features for Case-Based Reasoning

Artificial Intelligence (AI) decision aids are becoming increasingly utilized across domains, including for manpower and personnel decisions, because they offer effective and efficient mechanisms for collaboratively supporting human decision-makers. However, despite the computational, data-driven benefits of leveraging AI agents, prior studies highlight that bias may introduce glaring anomalies that may greatly hinder the decision-making process of such collaborative endeavors. Further, as AI systems become commonplace, humans often become over-reliant and complacent about AI decisions, allowing such biases to go undetected and unmitigated. Despite abundant data-driven and algorithmic bias mitigation solutions, several prior investigations have indicated that these technical solutions alone are insufficient and demonstrate the need for human intervention. While humans play a critical role in bias detection, research on affordances to boost bias detection and mitigation in human-AI collaborative contexts remains underexplored. To address this gap, this presentation discusses an investigation of the potential of three different cognitive forcing functions (CFFs), 30-second wait, update, and on-demand, which were chosen for the demonstrated power of CFFs in the medical context to boost analytical and deliberative thinking. We apply these to explore the potential of CFFs to enhance user interaction and interpretation of AI to support the bias detection capabilities of human users. Given the widespread use of AI tools to screen applicants for hiring and relevance to manpower decisions in Naval settings, we frame our investigation within a human resources context. Using a within-subjects factorial survey, we asked participants (N=152) to take on the role of a human resources assistant and work with an AI decision aid to evaluate candidates. The results indicate that these CFFs significantly influenced performance measures, including assessing the fairness of the AI’s decisions. We also found that these CFFs all had significant effects on increasing physical and perceptual interaction and interpretation measures aimed at facilitating greater bias detection capabilities in the process. While similar outcomes were observed across all the CFFs, we observed a greater impact on recognizing bias and enhanced trust associated with the update CFF. These empirical findings expand existing scholarship by playing a critical role in understanding how affordances that enhance bias detection and mitigation can be envisioned to boost the overall efficacy and efficiency of human-AI collaborative ventures, which can have widespread application across multiple different contexts in the Navy that leverage AI-driven decision-making. Such visions can be crucial for conceptualizing intentional designs to increase interaction and interpretation of AI decisions that help counter AI complacency, thereby boosting bias mitigation capabilities in human-AI collaboration.

Promoting Human Interpretation and Interaction to Mitigate Bias in Artificial Intelligence Assisted Decision Aids Through Intentional Interface Designs

Many different complex physics-based algorithms can be used to predict the quality of channels in radio frequency (RF) communications with varying levels of fidelity across many parameters. This project aims to use a generative adversarial network to accomplish the same prediction task. RadioML and GNURadio were used to generate datasets composed of short signal sections of IQ data for various modulations and SNRs. Two different conditional GANs, pix2pix and Wasserstein, were trained on paired and unpaired RF datasets containing several modulation schemes and channel effects in order to investigate the different GAN’s potential to learn the complex channel effects that can occur during RF transmission. After training on the synthetic data, the output of the conditional GANs qualitatively resembled the training data. As in the image domain, metrics for GAN outputs are difficult to quantify, and we are working on appropriate measures to rank GAN performance. This is a first step toward training a GAN that can qualitatively and quantitatively reproduce the transformation between the transmitted RF data and the received RF signal. An HF dataset labelled with the HF channel metadata (e.g. time of day, sun spots, etc.), the conditional parameters for the GAN, is being acquired for future training. While the HF data collection effort continues we are addressing issues of short, discrete IQ samples by investigating different GAN architectures that can enable the generation of longer sequences.

HF Observability GAN (HOGAN)

Naval surface vessels, both manned and unmanned, are equipped with multi-
modal sensor systems that must process and make sense of complex operating
environments for optimal mission performance and survivability. Autonomous
sensor resource management increases the effectiveness of a group of sensors and
the efficiency of each individual sensor. In this research we focus on optimizing
autonomous sensor resource management for an individual radar system. Radar
resource management is the process by which a radar system will allocate its
resources between various tasks, viz., scanning in order to detect previously un-
detected targets, as well as tracking previously detected targets. A sub-problem
of the resource management problem is radar time allocation, where the focus
is on determining how much time is to be spent on scanning for new targets,
and how much time is to be spent on tracking each target that has already been
detected. Because of the sheer volume of parameters in a radar system, tra-
ditional methods of solving these problems often involve heuristic algorithms,
or they utilize simple techniques that may achieve unsatisfactory performance.
However, as recent works have shown, advances in reinforcement learning (RL)
have opened up solutions in this space that can create significant improvements
in performance. Our ongoing joint industry-academia collaboration between
Syracuse University and Saab, Inc., which is a supplier of U.S. Navy and U.S.
Marine Corps radar systems, demonstrates the capabilities of RL to address the
time allocation problem in both a multi-target tracking setting and a scan while
tracking instance.
In multi-target tracking, we must decide on how to allocate the dwell times
of a particular radar such that tracking performance is maximized across a set
of targets in varying modes of flight operation. In the “scan while tracking”
instance, the problem is to determine the correct time allotment such that a
radar system is capable of discovering all targets in a scene while continuing to
accurately track all detected targets. We explore different RL algorithms along
with carefully designed cost functions to train agents capable of achieving state-of-the-art results on these problems. Further, we present results to demonstrate
the potential of these agents to generalize to other related problems in this
space. Simulation software has been implemented, and simulations are carried
out in near-realistic simulation environments (e.g., with targets whose velocities
change in magnitude and direction), demonstrating the potential for real-world
applications of these RL agents. We compare the RL approach with static and
heuristic time allocation strategies, especially in the context of the possibility
of sudden changes in the environment, with several new targets emerging unex-
pectedly during the radar surveillance operation, necessitating dynamic changes
to the relative magnitudes of time slots allocated to scanning and tracking var-
ious targets.

Reinforcement Learning based Resource Allocation for Radar Scanning and Multi-Target Tracking

Internet has become a ubiquitous influence in today’s age; however, internet activities have become vulnerable and detrimental due to malicious actions if not secured properly. For example, large scale data breaches such as those indicted at Yahoo and Equifax, essentially exploited the weaknesses of the traditional username/password authentication mechanism. Therefore, it has become imperative for us to find additional ways to protect our data, which also needs to be low-cost and unobtrusive. Existing authentication mechanisms can be categorized into what you know (knowledge-based), what you have (possession of physical things) and what you are (biometrics). Biometrics-based authentication can be further categorized into physiological and behavioral biometrics. Physiological biometrics are based on physiological characteristics of a person such as fingerprints, iris, facial recognition and palmprints, which can cumbersome due to the cost and usability. On the other hand, behavioral biometrics use the behavior of a person interacting with a device such as mouse dynamics (i.e., different mouse movements on a graphical user interface (GUI)), widget interactions (i.e., mouse cursor hovering over different icons or buttons on a GUI), keystroke dynamics, swipe, motion, walking or gait, all of which are inexpensive and unobtrusive techniques that can be leveraged to strengthen existing authentication without interfering with day-to-day computer operations. The experimental work of this proposal includes two novel contributions: 1) a survey of the state-of-the-art literature on mouse dynamics and widget interactions and 2) designing and validating fusion of mouse dynamics and widget interactions’ modalities on a realistic and popular website (i.e., PayPal and Facebook) utilizing our novel datasets. We performed literature survey on mouse dynamics with widget interactions along the following dimensions: tasks and experimental settings for data collection, taxonomy of raw attributes, feature extractions and mathematical definitions, publicly available datasets, algorithms (statistical, machine learning, and deep learning), fusion and performance, and limitations. Secondly, we experimented with trajectory driven window mapping to fuse mouse dynamics and widget interactions modalities using a total of 10 users for PayPal with an average of 12 minutes of data per user and a total of 15 users for Facebook with an average of 2 hours of data per user. Our mean EER rates with a score-level fusion of mouse dynamics and widget interactions are 7.64\% (SVM-rbf) and 3.25\% (GBM), for PayPal, and 5.49\% (SVM-rbf) and 2.54\% (GBM), for Facebook. We further improve the EER rates by combining decision scores from multiple consecutive trajectories, which yields a 0\% mean EER after 11 decision scores across all the users for both PayPal and Facebook

Multi-Agent Cooperative Game with a Learned Message Passing Interface

There is growing interest in the use of machine learning (ML) techniques to extract and classify signals of interest from passive sonar data. As with all machine learning applications a large body of data is required to train the ML model. Whilst a large body of open source acoustic data is available from a variety of sources and for different applications this data is often unlabelled, or labelled only for the task at hand. Marine mammal data recordings for example may be labelled with a species of interest but other sounds, such as vessels and other marine life, may not be labelled. For an effective classification model to be developed all sources in the recording must be labelled.
One solution to this problem is to manually label each recording, however the time required to create sufficient labels using this method can be prohibitive. This work follows a different approach, developing an algorithm to auto label large datasets using semi-supervised and active learning approaches. These make use of a small labelled dataset and minimal expert user interaction to create a large pseudo-labelled training dataset. Testing of pseudo-labelling has shown that it is possible to use this technique to create large training sets, however, Accuracy remains low. The introduction of active learning increases this Accuracy by guiding the model training.



Automating the Labelling of Underwater Audio Recordings

"With the rapid progression of technology, it becomes difficult for individuals to navigate the amount of data and systems at their disposal. To lessen the burden, technology can be leveraged to create an ambient intelligent environment where humans can naturally utilize systems with little prior training.  In this work, we demonstrate various technical applications that contribute to the effort of creating an environment that supports users by providing systems that enhance their performance. The features we developed allow the user to interact and control systems via hand gestures, action recognition, and verbal cues.

The capabilities have been implemented using various machine learning models and frameworks. For example, hand gesturing has been implemented with the MediaPipe posing framework, which allows for key point estimation on live camera feed. These key points allow us to identify the position of an individual’s hand in a relative space and can use this data to create interactive actions between the user and the system. Action recognition data is generated via an action classification model. This model estimates confidence scores for relevant action classification labels based on the real-time camera footage it receives as input. This data provides our system with more context about the individuals in our command centers, allowing us to tailor system interactions to specific actions. Lastly the verbal cues are implemented using a live transcript utilizing an automatic speech recognition model to transcribe live audio. This is fed into a dialogue manager which then determines the intent of the speaker using an intent classification model, which will determine what actions the dialogue manager should take or how the conversation should continue. Our goal is to expand our Human Computer Interaction (HCI) capabilities in order to achieve a fully ambient intelligent environment, which would allow its users to easily interact with the system and provide a more effective interactive experience."

Developing Interactive Ambient Intelligent Environments

This talk highlights the strategic vision for so called responsible autonomous AI, emphasizing a system of systems approach to integrating our perception systems, synthetic training systems, cognitive systems prospection, which is very central to cognition and critical for evidence based decision, and of course teaming and command and control based on assessments of the operational environment.

Next from NAML 2023

ATR for UUV Side-Scan and Synthetic Aperture Sonars

Similar lecture

Design, Demonstration, and Proof of Concept Using the Explainable Reinforcement Learning (XAI) of Soar for Combat Identification (CID)

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES