United States

Truly autonomous operation of unmanned underwater vehicles (UUVs) is of critical importance as the Navy seeks to implement its 30-year Distributed Maritime Operations strategy. During the course of a mission, unmanned systems face the need to make decisions that range from when, where, and how to position themselves and operate their mission systems in the face of uncertain, unknown, and/or changing environmental conditions. Many in-service systems, however, lack a well-established means of incorporating in-situ observations of environmental variables, such as local or global currents, into their path planning, guidance, and/or control decisions. As such, there is a need for technologies that can autonomously make environmentally aware decisions for the UUV in order to improve the probability of mission completion.

We describe here a deep reinforcement learning (RL) framework that partially addresses these issues by generating energy-minimizing guidance systems for UUVs. Specifically, the proposed framework outputs trained policy networks (agents) capable of mapping local current velocity measurements to heading, depth, and speed commands that yield nearly-optimal, minimum-energy transits through spatially varying current fields. Numerical simulations in two and three spatial dimensions demonstrate that the energy cost of transits executed under the guidance of a trained agent are better than dead-reckoning, a common in-service guidance scheme, and approach known optimal performance for a set of realistic test scenarios that were not included in the training process.


NAML 2024

A Novel Reinforcement Learning Framework for Improved Guidance and Power Management of Unmanned Underwater Vehicles

proximal policy optimization

guidance systems

unmanned underwater vehicles

optimal control

neural networks

reinforcement learning

technical paper

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


"The Intelligence Advanced Research Projects Activity (IARPA) SCISRS program addresses the critical need for detecting and characterizing RF anomalies indicative of secure data compromise in complex RF environments. The program focuses on identifying low probability of intercept (LPI) signals, altered or mimicked signals, and unintentional emissions. Expedition Technology (EXP) supports this initiative with our solution Sensing Hardware Emissions via Artificially Intelligent RF Reasoning Systems (SHEARRS). SHEARRS integrates traditional signal processing and machine learning in an end-to-end system to enhance RF security. This work builds off the foundational ML capabilities developed under the DARPA RFMLS program and ongoing transition sponsored by Naval Information Warfare Center (NIWC) Pacific. 

SHEARRS is designed to be scalable, computationally efficient, and adaptable to diverse radio hardware. The system operates by applying advanced algorithms to detect subtle, anomalous signals amidst a noisy background across a large frequency range of interest (100 MHz to 6 GHz) using commodity COTS software defined radios (SDR). It has been tested and validated on government RF testbeds, demonstrating its ability to identify LPI signals, recognize altered transmissions, and detect unintentional emissions from electronic devices. 

Our proposed presentation for the Eighth Annual Workshop on Naval Applications of Machine Learning will detail the technical architecture of SHEARRS, its operational workflow, and the methodologies employed for anomaly detection and characterization. This will include discussions of our modulation recognition neural network capabilities for narrow and wideband scenes, statistical signal processing for efficient detection of spectral anomalies using a fast sweeping 6 GHz spectrum monitor, and our end-to-end system pipeline for combining the outputs of multiple detection engines. We will discuss the results from the IARPA testbed evaluations, emphasizing the system's performance metrics, scalability, and adaptability. 

The integration of SHEARRS with the SCISRS program represents a significant advancement in securing data in complex RF environments. Our solution not only detects but also characterizes a wide spectrum of RF anomalies, providing a robust tool for national security missions operating in diverse and uncontrollable environments. The presentation aims to inform on the technical capabilities, applications, and future developments of SHEARRS within the context of the SCISRS program. "

Sensing Hardware Emissions and Anomalies via Artificially Intelligent RF Reasoning Systems (SHEAARRS)

NPS team developed a vessel tracking system that uses AIS, satelite imagery, and electromagnetics to monitor traffic in waterways, ports, and coastal areas. Working with open-source and vendor data, a common operating picture was provided that prevents AIS spoofing and also allows classification based on AIS and vessel electromagnetics.

Common Operating Picture of Maritime Vessels using AIS, Satellite Imagery and Electro Magnetics

A novel approach for pulse-Doppler radar moving target detection in range-Doppler maps is proposed. The proposed ensemble architecture is highly modular, generalizable, and extensible – allowing for customization and applicability to a wide variety of radar systems. It contains a suite of statistical, convolutional, and graph-based machine learning models which seamlessly combine to detect targets in real-world range-Doppler maps. The model has been shown to be adjustable and tunable to work on multiple real-world datasets, including those containing air-to-air targets and air-to-ground targets, and has a high probability of detection and low false alarm rate in all cases.

The RAnge-Doppler Detection (RADD) Model

This paper presents an eXplainable Deep Reinforcement Learning (XDRL) based strategy for solving the proposed problem of fleet-level aircraft maintenance scheduling (AMS) optimization. The XDRL-AMS considers various factors such as the aircraft’s initial status, mission requirements, maintenance resource capacity, and operational constraints to create a maintenance schedule for a specified period. The schedule aims to balance both mission readiness and cost reduction. We developed an RL environment, called AMS-Gym, using the OpenAI Gym toolkit specifically designed for this problem. AMS-Gym is highly flexible, allowing for easy extension to more complex scenarios and incorporating additional explanatory capabilities. The explainable RL capability was achieved by utilizing a decomposed reward Deep Q-Network (drDQN) algorithm. In the context of the AMS scenario, the drDQN consists of two parts: (i) a DQN that aims to maximize the mission accomplishment objective, and (ii) a DQN that aims to minimize the maintenance cost objective. As a result, the proposed drDQN strategy can generate real-time aircraft maintenance decisions, explain why those decisions were selected, and present the tradeoffs between the chosen action and non-selected alternatives. Experiment results show that the proposed drDQN performs well, providing an approximate solution to the vanilla DQN with a simpler structure while offering the ability to explain its decisions. In addition, a web-based prototype with an intuitive textual and visual user interface was developed to demonstrate the feasibility of the drDQN approach.

Scheduling Condition-based Maintenance: An Explainable Deep Reinforcement Learning Approach via Reward Decomposition

The Operational Surface Risk Indicators (OSRI) project, led by Commander, Naval Surface Forces (CNSF), aggregates disparate enterprise-wide data sources into DoN Jupiter’s common data environment. OSRI deploys machine learning models to identify significant risk factors and inform better decision-making. Leveraging machine learning and data visualization tools, OSRI seeks to enhance operational safety and readiness across the Surface Force. OSRI builds predictive models to estimate the likelihood of a mishap based on a ship’s state. This capability can lead to a better understanding of important factors that contribute to these events and can proactively arm decision-makers with the empirical data and knowledge to intervene and improve operational safety. 

OSRI measures risk using Total Event Cost (TEC) as reported in RMI. TEC indicates the severity of events and leads to a regression model. Since TEC is highly skewed, a few costly outliers could dominate the model. Therefore, OSRI defines risk as the base 10 logarithm of TEC. This measure of risk, along with a filter for limiting events to those that are operational, even beyond class A or B mishaps, allows us to expand our set of events so that we are not modeling only ‘black swan’ events, which is inherently difficult. 

OSRI makes monthly risk predictions based on data from DMDC (manning), ASCAS (organizational climate), TORIS (training), ADE (fit/fill), and NVR (ship characteristics). After assessing a variety of regression models, Random Forests was found to be the best under the preferred measure of model performance, Normalized Discounted Cumulative Gain (NDCG), which is a rank-based metric that encapsulates how decision-makers use the output of the model. OSRI’s dashboard displays model output and ship-specific indicators, enhancing leaders’ intuition about risk by providing unique data-driven insights.

Another novel aspect of the OSRI model is how safety under-reporting is handled. Many ships do not submit the required two safety-related lessons learned critiques per year. This carries over to RMI reporting, where some ships have not reported any mishaps for years, which begs the question are they safe or just not reporting? An autoregressive model would be invalid for ships that under-report and have an incomplete history of events, therefore OSRI bases each ship’s predicted risk on historical data from all other ships in the fleet excluding the ship under study. This method also addresses feedback that leadership does not want ships to be penalized in the model for reporting safety-related incidents. 

The OSRI team is paving the way with cutting-edge data aggregation and machine learning under the guidance and expertise of CNSF. Our risk modeling effort seeks to make a measurable impact for decision-makers across the Navy, with the goal of improving operational safety, enhancing the Surface Force’s operational posture, and promoting the health and well-being of Fleet Sailors.


Operational Safety Risk Indicators

Predicting the remaining useful life (RUL) of machinery, such as
NASA's Turbofan engines, is crucial for maintenance and reliability.
Traditionally, this task has relied on manual machine learning (ML)
approaches that require domain expertise and extensive feature
engineering. However, the emergence of AutoML (Automated Machine
Learning) has promised to simplify this process, making it accessible
to non-experts by automating tasks like model selection,
hyper-parameter tuning, and feature engineering. This study conducts a
comprehensive comparative analysis between traditional manual ML and
AutoML approaches for RUL prediction in Turbofan engines. The
Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset
is used to evaluate the performance of both methods. Various factors
such as predictive accuracy, model interpretability, and ease of use
are considered. The results of this study provide valuable insights
into the strengths and weaknesses of manual ML and AutoML in the
context of RUL prediction. This research contributes to a deeper
understanding of the applicability of AutoML in critical domains like
aerospace engineering and highlights when each approach may be
preferable. Ultimately, the study indicates that the manual LSTM model
has an accuracy advantage over the automated machine model, by
approximately 26%. However, the total time spent on manual models is
hundreds of times greater than AutoGluon.

A Comparative Study of Manual Machine Learning and AutoML Approaches for Predicting Remaining Useful Life in NASA's Turbofan Engine Degradation

Penetration testing, a proactive method for identifying vulnerabilities in cyberspace, has traditionally relied on human expertise, necessitating a deep understanding of cyber environments and the ability to think creatively in the face of complex security challenges. However, rapid advancements in machine learning have opened up new possibilities for automating many tasks. Our ongoing work explores the application of Large Language Models (LLMs) to drive an automated agent capable of conducting penetration tests, bridging the gap between static tools and dynamic human intuition in cybersecurity.
                                                                       
LLMs, with their impressive natural language understanding and generation capabilities, have demonstrated potential in various domains, ranging from text completion to complex problem-solving tasks. However, when it comes to penetration testing, these models face unique challenges. Penetration tests often involve a series of interconnected steps, requiring a deep understanding of the subject matter and unique network environment, strategic planning, and the ability to adapt to dynamic situations. LLMs, in their current state, struggle with tasks that necessitate exploration, strategic lookahead, and the retention of information over extended interactions.
                                                                       
Addressing these limitations, our work implements a novel approach, leveraging a hierarchical architecture with elements inspired by the "Tree of Thoughts" framework. This architecture breaks down complex penetration testing tasks into manageable, sequential steps, enabling the LLM to maintain focus, retain critical information, and make strategic decisions autonomously in cyberspace. The system is structured in a chain of command, consisting of the Strategic Planner, Operational Planner, Modules, and Submodules, each playing a vital role in guiding the agent through the penetration testing process. This division of responsibility allows for both greater levels of control from and understandable reporting to humans in the loop.
                                                                       
These techniques have enabled success in entry level challenges. This ongoing work will continue to explore the degree to which an LLM-enabled agent can solve HackTheBox pen testing challenges. This offers key insights into the roles that generative AI can and will play in the cyber domain. We responsibly investigate the potential dual use of this rapidly advancing technology given implications spanning across defense, industry, and society.

LLM-Directed Agent Models in Cyberspace

Generative AI has reached a cultural moment, creating a set of questions about how thought work will change in the future. In this talk we present a set of best practices for building and deploying Generative AI capabilities to existing workflows for maximal effect. Several considerations come into play, but a framework of common steps emerges for organizations to take full advantage, increasing efficiency while maintaining trust in the results. 

The considerations include process mapping, method selection, model selection, fine-tuning, workflow integration, iterative development, and trust guardrails. We will show how these considerations come together to allow for the effective application of these technologies. 

Applications to Navy use cases are myriad. Anywhere a thought worker uses technology and research as part of a work process can benefit from the thoughtful application of Generative AI. It has significant implications for knowledge management, document summary, and report generation among others. By teaming the machine with the thought worker, many tedious or time consuming tasks can be improved while maintaining quality and efficacy. This presentation will conclude with an example resulting from the development of a workflow enhancing Generative technology.

Building effective workflow accelerators using Generative AI

NIWC PAC Code 584 in Philadelphia has achieved success in developing tools in the NLP fields of predictive AI and generative AI. For generative AI, we developed a NIWC PAC custom GPT Chatbot that is now answering questions about documents provided by our Public Affairs Officer. The NISE-funded chatbot is a collaborative effort between Code 584 and the PAO’s office and will provide Distro A outputs in the first version, and then a customized chatbot for CUI scientific and engineering material in the second version. These will be the first GPT chatbots deployed as digital assistants for every Center employee and will serve as prototypes for future work, including efforts for the NIWC PAC vision of Making Kennedy Real.

The GPT chatbot is powered by an open-source large language model and uses machine-learning algorithms to analyze and understand the context of a conversation and then retrieve relevant information from NIWC PAC’s Technical Library, a digital archive of over 1,400 technical reports, technical documents, journal articles, and conference papers. The chatbot introduces enhanced problem-solving and accelerated innovation within the research and development process. These new solutions offer an opportunity to improve workflow while also disrupting old ways of doing business and generating new ways to think and act creatively, allowing us all to generate novel ideas and uncover new insights, accelerating the pace of innovation within our organization.

For predictive AI, Google’s open-source Bidirectional Encoder Representations from Transformers (BERT) model remains a workhorse as a foundational NLP model for the division. BERT has skyrocketed since Code 584 began using it in 2020, and over 150 research studies have improved on the BERT large language model. 

In the first predictive AI success story, the BERT-based Technology Transition Classifier (TTC) continues to serve our NIWC PAC Business Portfolio Managers in quarterly data calls. The TTC is an internal application that uses NLP to find technology transitions within NIWC PAC weekly highlight submissions. The TTC is a binary classifier, meaning it predicts if a weekly highlight either involves a technology transition or does not involve a technology transition. The TTC provides classification and a confidence level for each prediction. Each quarter, the application provides an analysis on all NIWC PAC Hub highlights to the BPMs with 96% accuracy.

Our second success in predictive AI is the Controlled Unclassified Information Categorization Automation Tool. The CUICAT is an NLP application that automatically detects and categorizes CUI in Microsoft Office documents and .pdfs. The tool can identify and label five categories of CUI: privacy, health, procurement, legal, and non-CUI; and was a top finisher in NAVSEA’s nationwide Challenge.gov competition. The CUICAT is now funded as a pilot project where we are expanding the CUI categories and targeting deployment on Microsoft Azure.

Lastly, NIWC PAC Code 584 recognizes that an often-overlooked aspect in AI development is how to get prototypes to the fleet. The Navy's adoption of AI should have a well-defined pathway to ensure the sustainment. To that end, Code 584 is working with the PEO for Digital and Enterprise Services who has a long history of driving IT transformation and is ideal to manage acquisition and longevity of an AI program. Last year, the PEO established a “Pilot-to-Production” (P2P) process to introduce AI tools to the Navy under a portfolio of AI pilot projects. The P2P acquisition pipeline has a three-year horizon from concept to IOC, but pilots can launch from any one of the three horizons depending on technical maturity and other factors. The process uses industry standard DevSecOps principles to instill rigor at every phase of the lifecycle. We believe PEO Digital’s P2P roadmap is well-suited for seeding prototypes and to sustain AI capabilities for the Fleet well into the future.

NIWC PAC Code 584 Machine Learning & Natural Language Processing: Review of Accomplishments & Future Direction

COSMOS is modern web application that enables automation/orchestration between COTS platforms that provides customers an ability to self-onboard, build and secure GovCloud Amazon Web Services (AWS) Impact Level (IL)-4/5 workloads within minutes. COSMOS provides an environment to achieve flexible and agile speed to capability in the face of rapidly evolving threats and missions, enabling mission owners to make the superior decisions necessary to win in battle.

COSMOS provides 200,000% faster onboarding than any other GovCloud environment; a Cloud Native Access Point architecture that provides accessibility from anywhere, any device; high availability standards; inheritable security through federated DoD M365 tenet identities; continuous monitoring for cloud resources; and on-demand Risk Management Framework (RMF) Authority to Operate (ATO) artifact generation to promote a continuous state of cybersecurity awareness and readiness.

Premium content

A Novel Reinforcement Learning Framework for Improved Guidance and Power Management of Unmanned Underwater Vehicles

Downloads

Next from NAML 2024

Sensing Hardware Emissions and Anomalies via Artificially Intelligent RF Reasoning Systems (SHEAARRS)

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES