United States

First-principles computer models or simulations are often used as forward models for training reinforcement learning (RL) agents because training on real-world systems is too expensive, slow, or otherwise impractical. Even the highest-fidelity models can, however, fail to account for all physical variables and these modeling errors can lead to poor performance when the simulation-trained agent is exposed to the real-world. A potential solution is to use data from the physical system to directly train a deep neural network as an approximation (surrogate) of the system’s underlying state-transition function. Given difficulties inherent in acquiring sufficient real-world training data, one advantageous approach is to first train a neural network surrogate using a first-principles model and then use transfer learning and a smaller set of real-world data to fine-tune the surrogate’s approximations. An RL agent can then be quickly and cheaply trained using the well-trained surrogate and can exhibit better real-world performance compared to an agent trained on a first-principles model alone. We propose a unique algorithm for generating training data appropriate for this type of surrogate, explain several important considerations in posing the learning objective function, describe the training techniques used to achieve outstanding performance for several surrogate models before and after transfer learning, and present results for an RL agent trained using a surrogate as the forward model. More specifically, we consider the problem of training RL agents to control an Unmanned Underwater Vehicle (UUV) in an environment characterized by current flow fields. We include discussion of how our novel data generation algorithm can be extended to work for a UUV navigating such an environment and explain how an egocentric state-space definition can improve the performance of an RL agent trained to control the UUV.

NAML 2023

Reinforcement Learning for UUV Control Using Stable and Accurate Neural Network Approximations of Motion Dynamics

technical paper

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


"Models which predict optical turbulence (as measured by Cn2) from macro-meteorological data are expensive to develop and often perform poorly when deployed in new environments.
Hybrid models combine these existing models with local measurement and a Gradient Boosted Decision Tree (GBDT) architecture to improve predictive power.
This architecture achieved approximately a 29% improvement over the baseline using only one day of measurement, and 68% using 180 days of observation.
"

Hybrid Optical Turbulence Models Using Machine Learning and Local Measurements

Internet has become a ubiquitous influence in today’s age; however, internet activities have become vulnerable and detrimental due to malicious actions if not secured properly. For example, large scale data breaches such as those indicted at Yahoo and Equifax, essentially exploited the weaknesses of the traditional username/password authentication mechanism. Therefore, it has become imperative for us to find additional ways to protect our data, which also needs to be low-cost and unobtrusive. Existing authentication mechanisms can be categorized into what you know (knowledge-based), what you have (possession of physical things) and what you are (biometrics). Biometrics-based authentication can be further categorized into physiological and behavioral biometrics. Physiological biometrics are based on physiological characteristics of a person such as fingerprints, iris, facial recognition and palmprints, which can cumbersome due to the cost and usability. On the other hand, behavioral biometrics use the behavior of a person interacting with a device such as mouse dynamics (i.e., different mouse movements on a graphical user interface (GUI)), widget interactions (i.e., mouse cursor hovering over different icons or buttons on a GUI), keystroke dynamics, swipe, motion, walking or gait, all of which are inexpensive and unobtrusive techniques that can be leveraged to strengthen existing authentication without interfering with day-to-day computer operations. The experimental work of this proposal includes two novel contributions: 1) a survey of the state-of-the-art literature on mouse dynamics and widget interactions and 2) designing and validating fusion of mouse dynamics and widget interactions’ modalities on a realistic and popular website (i.e., PayPal and Facebook) utilizing our novel datasets. We performed literature survey on mouse dynamics with widget interactions along the following dimensions: tasks and experimental settings for data collection, taxonomy of raw attributes, feature extractions and mathematical definitions, publicly available datasets, algorithms (statistical, machine learning, and deep learning), fusion and performance, and limitations. Secondly, we experimented with trajectory driven window mapping to fuse mouse dynamics and widget interactions modalities using a total of 10 users for PayPal with an average of 12 minutes of data per user and a total of 15 users for Facebook with an average of 2 hours of data per user. Our mean EER rates with a score-level fusion of mouse dynamics and widget interactions are 7.64\% (SVM-rbf) and 3.25\% (GBM), for PayPal, and 5.49\% (SVM-rbf) and 2.54\% (GBM), for Facebook. We further improve the EER rates by combining decision scores from multiple consecutive trajectories, which yields a 0\% mean EER after 11 decision scores across all the users for both PayPal and Facebook.

User Authentication by Fusion of Mouse Dynamics and Widget Interactions: Two Experiments with PayPal and Facebook

Estimating Sound Speed Profiles Using Vessel Source Signals of Opportunity

RippleGo is an AI-based voyage planner for the Inland Water Transportation industry.  In this talk, two AI-based technologies will be discussed and how each of the technologies improve safety and efficiency along the waterways.  One is a river level prediction model that forecasts river levels and the other is a travel time prediction model that forecasts time taken to go from a starting point to destination along the inland waterways.  Combining the 2 predictive technologies and other marine data sets, TRABUS developed a software-as-a-service application that enables operators in the inland waterways to plan and execute their voyages, safely and efficiently.  The talk will provide an overview of some of the core AI-driven capabilities of RippleGo.

RippleGo - First AI-Based Voyage Planning and Predictive Analytics Software for the US Inland Waterways

Coupled dynamical systems, especially those comprised of hysteretic elements, possess remarkable behavioral similarities to some aspects of neural activity in living systems. While some features of this similarity have been explored, the results have not yet been leveraged to improve machine learning techniques or artificial neural networks. In this work, a dynamical system of coupled overdamped bistable elements subject to a noise floor assists in information transfer. With the noise allowing the system elements to switch randomly between their stable states, the system response is quantified via a steady state probability distribution. A simple five element realization of this network is used to produce the analog equivalent of an XOR gate, an “XOR transform”, through the proper choice of coupling coefficients and controllable external biases. The demonstration of the XOR transform indicates the ability of the system to synthesize more general functions, a necessary prerequisite for machine learning and computing applications. Finally, a silicon implementation is proposed and simulated using a verified process model that could be fabricated to generate a working analog computing system.

Noise Driven Coupled Dynamic Systems–Towards Low Power Pattern Recognition and Machine Learning

Ensuring the success of future multi-domain operations (MDO) against a heavily militarized and techno-logically advanced adversary is a critical need for the Department of Defense. Raytheon Technologies (RTX) seeks to increase the productivity and impact of the warfighter in such peer-to-peer conflicts by reducing operator workload through sensor autonomy to speed the observe-orient-decide-act (OODA) loop and shorten the target kill chain. In this construct, the warfighter or analyst operates over-the-loop, focusing on complex mission decisions, and maintains decision-making control while supported by trusted AI/ML-enabled cognitive aids performing data intensive tasks within-the-loop that automatically process sensory data to produce decision-relevant information at the edge. This presentation introduces an autonomy vision called Cognitive Aids to Sensor Processing, Exploitation and Response, or CASPERTM and provides an overview of the Raytheon Applications of AI to Defense Systems or RAAIDS IR&D. For this effort, RTX has partnered with Virginia Polytechnic Institute (VT), Southern Methodist University (SMU) and the University of Texas San Antonio (UTSA). VT and SMU are working collaboratively to develop RL/ML-enabled algorithms that can autonomously control edge sensor operations to achieve greater automated detection, classification, and tracking of objects, optimizing both within an individual sensing system and across multiple systems and platforms that are sensing cooperatively. UTSA brings expertise in human-machine teaming and user experience design, which is not just a software development problem, but involves key disciplines such as cognitive science and psychology, human factors engineering, computer science, communication design, interaction design and usability engineering.

Raytheon Application of AI to Defense Systems

Deep Learning Physics Tools: Being Alert to Fatigue

"Jupiter, the Naval Enterprise Data and Analytics Environment, enables Sailors, Marines, and Civilians to integrate, share, and exploit data at speed and scale across the Warfighting, Business, and Readiness pillars for operational advantage and increased efficiency.  The Department of Navy Chief Data Officer and the Jupiter platform enable Artifical Intelligence across the Department of Navy.
"

Jupiter - AI and Data Enablement

Multi-Platform Space-Based AIS Fused RF Machine Learning

"Most AI/ML practitioners would agree that one of the top challenges they face is a lack of real-world data to use for research, training, and development. Scientists and engineers from NIWC Atlantic have been working to address this problem by supporting Task Force 59 to develop an infrastructure for collecting raw sensor data during maritime exercises and moving that data into a secure cloud store for later use by the fleet, S&T, and a variety of other stakeholders.
This presentation will provide an overview of the NIWC Atlantic involvement with Task Force 59 and Digital Horizon 2022, with specific focus on data collection and data lake creation. Engineering for this event was performed on a highly compressed timeline with numerous challenges and unknowns, and we will share the lessons learned which reflect an approach of pragmatism and initiative. 
"

Premium content

Reinforcement Learning for UUV Control Using Stable and Accurate Neural Network Approximations of Motion Dynamics

Downloads

Next from NAML 2023

Hybrid Optical Turbulence Models Using Machine Learning and Local Measurements

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES