United States

Control systems for unmanned undersea vehicles (UUVs) are typically implemented using proportional integral derivative (PID) control systems. PID control systems for UUVs are resource-intensive to tune because they require several engineers, marine operators, and ship crew working together to adjust the controller. Furthermore, PID controllers rely on complex dynamic system models that contain assumptions to reduce the computational complexity of the models but can degrade the controller&#39;s performance if an environmental condition is encountered that conflicts with an assumption. In this study, a deep reinforcement learning (DRL) control system based on the Deep Deterministic Policy Gradient (DDPG) algorithm is investigated for a UUV control system. The DDPG algorithm is model-free, meaning that a complex dynamic system model is not needed to learn and provide optimal control performance. DRL-based control systems are tuned autonomously, which can greatly reduce the resources needed for tuning. The drawback to deep reinforcement learning is that it is computationally and resource-intensive. This study investigates how mixed floating-point precision with loss scaling can be used to reduce the time and computational resources needed to train the DDPG agent. Numerical case studies are presented.

NAML 2022

Mixed Precision Reinforcement Learning for Control Simulation of Unmanned Undersea Vehicles

uuv control

deep reinforcement learning

reinforcement learning

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).
<br>
<br>



A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS 2022

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

Control systems for unmanned undersea vehicles (UUVs) are typically implemented using proportional integral derivative (PID) control systems. PID control systems for UUVs are resource-intensive to tune because they require several engineers, marine operators, and ship crew working together to adjust the controller. Furthermore, PID controllers rely on complex dynamic system models that contain assumptions to reduce the computational complexity of the models but can degrade the controller's performance if an environmental condition is encountered that conflicts with an assumption. In this study, a deep reinforcement learning (DRL) control system based on the Deep Deterministic Policy Gradient (DDPG) algorithm is investigated for a UUV control system. The DDPG algorithm is model-free, meaning that a complex dynamic system model is not needed to learn and provide optimal control performance. DRL-based control systems are tuned autonomously, which can greatly reduce the resources needed for tuning. The drawback to deep reinforcement learning is that it is computationally and resource-intensive. This study investigates how mixed floating-point precision with loss scaling can be used to reduce the time and computational resources needed to train the DDPG agent. Numerical case studies are presented.

technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Modern commercial navigation radars are cheap to purchase, however, with the use of DSP they can have high resolutions and complicated waveforms. This has lead them to proliferate across both civilian and military domains and to the RF band they operate in becoming increasingly congested. To improve situational awareness the ability to distinguish between different individual radars on a pulse-by-pulse basis is desired. This would provide the means to track military targets whilst ignoring civilian ones.
This work has investigated the application of SEI to this radar type, i.e. the ability to distinguish between individual commercial navigation radar. IQ data of a set of 30 radar from three common makes, 10 from each, were collected in an anechoic chamber using a wideband digitiser. Deep learning techniques were applied in order to perform classification. These techniques included the use of residual and inception networks, along with time-frequency transforms as a means of feature engineering. Results varied across the different makes with one make having a high F1 scores and two with considerably lower F1 scores; however, it is possible to distinguish between makes with ease.


Radar Specific Emitter Identification (SEI) Against Modern Commercial Navigation Radar

Many different complex physics-based algorithms can be used to predict the quality of channels in radio frequency (RF) communications with varying levels of fidelity across many parameters. High frequency (HF) channels are influenced by complicated, sparsely-measured processes including propagation (due to space, weather, platform movement, etc.), interference (due to human activity, automated processes, etc.), and receiver/transmitter hardware. We present the High Frequency Observability Generative Adversarial Network (HOGAN), a contextual GAN for HF channel modeling and data generation. GANs have demonstrated the capability to perform extremely complicated, difficult to quantify transforms without exhaustively curated datasets. HOGAN is currently trained using synthetic RadioML IQ data with the goal of training on over-the-air data collected in the field for passive forecasting of HF channels to estimate how transmitted signals will be received. HOGAN aims to build upon the success of the RFMLS program and other RF machine learning efforts at NIWC to lay the foundations for applied RF GANs.

HOGAN: High-Frequency Observability Generative Adversarial Network

High frequency (HF) is being re-invented across the DoD services as they envision a great power competition and evaluate the survivability of satellite systems. Maneuverability is key and an agile radio using artificial intelligence (AI) and machine learning (ML) could provide the agility needed across the HF spectrum. 

Specifically, the weakness of an automatic link establishment (ALE) systems [1] is that they only transmit on frequencies within a lookup table. If those frequencies are in use or jammed the ALE system will not connect. A solution is an ALE protocols that can scan the spectrum to find the best spectrum given an environmental condition. An environmental condition can be data collected from a radio receiver in a specific location (e.g., a ship sailing in a mission location). The data might include the information such as which frequencies are used, the patterns for when and how the frequencies are used, weather and terrain conditions for the location, and adversary activities such as jamming. Software defined radios (SDR) combined with recent AI/ML algorithms make it possible to define and adapt usable radio frequencies based on environment inputs. To meet the need, the objective of this proposal is to demonstrate an SDR system leveraging AI/ML algorithms such as predictive models, cognitive architecture, optimization algorithms, and game theory. For example, HF communications via the Ionosphere between 3-30 MHz is a dynamic part of the spectrum where ALE radio already exist. They use lookup tables and do not “see” the spectrum and cannot make choices base on the actual environment. The HF spectrum is the most challenging environment and could be the first users to adopt the concept.

Cognitive and Agile Radio

Cyber-attacks can escalate quickly, with serious ramifications. Once attackers have gained a foothold, they can exploit assets to move laterally within the network – often before human-led defence teams have had time to respond – gaining control of systems and information. ML/AI provides an opportunity to combat these attacks in a relevant time frame, through autonomous cyber defence.
Within Reinforcement Learning (RL), agents learn efficient strategies to win through their interaction with an environment and other agents. Behaviours do not need to be explicitly programmed – rather they are learned through multiple successes and failures. As such, RL lends itself well to complex problems where multiple courses of action could be appropriate or unknown to human operators.
CyberBattleSim (Microsoft) is an open-source experimentation platform and training environment for RL agents. It provides a high-level abstract simulation of enterprise networks in which Red and Blue team agents compete for control. Its simplicity allows the simulation to be readily modified to model new threats, vulnerabilities and attack patterns.
We report on the use of multi-agent RL to defend network infrastructure against an adversary attack within CyberBattleSim. We ground the abstract simulation in the real world using attack methodologies derived from APT threat groups and the MITRE ATT&CK framework. Red agents are rewarded for exploiting vulnerabilities and gaining control of nodes within the network. Blue agents can disrupt this by hardening or re-imaging nodes and are rewarded for maintaining control and availability of network resources. Blue agent collaboration is explored using centralized and decentralized communication strategies are explored to support effective multi-agent behaviour.

Autonomous network defence using Reinforcement Learning within CyberBattleSim

Machine learning requires large amounts of data. Despite the extensive availability of mission data collected from joint missions, military doctrine regularly compartmentalizes data to prevent accidental disclosure of operational details. Current procedures to consolidate data from different sources are manually intensive and vulnerable to insider manipulation. We desire solutions for cooperative secure learning that maintain accuracy while both protecting sensitive data sources and containing insider threats.

In this presentation, we describe a layered approach to secure, multi-party learning, in which each step in the learning process adds more security, and demonstrate its use on sensitive Security Operation Center (SOC) data. We will describe the components and flow through the system that allow for this layered security, for both single-party and multi-party learning. Our approach first combines automated anonymization and synthetic data generation to protect sensitive data with minimal loss in accuracy. We then extract feature embeddings that resist reverse engineering for sharing with the learning system via three options for privacy preserving models, and describe the training and testing process. Finally we show both performance and privacy metrics to prove this method's success in maintaining security without sacrificing performance. In most test cases, similar or better accuracy is achieved, with overall labeling accuracies ranging between 85% and 99%.

Finally, we conclude with discussion on how this work can be applied outside of cyber data, and how it can be used to make machine learning more robust in general.

Secure, Privacy-Preserving Machine Learning

Abstract – Using agent-based simulation and distributed cloud computing to test how massive, complex networks will react to scenarios that haven’t happened yet 

Studying complex systems that contribute to or impact the Navy mission is essential, but these systems cannot be fully understood by analyzing their individual components. If a complex system cannot be studied analytically, an alternative option is to explore the system through modeling and simulation. For government and commercial partners, Deloitte uses FutureScape as a modeling and simulation platform to create digital replicas of complex, real-world systems and test ways to improve them. 

In this presentation, we will hone-in on how to use FutureScape and build a controlled environment around Department of Defense and Navy use cases to answer what-if questions in an expeditious and cost-efficient manner. A few examples that we are actively working on with DoD partners include but are not limited to: 
-Create a digital testbed of the fuel supply chain network that enables the military to quantify the impact of disruptions on fuel supply and subsequently, readiness and mobilization times. 
-Model C4ISR and weapons systems and simulate and evaluate incursion scenarios to test the efficacy of layered defenses. 
-Model non-spatial networks to identify critical points in the network that, if disrupted, would have major cascading effects up or downstream. 

Recognizing there are significant investments being made by the DoD on modeling and simulation solutions for Naval Applications, our intent is to hone-in on a tested approach to generate data and insights about hypothetical scenarios. This will ultimately help sailors and Navy leadership evaluate the impact of potential disruptions, test strategies and contingency plans, mitigate risks, and make strategic choices impacting complex systems that make up their day-to-day operations.

Using Agent-Based Simulation, Machine Learning, and Distributed Cloud Computing to test how massive, complex networks will react to novel scenarios

Increasingly in cyber security, network traffic detection techniques leverage the advances of machine learning (ML) algorithms to gain significant improvement in performance. However, traditional ML based network traffic detection algorithms rely heavily on subject matter expertise for feature engineering (i.e., feature extraction). Restricting the ability of ML algorithms to identify patterns within the raw network traffic that are imperceptible to a human analyst. Therefore, we propose a novel ML based network traffic detection algorithm using the raw bytes of a network packet as features. Thus, minimizing the reliance on a subject matter expert for feature engineering.
Our prior work [1] demonstrates the feasibility of leveraging the raw bytes of a network packet as features for a 1D Convolutional Neural Network (CNN) and Feed Forward Neural Network (FNN) based network detection algorithm and respectively achieve an accuracy of 98.99% and 98.97%. However, neither the 1D CNN nor FNN foster a human analyst’s interpretability of the output prediction (i.e., malicious or benign). Thus, we propose the use of a Random Forest (RF) for prediction on raw bytes of a network packet. Similar to [2], the feature importance from our RF network classifier, assists the human analyst in interpretation of the output prediction. Furthermore, a structure is constructed, so that each value in a feature vector retains the same meaning (i.e., context) across all samples. Our feature vector is composed of select bytes from the TCP or UDP packet header. 
The UNSW-NB15 dataset is utilized to evaluate our novel RF network traffic classifier. Our Random Forest based raw network traffic detection achieves an accuracy of 98.96%. Additionally, the most important feature is the TCP offset header field. Thus, our method demonstrates the utility of using raw network traffic coupled with the RF reduces the need for a subject matter expert to perform feature engineering. Furthermore, the RF assists a human analyst in interpretation of the output prediction.

[1] M. De Lucia, P. E. Maxwell, N. D. Bastian, A. Swami, B. Jalaian, and N. Leslie, “Machine learning raw network traffic detection,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, Online Only, United States, Apr. 2021, p. 24. doi: 10.1117/12.2586114.
[2] J. Holland, P. Schmitt, N. Feamster, and P. Mittal, “New Directions in Automated Traffic Analysis,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event Republic of Korea, Nov. 2021, pp. 3366–3383. doi: 10.1145/3460120.3484758.

Random Forest based Raw Network Traffic Detection

Polyglot files employ a form of steganography to combine the valid structure of two or more file types. They evade detection by circumventing feature extraction — a process heavily reliant on correct file type identification. The objective of this work is to design high throughput, accurate machine learning / deep learning models for polyglot detection. These models detect polyglots prior to the file type-sensitive feature extraction used in current commercial off-the-shelf endpoint malware detection tools and browser-based file input sanitization systems, thereby protecting against polyglot malware without necessitating a complete overhaul of the existing tools/systems. Furthermore, these models allow us to quantify and catalog the presence of polyglot malware in existing malware repositories, thereby enhancing digital forensics capabilities. This presentation provides a technical overview of this subject area as well as our initial results.

Using Machine Learning to Enhance the Detection of Steganographic Malware

This work explores local congestion control methods for undersea acoustic networks. Congestion control policies are deployed and managed locally at each node acting as a router. Rather than accepting all traffic, intermediate routers can decide not to accept certain packets or to reduce the rate at which they receive packets while resolving the congestion they are experiencing. By not accepting packets, a backlog of packets is built in upstream queues. This backlog will eventually reach the packet sources and, similar to TCP, trigger a traffic rate reduction at the source application. Local packet queue management policies are also responsible for managing queue-space allocations among different traffic classes.These policies are developed based on a financial framework in which each traffic type is associated a commodity that is part of an investment portfolio. Each router, uses its local policy to optimize its portfolio based on the funds it has available, that is, its available data storage space. The space allocation profile is updated periodically via the solution of the celebrated Markowitz Portfolio Selection problem. A price model for packets based on local congestion indicators is used to quantify notions of return, profit and risk. Our framework uses a hybrid policy that combines the space allocation policy with a collection of traffic flow control policies implemented separately per traffic class. The traffic flow control policies are responsible for deciding what flows to pause and restart, and when to requests a reduced packet transmission from a neighboring transmitter. Traffic flow control policies use congestion indicators such as queue occupancy levels per traffic class, packet priority, and sojourn time statistics. These policies are learned via a reinforcement learning framework. The congestion control policy is coupled with a traffic shaping policy implemented at the same node. The traffic shaping policy is responsible for allocating available transmission bandwidth to each traffic type based on traffic priorities and quality-of-service (QoS) requirements. Initially our framework presumes an arbitrary traffic-shaping policy that does not coordinate with the local congestion control policy to decide what packets to prioritize. Numerical tests illustrating the behavior of several components of the proposed method will be discussed.

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Recent advances with artificial intelligence (AI) in industry and academia provide an opportunity to develop intelligent system agents capable of reliably competing against human experts in course-of-action analysis. The team trained agents using deep reinforcement learning in a prototype wargaming framework in naval scenarios. Team members conducted experiments to evaluate the performance of AI threat agents trained using multiple algorithms and parameter configurations in each scenario and identified configurations providing superior results. Trained agents were placed in a desktop wargaming framework to allow play against human opponents, enabling the team to identify areas for improvement. This presentation provides an overview of the wargaming framework, results of experimentation, and next steps.

Premium content

Mixed Precision Reinforcement Learning for Control Simulation of Unmanned Undersea Vehicles

Downloads

Next from NAML 2022

Radar Specific Emitter Identification (SEI) Against Modern Commercial Navigation Radar

Similar lecture

Multimodal Reinforcement Learning with Effective State Representation Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES