United States

In this paper we study the multi-agent pursuit-evasion problem, and present an extension of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) deep reinforcement learning algorithm. Previous pursuit-evasion advancements with MADDPG have focused on training capture strategies dependent on the restriction of evader movement with environmental features. We demonstrate a method to train pursuer agents to collaboratively surround and encircle an evader for reliable capture without a strategy rooted in environment entrapment (i.e. cornering). Our method utilizes a novel two-stage, variable-aggression, continuous reward function based on geometrical inscribed circles (incircles), along with a corresponding observation space, with agents operating in an entrapment-disadvantaged environment. Our results show reliable capture of an intelligent, superior evader by three trained pursuers in open space with our encircling strategy.

NAML 2022

Robotic Swarm Pursuit-Evasion Capture Strategy using Deep Reinforcement Learning - An Update

maddpg

swarm

robot

learning

The 9th Brazilian Conference on Intelligent Systems (BRACIS), the 16h Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), and the 12th Concurso de Teses e Dissertações em Inteligência Artificial e Computacional (CTDIAC) was held from 20 to 23 October, 2020.

BRACIS originated from the combination of the two most important scientific events in Brazil in Artificial Intelligence (AI) and Computational Intelligence (CI): the Brazilian Symposium on Artificial Intelligence - SBIA (22 editions), and the Brazilian Symposium on Neural Networks - SBRN (13 editions). ENIAC is a national forum for researchers, practitioners, educators and students to present and discuss innovations, trends, experiences, developments and work in progress in the fields of Artificial and Computational Intelligence. CTDIAC is a satellite event whose goals are to disseminate and award the best dissertations and theses (MSc and PhDs concluded in Brazilian universities from June 2018 to May 2020) in Artificial and Computational Intelligence areas.

In this year, the Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) will be held in conjunction with BRACIS 2020. This sysposium aims at integrating researchers, practitioners, developers, students and users to present their research results, to discuss ideas, and to exchange techniques, tools, and practical experiences – related to the Data Mining and Machine Learning areas.

BRACIS 2020

technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Many different complex physics-based algorithms can be used to predict the quality of channels in radio frequency (RF) communications with varying levels of fidelity across many parameters. High frequency (HF) channels are influenced by complicated, sparsely-measured processes including propagation (due to space, weather, platform movement, etc.), interference (due to human activity, automated processes, etc.), and receiver/transmitter hardware. We present the High Frequency Observability Generative Adversarial Network (HOGAN), a contextual GAN for HF channel modeling and data generation. GANs have demonstrated the capability to perform extremely complicated, difficult to quantify transforms without exhaustively curated datasets. HOGAN is currently trained using synthetic RadioML IQ data with the goal of training on over-the-air data collected in the field for passive forecasting of HF channels to estimate how transmitted signals will be received. HOGAN aims to build upon the success of the RFMLS program and other RF machine learning efforts at NIWC to lay the foundations for applied RF GANs.

HOGAN: High-Frequency Observability Generative Adversarial Network

High frequency (HF) is being re-invented across the DoD services as they envision a great power competition and evaluate the survivability of satellite systems. Maneuverability is key and an agile radio using artificial intelligence (AI) and machine learning (ML) could provide the agility needed across the HF spectrum. 

Specifically, the weakness of an automatic link establishment (ALE) systems [1] is that they only transmit on frequencies within a lookup table. If those frequencies are in use or jammed the ALE system will not connect. A solution is an ALE protocols that can scan the spectrum to find the best spectrum given an environmental condition. An environmental condition can be data collected from a radio receiver in a specific location (e.g., a ship sailing in a mission location). The data might include the information such as which frequencies are used, the patterns for when and how the frequencies are used, weather and terrain conditions for the location, and adversary activities such as jamming. Software defined radios (SDR) combined with recent AI/ML algorithms make it possible to define and adapt usable radio frequencies based on environment inputs. To meet the need, the objective of this proposal is to demonstrate an SDR system leveraging AI/ML algorithms such as predictive models, cognitive architecture, optimization algorithms, and game theory. For example, HF communications via the Ionosphere between 3-30 MHz is a dynamic part of the spectrum where ALE radio already exist. They use lookup tables and do not “see” the spectrum and cannot make choices base on the actual environment. The HF spectrum is the most challenging environment and could be the first users to adopt the concept.

Cognitive and Agile Radio

Cyber-attacks can escalate quickly, with serious ramifications. Once attackers have gained a foothold, they can exploit assets to move laterally within the network – often before human-led defence teams have had time to respond – gaining control of systems and information. ML/AI provides an opportunity to combat these attacks in a relevant time frame, through autonomous cyber defence.
Within Reinforcement Learning (RL), agents learn efficient strategies to win through their interaction with an environment and other agents. Behaviours do not need to be explicitly programmed – rather they are learned through multiple successes and failures. As such, RL lends itself well to complex problems where multiple courses of action could be appropriate or unknown to human operators.
CyberBattleSim (Microsoft) is an open-source experimentation platform and training environment for RL agents. It provides a high-level abstract simulation of enterprise networks in which Red and Blue team agents compete for control. Its simplicity allows the simulation to be readily modified to model new threats, vulnerabilities and attack patterns.
We report on the use of multi-agent RL to defend network infrastructure against an adversary attack within CyberBattleSim. We ground the abstract simulation in the real world using attack methodologies derived from APT threat groups and the MITRE ATT&CK framework. Red agents are rewarded for exploiting vulnerabilities and gaining control of nodes within the network. Blue agents can disrupt this by hardening or re-imaging nodes and are rewarded for maintaining control and availability of network resources. Blue agent collaboration is explored using centralized and decentralized communication strategies are explored to support effective multi-agent behaviour.

Autonomous network defence using Reinforcement Learning within CyberBattleSim

Machine learning requires large amounts of data. Despite the extensive availability of mission data collected from joint missions, military doctrine regularly compartmentalizes data to prevent accidental disclosure of operational details. Current procedures to consolidate data from different sources are manually intensive and vulnerable to insider manipulation. We desire solutions for cooperative secure learning that maintain accuracy while both protecting sensitive data sources and containing insider threats.

In this presentation, we describe a layered approach to secure, multi-party learning, in which each step in the learning process adds more security, and demonstrate its use on sensitive Security Operation Center (SOC) data. We will describe the components and flow through the system that allow for this layered security, for both single-party and multi-party learning. Our approach first combines automated anonymization and synthetic data generation to protect sensitive data with minimal loss in accuracy. We then extract feature embeddings that resist reverse engineering for sharing with the learning system via three options for privacy preserving models, and describe the training and testing process. Finally we show both performance and privacy metrics to prove this method's success in maintaining security without sacrificing performance. In most test cases, similar or better accuracy is achieved, with overall labeling accuracies ranging between 85% and 99%.

Finally, we conclude with discussion on how this work can be applied outside of cyber data, and how it can be used to make machine learning more robust in general.

Secure, Privacy-Preserving Machine Learning

Abstract – Using agent-based simulation and distributed cloud computing to test how massive, complex networks will react to scenarios that haven’t happened yet 

Studying complex systems that contribute to or impact the Navy mission is essential, but these systems cannot be fully understood by analyzing their individual components. If a complex system cannot be studied analytically, an alternative option is to explore the system through modeling and simulation. For government and commercial partners, Deloitte uses FutureScape as a modeling and simulation platform to create digital replicas of complex, real-world systems and test ways to improve them. 

In this presentation, we will hone-in on how to use FutureScape and build a controlled environment around Department of Defense and Navy use cases to answer what-if questions in an expeditious and cost-efficient manner. A few examples that we are actively working on with DoD partners include but are not limited to: 
-Create a digital testbed of the fuel supply chain network that enables the military to quantify the impact of disruptions on fuel supply and subsequently, readiness and mobilization times. 
-Model C4ISR and weapons systems and simulate and evaluate incursion scenarios to test the efficacy of layered defenses. 
-Model non-spatial networks to identify critical points in the network that, if disrupted, would have major cascading effects up or downstream. 

Recognizing there are significant investments being made by the DoD on modeling and simulation solutions for Naval Applications, our intent is to hone-in on a tested approach to generate data and insights about hypothetical scenarios. This will ultimately help sailors and Navy leadership evaluate the impact of potential disruptions, test strategies and contingency plans, mitigate risks, and make strategic choices impacting complex systems that make up their day-to-day operations.

Using Agent-Based Simulation, Machine Learning, and Distributed Cloud Computing to test how massive, complex networks will react to novel scenarios

Increasingly in cyber security, network traffic detection techniques leverage the advances of machine learning (ML) algorithms to gain significant improvement in performance. However, traditional ML based network traffic detection algorithms rely heavily on subject matter expertise for feature engineering (i.e., feature extraction). Restricting the ability of ML algorithms to identify patterns within the raw network traffic that are imperceptible to a human analyst. Therefore, we propose a novel ML based network traffic detection algorithm using the raw bytes of a network packet as features. Thus, minimizing the reliance on a subject matter expert for feature engineering.
Our prior work [1] demonstrates the feasibility of leveraging the raw bytes of a network packet as features for a 1D Convolutional Neural Network (CNN) and Feed Forward Neural Network (FNN) based network detection algorithm and respectively achieve an accuracy of 98.99% and 98.97%. However, neither the 1D CNN nor FNN foster a human analyst’s interpretability of the output prediction (i.e., malicious or benign). Thus, we propose the use of a Random Forest (RF) for prediction on raw bytes of a network packet. Similar to [2], the feature importance from our RF network classifier, assists the human analyst in interpretation of the output prediction. Furthermore, a structure is constructed, so that each value in a feature vector retains the same meaning (i.e., context) across all samples. Our feature vector is composed of select bytes from the TCP or UDP packet header. 
The UNSW-NB15 dataset is utilized to evaluate our novel RF network traffic classifier. Our Random Forest based raw network traffic detection achieves an accuracy of 98.96%. Additionally, the most important feature is the TCP offset header field. Thus, our method demonstrates the utility of using raw network traffic coupled with the RF reduces the need for a subject matter expert to perform feature engineering. Furthermore, the RF assists a human analyst in interpretation of the output prediction.

[1] M. De Lucia, P. E. Maxwell, N. D. Bastian, A. Swami, B. Jalaian, and N. Leslie, “Machine learning raw network traffic detection,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, Online Only, United States, Apr. 2021, p. 24. doi: 10.1117/12.2586114.
[2] J. Holland, P. Schmitt, N. Feamster, and P. Mittal, “New Directions in Automated Traffic Analysis,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event Republic of Korea, Nov. 2021, pp. 3366–3383. doi: 10.1145/3460120.3484758.

Random Forest based Raw Network Traffic Detection

Polyglot files employ a form of steganography to combine the valid structure of two or more file types. They evade detection by circumventing feature extraction — a process heavily reliant on correct file type identification. The objective of this work is to design high throughput, accurate machine learning / deep learning models for polyglot detection. These models detect polyglots prior to the file type-sensitive feature extraction used in current commercial off-the-shelf endpoint malware detection tools and browser-based file input sanitization systems, thereby protecting against polyglot malware without necessitating a complete overhaul of the existing tools/systems. Furthermore, these models allow us to quantify and catalog the presence of polyglot malware in existing malware repositories, thereby enhancing digital forensics capabilities. This presentation provides a technical overview of this subject area as well as our initial results.

Using Machine Learning to Enhance the Detection of Steganographic Malware

This work explores local congestion control methods for undersea acoustic networks. Congestion control policies are deployed and managed locally at each node acting as a router. Rather than accepting all traffic, intermediate routers can decide not to accept certain packets or to reduce the rate at which they receive packets while resolving the congestion they are experiencing. By not accepting packets, a backlog of packets is built in upstream queues. This backlog will eventually reach the packet sources and, similar to TCP, trigger a traffic rate reduction at the source application. Local packet queue management policies are also responsible for managing queue-space allocations among different traffic classes.These policies are developed based on a financial framework in which each traffic type is associated a commodity that is part of an investment portfolio. Each router, uses its local policy to optimize its portfolio based on the funds it has available, that is, its available data storage space. The space allocation profile is updated periodically via the solution of the celebrated Markowitz Portfolio Selection problem. A price model for packets based on local congestion indicators is used to quantify notions of return, profit and risk. Our framework uses a hybrid policy that combines the space allocation policy with a collection of traffic flow control policies implemented separately per traffic class. The traffic flow control policies are responsible for deciding what flows to pause and restart, and when to requests a reduced packet transmission from a neighboring transmitter. Traffic flow control policies use congestion indicators such as queue occupancy levels per traffic class, packet priority, and sojourn time statistics. These policies are learned via a reinforcement learning framework. The congestion control policy is coupled with a traffic shaping policy implemented at the same node. The traffic shaping policy is responsible for allocating available transmission bandwidth to each traffic type based on traffic priorities and quality-of-service (QoS) requirements. Initially our framework presumes an arbitrary traffic-shaping policy that does not coordinate with the local congestion control policy to decide what packets to prioritize. Numerical tests illustrating the behavior of several components of the proposed method will be discussed.

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Recent advances with artificial intelligence (AI) in industry and academia provide an opportunity to develop intelligent system agents capable of reliably competing against human experts in course-of-action analysis. The team trained agents using deep reinforcement learning in a prototype wargaming framework in naval scenarios. Team members conducted experiments to evaluate the performance of AI threat agents trained using multiple algorithms and parameter configurations in each scenario and identified configurations providing superior results. Trained agents were placed in a desktop wargaming framework to allow play against human opponents, enabling the team to identify areas for improvement. This presentation provides an overview of the wargaming framework, results of experimentation, and next steps.

Improving Course-of-Action Analysis in Naval Wargames with Reinforcement Learning

Control systems for unmanned undersea vehicles (UUVs) are typically implemented using proportional integral derivative (PID) control systems. PID control systems for UUVs are resource-intensive to tune because they require several engineers, marine operators, and ship crew working together to adjust the controller. Furthermore, PID controllers rely on complex dynamic system models that contain assumptions to reduce the computational complexity of the models but can degrade the controller's performance if an environmental condition is encountered that conflicts with an assumption. In this study, a deep reinforcement learning (DRL) control system based on the Deep Deterministic Policy Gradient (DDPG) algorithm is investigated for a UUV control system. The DDPG algorithm is model-free, meaning that a complex dynamic system model is not needed to learn and provide optimal control performance. DRL-based control systems are tuned autonomously, which can greatly reduce the resources needed for tuning. The drawback to deep reinforcement learning is that it is computationally and resource-intensive. This study investigates how mixed floating-point precision with loss scaling can be used to reduce the time and computational resources needed to train the DDPG agent. Numerical case studies are presented.

Premium content

Robotic Swarm Pursuit-Evasion Capture Strategy using Deep Reinforcement Learning - An Update

Downloads

Next from NAML 2022

HOGAN: High-Frequency Observability Generative Adversarial Network

Similar lecture

Interaffection of Multiple Datasets with Neural Networks in Speech Emotion Recognition

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES