United States

In this work, we study a cognitive radio network in which the objective is to maximize the throughput of a secondary user (SU) subject to a constraint on the collision probability with the primary user (PU). For a Markov transmission model of the PU, it has been shown that Age of Information (AoI) plays an important role in this optimization. We have previously developed a throughput-optimal secondary user sensing/transmission policy by solving the linear program for the optimal distribution of the PU state and the AoI of sensed information. This approach requires that the PU idle/transmission dynamics are known, which is not typically the case in practice. Therefore, we implement this approach within a reinforcement learning algorithm by learning the model and then applying the optimization to the learned model. Reinforcement learning algorithms are typically only concerned with maximizing rewards, and therefore do not account for the constraint of avoiding collisions with the PU. To control this cost as well as the reward, we apply a model-based reinforcement learning approach for constrained Markov Decision Processes (CMDP) based on Upper Confidence Reinforcement Learning (UCRL-CMDP), which has proven regret guarantees for the reward and cost. We compare this approach with a model-based (Dyna-Q) and a model-free (Actor-Critic) RL algorithm with a Lagrangian reward function to incorporate the constraint. Simulation results demonstrate that the UCRL-CMDP performs favorably compared with the other two approaches in terms of throughput and computational efficiency.

NAML 2022

Online Learning of the Secondary User Policy in a Cognitive Radio Network with Constraints on a Markov Primary User with Unknown Dynamics

age of information

constrained markov decision processes

cognitive radio networks

reinforcement learning

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



High frequency (HF) is being re-invented across the DoD services as they envision a great power competition and evaluate the survivability of satellite systems. Maneuverability is key and an agile radio using artificial intelligence (AI) and machine learning (ML) could provide the agility needed across the HF spectrum. 

Specifically, the weakness of an automatic link establishment (ALE) systems [1] is that they only transmit on frequencies within a lookup table. If those frequencies are in use or jammed the ALE system will not connect. A solution is an ALE protocols that can scan the spectrum to find the best spectrum given an environmental condition. An environmental condition can be data collected from a radio receiver in a specific location (e.g., a ship sailing in a mission location). The data might include the information such as which frequencies are used, the patterns for when and how the frequencies are used, weather and terrain conditions for the location, and adversary activities such as jamming. Software defined radios (SDR) combined with recent AI/ML algorithms make it possible to define and adapt usable radio frequencies based on environment inputs. To meet the need, the objective of this proposal is to demonstrate an SDR system leveraging AI/ML algorithms such as predictive models, cognitive architecture, optimization algorithms, and game theory. For example, HF communications via the Ionosphere between 3-30 MHz is a dynamic part of the spectrum where ALE radio already exist. They use lookup tables and do not “see” the spectrum and cannot make choices base on the actual environment. The HF spectrum is the most challenging environment and could be the first users to adopt the concept.

Cognitive and Agile Radio

Cyber-attacks can escalate quickly, with serious ramifications. Once attackers have gained a foothold, they can exploit assets to move laterally within the network – often before human-led defence teams have had time to respond – gaining control of systems and information. ML/AI provides an opportunity to combat these attacks in a relevant time frame, through autonomous cyber defence.
Within Reinforcement Learning (RL), agents learn efficient strategies to win through their interaction with an environment and other agents. Behaviours do not need to be explicitly programmed – rather they are learned through multiple successes and failures. As such, RL lends itself well to complex problems where multiple courses of action could be appropriate or unknown to human operators.
CyberBattleSim (Microsoft) is an open-source experimentation platform and training environment for RL agents. It provides a high-level abstract simulation of enterprise networks in which Red and Blue team agents compete for control. Its simplicity allows the simulation to be readily modified to model new threats, vulnerabilities and attack patterns.
We report on the use of multi-agent RL to defend network infrastructure against an adversary attack within CyberBattleSim. We ground the abstract simulation in the real world using attack methodologies derived from APT threat groups and the MITRE ATT&CK framework. Red agents are rewarded for exploiting vulnerabilities and gaining control of nodes within the network. Blue agents can disrupt this by hardening or re-imaging nodes and are rewarded for maintaining control and availability of network resources. Blue agent collaboration is explored using centralized and decentralized communication strategies are explored to support effective multi-agent behaviour.

Autonomous network defence using Reinforcement Learning within CyberBattleSim

Machine learning requires large amounts of data. Despite the extensive availability of mission data collected from joint missions, military doctrine regularly compartmentalizes data to prevent accidental disclosure of operational details. Current procedures to consolidate data from different sources are manually intensive and vulnerable to insider manipulation. We desire solutions for cooperative secure learning that maintain accuracy while both protecting sensitive data sources and containing insider threats.

In this presentation, we describe a layered approach to secure, multi-party learning, in which each step in the learning process adds more security, and demonstrate its use on sensitive Security Operation Center (SOC) data. We will describe the components and flow through the system that allow for this layered security, for both single-party and multi-party learning. Our approach first combines automated anonymization and synthetic data generation to protect sensitive data with minimal loss in accuracy. We then extract feature embeddings that resist reverse engineering for sharing with the learning system via three options for privacy preserving models, and describe the training and testing process. Finally we show both performance and privacy metrics to prove this method's success in maintaining security without sacrificing performance. In most test cases, similar or better accuracy is achieved, with overall labeling accuracies ranging between 85% and 99%.

Finally, we conclude with discussion on how this work can be applied outside of cyber data, and how it can be used to make machine learning more robust in general.

Secure, Privacy-Preserving Machine Learning

Abstract – Using agent-based simulation and distributed cloud computing to test how massive, complex networks will react to scenarios that haven’t happened yet 

Studying complex systems that contribute to or impact the Navy mission is essential, but these systems cannot be fully understood by analyzing their individual components. If a complex system cannot be studied analytically, an alternative option is to explore the system through modeling and simulation. For government and commercial partners, Deloitte uses FutureScape as a modeling and simulation platform to create digital replicas of complex, real-world systems and test ways to improve them. 

In this presentation, we will hone-in on how to use FutureScape and build a controlled environment around Department of Defense and Navy use cases to answer what-if questions in an expeditious and cost-efficient manner. A few examples that we are actively working on with DoD partners include but are not limited to: 
-Create a digital testbed of the fuel supply chain network that enables the military to quantify the impact of disruptions on fuel supply and subsequently, readiness and mobilization times. 
-Model C4ISR and weapons systems and simulate and evaluate incursion scenarios to test the efficacy of layered defenses. 
-Model non-spatial networks to identify critical points in the network that, if disrupted, would have major cascading effects up or downstream. 

Recognizing there are significant investments being made by the DoD on modeling and simulation solutions for Naval Applications, our intent is to hone-in on a tested approach to generate data and insights about hypothetical scenarios. This will ultimately help sailors and Navy leadership evaluate the impact of potential disruptions, test strategies and contingency plans, mitigate risks, and make strategic choices impacting complex systems that make up their day-to-day operations.

Using Agent-Based Simulation, Machine Learning, and Distributed Cloud Computing to test how massive, complex networks will react to novel scenarios

Increasingly in cyber security, network traffic detection techniques leverage the advances of machine learning (ML) algorithms to gain significant improvement in performance. However, traditional ML based network traffic detection algorithms rely heavily on subject matter expertise for feature engineering (i.e., feature extraction). Restricting the ability of ML algorithms to identify patterns within the raw network traffic that are imperceptible to a human analyst. Therefore, we propose a novel ML based network traffic detection algorithm using the raw bytes of a network packet as features. Thus, minimizing the reliance on a subject matter expert for feature engineering.
Our prior work [1] demonstrates the feasibility of leveraging the raw bytes of a network packet as features for a 1D Convolutional Neural Network (CNN) and Feed Forward Neural Network (FNN) based network detection algorithm and respectively achieve an accuracy of 98.99% and 98.97%. However, neither the 1D CNN nor FNN foster a human analyst’s interpretability of the output prediction (i.e., malicious or benign). Thus, we propose the use of a Random Forest (RF) for prediction on raw bytes of a network packet. Similar to [2], the feature importance from our RF network classifier, assists the human analyst in interpretation of the output prediction. Furthermore, a structure is constructed, so that each value in a feature vector retains the same meaning (i.e., context) across all samples. Our feature vector is composed of select bytes from the TCP or UDP packet header. 
The UNSW-NB15 dataset is utilized to evaluate our novel RF network traffic classifier. Our Random Forest based raw network traffic detection achieves an accuracy of 98.96%. Additionally, the most important feature is the TCP offset header field. Thus, our method demonstrates the utility of using raw network traffic coupled with the RF reduces the need for a subject matter expert to perform feature engineering. Furthermore, the RF assists a human analyst in interpretation of the output prediction.

[1] M. De Lucia, P. E. Maxwell, N. D. Bastian, A. Swami, B. Jalaian, and N. Leslie, “Machine learning raw network traffic detection,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, Online Only, United States, Apr. 2021, p. 24. doi: 10.1117/12.2586114.
[2] J. Holland, P. Schmitt, N. Feamster, and P. Mittal, “New Directions in Automated Traffic Analysis,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event Republic of Korea, Nov. 2021, pp. 3366–3383. doi: 10.1145/3460120.3484758.

Random Forest based Raw Network Traffic Detection

Polyglot files employ a form of steganography to combine the valid structure of two or more file types. They evade detection by circumventing feature extraction — a process heavily reliant on correct file type identification. The objective of this work is to design high throughput, accurate machine learning / deep learning models for polyglot detection. These models detect polyglots prior to the file type-sensitive feature extraction used in current commercial off-the-shelf endpoint malware detection tools and browser-based file input sanitization systems, thereby protecting against polyglot malware without necessitating a complete overhaul of the existing tools/systems. Furthermore, these models allow us to quantify and catalog the presence of polyglot malware in existing malware repositories, thereby enhancing digital forensics capabilities. This presentation provides a technical overview of this subject area as well as our initial results.

Using Machine Learning to Enhance the Detection of Steganographic Malware

This work explores local congestion control methods for undersea acoustic networks. Congestion control policies are deployed and managed locally at each node acting as a router. Rather than accepting all traffic, intermediate routers can decide not to accept certain packets or to reduce the rate at which they receive packets while resolving the congestion they are experiencing. By not accepting packets, a backlog of packets is built in upstream queues. This backlog will eventually reach the packet sources and, similar to TCP, trigger a traffic rate reduction at the source application. Local packet queue management policies are also responsible for managing queue-space allocations among different traffic classes.These policies are developed based on a financial framework in which each traffic type is associated a commodity that is part of an investment portfolio. Each router, uses its local policy to optimize its portfolio based on the funds it has available, that is, its available data storage space. The space allocation profile is updated periodically via the solution of the celebrated Markowitz Portfolio Selection problem. A price model for packets based on local congestion indicators is used to quantify notions of return, profit and risk. Our framework uses a hybrid policy that combines the space allocation policy with a collection of traffic flow control policies implemented separately per traffic class. The traffic flow control policies are responsible for deciding what flows to pause and restart, and when to requests a reduced packet transmission from a neighboring transmitter. Traffic flow control policies use congestion indicators such as queue occupancy levels per traffic class, packet priority, and sojourn time statistics. These policies are learned via a reinforcement learning framework. The congestion control policy is coupled with a traffic shaping policy implemented at the same node. The traffic shaping policy is responsible for allocating available transmission bandwidth to each traffic type based on traffic priorities and quality-of-service (QoS) requirements. Initially our framework presumes an arbitrary traffic-shaping policy that does not coordinate with the local congestion control policy to decide what packets to prioritize. Numerical tests illustrating the behavior of several components of the proposed method will be discussed.

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Recent advances with artificial intelligence (AI) in industry and academia provide an opportunity to develop intelligent system agents capable of reliably competing against human experts in course-of-action analysis. The team trained agents using deep reinforcement learning in a prototype wargaming framework in naval scenarios. Team members conducted experiments to evaluate the performance of AI threat agents trained using multiple algorithms and parameter configurations in each scenario and identified configurations providing superior results. Trained agents were placed in a desktop wargaming framework to allow play against human opponents, enabling the team to identify areas for improvement. This presentation provides an overview of the wargaming framework, results of experimentation, and next steps.

Improving Course-of-Action Analysis in Naval Wargames with Reinforcement Learning

Control systems for unmanned undersea vehicles (UUVs) are typically implemented using proportional integral derivative (PID) control systems. PID control systems for UUVs are resource-intensive to tune because they require several engineers, marine operators, and ship crew working together to adjust the controller. Furthermore, PID controllers rely on complex dynamic system models that contain assumptions to reduce the computational complexity of the models but can degrade the controller's performance if an environmental condition is encountered that conflicts with an assumption. In this study, a deep reinforcement learning (DRL) control system based on the Deep Deterministic Policy Gradient (DDPG) algorithm is investigated for a UUV control system. The DDPG algorithm is model-free, meaning that a complex dynamic system model is not needed to learn and provide optimal control performance. DRL-based control systems are tuned autonomously, which can greatly reduce the resources needed for tuning. The drawback to deep reinforcement learning is that it is computationally and resource-intensive. This study investigates how mixed floating-point precision with loss scaling can be used to reduce the time and computational resources needed to train the DDPG agent. Numerical case studies are presented.

Mixed Precision Reinforcement Learning for Control Simulation of Unmanned Undersea Vehicles

In this paper we study the multi-agent pursuit-evasion problem, and present an extension of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) deep reinforcement learning algorithm. Previous pursuit-evasion advancements with MADDPG have focused on training capture strategies dependent on the restriction of evader movement with environmental features. We demonstrate a method to train pursuer agents to collaboratively surround and encircle an evader for reliable capture without a strategy rooted in environment entrapment (i.e. cornering). Our method utilizes a novel two-stage, variable-aggression, continuous reward function based on geometrical inscribed circles (incircles), along with a corresponding observation space, with agents operating in an entrapment-disadvantaged environment. Our results show reliable capture of an intelligent, superior evader by three trained pursuers in open space with our encircling strategy.

Premium content

Online Learning of the Secondary User Policy in a Cognitive Radio Network with Constraints on a Markov Primary User with Unknown Dynamics

Downloads

Next from NAML 2022

Cognitive and Agile Radio

Similar lecture

Computational Modeling Of Trust Factors in Recognition of Warfighting Ships

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES