United States

Fast reorientation of space-based sensors is needed to increase ISR capacity and to respond to high-value targeting requests. Rapid reorientation maneuvers are realized by constructing and solving a nonlinear optimization problem that models the physics of the rotational motion as part of the ground process of mission operations. The solution trajectories are then uplinked to the spacecraft for attitude guidance. Attitude guidance commands for a spacecraft are a time-series of incremental rotations that are given to the attitude control system at the closed-loop feedback rate which is typically greater than 10 Hz.

In practical space systems, it is generally not possible to uplink reorientation commands as a finely sampled time-series (at or greater than 10 Hz) since this would quickly exhaust the capacity of a spacecraft’s command buffer. Instead, the current paradigm is to issue a reorientation command as a single time-tagged target orientation. Then, an on-board trajectory generator is used to construct an appropriate rotation command time-series at the closed-loop feedback rate which are then fed to the attitude guidance system. Typically, these on-board trajectory generators do not have the processing power to perform trajectory optimization and instead generate a feasible, but sub-optimal, time-series command set at the appropriate rate.

A potential solution to this problem is to compute and then compress the optimized time-series commands using an autoencoder. An autoencoder is an unsupervised machine learning application of a neural network that conducts a non-linear mapping of the inputs to a lower dimensional space, called the latent space, using the encoder portion of the autoencoder. It then reconstructs the same input from the latent space using the decoder portion of the autoencoder. With such a method, the optimized command sequence can be generated and encoded on the ground, then only the latent space is transmitted to the spacecraft and decoded on-board the spacecraft resulting in the reconstructed optimized rotation sequence at the closed-loop feedback rate. Given the time-series nature of the optimized command sequence, one appropriate architecture for the autoencoder is a long short term memory (LSTM) neural network autoencoder.

In this talk, we demonstrate the methodology for constructing and training the LSTM autoencoder and show the reconstruction error for different degrees of compression represented by decreasing dimensionality of the latent space. We then compare the reconstructed command sequence and the uncompressed time-optimal solutions using a flight simulation software at the closed-loop feedback rate. The result is a new and computationally efficient approach for implementing non-legacy maneuvers on board a spacecraft. There are numerous applications beyond spacecraft where this capability can benefit the DoD such as small unmanned vehicle operations and operations in degraded electromagnetic environments.

NAML 2022

A Method to Compress and Transmit Optimal Reorientation Commands to a Spacecraft Using Long Short-Term Memory Neural Network Autoencoders

spacecraft

long short-term memory neural network

autoencoder

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Abstract – Using agent-based simulation and distributed cloud computing to test how massive, complex networks will react to scenarios that haven’t happened yet 

Studying complex systems that contribute to or impact the Navy mission is essential, but these systems cannot be fully understood by analyzing their individual components. If a complex system cannot be studied analytically, an alternative option is to explore the system through modeling and simulation. For government and commercial partners, Deloitte uses FutureScape as a modeling and simulation platform to create digital replicas of complex, real-world systems and test ways to improve them. 

In this presentation, we will hone-in on how to use FutureScape and build a controlled environment around Department of Defense and Navy use cases to answer what-if questions in an expeditious and cost-efficient manner. A few examples that we are actively working on with DoD partners include but are not limited to: 
-Create a digital testbed of the fuel supply chain network that enables the military to quantify the impact of disruptions on fuel supply and subsequently, readiness and mobilization times. 
-Model C4ISR and weapons systems and simulate and evaluate incursion scenarios to test the efficacy of layered defenses. 
-Model non-spatial networks to identify critical points in the network that, if disrupted, would have major cascading effects up or downstream. 

Recognizing there are significant investments being made by the DoD on modeling and simulation solutions for Naval Applications, our intent is to hone-in on a tested approach to generate data and insights about hypothetical scenarios. This will ultimately help sailors and Navy leadership evaluate the impact of potential disruptions, test strategies and contingency plans, mitigate risks, and make strategic choices impacting complex systems that make up their day-to-day operations.

Using Agent-Based Simulation, Machine Learning, and Distributed Cloud Computing to test how massive, complex networks will react to novel scenarios

Increasingly in cyber security, network traffic detection techniques leverage the advances of machine learning (ML) algorithms to gain significant improvement in performance. However, traditional ML based network traffic detection algorithms rely heavily on subject matter expertise for feature engineering (i.e., feature extraction). Restricting the ability of ML algorithms to identify patterns within the raw network traffic that are imperceptible to a human analyst. Therefore, we propose a novel ML based network traffic detection algorithm using the raw bytes of a network packet as features. Thus, minimizing the reliance on a subject matter expert for feature engineering.
Our prior work [1] demonstrates the feasibility of leveraging the raw bytes of a network packet as features for a 1D Convolutional Neural Network (CNN) and Feed Forward Neural Network (FNN) based network detection algorithm and respectively achieve an accuracy of 98.99% and 98.97%. However, neither the 1D CNN nor FNN foster a human analyst’s interpretability of the output prediction (i.e., malicious or benign). Thus, we propose the use of a Random Forest (RF) for prediction on raw bytes of a network packet. Similar to [2], the feature importance from our RF network classifier, assists the human analyst in interpretation of the output prediction. Furthermore, a structure is constructed, so that each value in a feature vector retains the same meaning (i.e., context) across all samples. Our feature vector is composed of select bytes from the TCP or UDP packet header. 
The UNSW-NB15 dataset is utilized to evaluate our novel RF network traffic classifier. Our Random Forest based raw network traffic detection achieves an accuracy of 98.96%. Additionally, the most important feature is the TCP offset header field. Thus, our method demonstrates the utility of using raw network traffic coupled with the RF reduces the need for a subject matter expert to perform feature engineering. Furthermore, the RF assists a human analyst in interpretation of the output prediction.

[1] M. De Lucia, P. E. Maxwell, N. D. Bastian, A. Swami, B. Jalaian, and N. Leslie, “Machine learning raw network traffic detection,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, Online Only, United States, Apr. 2021, p. 24. doi: 10.1117/12.2586114.
[2] J. Holland, P. Schmitt, N. Feamster, and P. Mittal, “New Directions in Automated Traffic Analysis,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event Republic of Korea, Nov. 2021, pp. 3366–3383. doi: 10.1145/3460120.3484758.

Random Forest based Raw Network Traffic Detection

Polyglot files employ a form of steganography to combine the valid structure of two or more file types. They evade detection by circumventing feature extraction — a process heavily reliant on correct file type identification. The objective of this work is to design high throughput, accurate machine learning / deep learning models for polyglot detection. These models detect polyglots prior to the file type-sensitive feature extraction used in current commercial off-the-shelf endpoint malware detection tools and browser-based file input sanitization systems, thereby protecting against polyglot malware without necessitating a complete overhaul of the existing tools/systems. Furthermore, these models allow us to quantify and catalog the presence of polyglot malware in existing malware repositories, thereby enhancing digital forensics capabilities. This presentation provides a technical overview of this subject area as well as our initial results.

Using Machine Learning to Enhance the Detection of Steganographic Malware

This work explores local congestion control methods for undersea acoustic networks. Congestion control policies are deployed and managed locally at each node acting as a router. Rather than accepting all traffic, intermediate routers can decide not to accept certain packets or to reduce the rate at which they receive packets while resolving the congestion they are experiencing. By not accepting packets, a backlog of packets is built in upstream queues. This backlog will eventually reach the packet sources and, similar to TCP, trigger a traffic rate reduction at the source application. Local packet queue management policies are also responsible for managing queue-space allocations among different traffic classes.These policies are developed based on a financial framework in which each traffic type is associated a commodity that is part of an investment portfolio. Each router, uses its local policy to optimize its portfolio based on the funds it has available, that is, its available data storage space. The space allocation profile is updated periodically via the solution of the celebrated Markowitz Portfolio Selection problem. A price model for packets based on local congestion indicators is used to quantify notions of return, profit and risk. Our framework uses a hybrid policy that combines the space allocation policy with a collection of traffic flow control policies implemented separately per traffic class. The traffic flow control policies are responsible for deciding what flows to pause and restart, and when to requests a reduced packet transmission from a neighboring transmitter. Traffic flow control policies use congestion indicators such as queue occupancy levels per traffic class, packet priority, and sojourn time statistics. These policies are learned via a reinforcement learning framework. The congestion control policy is coupled with a traffic shaping policy implemented at the same node. The traffic shaping policy is responsible for allocating available transmission bandwidth to each traffic type based on traffic priorities and quality-of-service (QoS) requirements. Initially our framework presumes an arbitrary traffic-shaping policy that does not coordinate with the local congestion control policy to decide what packets to prioritize. Numerical tests illustrating the behavior of several components of the proposed method will be discussed.

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Recent advances with artificial intelligence (AI) in industry and academia provide an opportunity to develop intelligent system agents capable of reliably competing against human experts in course-of-action analysis. The team trained agents using deep reinforcement learning in a prototype wargaming framework in naval scenarios. Team members conducted experiments to evaluate the performance of AI threat agents trained using multiple algorithms and parameter configurations in each scenario and identified configurations providing superior results. Trained agents were placed in a desktop wargaming framework to allow play against human opponents, enabling the team to identify areas for improvement. This presentation provides an overview of the wargaming framework, results of experimentation, and next steps.

Improving Course-of-Action Analysis in Naval Wargames with Reinforcement Learning

Control systems for unmanned undersea vehicles (UUVs) are typically implemented using proportional integral derivative (PID) control systems. PID control systems for UUVs are resource-intensive to tune because they require several engineers, marine operators, and ship crew working together to adjust the controller. Furthermore, PID controllers rely on complex dynamic system models that contain assumptions to reduce the computational complexity of the models but can degrade the controller's performance if an environmental condition is encountered that conflicts with an assumption. In this study, a deep reinforcement learning (DRL) control system based on the Deep Deterministic Policy Gradient (DDPG) algorithm is investigated for a UUV control system. The DDPG algorithm is model-free, meaning that a complex dynamic system model is not needed to learn and provide optimal control performance. DRL-based control systems are tuned autonomously, which can greatly reduce the resources needed for tuning. The drawback to deep reinforcement learning is that it is computationally and resource-intensive. This study investigates how mixed floating-point precision with loss scaling can be used to reduce the time and computational resources needed to train the DDPG agent. Numerical case studies are presented.

Mixed Precision Reinforcement Learning for Control Simulation of Unmanned Undersea Vehicles

In this paper we study the multi-agent pursuit-evasion problem, and present an extension of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) deep reinforcement learning algorithm. Previous pursuit-evasion advancements with MADDPG have focused on training capture strategies dependent on the restriction of evader movement with environmental features. We demonstrate a method to train pursuer agents to collaboratively surround and encircle an evader for reliable capture without a strategy rooted in environment entrapment (i.e. cornering). Our method utilizes a novel two-stage, variable-aggression, continuous reward function based on geometrical inscribed circles (incircles), along with a corresponding observation space, with agents operating in an entrapment-disadvantaged environment. Our results show reliable capture of an intelligent, superior evader by three trained pursuers in open space with our encircling strategy.

Robotic Swarm Pursuit-Evasion Capture Strategy using Deep Reinforcement Learning - An Update

In this work, we study a cognitive radio network in which the objective is to maximize the throughput of a secondary user (SU) subject to a constraint on the collision probability with the primary user (PU). For a Markov transmission model of the PU, it has been shown that Age of Information (AoI) plays an important role in this optimization. We have previously developed a throughput-optimal secondary user sensing/transmission policy by solving the linear program for the optimal distribution of the PU state and the AoI of sensed information. This approach requires that the PU idle/transmission dynamics are known, which is not typically the case in practice. Therefore, we implement this approach within a reinforcement learning algorithm by learning the model and then applying the optimization to the learned model. Reinforcement learning algorithms are typically only concerned with maximizing rewards, and therefore do not account for the constraint of avoiding collisions with the PU. To control this cost as well as the reward, we apply a model-based reinforcement learning approach for constrained Markov Decision Processes (CMDP) based on Upper Confidence Reinforcement Learning (UCRL-CMDP), which has proven regret guarantees for the reward and cost. We compare this approach with a model-based (Dyna-Q) and a model-free (Actor-Critic) RL algorithm with a Lagrangian reward function to incorporate the constraint. Simulation results demonstrate that the UCRL-CMDP performs favorably compared with the other two approaches in terms of throughput and computational efficiency.

Online Learning of the Secondary User Policy in a Cognitive Radio Network with Constraints on a Markov Primary User with Unknown Dynamics

Mitigation of range-restricted communication blocks in a distributed swarm is of interest to the Navy, due to the existence of data sharing limitations in all types of sensors and missions carried out in communications-denied environments. If a distributed swarm is affected by a communications block or delay, individual members should still be able to cooperatively identify, evaluate, and act as a team to preserve mission success based on a number of critical parameters guiding the scenario. In our approach, individual members of a heterogeneous swarm in a communication blocked environment construct their own perceived Common Operating Picture (COP). We modified a previously proposed task allocation algorithm to account for incomplete member COPs, and implemented a communications-limiting algorithm in simulation space that emulates a real-world communications block. This block impacts the decision making and asset prioritization of the task allocation algorithm, and our work contributes to a better understanding of task allocation and swarm regrouping in a limited communications environment. Results suggest that compensating for a swarm’s incomplete COP in a limited communications environment achieves a higher likelihood of mission success, and that a quasi-centralized approach for task dissemination ultimately performed more reliably than a fully decentralized approach.

Performative Analysis of Sensor Limitations in Communications Denied Autonomous Swarms

Tactical mission planning for air combat missions is a human-intensive procedure to determine how to coordinate a diverse set of assets. This coordination becomes increasingly difficult due to the physical constraints of the system and the multiple time scales. This paper borrows from safety-critical control to formulate a mathematical description of a class of tactical mission plans. This mathematical description results in a Hamilton-Jacobi-Isaacs (HJI) partial differential equation (PDE). However, due to the curse of dimensionality, numerical solutions to this high dimensional PDE are computationally intensive. To produce a numerical solution, the HJI equation is solved using deep learning. This formulation illustrates the effectiveness and utility of differential games for mission planning through the solution of two games. The first game, the Air3D game, is a collision avoidance scenario between two vehicles. The second is the active target defense (ATD) game. An infinite time solution has been recently found in the controls literature. This paper determines a finite time approximation that is compared with the infinite time solution. In particular, the region of space that the defender protects in a finite time horizon is numerically estimated.

Premium content

A Method to Compress and Transmit Optimal Reorientation Commands to a Spacecraft Using Long Short-Term Memory Neural Network Autoencoders

Downloads

Next from NAML 2022

Using Agent-Based Simulation, Machine Learning, and Distributed Cloud Computing to test how massive, complex networks will react to novel scenarios

Similar lecture

Denoising Signal with Autoencoder

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES