United States

We demonstrate an innovative framework (CoEvSoarRL) that lever-ages machine learning algorithms to optimize and simulate a resilient and agile complex enterprise to improve the readiness and sustainment, as well as reduce the operational risk. The CoEv-SoarRL is an asymmetrical wargame simulation that leverages reinforcement learning and coevolutionary algorithms to improve total value of the enterprise. We address two of the key challenges: (1) the need to leverage holistic artificial intelligence to learn, optimization, and wargame (LAILOW) to simulate and improve the enterprise readiness; (2) the uncertainty and lack of data which require large-scale systematic what-if scenarios and analysis of alternatives to simulate and generate potential new data and unknown situations. Our CoEvSoarRL learns a model of an enterprise environment from historical data with Soar reinforcement learning. Then the Soar model is used to evaluate new decisions and operating conditions. We simulate the enterprise vulnerability (risk) and evolve new and more difficult operating conditions (tests);meanwhile we also coevolve better enterprise decision (solutions) to counter the tests. We apply lexical link analysis to discover probabilistic rules and patterns from historical data and use them for evaluating the feasibility of new data generation. We highlight proof-of-concept results from a US Marine Corps maintenance and supply chain data set, a cyber defense data set, and a force strategic lay down data set.
 



NAML 2022

Simulating a Complex Enterprise Using an Asymmetrical Wargame Simulation with Soar Reinforcement Learning, Coevolutionary Algorithms, and Lexical Link Analysis

cyber defense

strategic lay down

enterprise vulnerability

lexical link analysis

coevolutionary algorithms

soar reinforcement learning

asymmetrical wargame simulation

artificial general intelligence

synthetic data generation

risk

resilience

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Many — if not most — applications of artificial intelligence (AI) involve accurately tracking and identifying objects in the physical world. For example, shipboard systems could autonomously recognize and track nearby vessels, self-driving vehicles could identify and track cars and pedestrians, and search-and-rescue systems could automatically locate and track the lost or wounded. However, current tracking techniques have significant weaknesses that limit their applicability and robustness. Most machine learning systems require large-scale training datasets that are difficult or impossible to collect in many applications. These systems are also easily confused by novel scenarios and are vulnerable to adversarial attacks. To overcome these limitations, we draw inspiration from biological visual systems, which are remarkably robust in their ability to track and identify objects, even those that have not been seen before. In this talk, we will describe our current progress building biologically-inspired tracking systems. Our ultimate goal is to lay the groundwork to develop potentially revolutionary visual systems that are more robust and generalizable than today’s techniques.

This work was sponsored by a grant from Naval Surface Warfare Center Crane Division.

Building biologically-inspired object tracking systems

In the current rapidly developing environment of both technologies and techniques in electronic warfare (EW), it is essential for modeling and simulation (M&S) frameworks to keep up with the pace. When it comes to creating an accurate M&S environment, data is king. Collecting the necessary data through flight tests is not feasible due to the cost, availability of assets, and need for rapid turnaround. 
There are many tools that allow for quick creation of RF test data for the M&S environment, but the difference between testing on lab generated data and the real-world data can be the difference between life and death in the field. This is especially true for future cognitive EW systems which, as with all machine learning (ML) systems, will perform worse on noisy real-world data. The ML technique called Style Transfer is a powerful tool that can be leveraged in the EW M&S domain to improve the testing and evaluation (T&E) of the different systems involved in the kill chain.
Style transfer was developed as a tool for capturing the art styles of different prominent artists and transferring that style to other paintings and photographs. This model can show you what the Mona Lisa would look like if painted by Picasso, van Gogh, or Monet. This same technique can be used to modify lab generated, digitized radio frequency (RF) data to look more realistic. The styles that can be captured in this domain include the emitters, environmental noise, and receivers. Multiple styles can be applied to a single signal, which would allow a “clean” lab signal to be modified to look as if it were generated from a specific emitter, traveled through a noisy environment, and was received by a specific receiver. This technique can be applied to any digitized RF data, so it can be used to improve existing data sets. Also, with the rapid improvement of hardware, a model using this technique may be able to act as a real-time filter on generated data in M&S frameworks.
Style transfer provides a way to improve the realism of lab-generated RF data to improve the confidence in the results of T&E events. The technique is useful not only for current systems, but will continue to be relevant as new systems enter the battlefield, including cognitive systems.

Generating Realistic RF Test Data Through Style Transfer

Deep learning algorithms are powerful and present an opportunity to achieve high performance in computer vision applications. Unfortunately, these algorithms also require large quantities of data that can be difficult to assemble, particularly in the case of Defense applications. Obtaining conventional imagery that is relevant to defense applications is expensive and time-consuming. Synthetic data, therefore, has emerged not only as a viable option, but as a necessary component of defense algorithm development strategy. The U.S. Army Futures Command DEVCOM C5ISR Center, Research and Technology Integration Directorate, Modeling and Simulation Division is developing a variety of synthetic data generation capabilities. One of these approaches utilizes fully synthetic imagery generated by physics engines with high-fidelity inputs and models. In particular, we have created an IR image generation pipeline using the UnrealEngine (UE4) and an IR plugin. In order to evaluate the suitability of this synthetic imagery for algorithm training, we used it to train a version of YOLO to detect and classify real targets in field-collected imagery. This presentation outlines the image generation pipeline, the performance of various iterations of the UE4 imagery when evaluated against real data, and future steps for synthetic imagery development.

Evaluation of IR Imagery Generated by UnrealEngine for Algorithm Training

How best to leverage small datasets for machine learning tasks is an active area of research. Synthetic data has the potential to alleviate object rarity and long-tail distributions, provided the synthetic data introduces more signal than noise into the system. The IQT Labs Synthesizing Robustness project aimed to explore whether domain adaptation strategies (attempting to make the synthetic data look more “realistic”) can improve the efficacy of the synthetic data when it comes to localizing rare objects. In this project we explored generative adversarial network (GAN) techniques for domain adaptation on the publicly available RarePlanes satellite imagery dataset. 

Over a number of experiments we quantified the detection performance for rare aircraft as a function of training size per object class. We also note that strategically augmenting the real dataset is nearly as effective as adding synthetic data in the quest to improve the detection or rare object classes. Intriguingly, in certain cases, confusion between similar-appearing object classes is a more important factor than training dataset size in determining detection performance. Finally, we share some of the lessons learned from this project; for example: fully extracting the utility of synthetic data is a tricky and nuanced process, requiring effort and creativity.

Synthesizing Robustness: On The Nuances of Extracting Utility from Synthetic Data

Scalable synthetic sensor simulation is an important yet challenging open problem for ROV simulators to bridge the "reality gap" and "domain randomization gap" for machine learning and training. Utilizing machine vision techniques Blue Ring Imaging and SeeByte are constructing a pipeline for creation of synthetic underwater assets from camera and sonar data which is photoreal, randomized, and reflect the diversity of underwater environments for ROV CONOPs. In this talk Casey Sapp, CEO of Blue Ring Imaging, and Vincente Nguyen of SeeByte will provide a high level overview of synthetic data generation of sonar data using GANs (Generative Adversarial Networks) for ROV simulation and reinforcement learning.

Synthetic Data Generation for UUV Reinforcement Learning

The Defence Science and Technology Group (DST Group) have been assisting Navy to develop a new Intelligence, Surveillance and Reconnaissance (ISR) and targeting capability on-board ships. The research began as an attempt to automatically classify ships with Convolutional Neural Network (CNN) technology from images provided by Uninhabited Aerial Vehicle (UAV) camera streams. This effort expanded into an investigation surrounding how to deploy Machine Learning (ML) system(s) that would eventually be integrated into existing Defence combat and mission systems. After establishing an Artificial Intelligence (AI) eco-system the focus shifted towards using ML that would reduce the communications bandwidth required to share visual information across tactical networks. This naturally morphed into establishing robust and reliable models that operate within a defence context. These models would need to be resilient to adversarial attacks and present information to operators in language that was intuitive to a human and machine team (using eXplainable AI). DST Group has collaborated with the University of Adelaide and Defence to create an ensemble of models to dynamically recognise ships at sea in near real-time. At present this application includes: ABCNet; Deepsort; Detectron2; Detr; ProtoPNet; StDan; Recurrent Neural Networks (RNNs) to construct an attention network; a sophisticated Transformer network; and additional AI techniques to support interoperability. At present this research is an evolving concept demonstrator, aimed at generating visual tracks in near real-time together with an intuitive text description about each maritime asset detected within the scene. This talk will discuss the journey and demonstrate the progress made.

AI-Enabled Tactical Information Exploitation

Task scheduling on multiple channels is a mature field of research and is a critical component of modern multifunction radars. Management of the radar timeline can significantly impact the overall performance by directly affecting the ability to detect and track targets. Determining the optimal schedule given the channel availabilities and a set of search/track tasks can require a prohibitive amount of computation, especially when the radar scheduler needs to operate in real time. Suboptimal solutions can be generated with fast algorithms, including traditional methods such as Monte Carlo tree search; however, if the number of tasks is large, the search space may be too voluminous and the required performance may not be met. To enable fast, high-performance schedulers, machine learning can be employed. If there is statistical similarity among the different problems the scheduler solves, then the scheduler can learn from previous problems and operate on novel ones using knowledge not exploited by a classic scheduler. Given that optimal schedulers such as Branch-and-Bound are available, a sensible approach is to use supervised learning - saved scheduling problems can be solved offline and used to generate labeled data. This work will demonstrate how supervised learning can be used to create policy neural networks that outperform various traditional schedulers (even without integration into search algorithms) while still maintaining fast execution for real-time implementation. The scheduling problem will be cast into the form of a Markov decision process. A specific focus will be how the tasks are encoded into the observation space; it will be shown that intelligent design of this "feature extraction" can allow even policies with modest degrees-of-freedom to perform well. Different reward functions will also be considered. Implementation of the framework and algorithms discussed are available as a Python package, "task-scheduling" (https://spork.nre.navy.mil/nrl-radar/CRM/task-scheduling). Access to the source code is restricted to U.S. DoD members with accounts on NRE Fusion.

Markov Decision Process Design for Supervised Learning of Task Schedulers

The space domain is becoming ever more congested and complex. The number of satellite launches is growing exponentially; commercialisation of various space-based imaging services and the number of space-capable nations continues to rise; and adversaries increasingly operate in grey-zones, attempting to evade attribution of their activities. Simultaneously, the variety and quantity of available data is growing, providing a complex challenge for space analysists trying to build an accurate picture. Machine Learning (ML) techniques can offer assistance to analysts in managing these burdens: ML can exploit excess data to provide timesaving, wider-reaching, or novel capability, and release human effort to focus on complex or higher-priority issues.

This presentation demonstrates the feasibility and effectiveness of machine learning models for satellite characterisation. Dstl has developed models to predict the operational status and the bus (structural frame type) of satellites using ground-based time-series observations of reflected sun-light or reflected radar. We found classical ML models (e.g. Random Forests) performed as well as Deep Learning approaches: our LSTM (long short-term memory) models struggled to retain their high performance when generalising to non-iid (independent identically distributed), out-of-distribution samples. The full range of characterisation tasks is significantly broader than those investigated here and considerable scope remains for further low-TRL research.

Data was a critical and limiting factor throughout this work. Based on the sources available, Wide Field Lens Array telescopes offered the best data for LEO characterisation while traditional commercial world-wide networks of Newtonian telescopes offered the best data for GEO characterisation. Open-source labels were not always available to high enough accuracy, limiting model performance in some cases. Furthermore, open-source labels are not openly archived, complicating the use of historic observations.

Machine Learning for Satellite Characterisation

New operating concepts are focused on a distributed, smaller, lighter, and more agile tactical force in the littorals and the maritime. This requires a change in medical evacuation strategy. In future operating environments, aviation-based evacuation may be infeasible due to hostile conditions or proximity to land. This presentation discusses previous research on modeling UAV routing under adversary threat and its transition to current research on the routing of Unmanned Surface Vessels (USuV) under adversary threat in support of casualty evacuation. Computational experiments are designed using a grid-world environment with randomized demand (location of casualties) and adversary movements to explore different routing strategies through the network. Initial findings are discussed, including the limitations of explored models and considerations for the development of the grid-world.

Modeling Medical Evacuation USuV Routing Under Stochastic Demands and Adversary Movements Using Online Optimization

Current naval operations require sailors to make time-critical and high-stakes decisions based on uncertain situational knowledge in dynamic operational environments. Recent tragic events have resulted in unnecessary casualties and represent the decision complexity involved in naval operations and specifically highlight challenges within the OODA loop (Observe, Orient, Decide, and Assess). Kill chain decisions involving the use of weapon systems are a particularly stressing category within the OODA loop—with unexpected threats that are difficult to identify with certainty, shortened decision reaction times, and lethal consequences. An effective kill chain requires the proper setup and employment of shipboard sensors, the identification and classification of unknown contacts, the analysis of contact intentions based on kinematics and intelligence, awareness of the environment, and decision analysis and resource selection. This project is exploring the use of automation and artificial intelligence (AI) to improve naval kill chain decisions. The team is studying naval kill chain functions and developing specific evaluation criteria for each function for determining the efficacy of specific AI methods. The team is identifying AI methods and applying evaluation criteria to map specific AI methods to specific kill chain functions.

Premium content

Simulating a Complex Enterprise Using an Asymmetrical Wargame Simulation with Soar Reinforcement Learning, Coevolutionary Algorithms, and Lexical Link Analysis

Downloads

Next from NAML 2022

Building biologically-inspired object tracking systems

Similar lecture

Leverage Artificial Intelligence to Learn, Optimize, and Win (LAILOW) for the Marine Maintenance and Supply Complex System

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES