United States

We investigate the use of artificial intelligence-based techniques in learning to play a 2-player, real-time strategy game called Hunting-of-the-Plark. The game is challenging to play for both humans and artificial intelligence-based techniques because neither player can directly observe its opponent&#39;s moves at any time during the game, and has to rely on post hoc analyses of game moves and outcomes to infer the opponent&#39;s game-playing strategies and learn to play effectively against them. We analyze the performance of different deep reinforcement algorithms to train software agents that can play the game. We also investigate different techniques that can be used by players to improve their play including self-play, where each player first builds a model of the opponent from the history of opponent&#39;s perceived actions and then uses it to refine their own game-playing strategies, and, dynamic strategy evolution, where the players strategically adapt their own strategies to gain advantage over the opponent. Our experimental results show that the actor-critic learning algorithm is the most efficient in terms of training time and rewards received by both players. Our experiments also showed that while one of the players starts with a disadvantage due to the inherent asymmetric nature of the game rules, it can increase its score considerably by using improved techniques like self-play and dynamic strategy evolution.

NAML 2022

Learning to Play Simulated War-Games using AI: A Case Study of Deep Reinforcement Learning for Hunting of the Plark Game

game playing ai

self-play

reinforcement learning

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

IJCNLP-AACL 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

We investigate the use of artificial intelligence-based techniques in learning to play a 2-player, real-time strategy game called Hunting-of-the-Plark. The game is challenging to play for both humans and artificial intelligence-based techniques because neither player can directly observe its opponent's moves at any time during the game, and has to rely on post hoc analyses of game moves and outcomes to infer the opponent's game-playing strategies and learn to play effectively against them. We analyze the performance of different deep reinforcement algorithms to train software agents that can play the game. We also investigate different techniques that can be used by players to improve their play including self-play, where each player first builds a model of the opponent from the history of opponent's perceived actions and then uses it to refine their own game-playing strategies, and, dynamic strategy evolution, where the players strategically adapt their own strategies to gain advantage over the opponent. Our experimental results show that the actor-critic learning algorithm is the most efficient in terms of training time and rewards received by both players. Our experiments also showed that while one of the players starts with a disadvantage due to the inherent asymmetric nature of the game rules, it can increase its score considerably by using improved techniques like self-play and dynamic strategy evolution.

poster

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Communicable diseases can compromise the abilities of individual warfighters and the command-and-control and operational capabilities of military environments. Thus, there is a critical need within the military to be able to evaluate an individual’s risk of becoming ill before being sent on a physically difficult operation and to identify outbreaks of communicable diseases early in order to be able to contain, mitigate, and track its spread. This need is especially important when soldiers and personnel have to be kept in close quarters on a ship, in barracks or in other close quarter environments. Wearable health monitoring devices provide the capability to record a variety of real time physiological time-series data including, but not limited to: respiratory rate, respiratory rate variability, heart rate, steps, calories burned and state of mind. Theoretically these physiological measures may be used to predict the health status of individuals over the course of an acute respiratory infection (ARI) from healthy, infected, pre-symptomatic, symptomatic, and recovered. To test this hypothesis, we are using data collected from Spire HealthTag devices that were deployed during a community cohort contact tracing study conducted by the University of Maryland during the 2018-2019 influenza season. The objective of this study is to use the wearable data to predict the onset of an ARI due to viruses such as parainfluenza, coronavirus, adenovirus, influenza, and rhinovirus. With the future goal of pre-symptomatic illness detection in mind, for now the goal is to be able to classify between sick and healthy days. A random forest classification model and its results will be presented. 

Predicting of Health Status from Wearable Physiological Monitoring Data project

Engagement complexity, along with technological advances, is growing faster than shipboard technical upgrades. The conundrum is that because of the challenges of peer-to-peer engagements, upgrades to more powerful computers is needed, but will likely be unavailable in the next several years. Lighter processor power solutions need to be researched to address this issue. Because of the increase in Engagement complexity, users’ needs solutions that fit their current IT while in the operational theater to determine practical recommendations in achieving mission success. This means that a management aids and decision aids need to effectively run at lower computer processing power than needed for complex analytics in order to provide decision support. This research investigated the use of meta-models that can be ported to current IT level processing machines associated with deployed environments. The research focused on approaches that allowed for simulations of advanced, complex engagement issues using meta-models that can be processed to provide the highest likelihood of a successful recommendation that includes platforms following mixed sequences of actions to achieve end goals. Meta-models are the synthetic results of a multiple simulation. A meta-model consists of: (1) the input variables associated with the capabilities of a platform’s weapons, electronic warfare, communication links, and sensors (2) the variables numerical values and (3) statistically analyzed output. When the sequence of events is decomposed into statistically independent activities following a conditional probability construct, the meta-models can be rearranged based on the desired mission outcome. The arrangement of meta-models, related inputs and outputs, constitute a statistically-based sequence of events to obtain desired goals for analysis. In this manner, meta-models, i.e., input variables to output success statistics, are light weight data structures that can be queried and processed within a laptop or equivalent IT system. The research implemented meta-models to support a decision aid software application based on game theory for blue vs red wargaming analysis. It ran Monte Carlo simulations to support recommendations with statistical results. This decision aid is used on laptops at the Naval Postgraduate School for AI/ML experimentation. The tool using meta-models to help resolve engagement complexity using government restricted laptops, demonstrating potential value in the approach. This research shows evidenced that meta-models can be used for “real time” decision aid recommendations when limited to deployed IT environments, but need to be customized to one set of goals. Additionally, recommendations can be accompanied by Monte Carlo-based, statistical outcomes that represent results of complex engagement models running on processing-hindered government laptops.

Deploying Meta-Models to Drive “Real-Time,” On-Site Course-of-Action Recommendations

Naval War College (NWC) and other leadership facilities provide extensive training on “best” ways to use technology during a single battle and over a prolonged campaign. AI has now added another level of complexity, both in testing and developing decision aids. Test and Evaluation, Verification, Validation (TEVV) and Live Virtual Construct (LVC) operational facilities are charged with testing new technologies to a given set of requirements to ensure performance. The challenge and difference is noted when teaching and then operationally testing autonomous AI-enabled systems. Algorithms are very sensitive to the training data used and lack of alignment to what is taught during a war game, as represented in the training data, vs what is tested at the TEVV/LVC facility can cause AI performance failures during deployment. Additionally, are AI-based decision aids in alignment? Warfighters are being trained to have expectations of performance, but will the AI system be limited in its response? Will the decision aid provide adequate recommendations and has the AI-based system been adequately testing in terms of expectations? The solution offered in this paper is a schema structure that is automated and visually friendly in the approach used to represents an activity ontology, knowledge representation describing cause and effect tactical sequencing. The paper describes the schema used for knowledge representation as well as how it will capture past war game knowledge played by war gaming institutes. Examples are offered to illustrate the approach and convey how this type of ontology developed from war gaming can ensure TEVV/LVC test scripts are designed to operationally “stress test” autonomous AI-enabled systems following these equivalent tactical sequences.

Semantic methods for activity ontology for testing AI and developing decision aids

The Digital Aviation Readiness Technology Engine (DARTE) has proven highly successful in monthly predictions for readiness metrics, including the number of Mission Capable (MC) F/A-18 aircraft at the squadron level across the Naval Aviation Enterprise. We show that the DARTE method can be applied on a more granular timescale by using the method to predict squadron-level F/A-18 readiness on a daily basis, up to one week in advance. We quickly train multiple models that have different architectures and provide the model outputs to tree-based ensemble meta-learners, creating an ensemble model that can accurately predict the number of MC aircraft that a Naval F/A-18 squadron will have each day for the next week. Additionally, we interrogate the predictions with explainable AI (XAI) to provide decision makers and subject matter experts with information that they can use to derive actionable insights. Finally, these predictions and insights are displayed via dashboards.


Readiness Predictions for the MH-60 SeaHawk with the Digital Aviation Readiness Technology Engine

PMW/ISEA teams are dealing with thousands of trouble tickets daily, by prioritizing the tickets as they are submitted based upon how long the ticket is estimated to resolve will enable the help desk and the ISEA groups to make quicker decisions and prioritize their resources. Delta Duration Model was built with an ensemble structure to predict the time it will take for a ticket to resolve as the ticket ages. The Delta Duration Model uses up to 4 different AI/ML algorithms working together to determine duration of ticket upon submission. As the ticket ages, additional models are used to more accurately predict a resolution date. A model maintenance pipeline was created out of a need to track various accuracy metrics on the deployed models for the Delta Duration Model. Each week when new tickets have been submitted and predicted upon the maintenance pipeline tracks accuracy for each set of models within the pipeline. This monitors changes in the model’s accuracy over time and allows transparency when model drift has occurred.

Machine Learning to Predict and Monitor Trouble Ticket Duration

Research shows that genders are affected differently by length and word choices in job descriptions. Addition, research exhibits the existence of a gender gap where women tend to not apply unless they are 100% qualified for the job. Our hypothesis was that women do not know what the job description specifically states if it is filled with jargon, therefore do not feel like they are 100% qualified so they tend to be less likely to apply. To investigate this potential unconscious gender bias further, an experiment was performed that surveyed both Navy and non-Navy people to view three job descriptions in their respective STEM field. The survey asked them if they were interested in the job and if they would send this job to someone else. Additionally, they were asked if they would share the ad.

How Long, Jargon-filled Job Descriptions Affect Potential Candidate Interest Level

The U.S. maritime operations, particularly in the littorals, have been contested and dangerous. It is imperative to apply advanced analytics to help operational or tactical units execute distributed maritime operations (DMO) and expeditionary advanced base operations (EABO). The DMO and EABO concepts require flexible configurations and combinations of capabilities and tactics across multiple domains to win in complex contested environments. The DMO and EABO concepts also require to use advanced detection and counter-detection technologies.

Distributed and collaborative computing and networking have been associated with peer-to-peer models. The researchers apply the peer-to-peer and self-organizing collaborative learning agents (CLAs) to meet the requirements of DMO and EABO. As an example of EABO, the researchers show a use case for distributed Marine transportation units to handle transportation movement requests (TMRs). The researchers show how to data-mine historical data, discover capability and behavior patterns, then apply them to design new peer groups to balance the load and maintain much lower signatures. 

The researchers map the need of lowering operation signatures to a load balancing problem and applies lexical link analysis (LLA) for CLAs. The researchers further apply the principle of quantum entanglement and superposition into a framework of LLA quantum intelligence game (LLAQIG). LLAQIG helps a peer simultaneously achieve the Nash Equilibrium for itself and the optimal total social welfare for its peer-to-peer network. 

The LLAQIG is used to find the intrinsic value of a unit, (i.e., a unit possesses a requested capability and has the potential to become a high-value unit). The LLA groups and LLQIG quantum effect impact factor help a peer-to-peer network self-organize its units and make them appear more randomly connected, which therefore balances the load, lowers signatures, and avoids detection. The DoD operational or tactical units can potentially leverage the results for shaping flexible C2, organizational structures, and modernization.




Self-organizing and Load Balancing with Quantum Effect for Peer-to-Peer Collaborative Learning Agents and Flexible Command and Control and Organizational Structures

There is a pressing need to exploit fleeting opportunities continually--real time anticipatory ecosystem entropy offers a lens for these kinds of exploitations. We want to measure ecosystem entropy continually to make better, faster data-driven decisions which advantage one player over another. Combining entropy measures with process models, we seek to understand how dissipative structures interact with one another. Continual sliding window entropy measurements identify exploitation opportunities when a player’s ecosystem is especially chaotic or orderly. Process models provide turnkey simulation parameters which predict ecosystem next-states and identify exploitation actions. This concept may be applied to space, social media, critical infrastructure, urban, supply chain, macro-economic, natural, and geo-political ecosystems, as well. Efforts to date have neither examined ecosystems in a temporal process mining context using categorical data event logs nor analyzed interactions between ecosystem players in order to anticipate outcomes and improve decision making. A highly adaptable, domain agnostic information maneuver capability is possible. It will be possible to drive entropy purposefully by anticipating ecosystem changes or inflection points and make decisions which favor one party over the other.

Measuring and Exploiting Entropy Shifts in Time Series Ecosystems

NAVPLAN 2021 calls for distributed maritime operations (DMO), littoral operations in a contested environment (LOCE), and expeditionary advanced base operations (EABO). This project informs DMO, EABO, and LOCE objectives with technical designs for hardware, software, processing, and artificial intelligence/machine learning (AI/ML) at the tactical edge. The selected micro-service mesh architecture may additionally support naval tactical grid (NTG) objectives. Chassis and servers were optimized for tactical cloud far edge deployment with high performance in a low space, weight, power, and cost configuration (SWaP-C). AI/ML for the tactical edge operating environment was evaluated through a scenario for forward deployed sensor assessment for systems maintenance, situational awareness, and far edge decision support.

AI/ML in the Tactical Cloud for Forward Deployed Decision Support

Military tactical decision-making requires the ability to determine efficient optimized actions across multiple entities in complex environments, potentially with limited or denied communications. In recent years, researchers have applied various approaches to wargaming problems, including game theory, optimization, and more recently, deep reinforcement learning. Most deep reinforcement learning algorithms treat agents as decentralized with rewards that do not require cooperation across agents. In this research, we model a blue versus red war game scenario in an AI-gym environment. We add a stochastic game algorithm (COCO-Q) to a decentralized reinforcement learning algorithm (PRIMAL) to incorporate cooperation as a part of a goal-oriented model for a blue versus red simulation. 

PRIMAL is a decentralized pathfinding model used to implement pathways for warehouse drones. Each drone has a drop-off location, and it makes pathway decisions on its moves within a certain radius. This approach will provide a locally optimal pathway for each individual drone. COCO-Q is a stochastic game model with side payments. The side payments provide incentive for agents to work together for a common goal. These incentives are included when calculating the COCO value. The advantage of COCO-Q is that it considers both competitive and cooperative objectives, which allow the model to be shaped as a team advantage solution versus individual agents playing the same game independently. In this work, we introduce an extension of COCO-Q to a blue/red wargame that looks for a cooperative pathway for the given scenario and compare solutions to PRIMAL’s individual pathway solution.

Premium content

Learning to Play Simulated War-Games using AI: A Case Study of Deep Reinforcement Learning for Hunting of the Plark Game

Downloads

Next from NAML 2022

Predicting of Health Status from Wearable Physiological Monitoring Data project

Similar lecture

Transferable Dialogue Systems and User Simulators

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES