United States

For general-sum, n-player, strategic games with transferable utility, the COCO value provides a computable method to both 1) quantify the strategic value of a player; and 2) make cooperation rational through side payments. We give a simple formula to compute the COCO value in normal-form games. Next, we provide two methods to generalize the COCO values to stochastic (or Markov) games, and show that one of them may be computed using generalized Q-learning algorithms. Finally, an empirical validation is performed on stochastic grid-games with three or more players.

NAML 2023

N-Player COCO-Q Reinforcement Learning

cooperation

artificial intelligence

game theory

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Cyclostationary features such as high-order moments and cumulants are often applied in detecting and classifying digitally modulated signals. However, these predictor variables may not be mutually uncorrelated, raising the concern that potential multicollinearity may lead to redundant information and significantly affect quality prediction. This work explores machine learning and statistical methods to mitigate multicollinearity issues. We first perform multicollinearity diagnostics by evaluating variance inflation factors and eigendecomposition of the correlation matrix of the predictor variables. To combat multicollinearity, we apply feature transformation using the principal component analysis (PCA), a traditional dimensionality reduction technique that has been well-studied and is suitable for small feature sets. Our empirical results show that the chosen diagnostic tools effectively detected cyclostationary multicollinearity. Future work will include assessing PCA against existing methods for selection of moments and cumulants.

Multicollinearity Diagnostics on Cyclostationary Features

Navy divers are exposed to health risks while operating in marine waters contaminated with a wide variety of bacterial, viral, and protozoan pathogens. Timely and accurate analysis of measured bacterial levels to predict illness is rarely possible due to the costs involved. Available data integrated through the means of knowledge graphs can be transformed into knowledge that can help divers avoid exposure to biological hazards. A knowledge graph (also known as an ontology) is a graph-based data model that can express and integrate structural knowledge from multiple domains in the form of (head-entity, relationship, tail-entity) triples, where the entities represent nodes in the graph and the relationships represent edges that connect entities. Knowledge graphs can encode complex rules which are useful in defining and reasoning about the semantics of the entities and can lead to new knowledge and discoveries. For example, given the following two triples in a knowledge graph (Pseudomonas aeruginosa, causes, acute respiratory infection), (acute respiratory infection, isa, communicable disease), one can infer that (Pseudomonas aeruginosa, causes, communicable disease). While knowledge graphs can be useful in inferring new knowledge, extracting triples from large-scale data manually can be a tedious task and the resulting graphs can suffer from incompleteness. Currently, there are many knowledge graphs or ontologies readily available, including a water quality ontology (DOCE), an epidemiology ontology (EPO), an infectious disease ontology (IDO) and an exposure ontology (ExO), among others. However, there is no knowledge graph that links water quality to health risks posed by water activities such as diving. To fill in this gap, we have focused on the construction of a knowledge graph that links water quality to health risks by leveraging existing ontologies and large-scale data available in PubMed. We constructed a knowledge graph based on semantic triples automatically extracted from the Semantic MEDLINE database (SemMedDB), a PubMed-scale repository of biomedical triples. The entities in the SemMedDB triples correspond to Metathesaurus concepts in the Unified Medical Language System (UMLS), specifically concept unique identifiers (CUIs), while the relations are drawn from the UMLS semantic network. Using a set of manually identified CUIs that we found to be most relevant to water quality and health risks, we constructed a graph of approximately 75,000 semantic triples which contain at least one entity from the manually identified set. We subsequently used the constructed knowledge graph to study machine learning approaches for knowledge graph completion (KGC), to help identify new valid triples or new relationships between existing entities. Specifically, we experimented with the KG-BERT approach for graph completion by employing pre-trained BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robustly Optimized BERT Pre-training Approach) models and also equivalent models fine-tuned on a collection of water quality and health-risks abstracts retrieved from the Web of Science. Experimental results showed that the KG-BERT approach with BERT/RoBERTa models fine-tuned on the domain-specific corpus improves the performance of KG-BERT with pre-trained models. Furthermore, the best results overall were obtained with a fine-tuned RoBERTa model, which has an accuracy of 88.61% on the valid triple classification task.

Leveraging Existing Literature and Machine Learning to Build a Knowledge Graph Focused on Water Quality and Health Risks

Recent operational challenges have prompted a renewed interest in mission-engineering tools that can advance decision-making and innovation in defense. These tools have the ability to simulate high-fidelity combat engagements to analyze operational concepts, platforms, systems, and capabilities in multiple domains to develop optimal solutions to the intricate and difficult issues within the field of mission engineering. One such tool is the Advanced Framework for Simulation, Integration, and Modeling (AFSIM). AFSIM is a standard combat simulation tool that includes a variety of applications and plugins that can be used for systems engineering as well as engagement, mission, or campaign-level simulation. Due to software challenges, using AFSIM in combination with Reinforcement Learning (RL) poses particular challenges that must be overcome before RL will gain widespread use in the AFSIM community. This study proposes a novel way to utilize synthetic data created through AFSIM by using offline, or data-driven, RL.

Data-Driven Reinforcement Learning for Mission Engineering and Combat Simulation

We constructed a scenario using a previously developed framework to educate junior developers on the importance of incorporating statistical analysis in the development of autonomy behaviors. We added rudimentary perception and obstacle avoidance to the previously developed framework, adding variable parameters for the range the autonomous agent could perceive its environment, the braking response curve, and the maximum speed through which the agent could traverse its enivronment. By simulating the agent avoiding obstacles through an environment it has no a priori knowledge of and analyzing the frequencies with which the agent either collided or nearly collided with obstacles, we were able to identify patterns in the design parameters. For example, One-way ANOVA testing showed a statistically significant difference in collision avoidance success rate between the different braking curves across identical sensor ranges and maximum speeds (p=7.88e-24). Additionally, thresholds for agent survivability were identifiable by analyzing the sensor ranges and maximum speeds allowed in simulations. When combined with other core development best practices such as unit testing and version control, we believe we can better define intended behaviors and consequently improve the reliability of the behavior by conducting relevant statistical analysis on the design parameters of behaviors.

Learning How To Learn: Emphasizing Iterative Design in Good Autonomous Framework Development

The objective of this research project is to improve the warfighter’s maritime domain situational awareness by designing and developing a prototype payload for deployment on an unmanned or dismounted operator’s platform in support of various missions. The payload would have the ability to detect, track, and identify Signals of Interest (SOIs) by utilizing Convolutional Neural Networks (CNN) and Digital Signal Processing (DSP) techniques. The 1st year of this effort entailed developing a benchtop prototype capable of performing real-time SOI classification. The components of the prototype consisted of a Software Defined Radio (SDR) to stream the raw digitized In-phase and Quadrature (I/Q) components of the signal and a System on a Chip (SoC) development kit to perform real-time DSP and SOI classification via the CNN. The SOIs were classified based on modulation type, wireless protocol, and emitter type. Two network architectures were developed using a standard image approach and a complex value I/Q approach to evaluate the benefit of I/Q input format. DSP techniques such as power normalization, down sampling, up/down converting, and time averaging were implemented to elicit better generalization and accuracy of the trained CNN. This research project effort has been selected to continue into a second year. Future work will focus on Low Probability of Intercept (LPI) signal as the SOI and applying machine learning to both detection and classification of low and negative Signal to Noise Ratio (SNR) signals.

Spectrum Awareness Persistence Payload

Synthetic data generation is crucial to performative and predictive autonomy technology research. The efficient generation of robust synthetic data requires 3D Computer-Aided Design (CAD) models that correctly represent the target but sufficiently small as to promote agile computation. "Watertight" CAD models - models with no gaps or surface flaws - are necessary for computer simulations, generation of synthetic datasets for machine learning, and accurate synthetic data analysis. This is joint work with Dr. Igor Ternovskiy (AFRL/RYAT) and Ms. Elizabeth Sudkamp (AFRL/RYAP).

A Facet to Watertight CAD Converter

Naval Aviation Support Equipment (SE) PMAs lack an automated methodology to produce a failure mode and system identification. An automated capability would facilitate a consistent, repeatable, and time savings process across the sixteen SE PMA’s portfolio of equipment. The NAVAIR methodology of recording equipment maintenance entails completing a Maintenance Action Form (MAF). Deficiencies in the MAF process allow maintainers to enter overly generic systems and root causes for the maintenance action. However, the MAF includes underutilized fleet narratives that describe the equipment discrepancy and the corrective maintenance action. This research investigates whether an unsupervised machine-learning algorithm can process fleet text narratives and identify SE Failure Modes and system affected. 

The research proceeded by employing Natural Language Processing (NLP) to the MAF unstructured text data. The researchers acquired data comprising of several pieces of equipment (contained in separate files) of NAVAIR equipment that contains a variety of failure modes and systems. The unstructured maintenance text was preprocessed utilizing lemmatization, lowercasing characters, removing punctuations and stop words, replacing acronyms/abbreviations, and stripping white space. The researchers plan to apply a Latent Dirichlet Allocation (LDA) topic algorithm to the processed unstructured text iterated across a sequence of topic quantity hyper-parameters. A harmonic mean function can be used to identify the optimal number of hyper-parameter topics. The researchers will apply the LDA algorithm with the optimal number of hyper-parameter topics against the dataset. 

The objective of this project consists of determining whether NLP techniques can identify SE Failure Modes and System affected using unstructured text fields found in the MAF. The project is currently in progress and should result in an automated solution to extract accurate labels from the text data. This solution will allow the stakeholders to identify root causes of Support Equipment failure and apply the necessary resources to rectify the failure mode quickly and more efficiently.

Unsupervised Support Equipment (SE) Failure Mode Identification

We demonstrate an end-to-end neural network approach for acoustic source detection and classification on a linear sensor array. We show that the neural network learns optimal spatial and frequency filters as it performs beamforming-based signal enhancement and finite impulse response (FIR) filtering to isolate source-specific energy signatures. The trained network shows robust performance as a function of signal-to-noise ratio.

An End-to-end Neural Network for Acoustic Source Detection and Classification on a Linear Sensor Array

This study investigates the ability of deep learning methods to identify salience of radio frequency (RF) data. We primarily study this through sensitivity and are Unfortunately, RF data exists in the presence of noise and nuisance signals. We define noise as any perturbation on the data which obscures signal, while nuisance is a signal within the data we are not interested in.

Exploring Salience Metrics of Radio Frequency Data

Current and future methods depend on more data and computational power to ensure mine counter measure(MCM) maritime dominance. Using Synthetic Aperture Sonar(SAS) Side-Scan data, we show how we are tackling the challenge of limited training data.

Premium content

N-Player COCO-Q Reinforcement Learning

Next from NAML 2023

Multicollinearity Diagnostics on Cyclostationary Features

Similar lecture

Leverage Artificial Intelligence to Learn, Optimize, and Win (LAILOW) for the Marine Maintenance and Supply Complex System

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES