United States

Commanders and policymakers often require human-machine teaming on subjective decisions, often in a specific setting, a region or a specific population, and may call on engineers to rapidly redeploy a system to another setting. Subjective tests often lack ground truth, or even a reference standard, to calculate performance metrics such as accuracy and precision. Extensions to multiclass problems are even more challenging. We present a biologically-inspired resampling method to reframe these problems as a treatment effect analysis a specific result from the original method validation which will support rapid revalidation in a new setting. Military use cases are proposed, and specific analyses from medicine are presented, along with a software implementation for reference.

NAML 2024

Sample size estimation for multi-class, multi-rater human-machine teaming problems without ground truth, applications to military medicine and military affairs

resampling

machine-learning

validation

technical paper

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Discussions on building a framework for the development of Robust and Safe AI systems with extensions to Naval and Defense  applications

DoD operations increasingly seek to use AI-enhanced control systems to make rapid decisions at mission time, for both autonomy and more general command and control operations. This is motivated by a need for both automated and human control systems that can make decisions significantly faster than the dynamics of the real-world environments in which they operate. Despite the advances offered by AI-enhanced systems in terms of reaction time and their ability to integrate large amounts of available data into their decisions, these systems are often mistrusted, as a consequence of their decisions being both opaque and prone to making errors that a human operator would not. Reliability in fast-paced environments requires that decisions can be quickly substantiated in real-time to prevent such errors and build trust in these systems. We are developing methods for using AI-enhanced control systems as intelligent decision tools, helping operators better understand the context of any AI-derived recommendation and empowering operators to fully explore the supporting evidence for that recommendation. To accomplish this, we have extended a methodology for adversarial explanations to state-of-the-art reinforcement learning frameworks, including MuZero. Adversarial explanations are a technique for generating counterfactuals that call attention to salient portions of data that contributed to a given decision. While related work focuses on highlighting salient data, such highlights say nothing about the relationships amongst that data that led to a decision. In contrast, adversarial explanations result in a “texture” of data that can be intuitively understood by an operator who is familiar with the kinds of data used to make a decision, even if they are not familiar with the patterns in the data that justify one decision versus another. In our work, we have shown how these textures can be used to diagnose either if an AI agent is missing an important detail in its decision process, or if an operator is not considering a vital detail. Additionally, we provide mechanisms for evaluating the criticality of any given decision – that is, evaluating when the consequences of a wrong decision are potentially catastrophic to mission success. This criticality is tied to the timeframe in which a good decision must be made to avoid catastrophe; that is, it helps operators decide if a decision is not only important, but also urgent, allowing important but non-urgent decisions to receive additional analysis and consideration. Together, these techniques allow for the more trustworthy exploitation of AI capabilities in rapid environments, giving better insights into recommended decisions and empowering human operators to override those decisions when necessary.

Making AI a Better Partner for Trusted C2 Applications

Data Analytics Needs Psychology

The Scalable Asymmetric Lifecycle Engagement (SCALE) Embedded System Security (ESS) Trusted Artificial Intelligence (TAI) program is a workforce development effort within the Trusted and Assured Microelectronics (T&AM) program. The goal of TAI is to provide a novel workforce development model that engages current DOD/DIB/GOV scientists and engineers with ongoing research projects at partner universities. This exposes DOD/DIB/GOV employees to state-of-the-art research and helps them direct it towards DOD/DIB/GOV-relevant problems. It also allows students at the undergraduate and graduate level to interact with DOD/DIB/GOV employees with the goal of fostering interest in DOD/DIB/GOV employment, increasing the likelihood of recruiting a ready and flexible workforce cleared for work for the DOD/DIB/GOV. This effort provides interesting and challenging continuing education to government employees while forging close relationships between principal investigators and their labs, increasing opportunities for future collaborations and recruitment.

Trusted AI and Naval Workforce Development

As AI systems take on critical roles in high-stakes decisions, the ability to explain their decisions becomes crucial for assigning appropriate trust. Limitations for explaining black-box systems make interpretable AI methods an appealing alternative. Case-based reasoning (CBR) solves new problems by retrieving solutions to similar prior problems and adapting their solutions to fit the new problem circumstances, and is “naturally interpretable”, in that decisions can be justified by presenting the cases from which they were derived. However, there has been little study of when the benefits of CBR apply and how specifics of case presentation affect explanatory effectiveness. To assess factors affecting the usefulness of cases to explain decision-making, we conducted a human subjects study (N=110) to investigate how alternative types of explanations of a CBR system’s conclusions impact user trust in the CBR system. Our goal was to inform future design of case-based decision support systems by advancing understanding of how different system types and explanations affect the factors of goodness, convincingness and trust in the system.

Our experiment tested how two different types of interactions with a CBR system—one-shot, or interactive “conversational case-based reasoning”—and four types of explanation presentation affected subjects’ assessment of explanations of decisions of a case-based classification system. In our tests, augmenting presented cases with explicitly highlighted similarities and differences was helpful, but augmenting cases with nearby counterfactual cases was not. Whether the system interacted in a conversational manner was not found to impact scores, suggesting that the case itself is more important to users than how it was retrieved. To assess effects when erroneous conclusions were explained, we provided the user with an intentionally incorrect conclusion, without the user being made aware that the conclusion was incorrect, and found impact on scores for goodness, convincingness and trust of case-based explanations. This suggested that case-based explanations were not masking system errors. We also studied the effects of similarity of the presented case to the new problem, and found that it impacted convincingness and trust, underlining the importance of sufficient past cases.

The study illustrates that even for interpretable methods, specifics of explanation presentation matter. Results can be used by system designers to improve the quality of the system explanations. Also, by underlining that the key factor in goodness, convincingness, and trust is the quality of the cases themselves, the study provides support for case presentation as a natural vehicle for explanation.

This work was funded by the US Department of Defense (Contract W52P1J2093009), and by the Department of the Navy, Office of Naval Research (Award N00014-19-1-2655).


Interpretable Decision-Making By Case-Based Reasoning: Lessons from a Human Subjects Study

"Artificial intelligence (AI) and machine learning (ML) have grown in an unprecedented manner, leading to their widespread adoption in complex domain settings far removed from computer science and computer engineering. As these technologies have matured interest has increased in the development of artificial cognitive capabilities that would allow these new systems to act less like an automated appliance, and more like an interdependent teammate. Similarly, advances in team science and cognitive science have led to advances in our understanding of the patterns and frameworks human teammates utilize in communication, coordination, and adaptability to form effective teams even with diverse team composition, and novel task environments. While much of the literature on effective teaming has focused on the realm of human teaming, these advances have been noted as important to human-machine teams as well.
 
In order to be effective as interdependent teammates, instead of merely statistical appliances, machine co-performers need to develop skills for teaming intelligence including the establishment of common ground with human co-performers and understanding their purpose in joint activity (Clark, 1996).  In response to these needs, we propose a novel framework that integrates dynamic Self-Organizing Maps (SOMs) and the automated generation and extraction of team relevant ontologies to generate common ground and teaming intelligence aligned to the intent of human commanders. This framework enhances AI capabilities by providing contextually rich and explainable insights, fostering a shared understanding and seamless interaction between human operators and AI systems.  We demonstrate this framework using over open source intelligence data collected from gray zone conflicts in the domain of  Illegal, Unreported, and Unregulated (IUU) fishing activity, including 55 billion AIS messages representing over 200,000 industrial fishing vessels, carriers, and bunker vessels from 2012 - 2017 with spatiotemporal characteristics including over 100 billion GPS positions providing full labeled tracks of voyages.  "

Enhancing Explainability for Human-Machine Teams with Automatic Common Ground Establishment and Ontology Estimation

The time- and computational-cost of solving the atmospheric radiative transfer problem with sufficient spatial and temporal resolution for predictive-quality scene simulation has been a limiting factor for many decades. 
The advent of deep learning has shown its ability to dramatically accelerate the inference times, sometimes running 100s of times faster, while still achieving extremely high accuracy compared to traditional reference/standard numerical solvers. 
We will present unprecedented preliminary results of a novel deep-learning model architecture for hyperspectral atmospheric radiative transfer solution estimation along with a quantitative analysis of its performance, both in terms of accuracy/precision and runtime

Preliminary results for a Deep Learning Atmospheric Radiative Transfer Solver to accelerate Predictive Scene Simulation

Point measurements of optical turbulence rely on sensitive instruments that are often difficult to deploy in many environments. Furthermore, existing models of turbulence may not perform well in those environments. In a recent experiment at the Naval Postgraduate School, optical turbulence measurements using a sonic anemometer along with various atmospheric parameters (air temperature, humidity, solar flux, etc.) were collected over a ten-month period. 
Using the observed atmospheric parameters as predictors, we developed a bagged ensemble of regression trees using optical turbulence as the response. Feature selection was used to decide the parameter set. Overall, the model’s predicted values showed good agreement with observed values of turbulence derived from the sonic anemometer data.


Prediction of Optical Turbulence Using Machine Learning Regression Models

Atmospheric propagation effects, including optical turbulence, impact performance of Naval systems including optical communications, laser, and imaging systems. Cn2 is a common quantitative measure used to characterize optical turbulence and its impact on optical systems. Parametric models of turbulence (such as NAVSLaM) generally perform well under well-characterized, relatively homogeneous environments, however are not typically designed for challenging environments, with varying terrain height. Machine learning is well suited to modeling these kinds of scenarios.

In this presentation, we will demonstrate Cn2 predictions along an inhomogeneous test path, by building and applying specific ML (machine learning) models, using meteorological data as input parameters. We use approximately two years of data collected on the SPARTA test path in San Diego, CA, a path that is well instrumented with meteorological and turbulence sensors. The extensive dataset provides a large training set, and enables diurnal and seasonal trend analysis. We explore different combinations of inputs and rank them in order of their contribution and relative importance to the model prediction. A comparison between various ML algorithms, such as LightGBM (Light Gradient Boosting Machine) and Extra Tree, is performed to determine which best predicts Cn2. In addition to using meteorological data as inputs to the model, we also experiment with using images captured at a test range setup at NIWC Pacific in San Diego.

Novel Approaches to Modeling Turbulence in Inhomogeneous Environments

Atmospheric turbulence is a phenomenon resulting from temperature, distance, altitude, humidity, wind speed, pollution, and fog, which creates aberrations in laser optical systems that attempt to track a target. With atmospheric turbulences, the ability to focus radiated energy reliably and repeatedly at range, with precision and controllable effects, is hampered due to the aberrations created for long range telescopic systems. Current approaches to correct for aberrations commonly utilize a beacon projected onto the target. However, this approach is undesirable for deployed systems, since using a beacon reveals the location of the detector. The objective of this work is to develop a beaconless, imaging-based method of high-bandwidth wavefront sensor that corrects for atmosphere-induced aberrations in long range telescopic systems.

The breakdown of the workflow is as follows. In order to understand the atmospheric turbulence effects on long range telescopic systems, the first step entails creating the aberrations in images with Zernike polynomials, which are functions that create various distortions in aperture lenses. The approach includes synthetic generation of the dataset in MATLAB that varies the degree of phase and tilt, two controllable parameters that create the varying distortions. The second part investigates image compression techniques for storing sparse representations of the images. The objectives for image compression is to discover a representation that retains the needed information to perform the image correction while utilizing less memory storage and less computational time to process, since this correction will need to be performed in a real-time adaptive optics system. The third part involves performing the image deblurring of the aberrations. This step requires performing a deconvolution of the image by separating the image from the point spread function that induces the aberrations, and performing image restoration that removes the aberrations. This work presents results that ties in this closed-loop workflow for an adaptive optics system by comparing several image deblurring and compression approaches, accounting for execution speed and accuracy of the resulting image.

Premium content

Sample size estimation for multi-class, multi-rater human-machine teaming problems without ground truth, applications to military medicine and military affairs

Downloads

Next from NAML 2024

Discussions on building a framework for the development of Robust and Safe AI systems with extensions to Naval and Defense applications

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES