United States

The future of Naval Aviation is unmanned and eventually autonomous. This presentation focuses on the generation of certification for a ML algorithm to complete a task without a human in or on the loop: The autonomous aerial refueling task. With the support and cooperation of the National Airworthiness Council Artificial Intelligence Working Group I have published multiple academic papers, white papers, and several presentations on this topic. This presentation highlights my previous work area, and details a possible path forward. Standards and methods of compliance do not exist for certification officials to allow ML algorithms to operate without human oversite. Until these can be established use of ML without a human in or on the loop will remain a demonstration vice a fleet fielded system.

NAML 2023

Standards and Methods of Compliance for Autonomous Unmanned Aerial Refueling

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
<br>Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations
<br><br>

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2025

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

technical paper

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


"Baseline efforts in Performance in Noise indicated that tasks performed in a tactical situation break down when speech intelligibility decreases. Speech intelligibility decrements can be caused by excessive background noise or cluttered audio channel communications. The objective of this effort is to investigate the development of a communication aid using combined benefits of speech recognition, speech analysis, and Machine Learning (ML) tools for enhancing and filtering communications to achieve high speech intelligibility even when communications are poor due to the addition of noise and cluttered comms. The benefit to this technology is not only improved communications in hard to hear situations, but also allows for the appropriate use of hearing protection without sacrificing ability to hear verbal communications. 
The purpose of this effort is to investigate the development of assistive technology through speech recognition and Machine Learning (ML) tools to enhance and filter communications for Navy specific environments and test the technology on the ability of Sailors to successfully complete tasks in low speech intelligibility conditions. Traditional commercial off the shelf (COTS) speech recognition algorithms are not effective with the typical sentence structure of military communications. Our current efforts are building upon previous successes to train models capable of parsing and understanding our Warfighters to provide a useful assistive technology for improving their ability to communicate in lower speech intelligibility environments.
"

Performance in Noise, Automated Speech Recognition Communications Aid

"Concerns of trustworthiness, unintended social
harms, and unacceptable social and ethical viola-
tions undermine the promise of ML advancements.
Moreover, such risks in complex ML-driven sys-
tems are often difficult to foresee, as many only
arise over periods of time, across populations, and
at scale. These risks emerge through interactions
amongst ML development choices, the context of
model use, environmental factors, and the effects
of a model on its target. Systems safety engi-
neering is an established discipline with a proven
track record of identifying and managing such
risks in high-complexity sociotechnical systems.
In this work, we apply a state-of-the-art systems
safety approach to concrete ML applications with
notable social and ethical risks to demonstrate a
systematic means to argue for safe and trustwor-
thy ML in sociotechnical systems."

Applying Real Safety to Machine Learning Problems

The task of parsing, reading and analyzing the textual and structured information contained in scanned/digital-born documents, namely Visual Document Understanding (VDU) enabled by Optical Character Recognition (OCR), is an incredibly challenging task due to the diversity of layouts and formats, the complexity of template structures, and lack of support for low resource languages, dialects, and scripts. Traditionally, methods for VDU/OCR have relied on rule-based or conventional machine learning approaches, which does not only produce inferior results but also is extremely hard to generalize and fine-tune the solution for documents of interest. To address these shortcomings, our team proposes a VDU/OCR solution enabled by a unified text-image multimodal transformer to learn cross-modal representations. Our approach leverages pre-trained model families such as Language-Independent Layout Transformer (LiLT) a for both text-based document understanding using language models, and image-based document understanding using Visual Transformers, and fine-tune them for the languages, scripts, and document formats of interest. 

Deep Learning for Visual Document Understanding in Low Resource Languages

While the recent boom in Artificial Intelligence (AI) has given rise to the technology’s use and popularity across many domains, the same boom has exposed vulnerabilities of the technology to many threats that could cause the next “AI winter." AI is no stranger to “winters," or drops in funding and interest in the technology and its applications. Many in the field consider the early 1970’s as the first AI winter with another proceeding in the late 1990’s and early 2000’s. There is some consensus that another AI winter is all but inevitable in some shape or form, however, current thoughts on the next winter do not consider secure and robust AI and the implications of the success or failure of these areas. The emergence of AI as an operational technology introduces potential vulnerabilities to AI’s longevity. The National Security Commission on AI (NSCAI) report outlines recommendations for building secure and robust AI, particularly in government and Department of Defense (DoD) applications. However, are they enough to help us fully secure AI systems and prevent the next “AI winter”? An approaching “AI Winter” would have a tremendous impact in DoD systems as well as those of our adversaries. Understanding and analyzing the potential of this event would better prepare us for such an outcome as well as help us understand the tools needed to counter and prevent this “winter” by securing and robustifying our AI systems. In this talk, Harguess will introduce the following four pillars of AI assurance, that if implemented, will help us to avoid the next AI winter: security, fairness, trust, and resilience.

Is the Next Winter Coming for AI? Elements of Making Secure and Robust AI

"Social media and conversational messaging have created the largest untapped asset for public health enrichment and commercialization. This data is unstructured which makes it difficult to analyze with conventional technologies. Current natural language analytic methodologies are static, focusing on discerning what conversations are about. The next generational leap in analyzing human language is temporal trajectory modeling, offering the ability to both understand and predict where a conversation is going in the future. Temporal trajectories could enable the automated tracking of a person’s mental or emotional state over time to predict trends and outcomes like acts of aggression, suicide, attention deficit disorders, bipolar disorders, schizophrenia, and the early onset of Alzheimer’s. The goal is to show that these temporal trajectories can be recognized and predicted with accuracy that matches or improves upon human interpretations of the same data. The empirical evidence generated from this research will support the ultimate goal of early detection and/or prediction of future behaviors or mental health conditions. 

At the core of the approach being considered is the use of document vectors to model sentence similarities over time to track the trajectories of human thought and predict future behaviors and outcomes. The use of document vectors has already been proven to achieve high accuracy, particularly in the classification of documents. Document vectorization enables the classification of a discrete piece of a conversation with no relation to its entirety or forward path. A temporal trajectory classification model is an umbrella-like technology that predicts the trajectory of these discrete elements.  In essence, we will be creating a new ensemble learning model from discrete models that generate predictive outcomes and mindset directionality.  Key hurdles include the ability to incorporate linear proximity models into a multi-dimensional space while predicting semantic trajectories. This approach represents the next evolutionary step in natural language comprehension in the fields of mental health, public safety, national security and law enforcement.

The outcomes from the successful creation and deployment of such a technology could have far-reaching positive societal impacts, as well as strong commercial value. Imagine the improvements (saved lives, improved outcomes, efficiencies) that could be made in the diagnosis and treatment of a wide range of mental health issues plaguing our communities and institutions including suicidal ideation, radicalization, and homicidal ideation. Additionally, this research will establish a foundational technology for a wide range of other use cases including the trajectory of corporate communications to predict future bankruptcies, risk of non-compliance, shifts in employee morale, hostile work environments, corporate malfeasance, etc."

Mindset Trajectory Tracker

As AI continues to become part of our lives, it will become essential for people and AI to collaborate effectively. A key problem with current-generation AI is that it cannot always be trusted, and users do not know when they can or should trust it. In this presentation, I'll talk about the Trusted AI project, which is a major 3-year  collaboration between Indiana University, Notre Dame, Purdue University, and Naval Surface Warfare Center Crane Division. The project has two goals: to conduct novel research related to understanding how to improve AI and how to help people know when to trust (and not to trust) AI, and to develop the AI workforce by educating students at all levels about AI and security. The project involves 15 faculty PIs at the 3 universities and almost 50 students at all levels. 

Trusted AI Research and Workforce Development

K-Means Based Automated Behavior Recognition Using Random Forest Learning

"The San Diego Supercomputer Center was founded in 1985 as one of two national centers for supercomputing research and applications. It's mission is to ""translate innovation into practice"". Innovations range across computing, information, and data sciences, broadly defined, and practice includes any and all domains, with a special focus on STEM fields in addition to solving pressing societal challenges.
In this talk we will provide a brief overview of the center, followed by details on some of its AI & ML systems and activities. "

AI & ML at the San Diego Supercomputer Center (SDSC)

"Automatically detecting and classifying objects in remote sensing imagery can significantly improve intelligence analysis efficiency and throughput. Object detectors are typically trained on a specific set of object classes aligned with the intelligence problem of interest. The detection of unexpected, unusual, or novel objects is also a core intelligence task routinely performed by imagery analysts, but current detectors cannot perform this task. Novel objects are usually ignored by the detector, if they are not sufficiently similar to any known class, or are misclassified as a known class.

Novel object detection addresses this capability gap by discovering objects that are not within the set of known classes but are of intelligence significance. Remote sensing images often contain dozens or hundreds of object types, many of which are not in the training data. Many of these objects are irrelevant, while others may be quite important. Our method automatically alerts analysts to novel objects that are similar to the training set but are sufficiently different such that they may be a new class or a novel variant of a known class. Balancing these two constraints enables the detection of salient novelties, such as a new type of wing on a previously-known aircraft model, without overwhelming the analyst with nuisance novelties such as changes in vegetation.

The xView dataset contains WorldView-3 imagery with 60 object classes labeled in a two-level ontology, with 8 general classes in the top level, such as “Truck”, and 52 specific classes in the lower level, such as ""Truck with Flatbed"". We adapted this dataset for realistic novelty detection by considering 18 lower level classes as novel and removing images containing them from the training set. These novel classes are distributed across all 8 of the general categories. During testing, the system should detect instances from these 18 categories as novel objects, while also detecting all instances from the 34 known classes, declaring them to be non-novel, and correctly classifying them.

Our method is mathematically grounded in Extreme Value Theory (EVT) and utilizes the Extreme Value Machine (EVM) that we developed on the DARPA Science of Artificial Intelligence and Learning for Open-world Novelty project. EVM creates an EVT distribution over the distances between deep-learned feature embeddings for each class to provide a novelty score. We train an object detector, classifier, and EVM on the known classes, then estimate the novelty of each detection at test time. On XView this approach achieves a 16% AUC increase over a baseline method that thresholds the classifier scores.

Our novelty detection method can be used in various applications of interest to the Navy, such as detecting novel aircraft classes or attributes on the ground, in the air, or on a carrier ship; novel ship or vehicle detection; or detection of novel aircraft attributes, including novel ammunition or damage.
"

Novel Object Detection in Remote Sensing Images for Naval Applications

Two requirements for maintaining our lead in artificial intelligence (AI) are widespread development and the adoption of AI applications.  Both are risky and not yet common. However we have achieved them with many large organizations including Pfizer, Schlumberger, and GE Aviation.  We call it “Everyday AI”: embedding AI development and apps in everyday tasks and processes throughout an enterprise.  A key challenge is balancing data/AI/ML decentralized development with governance.  If there’s too much decentralization then apps aren’t trusted or adopted.  If there’s too much governance then decentralization doesn’t take hold and everyday tasks aren’t addressed.  This presentation covers eight “plays” organizations can run to achieve Everyday AI, a few of which are a data/AI/ML literacy program for all skill levels, catalog the top 20% data/AI/ML products as-is, and an industrial-scale AI operations automation workbench.

Premium content

Downloads

Next from NAML 2023

Performance in Noise, Automated Speech Recognition Communications Aid

Similar lecture

Using Large Language Models to Assist in Air Combat Mission Planning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES