United States

The Navy faces increasing challenges in detecting suspicious maritime behavior due to complexities in integrating diverse data sources and distinguishing genuine threats from routine activities. NextGen Federal Systems proposes a robust solution that uniquely integrates advanced weather forecasting and simulation capabilities into Knowledge Graph technology—a key differentiator enhancing maritime behavior analysis. Leveraging our experience developing high-resolution weather modeling solutions for programs like the Joint Simulation Environment (JSE) Weather Services and the Army Weather Operational Effects (WxOE) initiative, we address weather-related anomalies that often produce false positives.

By integrating data from sources such as AIS, SAR, EO imagery, RF signals, ship specifications, port logs, insurance databases, and sanctions lists, our approach provides a comprehensive understanding of ship behaviors. This multimodal data fusion enables analysts to identify patterns and anomalies with greater accuracy. Examples include disruptions in vessel observability due to adverse weather conditions or ship course deviations to avoid hazardous environments. By leveraging authoritative data like NOAA’s GOES satellite imagery and synthetic weather services, we ensure analysts can differentiate genuine threats from routine or weather-driven behaviors.

To enhance usability, we propose integrating Generative AI with Retrieval-Augmented Generation (RAG) to streamline querying and analysis. RAG enables analysts to pull relevant insights directly from the Knowledge Graph, providing contextual and actionable information in real time. Building on our expertise from the Synthetic Natural Battle Environment Engine (SyNBEE) SBIR project, this capability allows analysts to interact with the Knowledge Graph using natural language prompts. Our experience with simulation tools like Ansys STK enables us to model complex behaviors, such as analyzing RF signal-to-noise ratios (SNR) through atmospheric conditions to detect likely spoofing. Combined with our multimodal data integration, these tools help analysts accurately identify and model suspicious behaviors.

Further reinforcing our solution, our development of BIFROST—a modern, high-performance, one-stop web portal for the Air Force Weather Enterprise (AFWE)—showcases our ability to deliver user-centered design. By employing a Human-Centered Design (HCD) process and leveraging DevSecOps pipelines for rapid deployment, we ensure our solutions meet operational requirements at the speed of need.

Our solution aligns with the Navy&#39;s goals of applying machine learning for tactical platform data and advancing AI human factors. By fusing multimodal data, environmental modeling, simulation expertise, and user-centric design, we offer a scalable, impactful tool for maritime domain awareness, ensuring operational efficiency and decision superiority.


NAML 2025

NextGen&#39;s Multimodal Fusion: GenAI with Environmental and Contextual Modeling for Maritime Domain Awareness

weather forecasting

suspicious vessel behavior

weather modeling

multimodal data fusion

generative ai

maritime domain awareness

retrieval-augmented generation

anomaly detection

machine learning

knowledge graphs

The Navy faces increasing challenges in detecting suspicious maritime behavior due to complexities in integrating diverse data sources and distinguishing genuine threats from routine activities. NextGen Federal Systems proposes a robust solution that uniquely integrates advanced weather forecasting and simulation capabilities into Knowledge Graph technology—a key differentiator enhancing maritime behavior analysis. Leveraging our experience developing high-resolution weather modeling solutions for programs like the Joint Simulation Environment (JSE) Weather Services and the Army Weather Operational Effects (WxOE) initiative, we address weather-related anomalies that often produce false positives.

By integrating data from sources such as AIS, SAR, EO imagery, RF signals, ship specifications, port logs, insurance databases, and sanctions lists, our approach provides a comprehensive understanding of ship behaviors. This multimodal data fusion enables analysts to identify patterns and anomalies with greater accuracy. Examples include disruptions in vessel observability due to adverse weather conditions or ship course deviations to avoid hazardous environments. By leveraging authoritative data like NOAA’s GOES satellite imagery and synthetic weather services, we ensure analysts can differentiate genuine threats from routine or weather-driven behaviors.

To enhance usability, we propose integrating Generative AI with Retrieval-Augmented Generation (RAG) to streamline querying and analysis. RAG enables analysts to pull relevant insights directly from the Knowledge Graph, providing contextual and actionable information in real time. Building on our expertise from the Synthetic Natural Battle Environment Engine (SyNBEE) SBIR project, this capability allows analysts to interact with the Knowledge Graph using natural language prompts. Our experience with simulation tools like Ansys STK enables us to model complex behaviors, such as analyzing RF signal-to-noise ratios (SNR) through atmospheric conditions to detect likely spoofing. Combined with our multimodal data integration, these tools help analysts accurately identify and model suspicious behaviors.

Further reinforcing our solution, our development of BIFROST—a modern, high-performance, one-stop web portal for the Air Force Weather Enterprise (AFWE)—showcases our ability to deliver user-centered design. By employing a Human-Centered Design (HCD) process and leveraging DevSecOps pipelines for rapid deployment, we ensure our solutions meet operational requirements at the speed of need.

Our solution aligns with the Navy's goals of applying machine learning for tactical platform data and advancing AI human factors. By fusing multimodal data, environmental modeling, simulation expertise, and user-centric design, we offer a scalable, impactful tool for maritime domain awareness, ensuring operational efficiency and decision superiority.


NextGen's Multimodal Fusion: GenAI with Environmental and Contextual Modeling for Maritime Domain Awareness

technical paper

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event.You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

**Get Your Tickets for NAML 2025 - Complimentary to students, government, and military**

Dive into the world of machine learning research relevant to Naval applications.

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 

We are offering a one-time only discounted on-demand access to this NAML 2025 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml25ondemand.paperform.co/)

Please register!

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

In support of the navy’s logistics, this machine learning research project evaluated automation opportunities to increase operational efficiency and resiliency of the navy’s readiness in a contested logistics environment. The overall focus is two-fold: an upstream and a downstream approach. The “upstream” approach investigates increased automation, improvements in quality, de-duplication in the configuration records (i.e., anomaly removal) resulting in better configuration and provisioning records, as well as potentially reducing the stock that needs to be managed. The “downstream” approach encompasses current developments in sustainment (i.e., predict supply and maintenance needs, indicate anomalies, and support deep dives into system health). Together, these approaches result in an efficient and accurate configuration and provisioning profile in support of a spectrum of operational aspects of ships, ultimately reducing delay times and increasing uptime in alignment with one of the Navy (CNO and NAVSEA) key tenets of 80% surge cable fleet and on-time delivery.

Applications of Machine Learning to Smart Provisioning and Predictive Sparing

Many machine learning applications are constrained by limited quantity, quality, and variance of the collected real-world datasets used for training and evaluation. In this work, authors leveraged generative artificial intelligence techniques to extend the amount of data available to train and evaluate convolutional neural networks (CNNs) for object detection and classification. Stable Diffusion was used as the core augmentation algorithm alongside traditional machine vision techniques. A variety of augmentation techniques were compared in terms of their impact on training and evaluating CNNs. The augmented images were used in part to train CNNs to improve the performance of detection models when evaluated on real-world images. Additional experiments were conducted which quantified the prediction performance on real-world data by measuring the performance on similar synthetic data. Within these experiments, various ratios of synthetic and real-world images were used to train networks which were then evaluated on real-world and synthetic holdout datasets.

Comparison of augmentation techniques with Stable Diffusion for aircraft identification

The rising popularity of spatial computing has generated an increased demand for high fidelity 3D-modeling solutions. However, most of these applications assume laboratory-controlled image conditions, which relaxes the problem. On the contrary, real-world situations require algorithms to operate under unconstrained and opportunistic imaging conditions. As part of the Intelligence Advanced Research Projects Activity’s (IARPA) WRIVA program, Accenture is developing advanced novel view synthesis technology to enable new capabilities for mission planning and preparation, remote asset monitoring and inspection, visualization and virtual reality with emphasis on building solutions for unconstrained and non-cooperative image settings. 
We have developed a computer vision capability to perform novel view synthesis and high-fidelity 3D site modeling from limited imagery (in space and time) and a diverse set of imaging sensors while addressing several environmental, data gap and architectural challenges. In addition, we have investigated algorithms that can detect and mitigate image (meta)data artifacts to improve site model representation including time stamp prediction, camera pose restoration, and image artifact detection and mitigation. 
The 3D site modeling involves learning an accurate representation of the scene’s geometry and structure given the available imagery using a combination of techniques including structure-from-motion, monocular depth estimation, and Gaussian Splatting. This learned model is then used to render arbitrary viewpoints to synthesize novel views from arbitrary viewpoints. Additionally, the model can produce high fidelity 3D site representations, including depth maps and point clouds as byproducts for more explicit access to the scene’s 3D geometry.
In this talk we will describe how, in collaboration with world-class academic institutions, we are researching and developing state-of-the-art capabilities in 3D reconstruction and site modeling (e.g., Gaussian Splatting and Neural Radiance Fields) to become a viable alternative for use cases involving point clouds and classical photogrammetry. We will discuss how we are building site models and reconstructing large outdoor scenes given a set of unconstrained image sensors. We will further show how we are using generative AI to edit scenes (add/remove/edit objects) in a 3D sense to facilitate interpretability and exploration.
Additionally, we will show how our computational framework can generate fly-through and walk-through renders, while a user can freely navigate in them mapped environment using their mobile device or virtual reality headset.


BELLA – Building Navigable Geospatial Environments at Large scales with Limited Imagery Under Sensor Agnostic Conditions

The pandemic related increase in the overlap of work and personal device usage as well as resulting number of malicious attacks are helping attackers generate malware variants on various operating systems. Because of these attacks, the volume, type, and sophistication of malware are continuously increasing. Consequently, an improved platform-agnostic malware analysis system is a critical requirement to stop the rapid expansion of malicious activities and minimize
using various cyber security tools for different operating systems. We propose PAMDAC, a system for platform-agnostic malware detection and classification
that satisfies such a requirement. Typically, a platform-agnostic feature derived from the analyzed malware sample does not depend on the knowledge of the platform which the malware targets. In this comprehensive study, we look at the performance of different binary features for malware detection and classification on various platform (Android, Linux and Mac OS) datasets. Experimental results are quite
promising with highly accurate classification results on various malware datasets. Finally, we perform a case study to look at the effectiveness of using such
platform-agnostic features on the amalgamation of the aforementioned datasets for malware detection.

PAMDAC: Platform-Agnostic Malware Detection And Classification Based On Binary Features

The GenAI for Wargaming initiative, part of the CAIMANS project, explores the integration of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to automate the creation of military wargaming scenarios. Traditional scenario development is time-intensive, requiring extensive human effort to research, analyze, and synthesize operationally relevant content. This project aims to enhance wargaming efficiency by using AI to generate “Road to War” briefs, tactical vignettes, and executable simulation scripts in seconds rather than days.

In Sprint 1, the team developed an AI-driven pipeline that leverages LLMs to generate structured, contextually accurate wargaming scenarios. By incorporating RAG techniques and verified data sources (e.g., the Janes military database), the system minimizes inaccuracies and enhances realism. A fact-verification loop further reduces AI hallucinations, ensuring generated content aligns with real-world constraints.

A key innovation in this sprint is the automated translation of scenarios into Lua scripts for execution in the Command PE simulation environment. This allows rapid scenario instantiation, enabling real-time scenario iteration and dynamic course-of-action assessments. Additionally, the project explored AI applications for DIGICAT (a digital operational wargaming system) and began assessing LLM-driven optimization models for military applications.

Results demonstrate a significant reduction in manpower and time required for scenario generation, with outputs meeting operationally relevant accuracy and fidelity standards. This system will be tested in the Northwest Pacific 2025 wargame, providing real-world validation of AI-driven wargaming automation. Future research will focus on real-time scenario adaptation, intelligent agent behaviors, and reinforcement learning integration, further advancing the role of AI in military planning and strategic decision-making.

GenAI for Wargaming - Sprint 1 - Scenario Generation using LLMs

Wargaming, as a part of mission planning or strategic training, is an important but time-consuming process designed to ensure that any planned military courses of action (CoAs) are made with a comprehensive understanding of how encounters with enemy agents could unfold. Simulating the course of action of enemy agents is a vital aspect of wargaming, as this allows military strategists to play out different possible CoAs, identify weaknesses in their CoAs that enemies could exploit, and further strengthen their CoAs to account for a range of enemy responses. Advances in generative AI could take advantage of the creative strengths of Large Language Models (LLMs) to considerably speed up the wargaming process. 
To this end we present our progress on wargaming within GAME (Generative AI Mission Engineering), our project aiming to use LLMs for mission planning. We have developed the wargaming system we call NEXUS (Naval EXperimental Unit Simulation), which includes a battlespace framework for modern military combat. NEXUS updates the state of battle in accordance with command decisions and allows information about the state of battle to be packaged into a structured data format for an LLM to easily process. NEXUS also builds upon existing LLM-based military command simulators (Goecks & Waytowich, 2024). In the process of building NEXUS we evaluated how well different self-reflection frameworks enhanced CoA generation and found that the best-performing framework for benchmark problems with a single possible solution (Zhao & Huang, 2023) nonetheless produced underwhelming results when used for CoA generation in more open-world problems with many valid solutions. To explore different possible enemy actions, we prioritize the production of diverse CoAs by crafting commander “personas” with contrasting personalities (e.g., aggressive vs cautious). In conjunction with the ReAct (“reason before action”) framework (Yao & Zhao, 2023), our approach allows NEXUS to generate creative battle strategies within that persona’s specific role to enable testing CoAs that best fulfill mission objectives.

V. G. Goecks and N. Waytowich, ‘‘COA-GPT: Generative pre-trained transformers for accelerated course of action development in military operations,’’ in Proc. Int. Conf. Mil. Commun. Inf. Syst. (ICMCIS), Apr. 2024, pp. 1–24.

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents are experiential learners. arXiv preprint arXiv:2308.10144, 2023.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023.

Generative AI Mission Engineering: A Creative Approach

Background: Trust in AI systems is recognized as key for team performance, for the adoption and use of new technology, and to adapt, modernize, and maintain a competitive edge. This is especially important for military and civilian operations with AI systems. However, the nature of trust is elusive, complicated, and enmeshed in a web of subjectivity. Trust is amorphous, abstract, context-dependent, dynamic, often irrational, complicated by subjective observer (trustor) dependencies, complicated by subject (trustee) dependencies, and suffers from measurement difficulties because of its tacit nature (surveys and questionnaires are estimates, not truth, because people often say or think one thing and do another). 

Objectives: The overall goal of the Operational Trust in Mission Autonomy (OPTIMA) working group is to improve models and estimates of trust to understand, diagnose, and predict usage of AI systems. This will inform AI development, human training, and human-machine teaming. The goals of this work were to create an epistemological foundation for trust estimation by: 
(1) Explicitly defining and justifying foundational beliefs,
(2) Formalizing a trust estimation methodology, and
(3) Identifying trust and trustworthiness concepts and distilling them into a socio-technical ontology of antecedents, and an ontology of contractual considerations for context.

Approach: We approached trust from a philosophical and psychological perspective by first exploring trust’s subjectivity and prevailing viewpoints. Then, we conducted a conceptual analysis of 43 scientific and 12 non-scientific publications. We then used subject matter experts to group concepts and construct ontologies.

Results: The findings of this conceptual analysis revealed that organizational processes, procedures, regulations, and governance are important for trust, yet are under-represented in the scientific literature. We labeled these enterprise sources. From this, we constructed two novel ontologies of trust relationships: a trust antecedent ontology consisting of three socio-technical sources —human, system, and enterprise — and a contract ontology covering context. We used these components to make recommendations for future research, measurements, and modeling. 


Foundations for Operational Trust in Mission Autonomy

As automation becomes increasingly integrated into complex military tasks, its role in supporting human performance under fatigue warrants careful evaluation. A specific military use case in which automatic target recognition (ATR) is being integrated is undersea threat detection (UTD). These types of tasks demand sustained vigilance, accurate classification, and reliable metacognitive judgements. Fatigue, especially due to increased time awake, presents a significant challenge to sustaining high performance. This study investigated whether ATR enhances UTD performance under low fatigue conditions and protects against errors when operators are fatigued, as is common during fleet operations. 
Thirty-six active duty service members completed four sessions of a simulated UTD task, with and without an imperfect ATR system, over a ~24 hour wakeful period. Results showed that ATR did not enhance performance when participants were alert but mitigated the impact of fatigue on detection accuracy. However, fatigue led to decreased metacognitive sensitivity, reflected in greater confidence for false alarms and reduced trust in the ATR system. These findings suggest that while automation can protect basic task performance under fatigue, it does not prevent the degradation of higher-level cognitive processes, such as metacognitive accuracy and trust in automation. 
This study highlights the importance of understanding how automation interacts with cognitive states, especially under fatigue, to optimize its role in critical military operations.

Imperfect Automatic Target Recognition Stabilizes Post-Mission Analysis Performance but not Metacognition and Trust under Increased Time Awake

COSMOS is a state-of-the-art government cloud infrastructure. See firsthand how its capabilities can support DoD customers, boost efficiency, propel AI/ML adoption forward, and drive mission success.

COSMOS: Moving Mission Light Years Ahead

This session will discuss the MAPT ecosystem, its foundational capabilities as well as the critical issues as seen by defense and commercial sectors. Examples include energy efficiency, functional and supply chain security, workforce development and many others.  What will manufacturing and supply chain look like in next 2/5/10/20 years?

Premium content

NextGen's Multimodal Fusion: GenAI with Environmental and Contextual Modeling for Maritime Domain Awareness

Downloads

Next from NAML 2025

Applications of Machine Learning to Smart Provisioning and Predictive Sparing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES