United States

Large Language Models (LLMs) can generate highly detailed and insightful information, but they remain prone to making factual errors, often referred to as &quot;hallucinations.&quot; Even with the best current mitigation—grounding the LLM in data with Retrieval Augmented Generation (RAG)—hallucinations occur at rates of on order 10%, rendering them unreliable for mission-critical work. A new technique called RAG-Verification (RAG-V), can bring down the rate of LLM hallucinations by 2 orders of magnitude. In this talk, we describe the RAG-V framework and also unpack some surprising impacts that artificial intelligence can have on the tradecraft of intelligence analysis.

NAML 2025

Primer RAG-V: Efficient Performance Enhancement for Large Language Models

automatic.

improvement

evaluation

Large Language Models (LLMs) can generate highly detailed and insightful information, but they remain prone to making factual errors, often referred to as "hallucinations." Even with the best current mitigation—grounding the LLM in data with Retrieval Augmented Generation (RAG)—hallucinations occur at rates of on order 10%, rendering them unreliable for mission-critical work. A new technique called RAG-Verification (RAG-V), can bring down the rate of LLM hallucinations by 2 orders of magnitude. In this talk, we describe the RAG-V framework and also unpack some surprising impacts that artificial intelligence can have on the tradecraft of intelligence analysis.

poster

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event.You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

**Get Your Tickets for NAML 2025 - Complimentary to students, government, and military**

Dive into the world of machine learning research relevant to Naval applications.

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 

We are offering a one-time only discounted on-demand access to this NAML 2025 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml25ondemand.paperform.co/)

Please register!

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

The United States Navy is centralizing its analytic workloads within the Jupiter system, leveraging Databricks as the framework for managing its vast and sensitive data. This strategic move consolidates data and analytics under one platform, ensuring the integrity of Controlled Unclassified Information (CUI), Personally Identifiable Information (PII), and Protected Health Information (PHI), while significantly reducing the risk of data leaks. However, this transition has introduced challenges, as analysts and data scientists adapt to Jupiter’s workflows, often resulting in inefficiencies, increased manual efforts, and the potential for models to grow stale due to inadequate retraining processes.

This paper presents a comprehensive solution currently in use by the Naval Safety Command to address these challenges. The solution focuses on automating model retraining workflows to maintain predictive accuracy as new data becomes available. Specifically, we will demonstrate an end-to-end process using one of the Naval Safety Center’s models operating on data housed within Jupiter that is updated monthly. This process includes an automated Model Search, which systematically evaluates hundreds of candidate models using the latest data, and a Model Selection component, which identifies the top-performing model based on appropriate, predefined metrics.
In addition to detailing the automated workflows, this paper explores the critical task of identifying and employing the most suitable metrics to assess model performance, ensuring that the selected models remain relevant and accurate. By eliminating manual intervention and addressing the reliance on outdated validation statistics, this approach significantly reduces inefficiencies and enhances decision-making at all levels.

By showcasing the successful implementation of this process within the Naval Safety Command, this paper aims to provide a roadmap for Navy personnel to fully leverage Jupiter’s capabilities, maintain the predictive power of their models, and establish scalable, efficient workflows that uphold the Navy’s commitment to data integrity and operational excellence.

Automating Model Search and Selection for Continuous Prediction Accuracy in Dynamic Data Environments in Jupiter

This presentation covers the ethics of the new AI/ML movement, and dives into the grey areas where technology is pulling information from users. We will explore the different types of Big Data Analytics happening in the background, and showcase the relationships (and lack thereof) where human data is pulled from across the Internet. To keep up with the addition of AI in the world, new mental models need to be made - and continuously adjusted to keep up. How does this surgency of continuously evolving AI technology impact the everyday user? 

AI is pulling from Big Data across the internet, with common sources being continuously used as the AI's playpen. Individual companies are also taking their personal Big Data sets - collected from their users over the past several years - and adding varying degrees of AI in an attempt to incorporate "fresh" technology into their systems and earn more money, often with unknown side effects. Dive in with me to learn more!


Human Factors, Ergonomics, and Ethics During the AI Takeover

Summary: Oracle presents Graph enabled Retrieval Augmented Generation (RAG) in the database to improve search and identify trends and connections across a corpus of documents. Graph enabled RAG offers many benefits over Naïve RAG as outlined in this abstract. Researchers in academia and industry have developed and experimented with variations of Graph and RAG using several packages and environments to illustrate the concept. In this presentation, Oracle will demonstrate how organizations can leverage their documents in the database to use Graph enabled RAG to visualize trends in a corpus and to improve searchability.

Abstract: The information overload in today's digital age calls for a paradigm shift in how organizations interpret and leverage their data. Introducing Graph enabled RAG (Graph enabled RAG): a cutting-edge approach that transforms the traditional data analysis landscape. Graph RAG represents a significant advancement over Naive RAG, which relies on linear text and lacks the capacity to capture the intricate relationships within data. By harnessing the power of graph structures, Graph enabled RAG creates a dynamic network that brings context and connectivity to the forefront. This approach enables organizations to uncover hidden patterns, establish meaningful correlations, and access valuable insights trapped within their data.

Naïve RAG has helped organizations to search their enterprise documentation using natural language queries to address limitations of Large Language Model (LLM)s. These limitations include the scope of the LLM’s training data that may not allow the LLM to answer a user’s question correctly, traceability that allows the user to identify the source of information that led to an LLM’s response, and the ability to point to new organization specific documentation to inform the LLMs response rather than rely on retraining the model. 

Naïve RAG addresses the limitations stated above but does not fully identify overarching themes and concepts across a collection of documents. A user that provides a query to a RAG system may not receive the mostly relevant information to their query as the information returned may not capture thematic details that could inform their response more accurately. This is in part due to memory limitations relative to the large size of enterprise documents stores. Graph enabled RAG's success lies in its ability to capture the themes and concepts that underpin large corpora of documents. It employs a sophisticated technique to generate responses, incorporating relevant information from the dataset. This ensures that the generated content is not only factually accurate but also coherently structured, making it an invaluable tool for organizations seeking to leverage their data effectively.

With Graph enabled RAG, organizations can build robust information retrieval systems, construct insightful knowledge graphs, and develop intelligent recommendation engines. While researchers in industry and academia have demonstrated variations of Graph RAG, those techniques often require additional software libraries and unnecessary complexity for implementation. Oracle will present on how to perform Graph enabled RAG in the database with the assistance of Generative AI models.

Graph Enabled RAG for Improved Search Across Large Document Stores

Cognitive radar systems must efficiently allocate limited resources to maintain accurate situational awareness in dynamic environments. Traditional rule-based approaches often struggle to adapt to rapidly changing conditions, resulting in suboptimal target-tracking performance. In this study, we present a reinforcement learning-based framework for resource allocation in multi-target tracking tasks. By employing Deep Q-Networks (DQN), the radar system learns to dynamically allocate scanning resources across multiple tasks.

Our approach utilizes extended Kalman filter (EKF) estimates of target states, including position, velocity, and covariance matrix elements to inform the decision-making process. The reward function is designed to promote uncertainty reduction and maximize detection efficiency. Performance is evaluated in a Python-based simulation environment with varying numbers of targets, comparing DQN-based strategies against traditional equal allocation methods.

Results demonstrate that the DQN-based radar agent achieves improved target detection times and reduced covariance matrix traces in certain scenarios with moderate target density. However, performance declines when the environment becomes highly dynamic with many targets. Ongoing work focuses on refining reward structures, exploring alternative reinforcement learning architectures, and introducing multi-objective optimization to enhance radar performance across diverse operational conditions.

This research highlights the potential of reinforcement learning to significantly improve cognitive radar resource allocation, offering promising avenues for future applications in aerospace and defense contexts.

Reinforcement Learning Based Resource Allocation for Radar Scanning and Multi-target Tracking

Tremendous variations coupled with large degrees of freedom in UAV-based imaging conditions lead to a significant lack of data in adequately learning UAV-based perception models. Using various synthetic renderers in conjunction with perception models is prevalent to create synthetic data to augment the learning in the ground-based imaging domain. However, severe challenges in the austere UAV-based domain require distinctive solutions to image synthesis for data augmentation. In this work, we leverage recent advancements in neural rendering to improve static and dynamic novel-view UAV-based image synthesis, especially from high altitudes, capturing salient scene attributes. Finally, we demonstrate a considerable performance boost is achieved when a state-of-the-art detection model is optimized primarily on hybrid sets of real and synthetic data instead of the real or synthetic data separately.

UAV-SIM: Enhanced UAV-based Human Detection Using Novel Neural Rendering

Setting up a local Large Language Model (LLM) on a Navy RDT&E laptop can leverage open-source tools and models approved for DoD use. The integration of the LLM into Visual Studio Code as a chat bot, tab autocomplete, and code editor, can increase efficiency writing code and improve research capabilities. This enhanced workflow allows for accessing tools in offline environments and for asking questions of project documentation using RAG, while ensuring confidentiality of sensitive information.

This poster presentation provides details on setting up a local LLM and highlight the benefits for Navy software developers. Showcasing the challenges and solutions associated with using a local LLM on an RDT&E machine can lead to more Navy software developers, both ashore and afloat, utilizing this useful tool.

Local LLMs for Software Development on a Navy RDT&E Laptop Asset

Autonomous robot scanners can map and annotate environments for a variety of DoD relevant purposes, such as supporting installations, reverse engineering, and maintenance of facilities and ships. As scanners perform their scan in a fixed location, one key aspect is finding the optimal scanning waypoints to cover a given environment by locating landmark structures, such as stairs, doors, desks, ladders, and other prominent features. In this work, we use a SPOT robot to create our own maps and an Oak-D camera for “in-real-time” object detection. We show that object detection can be used to enhance the traversal of mapped environments by annotating landmarks and using the detections as insight for object avoidance.

Object Detection And Map Annotation For Autonomous Robot Scanning

In this work, we discuss the development and deployment of a robust autonomous system capable of performing diverse maritime tasks, including autonomous docking with human-in-the-loop interaction, under unknown dynamic conditions. Our approach leverages a data-driven modular design to facilitate a seamless transfer of autonomy across various maritime surface vessel platforms. This design addresses challenges associated with a priori system model identification, particularly under evolving system behaviors and shifting environmental conditions. The proposed learning-based platform integrates a deep Koopman system model to capture system dynamics and enable predictive control. A safety filter ensures the generation of safe velocities, which the recursive least squares model predictive controller (RLS-MPC) follows to achieve robust and reliable motion control. Dock detection is performed using a camera and AprilTag, providing precise and reliable localization for docking tasks. To support development and testing, we created a simulation environment in ROS and Gazebo, enabling realistic and scalable evaluation of the system under various scenarios. We introduce safety-aware Koopman-based adaptive MPC control architecture. The framework's efficacy is validated in both simulation and real-world experiments, demonstrating its capability in autonomous docking tasks. Results show consistent improvements in accuracy and robustness compared to approaches that overlook dynamic system changes, underscoring its potential for advancing maritime autonomy.

Safety-aware Koopman-based Adaptive MPC for Docking of Autonomous Surface Vehicles

"The increasing complexity of machine learning (ML) models used in longitudinal data analysis often hinders effective interpretability, a critical barrier in high-stakes environments like the US Navy. To address this, we propose a novel framework: Embedding Model Agnostic Human Readable Attribution (EMAHRA) into longitudinal analyses. This approach one-hot encodes the trends of longitudinal variables into human-readable descriptors, such as ""slow rise,"" ""abrupt jump,"" or ""steady decline,"" to provide model explanations in language comprehensible to human decision-makers.

By transforming raw data trends into descriptive, categorical variables, EMAHRA simplifies the interpretive layer of explainable AI (XAI). Importantly, this method enhances not only the explainability but also the consistency of model outputs. Our results demonstrate that encoding trends as human-readable descriptions mitigates the influence of minor fluctuations in data, focusing the analysis on overriding trends. This improvement was observed to reduce noise in the data, resulting in models that were both more stable and more consistent in their predictions.

Applications in a US Navy setting underscore the value of EMAHRA. For instance, in ship maintenance, this approach enables personnel to interpret predictive models identifying components at risk of failure. Instead of opaque outputs, models could explain predictions with statements like ""The hydraulic system shows a steady decline in pressure, indicating wear and tear."" Similarly, in operational readiness, longitudinal health data of personnel could be described as ""gradual improvement in fitness metrics,"" helping leaders allocate resources more effectively."

Embedding Model Agnostic Human Readable Attribution into Longitudinal Analyses for Better XAI

"Neuromorphic processing mimics the processing of neural structures in the brain. Neuromorphic computer architectures enable the development of sophisticated and novel capabilities in pattern recognition, event reasoning, resilient decision-making, machine learning, and unsupervised tasking for energy-efficient mobile combat vehicles. Neuromorphic sensors with several benefits are event-based cameras. They feature little latency and low power consumption and are offered as COTS solutions. Spiking neural networks (SNN) can be trained with the event-based datasets that are produced. This paper offers a comprehensive overview of neuromorphic computing, encompassing the latest hardware and software platforms as well as some datasets in the field. This effort is based on a NISE project in which we surveyed neuromorphic computing and SNNs and collected real-time data using DAVIS cameras.
DVS cameras are a type of event-based camera -- asynchronous sensors that sample light based on the scene dynamics, rather than on a clock that has no relation to the viewed scene. Their advantages are: very high temporal resolution and low latency (both in the order of microseconds), very high dynamic range3 (140 dB vs. 60 dB of standard cameras), and low power consumption. Hence, event-based cameras have a large potential for robotics and wearable applications in challenging scenarios for standard cameras, such as where high speed and high dynamic range are critical. Spikes that indicate a change in brightness are returned by DVS cameras, which concentrate on using brightness to identify an event. SNNs and DVS cameras are complementary in that they both handle inputs by default using event-based triggers. Therefore, for algorithmic implementation, DVS cameras and neuromorphic computing hardware may result in a small form factor AI-based sensing paradigm that is both natural and incredibly efficient.
Lava is an open-source SW platform for creating AI apps for Intel Loihi and other neuromorphic hardware designs.  Lava currently supports Intel's Loihi architecture and conventional CPUs, but its compiler and runtime can be extended to support other architectures. Third-party libraries such as Nengo, ROS, YARP, and others can be added to and modified with Lava. Additional tools enable users to profile workload power and performance, visualize complex networks, and assist with the float to fixed point conversions needed by many low-precision devices, including neuromorphic hardware.
SNNs were trained in the present investigation using dynamic vision sensor (DVS)-based datasets using LAVA and other open-source tools. This is mainly because, once available, LAVA hardware will offer significant performance benefits. We examined MNIST and N-MINST, two widely used opensource datasets that serve as a benchmark for creating Navy-specific proprietary datasets and are used to train SNNs. The use of these datasets for SNN training and implementation was covered in this paper. Information on the SNN MNIST Classifier: Using a multi-layer feed-forward classifier neural network that is easily constructed and uses pre-trained weights from the LAVA repository, without the need for convolutional layers. 2) SNN testing with twenty-five MNIST dataset images 3) A classification accuracy of 88% was obtained. 2) The second portion of the SNN training phase took a long time. By adding a few more open-source spiking neural component primitives to PyTorch, users can use the most recent and stable setup created with PyTorch, as well as spiking neural network components suitable with deep learning.Using these primitives, an SNN was built and trained on the entire N-MNIST dataset. The accuracy and runtime performance were documented during five epochs. The outcomes are extensive and impressive.
Although the aforementioned event-based datasets are publicly available, none of them were designed with dod data—such as drones—in a generic format in mind. Any project that requires a drone data set must be built from the ground up. This might mean not just spending the time and employing the drones to record, but also investing in a DVS camera. In cooperation with the SAUSR range at the Naval Weapons Station, drone movement footage was recorded. We were able to set them up so we could record their plans, which they required tested. "

Premium content

Primer RAG-V: Efficient Performance Enhancement for Large Language Models

Next from NAML 2025

Automating Model Search and Selection for Continuous Prediction Accuracy in Dynamic Data Environments in Jupiter

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES