United States

Recognizing the imperative to address the reliability and transparency issues of
Large Language Models (LLM), this work proposes an LLM maturity model
tailored for text-to-query applications. This maturity model seeks to fill the existing void in evaluating LLMs in such applications by incorporating dimensions
beyond mere correctness or accuracy. Moreover, this work introduces a realworld use case from the law enforcement domain and showcases QueryIQ, an
LLM-powered, domain-specific text-to-query assistant to expedite user workflows and reveal hidden relationship in data.

NAML 2024

Harvesting Trustworthy AI for DOD Superiority

trustworthy

mlops

large language model

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Decision tree models form the basis for many machine learning approaches and have several useful properties; however, single-tree applications are limited by the balance between predictive accuracy and generalization. We introduce a new method for increasing the generalized predictive performance of pre-trained decision tree models. Using this method, the structure of the tree is leveraged to derive a set of decision threshold-based weights controlling the relative contribution of prediction values. The final prediction is then the result of an aggregation of these weighted values. This “soft trees” approach is applied to a set of benchmark datasets and is shown to increase r2 performance over standard decision trees. We further explore the introduction of missing or categorical input features, utility in probabilistic classification, and applications to tasks in resource-limited hardware environments.

Improving decision tree generalization using threshold smoothing

An average enterprise may use over 100 applications (even if no one can actually count them all), some redundant, some outside the visibility of a documented enterprise architecture and certainly not all integrated. As a result, users spend hours moving information between applications and ensuring the consistency of that data to the extent of their desire and ability. 
This results in mindless repetition that takes its toll creating stress-related physical conditions costing trillions of dollars and introducing expensive errors into every industry. Data quality is a root cause of many of those errors. At a time when the volume and importance of data in every enterprise is growing at exponential rates, ways to ensure the quality and accuracy of enterprise data are needed; automation is one way which brings almost unbelievable speed to the processing of it.
Automation technologies build the foundation for expanding into cognitive automation with the integration of machine learning, deep learning, sensor technology, and other emerging technologies. Strengthening operational insights with the ability to combine analytics of larger sets of data with automation is an additional benefit of adoption. 
Within the context of digital transformation, automation refers to the methods and technologies used to generate software-based execution of operational processes or tasks within processes to minimize the involvement of humans. “Intelligent automation, also referred to as hyperautomation] is achieved by mimicking the capabilities that knowledge workers use in performing their work activities (e.g., language, vision, execution and thinking & learning).” (“Intelligent Automation,” Bornet, et al.) Multiple technologies support each of the four work activities; there is no single, pre-integrated, all-encompassing technology solution. 
There are many technologies that have been in use for years, such as optical character recognition and programmatic interfaces, that feed into newer automation technologies such as smart workflows, low-code/no-code software development, and robotic process automation. An enterprise automation platform will include many tools, connections, and reusable components that can link applications and systems of record, presenting a more powerful engagement layer to interface with customers and employees. As an enterprise gains in automation maturity, more elements of artificial intelligence, including machine learning and deep learning, can be integrated along with the data collected by biometrics. API management and application management are perhaps the oldest automation techniques and most found within an enterprise architecture. 
This poster breaks down the need for intelligent automation and shares the progression of technologies that are in. Use cases within the Navy mission will be shared along with NIWC successes.


Intelligent Automation; Why and How

HorizonHunt is a cutting-edge GenAI system designed to aggregate and assimilate multifaceted data and render it fully searchable through an intuitive conversational interface. The HorizonHunt data lake and AI multi-modal platform uniquely integrates image, audio, text, PDF, csv, and other data modalities at enterprise scale and speed, enabling end-users to intuitively query within and across datasets using plain English. 

Benefits include: faster time to insight, multi-modal data analysis, domain-specific insights, adaptive learning, and reliable intelligence for Naval operations. 

Prototype datasets include: 
Static Data – 9/11 Commission Report including: PDFs, Text, Audio, Images
Dynamic Data – GDELT including: Text, CSV, Long/Lat 


Looking beyond the obvious – utilizing generative AI and LLMs to investigate relationships within multimodal datasets within a data lake.

The objective of this project is to use machine learning convolutional neural networks (CNN) to classify marine mammal spectrograms. CNNs excel at image recognition and can identify features and patterns in images. Spectrograms of marine mammal acoustics can be treated as images and put into a standardized format to be used as an input into a CNN model for classification, allowing for a more structured representation of the data than raw time series. The Marine Mammal Monitoring (M3R) on Navy ranges has an extensive data set of both labeled and unlabeled recordings, online sources such as the Watkins marine mammal sound data base have a smaller extent of data but more accessible labeled data. By taking advantage of these online data sets a robust CNN model can be developed and applied to a subset of M3R data to make classifications of marine mammals on navy ranges. 


Marine Mammal Classification using CNNs 

Generative AI (GenAI) and Large Language Models (LLMs) have become increasingly prominent in National Security, especially in the Department of the Navy (DON). Navy CIO Jane Rathbun's September 6th memo, "DON Guidance on Generative AI & LLMs," highlights concerns about LLMs producing biased or inaccurate text. Understanding the importance of reducing biased results, SGS data scientists have researched approaches to mitigate “hallucinations.” The first is simple; set LLM temperature to zero. Temperature is a hyper-parameter that impacts output diversity or randomness. The second is also simple, but mundane, time consuming, and difficult to scale. Provide structured text during training. A problem facing the Navy is the majority of this text is unstructured and locked away in Portable Document Format (PDF) format. SGS data scientists have applied a suite of labeling tools to develop a significantly more effective and efficient method for parsing PDFs. This novel method allows us to train a model that adaptively learns how certain publications are structured, and is an approach to enable the DON's use of GenAI.

Navigating the Seas of Uncertainty: Mitigating Risks in Large Language Models (LLMs)

The Office of the Naval Inspector General (NAVINSGEN) independently and objectively inspects, investigates, and inquires into matters of importance to the Department of the Navy. This includes handling hotline complaints received in unpredicted volumes with various priority levels based on the importance and emergency level of the case. To improve the intake process, BAH developed a Machine Learning pipeline to convert complaint forms to structural data, clean text data, and leverage Natural Language Processing (NLP) models to predict and rank the priority level for the IG intake team. The solution is backed by Databricks and lives on Naval data framework, Jupiter on Advana. In finding the optimal NLP model, the team has experimented with various transformers and techniques combined with MLFlows. This solution allows decision makers in NAVINSGEN to speed up the complaint review process by automatically suggesting and flagging critical cases.

NLP Application on Naval IG Hotline using Jupiter Platform

Snowflake Cortex is an intelligent, fully managed service that offers access to industry-leading AI models, LLMs and vector search functionality to enable organizations to quickly analyze data and build AI applications. Snowflake Container Services enables developers to effortlessly deploy, manage and scale custom containerized workloads and models for tasks such as fine-tuning open-source LLMs using secure Snowflake-managed infrastructure with GPU, in partnership with NVIDIA, instances all within the boundary of their Snowflake account.


Obfuscating GenAI Complexities

A new data-adaptive model via machine learning is investigated as a prospective capability for ocean wave height estimation based on simulated ship motion time-series. Long Short-Term Memory (LSTM) neural networks are trained and tested with physics-based Large Amplitude Motion Program (LAMP) simulation data for the ship geometry and loading conditions of the David Taylor Model Basin (DTMB) Model 5415. From this initial study, data-adaptive machine learning appears to be a suitable candidate for continued investigation of estimation of the ocean wave field based on ship motions.

Ocean Wave Height Estimation via Machine Learning

Radio frequency fingerprinting (RFF) identifies radio frequency (RF) emitters based on signatures that the individual emitter’s hardware imprints into the RF signal. There have been significant advancements in RFF but most of the advancements investigated modern machine learning approaches to radio frequency signals where classic RF processing techniques and machine learning are combined. However, these methods often lack the robustness to operate in many DOD environments that are highly uncertain, complex, and dynamic which, in the RF communication application, is often due to the communication channel. To overcome this communication channel problem, this work begins an investigation of integrating physics-based and explainable models directly into the machine learning approach. This enables RF subject matter expertise to improve the RF fingerprinting results. In this work two approaches are investigated: networks that respect complex multiplication (complex networks); and data collections from highly variable channel conditions. 
Radio frequency fingerprinting has shown great potential for device authentication and access control in wireless communication networks. The signal intelligence (SIGINT) and data science communities have developed efficient methods to identify transmitters (Tx) based on unique signatures of individual Tx hardware. These signatures are due to intrinsic characteristics of the Tx hardware which unintentionally affect the waveforms leading to deviation from ideal waveforms. The hardware specific deviation from the ideal waveform is usually minor enough that the signal message is not impacted, the bit error rate (BER) is maintained, and the decoded messages remain intact. Since these characteristics are created by non-ideal components, such as parasitic circuitry, component tolerance, or temperature stability, they are hard to mimic in an electronic warfare (EW) when compared to software-level identification such as internet protocol (IP) or media access control (MAC) addresses.
As is the case with any wireless communication systems, the received signal is heavily impacted by noise and the channel i.e., attenuation, reflection, diffraction, and refraction paths. These effects will cause modifications of the received signal when compared to training signals which can cause a degradation in the classification performance. A RF reverberation chamber (R-chamber) is used to introduce data drift caused by multipath channel effects to the RF transmission. The R-chamber's tuner is rotated during the training stage to randomize the electromagnetic field inside the chamber and simulate a random multipath. 
Previous RF fingerprinting studies used deep neural network classifiers, and it has been observed that for Wi-Fi signals, RF fingerprinting was improved using complex-valued networks. Their study was in a structured environment with the only channel noise arising from atmospheric conditions. This work extends further explores the complex network concepts using more sophisticated channel noise generated using a R-chamber. We show an increase in performance when the network is tested in channel conditions not exposed to the training dataset demonstrating the utility of complex-valued network.

Radio Frequency Fingerprinting in Distorted Electromagnetic Environments

Thousands of vulnerabilities are found on all NAVWAR systems every year, by providing a way to prioritize which vulnerabilities to patch first based upon severity provides agents with a streamlined path to protect every system. RAVEN has built out a Machine Learning classification algorithm that utilizes Common Vulnerabilities and Exposures (CVEs) descriptions and key features linked to each CVE to predict likelihood of weaponization for each vulnerability. By identifying higher risk CVEs, cybersecurity help desk agents can prioritize patches needed for vulnerabilities on every system, investigate avenues that hackers can utilize to break into across the most vulnerable CVEs, and analyze the vulnerability landscape across the fleet. 

Before ships are ready for deployment, they must go through different event windows to ensure systems are up to date and working together with no issues. Reoccurring issues between different event windows on the same ship or system is common. The Continuity of Issues project was created to link reoccurring issues as they move across event windows. By creating a record linkage framework and Natural Language Processing techniques to look at every ticket’s discrepancy description we can match tickets across 5 disparate data sources. By enabling this viewpoint of common issues we can demonstrate the timeline of ACF installs compared to number of discrepancies within event windows, track reoccurring topics and issues over multiple time periods, and track average trend of Casualty Reports (CASREPS). 

PMW/ISEA teams are dealing with thousands of trouble tickets on a daily basis. The CASREP Prediction Model was developed to predict which Trouble Tickets are most likely to escalate into a CASREP or will take longer to resolve. This allows immediate identification of high-risk issues as soon as the trouble ticket is submitted. Enables team to prioritize response actions and resources, and potentially reduce system downtime. CASREP Prediction Model uses NAVWAR Help Desk ITSM data and five different AI/ML algorithms to work together to boost accuracy. 


Premium content

Harvesting Trustworthy AI for DOD Superiority

Next from NAML 2024

Improving decision tree generalization using threshold smoothing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES