United States

Generative AI (GenAI) and Large Language Models (LLMs) have become increasingly prominent in National Security, especially in the Department of the Navy (DON). Navy CIO Jane Rathbun&#39;s September 6th memo, &quot;DON Guidance on Generative AI &amp; LLMs,&quot; highlights concerns about LLMs producing biased or inaccurate text. Understanding the importance of reducing biased results, SGS data scientists have researched approaches to mitigate “hallucinations.” The first is simple; set LLM temperature to zero. Temperature is a hyper-parameter that impacts output diversity or randomness. The second is also simple, but mundane, time consuming, and difficult to scale. Provide structured text during training. A problem facing the Navy is the majority of this text is unstructured and locked away in Portable Document Format (PDF) format. SGS data scientists have applied a suite of labeling tools to develop a significantly more effective and efficient method for parsing PDFs. This novel method allows us to train a model that adaptively learns how certain publications are structured, and is an approach to enable the DON&#39;s use of GenAI.

NAML 2024

Navigating the Seas of Uncertainty: Mitigating Risks in Large Language Models (LLMs)

technical orders

genai

generative ai

readiness

hallucinations

Generative AI (GenAI) and Large Language Models (LLMs) have become increasingly prominent in National Security, especially in the Department of the Navy (DON). Navy CIO Jane Rathbun's September 6th memo, "DON Guidance on Generative AI & LLMs," highlights concerns about LLMs producing biased or inaccurate text. Understanding the importance of reducing biased results, SGS data scientists have researched approaches to mitigate “hallucinations.” The first is simple; set LLM temperature to zero. Temperature is a hyper-parameter that impacts output diversity or randomness. The second is also simple, but mundane, time consuming, and difficult to scale. Provide structured text during training. A problem facing the Navy is the majority of this text is unstructured and locked away in Portable Document Format (PDF) format. SGS data scientists have applied a suite of labeling tools to develop a significantly more effective and efficient method for parsing PDFs. This novel method allows us to train a model that adaptively learns how certain publications are structured, and is an approach to enable the DON's use of GenAI.

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this workshop.

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


The Office of the Naval Inspector General (NAVINSGEN) independently and objectively inspects, investigates, and inquires into matters of importance to the Department of the Navy. This includes handling hotline complaints received in unpredicted volumes with various priority levels based on the importance and emergency level of the case. To improve the intake process, BAH developed a Machine Learning pipeline to convert complaint forms to structural data, clean text data, and leverage Natural Language Processing (NLP) models to predict and rank the priority level for the IG intake team. The solution is backed by Databricks and lives on Naval data framework, Jupiter on Advana. In finding the optimal NLP model, the team has experimented with various transformers and techniques combined with MLFlows. This solution allows decision makers in NAVINSGEN to speed up the complaint review process by automatically suggesting and flagging critical cases.

NLP Application on Naval IG Hotline using Jupiter Platform

Snowflake Cortex is an intelligent, fully managed service that offers access to industry-leading AI models, LLMs and vector search functionality to enable organizations to quickly analyze data and build AI applications. Snowflake Container Services enables developers to effortlessly deploy, manage and scale custom containerized workloads and models for tasks such as fine-tuning open-source LLMs using secure Snowflake-managed infrastructure with GPU, in partnership with NVIDIA, instances all within the boundary of their Snowflake account.


Obfuscating GenAI Complexities

A new data-adaptive model via machine learning is investigated as a prospective capability for ocean wave height estimation based on simulated ship motion time-series. Long Short-Term Memory (LSTM) neural networks are trained and tested with physics-based Large Amplitude Motion Program (LAMP) simulation data for the ship geometry and loading conditions of the David Taylor Model Basin (DTMB) Model 5415. From this initial study, data-adaptive machine learning appears to be a suitable candidate for continued investigation of estimation of the ocean wave field based on ship motions.

Ocean Wave Height Estimation via Machine Learning

Radio frequency fingerprinting (RFF) identifies radio frequency (RF) emitters based on signatures that the individual emitter’s hardware imprints into the RF signal. There have been significant advancements in RFF but most of the advancements investigated modern machine learning approaches to radio frequency signals where classic RF processing techniques and machine learning are combined. However, these methods often lack the robustness to operate in many DOD environments that are highly uncertain, complex, and dynamic which, in the RF communication application, is often due to the communication channel. To overcome this communication channel problem, this work begins an investigation of integrating physics-based and explainable models directly into the machine learning approach. This enables RF subject matter expertise to improve the RF fingerprinting results. In this work two approaches are investigated: networks that respect complex multiplication (complex networks); and data collections from highly variable channel conditions. 
Radio frequency fingerprinting has shown great potential for device authentication and access control in wireless communication networks. The signal intelligence (SIGINT) and data science communities have developed efficient methods to identify transmitters (Tx) based on unique signatures of individual Tx hardware. These signatures are due to intrinsic characteristics of the Tx hardware which unintentionally affect the waveforms leading to deviation from ideal waveforms. The hardware specific deviation from the ideal waveform is usually minor enough that the signal message is not impacted, the bit error rate (BER) is maintained, and the decoded messages remain intact. Since these characteristics are created by non-ideal components, such as parasitic circuitry, component tolerance, or temperature stability, they are hard to mimic in an electronic warfare (EW) when compared to software-level identification such as internet protocol (IP) or media access control (MAC) addresses.
As is the case with any wireless communication systems, the received signal is heavily impacted by noise and the channel i.e., attenuation, reflection, diffraction, and refraction paths. These effects will cause modifications of the received signal when compared to training signals which can cause a degradation in the classification performance. A RF reverberation chamber (R-chamber) is used to introduce data drift caused by multipath channel effects to the RF transmission. The R-chamber's tuner is rotated during the training stage to randomize the electromagnetic field inside the chamber and simulate a random multipath. 
Previous RF fingerprinting studies used deep neural network classifiers, and it has been observed that for Wi-Fi signals, RF fingerprinting was improved using complex-valued networks. Their study was in a structured environment with the only channel noise arising from atmospheric conditions. This work extends further explores the complex network concepts using more sophisticated channel noise generated using a R-chamber. We show an increase in performance when the network is tested in channel conditions not exposed to the training dataset demonstrating the utility of complex-valued network.

Radio Frequency Fingerprinting in Distorted Electromagnetic Environments

Thousands of vulnerabilities are found on all NAVWAR systems every year, by providing a way to prioritize which vulnerabilities to patch first based upon severity provides agents with a streamlined path to protect every system. RAVEN has built out a Machine Learning classification algorithm that utilizes Common Vulnerabilities and Exposures (CVEs) descriptions and key features linked to each CVE to predict likelihood of weaponization for each vulnerability. By identifying higher risk CVEs, cybersecurity help desk agents can prioritize patches needed for vulnerabilities on every system, investigate avenues that hackers can utilize to break into across the most vulnerable CVEs, and analyze the vulnerability landscape across the fleet. 

Before ships are ready for deployment, they must go through different event windows to ensure systems are up to date and working together with no issues. Reoccurring issues between different event windows on the same ship or system is common. The Continuity of Issues project was created to link reoccurring issues as they move across event windows. By creating a record linkage framework and Natural Language Processing techniques to look at every ticket’s discrepancy description we can match tickets across 5 disparate data sources. By enabling this viewpoint of common issues we can demonstrate the timeline of ACF installs compared to number of discrepancies within event windows, track reoccurring topics and issues over multiple time periods, and track average trend of Casualty Reports (CASREPS). 

PMW/ISEA teams are dealing with thousands of trouble tickets on a daily basis. The CASREP Prediction Model was developed to predict which Trouble Tickets are most likely to escalate into a CASREP or will take longer to resolve. This allows immediate identification of high-risk issues as soon as the trouble ticket is submitted. Enables team to prioritize response actions and resources, and potentially reduce system downtime. CASREP Prediction Model uses NAVWAR Help Desk ITSM data and five different AI/ML algorithms to work together to boost accuracy. 


RAVEN Machine Learning Solutions for Trouble Ticket Support and Cybersecurity

Continuing the effort towards data-driven RF-channel estimation from last year, this project moves towards using vector-quantized variational auto-encoders (VQ-VAEs). VQ-VAEs have shown to be able to create high-fidelity images with more ease and control than generative adversarial networks (GANs). Using simulated GNU Radio datasets from RadioML, short samples of IQ data with various digital modulations and SNRs were used to train a conditional VQ-VAE-2 with a PixelSnail network to generate the codes. The VAE-generated samples show promise of reproducing the GNU Radio simulated channel effects when compared to the training data, SNR calculations, and modulation recognition classifier results. We look to use OpenAI’s Jukebox architecture to expand these short, discrete samples in order to model longer term channel effects. With this, we hope to be able to expand this to the complex environment of high frequency (HF) data.

RF Channel Estimation with VQ-VAEs

We introduce the Spatial Auditory Network Dataset for spatialized sound event detection (SED) research. This dataset was created using the Spatial Auditory Network Dataset Synthesis (SANDS) toolset, which provides a technique for rapidly synthesizing auditory network data using modern-day 3D game development libraries. A combination of the Unity game engine and the Steam audio plugin are used to virtualize sound propagation using ray-traced multi-path sound propagation that simulates occlusion, reflection, transmission, scattering, absorption, reverberation, and Doppler effects. The dataset consists of 10,000 realizations of up to five out of eight possible sound classes moving through a residential neighborhood environment.

Spatial Auditory Network Dataset Synthesis (SANDS)

Modulation recognition of radio frequency signals has become a canonical example of radio frequency machine learning (RFML). State-of-the-art approaches employ convolutional neural networks applied to input examples batched into a fixed set of timesteps and run on GPUs. For power limited or edge applications, emerging neuromorphic hardware promises enhanced flexibility and energy savings for RFML. However, straightforward transforms of CNNs onto neuromorphic processors result in networks with too many neurons relative to current hardware. To address this, we developed a transform which takes a trained artificial neural network operating on batched inputs to a neuromorphic-ready network operating on streaming data. This streaming transform trades neuron count for synapses, making it possible to fit larger neural networks onto current neuromorphic processors. We discuss the current status and future plans, as the project is ongoing.

Streaming Modulation Recognition for Neuromorphic RF Applications

The objectives of this project are to apply an automated semi-supervised natural language processing model to parse the unstructured text columns contained in the VIDS/MAF maintenance forms. Specifically, an NER model was trained to identify components and failure modes identified in these columns. The outputs from this/these model(s) would be directly used in SE Reliability Control Board processes to identify root cause issues with the most problematic equipment. This was previously a manual process, but this effort aims to automate some of the leg work required by the PMA and IPT teams.
The project used entirely unlabeled data and an automated process was created to label multiple entities of interest contained within the text data. This sped up the process of creating fairly accurate data to train the NER model on compared to a manual labeling effort. The stop words, failure modes, and components were also stored in separate files to be used in updating future models.
Models learned entirely from the raw work order data including all typos, naval lexicon, abbreviations, etc. The only modification of the unstructured text columns was that they were concatenated into a single column and stored as lowercase characters. Some records were excluded from being used as training data in order to remove noise from the final output and abide by stakeholder objectives.
The models learned from a single general dataset and performed well on larger datasets. Smaller datasets required some fine-tuning on the components and failure modes in order to better separate unique components which are not generally seen. This process is manual but can be quickly done using a bag of words and lookup table approach on the most common words. Given most of the words already have an associated label, this eliminates most words that we need to update.
Post-processing was also required to clean the output of the model into usable results, specifically the identification of failure modes and components associated to a maintenance work order. Priority on the unstructured text columns was used to decide which components and failure modes were most relevant to the work order. Similarity scores were also used to group together misspelled tokens and similar tokens that could be rolled into a single group. The final product includes source data with the component and failure mode labeled and visualizations of the most problematic components and the most frequently occurring failure modes. The option to drilldown into more detailed visualizations is also available as needed.


Support Equipment Failure Mode Identification – Name Entity Recognition (NER)

The Marine Corps Systems Command (MCSC) Project Officers (POs) deliver and sustain Command, Control, Communication, Computer, Intelligence, Surveillance, and Reconnaissance (C4ISR) systems for the Fleet Marine Forces (FMF). As part of that process, the MCSC PO must ensure the systems meet the technical and operational requirements of the FMF. This effort involves the creation of Test & Evaluation (T&E) artifacts. This T&E process is done independently for each system and results in the verification and validation of thousands of requirements using clustering and machine learning techniques.

Next from NAML 2024

NLP Application on Naval IG Hotline using Jupiter Platform

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES