United States

The Naval Postgraduate School Meteorology Department has a developed model, Navy Atmospheric Vertical Surface Layer Model (NAVSLaM), to predict optical turbulence in the surface layer (up to ~100 m above the ocean or land) based upon atmospheric measurements using simple, robust sensors. However, the Physics Department has developed machine learning models of optical turbulence using atmospheric measurements. This research measures optical turbulence over many months using sonic anemometers that served as the baseline to compare prediction from the models. Atmospheric parameters such as air temperature, wind speed, humidity at two different heights; solar flux and ground temperature were simultaneously collected. Those data were used as input for NAVSLaM and machine learning models to predict optical turbulence. We then compared the performance of these prediction models to each other by calculating the root-mean-square error with respect to the baseline data from the sonic anemometers. The results from this research will help determine which model is more reliable to the given environment.

NAML 2024

Comparison of Optical Turbulence Prediction Models

laser propagation

directed energy

optical turbulence

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


With the increasing reliance on collaborative and cloud-based systems, there is a drastic increase in attack surfaces and code vulnerabilities. Automation is key for fielding and defending software systems at scale. Researchers in Symbolic AI have had considerable success in finding flaws in human-created code. Also, run-time testing methods such as fuzzing do uncover numerous bugs. However, the major deficiency of both approaches is the inability of the methods to fix the discovered errors. They also do not scale and defy automation. Static analysis methods also suffer from the false positive problem – an overwhelming number of reported flaws are not real bugs. This brings up an interesting conundrum: Symbolic approaches actually have a detrimental impact on programmer productivity, and therefore do not necessarily contribute to improved code quality. What is needed is a combination of automation of code generation using large language models (LLMs), with scalable defect elimination methods using symbolic AI, to create an environment for the automated generation of defect-free code.

Double-edged sword of LLMs: mitigating security risks of AI-generated code

Dive into the realm of advanced AI with our poster presentation on integrating large language models and RAG architectures using Databricks. Discover how this cutting-edge technology transforms operational intelligence, offering unprecedented capabilities in data analysis, decision-making, and strategic operations. Learn how Databricks, with its robust data processing and AI tools, can harness the full potential of large language models, providing the Navy with a formidable tool in information management and operational efficiency. Join us to explore this synergy of technology and defense, paving the way for smarter, faster, and more secure Naval operations.

Empowering Military Intelligence with Next-Gen AI: Unleash the Power of Databricks and Large Language Models

Background: The Millennium Cohort Study (MCS) is a prospective cohort study designed to examine the long-term effects of military service with surveys administered every 3-5 years. Participant non-response to follow-up surveys is a potential threat to the validity and generalizability of study findings. In recent years, predictive analytics has emerged as a promising tool to identify predictors of non-response.

Methods: Among participants enrolled at baseline and met eligibility criteria (n=70,117), we utilized a temporal cross-validation method to predict response to the 2019-2021 MCS follow-up survey and compared prediction performance of six machine learning algorithms. Using Latent Class Analysis (LCA), we classified participants based on historical response to the baseline survey administered in 2001-2003 and follow-up surveys administered in 2004-2006, 2007-2008, 2011-2013, 2014-2016. We compared prediction performance with and without the LCA variable. Feature analysis was subsequently run on the best performing model to identify the most important predictors with and without LCA. 

Results: The LCA variable consisted of six distinct classes, comprised of consistent responders (web, paper, and mode switcher), consistent non-responders (early and late drop-offs), and inconsistent responders. Notably, the LCA classes were found to be statistically significant across various military and demographic characteristics. 

Random forest outperformed all other machine learning algorithms in both analyses, with and without LCA. Without the LCA variable, random forest outperformed the benchmark regression model by 9.3% in area under the curve (AUC) of the receiver operating characteristic (ROC) curve and 7.7% in AUC of precision-recall. Including the LCA variable, all six algorithms performed comparably with one another, with random forest outperforming the benchmark regression model by 2.3% ROC AUC and 1.6% for precision-recall AUC. Moreover, inclusion of the LCA variable increased the ROC AUC for all algorithms by 10% or more, except for random forest and interaction forest which improved by approximately 5% in ROC AUC.

Feature analysis indicated LCA was the most important predictor in the model. In general, participants identified as consistent responders were predicted to respond to follow-up surveys (predicted probability, ŷ > 0.95), while consistent non-responders were predicted to not respond to follow-up surveys (ŷ < -0.84). Predicting response behavior among inconsistent responders appeared to be the most challenging (ŷ = 0.41).

Conclusion: Our findings suggest that historical response patterns can significantly improve the ability to predict participant response to follow-up surveys. Machine learning algorithms may be valuable when historical data are not available. Implementing these methods in longitudinal studies can enhance outreach efforts by strategically targeting participants and ultimately boost survey response and mitigate non-response.

Enhancing Targeted Outreach for Longitudinal Surveys: Predictive Analytics for Participant Response in the Millennium Cohort Study

Recognizing the imperative to address the reliability and transparency issues of
Large Language Models (LLM), this work proposes an LLM maturity model
tailored for text-to-query applications. This maturity model seeks to fill the existing void in evaluating LLMs in such applications by incorporating dimensions
beyond mere correctness or accuracy. Moreover, this work introduces a realworld use case from the law enforcement domain and showcases QueryIQ, an
LLM-powered, domain-specific text-to-query assistant to expedite user workflows and reveal hidden relationship in data.

Harvesting Trustworthy AI for DOD Superiority

Decision tree models form the basis for many machine learning approaches and have several useful properties; however, single-tree applications are limited by the balance between predictive accuracy and generalization. We introduce a new method for increasing the generalized predictive performance of pre-trained decision tree models. Using this method, the structure of the tree is leveraged to derive a set of decision threshold-based weights controlling the relative contribution of prediction values. The final prediction is then the result of an aggregation of these weighted values. This “soft trees” approach is applied to a set of benchmark datasets and is shown to increase r2 performance over standard decision trees. We further explore the introduction of missing or categorical input features, utility in probabilistic classification, and applications to tasks in resource-limited hardware environments.

Improving decision tree generalization using threshold smoothing

An average enterprise may use over 100 applications (even if no one can actually count them all), some redundant, some outside the visibility of a documented enterprise architecture and certainly not all integrated. As a result, users spend hours moving information between applications and ensuring the consistency of that data to the extent of their desire and ability. 
This results in mindless repetition that takes its toll creating stress-related physical conditions costing trillions of dollars and introducing expensive errors into every industry. Data quality is a root cause of many of those errors. At a time when the volume and importance of data in every enterprise is growing at exponential rates, ways to ensure the quality and accuracy of enterprise data are needed; automation is one way which brings almost unbelievable speed to the processing of it.
Automation technologies build the foundation for expanding into cognitive automation with the integration of machine learning, deep learning, sensor technology, and other emerging technologies. Strengthening operational insights with the ability to combine analytics of larger sets of data with automation is an additional benefit of adoption. 
Within the context of digital transformation, automation refers to the methods and technologies used to generate software-based execution of operational processes or tasks within processes to minimize the involvement of humans. “Intelligent automation, also referred to as hyperautomation] is achieved by mimicking the capabilities that knowledge workers use in performing their work activities (e.g., language, vision, execution and thinking & learning).” (“Intelligent Automation,” Bornet, et al.) Multiple technologies support each of the four work activities; there is no single, pre-integrated, all-encompassing technology solution. 
There are many technologies that have been in use for years, such as optical character recognition and programmatic interfaces, that feed into newer automation technologies such as smart workflows, low-code/no-code software development, and robotic process automation. An enterprise automation platform will include many tools, connections, and reusable components that can link applications and systems of record, presenting a more powerful engagement layer to interface with customers and employees. As an enterprise gains in automation maturity, more elements of artificial intelligence, including machine learning and deep learning, can be integrated along with the data collected by biometrics. API management and application management are perhaps the oldest automation techniques and most found within an enterprise architecture. 
This poster breaks down the need for intelligent automation and shares the progression of technologies that are in. Use cases within the Navy mission will be shared along with NIWC successes.


Intelligent Automation; Why and How

HorizonHunt is a cutting-edge GenAI system designed to aggregate and assimilate multifaceted data and render it fully searchable through an intuitive conversational interface. The HorizonHunt data lake and AI multi-modal platform uniquely integrates image, audio, text, PDF, csv, and other data modalities at enterprise scale and speed, enabling end-users to intuitively query within and across datasets using plain English. 

Benefits include: faster time to insight, multi-modal data analysis, domain-specific insights, adaptive learning, and reliable intelligence for Naval operations. 

Prototype datasets include: 
Static Data – 9/11 Commission Report including: PDFs, Text, Audio, Images
Dynamic Data – GDELT including: Text, CSV, Long/Lat 


Looking beyond the obvious – utilizing generative AI and LLMs to investigate relationships within multimodal datasets within a data lake.

The objective of this project is to use machine learning convolutional neural networks (CNN) to classify marine mammal spectrograms. CNNs excel at image recognition and can identify features and patterns in images. Spectrograms of marine mammal acoustics can be treated as images and put into a standardized format to be used as an input into a CNN model for classification, allowing for a more structured representation of the data than raw time series. The Marine Mammal Monitoring (M3R) on Navy ranges has an extensive data set of both labeled and unlabeled recordings, online sources such as the Watkins marine mammal sound data base have a smaller extent of data but more accessible labeled data. By taking advantage of these online data sets a robust CNN model can be developed and applied to a subset of M3R data to make classifications of marine mammals on navy ranges. 


Marine Mammal Classification using CNNs 

Generative AI (GenAI) and Large Language Models (LLMs) have become increasingly prominent in National Security, especially in the Department of the Navy (DON). Navy CIO Jane Rathbun's September 6th memo, "DON Guidance on Generative AI & LLMs," highlights concerns about LLMs producing biased or inaccurate text. Understanding the importance of reducing biased results, SGS data scientists have researched approaches to mitigate “hallucinations.” The first is simple; set LLM temperature to zero. Temperature is a hyper-parameter that impacts output diversity or randomness. The second is also simple, but mundane, time consuming, and difficult to scale. Provide structured text during training. A problem facing the Navy is the majority of this text is unstructured and locked away in Portable Document Format (PDF) format. SGS data scientists have applied a suite of labeling tools to develop a significantly more effective and efficient method for parsing PDFs. This novel method allows us to train a model that adaptively learns how certain publications are structured, and is an approach to enable the DON's use of GenAI.

Navigating the Seas of Uncertainty: Mitigating Risks in Large Language Models (LLMs)

The Office of the Naval Inspector General (NAVINSGEN) independently and objectively inspects, investigates, and inquires into matters of importance to the Department of the Navy. This includes handling hotline complaints received in unpredicted volumes with various priority levels based on the importance and emergency level of the case. To improve the intake process, BAH developed a Machine Learning pipeline to convert complaint forms to structural data, clean text data, and leverage Natural Language Processing (NLP) models to predict and rank the priority level for the IG intake team. The solution is backed by Databricks and lives on Naval data framework, Jupiter on Advana. In finding the optimal NLP model, the team has experimented with various transformers and techniques combined with MLFlows. This solution allows decision makers in NAVINSGEN to speed up the complaint review process by automatically suggesting and flagging critical cases.

Premium content

Comparison of Optical Turbulence Prediction Models

Next from NAML 2024

Double-edged sword of LLMs: mitigating security risks of AI-generated code

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES