A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines.1 Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

Visually Grounded Continual Language Learning with Selective Specialization

The presentation titled "CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text" introduces a novel framework designed to enhance the processing and understanding of procedural text. CLMSM stands out by its unique approach of utilizing metadata from procedures, integrating a Contrastive Learning strategy for global context comprehension, and employing Masked Step Modeling to capture local context nuances. This framework addresses the inherent challenge in procedural text processing – the need for a temporal understanding and tracking of entities across steps, a critical aspect that conventional pre-training methods like Masked Language Modeling (MLM) often overlook.

In this presentation, we demonstrate how CLMSM significantly outperforms various baseline models in entity tracking tasks, underscoring the importance of in-domain pre-training for accurately learning temporal and entity-based information in procedures. The results are particularly notable in scenarios requiring the tracking of entities with implicit locations, highlighting CLMSM’s advanced knowledge retention capabilities. Additionally, we present insights into CLMSM's adaptability to out-of-domain tasks, further establishing its versatility. This framework not only sets a new standard in procedural text understanding but also opens avenues for future research, particularly in extending its applicability to other domains with procedural text, potentially unveiling new challenges and opportunities.

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Open-domain dialogue system usually requires different sources of knowledge to generate more informative and evidential responses. However, existing knowledge-grounded dialogue systems either focus on a single knowledge source or overlook the dependency between multiple sources of knowledge, which may result in generating inconsistent or even paradoxical responses. To incorporate multiple knowledge sources and dependencies between them, we propose SAFARI, a novel framework that leverages the exceptional capabilities of large language models (LLMs) in planning, understanding, and incorporating under both supervised and unsupervised settings. Specifically, SAFARI decouples the knowledge grounding into multiple sources and response generation, which allows easy extension to various knowledge sources including the possibility of not using any sources. To study the problem, we construct a personalized knowledge-grounded dialogue dataset Knowledge Behind Persona (KBP), which is the first to consider the dependency between persona and implicit knowledge. Experimental results on the KBP dataset demonstrate that the SAFARI framework can effectively produce persona-consistent and knowledge-enhanced responses.

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

Motivational Interviewing is a counselling style that requires skillful usage of reflective listening and engaging in conversations about sensitive and personal subjects. In this paper, we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections on user responses. We conduct a two-step human evaluation where we first independently assess the generated reflections based on four criteria essential to health counseling; appropriateness, specificity, naturalness, and engagement. In the second step, we compare the overall quality of generated and human-authored reflections via a ranking evaluation. We use GPT-4, BLOOM, and FLAN-T5 models to generate motivational interviewing reflections, based on real conversational data collected via chatbots designed to provide support for smoking cessation and sexual health. We discover that GPT-4 can produce reflections of a quality comparable to human-authored reflections. Finally, we conclude that large language models have the potential to enhance and expand reflections in predetermined health counseling chatbots, but a comprehensive manual review is advised.

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

Recently, Temporal Knowledge Graph Forecasting (TKGF) has emerged as a pivotal domain for forecasting future events. Unlike black-box neural network methods, rule-based approaches are lauded for their efficiency and interpretability. For this line of work, it is crucial to correctly estimate the predictive effectiveness of the rules, i.e., the confidence. However, the existing literature lacks in-depth investigation into how confidence evolves with time. Moreover, inaccurate and heuristic confidence estimation limits the performance of rule-based methods. To alleviate such issues, we propose a framework named \textbf{TempValid} to explicitly model the temporal validity of rules for TKGF. Specifically, we design a time function to model the interaction between temporal information with confidence. TempValid conceptualizes confidence and other coefficients as learnable parameters to avoid inaccurate estimation and combinatorial explosion. Furthermore, we introduce a \textit{rule-adversarial negative sampling} and a \textit{time-aware negative sampling} strategies to facilitate TempValid learning. Extensive experiments show that TempValid significantly outperforms previous state-of-the-art (SOTA) rule-based methods on six TKGF datasets. Moreover, it exhibits substantial advancements in cross-domain and resource-constrained rule learning scenarios.

Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting

Recognizing the imperative to address the reliability and transparency issues of
Large Language Models (LLM), this work proposes an LLM maturity model
tailored for text-to-query applications. This maturity model seeks to fill the existing void in evaluating LLMs in such applications by incorporating dimensions
beyond mere correctness or accuracy. Moreover, this work introduces a realworld use case from the law enforcement domain and showcases QueryIQ, an
LLM-powered, domain-specific text-to-query assistant to expedite user workflows and reveal hidden relationship in data.

Harvesting Trustworthy AI for DOD Superiority

he SeaVision maritime awareness tool provides an unclassified common operating picture for the U.S. Navy and many partner nations. It was developed at Naval Information Warfare Center (NIWC) Pacific and the U.S. Department of Transportation, Volpe Center. SeaVision has a complex interface, and training materials span many hundreds of pages. This makes learning to use SeaVision a difficult task, especially for our foreign partners. 

We used open-source LLMs to develop a chatbot that can answer questions about how to use the SeaVision tool. With this chatbot, new users can ask a question and quickly learn how to use any of SeaVision's many features. We provide the LLM context by inserting text excerpts from training documents and transcripts of training videos. Our system's architecture combines both a retrieval augmented generation (RAG) system and a custom memory module. Our RAG system uses a FAISS vector database to store the training files as embedded passages; cosine similarity was used as our metric for retrieval. The custom memory module keeps track of past questions, answers and contexts. When memory is limited, the module favors preserving the newest question and answer pairs while discarding the oldest context available. The memory module also acts as a context gatekeeper by stopping duplicate contexts from being fed into the prompt. 

Although question answering is important to being able to operate SeaVision, it is only a small part of the larger picture. We plan to expand the functionality of this chatbot to perform atomic tasks carried out by maritime domain analysts such as getting the current locations of ships, the paths ships have followed through time, port call histories, inspection histories, warnings about potentially unsafe or insecure shipping, and much more with the goal of combining multiple atomic tasks into single common tasks. We plan to demonstrate how this type of tool can automate tasks and greatly improve Warfighter performance. Many other U.S. Navy systems have similarly complicated interfaces, and chatbots could improve Warfighter performance for these systems as well.

Training SeaVision Operators using Large Languages Models

We apply the concept of differentiable simulation to enhance machine learning for passive sonar. Our approach uses a differentiable ambient noise model that can be used as a neural network layer, allowing it to learn optimal data augmentations through interpretable, tunable parameters during training. We use this augmentation layer to adversarially train a neural network that detects acoustic emissions from a merchant ship under varying signal-to-noise ratios (SNRs) and weather conditions. Our adversarial training shows a remarkable ability to generalize performance to low SNR signals as compared to non-adversarial baseline training.

Adversarial Machine Learning Training for Signal-to-Noise Generalization in Passive Undersea Acoustics

The Naval Postgraduate School Meteorology Department has a developed model, Navy Atmospheric Vertical Surface Layer Model (NAVSLaM), to predict optical turbulence in the surface layer (up to ~100 m above the ocean or land) based upon atmospheric measurements using simple, robust sensors. However, the Physics Department has developed machine learning models of optical turbulence using atmospheric measurements. This research measures optical turbulence over many months using sonic anemometers that served as the baseline to compare prediction from the models. Atmospheric parameters such as air temperature, wind speed, humidity at two different heights; solar flux and ground temperature were simultaneously collected. Those data were used as input for NAVSLaM and machine learning models to predict optical turbulence. We then compared the performance of these prediction models to each other by calculating the root-mean-square error with respect to the baseline data from the sonic anemometers. The results from this research will help determine which model is more reliable to the given environment.

Comparison of Optical Turbulence Prediction Models

With the increasing reliance on collaborative and cloud-based systems, there is a drastic increase in attack surfaces and code vulnerabilities. Automation is key for fielding and defending software systems at scale. Researchers in Symbolic AI have had considerable success in finding flaws in human-created code. Also, run-time testing methods such as fuzzing do uncover numerous bugs. However, the major deficiency of both approaches is the inability of the methods to fix the discovered errors. They also do not scale and defy automation. Static analysis methods also suffer from the false positive problem – an overwhelming number of reported flaws are not real bugs. This brings up an interesting conundrum: Symbolic approaches actually have a detrimental impact on programmer productivity, and therefore do not necessarily contribute to improved code quality. What is needed is a combination of automation of code generation using large language models (LLMs), with scalable defect elimination methods using symbolic AI, to create an environment for the automated generation of defect-free code.

Double-edged sword of LLMs: mitigating security risks of AI-generated code

Dive into the realm of advanced AI with our poster presentation on integrating large language models and RAG architectures using Databricks. Discover how this cutting-edge technology transforms operational intelligence, offering unprecedented capabilities in data analysis, decision-making, and strategic operations. Learn how Databricks, with its robust data processing and AI tools, can harness the full potential of large language models, providing the Navy with a formidable tool in information management and operational efficiency. Join us to explore this synergy of technology and defense, paving the way for smarter, faster, and more secure Naval operations.

Empowering Military Intelligence with Next-Gen AI: Unleash the Power of Databricks and Large Language Models

An average enterprise may use over 100 applications (even if no one can actually count them all), some redundant, some outside the visibility of a documented enterprise architecture and certainly not all integrated. As a result, users spend hours moving information between applications and ensuring the consistency of that data to the extent of their desire and ability. 
This results in mindless repetition that takes its toll creating stress-related physical conditions costing trillions of dollars and introducing expensive errors into every industry. Data quality is a root cause of many of those errors. At a time when the volume and importance of data in every enterprise is growing at exponential rates, ways to ensure the quality and accuracy of enterprise data are needed; automation is one way which brings almost unbelievable speed to the processing of it.
Automation technologies build the foundation for expanding into cognitive automation with the integration of machine learning, deep learning, sensor technology, and other emerging technologies. Strengthening operational insights with the ability to combine analytics of larger sets of data with automation is an additional benefit of adoption. 
Within the context of digital transformation, automation refers to the methods and technologies used to generate software-based execution of operational processes or tasks within processes to minimize the involvement of humans. “Intelligent automation, also referred to as hyperautomation] is achieved by mimicking the capabilities that knowledge workers use in performing their work activities (e.g., language, vision, execution and thinking & learning).” (“Intelligent Automation,” Bornet, et al.) Multiple technologies support each of the four work activities; there is no single, pre-integrated, all-encompassing technology solution. 
There are many technologies that have been in use for years, such as optical character recognition and programmatic interfaces, that feed into newer automation technologies such as smart workflows, low-code/no-code software development, and robotic process automation. An enterprise automation platform will include many tools, connections, and reusable components that can link applications and systems of record, presenting a more powerful engagement layer to interface with customers and employees. As an enterprise gains in automation maturity, more elements of artificial intelligence, including machine learning and deep learning, can be integrated along with the data collected by biometrics. API management and application management are perhaps the oldest automation techniques and most found within an enterprise architecture. 
This poster breaks down the need for intelligent automation and shares the progression of technologies that are in. Use cases within the Navy mission will be shared along with NIWC successes.


Underline digital video library

Visually Grounded Continual Language Learning with Selective Specialization

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting

Harvesting Trustworthy AI for DOD Superiority

Training SeaVision Operators using Large Languages Models

Adversarial Machine Learning Training for Signal-to-Noise Generalization in Passive Undersea Acoustics

Comparison of Optical Turbulence Prediction Models

Double-edged sword of LLMs: mitigating security risks of AI-generated code

Empowering Military Intelligence with Next-Gen AI: Unleash the Power of Databricks and Large Language Models

Intelligent Automation; Why and How

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES