Motivational Interviewing is a counselling style that requires skillful usage of reflective listening and engaging in conversations about sensitive and personal subjects. In this paper, we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections on user responses. We conduct a two-step human evaluation where we first independently assess the generated reflections based on four criteria essential to health counseling; appropriateness, specificity, naturalness, and engagement. In the second step, we compare the overall quality of generated and human-authored reflections via a ranking evaluation. We use GPT-4, BLOOM, and FLAN-T5 models to generate motivational interviewing reflections, based on real conversational data collected via chatbots designed to provide support for smoking cessation and sexual health. We discover that GPT-4 can produce reflections of a quality comparable to human-authored reflections. Finally, we conclude that large language models have the potential to enhance and expand reflections in predetermined health counseling chatbots, but a comprehensive manual review is advised.

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

Conversational dense retrieval has shown to be effective in conversational search. However, a major limitation of conversational dense retrieval is their lack of interpretability, hindering intuitive understanding of model behaviors for targeted improvements. This paper presents CONVINV, a simple yet effective approach to shed light on interpretable conversational dense retrieval models. CONVINV transforms opaque conversational session embeddings into explicitly interpretable text while faithfully maintaining their original retrieval performance as much as possible. Such transformation is achieved by training a recently proposed Vec2Text model based on the ad-hoc query encoder, leveraging the fact that the session and query embeddings share the same space in existing conversational dense retrieval.

To further enhance interpretability, we propose to incorporate external interpretable query rewrites into the transformation process. Extensive evaluations on three conversational search benchmarks demonstrate that CONVINV can yield more interpretable text and faithfully preserve original retrieval performance than baselines. Our work connects opaque session embeddings with transparent query rewriting, paving the way toward trustworthy conversational search.

Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding

Depression is a widespread mental health disorder affecting millions worldwide. In the age of social media, it is crucial to leverage technology to detect, help, and support individuals displaying symptoms online. Our investigation revealed that social media, particularly Reddit, has a wide demographic of people and hosts a variety of discussion topics, including a significant number of posts on mental health. Using the large amounts of data on Reddit, the goal is to train and develop various machine learning models to recognize manifestations of depression and evaluating their performance. These models include machine learning and deep learning methods, such as Support Vector Machines (SVMs), transformers, and various Recurrent Neural Network (RNN) architectures — simple RNNs, Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) units — for sentiment analysis and depression detection on Reddit. The objective is to evaluate and enhance the models’ accuracy. Early detection facilitated by such models could potentially mitigate the impact of mental health disorders through timely intervention and support mechanisms.

Using Machine Learning and Deep Learning for Depression Detection on Social Media

Anxiety disorder (AD) is a condition associated with feelings of dread and an incapability to perform daily tasks. Symptoms can be physical, such as a rapid pulse and sweating, and emotional, including constant stress and difficulty concentrating. Anxiety disorders are subjective, variable and often overlap with underlying conditions, complicating detection and treatment. This study focuses on three factors, implicit memory, reaction time, and reaction to a perceived threat—all measured in previous studies and correlated with anxiety levels. Our goal is to determine if anxiety presence can be identified, using three games and one simulation. The first game centers around implicit memory, enabling us to perform tasks subconsciously. Anxious individuals exhibit an implicit memory bias for threat-related information. The second game will associate a level of anxiety to reaction times. The third game first makes the user complete a regular task and progressively introduces a threat to the successful completion of the task. Attentional bias shows that anxious people are more likely to respond to threat-related stimuli rather than stick to their task. Finally, the user will take part in an artificial anxious situation. In the first stage, the user plays a regular game. In the second, distracting stimuli appear. In the third, the user’s available resources decrease. Throughout the stages, both the user’s effectiveness and efficiency will be evaluated. If the user experiences anxiety, efficiency should decrease in stage 2 and effectiveness should decrease in stage 3. Based on their results, they will receive an assessment of their optimal stress level for highest performance. The results from all of three games will be compiled to provide an anxiety assessment. The player will then be prompted to complete the Hamilton Anxiety Rating Scale (HAM-A) which will be used to verify whether the game results match their true anxiety levels.


Using Interactive Games as Tools for Gauging Anxiety

Naval aviation assets on aircraft carriers such as aircraft, catapults and arresting gear are tracked through information systems that provide situational awareness to sailors. The current concept of operations requires highly skilled workstation operators to manage everything from aircraft status to flight schedules to keep information up to date. Operators receive information over radio communication channels from flight deck personnel. This process requires the operator to be dedicated to listening to updates and manually entering what they hear into workstations. This project developed an automated speech recognition prototype that can automatically listen into the audio communication pipeline and add information into computer systems.  
A speech recognition pipeline was created to recognize the standard phrases spoken by aircraft carrier flight deck personnel. Outputs can be used to correctly identify the actions that need to be performed to update computer systems automatically. The pipeline includes speech to text Automatic Speech Recognition (ASR) models and speech enhancement followed by post processing and similarity matching techniques to increase accuracy. Recursion was implemented to re-run the pipeline with a larger speech model if the result does not match a standard flight deck phrase. The team made use of Whisper, a transformer-based model with various model sizes ranging from 39 million to 1.56 billion trained parameters. The pipeline was tested in a reverberation chamber using flight deck phrases under various noise levels and model sizes. 
The team is currently working on fine-tuning the model and creating synthetic data. Data generation used by the team include the ability to fabricate semi-realistic speech in 108 varying voices and dialects. The team has tested fine-tuning efforts on a hold-out percentage of the dataset but has plans to continue fine-tuning the whisper model on both real-world and synthetic data to note results on increased model performance. While high-decibel noisy recordings were obtained from a laboratory, the researchers hope to obtain more real-world data and ultimately test the system either in a simulated environment or the flight deck environment. 


Aircraft Carrier Flight Deck Information Updates using Speech Recognition 

Typically, effective object detection in images requires thousands or tens of thousands of labeled instances. Crowd-sourced data has been used to train object detectors for commercial purposes, but the DoD cannot use this same approach to train neural networks with supervised learning. Either there is an insufficient number of instances of DOD-relevant data, or curating and labeling that many images is too cost and time prohibitive to be of use. ​

For this reason, few-shot object detection can be used to limit training data to tens or hundreds of instances rather than thousands upon thousands. Specifically, our experiments adapt few-shot object detection to the Naval domain. We set up two experiments where in the first experiment, we use a large, common dataset to train a base network followed by training with a small sub-group of novel classes of data; and, in the second experiment, we test another few-shot network, DE-Vit, which can use few-shot methods to find new objects in an image. ​

There are many methods that can be used to train neural networks with fewer data. These experiments are conducted as a part of the SAVANT project—which uses an ensemble of semi-supervised methods to build upon datasets. It will serve as a test bed for semi-supervised methods while creating some much-needed labeled Naval data.

Adapting Few-Shot Learning To The Naval Domain

Contextual query rewriting (CQR) is a crucial component in Conversational AI agents, leveraging the contextual information from previous user-agent conversations to improve the comprehension of current user intent. However, traditional CQR methods often concentrate on supervised fine-tuning only, neglecting the opportunities to learn from user feedback to align with user preferences. Inspired by recent advances in learning from human feedback (LHF), this paper proposes a novel Preference Aligned Contextual Query Rewriting (PA-CQR) framework to enhance the CQR model's capability in generating user preference-aligned rewrites. This paper also investigates the efficacy of various state-of-the-art feedback learning algorithms on the CQR task, and proposes a novel Dynamic Direct Preference Optimization (Dynamic DPO) algorithm to better adapt the DPO algorithm to large-scale CQR training. Experiments on large-scale real-world CQR data set demonstrate the superiority of the proposed PA-CQR framework and the Dynamic DPO.

Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning | VIDEO

A Personalized Query Rewriting system strives to minimize defective queries to ensure robust conversational functionality by considering individual user behavior and preferences. It's designed as a search-based system, maintaining a user index of past successful interactions with the conversational AI. However, this method faces challenges with unseen interactions, which refers to novel user interactions not covered by the user's historical index.

This paper introduces our Collaborative Query Rewriting approach, which utilizes underlying topological information to assist in rewriting defective queries arising from unseen user interactions. This approach begins by constructing a "User Feedback Interaction Graph" (FIG) using historical user-entity interactions. Subsequently, we traverse through the graph edges to establish an enhanced user index, referred to as the "collaborative user index". 

This paper then further explores the use of Large Language Models (LLMs) in conjunction with graph traversal, leading to a significant increase in index coverage for unseen interactions. The effectiveness of our proposed approach has been proven through experiments on a large-scale real-world dataset and online A/B experiments.

Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding

In the context of a voice assistant system, steering refers to the phenomenon in which a user issues a follow-up command attempting to direct or clarify a previous turn. We propose STEER, a steering detection model that predicts whether a follow-up turn is a user's attempt to steer the previous command. Constructing a training dataset for steering use cases poses challenges due to the cold-start problem. To overcome this, we developed heuristic rules to sample opt-in usage data, approximating positive and negative samples without any annotation. Our experimental results show promising performance in identifying steering intent, with over 95% accuracy on our sampled data. Moreover, STEER, in conjunction with our sampling strategy, aligns effectively with real-world steering scenarios, as evidenced by its strong zero-shot performance on a human-graded evaluation set. In addition to relying solely on user transcripts as input, we introduce STEER+, an enhanced version of the model. STEER+ utilizes a semantic parse tree to provide more context on out-of-vocabulary words, such as named entities that often occur at the sentence boundary. This further improves model performance, reducing error rate in domains where entities frequently appear, such as messaging. Lastly, we present a data analysis that highlights the improvement in user experience when voice assistants support steering use cases.

STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants | VIDEO

This paper introduces WordArt Designer, a user-driven framework for artistic typography synthesis, relying on the Large Language Model (LLM). The system incorporates four key modules: the LLM Engine, SemTypo, StyTypo, and TexTypo modules. 1) The LLM Engine, empowered by the LLM (e.g. GPT-3.5), interprets user inputs and generates actionable prompts for the other modules, thereby transforming abstract concepts into tangible designs. 2) The SemTypo module optimizes font designs using semantic concepts, striking a balance between artistic transformation and readability. 3) Building on the semantic layout provided by the SemTypo module, the StyTypo module creates smooth, refined images. 4) The TexTypo module further enhances the design's aesthetics through texture rendering, enabling the generation of inventive textured fonts. Notably, WordArt Designer highlights the fusion of generative AI with artistic typography. Experience its capabilities on ModelScope: https://www.modelscope.cn/studios/WordArt/WordArt.

WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models

The text prediction (TP) workflow calls a Large Language Model (LLM), almost, after every character to get subsequent sequence of characters, till user accepts a suggestion. The confidence score of the prediction is commonly used for filtering the results to ensure that only correct predictions are shown to user. As LLMs require massive amounts of computation and storage, such an approach incurs network and high execution cost. So, we propose a Model gatekeeper (GK) to stop the LLM calls that will result in incorrect predictions at client application level itself. This way a GK can save cost of model inference and improve user experience by not showing the incorrect predictions. We demonstrate that use of a model gatekeeper saved approx 46.6% of COGS for TP, at the cost of approx 4.5% loss in character saving. Use of GK also improved the efficiency (suggestion rate) of TP model by 73%.

Gatekeeper to save COGS and improve efficiency of Text Prediction

In human-computer conversations, extracting entities such as names, street addresses and email addresses from speech is a challenging task. In this paper, we study the impact of fine-tuning pre-trained speech encoders on extracting spoken entities in human-readable form directly from speech without the need for text transcription. We illustrate that such a direct approach optimizes the encoder to transcribe only the entity relevant portions of speech ignoring the superfluous portions such as carrier phrases, or spell name entities. In the context of dialog from an enterprise virtual agent, we demonstrate that the 1-step approach outperforms the typical 2-step approach which first generates lexical transcriptions followed by text-based entity extraction for identifying spoken entities.

Underline digital video library

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding

Using Machine Learning and Deep Learning for Depression Detection on Social Media

Using Interactive Games as Tools for Gauging Anxiety

Aircraft Carrier Flight Deck Information Updates using Speech Recognition

Adapting Few-Shot Learning To The Naval Domain

Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning | VIDEO

Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding

STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants | VIDEO

WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models

Gatekeeper to save COGS and improve efficiency of Text Prediction

E2E Spoken Entity Extraction for Virtual Agents | VIDEO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES