Recording the neural activity of multiple and large neural populations in the brain is becoming widely available with modern recording techniques. It still remains a challenge, however, to understand how distinct and anatomically different neural populations interact with each other to control behaviour. We propose a new method to discover causal interactions between neural populations based on recurrent switching dynamical systems. We introduce an extended dynamics model that incorporates the current time-step when calculating the latent state variables. We also introduce an acyclicity constrain in learning the parameters of the model. These mechanisms enable rich causal interactions between neuronal populations to be identified from the learned model. Our model outperforms previous work on discovering interactions between neural populations in simulated datasets, without sacrificing the prediction performance of firing rates. We also apply our method on real neural recordings from two macaque monkey brains performing a behavioral task, and show that the proposed method is able to detect causal interactions between brain regions related to the different time windows of the task.

Discovering Low-Dimensional Causal Pathways between Multiple Interacting Neuronal Populations

The self-supervised Masked Image Modeling (MIM) schema, following "mask-and-reconstruct" pipeline of recovering contents from masked image, has recently captured the increasing interest in the community, owing to the excellent ability of learning visual representation from unlabeled data. Aiming at learning representations with high semantics abstracted, a group of works attempts to reconstruct non-semantic pixels with large-ratio masking strategy, which may suffer from "over-smoothing" problem, while others directly infuse semantics into targets in off-line way requiring extra data. Different from them, we shift the perspective to the Fourier domain which naturally has global perspective and present a new Masked Image Modeling (MIM), termed Geminated Gestalt Autoencoder (Ge^2-AE) for visual pre-training. Specifically, we equip our model with geminated decoders in charge of reconstructing image contents from both pixel and frequency space, where each other serves as not only the complementation but also the reciprocal constraints. Through this way, more robust representations can be learned in the pre-trained encoders, of which the effectiveness is confirmed by the juxtaposing experimental results on downstream recognition tasks. We also conduct several quantitative and qualitative experiments to investigate the learning behavior of our method. To our best knowledge, this is the first MIM work to solve the visual pre-training through the lens of frequency domain. 

The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

Conversations emerge as the primary media for exchanging ideas and conceptions. From the listener’s perspective, identifying various affective qualities, such as sarcasm, humour, and emotions, is paramount for comprehending the true connotation of the emitted utterance. However, one of the major hurdles faced in learning these affect dimensions is the presence of figurative language, viz. irony, metaphor, or sarcasm. We hypothesize that any detection system constituting the exhaustive and explicit presentation of the emitted utterance would improve the overall comprehension of the dialogue. To this end, we explore the task of Sarcasm Explanation in Dialogues, which aims to unfold the hidden irony behind sarcastic utterances. We propose MOSES, a deep neural network which takes a multimodal (sarcastic) dialogue instance as an input and generates a natural language sentence as its explanation. Subsequently, we leverage the generated explanation for various natural language understanding tasks in a conversational dialogue setup, such as sarcasm detection, humour identification, and emotion recognition. Our evaluation shows that MOSES outperforms the state-of-the-art system for SED by an average of ∼2% on different evaluation metrics, such as ROUGE, BLEU, and METEOR. Further, we observe that leveraging the generated explanation advances three downstream tasks for affect classification – an average improvement of ~14% F1-score in the sarcasm detection task and ∼2% in the humour identification and emotion recognition task. We also perform extensive analyses to assess the quality of the results.

Explaining (Sarcastic) Utterances to Enhance Affect Understanding in Multimodal Dialogues

In recent years, disinformation and fake news have become a global phenomenon, particularly so during the pandemic, COVID-19. The wide dissemination of fake news can have detrimental societal effects on individuals and the society. Despite the recent progress in detecting fake news, it remains a challenging task due to its complexity, diversity, multi-modality, speed, and costs of fact-checking or annotation. In this talk, we look at some challenges and lessons learned when exploring effective and explainable detecting strategies of fake news, and discuss challenges in fake news research and the need for interdisciplinary research.

Combating Disinformation on Social Media and Its Challenges: A Computational Perspective

A significant number of studies apply acoustic and linguistic characteristics of human speech as prominent markers of dementia and depression. However, studies on discriminating depression from dementia are rare. Co-morbid depression is frequent in dementia and these clinical conditions share many overlapping symptoms, but the ability to distinguish between depression and dementia is essential as depression is often curable. In this work, we investigate the ability of clustering approaches in distinguishing between depression and dementia from human speech. We introduce a novel aggregated dataset, which combines narrative speech data from multiple conditions, i.e., Alzheimer's disease, mild cognitive impairment, healthy control, and depression. We compare linear and non-linear clustering approaches and show that non-linear clustering techniques distinguish better between distinct disease clusters. Our interpretability analysis shows that the main differentiating symptoms between dementia and depression are acoustic abnormality, repetitiveness (or circularity) of speech, word finding difficulty, coherence impairment, and differences in lexical complexity and richness.

Data-driven Approach to Differentiating between Depression and Dementia from Noisy Speech and Language Data

We present a simple yet effective method to train a named entity recognition (NER) model that operates on business telephone conversation transcripts that contain noise due to the nature of spoken conversation and artifacts of automatic speech recognition. We first fine-tune LUKE, a state-of-the-art Named Entity Recognition (NER) model, on a limited amount of transcripts, then use it as the teacher model to teach a smaller DistilBERT-based student model using a large amount of weakly labeled data and a small amount of human-annotated data. The model achieves high accuracy while also satisfying the practical constraints for inclusion in a commercial telephony product: realtime performance when deployed on cost-effective CPUs rather than GPUs.

An Effective, Performant Named Entity Recognition System for
Noisy Business Telephone Conversation Transcripts

Humans naturally produce nonverbal signals such as facial expressions, body movements, hand gestures, and tone of voice, along with words, to communicate their messages, opinions, and feelings. Considering robots are progressively moving out from research laboratories into human environ- ments, it is increasingly desirable that they develop a similar social intelligence. Therefore, equipping social robots with non- verbal communication skills has been an active research area for decades, where data-driven, end-to-end learning approaches have become predominant in recent years, offering scalabil- ity and generalisability. However, most of these approaches consider a single character, modelling intrapersonal dynamics only. In this paper, we propose a method based on condi- tional Generative Adversarial Networks, intending to generate behaviours for a robot in affective dyadic interactions. Our method takes as an input the audio of a target person together with the nonverbal signals of their interacting partner, modelled by a novel Context Encoder, to generate appropriate body gestures. We evaluate our method on the multimodal JESTKOD dataset that comprises dyadic interactions under agreement and disagreement scenarios. The experimental results show that Context Encoder can better contribute to the prediction of co- speech gestures in agreement situations.

Agree or Disagree? Generating Body Gestures from Affective Contextual Cues during Dyadic Interactions

Collaborative storytelling has long been a goal of social robotics, however, much of this research is limited in interactivity or assumes that story content is curated. In this paper, we present a working fully-automatic collaborative storytelling robot, which can collaborate with a person to create a unique, improvised story by using a large-scale neural language model to dynamically generate continuations to a story. Because effective storytelling requires engaging the emotions of participants, we explore several modalities of procedurally-generated expressivity: 1. an expressive text-to-speech voice with several delivery styles, 2. physical and verbal reactions performed by the robot, and 3. an external display used to show instructions and graphics during storytelling.

To understand the issues associated with improvised collaborative storytelling with a social robot, we conduct an online survey and elicitation study with a group of online observers of collaborative storytelling gameplay, comparing several expressivity strategies in terms of storytelling-related characteristics, expressivity characteristics, and personality traits as measured by RoSAS. This evaluation showed that expressivity strategies using both emotive voice and performed reactions were perceived to be more competent storytellers and more strongly associated with positive personality traits.

Robot-mediated Group Instruction for Children with ASD: A Pilot Study

As robotic systems become more interactive and complex, there is a need to standardise interfaces and simplify development processes. This is particularly pertinent in the field of manufacturing, where human-robot collaboration is on the increase, but where standards and proprietary software are key barriers to deployment and adoption. In this article we present the ROSIE Adapter, a generalpurpose, modular adapter developed in ROS designed to support the creation and connection of industry-ready digital twins. Together with our previous work on the modular CSI digital twin framework, we demonstrate how the ROSIE Adapter creates a versatile “plug-and-play” interface that simplifies the development of new robotic processes, and improves accessibility to novice users. Furthermore, the adaptor supports integration of intuitive interface devices, such as speech and augmented reality interfaces, which enable more natural collaboration. We describe the adaptor and its use in two real-world applications, demonstrate the ease of use via a three-day hackathon event, and provide results showing the faithfulness of the arising digital twins to their connected physical systems.

ROSIE: A ROS adapter for a Modular Digital Twinning Framework

In ongoing and consecutive conversations with persons, a social robot has to determine which aspects to
remember and how to address them in the conversation. In the health domain, important aspects concern the health-related goals, the experienced progress (expressed sentiment) and the
ongoing motivation to pursue them. Despite the progress in speech technology and conversational agents, most social robots lack a memory for such experience sharing. This paper presents the design and evaluation of a conversational memory for personalized behavior change support conversations on healthy nutrition via memory-based motivational rephrasing. The main hypothesis is that referring to previous sessions improves motivation and goal attainment, particularly when references vary. In addition, the paper explores how far motivational
rephrasing affects user’s perception of the conversational agent (the virtual Furhat). An experiment with 79 participants was conducted via Zoom, consisting of three conversation sessions. The results showed a significant increase in participants’ change in motivation when multiple references to previous sessions were
provided.

Combining Context Awareness and Planning to Learn Behavior Trees from Demonstration

Social robots are often critiqued as being too ‘robotic’ and unemotional. For affective human-robot interaction (HRI), robots must detect sentiment and express emotion and empathy in return. We explored the extent to which people can detect emotions, empathy and sentiment from speech expressed by a computer system, with a focus on changes in prosody (pitch, tone, volume) and how people identify sentiment from written text, compared to a sentiment analyzer. 89 participants identified empathy, emotion and sentiment from audio and text embedded in a survey. Empathy and sentiment were best expressed in the audio, while emotions were the most difficult detect (67%, 67% and 42% respectively). We found moderate agreement (70%) between the sentiment identified by the participants and the analyzer. There is potential for computer systems to express affect by using changes in prosody, as well as analyzing text to identify sentiment. This may help to further develop affective capabilities and appropriate responses in social robots, in order to avoid ‘robotic’ interactions. Future research should explore how to better express negative sentiment and emotions, while leveraging multi-modal approaches to HRI.

ClassMate Robot: A Robot to Support Teaching and Learning Activities in Schools

Robot-initiated touch is a promising mode of expression that would allow robot caregivers to perform physical tasks (instrumental touch) and provide comfort (affective touch) in healthcare settings. To understand the factors that shape how people respond to touch from a robotic caregiver, we conducted a crowdsourced study (N=163) examining how robot-initiated touch (present or absent), the robot’s intent (instrumental or affective), robot appearance (Nao or Stretch), and robot tone (empathetic or serious) impact the perceived quality of care. Results show that participants prefer instrumental to affective touch, view the robot as having greater social attributes (higher warmth, higher competence, and lower discomfort) after robot-initiated touch, are more comfortable interacting with the human-like Nao than the more machine-like Stretch, and favor consistent robot tone and appearance. From these results, we derived three design guidelines for caregiving robots in healthcare settings.

Underline digital video library

Discovering Low-Dimensional Causal Pathways between Multiple Interacting Neuronal Populations

The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

Explaining (Sarcastic) Utterances to Enhance Affect Understanding in Multimodal Dialogues

Combating Disinformation on Social Media and Its Challenges: A Computational Perspective

Data-driven Approach to Differentiating between Depression and Dementia from Noisy Speech and Language Data

An Effective, Performant Named Entity Recognition System for Noisy Business Telephone Conversation Transcripts

Agree or Disagree? Generating Body Gestures from Affective Contextual Cues during Dyadic Interactions

Robot-mediated Group Instruction for Children with ASD: A Pilot Study

ROSIE: A ROS adapter for a Modular Digital Twinning Framework

Combining Context Awareness and Planning to Learn Behavior Trees from Demonstration

ClassMate Robot: A Robot to Support Teaching and Learning Activities in Schools

Learning Personalized Human-Aware Robot Navigation Using Virtual Reality Demonstrations from a User Study

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES