Several popular sequence-based and pretrained language models have been found to be successful for text-driven prediction of brain activations. However, these models still lack long-term cognitive plausibility as well as insights on the underlying neural substrate mechanisms. This paper studies the influence of context representations of different language models such as sequence-based models: Long short-term memory networks (LSTMs) and ELMo, and popular pretrained Transformer language model (Longformer). In particular, we study how the internal hidden representations align with the brain activity observed via fMRI when the subjects listen to several narrative stories. We use brain imaging recordings of subjects listening to narrative stories to interpret word and sequence embeddings. We further investigate how the representations of language models layers reveal better semantic context during listening. Experiments across all language model representations provide the following cognitive insights: (i) the representations of LSTM cell states are better aligned with brain recordings than LSTM (hidden state), the cell state activity can represent more long-term information, (ii) the representations of ELMo and Longformer display a good predictive performance across brain regions for listening stimuli; (iii) Posterior Medial Cortex (PMC), Temporo-Parieto-Occipital junction (TPOJ), and Dorsal Frontal Lobe (DFL) have higher correlation versus Early Auditory (EAC) and Auditory Association Cortex (AAC).

Long-Term Plausibility of Language Models and Neural Dynamics during Narrative Listening

How does the brain represent different modes of information? Can we design a system that can automatically understand what the user is thinking? We can make progress towards answering such questions by studying brain recordings from devices such as functional magnetic resonance imaging (fMRI). The brain encoding problem aims to automatically generate fMRI brain representations given a stimulus. The brain decoding problem is the inverse problem of reconstructing the stimuli given the fMRI brain representation. Both the brain encoding and decoding problems have been studied in detail in the past two decades and the foremost attraction of studying these solutions is that they serve as additional tools for basic research in cognitive science and cognitive neuroscience. Recently, inspired by the effectiveness of deep learning models for natural language processing and computer vision, such models have been applied for neuroscience as well. In this tutorial, we plan to discuss different kinds of stimulus representations, and popular encoding and decoding architectures in detail. The tutorial will provide a working knowledge of the state of the art methods for encoding and decoding, a thorough understanding of the literature, and a better understanding of the benefits and limitations of encoding/decoding with deep learning.

Deep Learning for Brain Encoding and Decoding (1)

How pre-trained transformer-based language models perform grounded language acquisition through cross-situational learning (CSL) remains unclear.
In particular, it is still not understood how meaning concepts are captured from complex sentences, along with learning language-based interactions,
could benefit the field of human-robot interactions and help understand how children learn and ground language.
In our current work, we study cross-situational learning to understand the mechanisms enabling children to learn rapidly sequence of words to meaning concepts: the two sequence-based models (i) Echo State Networks (i.e., Reservoir Computing), and (ii) Long-Short Term Memory Networks (LSTM) makes this mapping.
We consider the three different input representations: (i) One-Hot encoding, (ii) BERT fine-tuned on Juven+GOLD corpus, and (iii) Google BERT.
We investigate which of these three representations better predict the stimulated vision as a function of sentences describing the scenes using two models.
Using our approach, we test two datasets: Juven and GoLD, and present how these models generalize only after a few hundred partially described scenes via cross-situational learning.
We find that both One-Hot encoding and BERT fine-tuned representations (for both models) significantly improve the predictions.
Moreover, we argue that these models are able to learn complex relations between the contexts in which a word appears and their corresponding meaning concepts, handling polysemous and synonymous words.
This aspect could be incorporated into a human-robot interaction study that examines grounding language to objects in a physical world and poses a challenge for researchers to investigate better the use of transformer models in robotics and HCI.

Learning to Parse Sentences with Cross-Situational Learning using Different Word Embeddings Towards Robot Grounding

FT47: Neuroscience

technical paper

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

CogSci 2022

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





**Organizers and Presenters:** Subba Reddy Oota, IIIT Hyderabad; Jashn Arora, IIIT Hyderabad; Manish Gupta, IIIT Hyderabad; Raju Bapi, IIIT Hyderabad; Mariya Toneva, Carnegie Mellon University
 
 How does the brain represent different modes of information? Can we design a system that can automatically understand what the user is thinking? We can make progress towards answering such questions by studying brain recordings from devices such as functional magnetic resonance imaging (fMRI). The brain encoding problem aims to automatically generate fMRI brain representations given a stimulus. The brain decoding problem is the inverse problem of reconstructing the stimuli given the fMRI brain representation. Both the brain encoding and decoding problems have been studied in detail in the past two decades and the foremost attraction of studying these solutions is that they serve as additional tools for basic research in cognitive science and cognitive neuroscience. Recently, inspired by the effectiveness of deep learning models for natural language processing and computer vision, such models have been applied for neuroscience as well. In this tutorial, we plan to discuss different kinds of stimulus representations, and popular encoding and decoding architectures in detail. The tutorial will provide a working knowledge of the state of the art methods for encoding and decoding, a thorough understanding of the literature, and a better understanding of the benefits and limitations of encoding/decoding with deep learning.

Tutorial 1: Deep learning for brain encoding and decoding

tutorial

SpLU-RoboNLP: Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

workshop paper

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

IJCNLP-AACL 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

Subba Reddy Oota

3

1

SHORT BIO

Presentations

Long-Term Plausibility of Language Models and Neural Dynamics during Narrative Listening

Deep Learning for Brain Encoding and Decoding (1)

Learning to Parse Sentences with Cross-Situational Learning using Different Word Embeddings Towards Robot Grounding

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES