Understanding the extent to which the perceptual world can be recovered from language is a fundamental problem in cognitive science. We reformulate this problem as that of distilling psychophysical information from text and show how this can be done by combining large language models (LLMs) with a classic psychophysical method based on similarity judgments. Specifically, we use the prompt auto-completion functionality of GPT3, a state-of-the-art LLM, to elicit similarity scores between stimuli and then apply multidimensional scaling to uncover their underlying psychological space. We test our approach on six perceptual domains and show that the elicited judgments strongly correlate with human data and successfully recover well-known psychophysical structures such as the color wheel and pitch spiral. We also explore meaningful divergences between LLM and human representations. Our work showcases how combining state-of-the-art machine models with well-known cognitive paradigms can shed new light on fundamental questions in perception and language research.

What Language Reveals about Perception: Distilling Psychophysical Knowledge from Large Language Models

Mental imagery is theorized to play a key role in mood and mood disorders due to the emotional impact of visualizations and biases in the processing of negative versus positive imagery. Although differences in emotional imagery have been linked to mental health outcomes, it is unclear if individuals experiencing emotional distress differ in their baseline ability to generate mental images (i.e., ‘imagery ability’). Recent research has highlighted linkages between imagery ability and facets of trait mindfulness, such as the tendency to observe and describe inner thoughts. Thus, we suspected that individual differences in trait mindfulness may help explain inconsistent findings regarding the relationship between imagery ability and emotional distress. A path analysis revealed that trait mindfulness significantly and fully mediated the relationship between imagery vividness and depression, indicating that mindfulness is a critical aspect of imagery phenomenology, as well as emphasizing the importance of mindfulness to mental health.

The relationship between individual differences in mental imagery vividness and emotional distress

Images are a powerful medium known to induce empathy and emotional response in people. In political protests it has the power for a people-initiated policy change and signifies the deep symbolism of a political system. In this study, we aim to quantify the range of emotional connection a person experiences for photographs of a farmers' protest.The protest was the headlines in all media at the time this experiment was conducted and had polarized public opinion. Each photograph is identified to have a set of physical and semantic features. The three selected features were presence of police, gender and close-up (vs.long-shot) in the frame. The intensity on a range of emotions (fear, disgust, anger, sadness, optimism, pessimism, surprise, shock, happiness, and respect) experienced by the viewer for each feature was collected. By statistical and dimensionality analyses, we isolate and identify influencing factors in an image. We found that the presence of police in aggressive actions and close-up shots of had the highest variation in the emotional responses of participants. Interestingly, the gender of the protesters did not show statistically significant effects. The findings from the exploratory investigation highlights the powerful role photographic features have on emotional responses of people, an understudied but critical factor in a world immersed in social media.

Exploring Empathy and a Range of Emotions Towards Protest Photographs

Research in event cognition has focused on how people perceive
and remember events under experimental conditions. This
research study aims to explore the temporal duration of
self-reported events from daily life (Sreekumar, et al., 2018;
Zhuang, et al., 2012). The small amount of prior work that
exists suggests that daily event durations have a Gaussian
distribution and that people have prior beliefs that reflect this
reality (Griffiths & Tenenbaum, 2006). Forty-eight participants
provided activity duration data as they went about their
everyday lives for 14 days. Descriptive analyses and activity
duration modeling (mixture models of gaussian, gamma, normal
and exponential distributions) were used to characterize event
durations within activity types. Results show that most of the
events present an exponential pattern of durations, while others
show a bimodal pattern. Although some preplanned events have
a characteristic time, many daily events have a substantial
exponential component.

How long are real-life events?

Threatening stimuli capture visual attention more rapidly than benign stimuli. The canonical interpretation of this robust finding is that the brain evolved a “fear module” enabling rapid detection of threats common at the time of mammalian evolution, such as snakes and spiders. This rapid attentional capture is thought to enable prioritized processing of threatening stimuli, providing a survival advantage, and is assumed to be universal. However, these findings have been documented almost entirely in WEIRD (white, educated, industrialized, rich, and democratic) populations. Here, we address this gap by examining threat detection in a remote African culture, the Himba. Using a touch screen visual search task, we found that both evolutionary-relevant (snakes and spiders) and modern threats (knives and syringes) captured attention more rapidly than benign stimuli. To our knowledge, this is the first study showing that the same kind of threats that rapidly capture visual attention in the West also rapidly capture visual attention in the Himba.

Visual attention to threat in the Himba, a remote people of Namibia

How possessions gain personal meaning has been a topic of interest to diverse fields studying the mind. Traditionally, the personal value of possessions is hypothesized to be driven by their special histories (Belk, 1988). We provide evidence that people value possessions for their ability to regulate distress. During the pandemic, people (N=333) reported having coping-possessions that helped them to regulate emotions (49%), solve problems (26%), or both (25%). Coping-possessions had high personal and monetary values (M=4.82 and 4.45, respectively; one-sample t-tests against midpoint, ps < .001) with problem-focused possessions having higher monetary values than emotion-focused ones (p < .001). The majority of coping-possessions, however, had mundane histories (e.g., bought at a store) rather than noteworthy ones (p < .001), suggesting that the emotional functions of objects contribute to their perceived personal value. This work also has implications for how adults use objects to regulate their emotions (Timpano & Port, 2021).

The emotional value in objects: Insights from coping with the COVID-19 pandemic

Static and Dynamic Speaker Modeling based on Graph Neural Network for Emotion Recognition in Conversation

There is a great interest in storytelling for tourism experience development. An objective of storytelling is to engage people (Gabriel, 2000) by sparking emotions. Storytelling involves loading tourism experiences in a way that engages visitors’ emotions.Emotions are regarded as core building blocks of an experience (Bastiaansen, et al., 2019). It is assumed that an experience involving emotional engagement through storytelling should translate into a memorable experience. Zak (2015) asserts that emotions and identification lead to empathy, then to narrative transportation and better recall of the story. However, little is known about how storytelling in tourism experiences leads to emotions and memories.

In this study, we compared the emotional impact and the memories of a (tourism) experience withoutstorytelling(mainly providing facts and information) to onewith storytelling(using specific storytelling techniques).

The Van Gogh Heritage Foundation is currentlyredesigning three Van Gogh visitor centers located in the south of the Netherlands where Van Gogh spent most of his life. For two locations, we wrote a factual “non-story” version, and an emotional “story” version of certain events that took place in Vincent’s (early) years. We then created four 6-minutes videos, consisting of images, and a voiceover.

Forty participants had their brain activity (EEG) and skin conductivity monitored while watching two of the four videos. Participants were asked to evaluate the videos immediately afterwards. One week after the experiment, 26 participants were interviewed about their memories of the videos.

Results show that the “stories” led to a significantly higher level of narrative transportation than the “non-stories”. Furthermore, the “stories” lead to empathy for certain story’s characters, and to stronger and more vivid memories.

The presentation will include theoretical and managerial implications. We will show how the outcomes have found their way into the redesign of the visitor centers.

Storytelling and tourism experience: Emotions around, and memories of "stories" and "non-stories" regarding Vincent Van Gogh

Physiological data are nowadays frequently used to recognize the
affective state of subjects while performing different tasks. Automatic
recognition of a stressful state as a consequence of a high level
of cognitive load is significant to prevent illnesses like depression,
anxiety and sleep disorders that are often due to excessive workload.
The spread of wearable sensors that are increasingly reliable and
comfortable makes them easy to use in day-life activities. However,
due to the nature of experiments that involve subjects, the cardinality
of the acquired data is often low, making difficult to train
deep learning methods from the scratch. In this paper we consider
the photopletismography (PPG) that measures the blood volume
registered just under the skin, which can be used to obtain the heart
rate of the subject. It is well known that PPG data are particularly
relevant to detect high level of arousal that is activated by stress.
We show that, converting monodimensional photopletismography
(PPG) data into bidimensional signals it is possible to apply a pretrained
CNN, obtaining deep features that outperform handcrafted
ones in classification tasks, especially introducing feature selections
strategies to avoid curse of dimensionality.

A Deep Learning Approach to Recognize Cognitive Load using PPG Signalshttps://dl.acm.org/doi/10.1145/3453892.3461625

Research suggests that perspective-taking in children is associated with greater cognitive and emotion understanding as well as parent use of mental state language—words concerning inner mental states or feelings. However, prior research shows conflicting results on which categories of mental state language—cognition (e.g., think, know), emotion (e.g., happy), or desire (e.g., want)—facilitate children’s perspective-taking. The current study examines the relationship between parent mental state language and children’s perspective-taking.

We are coding parent mental state language categories from a wordless book task with two-year-old children and assessing children’s perspective-taking using a card task (Flavell et al., 1981). Preliminary results (N=7, planned 41) show that, on average, parents use more cognition words per session than emotion and desire words and, on average, children score at chance on the perspective-taking task. Differences between mental state categories on children’s perspective-taking skills as well as individual variability will be analyzed.

The association between types of caregiver mental state language and toddler perspective-taking

During social interactions, we adapt our behavior in anticipation and response to the actions of our partners. This project explores whether anticipating that we are going to do something together also affects how we perceive objects in the world. In 4 Experiments, children and adults estimated the weight of objects when they anticipated that they will be lifted alone and when they thought they would have help. Results showed that the estimated weight of objects varied for adults and children depending on whether they thought they would lift alone or together. Children reliably perceived objects as lighter when they expected help from an adult parent compared to lifting them alone or with the help of a sibling. Parents, in turn, estimated objects would be heavier when they were helping their child. Overall, results suggest that we take into account the costs and benefits of helping another person when acting together.

Costs and Benefits of Helping; How Children and Adults Estimate the Intended Effort of Acting Together

The unprecedented spread of smartphone usage and its various boarding sensors have been garnering increasing interest in automatic mental health detection. However, there are two major barriers to reliable mental health detection applications that can be adopted in real-life: (a)The outputs of the complex machine learning model are not explainable, which reduces the trust of users and thus hinders the application in real-life scenarios. (b)The sensor signal distribution discrepancy across individuals is a major barrier to accurate detection since each individual has their own characteristics. We propose an explainable mental health detection model. Spatial and temporal features of multiple sensory sequences are extracted and fused with different weights generated by the attention mechanism so that the discrepancy of contribution to classifiers across different modalities can be considered in the model. Through a series of experiments on real-life datasets, results show the effectiveness of our model compared to the existing approaches.

Underline digital video library

What Language Reveals about Perception: Distilling Psychophysical Knowledge from Large Language Models

The relationship between individual differences in mental imagery vividness and emotional distress

Exploring Empathy and a Range of Emotions Towards Protest Photographs

How long are real-life events?

Visual attention to threat in the Himba, a remote people of Namibia

The emotional value in objects: Insights from coping with the COVID-19 pandemic

Static and Dynamic Speaker Modeling based on Graph Neural Network for Emotion Recognition in Conversation

Storytelling and tourism experience: Emotions around, and memories of "stories" and "non-stories" regarding Vincent Van Gogh

A Deep Learning Approach to Recognize Cognitive Load using PPG Signalshttps://dl.acm.org/doi/10.1145/3453892.3461625

The association between types of caregiver mental state language and toddler perspective-taking

Costs and Benefits of Helping; How Children and Adults Estimate the Intended Effort of Acting Together

Personalized multi-task attention for multimodal mental health detection and explanation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES