Canada

This study addresses the interaction between language and event cognition. Previous research has shown that viewers follow perceptual and conceptual features of event structure, and these features influence how they process events. Here, we examine whether events are processed in different ways when preceded by descriptions that contrast in lexical aspect. In our experiment, viewers watched fifteen short movie clips and were asked to indicate whether they saw an interruption. Interruptions occurred in either the midpoint or the late point of the events. We found that telic vs. atelic sentences shown prior to the movie clips influenced event construal and hence the detection of interruptions at midpoints vs. endpoints. This novel finding supports a mapping between lexical aspect in language and temporal structure-building in event apprehension. Our study provides direct experimental evidence for the role of language as a top-down conceptual cue in event processing.

CogSci 2022

Linguistic aspect constrains event apprehension

poster

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





How people learn about events often varies with some events perceived in their entirety and others are inferred based on the available evidence. Here, we investigate how children and adults linguistically encode the sources of their event knowledge. We focus on Turkish – a language that obligatorily encodes source of information for past events using two evidentiality markers. Children (4- to 5-year-olds and 6- to 7- year-olds) and adults watched and described events that they directly saw or inferred based on visual cues with manipulated degrees of indirectness. Overall, participants modified the evidential marking in their descriptions depending on (a) whether they saw or inferred the event and (b) the indirectness of the visual cues giving rise to an inference. There were no differences across age groups. These findings suggest that Turkish-speaking adults’ and children’s use of evidential markers are sensitive to the indirectness of the inferential evidence for events.

Linguistic Encoding of Inferential Evidence for Events

Like verbal discourse, visual narratives use various techniques to motivate readers to infer unstated events. One technique of sponsoring bridging inferences in visual sequences is to replace climactic events with an “onlooker”, a bystander viewing the off-panel climax reacting with textual and/or bodily cues to varying degrees. In two self-paced reading experiments, we explored how onlookers sponsored inferencing. Experiment 1 compared five types of onlookers ranging from a passive face with no cues to explicit facial or textual cues, and Experiment 2 investigated the effect of zooming in on onlookers’ bodily cues. Less explicit onlookers are processed faster, but at the next panel all onlookers required equal updating costs and zooms elicited further costs. Moreover, textual cues facilitated understanding. Overall, these findings show that complexity and explicitness of information in visual narratives affect subsequent event inferencing.

A Closer Look at Bridging Inferences: Onlookers and Framing in Visual Narratives

Recent evidence indicates that children represent and learn multiple meanings of ambiguous words from early in development (e.g., mail letter, alphabetic letter). This raises the question of which naturalistic factors might allow young children to resolve lexical ambiguities. Previous research has shown that children’s processing of ambiguous words is facilitated by verb-related information. However, it is still unclear whether such facilitation comes from bottom-up (lexical associations) or top-down information sources (verb-event structures). In this study, we leveraged a large sense-annotated child-directed speech corpus to disentangle the effect of bottom-up lexical and top-down event structure cues. Preliminary results show that 4-year-olds might rely on verb-event structures when these are put in competition with lexical association. We discuss implications for theories of sentence parsing and word learning.

The Role of Verb-Event Structure in Children’s Lexical Ambiguity Resolution

Over the last few decades, Causal Model Theory (CMT) has become a dominant framework for human causal-based reasoning, including categorization and inference. CMT prescribes how people should reason about probabilistic events in terms of causal models. In typical causal-based categorization experiments, subjects are provided with verbal descriptions of causally linked features, generally including probabilistic information. Another line of research focuses on perceived or experienced causal events, rather than on verbal descriptions. In this work we asked whether effects which are consistent with CMT, and that have been obtained with verbal descriptions, generalize to visually perceived events. In two experiments, we presented subjects with videos of a 3D A→B causal event rather than verbal descriptions. In Exp. 1, we found that subjects who saw the causal event did not show the coherence effect in categorization (i.e., subjects tend to rate the null ¬A¬B event as a category member). However, subjects who did see the null event during training did show the effect. In Exp. 2, we ruled out the possibility that Exp. 1’s results were simply an effect of how frequently events were experienced during training. We conclude that a one-shot perceived causal event is not sufficient for people to show causal-based reasoning as CMT predicts.


Categorizing perceived causal events

Decades of sentence comprehension research have focused on the cognitive factors that determine the processing difficulty. In this work, we investigate how locality and interference effects interact with each other in Hindi, a head-final language with flexible word order and a rich case-marking system. Using linear regression, we examine the extent to which the dependency distance of a sentence, which is based on retrieving previously-stored elements in the working memory can be predicted by various other backward-looking measures of processing complexity. We evaluate how dependency distance is influenced by similarity-based interference, case density, information structure, and finally, a forward surprisal measure proposed recently to model planning processes in comprehension and production systems. Overall, our results indicate that similarity-based interference and case density are significant positive predictors of dependency length, lending credence to the view in the literature that the mechanisms underlying locality might be driven by interference.


Interference and Case Marker Effects in Dependency Locality: Insights from Hindi

Research on cognition in people with impairments is still insufficient to account for cognitive diversity. This study addresses the issue by making the first attempt to examine synthetic speech perception in congenitally blind versus sighted participants. With the rising prevalence of modern speech technologies, the study of synthetic speech perception across various groups becomes indispensable. Although recent research revealed certain advantages of blind listeners compared to sighted ones, these studies only addressed natural speech perception. Using a speeded AX discrimination task in a controlled experiment, we test how the two groups perceive signal distortions resulting from differing synthetic speech qualities, generated using neural networks. Results show that blind participants had significantly better discrimination, especially in the difficult condition involving small distortions. This suggests that blind participants were sensitive to deviations that remained imperceptible to sighted participants. Theoretical implications for speech perception and practical insights for Text-to-Speech technologies will be discussed.


Do we all perceive synthetic speech in the same way? Testing congenitally blind versus sighted listeners

In describing images, visual and linguistic processes coordinate with each other and they both proceed in a linear fashion. Three main theories about the nature of the relation between these processes and how they unfold over time have been proposed in the literature of 'linearization'. In this work, we investigate the hypotheses put forward by these theories utilizing a corpus of spoken image descriptions with speakers' eye-movement data. We explore the time alignment between the fixations on objects and the utterance of the corresponding nouns in the data. In contrast to previous studies, this dataset allows us to inspect unrestricted language production in the context of real scenes on a larger scale. We find both confirming and conflicting evidence for each of the theories in question, suggesting that the intricate relation between eye movements and language production may involve mechanisms proposed by all three theories.


Time Alignment between Gaze and Speech in Image Descriptions: Exploring Theories of Linearization

How do children within the autism spectrum (ASD) integrate linguistic and visual information during real-time processing of sentences and dynamic scenes? We employed a visual word paradigm with sentences containing different verb types (causatives such as 'crack'; perception/psychological verbs such as 'examine') and with different scene contexts (agent in the scene moving towards a target object or remaining neutral). Participants were a group of 10-year old ASD and typically developing children (TD). We reasoned that the two dominant theories about ASD cognitive processing style—the weak central coherence theory (WCC) and the enhanced perceptual function (EPF)—make different predictions on how and where information about linguistic and visual information may be integrated. WCC predicts that ASD children would have greater difficulty with sentence and scene semantics while EPF predicts an early, post-perceptual integration via verb-specific and object (but not full scene) representations. The results suggest that ASD children differ from TD children by computing the link between verb and object faster, suggesting an enhanced post-perceptual integration.


Enhanced Visual-Linguistic Interaction in Children within the Autism Spectrum: Evidence from the Visual-World Paradigm with Dynamic Scenes

During speech perception, humans constantly sense multiple sources of information. As so, the McGurk effect has been typically conceived as a prototypical example of language multi-modal integration, specifically, it has been used to study audio-visual
integration during speech. The McGurk effect arises when in-congruent audio-visual stimuli are paired and perceived as a different syllable ( auditory /ba/ +
visual /ga/ = percept “da” ).

We developed a hierarchical computer model, based on self-organizing maps and Hebbian learning to study speech multi-modal integration. Our architecture allows studying the McGurk effect purely on bottom-up processing of audio-visual information.
We trained several versions of the model and measured the activation similarity between
McGurk alike stimuli and congruent ones by means of mutual information. Our results suggest that the illusory percept arises from the best congruent representation that reduces uncertainty. Furthermore, the reliability of each sensory modality determines the best congruent representation.


Multimodal integration as predictions. An explanation of the McGurk effect

Correlation judgments are at the core of belief formation. In previous studies of correlation judgment from 2D scatterplots, observers underestimate correlations, and display stronger underestimation biases when the scatterplot is shown in a landscape view than in a portrait view. Yet, it is unclear how these biases arise. Here, we propose that observers are Bayesian learners who perform “mental regression” using the observed data points in graph. Accordingly, judgment errors can arise from biased visual information sampling. We test our model’s predictions with two eye-tracking experiments and find that the Bayesian learning model, applied to information obtained from visual fixation data, replicates classic behavioral findings. The model also predicts trial-level estimation biases at a high accuracy level. Our study shows how computational models trained on process-level data can shed light on the cognitive mechanisms underlying belief formation, and yield theory-driven practical implications for data visualization and statistical communication.

Premium content

Linguistic aspect constrains event apprehension

Downloads

Next from CogSci 2022

Linguistic Encoding of Inferential Evidence for Events

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES