Canada

Machine learning models that automatically assess reasoning quality are trained on human-annotated written products. These “gold-standard” corpora are typically created by prompting annotators to choose, using a forced choice design, which of two products presented side by side is the most convincing, contains the strongest evidence or would be adopted by more people. Despite the increase in popularity of using a forced choice design for assessing quality of reasoning (QoR), no study to date has established the validity and reliability of such a method. In two studies, we simultaneously presented two products of reasoning to participants and asked them to identify which product was ‘better justified’ through a forced choice design. We investigated the criterion validity and inter-rater reliability of the forced choice protocol by assessing the relationship between QoR, measured using the forced choice protocol, and accuracy in objectively answerable problems using naive raters sampled from MTurk (Study 1) and experts (Study 2), respectively. In both studies products that were closer to the correct answer and products generated by larger teams were consistently preferred. Experts were substantially better at picking the reasoning products that corresponded to accurate answers. Perhaps the most surprising finding was just how rapidly raters made judgements regarding reasoning: On average, both novices and experts made reliable decisions in under 15 seconds. We conclude that forced choice is a valid and reliable method of assessing QoR.

CogSci 2022

Measuring Quality of General Reasoning

poster

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





The study presents the experimentally developed technique to determine how and where semantic categories are represented in the human brain. For this purpose, the combination of linguistic annotation of oral texts in Russian with the registration of BOLD signal in functional MRI subjects’ testing was applied. The goal is to correlate groups of words in input texts and patterns of brain activation. The algorithm of the texts’ markup consisted of four sequential procedures: (1) time annotation, (2) lemmatization, (3) vectorization, (4) indexation of words-attributes.
Regarding the issue of brain activity allocation, the corresponding activation distributed nature can be noted. It proves our hypothesis that conceptual information during text comprehension is processed in different (however specific) areas of the brain and is not compactly localized.
The main result is that the allocated lexical clusters are composed of paradigmatically close words (synonyms, hyponyms) and situationally related words.

Russian-language semantic categories and their brain activity correspondence

In the study of human language comprehension, the N400 brain potential is often used as an index of on-line processing of meaning. However, a more precise account of its underlying computational process is still debated. Previous neural network modelling work simulated N400 amplitudes as change in a probabilistic representation corresponding to an internal implicit prediction error or Bayesian surprise at the level of meaning (Rabovsky, Hansen, & McClelland, 2018). Here we further test this account using a Bayesian learner model to simulate semantic processing of nouns from different semantic categories presented in an oddball like roving paradigm. We show that Bayesian surprise at the level of meaning significantly predicts N400 amplitudes and does so to a significantly higher degree than a non-Bayesian baseline. This sets the N400 in relation to ERP mismatch negativities in perceptual oddball paradigms, which have featured prominently in Bayesian accounts of brain function.

N400 amplitudes as Bayesian surprise at the level of meaning

Several popular sequence-based and pretrained language models have been found to be successful for text-driven prediction of brain activations. However, these models still lack long-term cognitive plausibility as well as insights on the underlying neural substrate mechanisms. This paper studies the influence of context representations of different language models such as sequence-based models: Long short-term memory networks (LSTMs) and ELMo, and popular pretrained Transformer language model (Longformer). In particular, we study how the internal hidden representations align with the brain activity observed via fMRI when the subjects listen to several narrative stories. We use brain imaging recordings of subjects listening to narrative stories to interpret word and sequence embeddings. We further investigate how the representations of language models layers reveal better semantic context during listening. Experiments across all language model representations provide the following cognitive insights: (i) the representations of LSTM cell states are better aligned with brain recordings than LSTM (hidden state), the cell state activity can represent more long-term information, (ii) the representations of ELMo and Longformer display a good predictive performance across brain regions for listening stimuli; (iii) Posterior Medial Cortex (PMC), Temporo-Parieto-Occipital junction (TPOJ), and Dorsal Frontal Lobe (DFL) have higher correlation versus Early Auditory (EAC) and Auditory Association Cortex (AAC).

Long-Term Plausibility of Language Models and Neural Dynamics during Narrative Listening

Sequential learning in artificial neural networks is known to trigger catastrophic interference (CI), where previously learned skills are forgotten after learning new skills. This is in direct contrast to humans’ ability to learn increasingly complex skills across the lifespan without major instances of CI. The present work builds on techniques for mitigating CI that have been proposed in prior work. Anderson and Schooler (1991) first documented that the memory environment has a lawful structure. Following from their observation, we constructed a training environment where previously mastered tasks (Boolean functions) decrease in frequency over time according to a power law. It was predicted that training in this environment would (1) mitigate CI, (2) replicate human performance in learning curves following a power law of practice, and (3) promote positive transfer of training to new skills, all without the need to posit additional mechanisms. The present results support all three predictions.

Catastrophic interference in neural network models is mitigated when the training data reflect a power-law environmental structure

Research shows the existence of a small-number system (~1-3) invoking parallel individuation, and a large-number system (~4+) based on Weberian magnitude estimation. A 128-channel EEG system was used to investigate ERPs while participants detected numerical changes in dot arrays during an oddball task.
During “No Change” trials, the N170 over the parietal-occipital-temporal areas showed distinct waveforms to the habituation of small (but not large) numbers, indicating that more items encoded in working memory leads to stronger N170 amplitudes. We also observed decreased accuracy for increasing vs. decreasing changes in magnitude.
Converging over the mid-parietal area (Pz), lower P3b amplitudes were observed for harder, numerically-larger conditions, while Increasing-Large changes showed longer reaction times compared to Decreasing-Large.
Our findings suggest a neurobehavioral differentiation in perceiving small vs. large numbers and increasing vs. decreasing change at early stages of processing, and a later stage that involves higher-order numerical processing linked to context-updating.

The Neurobehavioral Basis of Parallel Individuation and Numerical Approximation

Research shows that participants predict their memory performance to be lower when they experience disfluencies during encoding, even though encoding disfluency does not always affect actual memory performance. Bizarre statements are typically encoded slower than common statements, which constitutes an example of an encoding disfluency. The current study investigated how disfluencies during encoding for bizarre and common statements affect actual and predicted memory performance from a metacognitive perspective. Across two experiments under intentional learning instructions, participants made either memory predictions or vividness ratings for bizarre and common statements, followed by a free recall task. Participants predicted to remember common statements more than bizarre statements for both Experiment 1 (self-paced encoding) and Experiment 2 (experimenter-paced encoding), even though the actual memory performance was higher for bizarre than common statements. This demonstrates a metacognitive illusion for the bizarreness effect, similar to other manipulations of encoding disfluency.

Bizarreness Effect and Its Relation to Memory and Metamemory

The new Sans Forgetica (SF) typeface was designed to promote desirable difficulty. Here, we investigate whether SF improves memory for words with within-subject designs. Participants studied words in Arial and SF (Exp 1 and 2) and completed old-new recognition tests where words retained their study fonts (Exp 1) or were in either Arial or SF (Exp 2). They had significantly better recognition (hit rate) in SF than in Arial (Exp 1) and significantly higher sensitivity indexes (d’) when words were tested in SF than in Arial (Exp 2). While encouraging, further examination of these results (e.g., response bias) suggest a less straightforward interpretation. Thus, we have reservations for the effectiveness and use of SF for improving word memory.

Does Sans Forgetica font facilitate word memory?

Making judgments of learning (JOLs) can sometimes directly modify subsequent memory, which is termed JOL reactivity. To explain this phenomenon, the changed-goal hypothesis posits that JOLs increase participants’ awareness of differences in item difficulties and prompt them to focus more on easier items at the cost of harder items. The cue-strengthening hypothesis states that making JOLs enhances cues that inform JOLs and benefits memory performance if memory tests are sensitive to those cues. In Experiment 1, we found that immediate JOLs produced positive reactivity for related word pairs but prestudy JOLs did not, although both types of JOLs signaled differences in item difficulty. In Experiment 2, we manipulated target-target relatedness between consecutive word pairs and administered either free or associative recall tests. Only list-level but not item-level JOLs produced positive reactivity for target-target related pairs with free recall. Thus, both experiments support the cue-strengthening hypothesis more than the changed-goal hypothesis.

Changed-goal or cue-strengthening? An investigation into the underlying mechanisms of judgment of learning reactivity

Studying false recognition is highly important not only in furthering our general understanding of memory but also in applied situations like eyewitness testimony. The Deese Roediger McDermott (DRM) paradigm, utilizing study lists of words all semantically related to a theme word that is not actually presented, has been shown to be highly likely to produce false recognition of the theme word (critical lure) at test (Roediger & McDermott, 1995). Prior studies have aimed to combat this error by providing feedback, however the feedback provided was either delayed or only given for errors to the critical lure. Our study provides immediate feedback for every memory judgment given in a DRM recognition task, allowing subjects to evaluate their active memory traces. The results suggest that immediate feedback (compared to delayed) is more effective at reducing false recognitions, possibly due to a learned reliance on memory features that are diagnostic of true memories.

Immediate Feedback Decreases False Recognition in the DRM Paradigm

Failing to come up with a word or name is a fairly common experience that is exacerbated in older adulthood and among populations with language impairments, and yet the mechanisms underlying lexical retrieval remain fairly understudied. In this work, we introduce and evaluate a series of nested computational models of lexical retrieval that combine semantic representations derived from a distributional semantic model with a process model to account for behavioral performance in a primed lexical retrieval task. The models were tested on a behavioral data set where participants attempted to retrieve answers to descriptions of low-frequency words and were provided a semantically and/or phonologically related prime word before the retrieval attempt. Model comparisons indicated that a model that emphasized semantic activations from the description and phonological activations from the prime word best accounted for the overall data. Additionally, incorrect responses and metacognitive judgments indicating that participants had other words in mind were associated with models that instead emphasized semantic activations from the prime word. Taken together, these results identify the locus of lexical retrieval failures and offer the opportunity to investigate broader questions about semantic memory retrieval.

Premium content

Measuring Quality of General Reasoning

Downloads

Next from CogSci 2022

Russian-language semantic categories and their brain activity correspondence

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES