Canada

When we read research findings, what facilitates consideration of different possible outcomes? 112 Amazon Mechanical Turk Workers read about four psychological studies, and either predicted the results, received the results without explanation, or received the results with a plausible explanation. We hypothesized that 1. receiving results would increase the difficulty of explaining alternative outcomes, and that this would be amplified by receiving explanations, 2. As difficulty explaining alternatives increases, participants would be less surprised by actual results and predict higher likelihood of replication. We did not find the expected differences in difficulty across conditions. However, across all four research studies, surprise was negatively correlated with likelihood of replication, indicating that more surprising results are considered less likely to replicate. We consider reasons why our manipulation did not affect difficulty of explaining alternative outcomes, and discuss implications for how research can best be presented to promote consideration of alternative outcomes.

CogSci 2022

Alternative Outcomes, Surprise, and Predicting Replication of Psychological Research

poster

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





People often evaluate explanations by considering various ‘explanatory virtues’, such as an explanation’s simplicity (i.e. the number of unexplained causes referred to). Simplicity has been thought to guide these evaluations by providing a cue to the inputs of Bayesian inference (priors and likelihoods), thus indirectly helping compute the outputs: the posterior probability of an explanation being true. Yet simplicity may also play other, more direct, roles in explanation evaluations. While study 1 supported the mediating role of priors and likelihoods in people’s simplicity preferences, study 1 and 2 found that participants still preferred simpler explanations after statistically controlling for priors and likelihoods (either elicited or provided). These results suggest that simplicity guides explanation evaluations not just by providing cues to the inputs of Bayesian inference, but also by serving as a direct cue to the outputs of these inferences – perhaps providing a simplifying heuristic for these evaluations.

Simplicity beyond probability: Simplicity’s role in evaluating explanations goes beyond providing cues to priors and likelihoods

How do humans explain and cognize visual information? Why do AI explanations in radiology, despite their remarkable accuracy, fail to gain human trust? In a study of 13 radiology practitioners, we found that AI explanations of x-rays differ from human explanations in 3 ways. The first concerns visual reasoning and evidence: how humans get other humans to see an interpretation’s validity. Machine learned classifications lack this evidentiary grounding, and consequently XAI explanations like heat maps fail to meet many users' needs. The second concerns the varying needs of interlocutors. Predictably, explanations suitable for experts and novices differ; presuppositions on explainee knowledge and goals inform explanation content. Pragmatics matter. The third difference concerns how linguistic terms and phrases are used to hedge uncertainty. There is no reason XAI might not satisfy these human requirements. To do so, however, will require deeper theories of human explanation.

Cognitive Differences in Human and AI Explanation

In some contexts, human learning greatly exceeds what the sparsity of the available data seems to allow, while in others, it can fall short, despite vast amounts of data. This apparent contradiction has led to separate explanations of humans being equipped either with background knowledge that enhances their learning or with suboptimal mechanisms that hinder it. Here, we reconcile these findings by recognising learners can be uncertain about two structural properties of environments: 1) is there only one generative model or are there multiple ones switching across time; 2) how stochastic are the generative models. We show that optimal learning under these conditions of uncertainty results in learning trade-offs: e.g., a prior for determinism fosters fast initial learning but renders learners susceptible to low asymptotic performance, when faced with high model-stochasticity. Our results reveal the existence of optimal-paths-to-not-learning and reconcile within a coherent framework, phenomena previously considered disparate.

Optimal learning under structural environmental uncertainty reveals inherent learning trade-offs

How effective do observers expect other problem solvers to be? What makes a decision seem "human"?" We addressed this question in the context of the Traveling Salesman Problem (TSP), a decision problem in which a perfectly optimal solution is intractable, but for which various kinds of approximate solutions are available. We conducted a series of experiments involving both "production tasks" in which we asked subjects to solve the TSP, and "perception tasks", in which we asked subjects to judge others' solutions to the TSP, rating them for intelligence or humanness. Results suggest that observers expect human solutions to be less optimal than algorithmic solutions: observers expect human problem solvers to exhibit a combination of local and global solution criteria, and to use a short look-ahead window when choosing a solution. These results shed light on human models of other humans' minds, a fundamental problem in social interaction and robotics.

How Optimal is Too Optimal? Expectations About Performance in the Traveling Salesman Problem

Knowledge of repeating patterns is foundational for early mathematical thinking. While interventions that rely on direct instruction help children to master specific patterns, they often struggle to transfer this knowledge to new patterns. This paper investigates exploration as a learning strategy for abstract patterning and improving knowledge transfer. In a yoked, between-subjects design, 5- and 6-year-old children (n = 90) were tasked with finding up to three hidden stars in a repeating shape pattern (ABB). Exploration participants (n=45) explored pattern materials themselves, while demonstration participants (n=45) observed the experimenter revealing the contents of each shape in the same order as the exploration counterpart. In a transfer task, children saw the same repeating pattern with different stimuli. We found that 6-year-olds significantly outperformed 5-year-olds in the exploration condition and used more sophisticated patterning strategies. These findings suggest that exploration may support and extend older children’s emerging understanding of underlying repeating pattern units.

Exploration as a Learning Strategy to Support Children's Pattern Learning

Research suggests the processes involved in analogical transfer are not unlike those of insightful problem solving. This study tests whether encoding cross-domain analogies leads to more feelings of insight and higher rates of far transfer. On each trial in the present study, subjects induced a conceptual relationship from a set of three verbal analogy pairs (30 trials). Half of the trials had a semantically distant (i.e. expansive) final pair, and half of the trials had a semantically close (i.e. consistent) final pair. Subsequently, subjects attempted to complete a novel far analogy for each of the previously learned relations. Far transfer performance on new analogies was predicted by the semantic vector distance of the originally encoded relation. Subjects also reported significantly stronger aha moments at the moment of transfer for accurate transfer responses compared to inaccurate responses, which suggests that the aha response may provide metacognitive information about solution accuracy.

Far Analogical Transfer Is Associated with Aha Moments

The Aha-Accuracy effect refers to the finding that experiencing an Aha! moment is associated with reaching correct solutions on insight problems. Because this effect has generally been demonstrated with verbal problems, this study tested for this effect on spatial problems (matchstick arithmetic). In addition, this study also explored the effect of hints on the Aha! experience and the Aha-Accuracy effect. Overall, there was no Aha-Accuracy effect in the no-hint control condition. There was an Aha-Accuracy effect in the hint condition, but it was limited to problems with solutions that were not directly cued by the hint. When the hint was directly relevant for solution, then many participants were able to reach a correct solution without an Aha! experience. These findings provide evidence that providing hints may not simply increase the likelihood of reaching a solution, but it may also alter the Aha! experience.

Hints and the Aha-Accuracy Effect in Insight Problem Solving

Machine learning models that automatically assess reasoning quality are trained on human-annotated written products. These “gold-standard” corpora are typically created by prompting annotators to choose, using a forced choice design, which of two products presented side by side is the most convincing, contains the strongest evidence or would be adopted by more people. Despite the increase in popularity of using a forced choice design for assessing quality of reasoning (QoR), no study to date has established the validity and reliability of such a method. In two studies, we simultaneously presented two products of reasoning to participants and asked them to identify which product was ‘better justified’ through a forced choice design. We investigated the criterion validity and inter-rater reliability of the forced choice protocol by assessing the relationship between QoR, measured using the forced choice protocol, and accuracy in objectively answerable problems using naive raters sampled from MTurk (Study 1) and experts (Study 2), respectively. In both studies products that were closer to the correct answer and products generated by larger teams were consistently preferred. Experts were substantially better at picking the reasoning products that corresponded to accurate answers. Perhaps the most surprising finding was just how rapidly raters made judgements regarding reasoning: On average, both novices and experts made reliable decisions in under 15 seconds. We conclude that forced choice is a valid and reliable method of assessing QoR.

Measuring Quality of General Reasoning

The study presents the experimentally developed technique to determine how and where semantic categories are represented in the human brain. For this purpose, the combination of linguistic annotation of oral texts in Russian with the registration of BOLD signal in functional MRI subjects’ testing was applied. The goal is to correlate groups of words in input texts and patterns of brain activation. The algorithm of the texts’ markup consisted of four sequential procedures: (1) time annotation, (2) lemmatization, (3) vectorization, (4) indexation of words-attributes.
Regarding the issue of brain activity allocation, the corresponding activation distributed nature can be noted. It proves our hypothesis that conceptual information during text comprehension is processed in different (however specific) areas of the brain and is not compactly localized.
The main result is that the allocated lexical clusters are composed of paradigmatically close words (synonyms, hyponyms) and situationally related words.

Russian-language semantic categories and their brain activity correspondence

In the study of human language comprehension, the N400 brain potential is often used as an index of on-line processing of meaning. However, a more precise account of its underlying computational process is still debated. Previous neural network modelling work simulated N400 amplitudes as change in a probabilistic representation corresponding to an internal implicit prediction error or Bayesian surprise at the level of meaning (Rabovsky, Hansen, & McClelland, 2018). Here we further test this account using a Bayesian learner model to simulate semantic processing of nouns from different semantic categories presented in an oddball like roving paradigm. We show that Bayesian surprise at the level of meaning significantly predicts N400 amplitudes and does so to a significantly higher degree than a non-Bayesian baseline. This sets the N400 in relation to ERP mismatch negativities in perceptual oddball paradigms, which have featured prominently in Bayesian accounts of brain function.

Premium content

Alternative Outcomes, Surprise, and Predicting Replication of Psychological Research

Downloads

Next from CogSci 2022

Simplicity beyond probability: Simplicity’s role in evaluating explanations goes beyond providing cues to priors and likelihoods

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES