From Images to Symbols: Drawing as a Window into the Mind (W3.6)

The ability to represent semantic structure in the environment — objects, parts, and relations — is a core aspect of human visual perception and cognition. Here we leverage recent advances in program synthesis to develop an algorithm for learning the part-based structure of drawings as represented by graphics programs. This algorithm iteratively learns a library of abstract subroutines that can be used to more compactly represent a set of drawings by capturing common structural elements. Our experiments explore how this algorithm exploits statistical reg- ularities across drawings to learn new subroutines. Together, these findings highlight the potential for understanding human visual concept learning via program-like abstractions.

Learning part-based abstractions for visual object concepts

What visualization strategies do people use to communicate abstract knowledge to others? We developed a drawing paradigm to elicit visual explanations about novel machines and obtained detailed annotations of the semantic information conveyed in each drawing. We found that these visual explanations contained: (1) greater emphasis on causally relevant parts of the machine, (2) less emphasis on structural features that were visually salient but causally irrelevant, and (3) more symbols, relative to baseline drawings intended only to communicate the machines' appearance. However, this overall pattern of emphasis did not necessarily improve naive viewers' ability to infer how to operate the machines, nor their ability to identify them, suggesting a potential mismatch between what people believe a visual explanation contains and what may be most useful. Taken together, our findings advance our understanding of how communicative goals constrain visual communication of abstract knowledge across behavioral contexts.

How do the semantic properties of visual explanations guide causal inference?

Symbolic representations of number are instrumental to mathematical reasoning and many aspects of social organization. What explains their emergence in human cultures? To understand how functional and cognitive constraints impact people’s communication about number, we used a drawing-based reference game to investigate how human dyads coordinated to form novel number systems. We found a systematic bias towards symbols exploiting 1-to-1 correspondence to objects in visual arrays, and that this strategy was contingent on the communicative relevance of number. Moreover, the meaning of these symbols was transparent to third party observers not present during their production. Finally, model-based analyses of these symbols' visual properties suggest that the ability to decode exact quantity from them may rely on perceptual processing mechanisms beyond those sufficient for object recognition. These findings contribute to our understanding of how both communicative need and capacity for visual abstraction constrain the emergence of iconic representations of exact number.

Improvised Numerals Rely on 1-to-1 Correspondence

Compositionality is a core feature of human cognition and behavior. People readily decompose visual objects into parts and complex procedures into subtasks. Here we investigate how these two abilities interact to support learning in a block-tower assembly experiment. We measured the way participants segmented these towers based on shape information alone, and asked how well the resulting parts explained the procedures other participants used to build them. We found that people decomposed these shapes in consistent ways and the most common parts appeared especially frequently as subroutines in the assembly experiment. Moreover, we found that the subroutines participants used converged over time, reflecting shared biases toward certain ways of reconstructing each tower. More broadly, our findings suggest important similarities between the perceptual and procedural abstractions humans use to perceive and interact with objects in their environment.

Connecting perceptual and procedural abstractions in physical construction

What computations enable humans to leap from mere obser- vations to rich explanatory theories? Prior work has focused on stochastic algorithms that rely on random, local perturba- tions to model the search for satisfactory theories. Here we introduce a new approach inspired by the practice of ‘debug- ging’ from computer programming, whereby learners use past experience to constrain future proposals, and are thus able to consider large leaps in their current theory to fix specific de- ficiencies. We apply our ‘debugging’ algorithm to the mag- netism domain introduced by (Ullman, Goodman, & Tenen- baum, 2010) and compare its efficiency and accuracy to their stochastic-search algorithm. We find that our algorithm not only requires fewer iterations to find a solution, but that the solutions it finds more reliably recover the correct latent theo- ries, and are more robust to sparse data. Our findings suggest the promise of such constraint-based approaches to emulate the way humans efficiently navigate large, discrete hypothesis spaces.

Theory Acquisition as Constraint-Based Program Synthesis

Many real-world tasks require agents to coordinate their behavior to achieve shared goals. Successful collaboration requires not only adopting the same communicative conventions, but also grounding these conventions in the same task-appropriate conceptual abstractions. We investigate how humans use natural language to collaboratively solve physical assembly problems more effectively over time. Human participants were paired up in an online environment to reconstruct scenes containing two block towers. One participant could see the target towers, and sent assembly instructions for the other participant to reconstruct. Participants provided increasingly concise instructions across repeated attempts on each pair of towers, using more abstract referring expressions that captured each scene's hierarchical structure. To explain these findings, we extend recent probabilistic models of ad hoc convention formation with an explicit perceptual learning mechanism. These results shed light on the inductive biases that enable intelligent agents to coordinate upon shared procedural abstractions.

Learning to communicate about shared procedural abstractions

People can produce drawings of specific entities (e.g., Garfield), as well as general categories (e.g., ``cat''). What explains this ability to produce such varied drawings of even highly familiar object concepts? We hypothesized that drawing objects at different levels of abstraction depends on both sensory information and representational goals, such that drawings intended to portray a recently seen object preserve more detail than those intended to represent a category. Participants drew objects cued either with a photo or a category label. For each cue type, half the participants aimed to draw a specific exemplar; the other half aimed to draw the category. We found that label-cued category drawings were the most recognizable at the basic level, whereas photo-cued exemplar drawings were the least recognizable. Together, these findings highlight the importance of task context for explaining how people use drawings to communicate visual concepts in different ways.

Visual communication of object concepts at different levels of abstraction

Curiosity is a fundamental driver of human behavior, and yet because of its open-ended nature and the wide variety of behaviors it inspires in different contexts, it is remarkably difficult to study in a laboratory context. A promising approach to developing and testing theories of curiosity is to instantiate them in artificial agents that are able to act and explore in a simulated environment, and then compare the behavior of these agents to humans exploring the same stimuli. Here we propose a new experimental paradigm for examining children's -- and AI agents' -- curiosity about objects' physical interactions. We let them choose which object to drop another object onto in order to create the most interesting effect. We compared adults' (N=155) and children's choices (N=66; 3-7 year-olds) and found that both children and adults show a strong preference for choosing target objects that could potentially contain the dropped object. Adults alone also make choices consistent with achieving support relations. We contextualize our results using heuristic computational models based on 3D physical simulations of the same scenarios judged by participants.

Predicting children's and adults' preferences in physical interactions via physics simulation

Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgoals is computationally costly. What algorithms might enable us to reap the benefits of planning using subgoals while minimizing the computational overhead of selecting them? We propose visual scoping, a strategy that interleaves planning and acting by alternately defining a spatial region as the next subgoal and selecting actions to achieve it. We evaluated our visual scoping algorithm on a variety of physical assembly problems against two baselines: planning all subgoals in advance and planning without subgoals. We found that visual scoping achieves comparable task performance to the subgoal planner while requiring only a fraction of the total computational cost.Together, these results contribute to our understanding of how humans might make efficient use of cognitive resources to solve complex planning problems.

Visual scoping operations for physical assembly

Curiosity drives much of human behavior, but its open-ended nature makes it hard to study in the laboratory. Moreover, computational theories of curiosity -- models of how intrinsic motivation promotes complex behaviors -- have been challenging to test because of technical limits. To circumvent this problem, we develop a new way to assess intrinsic motivation for building: we assume people build what they find interesting, so we asked them to rate the "interestingness" of visual stimuli -- in this case, simple block towers.
Adults gave a range of ratings to towers built by children, with taller towers rated higher.
To probe interestingness further, we developed controlled tower stimuli in a simulated 3D environment. While tower height predicted much of the variation in ratings, people also favored more precarious towers, as inferred from geometric features and simulated dynamics. These ratings and features therefore give a clear target for computational accounts of curiosity to explain.

Measuring and predicting variation in the interestingness of physical structures

**Organizers and Presenters:** Kushin Mukherjee, University of Wisconsin-Madison; Holly Huey, University of California, San Diego; Timothy Rogers, University of WisconsinMadison; Judith Fan, University of California, San Diego
 
 
Drawing is a powerful cognitive technology for creating external representations of thought. While drawings have long provided inspiration to researchers in many areas of cognitive science, including psychology, artificial intelligence, and neuroscience, these communities have not generally had opportunities to interact and share insights. The goal of this workshop is to bring together perspectives from multiple disciplines to explore the question of how humans use drawings to communicate knowledge, catalyzing new opportunities for multidisciplinary collaboration. We are introducing a novel “flipped” format wherein we will hold three virtual seminars in the weeks leading up to CogSci 2022, each highlighting insights from Machine Learning, Neuroscience, and Developmental Science, respectively. Holding these thematic seminars in advance will enable us to attract a broader audience for our event and focus on promoting informal interaction among workshop attendees at the on-site event.

Workshop 3: From images to symbols: Drawing as a window into the mind

workshop paper

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
 
 [**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
 [**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

CogSci 2022

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 






**For a full list of posters in this session, click the See All link on the right side of the page.**

Poster Session 1 -  A: AI, Computer Science and Computer Models

poster

### Looking for Cogsci 2023? Please [click here](https://underline.io/events/405/reception) to switch to 2023 edition.


**Dear Colleagues,**

Welcome to the 43rd Annual Conference of the Cognitive Science Society! 

This year’s conference brings together the latest research from all Cognitive Science and also highlights this year’s theme **Comparative Cognition: Animal Minds.** 

The program features four plenary speakers: Friederike Range (University of Veterinary Medicine, Vienna), Onur Güntürkün (Ruhr University Bochum), Julia Fischer (University of Göttingen), and Erich Jarvis (The Rockefeller University). The keynotes and invited symposia showcase the broad spectrum of ideas that encompass the conference theme of comparative cognition. 

Further, the program includes the Rumelhart Prize keynote presentation by Susan Goldin-Meadow (University of Chicago), and symposia to celebrate the winner of the Elman Prize: Wei Ji Ma (New York University), the Heineken Award winner Robert Zatorre (McGill University), and the Robert J. Glushko Dissertation Prizes session.

The program committee for CogSci 2021 received 1,025 submissions, including 710 full papers, 275 member abstracts, 8 publication-based talks, as well as 15 proposals for symposia, and 17 for workshops and tutorials. After a rigorous review process, the committee selected 152 papers for oral presentation and inclusion in the conference proceedings (14.8%), 332 papers for poster presentation and inclusion in the proceedings (32.4%), and 175 papers for poster presentation with inclusion of abstracts in the proceedings (17.1%) We also selected 275 submitted member abstracts. In addition, we accepted 6 publication-based talks, 12 symposia, and 9 workshops and tutorials to make for an exciting and inclusive program.

We hope that you enjoy the conference this year! 

**Your Program Chairs,**
![](https://assets.underline.io/uploads/markdown_image/1/image/99984908caba5e6453e5fc5ba34f3507.jpg)

**Learn more about the Vienna Cognitive Science Hub at the University of Vienna, hosts of CogSci 2021**

<iframe width="560" height="315" src="https://www.youtube.com/embed/464h1m3iU68" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

To review the proceedings for the conference or review the searchable program, please visit the [website](https://cognitivesciencesociety.org/program/).

CogSci 2021

The Cognitive Science Society brings together researchers from around the world who hold a common goal: understanding the nature of the human mind. The mission of the Society is to promote Cognitive Science as a discipline, and to foster scientific interchange among researchers in various areas of study, including Artificial Intelligence, Linguistics, Anthropology, Psychology, Neuroscience, Philosophy, and Education. The Society is a non-profit professional organization and its activities include sponsoring an annual conference and publishing the journals Cognitive Science and TopiCS.

**Date: **Wednesday, July 28, 2021 
**Time: **17:20 CEST - 19:00 CEST 
**Session Description: ** Review the posters you are interested in, watch the videos, and even leave messages for the authors in the Q&A section before joining the poster presenters and exhibitors in Gather.town for live discussion. 
**For a full list of posters in this session, click the See All link on the right side of the page.**

Poster Session 2 -  D: Education, Development and Perspectives in Cognitive Science

Poster Session 3 -  A: AI, Computer Science and Computer Models

**Session Chair: **Agnieszka Konopka 
**Technical Session Chair: **Kimia Goderle 
**Session Description: ** The session is comprised of 15 minute talks followed by up to 5 minutes of live Q&A time for each presenter.

IP2: Communication

technical paper

**Session Chair: **Daphna Buchsbaum 
**Technical Session Chair: **Alon Hafri 
**Session Description: ** The session is comprised of 15 minute talks followed by up to 5 minutes of live Q&A time for each presenter.

IP8: Development models

**Session Chair: **Vikranth Bejjanki 
**Technical Session Chair: **Abby Clements 
**Session Description: ** The session is comprised of 15 minute talks followed by up to 5 minutes of live Q&A time for each presenter.

IP28: Perception 2

**Session Chair: **Caren Walker 
**Technical Session Chair: **Jonathan Kominsky 
**Session Description: ** The session is comprised of 15 minute talks followed by up to 5 minutes of live Q&A time for each presenter.

Judith Fan

11

14

2

Presentations

From Images to Symbols: Drawing as a Window into the Mind (W3.6)

Learning part-based abstractions for visual object concepts

How do the semantic properties of visual explanations guide causal inference?

Improvised Numerals Rely on 1-to-1 Correspondence

Connecting perceptual and procedural abstractions in physical construction

Theory Acquisition as Constraint-Based Program Synthesis

Learning to communicate about shared procedural abstractions

Visual communication of object concepts at different levels of abstraction

Predicting children's and adults' preferences in physical interactions via physics simulation

Visual scoping operations for physical assembly

Measuring and predicting variation in the interestingness of physical structures

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES