Humans tend to privilege an intermediate level of categorization, known as the basic level, when categorizing objects that exist in a conceptual hierarchy. Domain experts demonstrate a downward shift in their object categorization behaviour, recruiting subordinate levels in a conceptual hierarchy as readily as conventionally basic categories (Tanaka & Philibert, 2022; Tanaka & Taylor, 1991). Do large language models show similar behavioural changes when prompted to behave in an expert-like way? We test whether GPT-4 with Vision (GPT-4V, OpenAI, 2023a) demonstrates downward shifts using an object naming task and eliciting expert personas using the model’s system prompt. We find evidence of downward shifts when expert system prompts are used, suggesting that expert-like behaviour can be elicited from GPT-4V using prompting. We also find that there is an unpredicted upward shift in areas of non-expertise in some cases. These findings suggest that in the default case, GPT-4V is not a novice: instead, it behaves at default with a median level of expertise, while further expertise can be primed or forgotten through textual prompts. These results open the door for GPT-4V and similar models to be used as tools for studying differences in the behaviour of experts and novices even within the same large language model.

Authors:

Cara Su-Yi Leong: New York University; Brenden Lake: NYU