Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world. Grounded language models have had success in learning to connect concrete categories like nouns and adjectives to the world via images and videos, but can struggle to isolate the meaning of verbs from the context in which they typically occur, e.g. a person walking across a tennis court is labeled “swinging a tennis racket". We are interested in building representations of trajectories (i.e. position and rotation of objects over time), and evaluating the extent to which they encode verb semantics. To achieve this, we build a procedurally generated agent-object-interaction dataset, and obtain human annotations for the verbs that occur. We find that trajectory data correlates as-is with verbs like fall, and can be abstracted through self-supervised pretraining to representations that capture nuanced differences in verb meaning, such as roll vs. slide, while falling short on other distinctions like fall vs. drop.

Do Trajectories Encode Verb Meaning?

We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal child is likely to have access: That is, naturalistic, spontaneous speech paired with richly grounded visuospatial context. We use the collected data to compare several distributional semantics models for verb learning. We evaluate neural models based on 2D (pixel) features as well as feature-engineered models based on 3D (symbolic, spatial) features, and show that neither modeling approach achieves satisfactory performance. Our results are consistent with evidence from child language acquisition that emphasizes the difficulty of learning verbs from naive distributional data. We discuss avenues for future work on cognitively-inspired grounded language learning, and release our corpus with the intent of facilitating research on the topic.

Dylan Ebert

2

2

SHORT BIO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES