Distributional models (such as neural network language models) have been successfully used to model a wide range of linguistic semantic behaviors. However, they lack a way to distinctly represent different kinds of semantic relations within a single semantic space. Here, we propose that neural network language models can sensibly be interpreted as representing syntagmatic (co-occurrence) relations using their input-output mappings, and as representing paradigmatic (similarity) relations using the similarity of their internal representations. We tested and found support for this hypothesis on four neural network architectures (SRNs, LSTMs, Word2Vec and GPT-2) using a carefully constructed artificial language corpus. Using this corpus, we show that the models display interesting but understandable differences in their ability to represent these two kinds of relationships. This work demonstrates distributional models can simultaneously learn multiple kinds of relationships, and that systematic investigation of these models can lead to a deeper understanding of how they work.

Authors:

Jingfeng Zhang: University of Illinois Urbana Champaign; Jon Willits: University of Illinois at Urbana-Champaign