When given the opportunity, humans naturally engage in anthropomorphism, which may reflect a bias to engage in mentalistic attributions in understanding social interactions. In this experiment, we evaluate whether neural activity in social perceptual brain regions can be explained by perceptual cues of agency and interactivity, or by semantic models of written descriptions of Heider-Simmel style animations. Models were compared in representational similarity space using variance partitioning of the neural response from the STS, TPJ, and PCC. The right STS and TPJ were best explained by perceptual models of distance between the agents, an indicator of interactivity, and separately by the similarity structure of the free responses, which captured both action and interaction terms. Together, these results implicate the importance of contextual framing, either through perceptual features of interactivity or social context as implied by the nature of interactions, as defining features in neural representations of interactivity.