Thailand

In this work we examine the use of Large Language Models (LLMs) in the challenging setting of acting as a Minecraft agent. We apply and evaluate LLMs in the builder and architect settings, introduce clarification questions and examining the challenges and opportunities for improvement. In addition, we present a platform for online interaction with the agents and an evaluation against previous works.

ACL 2024

Large Language Models as Minecraft Agents

llms

conversational agents

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructions across a diverse range of virtual 3D environments, including curated research environments as well as open-ended, commercial video games. Our goal is to develop an instructable agent that can accomplish anything a human can do in any simulated 3D environment. In this paper we describe our motivation and goal, the initial progress we have made, and promising preliminary results on several diverse research environments and a variety of commercial video games.

Scaling Instructable Agents Across Many Simulated Worlds

The popularity of role-playing video games (RPGs) continues to increase over time. These types of games often afford players multiple options for how they wish to accomplish any given task, which adds to their appeal. Most commonly, RPGs employ a finite state dialogue management system taking the form of a graph structure, where the set of possible utterances at a given node depend on the current game state. From the player's perspective, these appear as a list of pre-written dialogue options that can be chosen to advance the conversation, thus limiting a player's role-playing to the available options. In this paper, we investigate a hybrid dialogue management approach, which allows players more freedom while staying true to the dialogue graph created by the game's designers. Instead of seeing a list of options, players are provided an open-ended text field. A virtual game master (GM) is then tasked with either mapping the player input to one of the dialogue options, or providing in-character feedback to guide players into typing an utterance which the virtual GM can successfully map to an dialogue option. As a preliminary study, we investigate simulated players with a suite of virtual GMs, including a random baseline within the video game Disco Elysium: The Final Cut.

Hybrid Dialogue Management for Task-Oriented Dialogue in Video Games

We present PLUGH a modern benchmark that currently consists of 5 tasks, each with 125 input texts extracted from 48 different games and representing 61 different (non-isomorphic) spatial graphs to assess the abilities of Large Language Models (LLMs) for spatial understanding and reasoning. Our evaluation of API-based and open-sourced LLMs shows that while some commercial LLMs exhibit strong reasoning abilities, open-sourced competitors can demonstrate almost the same level of quality; however, all models still have significant room for improvement. We identify typical reasons for LLM failures and discuss possible ways to deal with them. Datasets and evaluation code are released.

PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models

In a conversation between two parties, the speaker intends to convey a concept through language while the listener interprets this utterance based on context including their sociocultural background. Differences in culture may lead to differences in interpretations of the meaning of words, resulting in "pragmatic failure": for example, "football" refers to two distinct sports in the United States and in Europe. We seek to understand how language agents may overcome this lack of common ground through the collaborative game Codenames Duet \cite{cross-cultural-codes}, where players have constrained communication and must overcome conceptual and pragmatic misunderstandings to win. By adapting the Rational Speech Acts (RSA) framework \cite{GOODMAN2016818} to model efficient communication and pragmatic reasoning while simulating cross-cultural communication, we introduce a new method RSA+C3 that is able to adapt to players of different cultures without shared common ground. Our contributions towards resolving pragmatic failure in the communicative environment of Codenames Duet are: (1) modeling Codenames players through contrastive learning of an embedding space and LLM prompting, (2) studying cultural differences reflected in our trained models, and (3) demonstrating that our method RSA+C3 can ease cross-cultural communication in gameplay by inferring sociocultural context from interaction.

Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames

State-of-the-art Large Multi-Modal Models (LMMs) have demonstrated exceptional capabilities in vision-language tasks. Despite their advanced functionalities, the performances of LMMs are still limited in challenging scenarios that require complex reasoning with multiple levels of visual information. Existing prompting techniques for LMMs focus on either improving textual reasoning or leveraging tools for image preprocessing, lacking a simple and general visual prompting scheme to promote vision-language coordination in LMMs. In this work, we propose Scaffold prompting that scaffolds coordinates to promote vision-language coordination. Specifically, Scaffold overlays a dot matrix within the image as visual information anchors and leverages multi-dimensional coordinates as textual positional references. Extensive experiments on a wide range of challenging vision-language tasks demonstrate the superiority of Scaffold over GPT-4V with the textual CoT prompting.

Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models

Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master's role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game "Jim Henson's Labyrinth: The Adventure Game." Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.

You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we compare how the strategies of daily word-guessing game players have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length, and generate repetitions, as well as hallucinations of non-existent words and inflections.

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

DAGGER is a synthetically generated dataset for creating text adventure games from fiction and for generating prose from game states. It extends the LIGHT dataset by enhancing the original game locations with new characters and items generated using GPT-4. Additionally, each location is paired with a piece of short fiction, totaling 259,582 words across the entire dataset. DAGGER was created using a novel methodology that leverages the capabilities of GPT-4 and a new tool called DataDreamer, which simplifies the implementation of complex prompting workflows and enables the generation of high-quality synthetic training data. The DAGGER dataset is used to train two baseline models: one that converts game states into fictional narrative descriptions and another that predicts game states from fiction. These models demonstrate the usefulness of the dataset for tasks involving the mapping between stories and playable text adventure games. We make the dataset, the trained models, and a Python game engine publicly available to facilitate further research.

DAGGER: Data Augmentation for Generative Gaming in Enriched Realms

The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback.\footnote{The minigame's code and the user study's game logs and video recordings will be publicly released on acceptance.} On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Text-based adventure game generation has attracted considerable interest since the advancements in large language models. However, even state-of-the-art (SOTA) large language models (LLMs) face challenges in generating semantically and logically coherent content, and often fail to align with human intentions. In our work, we propose a approach for enhancing content coherence, additionally allowing humans in the loop to better match their intentions. Specifically, we generate game components, including locations, characters, items, actions, and blocks, and integrate these components with semantic and logical constraints. Meanwhile, human game creators can intervene and modify the components to suit their preferences. Evaluation results demonstrate that our procedural world-generation approach both enriches the content and satisfies semantic and logical requirements compared to the SOTA LLMs. Moreover, the human-in-the-loop feature gives game creators more control over generated content.

Premium content

Large Language Models as Minecraft Agents

Downloads

Next from ACL 2024

Scaling Instructable Agents Across Many Simulated Worlds

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES