Thailand

Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master&#39;s role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game &quot;Jim Henson&#39;s Labyrinth: The Adventure Game.&quot; Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.

ACL 2024

You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

text adventure game

tabletop role-playing game

function calling

large language model

artificial intelligence

natural language processing

Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master's role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game "Jim Henson's Labyrinth: The Adventure Game." Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we compare how the strategies of daily word-guessing game players have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length, and generate repetitions, as well as hallucinations of non-existent words and inflections.

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

DAGGER is a synthetically generated dataset for creating text adventure games from fiction and for generating prose from game states. It extends the LIGHT dataset by enhancing the original game locations with new characters and items generated using GPT-4. Additionally, each location is paired with a piece of short fiction, totaling 259,582 words across the entire dataset. DAGGER was created using a novel methodology that leverages the capabilities of GPT-4 and a new tool called DataDreamer, which simplifies the implementation of complex prompting workflows and enables the generation of high-quality synthetic training data. The DAGGER dataset is used to train two baseline models: one that converts game states into fictional narrative descriptions and another that predicts game states from fiction. These models demonstrate the usefulness of the dataset for tasks involving the mapping between stories and playable text adventure games. We make the dataset, the trained models, and a Python game engine publicly available to facilitate further research.

DAGGER: Data Augmentation for Generative Gaming in Enriched Realms

The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback.\footnote{The minigame's code and the user study's game logs and video recordings will be publicly released on acceptance.} On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Text-based adventure game generation has attracted considerable interest since the advancements in large language models. However, even state-of-the-art (SOTA) large language models (LLMs) face challenges in generating semantically and logically coherent content, and often fail to align with human intentions. In our work, we propose a approach for enhancing content coherence, additionally allowing humans in the loop to better match their intentions. Specifically, we generate game components, including locations, characters, items, actions, and blocks, and integrate these components with semantic and logical constraints. Meanwhile, human game creators can intervene and modify the components to suit their preferences. Evaluation results demonstrate that our procedural world-generation approach both enriches the content and satisfies semantic and logical requirements compared to the SOTA LLMs. Moreover, the human-in-the-loop feature gives game creators more control over generated content.

WorldWeaver: Procedural World Generation for Text Adventure Games using Large Language Models

We formally introduce a improvisational wordplay game called Connections to explore reasoning capabilities of AI agents. Playing Connections combines skills in knowledge retrieval, summarization and awareness of cognitive states of other agents. We show how the game serves as a good benchmark for social intelligence abilities of language model based agents that go beyond the agents' own memory and deductive reasoning and also involve gauging the understanding capabilities of other agents. Finally, we show how through communication with other agents in a constrained environment, AI agents must demonstrate social awareness and intelligence in games involving collaboration.

Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

We explore the strategic capabilities of generative agents in a competitive game that simulates the television show Survivor. Large Language Models have been shown to act as intelligent agents through the addition of external cognitive architectures, but it is still unknown how these agents perform in competitive, multi-agent environments. We suggest a framework, built on top of a frozen large language model, GPT-4, for designing generative agents in competitive, episodic environments and we evaluate their game performance. We provide new modules allowing agents to set and evaluate strategic goals, develop theories of mind about other agents, and embody descriptive personas that affect their behavior. Across many simulations, we show that agents with varied cognitive abilities exhibit altered behaviors and that goal-driven agents display an emergent capacity to rapidly utilize information perceived from their environment.

Outwit, Outplay, Out-Generate: A Framework for Designing Strategic Generative Agents in Competitive Environments

Interactive fiction (IF) games are a genre of games where the player interacts with the fictional world via text-based commands, solving puzzles primarily by exploring the world and using items they collect along the way. Although there has been much work on playing IF using AI, there is relatively less work on the creation of such games using AI. While large language models (LLMs) have made the generation of text far easier in the past several years, they still struggle to be able to generate such highly-structured and consistent story worlds that one might see in IF. We present a three-part system called BERALL, which generates unique text adventure games by 1) maintaining the current state story world, 2) using retrieval-augmented generation (RAG) to create relevant location descriptions, and 3) combining these components to create a coherent experience for the player. Our results show that our approach is effective at generating room and story descriptions from setting and knowledge graphs, demonstrating the benefits of LLMs in IF. We find challenges remain in maintaining current game state due in part to LLMs not understanding the impact of actions on knowledge graphs.

BERALL: Generating Retrieval-augmented State-based Interactive Fiction Games

Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments, yet their creativity remains underexplored. This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs. In Balderdash, players generate fictitious definitions for obscure terms to deceive others while identifying correct definitions. Our framework enables multiple LLM agents to participate in this game, assessing their ability to produce plausible definitions and strategize based on game rules and history. We implemented a centralized game engine featuring various LLMs as participants and a judge LLM to evaluate semantic equivalence. Through a series of experiments, we analyzed the performance of different LLMs, examining metrics such as True Definition Ratio, Deception Ratio, and Correct Guess Ratio. The results provide insights into the creative and deceptive capabilities of LLMs, highlighting their strengths and areas for improvement. Specifically, the study reveals that infrequent vocabulary in LLMs' input leads to poor reasoning on game rules and historical context.

Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash

Text-adventure games and text role-playing games are grand challenges for reinforcement learning game playing agents. Text role-playing games are open-ended environments where an agent must faithfully play a particular character. We consider the distinction between {\em characters} and {\em actors}, where an actor agent has the ability to play multiple characters. We present a framework we call a {\em thespian agent} that can learn to emulate multiple characters along with a soft prompt that can be used to direct it as to which character to play at any time. We further describe an attention mechanism that allows the agent to learn new characters that are based on previously learned characters in a few-shot fashion. We show that our agent outperforms the state of the art agent framework in multi-character learning and few-shot learning.

Thespian: Multi-Character Text Role-Playing Game Agents

Strategic social deduction games serve as valuable testbeds for evaluating the understanding and inference skills of language models, offering crucial insights into social science, artificial intelligence, and strategic gaming. This paper focuses on creating proxies of human behavior in simulated environments, with “Among Us” utilized as a tool for studying simulated human behavior. The study introduces a text-based game environment, named AmongAgents, that mirrors the dynamics of “Among Us”. Players act as crew members aboard a spaceship, tasked with identifying impostors who are sabotaging the ship and eliminating the crew. Within this environment, the behavior of simulated language agents is analyzed. The experiments involve diverse game sequences featuring different configurations of Crewmates and Impostor personality archetypes. Our work demonstrates that state-of-the-art large language models (LLMs) can effectively grasp the game rules and make decisions based on the current context. This work aims to promote further exploration of LLMs in goal-oriented games with incomplete information and complex action spaces, as these settings offer valuable opportunities to assess language model performance in socially driven scenarios.

Premium content

You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

Downloads

Next from ACL 2024

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES