Thailand

At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we compare how the strategies of daily word-guessing game players have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length, and generate repetitions, as well as hallucinations of non-existent words and inflections.

ACL 2024

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

word game

wordle

llms

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

DAGGER is a synthetically generated dataset for creating text adventure games from fiction and for generating prose from game states. It extends the LIGHT dataset by enhancing the original game locations with new characters and items generated using GPT-4. Additionally, each location is paired with a piece of short fiction, totaling 259,582 words across the entire dataset. DAGGER was created using a novel methodology that leverages the capabilities of GPT-4 and a new tool called DataDreamer, which simplifies the implementation of complex prompting workflows and enables the generation of high-quality synthetic training data. The DAGGER dataset is used to train two baseline models: one that converts game states into fictional narrative descriptions and another that predicts game states from fiction. These models demonstrate the usefulness of the dataset for tasks involving the mapping between stories and playable text adventure games. We make the dataset, the trained models, and a Python game engine publicly available to facilitate further research.

DAGGER: Data Augmentation for Generative Gaming in Enriched Realms

The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback.\footnote{The minigame's code and the user study's game logs and video recordings will be publicly released on acceptance.} On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Text-based adventure game generation has attracted considerable interest since the advancements in large language models. However, even state-of-the-art (SOTA) large language models (LLMs) face challenges in generating semantically and logically coherent content, and often fail to align with human intentions. In our work, we propose a approach for enhancing content coherence, additionally allowing humans in the loop to better match their intentions. Specifically, we generate game components, including locations, characters, items, actions, and blocks, and integrate these components with semantic and logical constraints. Meanwhile, human game creators can intervene and modify the components to suit their preferences. Evaluation results demonstrate that our procedural world-generation approach both enriches the content and satisfies semantic and logical requirements compared to the SOTA LLMs. Moreover, the human-in-the-loop feature gives game creators more control over generated content.

WorldWeaver: Procedural World Generation for Text Adventure Games using Large Language Models

We formally introduce a improvisational wordplay game called Connections to explore reasoning capabilities of AI agents. Playing Connections combines skills in knowledge retrieval, summarization and awareness of cognitive states of other agents. We show how the game serves as a good benchmark for social intelligence abilities of language model based agents that go beyond the agents' own memory and deductive reasoning and also involve gauging the understanding capabilities of other agents. Finally, we show how through communication with other agents in a constrained environment, AI agents must demonstrate social awareness and intelligence in games involving collaboration.

Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

We explore the strategic capabilities of generative agents in a competitive game that simulates the television show Survivor. Large Language Models have been shown to act as intelligent agents through the addition of external cognitive architectures, but it is still unknown how these agents perform in competitive, multi-agent environments. We suggest a framework, built on top of a frozen large language model, GPT-4, for designing generative agents in competitive, episodic environments and we evaluate their game performance. We provide new modules allowing agents to set and evaluate strategic goals, develop theories of mind about other agents, and embody descriptive personas that affect their behavior. Across many simulations, we show that agents with varied cognitive abilities exhibit altered behaviors and that goal-driven agents display an emergent capacity to rapidly utilize information perceived from their environment.

Outwit, Outplay, Out-Generate: A Framework for Designing Strategic Generative Agents in Competitive Environments

Interactive fiction (IF) games are a genre of games where the player interacts with the fictional world via text-based commands, solving puzzles primarily by exploring the world and using items they collect along the way. Although there has been much work on playing IF using AI, there is relatively less work on the creation of such games using AI. While large language models (LLMs) have made the generation of text far easier in the past several years, they still struggle to be able to generate such highly-structured and consistent story worlds that one might see in IF. We present a three-part system called BERALL, which generates unique text adventure games by 1) maintaining the current state story world, 2) using retrieval-augmented generation (RAG) to create relevant location descriptions, and 3) combining these components to create a coherent experience for the player. Our results show that our approach is effective at generating room and story descriptions from setting and knowledge graphs, demonstrating the benefits of LLMs in IF. We find challenges remain in maintaining current game state due in part to LLMs not understanding the impact of actions on knowledge graphs.

BERALL: Generating Retrieval-augmented State-based Interactive Fiction Games

Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments, yet their creativity remains underexplored. This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs. In Balderdash, players generate fictitious definitions for obscure terms to deceive others while identifying correct definitions. Our framework enables multiple LLM agents to participate in this game, assessing their ability to produce plausible definitions and strategize based on game rules and history. We implemented a centralized game engine featuring various LLMs as participants and a judge LLM to evaluate semantic equivalence. Through a series of experiments, we analyzed the performance of different LLMs, examining metrics such as True Definition Ratio, Deception Ratio, and Correct Guess Ratio. The results provide insights into the creative and deceptive capabilities of LLMs, highlighting their strengths and areas for improvement. Specifically, the study reveals that infrequent vocabulary in LLMs' input leads to poor reasoning on game rules and historical context.

Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash

Text-adventure games and text role-playing games are grand challenges for reinforcement learning game playing agents. Text role-playing games are open-ended environments where an agent must faithfully play a particular character. We consider the distinction between {\em characters} and {\em actors}, where an actor agent has the ability to play multiple characters. We present a framework we call a {\em thespian agent} that can learn to emulate multiple characters along with a soft prompt that can be used to direct it as to which character to play at any time. We further describe an attention mechanism that allows the agent to learn new characters that are based on previously learned characters in a few-shot fashion. We show that our agent outperforms the state of the art agent framework in multi-character learning and few-shot learning.

Thespian: Multi-Character Text Role-Playing Game Agents

Strategic social deduction games serve as valuable testbeds for evaluating the understanding and inference skills of language models, offering crucial insights into social science, artificial intelligence, and strategic gaming. This paper focuses on creating proxies of human behavior in simulated environments, with “Among Us” utilized as a tool for studying simulated human behavior. The study introduces a text-based game environment, named AmongAgents, that mirrors the dynamics of “Among Us”. Players act as crew members aboard a spaceship, tasked with identifying impostors who are sabotaging the ship and eliminating the crew. Within this environment, the behavior of simulated language agents is analyzed. The experiments involve diverse game sequences featuring different configurations of Crewmates and Impostor personality archetypes. Our work demonstrates that state-of-the-art large language models (LLMs) can effectively grasp the game rules and make decisions based on the current context. This work aims to promote further exploration of LLMs in goal-oriented games with incomplete information and complex action spaces, as these settings offer valuable opportunities to assess language model performance in socially driven scenarios.

AmongAgents: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game

Role-playing games (RPGs) provide players with a rich, interactive world to explore. Dialogue serves as the primary means of communication between developers and players, manifesting in various forms such as guides, NPC interactions, and storytelling. While most games rely on written scripts to define the main story and character personalities, player immersion can be significantly enhanced through casual interactions between characters. With the advent of large language models (LLMs), we introduce a dialogue filler framework that utilizes LLMs enhanced by knowledge graphs to generate dynamic and contextually appropriate character interactions. We test this framework within the environments of Final Fantasy VII Remake and Pokémon, providing qualitative evidence that demonstrates GPT-4’s capability to act as specific characters and generate dialogue. However, some flaws remain, such as GPT-4 being overly positive when portraying Cloud. This study aims to assist developers in crafting more nuanced filler dialogues, thereby enriching player immersion and enhancing the overall RPG experience.

Premium content

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

Downloads

Next from ACL 2024

DAGGER: Data Augmentation for Generative Gaming in Enriched Realms

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES