One major challenge in reinforcement learning (RL) is the large amount of steps for the RL agent needs to converge in the training process and learn the optimal policy, especially in text-based game environments where the action space is extensive. However, non-player characters (NPCs) sometimes hold some key information about the game, which can potentially help to train RL agents faster. Thus, this paper explores how to interact and converse with NPC agents to get the key information using large language models (LLMs), as well as incorporate this information to speed up RL agent's training using knowledge graphs (KGs) and Story Shaping.
Reinforcement learning (RL) has demonstrated remarkable efectiveness in solving intricate decision-making tasks, but its trial-and-error approach often leads to slow convergence to the optimal policy. In text-adventure games, NPCs possess crucial information that could spare the agent from extensive trial-and-error. Utilizing this prior knowledge could significantly reduce the agent’s policy search space, making it more eficient by breaking down complex tasks into smaller, focused objectives. For instance, knowing that ”killing the dragon” requires a sword allows the agent to concentrate on finding the sword directly, rather than wasting steps exploring how to defeat the dragon.
Large Language Models (LLMs) are incredibly capable of conversational tasks and are highly configurable using prompting techniques. Thus, we chose to use them as Meanwhile, they are not as eficient as RL agent in terms of searching for the optimal chain of actions. Therefore, we chose to keep the RL agent as the main component responsible for searching for the optimal policy while prised of LLMs.
The RL agent acts as an action module and the LLMs
act as a dialogue module. Yet, we still need to find a
way to bridge these two modules, i.e. incorporating the
information that the dialogue module retrieves into the
action module. For this purpose, we turn to the technique
of Story Shaping[
CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) module to imitate the optimal trajectory.
In this paper, we propose Dialogue Shaping, a
framework that is able to extract useful information through
conversation with NPCs, and then convert the
information into knowledge graphs which are then used to speed
up RL agent’s convergence to optimal policy by using
the Story Shaping technique[
Text games involve turn-based interactions where players read descriptions of the game’s environment in natural language and respond with short text-based actions. These games can be described using partially-observable Markov Decision Processes, denoted as ⟨, , , , Ω, , ⟩ ing possible states, transition probabilities, vocabulary , representtion, and discount factor. The RL agent’s goal is to learn a policy () →
to maximize expected future rewards.
The power of Large
in recent years due to their advanced ability to adapt to
numerous downstream tasks. ChatGPT, an LLM chatbot
created by OpenAI, ofers diverse interaction modes, and
users can engage with it by providing prompts for acting
as the NPC and the agent in text games [
ceur-ws.org useful game information and incorporate them into the game as extra reward signals to guide the RL agent.
A knowledge graph consists of triples ⟨, , ⟩ , capturing information about entities, their attributes, and relationships. Our method uses two types of KGs: internal KG and target KG.
During RL exploration in the text game, the internal
KG represents the agent’s current state, including the
room it’s in and the objects it possesses[
The target KG describes the final state the agent must achieve to win the game, specifying the last room the agent should be in and the required objects. This KG is generated before training and stays unchanged.
KGA2C [
description of the current room the agent is located in, the agent’s current inventory, feedback of the environment from the agent’s last action, and agent’s last action.
Story Shaping, proposed by Peng et al., is a technique that helps the RL agent infers tacit knowledge on how to accomplish a task. For each training step in the game, Story Shaping gives the RL agent an extra reward signal (in addition to the game environment’s reward signal) based on the similarity between agent’s current internal KG and target KG, and therefore encourage the agent to perform actions that will make its internal KG similar to the target KG. The target KG in this paper is generated by prompting the ChatGPT agent and it represents a summary of the hints the ChatGPT agent learns through talking to the ChatGPT NPC.
We open one session of ChatGPT, as shown in Figure 2, and prompted it to be the non-player character. The NPC is provided with general information about the game, including the layout and the available objects, as well as the hints to win the game. One example of hints is getting a sword in the Artillery room is a prerequisite to kill the dragon.
ChatGPT is prompted to be the player agent in the game. The ChatGPT agent is provided with its goal in the game (e.g. kill the dragon) and general instructions on how to converse with the NPC (e.g. ask questions based on previous given answers). We did not reveal any game details in the prompts for the ChatGPT agent, because it is expected to gain those information by asking questions to the ChatGPT NPC.
After the dialogue with NPC, we train a KGA2C agent to play the game. In order to incorporate the information learned by the ChatGPT agent during conversation with NPC into KGA2C agent’s training, we prompt ChatGPT agent to generate a knowledge graph and use it as a target knowledge graph for the Story Shaping KGA2C agent. The pipeline for our method is shown in Figure 3.
We conducted experiments to test our assumption that the dialogue is able to accelerate KGA2C agent’s convergence by generating the target knowledge graph.
We open one session of ChatGPT and prompt it to be the NPC of the game. Table 1 shows the NPC prompt for one game. We also open another separate session of ChatGPT and prompt it to be the agent of the game with a goal in mind. Table 2 shows the agent prompt for one game.
You are an NPC in a text-adventure game. You and the agent are both in the game. For each step, waits for the agent to ask questions, then you should provide a correct answer based on the information about the game given as follow: Layout: [room1]-east-[room2], [room2]-east-[room3], etc. (A-east-B means A is to the east of B) Goal and prerequisite: A dragon is in the dungeon. The only way to kill the dragon is to use a sword and there is no other way.
Object information: [object1], [object2] is in [room1]. [object3], [object4], [object5] is in [room2]. [room3] has no objects...
You are an agent in a text-adventure game. You and the NPC are both in the game. Your goal is to kill the dragon. For each step, you should ask questions to the NPC in order to get the information on how to kill the dragon.
Ask a new set of questions based on the current observation and answers given to the previous set of questions according to the following rule: 1. ask similar and follow-up questions to previous questions that have a ”yes” answer. 2. Avoid asking similar and follow-up questions to previous questions that have a ”no” answer.
Then, the dialogue begins as the agent comes up with a set of questions and the NPC provides answers to them back and forth. ChatGPT NPC proves to be a reliable game database, correctly responding to queries about questions as shown in the undesired dialogue in Table 3.
After the dialogue ends and the ChatGPT agent retrieved information on how to reach the goal, we prompt it to convert that information into a textual knowledge graph representation as shown in Table 4. We then filter the edges in the knowledge graph by only including ones that have ”you” as a subject, because we are only interested in what actions the agent has to perform to reach to goal.
Finally, we use this filtered knowledge graph as the target knowledge graph to ”shape” the Story Shaping KGA2C agent behaviors.
We generate each game using the LIGHT framework
[
For every game, the KGA2C agent is trained for 100,000 steps. After every 450 steps, the agent is evaluated for 50 episodes with 10 random seeds. We gather metrics like average and standard deviation of the test scores achieved for those 50 episodes, like in Figure 4. The maximum step limit for a single episode is 75 steps, while the optimal path for all games usually takes around 10 steps.
We trained the baseline KGA2C agent and the one with Story Shaping assistance for each game. Baseline KGA2C agent only receives reward signals that are built into the game mechanism (i.e. reaching the final goal), whereas the Story Shaping KGA2C agent receives additional reward signals when its internal knowledge graph overlaps with the target knowledge graph which is generated by the dialogue module (i.e. complete the prerequisite of the goal). 5.3. Results room and object locations. Moreover, when the ChatGPT Figure 4 showed the average test score and its standard agent makes ungrounded assumptions about the game deviation of the baseline KGA2C agent and Story Shap(like ”Does the barkeeper possess any knowledge about ing KGA2C agent equipped with target knowledge graph dragon’s weaknesses” while there is no barkeeper) in its generated from the dialogue during training for game 1. questions, the ChatGPT NPC is able to recognize (finding The Story Shaping KGA2C agent outperformed the baseout that the game does not have a barkeeper) and negate line in all games. In all games, the Story Shaping agent them. converged to the optimal policy (gaining maximum score
In evaluating the performance of ChatGPT agent, we of 15) much faster than the baseline. In game 1, the Story aim to minimize the number of exchanges with the Chat- Shaping KGA2C agent converged to the optimal policy GPT NPC while retrieving hints on winning the game. after trained for around 10000 steps despite a temporary We found out that ChatGPT agent is much more likely drop in average scores around step 30000, while the baseto hallucinate by coming up with ungrounded questions line agent took around 90000 training steps to learn the without explicit instructions on how to ask the optimal optimal policy, according to figure 4. Moreover, almost questions in our prompt. As shown in the desired dia- at all the training steps, the standard deviation score logue in Table 3, when we include those explicit instruc- range of the Story Shaping agent is disjoint from that of tions in the prompt, it is able to ground its inquiries. Oth- the baseline, meaning that the Story Shaping agent can erwise, it will fail to follow up on the previous questions consistently achieve higher score than the baseline. that have a ”yes” answer and endlessly ask ungrounded
Through evaluation of our technique across a range of text games, we have shown that the dialogue module is able to extract key game information which might take a traditional action based RL agent tens of thousands of steps to learn. Moreover, we show that the dialogue module is able to pass along those key information and guide the action agent through knowledge graph and Story Shaping technique efectively and reliably. Thus, we have proven the substantial potential of the dialogue component to greatly speed up RL agent’s convergence to the optimal policy. Future work might further exploit this potential by exploring approaches like few-shot prompting or finetuning LLMs to more efectively retrieve useful information from the NPC.