Summary on “Hybrid Neuro-Symbolic Approach for Text-Based Games using Inductive Logic Programming”

Abstract

In this paper, I briefly describe the summary of my work titled - Hybrid Neuro-Symbolic Approach for TextBased Games using Inductive Logic Programming. Text-based games (TBGs) have emerged as an important test-bed, requiring reinforcement learning (RL) agents to combine natural language understanding with reasoning. A key challenge for agents solving this task is to generalize across multiple games and shows good results on both seen and unseen objects. To tackle these issues, we have designed a hybrid neuro-symbolic framework for TBGs that uses symbolic reasoning along with the neural RL model. We also use WordNet as an external commonsense knowledge source to bring information to generalize the hypothesis. We have tested our work on diferent settings on TWC games and showed that the agents that incorporate the neuro-symbolic hybrid approach with the generalized rules outperform the baseline agents.

Keywords

• eol>Reinforcement Learning
• Text-based Games
• Inductive Logic Programming
• Answer Set Programming

1. Summary

Natural language plays a crucial job in human intelligence and cognition. TBGs become appropriate simulation environments for studying the language-informed sequential decisionmaking process as the states and actions in these games are described in natural language. So, to solve these games an agent needs the skill of both natural language processing (NLP) and reinforcement learning (RL). At a high level, the existing agents can be classified into two classes - (a) rule-based agents, and (b) neural agents. Both have advantages and disadvantages. Rule-based models are very eficient in doing multi-hop reasoning and specially commonsense reasoning, however, they rely heavily on pre-defined knowledge and are not greatly scalable. On the other hand, neural agents show good scalability and can be trained from scratch, although they perform poorly on unseen data and the policies are also not interpretable.

In our paper [ 1 ], we introduce a hybrid neuro-symbolic (HNS) architecture for TBGs that utilizes the positive features of both the neural and the symbolic agents. Instead of using predefined prior knowledge, the symbolic agent in HNS learns the symbolic policies by leveraging the reward and action pairs while playing the game. This allows the policies to be interpretable g n iod Word cn Embeddings E tex GRU t n o C

GRU

St-1

Context

Encoding g n i cod Word nE Embeddings n itco GRU

A Neural Action Selector OBSERVATION(Ot ) You have entered into a bedroom. You see a chest drawer and a

wardrobe. The wardrobe is open.

INVENTORY: You are carrying a shirt.

Admissible Actions

(e.g., open chest-drwawer, insert shirt into wardrobe)

Parser + IE

r o tc Symbolic Facts lee Generator S ino Fact Generalizer tcA (using WordNet) lico ASP Solver bm (s(CASP)) y S

Rulest-1

Rules Set

Action Selector

Action

Environment

Action + Reward

Rulest

Rule

ILP Generalizer Symbolic Rule Learner and very natural. Importantly, the rules are learned as default theories so that the agent can do non-monotonic reasoning. Also, we lift the rules using WordNet and that gives more generalization capabilities to the rules. The neural part of an HNS agent is responsible for doing the exploration in the environment and is used in the scenarios where the symbolic agent fails to provide an action (due to a lack of learned rules).

The goal of our paper was to show how a neural and a symbolic agent can work together in an RL environment for the TBGs. The neural agents are good at exploration whereas the symbolic agents are good at learning interpretable policies that ofer rewards and apply them to select a candidate set of actions. Keeping it as a motivation, we try to capitalize the power of both agents to get better results. The main idea is to use the symbolic agent to learn the policies in the form of logic rules and apply them using an ASP solver. When the symbolic agent fails to provide a good action, then the neural agent takes care of it as a fallback. In other words, the action selector gives priority to the symbolic agent over the neural. Figure 1 illustrates the components of our HNS architecture and shows an overview of the decision making process.

This framework has been tested on Text-World-Commonsense games and we show that the agents that incorporate the neuro-symbolic hybrid approach with the generalized rules outperform the baseline agents. The performance results and more details about this work can be found in our paper [ 1 ].

2. Conclusion

This summary gives a high-level overview of our works on Text-based games and how we try to incorporate the benefits from both neural and symbolic agents to build hybrid models. Our architecture in figure 1 shows the decision flow of an agent. We are currently working on it to improve its performance and trying to cover harder games.

References

[1] K. Basu , K. Murugesan , M. Atzeni , P. Kapanipathi , K. Talamadupula , T. Klinger , M. Campbell, M. Sachan , G. Gupta , A hybrid neuro-symbolic approach for text-based games using inductive logic programming , in: Combining Learning and Reasoning: Programming Languages, Formalisms, and Representations , 2021 .