=Paper=
{{Paper
|id=Vol-3193/short2GDE
|storemode=property
|title=Summary on “Hybrid Neuro-Symbolic Approach for Text-Based Games using Inductive Logic”
|pdfUrl=https://ceur-ws.org/Vol-3193/short2GDE.pdf
|volume=Vol-3193
|authors=Kinjal Basu
|dblpUrl=https://dblp.org/rec/conf/iclp/000222
}}
==Summary on “Hybrid Neuro-Symbolic Approach for Text-Based Games using Inductive Logic”==
https://ceur-ws.org/Vol-3193/short2GDE.pdf
Summary on “Hybrid Neuro-Symbolic Approach for
Text-Based Games using Inductive Logic
Programming”
Kinjal Basu1,2
1
University of Texas at Dallas, Richardson, USA
2
IBM Research, NY, USA
Abstract
In this paper, I briefly describe the summary of my work titled - Hybrid Neuro-Symbolic Approach for Text-
Based Games using Inductive Logic Programming. Text-based games (TBGs) have emerged as an important
test-bed, requiring reinforcement learning (RL) agents to combine natural language understanding with
reasoning. A key challenge for agents solving this task is to generalize across multiple games and
shows good results on both seen and unseen objects. To tackle these issues, we have designed a hybrid
neuro-symbolic framework for TBGs that uses symbolic reasoning along with the neural RL model. We
also use WordNet as an external commonsense knowledge source to bring information to generalize the
hypothesis. We have tested our work on different settings on TWC games and showed that the agents
that incorporate the neuro-symbolic hybrid approach with the generalized rules outperform the baseline
agents.
Keywords
Reinforcement Learning, Text-based Games, Inductive Logic Programming, Answer Set Programming.
1. Summary
Natural language plays a crucial job in human intelligence and cognition. TBGs become
appropriate simulation environments for studying the language-informed sequential decision-
making process as the states and actions in these games are described in natural language.
So, to solve these games an agent needs the skill of both natural language processing (NLP)
and reinforcement learning (RL). At a high level, the existing agents can be classified into two
classes - (a) rule-based agents, and (b) neural agents. Both have advantages and disadvantages.
Rule-based models are very efficient in doing multi-hop reasoning and specially commonsense
reasoning, however, they rely heavily on pre-defined knowledge and are not greatly scalable. On
the other hand, neural agents show good scalability and can be trained from scratch, although
they perform poorly on unseen data and the policies are also not interpretable.
In our paper [1], we introduce a hybrid neuro-symbolic (HNS) architecture for TBGs that
utilizes the positive features of both the neural and the symbolic agents. Instead of using pre-
defined prior knowledge, the symbolic agent in HNS learns the symbolic policies by leveraging
the reward and action pairs while playing the game. This allows the policies to be interpretable
2nd Workshop on Goal-directed Execution of Answer Set Programs (GDE’22), August 1, 2022
$ Kinjal.Basu@ibm.com (K. Basu)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
OBSERVATION(Ot )
Parser + IE
Context Encoding
You have entered into a
Symbolic Action Selector
Word bedroom. You see a
Embeddings chest drawer and a Symbolic Facts
wardrobe. The Generator
GRU wardrobe is open.
Fact Generalizer
St-1 INVENTORY: You are (using WordNet)
GRU carrying a shirt.
ASP Solver
Context Admissible Actions (s(CASP))
Encoding (e.g., open chest-drwawer,
Rulest-1
Action Encoding
insert shirt into wardrobe)
Word Rules Set
Embeddings Action Selector
Action Rulest
Action
GRU + Reward Rule
Environment ILP Generalizer
Neural Action Selector Symbolic Rule Learner
Figure 1: Overview of the HNS agent’s decision making at any given time step. The Hybrid Neuro-
Symbolic architecture mainly consist of 5 modules - (a) Context Encoder encodes the observation to
dynamic context,⨁︀
(b) Action Encoder encodes the admissible actions, (c) Neural Action Selector combines
(a) and (b) with operator, (d) Symbolic Action Selector returns a set of candidate actions, and (e)
Symbolic Rule Learner uses ILP and WordNet based rule generalization to generate symbolic rules.
and very natural. Importantly, the rules are learned as default theories so that the agent
can do non-monotonic reasoning. Also, we lift the rules using WordNet and that gives more
generalization capabilities to the rules. The neural part of an HNS agent is responsible for doing
the exploration in the environment and is used in the scenarios where the symbolic agent fails
to provide an action (due to a lack of learned rules).
The goal of our paper was to show how a neural and a symbolic agent can work together
in an RL environment for the TBGs. The neural agents are good at exploration whereas the
symbolic agents are good at learning interpretable policies that offer rewards and apply them
to select a candidate set of actions. Keeping it as a motivation, we try to capitalize the power of
both agents to get better results. The main idea is to use the symbolic agent to learn the policies
in the form of logic rules and apply them using an ASP solver. When the symbolic agent fails
to provide a good action, then the neural agent takes care of it as a fallback. In other words,
the action selector gives priority to the symbolic agent over the neural. Figure 1 illustrates the
components of our HNS architecture and shows an overview of the decision making process.
This framework has been tested on Text-World-Commonsense games and we show that
the agents that incorporate the neuro-symbolic hybrid approach with the generalized rules
outperform the baseline agents. The performance results and more details about this work can
be found in our paper [1].
2. Conclusion
This summary gives a high-level overview of our works on Text-based games and how we try
to incorporate the benefits from both neural and symbolic agents to build hybrid models. Our
architecture in figure 1 shows the decision flow of an agent. We are currently working on it to
improve its performance and trying to cover harder games.
References
[1] K. Basu, K. Murugesan, M. Atzeni, P. Kapanipathi, K. Talamadupula, T. Klinger, M. Campbell,
M. Sachan, G. Gupta, A hybrid neuro-symbolic approach for text-based games using
inductive logic programming, in: Combining Learning and Reasoning: Programming
Languages, Formalisms, and Representations, 2021.