-
OVERVIEW
Arti cial intelligence (AI) is increasingly studied in many
elds such as philosophy, law and decision making. One of
the approaches to AI is the use of agent and multi-agent
systems. Agents are key element for building complex
largescale distributed systems[
9
]. In multi-agent systems, each
agent interacts with the environment and communicates
with other agents in order to achieve the designated goal.
Communication means to share and exchange information,
cooperate and coordinate with each other in order to achieve
a common goal.
Argumentation is a type of communication between agents
and a process attempting to form an agreement about what
to believe. There has been increasing research in
argumentation and dialogue systems in the past decade[
23
]. The agent
as a dialogue participant needs sophisticated dialogue
strategies in order to make high quality dialogue contributions.
By reviewing the state of art literature in computerised
dialogue systems (e.g.[
21
];[
22
]), it is observed that their dialogue
strategies (i.e. strategic heuristics) are hardwired into the
computational agent. One of the main issues with this is that
an agent might be incapable of dealing with new dialogue
situations that have not been coded, and indeed this is an
impossible task given the dynamic nature of argumentation.
It would be ideal to make an agent search for an optimal
strategy by itself e.g. via trial and error, and thus the agent
with the best strategy will win the argument [
8
].
Machine learning has an important role to play in order
to meet these challenges. To make agents learn the dialogue
strategies, it would be more exible for them to make an
argument through exploration (trial and error). It is believed
that learning can make agents more exible to adapt to new
environments and new dialogue situations. One of the popular
machine learning approaches with regards to learning agents
is known as reinforcement learning (RL).
Reinforcement learning focuses on how to map an action
for each state by interacting with the environment and
observing the state change[
15
]. Sutton and Barto[
15
] de ne
reinforcement learning as an agent learning what to do and
how to connect each situation with an action to maximise
the cumulative reward. The learner or agent is not told what
action should be taken, rather the learner needs to explore a
policy that yields the maximum cumulative reward by
trying them out. In reinforcement learning, the agent interacts
with the environment by taking an action and receiving a
agent learn to argue there is a need to identify states, actions,
environment and the rewards. In this research abstract
argumentation systems (AAS) is initially used [
5
] to represent
the argumentation. Reasons are listed as follows:
(1) It has the ability to represent informal human
reasoning in a way that a computer can perform calculation.
In this way, argumentation bridges the gap between
human and machine reasoning[
11
].
(2) It is easier to compute acceptable arguments in
order to evaluate variance argument semantics e.g.
grounded extension.
(3) It provides a great opportunity for the agent to
explore the relationship between arguments.
(4) It is a powerful method to solve problems since it
can be easily implemented in logic programming[
5
].
The classical state representation of agents in literature (e.g.
[
19
];[
4
]) involves states being represented as nodes in the
argumentation graph and action by the attack relation between
arguments.
The main objective of our research is to investigate whether
reinforcement learning agent can be used to create an
argumentation AI with improved performance and e ciency
comparable to state-of-the-art systems. Performance is related
to how well the agent learns over time. The measurement
of performance for a good argumentation; for instance, is
whether argumentation can be won or lost or how many
arguments from a learning agent obtains accepted against other
heuristic strategy agents. E ciency is related to whether the
agent can learn within a limited or insu cient time. So the
aim is to nd out if the agent can learn rapidly or not. It
should also ensure that the agent obtains full knowledge from
the environment so as to be able to use an e cient method
to nd an optimal decision for each state [
17
].
The light of this hypothesis, the following steps will be
taken:
(1) Initially, a basic abstract argument game model is
used due to its simplicity in implementing arguments.
This in turn makes it possible to investigate how
reinforcement learning can be applicable to a simple
dialogue scenario.
(2) Evaluation of an argumentation setting with a human
or another AI agent by observing learning
performance over time.
(3) Investigating suitable means for reinforcement
learning of a complicated dialogue scenario and studying
the results in order to generalise the RL method.
Complicated dialogue scenario involves more move
types e.g. questions, challenges, assertion, withdrawal
and moves from abstract argument level to
propositional level.
This work will also investigate other di erent scenario
such as backtracking ([
14
];[
16
]), arguments content, weight
of individual argument amongst others[
6
]. Additionally,
challenging issue such as states representation[
1
] as well as reward
function will also be explored.
To prove the hypothesis, we have built the argumentation
software to facilitate experiment for reinforcement learning
agent arguing against di erent agents. A software testbed,
Argumento+, named after its predecessor Argumento as
reported in[
24
], has been built using the Java programming
language. Argumento+ contains the RL agent as well as
three other agents namely, random, maximum probability
utility and minimum probability utility agent for the sake
of the evaluation. The agents play abstract argument games.
RL agent plays game against them to maximise the
cumulative reward by winning more games. Indeed, if RL agent
win the game, it will receive rewards based on the number
of acceptable arguments i.e. grounded extensions. We
considered grounded extension because it contains an argument
that has no doubt in comparison with other arguments [
19
].
Consequently, it will be a more acceptable argument.
We have performed an initial experiment to investigate
whether RL agent learns to argue against baseline agents
[
1
]. RL agent adopts a commonly used RL method, that
is Q-learning algorithm. The aim of Q-learning is to allow
an agent to learn through experience and map each state
with an action by choosing the maximum value from the
Q-table which is updated after each episode. The initial
experiment and evaluation generally encourage the adopting
of reinforcement learning agent in argumentation with a
long term delayed reward which are considered as grounded
extensions [
1
].
In the future, this work will attempt to suggest ways to
improve the RL agent performance by carrying out further
works on the initial experiment results. The state
representation of the arguments still needs to be more sophisticated
in order to make each one of them unique [
1
], and as a
result this will make it easy for the agent to distinguish
between states. Even though initial suggestion pointed at
making the state a combination of the current state and
previous state, it is still di cult to uniquely identify each
state. To sort out this issue, it will be worth
investigating if this can be resolved by representing each state as:
(levelOf T ree; agentID; currentState; previousState)
Backtracking ([
14
];[
16
]) will also be considered to improve
the simple argument game by developing some game rules
in [
18
]. Moreover, to make the game more competitive and
e ective it is important to make the agent consider the
opponent's strategy[
7
]. Hence, the learning agent needs to consider
how to learn to argue with the opponent by expanding its
knowledge base with new arguments. In addition, in
complex argumentation scenario, we need to consider moving
from high level abstraction to the argument contents by
using propositional logic. Weighted arguments will also be
considered in this research since some arguments are more
important than others. We will consider choosing a suitable
argument model for the complicated scenario. There are many
models; for instance, Prakken's dialogue game P ersuasion
with dispute [
13
], Bench-Capon T DG dialogue game ([
2
];[
3
]),
DC by Mackenzie [
10
], Utilisation by Moore in [
12
] and DE
system (Yuan et al. [
20
]), all of these models will be critically
reviewed.