Artificial intelligence (AI) is increasingly studied in many fields such as philosophy, law and decision making. One of the approaches to AI is the use of agent and multi-agent systems. Agents are key element for building complex largescale distributed systems [9]. In multi-agent systems, each agent interacts with the environment and communicates with other agents in order to achieve the designated goal. Communication means to share and exchange information, cooperate and coordinate with each other in order to achieve a common goal.

Argumentation is a type of communication between agents and a process attempting to form an agreement about what to believe. There has been increasing research in argumentation and dialogue systems in the past decade [23]. The agent as a dialogue participant needs sophisticated dialogue strategies in order to make high quality dialogue contributions. By reviewing the state of art literature in computerised dialogue systems (e.g. [21]; [22]), it is observed that their dialogue strategies (i.e. strategic heuristics) are hardwired into the computational agent. One of the main issues with this is that an agent might be incapable of dealing with new dialogue situations that have not been coded, and indeed this is an impossible task given the dynamic nature of argumentation. It would be ideal to make an agent search for an optimal strategy by itself e.g. via trial and error, and thus the agent with the best strategy will win the argument [8].

Machine learning has an important role to play in order to meet these challenges. To make agents learn the dialogue strategies, it would be more flexible for them to make an argument through exploration (trial and error). It is believed that learning can make agents more flexible to adapt to new environments and new dialogue situations. One of the popular machine learning approaches with regards to learning agents is known as reinforcement learning (RL).

Reinforcement learning focuses on how to map an action for each state by interacting with the environment and observing the state change [15]. Sutton and Barto [15] define reinforcement learning as an agent learning what to do and how to connect each situation with an action to maximise the cumulative reward. The learner or agent is not told what action should be taken, rather the learner needs to explore a policy that yields the maximum cumulative reward by trying them out. In reinforcement learning, the agent interacts with the environment by taking an action and receiving a reward for the action taken as seen in figure 1. To make an Figure 1: Reinforcement learning agent-environment interaction agent learn to argue there is a need to identify states, actions, environment and the rewards. In this research abstract argumentation systems (AAS) is initially used [5] to represent the argumentation. Reasons are listed as follows:

(1) It has the ability to represent informal human reasoning in a way that a computer can perform calculation. In this way, argumentation bridges the gap between human and machine reasoning [11]. (2) It is easier to compute acceptable arguments in order to evaluate variance argument semantics e.g. grounded extension. (3) It provides a great opportunity for the agent to explore the relationship between arguments. (4) It is a powerful method to solve problems since it can be easily implemented in logic programming [5]. The classical state representation of agents in literature (e.g. [19]; [4]) involves states being represented as nodes in the argumentation graph and action by the attack relation between arguments.

The main objective of our research is to investigate whether reinforcement learning agent can be used to create an argumentation AI with improved performance and efficiency comparable to state-of-the-art systems. Performance is related to how well the agent learns over time. The measurement of performance for a good argumentation; for instance, is whether argumentation can be won or lost or how many arguments from a learning agent obtains accepted against other heuristic strategy agents. Efficiency is related to whether the agent can learn within a limited or insufficient time. So the aim is to find out if the agent can learn rapidly or not. It should also ensure that the agent obtains full knowledge from the environment so as to be able to use an efficient method to find an optimal decision for each state [17].

The light of this hypothesis, the following steps will be taken:

(1) Initially, a basic abstract argument game model is used due to its simplicity in implementing arguments. This in turn makes it possible to investigate how reinforcement learning can be applicable to a simple dialogue scenario. (2) Evaluation of an argumentation setting with a human or another AI agent by observing learning performance over time. (3) Investigating suitable means for reinforcement learning of a complicated dialogue scenario and studying the results in order to generalise the RL method. Complicated dialogue scenario involves more move types e.g. questions, challenges, assertion, withdrawal and moves from abstract argument level to propositional level.

This work will also investigate other different scenario such as backtracking ( [14]; [16]), arguments content, weight of individual argument amongst others [6]. Additionally, challenging issue such as states representation [1] as well as reward function will also be explored.

To prove the hypothesis, we have built the argumentation software to facilitate experiment for reinforcement learning agent arguing against different agents. A software testbed, Argumento+, named after its predecessor Argumento as reported in [24], has been built using the Java programming language. Argumento+ contains the RL agent as well as three other agents namely, random, maximum probability utility and minimum probability utility agent for the sake of the evaluation. The agents play abstract argument games. RL agent plays game against them to maximise the cumulative reward by winning more games. Indeed, if RL agent win the game, it will receive rewards based on the number of acceptable arguments i.e. grounded extensions. We considered grounded extension because it contains an argument that has no doubt in comparison with other arguments [19]. Consequently, it will be a more acceptable argument.

We have performed an initial experiment to investigate whether RL agent learns to argue against baseline agents [1]. RL agent adopts a commonly used RL method, that is Q-learning algorithm. The aim of Q-learning is to allow an agent to learn through experience and map each state with an action by choosing the maximum value from the Q-table which is updated after each episode. The initial experiment and evaluation generally encourage the adopting of reinforcement learning agent in argumentation with a long term delayed reward which are considered as grounded extensions [1].

In the future, this work will attempt to suggest ways to improve the RL agent performance by carrying out further works on the initial experiment results. The state representation of the arguments still needs to be more sophisticated in order to make each one of them unique [1], and as a result this will make it easy for the agent to distinguish between states. Even though initial suggestion pointed at making the state a combination of the current state and previous state, it is still difficult to uniquely identify each state. To sort out this issue, it will be worth investigating if this can be resolved by representing each state as: (levelOf T ree, agentID, currentState, previousState)

Backtracking ( [14]; [16]) will also be considered to improve the simple argument game by developing some game rules in [18]. Moreover, to make the game more competitive and effective it is important to make the agent consider the opponent's strategy [7]. Hence, the learning agent needs to consider how to learn to argue with the opponent by expanding its knowledge base with new arguments. In addition, in complex argumentation scenario, we need to consider moving from high level abstraction to the argument contents by using propositional logic. Weighted arguments will also be considered in this research since some arguments are more important than others. We will consider choosing a suitable argument model for the complicated scenario. There are many models; for instance, Prakken's dialogue game P ersuasion with dispute [13], Bench-Capon T DG dialogue game ( [2]; [3]), DC by Mackenzie [10], Utilisation by Moore in [12] and DE system (Yuan et al. [20]), all of these models will be critically reviewed.