Reinforcement learning is a form of machine learning where an intelligent agent learns to make decisions by interacting with some environment. The agent may have no prior knowledge of the environment and discovers it through interaction. For every action that the agent takes, the environment gives a reward signal that is used to measure how good or bad that action was. In this way, the agent learns which are more favorable actions to take in every state of the environment. There are different approaches to solve a reinforcement learning problem, but one drawback that arises during this process is the tradeoff between exploration and exploitation. In this work we focus on studying different exploration strategies and compare their effect in the performance of an intelligent tutoring system that is modeled as a reinforcement learning problem. An intelligent tutoring system is a system that helps in the process of teaching and learning by adapting to student needs and behaving differently for each student. We train this system using reinforcement learning and different exploration strategies and compare the performance of training and testing to find which is the best strategy.
that apply techniques from the field of
Artificial Intelligence to provide better support
for the users of the system [
Traditional tutoring systems use the one-tomany way of presenting the learning materials to the students.
_________________________________ Proccedings of RTA-CSIT 2021, May 2021, Tirana, Albania EMAIL: jezuina.koroveshi@fshn.edu.al (A.1); ana.ktona@fshn.edu.al (A.2);
2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
In this approach every student is given the
same materials to learn regardless of his/her
needs and preferences. These systems are not
well suited for all students because they may
come from different backgrounds, may have
different learning styles and do not absorb the
lessons with the same peace. An intelligent
tutoring system customizes the learning
experience that the student perceives by taking
into consideration factors such as pre-existing
knowledge, learning style and student
progress. According to [
Reinforcement learning is a form of
machine learning that is based on learning
from experience. The learner is exposed to
some environment, for which he may or may
not have information, starts making decisions
and gets some feedback that gives information
telling how good or bad that decision was.
Based on the feedback from the environment
the learner learns which decisions are more
favorable to take. This class of machine
learning has been used in modeling and
building intelligent tutoring systems such as in
the works from [
The remainder of this paper is organized as follows: in section 2 we give an overview on reinforcement learning, in section 3 we describe the model that we have used to build an intelligent tutoring system, in section 4 we give the experimental results of training the model using different exploration strategies and is section 5 we give the conclusions of our work.
Reinforcement learning is a form of
machine learning in which the learner learns
some sequence of actions by interacting with
the environment. The learner is in a state of the
environment, takes some action that moves it
from that state to another and after each action
the environment gives a reward signal. This
reward signal is used to learn which are the
best states to be in, and therefore learn which
action to take in order to go in those states. A
reinforcement learning problem can be
modeled as a Markov Decision Process
(MDP). A MDP is a stochastic process that
satisfies the Markov Property. In a finite MDP,
the set of states, actions and rewards have a
finite number of elements. Formally, a finite
MDP can be defined as a tuple M = (S, A, P,
R, γ), where:
S is the set of states: S
sn).
A is a set of actions: A
an).
γ ∈ [
the tradeoff between exploration and exploitation [11]. As given by [11]: “To obtain a lot of reward, a reinforcement learning agent must prefer actions that it has tried in the past and found to be effective in producing reward.
But to discover such actions, it has to try actions that it has not selected before. The agent has to exploit what it has already = (s1, s2, ex…pe,rienced in order to obtain reward, but it
also has to explore in order to make better …, action selections in the future”. There are different strategies that can be used to handle this problem: on the reward in comparison to immediate rewards. P defines the probability of transitions from s to s’ when taking action a in state s:
Pss’ = Pr{st+1 = sts=’ s|, at = a} R defines the reward function for each of the transitions, the reward we get if we take action a in state s and end up in state s’: Rss’ = E{rt+1 | st = s, at = a, st+1 = s’} The goal of the agent is to maximize the total reward it receives. The agent should maximize the total cumulative reward it receives in the long run, not just the immediate reward [11]. The expected discounted reward is defined as follows by [11]: Gt = Rt+1 + γ t+R2 + 2γRt+3 + … =
γk Rt+k+1 The sequence of states that end up in a terminal state is called an episode. The general process of RL may be defined as follows: 1. At each time step t, the agent is in a state s(t). 2. The agent choses one of the possible actions in this state, a(t), and applies that action. 3. After applying the action, the agent transitions in a new state s(t+1) and gets a numerical reward r(t) from the environment. 4. If the new state is not terminal, the agent repeats the step 2, otherwise the episode is finished. 2.1 Exploration dilemma and exploitation 1. Random policy: during the training concepts, student knowledge and how they are process the agent always chooses related to each other. The student starts random actions. This means that it learning the course material. The system gives always explores and does not exploit the student a lesson that teaches some what it has already learned. concepts. Depending on the student ability to 2. Greedy policy: during the training learn, he/she may learn these concepts or not. process the agent always chooses the If the student does not learn all the concepts given by the current lesson, the system cannot action that gives the best reward. In give him/her a new lesson. So, the system this way, it is always exploiting the should make sure that the student has absorbed knowledge that has gained and uses it all the material given by the current lesson to choose the action that gives the best before giving the next one. We propose the use reward. of reinforcement learning to train the 3. Epsilon-greedy: this method balances pedagogical module that based on student the tradeoff between exploration and knowledge and the concepts that are taught by exploitation. With probability epsilon each lesson to decide what lesson to give ε it chooses a random action, and withhim/her. The system will start by giving the probability 1- ε it chooses the best first lesson, and then following the student action. The epsilon value decreases progress will give every other lesson until the with time reducing exploration and end of the course. To model this as a reinforcement learning problem, we need to increasing exploitation in order to define the set of states, actions and rewards. In make use of the knowledge is gained. [15] we have given a definition of those 4. Boltzmann (soft-max) exploration: elements that create a framework for doing the one problem of the epsilon-greedy training using reinforcement learning method is that the exploration action is approach. One problem that arises when selected uniform randomly from the dealing with reinforcement learning is the fact set of actions. This means that it is that in order to do the training, it is required a equally likely to choose the worst relatively large number of iterations and data. appearing action and the second-best This cannot be achieved using real students, appearing action. The Boltzmann because the process would be very long. In exploration uses the Boltzmann [15] we have proposed the use of a simulated distribution [12] to assign a student that can be used during the training probability to the actions Pt(a): process. The student has some ability to learn which is given in the form of a learning probability, and this defines his/her ability to learn every concept that is taught by the lessons of the course.
agent selects4. Experimental results
The model that we propose focuses on the pedagogical module of the intelligent tutoring system. This is a system for teaching lessons of Python programming language based on concepts and student knowledge. The learning material is composed of lessons. Every lesson teaches some concepts and may require some previous concepts to be known by the student. In [14] we give a definition of lessons, We have done the training in a simulated environment by simulating the behavior of the student. For every episode the student starts with knowing random concepts, and the system tries to learn what is the next lesson to give. We have used the DQN algorithm as given by [13], using memory replay and target network. Figure 1 gives the architecture of the target and train networks. The hyper parameters used during the training are given in the Figure 2. The training is done using different exploration strategies for the same number of episodes. For each of the strategies we give the total reward received for every episode during the training process in figures 3, 4, 5, 6. After we performed the training, we have tested the performance of each of the models learned by using them in simulations, for 100 episodes with a student that knows random concepts and learning probability the same as the one used during the training process. For each of the tests, we show the total reward received and the length for each episode of the training in figures 7 to 14.
In this work we have compared the performance of different exploration strategies used in training an intelligent tutoring system using reinforcement learning. We took into consideration 4 strategies: random, greedy, epsilon-greedy and Boltzmann (soft-max). For each of the strategies used, we have considered the reward gained for every episode during the training and testing, to evaluate which one performed better. We saw that during the training phase, random and greedy strategies performed worse.
The reward was negative for every episode, which means that they chose the worst action for most of the time. For the random policy this means that it always explores and never exploits the knowledge. For the greedy policy this means that it always tries to exploit its knowledge, but it never explores for new actions that may be more profitable. On the other hand, the epsilon-greedy and Boltzmann strategies performed best during the training phase, with Boltzmann strategy getting slightly higher rewards. These strategies use a combination of exploration and exploitation, which makes them perform better.
During the testing phase we see that greedy policy performs worse than every other policy. This shows that the system has not learned anything during the training phase. Random and epsilon-greedy policies performed well during the testing phase with almost the same reward gained. Even though random policy performed poorly during the training phase, it did quite well during testing, meaning that the high level of exploration learned some good actions. The Boltzmann policy was the best during the testing phase, getting the highest reward values. This shows that this policy learned better which are the best actions to take. Also, comparing the episode length during the testing phase, Boltzmann strategy has the shortest episode lengths. This shows that it finishes each episode without reaching the episode length limit, meaning that it finishes the episode faster because it takes the right actions.