We argue in this position paper that simulation is an important tool to support the design of technology to support lifelong learning. We discuss various roles that simulation can play in helping the design of technology for lifelong learning, and then present some issues that must be dealt with in building simulations in this context along with some preliminary insights into how to deal with these issues. We illustrate our discussion using SimDoc, a simulation we have developed of a doctoral program, a real world longer-term mentoring environment.
The ubiquitous nature of technology means the time is ripe to
use technology to support lifelong learning [
Technology can already support lifelong learning. One way is via collaborative learning environments aimed at design tasks and information sharing [6]. Another is through mobile technology that has the capacity to enable learners to access learning material from any location while at the same time facilitating communication between learners and their peers or their instructors (mentors) [7]. Technology has been designed that can help learners of all ages to participate in lifelong learning through seeking help and social support, following up recommendations about content, scaffolding of learning, and finding mentors. However, most advanced learning technology research and most educational institutes’ use of technology to support learning have focused on shorter term learning episodes [8]. Such research has led to the development of various systems that are helping thousands of learners in numerous learning contexts and domains.
Perhaps the most ambitious approach to developing technology to support learning is in the area of artificial Copyright held by the author(s). Use permitted under the CC-BY license CreativeCommons.org/licenses/by/4.0/ intelligence in education (AIED), with its focus on personalization, deep modelling, and innovative pedagogical approaches. There have been many successes in AIED, but, again, mostly in restricted domains and shorter term learning contexts. As AIED has begun to venture into lifelong learning, it has faced new challenges. Two of these challenges concern the design and evaluation stages of AIED systems meant to support lifelong learning. Design is costly [9], and it is often impossible to run closely controlled experiments [10].
Given these design and evaluation challenges, it is important to
find a cheaper, faster, and more flexible approach for evaluating
design decisions underlying AIED systems for supporting lifelong
learning. Simulation is a promising approach, analogous to the use
of wind tunnels to evaluate the aerodynamics of various airplane
components [
Use of simulation within AIED research is not a new idea. In the mid-1990s VanLehn, Ohlsson, and Nason [12] asserted that technological advances had made it possible to create simulated pedagogical agents that could exhibit human-like behavior. They identified three main uses of simulation in learning environments: (i) simulation can provide an environment for human instructors to practice their teaching methods; (ii) simulation can present an environment for evaluating different pedagogical instructional designs; (iii) simulated learners can act as learning companions for human learners. Our use of simulation is of type (ii), which has not had nearly as much research as type (iii). However, there has been some type (ii) research, including the development of complex simulation models such as SimStudent [13], simple model simulations such as [14], and medium complexity model simulations such as those provided in [15], [16]. 2
As in the development of AIED systems themselves, most simulations supporting the design of learning technology are focused on short learning episodes (measured at most in months) and covering well-defined subject matter [14]. There is a lot of knowledge and experience on how these subjects are taught and learned that can be drawn upon to inform a simulation [13]. It is therefore often possible to anticipate potential learning outcomes. In a lifelong learning context simulation would enable advanced learning technology researchers to conduct experiments and shed light on AIED systems that are otherwise impractical to investigate because of the nature of the environment, in particular the lengthy time scales involved [17]. Simulation can be used to replicate lifelong learning by modeling a domain’s key characteristics and behavior over a span of time [15]. Further, simulation makes it possible to evaluate AIED systems for supporting lifelong learning without having to wait a lifetime for results [16]. Such evaluation is crucial in determining the implications of using a lifelong learning system. 2.1
When using simulation to explore design issues in building systems to support lifelong learning, system designers need to make decisions as to what system components to include in the final system design, and they need to explore various options in how these components behave [15], [18]. With simulation, designers can use simulated learners embedded in a simulated version of the desired system, exploring the impact of various components and discovering implications of various design decisions [17]. Further, designers can examine the impact of including or excluding certain learner attributes by creating simulated learners with various characteristics, and by manipulating the distribution of the learner population in various ways [16]. In addition, simulation enables system designers to inform various system parameter values [14]. 2.2
The initial exploration of design decisions is a kind of formative evaluation in that it involves examining the behaviour of various versions of an AIED system being developed to detect potential issues and opportunities before actually deploying the real system. But, it is also a kind of summative evaluation in that the outcomes of a particular simulation design are analyzed after the simulation runs. The patterns detected then inform decisions in the next iteration of simulation design. Real world learning environments also use the summative evaluation of one version of a learning system to inform the next, but the design cycles are much longer than in a simulation, the factors at play are much more numerous and interdependent, and there is often little flexibility in what changes can be made in a system design. Simulation thus provides an opportunity to explore formative and summative evaluation and to find interesting connections between the two. We return to this when exploring simulation lifecycle issues below.
There are other research issues associated with formative evaluation such as exploring ways of hooking simulated learners to other systems to explore those systems’ functionality. Such systems might be recommender systems, help systems, or mentoring systems. In addition, examining how simulation can draw from existing learning management systems (LMSs like BlackBoard or Moodle), or online courses like MOOCs (which have large amounts student data) is an opportunity to explore how real world data and simulation data can be mutually informative.
Finally, simulation enables the capturing of fine grained simulation data that allows exploration of many aspects of learning and teaching by data mining the simulation data for interesting patterns. Because by definition the simulation is a simplified model of the complex real world, these patterns can often emerge more clearly than in the real world where noise and the interactions of thousands of variables can obscure important relationships. Of course, it must be kept in mind that any simulation is merely a prediction for the real world and any patterns found in simulated data eventually need confirmation with actual data gathered in a real life learning scenario. 2.3
There are many other factors that affect learning outcomes beyond system design, pedagogical instructional design, and learning content. An example is the social interaction aspect of learning. Simulation enables experimentation with different pedagogical approaches and interaction patterns among learners and teachers to see the impact on learning outcomes. In particular, with simulation it is possible to explore hypothetical “what if” scenarios. These hypothetical scenarios can include situations where there is, as yet, no real world data; situations where it would take too long to get real world data (a standard feature of lifelong learning contexts); artificial configurations of (simulated) learners and (simulated) teachers that could never occur in the real world but that bring out various patterns that could be illuminating; learning environments where certain kinds of support tools are posited (even if not yet built) to see their effect; and so on. The ability to create “what if” scenarios is a major advantage that simulation provides. 3
Before using simulation to explore questions concerning supporting lifelong learning, a system designer must take into account several important factors, which we will discuss in this section. In section 4 we will then illustrate our discussion with lessons drawn from SimDoc, a simulation of a doctoral program, designed by the first author in his Ph.D. thesis [19]. We first discuss important issues to consider when designing and building a lifelong learning simulation model. We then end this section by providing a description of SimDoc, a model of a longer-term learning environment, the doctoral program. 3.1
Simulation model fidelity refers to the degree of similarity between a simulation model and the real world phenomena under study. Different researchers have demonstrated that it is possible to use different levels of model fidelity to gain insight into various pedagogical research issues. While Champaign and Cohen [14] used a very low fidelity model, Matsuda et al. [13] used a simulation model with high cognitive fidelity to explore the impact of personalized learning experiences. Medium fidelity simulation modelling has been used in [15] to uncover interesting results.
A system designer must consider the level of model fidelity they want to use before starting the simulation study. This is particularly important because the level of simulation fidelity affects the interpretation of the simulation results. The fidelity should be detailed enough to allow appropriate exploration of the research issues the designer is investigating using the simulation model. 3.2
To successfully use simulation to explore issues concerning supporting lifelong learning, a system designer must have clear research questions. This requires a system designer to think critically about the focus of the research and the design of expected simulation experiments. This will allow the system designer to more fully understand the structure of the target learning environment. Knowing the questions of interest also allows the designer to decide on which elements of the environment and learners to model, as well as the level of simulation model fidelity. Ultimately, this will direct the system designer’s search for data to use to inform the simulation model’s attributes, parameters, key assumptions, and algorithms.
A key issue to focus on here is the availability of data, which can be a big challenge in lifelong learning contexts. A system designer needs to consider beforehand if there are data concerning the phenomena under investigation. Are the data easily accessible? If yes, are the data from a single source or multiple sources? If multiple sources, how do we integrate data from different sources? What is the alternative if data is not accessible: can information be derived from known policies or procedures in the target environment, from related empirical research, from “commonsense” considerations? If data is available, a system designer needs to consider what approaches to use in identifying, collecting, and analyzing the data. After identifying research questions and sources of data, a system designer can then formulate a conceptual model and build the system incrementally until it behaves like the real world system being modelled. 3.3
Often a simulation model will have parameters whose values cannot be directly derived from available raw data. There are at least two ways to determine the parameter values for the missing data. One approach is to use commonsense assumptions. Another way is to use calibration to systematically derive the values for the missing data. Calibration is the process of adjusting numerical parameters in the computational model for the purpose of improving the match between the simulation output and data from the real world system [20]. While performing calibration, a system designer can only vary attributes and parameters that are not yet assigned values from other sources (which is why calibration is another way of informing a simulation). Calibration helps build a well-informed simulation model whose behaviour is statistically like the real world to a designer’s desired level of significance.
One open research issue concerning calibration is how a system designer handles multiple sources of data: is it better to choose one source, to compute an average across multiple sources, or to run multiple versions of the same simulation, each with a different source? 3.4
Once calibration has been performed to tune the parameters, it is important for the system designer to validate the resulting “best tuned” simulation model. Validation involves checking that a calibrated simulation model’s output and behavior are statistically like the output and behavior of the real world system under study [21]. This process necessitates prudent experimentation in order to ascertain that the model works as expected. A single simulation run is adequate when the simulation is based on a deterministic model. However, when the simulation model contains stochastic elements, many runs of the simulation model are needed that yield outputs that are both relatively stable but also have appropriate variability. “Stability” means that over time the average of the aggregate outputs of the simulation runs are statistically like the outputs of the real world system. “Appropriate variability” means that the simulation outputs of the various runs vary enough that overfitting hasn’t occurred. A key research issue here is determining how many such simulation runs are necessary in order to consider a simulation model validated. Another issue is how often should validation be done. We suggest that validation is necessary whenever a simulation model is revised. 3.5
Once a system designer has built, calibrated, and validated a simulation model, multiple experiments can be run in a relatively short amount of time. In addition to this advantage of time saving, a researcher can create hypothetical experimental setups to explore “what if” questions, as discussed above.
Once the simulation experiments are finished, the next question is, can the simulation be reused to explore new questions? Research issues that a researcher needs to address when reusing a simulation concern identifying the exact focus of the reuse. Is it based on improving the simulation model’s fidelity level such as adding new data derived from the literature, or further probing of the real-world setting, or adding new data derived from insights learned from observing the results of the first simulation experiments? Is the emphasis on change in the simulation model itself, which might involve adding or subtracting parameters, learner model attributes, or the number of agent types? Is it necessary to rebuild the simulation from scratch or is it possible to build on the existing model? At the very least, the next iteration of the simulation can draw on lessons gleaned from the previous iteration(s), but any new simulation, even if built on an existing model, almost certainly requires performing anew the steps of informing, calibrating, and validating the model. 4
In this section we will discuss how the factors introduced in section 3 played out in the design of our SimDoc simulation of a doctoral program [16], [18], [19]. We designed SimDoc to explore issues in Ph.D. students’ time in program and dropout rates. We decided on a medium fidelity simulation since we wanted to model a number of real world attributes (more than low fidelity models that explore interactions of one or two parameters as in [14]), but we didn’t have access to vast amounts of fine-grained data (as in [13]) that would have allowed high fidelity modelling.
Here is a very brief overview of SimDoc’s conceptual model. SimDoc has five key components: agents, normative rules, dialogic rules, events, and scenes based on features for building an electronic institution proposed by Esteva et al. [22] as illustrated in Figure 1. We modeled SimDoc’s entities following the agentbased modeling (ABM) [23] technique. Using ABM enables modelers to capture and represent characteristics of modeled elements on an individual basis.
In SimDoc, we modeled two types of agents representing students and faculty supervisors. Each of these agent types plays different roles within SimDoc. We use the normative model to capture the complex characteristics of a doctoral program that result from various interactions that happen between different doctoral program elements. These normative rules inform the behaviour and evaluation functions at an appropriate level of fidelity for the research issues we wish to investigate. The dialogic model represents the interaction strategies used within SimDoc. We used the notion of a scene to capture a single interaction that happens between two agents (such as a supervisor and student). To capture the various events that take place within the doctoral program (e.g. taking comprehensive exams or defending thesis proposals), we used an event model. These events trigger action and reactions by agents.
In SimDoc we wanted to investigate ways of improving student outcomes (shorter time in program, fewer dropouts) through exploring a variety of supervisor-student mentoring relationships. We therefore informed SimDoc appropriately first by obtaining 10-years’ worth of raw data gathered about student progress in the University of Saskatchewan doctoral program (the “UofS dataset”). We augmented this “baseline” data with information derived from milestones and policies of the University of Saskatchewan doctoral program and with information derived from research studies identifying supervisor and student types.
We then used system calibration processes (described in 3.3) that allowed us to determine values of several otherwise unassigned parameters in the model. Specifically, we calibrated the system to match the progress of students in the baseline UofS dataset. In our SimDoc simulation the calibration process resulted in a tuning of parameters that matched the UofS dataset with 93% confidence.
We then moved on to validate this “best matching” model, to determine that it was, over time, statistically consistent with the UofS baseline dataset (it had stability), but also had statistical variability to ensure that it wasn’t overfitted (it had appropriate variability). To do this validation, we devised an algorithm that uses Levene, Chi-Square, and ANOVA tests cumulatively on a set of simulation runs and stops when the appropriate statistical properties of the simulation runs have been fulfilled (see [24] for more details). This algorithm stops after a certain number of runs n, which in our SimDoc model turned out to be 100 runs.
We then used this number (100) as the appropriate number of runs that we should make for each simulation experiment, as we explored various supervisor-student interactions as they affected time in program and drop out rates in various “what if” scenarios. We could not do a new validation to determine the number of runs appropriate for each experiment since each “what if” scenario was purely hypothetical where all supervisors were of one type and all students of another (to help expose interesting patterns), not situations with any real world baseline data. In the end, we ran over two dozen experiments. This resulted in over 2400 runs, each a small mini-experiment, that shed interesting light on supervisorstudent interactions and their effect on time in program and dropout rates. For extensive details on these experiments (and all other aspects of SimDoc) see [19]. 5
An open issue associated with using simulation to understand system designs for supporting lifelong learning is identifying the contexts where the use of simulation is advantageous. What learning environment features are favorable for the use of simulation? One important aspect is the availability of some baseline data to inform the simulation. Another seems to be having relatively unambiguous questions to be answered. What other factors are important?
Another issue is to determine when a simulation is accurate enough to be believed. We have identified two features: stability and appropriate variability of the outputs from run to run of the simulation. But, there must be other factors too. The ultimate ‘reality check’ is that designs of lifelong learning support tools arising from simulation studies actually work in the real world.
Ultimately, a simulation is not a one-off. In most design scenarios there will be an iterative design/experiment cycle with simulation in the loop. The initial iteration is instrumental in determining what aspects of the model to focus on and probably what facets to discard. It is hoped that over successive generations of simulation design (with possible real world spin off applications along the way), the simulation can move gradually from medium to high fidelity as data is gathered during each cycle and new capabilities are integrated into the simulation model. Moreover, going forward there should an increasing number of datasets gathered over longer-term use (in MOOCs, in forums such as stack overflow and in other online learning contexts) that would inform a simulation model and provide baseline data for calibration and validation. Exploring how a simulation model evolves through many cycles is an important direction for simulation research.
In conclusion our overall proposition is that simulation should be an essential tool in a lifelong learning system designer’s toolkit. Simulation can be used to explore various aspects of learners’ knowledge acquisition and development, the effects of possible changes in a learning system’s design, the implications of various types of learner interactions with different people and learning environments as they go through life, even hypothetical “what if” scenarios that cannot exist in the real world but nevertheless shed light on issues in lifelong learning.
ACKNOWLEDGMENTS We would like to thank the University of Saskatchewan Data Warehouse team for allowing us to access their dataset, and the Natural Sciences and Engineering Research Council of Canada (NSERC) for funding our research.