As in any other field where there is a massive increase in product variety, in Technology Enhanced Learning (TEL) there is also a need for better findability of (mainly digital) learning resources. For instance, during the past few years, numerous repositories with digital learning resources have been set up (Tzikopoulos et al., 2007). A prominent European example is European Schoolnet's Learning Resource Exchange (http://lreforschools.eun.org) that federates more than 43,000 learning resources from 25 different content providers in Europe and beyond. The US examples are repositories such as MERLOT (http:// www.merlot.org) that has more than 20,000 learning resources (and about 70,000 registered users) and OER Commons (http://www.oercommons.org) with about 18,000 resources. Apart from learning content, learning resources may also include learning paths (that can help them navigate through appropriate learning resources) or relevant peer-learners (with whom collaborative learning activities can take place).
In this plethora of online learning resources available, and considering the various opportunities for interacting with such resources that often occur in both formal and non-formal settings, all user groups of TEL systems can benefit from services that help them identify suitable learning resources from a potentially overwhelming variety of choices. As a consequence, the concept of recommender systems has already appeared in the TEL-domain. Latest efforts to identify relevant research in this field, and to bring together researchers working on similar topics, have been the annual workshop series of Social Information Retrieval for Technology Enhanced Learning (SIRTEL), and a Special Issue on Social Information Retrieval for TEL in the Journal of Digital Information (Duval et al., 2009). These efforts resulted in a number of interesting conclusions, the main ones being that: a) There is a large number of recommender systems that have been deployed (or that are currently under deployment) in TEL settings; b) The information retrieval goals that TEL recommenders try to achieve are often different to the ones identified in other systems (e.g. product recommenders); c) There is a need to identify the particularities of TEL recommender systems, in order to elaborate on methods for their systematic design, development and evaluation.
In this direction, the present chapter attempts to provide an introduction to issues related to the deployment of recommender systems in TEL settings, keeping in mind the particularities of this application domain. The main contributions of this chapter are the following:
• Discuss the background of recommender systems in TEL, particularly in relation to the particularities of TEL context. • Reflect on user tasks that are supported in TEL settings, and how they compare to typical user tasks in other recommender systems. • Review related work coming from adaptive educational hypermedia (AEH) systems and the learning networks (LN) concept. • Assess the current status of development of TEL recommender systems.
• Provide an outline of particularities and requirements related to the evaluation of TEL recommender systems that can provide a basis for their further application and research in educational applications.
Technology Enhanced Learning and the analysis of the data it generates take place in different types of educational settings which are called macro-context (Vuorikari & Berendt, 2009). It generally has significant influence on what user actions are possible and how they can be interpreted. Examples of these dimensions of macro-context include dimensions such as educational level, formal and informal learning, delivery setting and different user roles.
Examples of the educational level are K-12 education, Higher Education (HE), Vocational Education and Training (VET) and workplace training. A formal setting for learning includes learning offers from educational institutions (e.g. universities, schools) within a curriculum or syllabus framework, and is characterised as highly structured, leading to a specific accreditation and involving domain experts to guarantee quality. This traditionally occurs in teacher-directed environments with person-to-person interactions, in a live and synchronous manner.
An informal setting, on the other hand, is described in the literature as a learning phase of so-called lifelong learners who are not participating in any formal learning and are responsible for their own learning pace and path (Colley, Hodkinson & Malcom, 2002;Longworth, 2003). The learning process depends to a large extent on individual preferences or choices and is often self-directed (Brockett & Hiemstra, 1991). The resources for informal learning might come from sources such as expert communities, work context, training or even friends might offer an opportunity for an informal competence development.
The TEL involvement can be characterised by the provision of blended learning opportunities to purely distant educational ones (Moore, 2003). Blended learning combines traditional face-to-face learning with computer-supported learning (Graham, 2005). Distance education, on the other hand, can be delivered using TEL environments in either synchronous or asynchronous ways. Traditionally, distance learning was more related to self-paced learning and learning-materials interactions that typically occurred in an asynchronous way (Graham, 2005). However, live streaming and virtual, personal learning environments (e.g. Web 2.0) have facilitated the development of synchronous distance learning services in formal educational settings.
Lastly, different actors and needs can be identified in TEL. A distinction can be made between the teacher-directed interaction and learner-directed learning processes. This has ramifications concerning the intended users of TEL environments. This thesis, for example, considers teachers as main users of the system.
While macro-context has large implications for interpretation and design, its aspects are fairly agreed-upon, and it is comparatively easy to measure. Microcontext is a more contested notion and more difficult to measure. However, while macro-context is domain-specific, concepts for micro-context range over more diverse fields.
In the past, the development of recommender systems has been related to a number of relevant user tasks that the recommender system supports within some particular application content. More specifically, Herlocker et al. (2004) have related popular (or less popular) user tasks with recommendation goals:
• Annotation in Context. Providing recommendations while the user is carrying out some other tasks. E.g. Web-recommenders that provide predictions about existing links in the user's typical browsing environment. • Find Good Items. The core recommendation task, recommending users with a number of suggested items. E.g. systems where good items are recommended, often without explaining why these ones are chosen (e.g. showing predicted rating values). • Find All Good Items. Providing recommendations in domains where information completeness is a critical factor (e.g. health or legal cases). It concerns recommending users with an exhaustive list of all relevant items. • Recommend Sequence. Very relevant in systems where users are "consuming" items in a sequence (i.e. one after the other), such as personalised radio and TV applications. It concerns the recommendation of a whole sequence of items, instead of simply a subset of relevant ones. • Just Browsing. Relevant in cases where recommendation is not supporting relevant or "equally good" items, but is trying to expand the search coverage with novel or serendipitous suggestions. It provides a recommended list of good items or some annotation in context, but the rationale for the recommendation is different. • Find Credible Recommender. Relevant in the early stages of getting familiarised with a recommender system, when users want to explore and validate the credibility of the system. Good items or annotations in content can be provided, but the rationale of recommendation may differ (e.g. providing very few novel or serendipitous suggestions that could surprise the users).
Generally speaking, most of the above identified recommendation goals and user tasks are valid in the case of TEL recommender systems as well. For example, a recommender system supporting learners to achieve a specific learning goal, "providing annotation in context" or "recommending a sequence" of learning resources are relevant tasks. However, in comparison to the typical item recommendation scenario, there are several particularities to be considered regarding what kind of learning is desired, e.g. learning a new concept or reinforce existing knowledge may require different type of learning resources. Moreover, for learners with no prior knowledge in a specific domain, relevant pedagogical rules such as Vygotsky's "zone of proximal development" should be applied, e.g. 'recommended learning objects should have a level slightly above learners' current competence level', (Vygotsky 1978).
Different from buying products, learning is an effort that often takes more time and interactions compared to a commercial transaction. Learners rarely achieve a final end state after a fixed time. Instead of buying a product and then owning it, learners achieve different levels of competences that have various levels in different domains. In such scenarios, what is important is identifying the relevant learning goals and supporting learners in achieving them. On the other hand, depending on the context, some particular user task may be prioritised. This could call for recommendations whose time span is longer than the one of product recommendations, or recommendations of similar learning resources, since recapitulation and reiteration are central tasks of the learning process (McCalla 2004).
As for teacher-centred learning context, different tasks need to be supported. These tasks can be broadly distinguished into the ones related to the preparation of lessons, the delivery of the lesson (i.e. the actual teaching), and the ones related to the evaluation. For instance, to prepare a lesson the teacher has certain educational goals to fulfil and needs to match the delivery methods to the profile of the learners (e.g. their previous knowledge). Lesson preparation can include a variety of information seeking tasks, such as finding content to motivate the learners, to recall existing knowledge, to illustrate, visualise and represent new concepts and information. The delivery can be supported in using different pedagogical methods (either supported with TEL or not), whose effectiveness is evaluated according to the goals set. A TEL recommender system could support one or more of these tasks, leading to a variety of recommendation goals.
Thus, although the previously identified user tasks and recommendation goals can be considered valid in a TEL context, there are several particularities and complexities. This means that simply transferring a recommender system from an existing (e.g. commercial) content to TEL may not accurately meet the needs of the targeted users. In TEL, careful analysis of the targeted users and their supported tasks should be carried out, before a recommendation goal is defined and a recommender system is deployed. This means that the TEL recommendation goals can be rather complex: for example, a typical TEL recommender system could suggest a number of alternative learning paths throughout a variety of learning resources, either in the form of learning sequences or hierarchies of interacting learning resources. This should take place in a pedagogically meaningful way that will reflect the individual learning goals and targeted competence levels of the user, depending on proficiency levels, specific interests and the intended application context.
Therefore, the task analysis of TEL recommender systems has to consider a number of context variables such as user attributes, domain characteristics, and intelligent methods that can be engaged to provide personalised recommendations. Extensive work on these topics has been carried out in the past, in the area of adaptive educational hypermedia systems.
Web systems generally suffer from the inability to satisfy the heterogeneous needs of many users. To address this challenge, a particular strand of research that has been called adaptive web systems (or adaptive hypermedia) tried to overcome the shortcomings of traditional 'one-size-fits-all' approaches by exploring ways in which Web-based could adapt their behaviour to the goals, tasks, interests, and other characteristics of interested users (Brusilovsky & Nejdl, 2004). A particular category of adaptive systems has been the one dealing with educational applications, called adaptive educational hypermedia (AEH) systems. Since one can say that AEH systems address issues of high relevance to TEL recommender systems, this section provides a brief overview of related work, trying to identify commonalities and differences that could be of relevance for TEL recommenders.
Adaptive web systems belong to the class of user-adaptive software systems (Schneider-Hufschmidt et al., 1993). According to (Oppermann, 1994) a system is called adaptive "if it is able to change its own characteristics automatically according to the user's needs". Adaptive systems consider the way the user interacts with the system and modify the interface presentation or the system behaviour accordingly (Weibenzahl, 2003). Jameson (2001) adds an important characteristic: "A user-adaptive system is an interactive system which adapts its behaviour to each individual user on the basis of nontrivial inferences from information about that user".
Adaptive systems help users find relevant items in a usually large information space, by essentially engaging three main adaptation technologies (Brusilovsky & Nejdl, 2004): adaptive content selection, adaptive navigation support, and adaptive presentation. The first of these three technologies comes from the field of adaptive information retrieval (IR) (Baudisch, 2001) and is associated with a search-based access to information. When the user searches for relevant information, the system can adaptively select and prioritise the most relevant items. The second technology was introduced by adaptive hypermedia systems (Brusilovsky, 1996) and is associated with a browsing-based access to information. When the user navigates from one item to another, the system can manipulate the links (e.g., hide, sort, annotate) to guide the user adaptively to most relevant information items. The third technology has its roots in the research on adaptive explanation and adaptive presentation in intelligent systems (Moore and Swartout, 1989;Paris, 1988). It deals with presentation, not access to information. When the user gets to a particular page, the system can present its content adaptively.
As Brusilovksy (2001) describes, educational hypermedia was one of the first application areas of adaptive systems. A number of pioneer adaptive educational hypermedia systems were developed between 1990 and 1996, which he roughly divided into two research streams. The systems of one of these streams were created by researchers in the area of intelligent tutoring systems (ITS) who were trying to extend traditional student modelling and adaptation approaches developed in this field to ITS with hypermedia components (Beaumont, 1994;Brusilovsky, Pesin, & Zyryanov, 1993;Gonschorek, & Herzog, 1995;Pérez, Lopistéguy, Gutiérrez, & Usandizaga, 1995). The systems of the other stream were developed by researchers working on educational hypermedia in an attempt to make their systems adapt to individual students (De Bra, 1996;de La Passardiere, & Dufresne, 1992;Hohl, Böcker, & Gunzenhäuser, 1996;Kay, & Kummerfeld, 1994). AEH research has often followed a top-down approach, greatly depending on expert knowledge and involvement in order to identify and model TEL context variables. For example, Cristea (2005) describes a number of expertise-demanding tasks when AEH con-tent is authored: initially creating the resources, labelling them, combining them into what is known as a domain model; then, constructing and maintaining the user model in a static or dynamic way, since it is crucial for achieving successful adaptation in AEH. Generally speaking, in AEH a large amount of user-related information (characterising needs and desires) has to be encoded in the content creation phase. This can take place in formal educational settings when the context variables are usually known, and there is a large amount of AEH research (e.g. dealing with learner and domain models) that can be considered and reused within TEL recommender research. On the other hand, in non-formal settings less expertdemanding approaches need to be explored.
Another strand of work includes research where the context variables are extracted from the contributions of the users. A category of such systems includes learning networks, which connect distributed learners and providers in certain domains (Koper & Tattersall, 2004;Koper et al., 2005). The design and development of learning networks is highly flexible, learner-centric and evolving from the bottom upwards, going beyond formal course and programme-centric models that are imposed from the top downwards. A learning network is populated with many learners and learning activities provided by different stakeholders. Each user is allowed to add, edit, delete or evaluate learning resources at any time. The concept of learning networks (Koper, Rusman, Sloep, 2005) provides meth-ods and technical infrastructures for distributed lifelong learners to support their personal competence development. It takes advantages of the possibilities of the Web 2.0 developments and describes the new dynamics of learning in the networked knowledge society. A learning network is learner-centred and its development emerges from the bottom-up through the participation of the learners. Emergence is the central idea of the learning network concept. Emergence appears when an interacting system of individual actors and resources self-organises to shape higher-level patterns of behavior (Gordon, 1999;Johnson, 2001;Waldrop, 1992).
We can imagine users (e.g. learners) interacting with learning activities in a learning network while their progress is being recorded. Indirect measures like time or learning outcomes, and direct measures like ratings and tags given by users allow identify paths in a learning network which are faster to complete or more attractive than others (e.g. Drachsler at al., 2009;Vuorikari & Koper, 2009). This information can be fed back to other learners in the learning network, providing collective knowledge of the 'swarm of learners' in the learning network. Most learning environments are designed only top-down as oftentimes their structure, learning activities and learning routes are predefined by an educational institution.
Learning networks, on the other hand, take advantage of the user-generated content that is created, shared, rated and adjusted by using Web 2.0 technologies. In the field of TEL several European projects address these bottom-up approaches of creating and sharing knowledge. A large EU-initiative that addresses the creation of informal learning networks is the TENcompetence project (Wilson et al., 2008).
Another category of systems that formulate and define their context variables following a bottom-up approach, are Mash-Up Personal Learning Environments (MUPPLE) (Wild et al., 2008). First such approaches were created by (Liber, 2000;Liber & Johnson, 2008;Wild et al., 2008;Wilson, 2005). The iCamp EU initiative explicitly addresses the integration of Web2.0 sources into MUPPLE, by creating a flexible environment that allows learners to create their own environments for certain learning activities. MUPPLEs are a kind of instance of the learning network concept and therefore share several characteristics with it. They also support informal learning as they require no institutional background and focus on the learner instead of institutional needs like student management or assessments. The learners do not participate in formal courses and neither receive any certification for their competence development. A common problem for MUPPLEs is the amount of data that is gathered already in a short time frame and the unstructured way it is collected. This can make the process of user and domain modelling demanding and unstructured. On the other hand, this is often the case in recommender systems as well, when user and item interactions are explored, e.g. in order to identify user and item similarities.
Many of the AEH systems address formal learning (e.g. Aroyo et al. 2003;De Bra et al. 2002;Kravcik et al. 2004), have equally fine-granulated knowledge domains and can therefore offer personalised recommendations to the learners. They take advantage of technologies like metadata and ontologies to define the relationships, conditions, and dependencies of learning resources and learner models. These systems are mainly used in 'closed-corpus' applications (Brusilovsky & Henze, 2007) where the learning resources can be described by an educational designer through semantic relationships and is therefore a formal learning offer. As mentioned before, in formal educational settings (such as universities) there are usually wellstructured formal relationships like predefined learning plans (curriculum) with locations, student/teacher profiles, and accreditation procedures. All this metadata can be used to recommend courses or personalise learning through the adaptation of the learning resources or the learning environment to the students (Baldoni et al. 2007). One interesting direction in this research is the work on adaptive sequencing which takes into account individual characteristics and preferences for sequencing learning resources (Karampiperis & Sampson, 2005). In AEH there are many design activities needed before the runtime and also during the maintenance of the learning environment. In addition, the knowledge domains in the learning environment need to be described in detail. These aspects make adaptive sequencing and other adaptive hypermedia techniques less applicable for TEL recommendation, where informal learning networks emerge without any highly structured domain model representation.
In informal learning networks, mining techniques need to be used in order to create some representation of the user or domain model. For instance, prior knowledge in informal learning is a rather diffuse parameter because it relies on information given by the learners without any standardisation. To handle the dynamic and diffuse characteristic of prior knowledge, and to bridge the absence of a knowledge domain model, probabilistic techniques like latent semantic analysis are promising (van Bruggen et al., 2004). The absence of maintenance and structure in informal learning is also called the 'open corpus problem'. The open corpus problem applies when an unlimited set of documents is given that cannot be manually structured and indexed with domain concepts and metadata from a community (Brusilovsky and Henze 2007). The open corpus problem also applies to informal learning networks. Therefore, bottom-up recommendation techniques like collaborative filtering are more appropriate because they require nearly no maintenance and improve through the emergent behaviour of the community. Drachsler, Hummel and Koper (2008) analysed how various types of collaborative filtering techniques can be used to support learners in informal learning networks.
Following their conclusions we have to consider the different environmental conditions of informal learning, such as the lack of maintenance and less formal struc-tured learning objects, in order to provide an appropriated navigation support to recommender systems. Learning networks are mainly structured by tags and ratings given by their users, being therefore in contrast with the institutionalised Virtual Learning Environments (VLEs) like Moodle or Blackboard that are used to better manage learning activities and distribute learning resources to learners.
Besides the already mentioned differences for prior knowledge in informal learning, there are also differences in the data sets which are derived from environmental conditions. Normally, the numbers of ratings obtained in recommender systems is usually very small compared to the number of ratings that have to be predicted. Effective prediction by ratings based on small amounts is very essential for recommender systems and has an effect on the selection of a specific recommendation technique. Formal learning can rely on regular evaluations of experts or students upon multiple criteria (e.g., pedagogical quality, technical quality, ease of use) (Manouselis et al., 2007), but in informal learning environments such evaluation procedures are unstructured and few. Formal learning environments like universities often have integrated evaluation procedures for a regular quality evaluation to report to their funding body. With these integrated evaluation procedures more dense data sets can be expected. As a conclusion, the data sets in informal learning context are characterised by the "Sparsity problem" caused by sparse ratings in the data set. Multi-criteria ratings could be beneficial for informal learning to overcome the "Sparsity problem" of the data sets. These multi-criteria ratings have to be reasonable for the community of lifelong learners. The community could rate learning resources on various levels, such as required prior knowledge level (novice to expert), the presentation style of learning resources, and even the level of attractiveness, because keeping students satisfied and motivated is a vital criteria in informal learning. These explicit rating procedures should be supported with several indirect measures, such as "Amount of learners using the learning resource", "Amount of adjustments of a learning resources", in order to measure how up-to-date the learning resource is.
Informal learning is therefore different from well-structured domains, like formal learning. Recommender systems for informal learning have no official maintenance by an institution, mostly rely on its community and most of the time do not contain well-defined metadata structures. Moreover, where formal learning is characteristically top-down designed and develop learning offers (closed-corpus), informal learning offers are emerging from the bottom-up through the communities (open-corpus). Therefore, it will be difficult to transfer and apply recommender systems even from formal to non-formal settings (and vice-versa), since user tasks and recommendation goals are often substantially different.
In the TEL-domain a number of recommender systems have been introduced in order to propose learning resources to users. Such systems could potentially play an important educational role, considering the variety of learning resources that are published online and the benefits of collaboration between tutors and learners (Recker & Wiley, 2000;Recker & Wiley, 2001;Kumar, al., 2005). The following tables provides a selection of some typical approaches, as well as an assessment of their status of development and evaluation.
One of the first attempts to develop a collaborative filtering system for learning resources has been the Altered Vista system (Recker & Walker, 2003;Recker et al., 2003;Walker et al., 2004). The aim of this study was to explore how to collect user-provided evaluations of learning resources, and then to propagate them in the form of word-of-mouth recommendations about the qualities of the resources. The team working on Altered Vista explored several relevant issues, such as the design of its interface (Recker & Wiley, 2000), the development of non-authoritative metadata to store user-provided evaluations (Recker & Wiley, 2001), the design of the system and the review scheme it uses (Recker & Walker, 2003), as well as results from pilot and empirical studies from using the system to recommend to the members of a community both interesting resources and people with similar tastes and beliefs (Recker et a., 2003;Walker et al., 2004).
Another system that has been proposed for the recommendation of learning resources is the RACOFI (Rule-Applying Collaborative Filtering) Composer system (Anderson et al., 2003;Lemire et al., 2005;Lemire, 2005). RACOFI combines two recommendation approaches by integrating a collaborative filtering engine, that works with ratings that users provide for learning resources, with an inference rule engine that is mining association rules between the learning resources and using them for recommendation. RACOFI studies have not yet assessed the pedagogical value of the recommender, nor do they report some evaluation of the system by users. The RACOFI technology is supporting the commercial site inDiscover (http://www.indiscover.net) for music tracks recommendation. In addition, other researchers have reported adopting RACOFI's approach in their own systems as well (Fiaidhi, 2004).
The QSIA (Questions Sharing and Interactive Assignments) for learning resources sharing, assessing and recommendation has been developed by Rafaeli et al. (2004;2005). This system is used in the context of online communities, in order to harness the social perspective in learning and to promote collaboration, online recommendation, and further formation of learner communities. Instead of developing a typical automated recommender system, Rafaeli et al. chose to base QSIA on a mostly user-controlled recommendation process. That is, the user can decide whether to assume control on who advises (friends) or to use a collaborative filtering service. The system has been implemented and used in the context of several learning situations, such as knowledge sharing among faculty and teaching assistants, high school teachers and among students, but no evaluation results have been reported so far (Rafaeli et al., 2004;2005).
In this strand of systems for collaborative filtering of learning resources, the CYCLADES system (Avancini & Straccia, 2005) has proposed an environment where users search, access, and evaluate (rate) digital resources available in repositories found through the Open Archives Initiative (OAI, http://www.openarchives.org). Informally, OAI is an agreement between several digital archives providers in order to offer some minimum level of interoperability between them. Thus, such a system can offer recommendations over resources that are stored in different archives and accessed through an open scheme. The recommendations offered by CYCLADES have been evaluated through a pilot study with about 60 users, which focused on testing the performance (predictive accuracy) of several collaborative filtering algorithms.
A related system is the CoFind prototype (Dron et al., 2000a;Dron et al., 2000b). It also used digital resources that are freely available on the Web but it followed a new approach by applying for the first time folksonomies (tags) for recommendations. The CoFind developers stated that predictions according to preferences were inadequate in a learning context and therefore more user driven bottom-up categories like folksonomies are important. A typical, neighbourhood-based set of collaborative filtering algorithms have been tried in order to support learning object recommendation by Manouselis et al. (2007). The innovative aspect of this study is that the engaged algorithms have been multi-attribute ones, allowing the recommendation service to consider multi-dimensional ratings that users provider on learning resources.
A different approach to learning resources' recommendation has been followed by Shen & Shen (2004). They have developed a recommender system for learning objects that is based on sequencing rules that help users be guided through the concepts of an ontology of topics. The rules are fired when gaps in the competencies of the learners are identified, and then appropriate resources are proposed to the learners. A pilot study with the students of a Network Education college has taken place, providing feedback regarding the users' opinion about the system.
Tang and McCalla proposed an evolving e-learning system, open into new learning resources that may be found online, which includes a hybrid recommendation service (Tang & McCalla 2003;2004a;2004b;2004c;2005). Their system is mainly used for storing and sharing research papers and glossary terms among university students and industry practitioners. Resources are described (tagged) according to their content and technical aspects, but learners also provide feedback about them in the form of ratings. Recommendation takes place both by engaging a Clustering Module (using data clustering techniques to group learners with similar interests) and a Collaborative Filtering Module (using classic collaborative filtering techniques to identify learners with similar interests in each cluster). The authors studied several techniques to enhance the performance of their system, such as the usage of artificial (simulated) learners (Tang & McCalla, 2004c). They have also performed an evaluation study of the system with real learners & McCalla, 2005).
A rather simple recommender system without taking into account any preferences or profile information of the learners was applied by Janssen et al. (2005). However, they conducted a large experiment with a control group and an experimental group. They found positive effects on the effectiveness (completion rates of learning objects) though not on efficiency (time taken to complete the learning resources) for the experimental group as compared to the control group. Nadolski et al. (2009) created a simulation environment for different combination of recommendation algorithms in hybrid recommender system in order to compare them against each other regarding their impact on learners in informal learning networks. They compared various cost intensive ontology based recommendation strategies with light-weight collaborative filtering strategies. Therefore, they created treatment groups for the simulation through combining the recommendation techniques in various ways. Nadolski et al. tested which combination of recommendation techniques in recommendation strategies had a higher effect on the learning outcomes of the learners in a learning network. They concluded that the light-weight collaborative filtering recommendation strategies are not as accurate as the ontology-based strategies but worth-while for informal learning networks when considering the environmental conditions like the lack of maintenance in learning networks. Nadolski et al. study confirmed that providing recommendations leads towards more effective, more satisfied, and faster goal achievement than no recommendation. Furthermore, their study reveals that a light-weight collaborative filtering recommendation technique including a rating mechanism is a good alternative to maintain intensive top-down ontology recommendation techniques.
Moreover, the ISIS system adopts a hybrid approach for recommending learning resources is the one recently proposed by Hummel et al. (2006). The authors build upon a previous simulation study by Koper (2005) in order to propose a system that combines social-based (using data from other learners) with informationbased (using metadata from learner profiles and learning activities) in a hybrid recommender system. They also designed an experiment with real learners. Drachsler (accepted) recently reported the experimental results the ISIS experiment. They found a positive significant effect on efficiency (time taken to complete the learning objects) of the learners after a runtime of four months. It is a very good example of a system that is following the latest trends in learning specifications for representing learner profiles and learning activities.
The same group recently developed a recommender system called ReMashed (Drachsler et. al, 2009a,b) that addresses learners in informal learning networks. They created a mash-up environment that combines sources of users from different Web2.0 services like flickr, delicious.com or Sildeshare. Again they applied a hybrid recommender system that takes advantage of the tag and rating data of the combined Web2.0 sources. The tags that are already given to the Web2.0 sources are used for the cold-start of the recommender system. The users of ReMashed are able to rate the emerging data of all users in the system. The ratings are used for classic collaborative filtering recommendations based on the Duine prediction engine (Van Setten, M., 2005). The same approach is followed by the proposed Learning Object Recommendation Model (LORM) that also follows a hybrid recommendation algorithmic approach and that describes resources upon multiple attributes, but has not yet reported to be implemented in an actual system (Tsai et al., 2006).
Finally, there have been some recent proposals for systems or algorithms that could be used to support recommendation of learning resources. These included and a case-based reasoning recommender that Gomez-Albarran & Jimenez-Diaz (2009) recently proposed.
Nevertheless, despite the increasing number of systems proposed for recommending learning resources, a closer look to the current status of their development and evaluation reveals the lack of systematic evaluation studies in the context of reallife applications. As Table 1 indicates: • More than half of the proposed systems (10 out of 16) still remain at a design or prototyping stage of development; • Only 7 systems have been evaluated through trials that involved human users.
Another interesting observation is that very often, experimental investigation of the recommendation algorithms does not take place. This is a common evaluation practice in systems examined for other domains (e.g. Breese et al., 1998;Deshpande & Karypis, 2004;Papagelis & Plexousakis, 2005;Herlocker et al., 2002), which indicate that careful testing and parameterisation has to be carried out before a recommender system is finally deployed in a real setting. One of the main reasons is that the performance of recommendation algorithms seems to be dependent on the particularities of the application context, therefore, it is advised to experimentally analyse various design choices for a recommender system, before its actual deployment. (Recker & Walker, 2000;Recker & Wiley, 2000;Recker & Walker, 2003;Recker et al., 2003;Walker et al., 2003) Full system
Interface, Algorithm, System usage Human users RACOFI (Anderson et al., 2003;Lemire et al., 2005)
Algorithm System designers QSAI (Rafaeli et al., 2004;Rafaeli et al., 2005) Full system --
Learning object sequencing (Shen & Shen, 2004) Prototype
Evolving e-learning system (Tang & McCalla, 2003;2004a;2004b;2004c;2005) Full system Algorithm, System usage Simulated users, Human users ISIS -Hybrid Personalised Recommender System (Drachsler et al., 2009) Prototype Algorithm, System usage Human users
Recommendation Service (Manouselis et al., 2007) Prototype
Learning Object Recommendation Model (Tsai et al., 2006) Design --RecoSearch (Fiaidhi, 2004) Design --Simulation environment (Nadolski et al., 2009) Full system Algorithm Simulated users ReMashed (Drachsler et al., 2009a,b) Full system Algorithm, System usage Human users
This papers provides an excerpt of a chapter that will appear later in the First Handbook on Recommender Systems. It offers an introduction to the issues related to the deployment of recommender systems in the TEL settings emphasising the particularities of this application domain. To our knowledge, this is the first study attempting to systematically cover the design and deployment of recommender systems in the TEL settings. Nevertheless, it can only provide a brief overview of related issues, leaving several aspects to be further explored and researched.
The paper first discussed the context in which TEL recommenders are deployed, and reflected on related user tasks and recommendation goals. A review of related work coming from the research strands of Adaptive Educational Hypermedia and Learning networks has been provided, with a particular emphasis on how it applies to TEL recommenders for formal and informal learning settings. Then, a survey of TEL recommenders proposed in the literature was presented with a critical view on the actual implementation of systems. This paper has left out the part with a particular emphasis on the evaluation and the discussion on evaluation requirements and issues for TEL recommender systems.
The main research challenge for the future is the one of the systematic development and evaluation of TEL recommender systems. In addition, for the various groups of researchers involved in TEL, a number of topics are of high research interest. For example, the recommendation support for learners in formal and informal learning that takes advantage of contextualised recommender systems has become an important one. These recommender systems, also called context-aware recommender Systems (Lemire et al, 2005), use for example geographical location of a user to recommend relevant resources. Such contextualisation becomes im-portant in situations where multilingual educational resources are recommended from a federation of repositories from a number of countries with different learning standards and/or institutions with different curricula (Vuorikari & Ochoa, 2009). Additionally, context awareness could include pedagogical aspects like prior knowledge, learning goals or study time to embed pedagogical reasoning into collaborative filtering driven recommendations.
Another promising approach is the use of multi-criteria input for recommender system in TEL. Users (learners and teachers) can not only rate learning resource based on the level of complexity, curriculum alignment or how much time is required to cover the learning material, but input also could be inferred from different implicit sources. Such multidimensional input can potentially have a high impact on the suitability of recommendations. A related problem is the lack of TEL specific data sets for informal and formal learning. Different to the recommender system world, where many data sets are available (e.g. MovieLens, BookCrossing, Jester Collaborative Filtering Dataset), the TEL community is still working with rather small home-made data sets, which are rarely public available.
Research of N. Manouselis was funded with support by the European Commission and more specifically, the project ECP-2006-EDU-410012 'Organic.Edunet: A Multilingual Federation of Learning Repositories with Quality Content for the Awareness and Education of European Youth about Organic Agriculture and Agroecology' of the eContentplus Programme. Research of H. Drachsler was funded with support by the European Commission and more specifically, the project IST 027087 'TENCompetence' of the FP6 Programme. Riina Vuorikari thanks the HS-säätiö for the stipend.