The seeking of teaching resources on the web is a very common practice among instructors. Some on-line systems aim to support instructors in this complex and delicate task. In fact, quality of teaching is without doubt an important factor for learning, and an important aspect of teaching is the delivery of quality learning material to students. Remarkable results have been achieved in the reuse and recommendation of teaching resources, producing the corner stone for Information Retrieval (IR) in education. Our research aims to connect those findings with web resources, overcoming the current limitation of analysing resources from educational repositories only. Firstly, this study addresses the problem of identifying those attributes of a teaching context that actually carry on a piece of information useful for rating web resources for teaching. Then, the project works on the design of WebEduRank, a ranking principle for rating webresources according to the teaching requirements and needs of an instructor. So far, the research has already identified the attributes that are expected to positively inform the WebEduRank. Using these attributes, the design of the WebEduRankis now completed and the methodology for its validation is already devised and implemented.
Learning Object Repositories (LOR) are a very reliable source of Learning Objects
(LO), where the recommendation approaches are facilitated by the metadata that those
objects come with [
Focusing on the applications of IR techniques for the retrieval of resources for teaching, it would be interesting to do not limit this search in resources hosted by LORs or other educational datasets, but to explore also the web. In practice, we are talking of going out on the web where we cannot rely on the fact that we have educational metadata like in LORs, and we do not even know if they actually are educational resources.
This research mainly aims to propose a method, called WebEduRank, for ranking or re-ranking web resources according to their suitability for a teaching context. The desired goal for the novel ranking principle is to provide a higher ranking of those web resources that present characteristics of interest for a teaching context. 2
Learning Object is currently the most relevant approach for the description of
educational characteristics of digital content, and it is adopted by most of the recommender
systems of educational resources [
Learning Object has been widely used by recommender systems for achieving
different recommendation goals. Some studies focused on recommending Learning
Objects according to course structure and learning styles, among other aspects [
The metadata of Learning Objects can also facilitate the process of query expansion
for a more effective retrieval of resources [
This section aims to present the process for designing a teaching context towards the proposal of WebEduRank. The teaching context has the delicate role of representing the teaching requirements and experiences of instructors. This task is very important for proposing a ranking principle of web resources able to take into account the educational context of a profiled instructor. In order to achieve this goal, the teaching context should exhaustively represent the teaching requirements of an instructor that the web resources have to comply with.
From an analysis of a review study of instructors’ knowledge involved in the
process of teaching (called teaching knowledge), we have found that most of the
contributions refer to some important categories of teaching knowledge such as Content
Knowledge, Pedagogical Knowledge and Pedagogical Content Knowledge among others [
After the review of literature about the teaching knowledge and Learning Object, a first draft of the teaching context is presented. We suggest a teaching context TC of a course formed by Prerequisite Knowledge (PK), Concept Name (CN), Course Title (CT), Education Level (ED-LEV), Difficulty (DIFF), Starting Knowledge (SK), Target Knowledge (TK). 3.1
This evaluation is only a first refining of the teaching context, so that the WebEduRank can be focused on only those aspects that are likely to be useful for our purposes. Then, the final evaluation of the WebEduRank is the final evidence whether or not such aspects have been useful and exhaustive. The evaluation has been carried out by surveying instructors about the usefulness of the attributes of the proposed teaching context for ranking web resources for teaching.
The research question that piloted the formulation and structure of the questionnaire is the following:
RQ1: What information of the proposed Teaching Context is perceived useful by instructors for informing the WebEduRank for the retrieval of web resources for teaching? For addressing the research question, the questionnaire simply focused on presenting the attributes of the Teaching Context to the participants and asking them the perceived usefulness and comprehensiveness of such information for the process of ranking teaching resources. So far, a total of 33 responses have been collected; most of the participants have been recruited by e-mail. Such number of participants already allows us to conduct some statistical analysis with a good degree of significance.
Structure of the questionnaire We approached the research question with a questionnaire of 11 items divided into three constructs as follow: – General questions (3). – Attitude in using the Internet for seeking teaching material (6). – Perceived usefulness of the attributes of the teaching context (2).
The first construct provides some general information about the sample such as age and teaching experience. We assume that instructors with different age, teaching experience and place where they teach consider a teaching context similarly. Hence, this first construct is only for describing the sample and for further analysis of the results.
Instead, the second construct shows the current attitude, and so practical experience, of instructors in using current on-line systems for seeking teaching resources. This aspect should also tell us if the sample massively use the Internet for such purposes, or it is just a beginner user in this regard. We believe that answers provided by instructors that massively use the Internet for teaching can provide us with more reliable and, more importantly, experienced feedback. In this construct, there are six 5-point Likert scale questions.
Finally, the last construct is related to the perceived usefulness of the attributes in the proposed teaching context for seeking teaching resources on the web. This evaluation is based on what information instructors usually consider when they look for resources on the web; information used in different phases of the search, from formulating the query to browsing the results. Two items belong to this construct. The first item is a multiplechoice question where instructors are asked to select those attributes that they think useful when searching teaching resources on the web for a specific concept. Participants have to choose at least one option, where also the ’none of the above’ option is available. There is an open field for additional considerations by participants. The second question is a matrix of 5-points Likert scales, where each row represents one attribute of the proposed Teaching Context; participants are required to assign a relevance score to each attribute. Also for this question, there is an open field for additional comments from the participants.
Data analysis For analysing the results of the questionnaire, statistics for measuring the perceived usefulness of respondents and repeatability of the results is used. The questions involved in this task are 5-points Likert scales, so we can use statistical hypotheses test methods for rejecting or not the null hypothesis that an attribute in the Teaching Context is believed useless by the population of instructors, not only the sample of this study. The null hypothesis can be stated as follows:
H0 : µrelevance 6 3:0 (1) If H0 is rejected, it means that the research hypothesis, namely the true mean relevance for the population is greater than 3.0, is retained. A value greater than 3.0 in a 5-points Likert scale is a positive response to the relevance of the attribute. The null hypothesis is tested for all the attributes in the Teaching Context.
The application of t for this study is two-fold: i) retain or reject H0 for the population, and ii) compute a confidence interval for the population mean of the relevance of all the attributes in the Teaching Context.
ceived usefulness of the attributes of the Teaching Context according to the surveyed instructors (N = 33). We use t distributions for testing H0 for all the attributes of the Teaching Context. For each attribute, it is reported whether or not H0 (it means the attribute is not relevant) is retained or rejected.
Two questions of the questionnaire are focused on the relevance of the attributes of the Teaching Context; only the most important one for the purposes of the WebEduRank is presented here.
By means of 5-point Likert scales, participants evaluated the relevance of all the attributes of the Teaching Context, in order to capture to what extent the attributes of the proposed Teaching Context actually represent important aspects of a teaching context, independently of the specific retrieval task. For each attribute, all the participants are required to express the relevance. Looking at the results achieved so far, other than the attributes country and course title, all the attributes got a very positive outcome.
In order to reject H0 for the degree of freedom equal to 32 (N-1), t must be greater than 2.738 at .01 significance level. All the attributes but course title and country rejected H0, presenting a t-value significantly over 2.738 and p-values smaller than .05.
course title is a borderline case where H0 is retained but the p-value is just over .05 and the lower bound of the confidence interval mean for the population is 2.995, slightly lower than the desired value of 3.0. Totally different is the situation for the attribute country, where the lower bound of the population mean is 2.348, remarkably lower than 3.0. For this reason, only the attribute country is removed. 4
The WebEduRank The WebEduRank aims to rate web pages according to some aspects of the teaching of an instructor. The rating should reflect the suitability of a web resource for teaching a certain concept in a particular teaching context or situation. A first note is that the proposed principle is not much focused on the ranking of the resources according to a topic. Google is already able to do it very well and with a remarkable performance. The main problem that the WebEduRank is trying to solve is the ordering of a set of web resources according to their suitability for teaching in a certain context. At this stage of the research, we consider as input a set of web resources that is assumed to be about the topic of a query, for example the results of a Google interrogation. The novel contribution of the WebEduRank would be the re-ranking of the web resources about a topic according to the teaching context of an instructor. This task is not simple at all, mainly because of the problematic structure of a web resource and the detection of educational attributes by text-analysis only. The structure of web resources, sometimes even unstructured, is a very well-known problem, challenging many activities around the text-analysis of web pages including information retrieval. However, this issue is better addressed by studies related to semantic web and web crawling, which are not research areas of this project.
Instead, the detection of educational attributes by text-analysis is a main problem to be addressed in this research. In order to achieve this goal, the WebEduRank has to be able to i) capture educational traits of web resources, and then ii) use such traits for ranking them accordingly to the teaching context of the instructor. These two problems are addressed in this research, where the traits to be extracted from web resources are those included in the Instructor Profile presented by this thesis. 4.1
According to the finding in Section 3, during the search of teaching resources on the web, an instructor knows that he is looking for resources about a concept (CN) for a particular course (CT), when some prerequisite knowledge (PK) is assumed to be already learnt by the students. Implicitly, the resources have to comply with the education level (ED-LEV) of the class with a certain level of difficulty (DIFF). The course may also be a bit different in terms of concepts taught in it because of the starting knowledge (SK) of a class. Finally, the instructor knows what is the target knowledge (TK) of the course. Some of these elements, such as PK, SK and TK can be derived by analysing the concept map of the teaching context, where the SK can be represented by source concepts (concepts with no prerequisites) in the map, and the TK can consist of the leaves of the concept map.
The proposed WebEduRank should replicate the aforementioned behaviour of
instructors when ranking web resources for teaching. The Teaching Context here
presented can provide the WebEduRank with such information, now the problem is how to
use these attributes for analysing web resources accordingly to the process previously
described. The purpose is to use these data for ranking the resources, not for
formulating or expanding the user query. It would be interesting to deeply explore also these
aspects, but they are out of the reach of this Ph.D. project. Only a shallow
investigation has been conducted within this research, and the results are presented in [
The next challenge is in which parts of a web page the principle should look for that
information. We can expect that some information of the teaching context is more likely
to be found in the links of a web page (e.g. prerequisite knowledge or concepts) than
the body or the title. Usually, for ranking web pages the body is the part of the page that
is mostly analysed beside to some metadata if available. However, within the body of a
web page we can find different tags that may contain different kind of information. For
example, links (expressed by the html tag a) can have as text the name of other related
concepts. So, we want to distinguish the text that comes from the following four parts of
a web page: title, body, links and highlights. We believe that these parts of a web page
can contain different attributes of the Teaching Context, reflecting a more informative
information than just finding the information in the whole body-text. The identification
of those four parts of the web pages is based on associating some HTML tags that are
usually used for expressing those parts of a web page. Table 1 shows the HTML tags
that are extracted from a web page and whose text is used for composing the four texts
of the web page that the WebEduRank will analyse.
The last step for the design of the WebEduRank is defining the way of how the attributes
of the Teaching Context are matched or searched into the four texts representing the web
page. As anticipated earlier, we expect that the attributes can appear differently, we can
say with a certain likelihood, into the four components of the web page defined in this
research. In this stage, we have 7 attributes of the Teaching Context considered over
the rating process of WebEduRank. We propose the Expectancy Appearance Matrix
(EAM) for representing the expectation that an attribute appears into a component of
the web page. Formally, EAM is a 4x7 matrix, where the rows are the four components
of the web pages analysed by the WebEduRank, and the columns are the attributes of the
Teaching Context. The element of ai j 2 EAM is the likelihood that the j-th attribute is
found in the i-th part of the web page.The events of finding or not an attribute in a certain
part of the web page are independent of each other, because an attribute can appear in a
section of the web page without any correlation with the appearance in other sections. In
this regard, the definition of the elements of the EAM raises a new challenge, allowing
the tuning of the WebEduRank for discovering the most appropriate values for each
element of the matrix. The analysis of the dataset DAJEE [
The idea behind the WebEduRank is to find a match between the text of a web resource and the text of the attributes of the Teaching Context, being the values of the Teaching Context are texts for describing the different aspects of the teaching. The matching of the texts is performed dividing the web page into four sections providing different information about the content of the web page. The WebEduRank basically analyses the frequency of the terms of the values of the attributes of the Teaching Context with a weighting system based on the values in the EAM. As introduced in Section 4.2, the columns EAM indicate the expectancy that an attribute of the Teaching Context of an instructor appears in the four sections of the web page. Hence, the frequency of the terms of an attribute j-th into a section i-th of the web page is weighted by the expectancy value ai j reported by the EAM. In practice, each column of the EAM is a vector stating the importance of considering an attribute of the Teaching Context along the four sections of a web page. 5
This section presents the experiment for validating the WebEduRank. Part of this experiment consists of collecting teaching contexts from instructors. Then, instructors are asked to provide the rating of a set of web resources according to their suitability for teaching a concept in the context defined by them. This preliminary phase has to be conducted because the teaching context used for informing the WebEduRank is a novel approach, so there are no datasets that we can use for our purposes. For this reason, we need instructors for defining a collection of teaching contexts and web resources so that we can run a number of experiments for tuning and validating our new proposal against current practice and baseline methods. In addition, we do not have the capability of Google in indexing most of the web pages on the web, so the data collected are about web pages retrieved by Google after a query submitted by instructors. In this way, the WebEduRank proposes a ranking of the same resources presented by Google, overcoming the issue of gathering web pages to be rated by instructors in a particular teaching context.
The goal of this evaluation is two-fold: tuning the WebEduRank with different settings of the EAM and proving its improvement in the ranking of web resources for teaching compared to some baselines. As said in the previous section, the EAM is a critical aspect of the WebEduRank, so it needs to be carefully tuned for achieving the best performance that the WebEduRank can actually reach. For this reason, we can have a number of different settings of the ranking principle to be evaluated against themselves first, and then against other approaches. Therefore, we can formulate the following research question:
RQ2: Given a set of web resources about a topic, can the WebEduRank offer an educational ranking of the web resources better than current practices? 5.1
For addressing the research question, the experiments need a set of data on which the
benchmark systems and our proposal can be run. Usually, web search results produced
by an IR system are compared to what the users of those systems actually believe
useful and relevant for their query. In this regard, it has been pointed out a small but yet
important difference between usefulness and relevance of an object for the web search
[
In this study we need to undertake the data collection about the usefulness of some
web resources for teaching of a concept in a certain teaching context; this is the purpose
(or user need) in our case. For this reason, we ask instructors to i) define a teaching
context of their interest, which includes a concept map, ii) formulate a query for retrieving
web resources for a concept of the concept map using Google, and iii) rate the retrieved
resources according to their usefulness for teaching the concept in the given teaching
context. In practice, instructors are the external assessors of the web pages presented by
Google, where the same assessors define the teaching context and formulate the query.
We believe that this protocol for the data collection cover the three points highlighted by
[
Once the data collection phase is completed, we can finally run a number of experiments for validating our proposal. Traditionally, when evaluating an IR system the rating of resources provided by the external assessors are considered the most reliable and truthful ratings. A number of metrics are usually involved for measuring how close to or far from the assessors’ ratings is the ranking or order of the resources produced by the novel IR system. 5.2
A first experiment is used for comparing the performance of the different tuning of the
WebEduRank and some baselines. The goal of this experiment is to have evidence of
the improved ranking produced by the novel approach against some simple baseline
methods. The baseline methods consist of two scoring approaches: i) the documents are
scored based only on the terms in the query provided by the instructors during the data
collection phase, ii) plain WebEduRank where TF-IDF score of the documents is applied
using a query composed by the values of the attributes of the Teaching Context. More
details about the benchmark systems and baseline are reported in the next subsection.
For this first experiment, prediction-accuracy metrics are more suitable for electing the
best approach. In particular, Root Mean Squared Error (RMSE) and Mean Average
Precision (MAP) [
RMSE = √ 1
å (ri ri)2 jitemsj i2items b (2) where items is the set of web resources rated by instructors during the phase of data collection, ri is the rating for the resource i-th given by the user and ri is the rating for b the i-th object predicted by the system. The latter metric simply computes the mean of the average precision of the results presented by the subject system. In this regard, a resource is considered relevant when it has been given a rate of 3 out of 5 by the external assessors. During the collection data phase, the first ten results presented by Google have been rated by the assessors, so we decide to average the top-3 precision measure of the result sets. The results are ordered by descent rating as produced by the subject system. It is expected that a configuration of the WebEduRank performs better than the baselines according to these two metrics. The best tuning of the WebEduRank will be tested against the benchmark systems in the next experiment.
During the second experiment, the WebEduRank is tested against Google which is
pointed out to be the most popular system for retrieving teaching resources [
The DCG metric tells us to what extent the produced ranking reflects the relevance
of the resources, by discounting the gain for users of recommending relevant resources
at lower positions. The formula of DCG given a list of items J is the following [
DCG = max(1;log2 j) where CG[ j] expresses the cumulative gain that the user has from being recommended the item at position j within the list of items J. The gain values are cumulative, so, for each J, it is built a Gain vector (G) of the 10 presented items (ten is the number of resources rated for each query). The element G[ j] is the user-relevance of the item returned in position j. Then, the Cumulated Gain (CG) is simply the sum of the gains from position 1 to j, recursively defined as follows:
CG[ j] =
{G[
CG[ j 1] + G[ j]; otherwise (3) (4)
Similarly, tAP works on the positions of the items but with a different approach. It
measures the differences in the ordering of resources as provided by the user and the
one produced by the subject system. This particular metric is more sensible of
differences happening in the highest positions of the list than the ones in the bottom of the
list. Hence, tAP is more strict when resources at the top positions of the user ranking
appear differently or appear at the lowest positions in the order generated by the subject
method. The formula of this metric is [
J 2
J C(i)
å(
1 i=2 i 1
where C(i) is the number of correctly ranked items above position i.
The only benchmark system used in this study is Google. We do not need to spend
many words for justifying this decision, being Google the most reliable search engine
also for teaching. However, before testing our proposal against Google, we have to
show some improvements that WebEduRank can offer compared to some baseline
approaches. Also, this comparison allows us to test the WebEduRank with different tuning
of the EAM. In this study, two baseline approaches are used. The first one does not use
any information from the Teaching Context and it is used with the aim of motivating the
usage of the Teaching Context for informing the WebEduRank. The second approach
is pretty similar to the method of the WebEduRank, but without using the EAM. Both
methods use the same score function for predicting the rating of the items, based on
the TF-IDF score of a document given the query. The only difference is in the
formulation of the query; the first approach uses the same query used by instructors during
the collection data phase (no usage of Teaching Context), while the second approach
builds the query using the attributes of the Teaching Context by following the structure
of the fifth query presented in Table 2 of [
A successful proposal of a ranking principle of web resources for educational users (i.e. students and instructors) would be a very important improvement in the field of recommender systems in Technology Advanced Learning (TEL). As discussed along this contribution, the predominant limitation of current solutions in TEL against Google or other generic search engines is the number of resources and data that these systems can handle. The proposal of this doctoral work will put a first important step towards the consideration of web resources into the recommendation process of teaching resources.
As part of the proposal of this research, the definition of the teaching context, which
is expected to positively inform the proposed WebEduRank, may finally concentrate the
proposal of new recommendation approaches on those aspects that are actually
important for such task. Finally, the impact of building such teaching context should be
evaluated. After a first analysis of the data about MOOCs from Coursera offered by the
DAJEE dataset [