Non-expert users find complex to gain richer insights into the increasingly amount of available data. Advanced data analysis techniques, such as data mining, are difficult to apply due to the fact that (i) a great number of data mining algorithms can be applied to solve the same problem, and (ii) correctly applying data mining techniques always requires dealing with the data quality of sources. Therefore, these non-expert users must be informed about what data mining techniques and parameters-setting are appropriate for being applied to their sources according to their data quality. To this aim, we propose the construction of an automatic recommender built using a knowledge base which contains information about previously solved data mining tasks. The construction of the knowledge base is a critical step in the recommender design. We propose a model-driven approach for the development of a knowledge base, which is automatically fed by a Taverna workflow. Experiments are conducted to show the feasibility of our knowledge base as a resource in an online educational platform, in which instructors of e-learning courses are non-expert data miners who need to discover how their courses are used in order to make informed decisions to improve them.
The increasing availability of data is a great opportunity for everyone to take
advantage of their analysis. Physicians in hospitals, lawyers in the law business, teachers
in high schools or universities, or even regular citizens, would be interested in
applying advanced data analysis techniques to make informed decisions in their daily life.
Importantly, data mining is one of the most prominent technique to discover implicit
knowledge patterns, thus gaining richer insights into data [
However, non-expert users may find complex to apply data mining techniques to
obtain useful results, due to the fact that it is an intrinsically complex process [
User-friendly data mining [
The process of building the data-mining knowledge base starts when a dataset is selected as new source in our Taverna workflow. Then, a set of data quality criteria are measured when some mining algorithms are applied to the dataset. This information is stored in a model which is automatically created by using a model-driven approach. An overview of our approach is shown in Figure 1.
It is worth noting that our knowledge base can be used (i) directly, by non-expert data miners that have certain expertise in data management; or (ii) indirectly, by using a kind of “recommender” that query the knowledge base to guide non-expert data miners by suggesting the best algorithm to be applied to the data, or even to guide experts data miners by suggesting, for example, the algorithms they should use at the beginning of their study.
Some experimentation is conducted in order to evaluate our knowledge base as
a resource for a non-expert data miner in an online educational context: instructors
of e-learning courses are non-expert data miners who need to discover whom and
how their courses are used in order to improve them. Data mining is being profusely
used [
This paper is therefore a step forward to realize the user-friendly data mining. Specifically, the contributions of this paper are as follows: 1. A metamodel that contains those useful concepts for representing models with information about data mining experiments: data’s sources metadata, results of data mining algorithms, and values of data quality criteria. 2. A knowledge base as a repository of models that contains the data mining information. 3. A Taverna workflow for providing a mechanism to obtain all the information to automatically create our knowledge base. 4. A set of experiments addressed to build a recommender are shown as proof of feasibility of our approach
The remainder of this paper is structured as follows: an overview of the related work is presented in section 2. Our knowledge base is introduced in section 3, while the conducted experiments are described in section 4. Finally, conclusions and future work are sketched in section 5. 2
The data mining algorithm selection is at the core of the knowledge discovery
process [
Regarding data mining workflows, the KDDONTO ontology [
There are some projects that allow scientific community to contribute with their
experimentation in improving the knowledge discovery process. The Machine Learning
Experiment Database developed by University of Leuven [
Unlike our proposal, both projects are oriented to support expert data miners. Our knowledge base would help naive data miners and non-experts users to have a kind of guidance about which techniques can or should be used and in which contexts.
Furthermore, although ontologies used in the aforementioned approaches are very
useful for providing semantics, they lack mechanisms for automating the management
(and interchange) of metadata, such as metamodeling [
Our aim in this work is creating a metamodel inspired by the aforementioned
data mining ontologies that allows us to create a set of models to create a knowledge
base for data mining. Moreover, in previous experiments we have demonstrated the
influence of data quality in the results obtained when applying techniques of data
mining [
Our knowledge base brings the results on executing data mining processes on many data sets. It can be therefore used as a resource to keep information about the behavior of different data mining algorithms with regard of the data sources quality and general characteristics of the data set. Collected information can be useful for supporting non-expert users in a decision making process and which is the best data mining algorithm to apply according to the available data. To this aim, our knowledge base contains the following information: Information from input data sets. Metadata from the data sets must be known, as number of attributes and instances, as well as the corresponding data types.
the execution of a data mining algorithm is acquired: data mining technique being executed, predicted attribute and their results.
Data quality criteria. Several quality criteria from the data sets must be measured.
Quality criteria can be related to data sets (e.g. percentages of null values), as well as fields (e.g. field correlation). 3.1
The development of our data mining knowledge base is driven by the development of a scientific workflow. This workflow is in charge of (i) collecting all the required information for our knowledge base (as previously stated), (ii) creating the knowledge base, and (iii) implementing a recommender for data mining algorithms based on our knowledge base.
Scientific workflows are largely recognized as useful paradigms to describe, drive, and share information about experiments5. Specifically, Taverna Workbench6 is used in our approach. Taverna is part of myGrid project7, that aims to produce and use a suite of tools designed to allow international communities to publish and share information.
Our workflow has as a main objective the datasets processing in order to create models to conform the knowledge base. To this end, the workflow begins with the loading of the data source (e.g. .arff files8) on which will be applied a set of data mining algorithms. Then, the type of data mining technique must be executed9. Next step is about to obtain a predicted attribute (usually the last column). Subsequently, in order to have a visual output in the workflow, expert user can select the resulting algorithm values (e.g. correctly classified instances, mean absolute error, precision, etc.), although all these results are part of the obtained model, and all data mining algorithms are executed, leading to a result set. Simultaneously, the workflow measures the quality criteria values of the data source according to some quality criteria. The workflow can be run manually or configured by command line.
Once required information is acquired, the knowledge base is generated as explained in the following subsection. 3.2
Our knowledge base aims to represent in a structured and homogeneous manner all
the necessary data mining concepts. Following the model-driven paradigm [
The aim of our metamodel is being as generic as possible. Therefore, any data related to the aforementioned information about data mining experiments (metadata of data sources, results of data mining algorithms, and values of data quality criteria) is adequately represented in a model. Our models are not restricted to a certain quality criteria, since the metamodel support creating new quality criteria in each model as required. The definition of our metamodel (see Fig. 3) is based on an analysis of several ontologies (see Section 2): DMKBModel. This is the main class that contains the other useful elements for representing a Data Mining Knowledge Base (DMKB). The DMKBModel class allows the specification of a model in which the following information can be stored: input data sets, metadata, data mining algorithms, parameter-setting, data mining results generated when the Taverna workflow is executed, and data quality criteria. 5 http://en.wikipedia.org/wiki/Scientific workflow system 6 http://www.taverna.org.uk/ 7 http://www.myGrid.org.uk 8 Attribute-Relation File Format (ARFF), a file format used by the data mining tool
Weka [
Fig. 2. Our Taverna workflow.
DataSet. It describes data sets used for generating the information included in the knowledge base. Each DataSet is composed of different fields. Also, each data set contains a category and a set of metadata.
Field. It represents a piece of data contained in the DataSet. This piece of data is identified by a name. Also, the kind of field must be defined (by means of an enumeration called FieldKind) and its type (by means of an enumeration called FieldType). This class contains a set of data quality values that are related to the field.
FieldKind. It is an enumeration class for defining the general kind of values that the field instances may have (continuous, categorical or mixed).
FieldType. It is an enumeration class for representing the type of each Field (numeric, date, nominal or string) DataMiningResults. This class represents values of measures for each data set after executing an algorithm (e.g. accuracy).
Algorithm. This class represent information about executed data mining algorithms.
Each algorithm belongs to a specific technique. (e.g. NaiveBayes, J48, RandomTree or Adaboost ).
Parameter. It is a class that represents values of initial parameters when executing an algorithm. This class contains the name of the parameter and a value. Technique. This class defines a set of existing data mining techniques (e.g. a tree, a probability matrix, etc.). It contains a subgroup attribute in case that the algorithm requires to be further classified.
ProblemKind. It defines the different kinds of problem with which the user need is satisfied (e.g. classification, prediction, clustering, etc.).
DataQualityCriteria. It is an abstract class that represents information related to the different criteria that can be presented either in a DataSet (DatasetDataQualityValue) or in each Field (FieldDataQualityValue). For each data quality criteria, a ComputationMode is defined to described how it is calculated (e.g. Pearson correlation method), and a MeasuringUnit that represent the corresponding unit of measure.
DatasetDataQualityValue This class inherits from the DataQualityCriteria class and defines data quality value criteria for a Dataset.
FieldDataQualityValue It inherits from the DataQualityCriteria class and represents a value for specific Field class.
As aforementioned, our Taverna workflow is in charge of handling the model-driven generation of the data mining knowledge base from the acquired information.
When a dataset is processed, all the acquired information is saved in a model conforming to the metamodel presented in Fig. 3. A set of transformations has been developed for creating the models that are integrated in the knowledge base. These transformation are executed in Taverna by means of a Web service.
Our model-driven approach is built on top of the Eclipse Framework10, which is an open source project conceived as a modular platform which can be extended in order to add features to the development environment. Specifically, transformation tasks for 10 http://www.eclipse.org generating models have been supported with the use of Java facilities provided by the Eclipse Modeling Framework (EMF)11. The Java code in listing 1.1 shows an excerpt of the transformation in charge of creating a model within the knowledge base. For each of the data mining algorithms executed by the workflow, the following classes are generated: DataMiningResult, Algorithm, Technique, and ProblemKind; as well as the required existing relationships among them: hasDMResults, algorithms, technique, and problemKind. Finally, the model (represented by means of a XMI file) is created. g ResourceSet r s = new ResourceSetImpl ( ) ; r s . getResourceFactoryRegistry ( ) . getExtensionToFactoryMap ( ) . put ( "xmi" , new XMIResourceFactoryImpl ( ) ) ; Resource r e s o u r c e = r s . createResource (URI . createFileURI ( " ouput generated /" + ds . getName ( ) + " . xmi" ) ) ; r e s o u r c e . getContents ( ) . add ( model ) ;
Code 1.1. Segment of Java code to create a model. dataset, in this case 305).
Our knowledge base is composed by the set of models obtained after running the Taverna workflow for each input data set. These will be the data source which allows us to build our recommender. 3.3
A recommender system takes as input a collection of cases, each belonging to one of
a small number of classes and described by its values for a fixed set of attributes, and
output a classifier that can accurately predict the class to which a new case belongs
(ref [
Our approach has been evaluated in the e-learning domain by carrying out a
experiment. The methodology followed comprises the steps listed below:
1. Selection of courses and data extraction from e-learning platforms.
2. Generation of 96 data sets as described in Sect. 4.1
3. Building of 1152 classification models from the application of 12 classification
algorithms on 96 out of 99 data sets. The rest were used for testing.
4. Extraction of meta-features of each data set
5. Creation of data sets with the meta-features of each data set adding as class
attribute the algorithm or algorithms which achieved the highest accuracy.
6. Building of a recommender of algorithms from our data sets with the meta-features
chosen. We rely on meta-learning to build our recommender since this technique
has been demonstrated suitable to assist users to choose the best algorithm for a
problem at hand [
In what follows, we describe the data sets and classifiers used in our experiment, along the process of building our knowledge base. Next, we explain the building of our recommender in order to show the feasibility of our proposal. 4.1
In our experiments, we used data from eight courses hosted in e-learning platforms at University of Cantabria (Spain): (i) one course, entitled “Introduction to multimedia methods” offered in three academic years (2007-2010) with 70 students enrolled in average and hosted in the Blackboard e-learning platform ; (ii) seven computer science courses taught in the 2007-2008 academic year with a total of 432 enrolled students and hosted in the Moodle Learning Content Management System ; (iii) six courses oriented to train transversal skills imparted during the first semester of 2013 with a range from 20 to 126 learners per course, also hosted in Moodle; and (iv) a semipresential course entitled ”Mathematics for economists” with 465 students enrolled. Training data sets We defined 23 data sets with information extracted from platforms logs. Each instance in every data set represents the activity of a student in an academic year together with the final mark obtained in the course. Two different groups of data sets are considered: the training data set (used to generate the experiments to feed our knowledge base), and the test data sets (used to evaluate the recommender).
In order to have enough data sets for our experimentation, and taking into account generally data from virtual learning environments are clean, we built new data sets performing some controlled perturbations to the original datasets. The new data sets have the quality degraded, which allow us to assess if the meta-features chosen are suitable for this purpose. Furthermore, as the process to be performed by the expert should be about a monitored data set which allow validating the behavior of the algorithms under variations of the quality of data.
We generated 96 data sets from them. First we created 3 data sets with data from multimedia course establishing the class attribute with values pass or fail, and another one as the union of these three. The same process was carried out with the programming course, the ”Mathematics for economists” course and the transversal courses. Next, we generated 4 discretised data sets from the previous bi-class data sets using PKIDiscretize from Weka, and 4 data sets more but these partially discretised. Besides, we created two data sets with 4 classes (fail, pass, good, excellent) and one with 5 classes (drop-out, fail, pass, good, and excellent). These are our 23 original data sets whose main features are shown in Table 1. Data sets numered from 1 to 11 correspond to the “Introduction to multimedia methods”, those from 12 to 15 correspond to the computer science courses, data set 16 and 17 are from the ”Mathematics for economists” course and finally data sets numbered from 18 to 23 correspond to the transversal courses.
Then, we generated 72 data sets by adding to the first eighteen data sets from Table 1 a 10, 20, 30 and 40% of missing values. And finally, we created 4 data sets more by applying SMOTE algorithm on 2 of our original data sets with the following proportion of balancing class: 80-20%, 85-15%, 70-10% and 90-10%. Test data sets Our test data sets are described in Table 2. As can be observed, we chose three data sets with different meta-features: the first one contains the activity carried out by the students in the 2009-2010 academic year in the “Introduction to Multimedia” course (mult2class2010), it is bi-class and all attributes except the class, are numerical; the second one, collects the activity performed in the three editions of Multimedia course degraded with a 10% of missing values (multGlobalActivity); and finally, the third one gathers data from the six transversal courses mentioned above (tranversalDS) in an unique file. It is bi-class, balanced, without structural nulls, with 2 nominal and 4 numerical attributes.
They were used to evaluate the feasibility of our knowledge base for building a classifier which helps the end-user in the selection of the best algorithm. 4.2
Due to the existence of different classification algorithms, 12 different classifiers
provided by Weka (trees, rules, bayesian, lazy and ensemble) were executed on the
training data sets in order to feed the knowledge base. These classifiers were selected
taking into account the most frequently used data mining algorithms [
The meta-features used in this experimentation can be classified in three groups: general, quality-related and based on information theoretic features. In particular, we selected the number of attributes and instances in the data set, the number of categorical and numerical attributes, the type of data in the data set (numeric, nominal or mixed) and the number of classes. Regarding quality, we chose completeness (percentage of null values) and finally, we used class entropy in order to establish if the class was balanced or not. We defined three possible values for this attribute: balanced, quite unbalanced, highly unbalanced.
Next, we explain how was calculated these two last meta-features.
Missing values Structural null values are considered [
npi)2 npi where { o2bs: { fi: number of observed frequencies. { pi: number of expected frequencies.
{ n is the number of categories to be considered. 4.4
Our knowledge base was fed with results of the training data sets. Each one of the classifiers enumerated in Section 4.2 was applied to the 96 training data sets described in Section 4.1. Results were stored in the knowledge base, together with their corresponding meta-features described in Section 4.3. This means that 1152 different models (96 12) were generated. 4.5
The knowledge base is used by a recommender for selecting the best classifier for an input test data set. Therefore, the goal of this experiment is twofold: on one hand, knowing if the generated knowledge base supports the recommender in its task, and on the other hand, evaluating the goodness of our recommender.
Before knowing which are the best classifiers for each of the test data sets, we performed a clustering process using kMeans on the meta-features of the training data sets in order to discover if there were well defined patterns that we could remark. In table 3 we show the results of the 5 clusters obtained. As can be observed, cluster0 collects the data sets with a high number of instances and the nominal attributes and null instances. Cluster1 contains those data sets with the lowest number of instances and a high number of numerical attributes. Cluster2 and cluster4 are very similar, both with a high number of instances and a 100% of numerical attributes, but differ in the degree of balance, cluster2 gathers quite unbalanced instances and cluster4, highly unbalanced instances. Finally, cluster3 contains instances with a high number of attributes and the highest number of nominal values. This analysis shows that we have a suitable collection of data sets, that means, it is representative enough.
Next, we built classifiers for our test data sets in order to know which one is the technique that best classifies each one. So that, we applied the 12 selected classifiers to the test data sets and these were ranked according to its accuracy. The best algorithms of this ranking are shown in Table 4. The table must be read as follows: the classifier which obtains the best accuracy for the mult2class2010 data set is NaiveBayes, which is followed by RandomForest and NNge, and quite far by KNN, J48 and BayesNet.
Next, we built two different recommenders using J48 and NaiveBayes algorithms, respectively. The meta data set used contained 111 instances, that means, one instance with the meta-features of each data set together the best algorithm which performed the classification task. Since some data sets were classified by more than one algorithm with the same accuracy, these appears twice, once with each algorithm. The data set considered for this task contained the instances of our knowledge base corresponding to the four classifiers that achieved more times the better results, which are (NaiveBayes, J48, Jrip and BayesNet ).
The recommendation given for each data set by each recommender is shown in Table 5. As can be observed, the recommender based on J48 recommends, for multGlobalActivity data set, one of the best classifiers, Jrip; and the best one for mult2class2012 and transversalDS datasets, NaiveBayes and J48 respectively. The recommender based onNaiveBayes recommends one of the best classifiers for the multGlobalActivity dataset, J48, and for transversalDS data set, Jrip. Thus, we conclude that these recommenders select one of the best classification algorithms.
Finally, we built another recommender, in this case, we used the 12 classifiers described in Section 4.2. Results are shown in Table 6. In multGlobalActivity data set, the recommender based on J48 recommends to use Jrip, which is one of the best algorithms to classify this data set. Moreover, for transversalDS data set, it recommends the best classifier, J48. The recommender based on NaiveBayes also recommends one of the best algorithms for mult2class2010 : RandomForest. However, the results are worse than in previous experiment in which we only considered four classifiers for our predictive attribute. This happens because, in this case, RandomForest appears in knowledge base as the best algorithm in the 25% of the cases, which is a high percentage over 12 possible classifiers. For transversalsDS data set, it also recommends RandomForest, which is the 4th better classifier for this data set over 12.
These results demonstrate that our proposal is feasible although it is necessary to have a higher number of experiments in order to get a more general model. It is a little problem in e-learning context because although there are lots of courses hosted in e-learning platforms, not all courses can be used since it is necessary to know how the courses were designed and exploded by learners to be considered to predict performance.
We used other techniques based on landmarking [
The application of data mining techniques are commonly known as a hard process generally based on trial and error empirical methods. As a consequence they can only be applied by a small minority of experts. In this paper, a knowledge base is defined that contains information of previous data mining experiments in order to provide guidance to non-expert users to apply data mining techniques. To generate our knowledge base, a model-driven approach is defined, based on a Taverna workflow. As shown in our experiments, our knowledge base can be useful as a resource for nonexpert data miners. The best classifiers can be recommended most of times from a set of 4 classifiers (NaiveBayes, J48, Jrip, and BayesNet ) in order to predict the performance of students in our e-learning scenario. Moreover, in one of these cases, our knowledge base supports in recommending the best algorithm for two of the data sets. Although, the number of good recommendations were worse when the set of classifiers is 12, these results encourage us to continue researching in order to improve how the recommender can use the knowledge base in a better manner. As future work, we plan to conduct more experiments in order to study how to obtain better results when more classifiers are considered. Regardless the recommender can provide good results to a non-expert user with significantly low effort, more complex recommenders that improve these results could be developed.
Acknowledgments. This work has been partially funded by IN.MIND project from University of Alicante (Spain).