Classification is the process of labeling data items as belonging to a given class from a model which is built from a selected set of data. The classification methods promis-ingly and effectively exploit the classification of cultural heritage unstructured data objects. There is a variety of classification methods available in contemporary litera-ture. The choice of a suitable and effective classification method for the classification of cultural heritage data is challenging and nontrivial. In this paper, we are providing an empirical study of classification methods to investigate the performance of a cul-tural heritage database. The classification algorithms for under focus empirical study are namely i.e. Bayesian (BayesNet, NaiveBayes, and NaiveBayesUpdateable), Function (Logistic, Multilayer Perceptron, Simple Logistic, and SMO), Lazy (IBK, KStar, and LWL), Meta (Bagging, Regression, LogitBoost, MultiClassClassifier, and Mul-tiClassClassifierUpdateable) and Rule-based (DecisionTable, JRip, OneR, and PART) classifiers provided in the Weka implementation by considering the av-erage values of each classification method category. This paper contributes in three aspects; 1) to provide a comparison of state-of-the-art classification methods on cultural heritage database; 2) secondly provides evaluation performance within the classification methods category as well as intra categories; 3) classification methods evaluation on the state-of-the-art evaluation metrics. Under focus study use speaker accent recognition data set available publically in UCI Ma-chine Learning repository for all the selected classification methods. The implemen-tation of the selective classifiers in Weka is exploited for the empirical study. The performance of classifiers is evaluated in terms of TP Rate, FP Rate, Precision, Re-call, F-Measure, MCC, ROC Area, PRC Area, and Kappa statistic.
Cultural heritage resources most valuable, non-renewable, scarce, and finite of any specific civilization era in the world. The cultural heritage represents our collective memory, shapes our identity, and also drives the economy [1, 2]. A reliable medium for history and knowledge transferring of a specific civilization is cultural heritage resources in the world. There are vast and diverse types of cultural heritage like paintings, sculptures, coins, manuscripts, monuments, archaeological sites, historical buildings, etc [3, 4]. These cultural heritage resources' accurate and efficient classification is valuable and useful for tourism and future generations. The contemporary literature reveals that there are several classification methods for processing variable performance. The choice of an efficient, accurate, and robust classification method is a challenging and non-trivial task.
The contemporary literature shows that there are several comparative studies done to evaluate the performance of classification methods in different domains like image processing, computer vision, disease detection systems, predictive systems, etc. Jia Wu et.al provided an empirical comparative study on classification methods on public data sets in [5]. S.Taruna and Mrinal Pandey a performance analysis of Classification Techniques to predict the performance of students in the academic field in[6]. Arvinder Kaur and Inderpreet Kaur evaluated the performance of classifiers to predict the faults in open source projects in [7]. Zulfiqar Ali and Waseem Shahzad provided the comparative analysis ACO based Rule Miners in [8]. Zulfiqar Ali et.al presents an empirical study to evaluate the performance of associative classifiers on imbalanced Datasets in KEEL in [9]. Radmila Jankovic et.al investigated the performance of ensemble classifiers on energy consumption in the Balkans in [10]. Zulfiqar Ali et.al provided the empirical performance analysis of machine learning methods to estimate the software development of effort in [11]. Yuanshun Yao et.al provided the analysis of machine learning-based classification models in the context of the complexity of the algorithms and performance [12]. Shelja Sharma et.al evaluated the performance of Bayesian classifiers on public data sets and used evaluation metrics accuracy, kappa statistics, and absolute mean error [13]. Umar Ishfaq et.al presents a comparative analysis of machine learning and deep learning-based classifiers for multiclass prediction in [14]. Chris Zhu et.al exploited augmented reality in the domain of cultural heritage assets in [15].
The contemporary literature reveals that several studies have been proposed for empirical comparison of classification methods in various domains of knowledge like the medical field, image classification, medical image classification, etc. However, these studies do not fully analyze the classifier performance on a cultural heritage database. It demands a more comprehensive empirical study for the classification methods to reveal the comparative performance analysis on cultural heritage databases. This study provides the empirical performance analysis of diverse and widely used classification approaches in the domain of cultural heritage in terms of state-of-the-art metrics.
Therefore, the objectives of this paper include: 1. To compare classification models in the context of a cultural heritage database 2. Evaluate the performance of various types of classification methods within the specific classifiers’ category as well as in the other classification categories. 3. The performance of classifiers is evaluated in terms of TP Rate, FP Rate, Precision, Recall,
F-Measure, MCC, ROC Area, PRC Area, and Kappa statistic.
The paper is organized as follows: Section 2 describes Related Work. Section 3 provides the firsthand introduction of selected classification models. Section 4 describes the data and methodology exploited for empirical study in this paper. Section 5 consists of the analysis performed on the cultural heritage data set. Finally, Section 6 concludes the under-focus comparative study.
.
The contemporary literature reveals that several studies have been proposed for empirical comparison of classification methods in various domains of knowledge like the medical field, image classification, medical image classification, etc. However, these studies do not fully analyze the classifier performance on a cultural heritage database. Marijana ´ Cosovi´ et.al [3] provided a review of classification methods exploited to classify the cultural heritage data objects. This article surveys the contemporary literature for the classification of cultural heritage data and provided the taxonomy for cultural heritage information. Jose Llamas et.al [16] used a convolutional neural network for the classification of images obtained from the architectural heritage. Various deep learning-based classification approaches are exploited and evaluated. Uday Kulkarni et.al proposed a crowdsourcing platform and designed a transfer learning-based classification model in [17]. This crowdsourcing platform collects Indian digital heritage space monuments data, performs classification tasks on images, and labels images retrieved based on queries. Radmila Jankovi'c provided the performance analysis of decision tree-based classification methods by using the Weka data mining tool in [18]. In this, J48, Hoeffding Tree, Random Tree, and Random Forest algorithms are applied for the heritage image classification task.
Marijana Ćosović and Radmila Janković used a Convolutional Neural Network for the classification of cultural heritages images in [19]. The CNN approach evaluated architectural heritage images i.e. bell tower, stained glass, vault, column, outer dome, altar, apse, inner dome, flying buttress, and a gargoyle in this study. Feng Zhipeng and Hamdan Gani [20] provided a new Indonesia cultural events dataset and applied a deep learning-based Hyperparameter Optimization approach for the classification of images related to cultural events. The Keras library is used for the development of the CNN model and experiments are performed in Python v.3.7. Sathit Prasomphan attempted to develop a cultural heritage information management system by using the deep learning approach [21]. Waqar Ali et.al. applied contextual recommendation systems in [22, 23]. A federated learning approach for privacy protection in heritage data recommendation is proposed in [24].
The story of the archaeological site is developed by using machine learning and image processing through features of the archaeological site and linked to the era of the ancient monument's architecture. Heri Kurniawan et.al proposed a framework, called eCultural Heritage and Natural History (eCHNH) in [4]. The proposed framework (eCHNH) is developed based on Zachman Architecture Framework.
A classifier is a machine learning model that is used to discriminate different objects based on certain features. The following subsections provide the basic description of the classification methods used in the focus empirical performance analysis on the cultural heritage database. 3.1.
We have selected BayesNet, NaiveBayes, and NaiveBayesUpdateable classification learning algorithms for the comparative and empirical study in the field of cultural heritage. The Bayesian family of classifiers works based on the Bayes theorem. Table 1 shows the comparative performance analysis of Bayesian-based classifiers namely as BayesNet [25], NaiveBayes [26], and NaiveBayesUpdateable [27] given Weka. 3.2.
The function-based classification methods work to estimate the performance of the function like LibSVM, Logistic, Multilayer Perceptron, RBF Network, Simple Logistic, SMO, SPegasos and voted perceptron. For the comparative analysis of the cultural heritage database, the selected Function-based classifiers are namely i.e. Logistic[28], MultilayerPerceptron [29], Simple Logistic[30], and SMO [31] implemented in Weka 3.3.
The paradigm of the machine in which the process of generalization of the training is delayed until a query is made to the system is known as instance-based learning or Lazy learning classification methods [32]. In this article, we have selected tree instance-based learning classification methods i.e. IBk [33], KStar [34], and LWL[35]. The Weka implementation is exploited for this empirical research study. 3.4.
The paradigm of classification techniques exploits the rules like conjunctive rules for the task of classification [36]. The rule-based classification method is also known as the separate–and–conquer approach. This type of classification method follows an iterative process consisting of the generation of rules that covers a subset of the training samples. Then starts removing all examples covered by the rule from the training set.
For the comparative performance analysis, four Rule-based classifiers are selected namely as DecisionTable [37], JRip [38], OneR [39], and PART [40] implemented in Weka. 3.5.
The type of classification algorithms that use or combine multiple algorithms are known as Meta classification methods like AdaBoostM1, Attribute Selected Classifier, Bagging, Grid Search, Metacast, etc. This study selected five meta classifiers that are Bagging [41] , Regression [28], LogitBoost [42] , MultiClassClassifier and MultiClassClassifierUpdateable implemented in Weka Data Mining tool.
This empirical study uses a speaker accent recognition data set (https://archive.ics.uci.edu/ml/datasets/Speaker+Accent+Recognition) [UCI Machine Learning Repository] for all the classification methods under the focus of the empirical study. The number of instances 329, number of attributes 12 and number of web hits 24149. All attributes of the data set are real and multivariate. We have used 10-fold data validation for the training and testing in this empirical analysis.
For experimentation, we use Weka 3.9 implementation of classification methods. For empirical analysis investigation, we used Bayesian-based Classifiers (BayesNet, NaiveBayes, and NaiveBayesUpdateable) [26], Instance-Based Classifiers (IBk, KStar, LWL), Regression-based Classifiers (Logistic, MultilayerPerceptron, SimpleLogistic and SMO) and Rule-Based Classifiers (DecisionTable, JRip, OneR, PART).
This section provides the analysis of classification methods on cultural heritage data-base namely the "speaker accent recognition data set" publically available. The subsec-tions of this section discuss the performance of classification methods belonging to the major five classification families i.e. Bays Based Classifiers, SVM-based Classifiers, Instance-Based Classifiers, Rule-Based Classifiers, and Meta Classification Methods. All classification algorithms are evaluated on the same and single heritage data set. The performance of each algorithm is measured in terms of TP Rate, FP Rate, Precision, Recall, F-Measure, MCC ROC Area, PRC Area, and Kappa statistic. 5.1.
The Table 5 shows the comparative performance analysis of Meta based classifiers namely as Bagging [41] , Regression [28], LogitBoost [42] , MultiClassClassifier and MultiClassClassifierUpdateable implemented in Weka. The performance of Bayesian classifiers is measured in terms of TP Rate, FP Rate, Precision, Recall, F-Measure, MCC ROC Area, PRC Area, and Kappa statistic as shown in Table 5. The performance of each method is also represented in terms of average, maximum, and minimum values. The results depict that the performance of Regression is promising to other competitive classifiers as shown in Table 5 with boldface values.
Meta Classification Methods
Averag e
Classification plays an important role in data mining and machine learning for the discovery of valuable and nontrivial knowledge from labeled data items. The classification methods promisingly and effectively exploit the classification of cultural heritage unstructured data objects. There is a variety of classification methods available in contemporary literature. This paper provides, an empirical performance analysis of Classification algorithms namely Bayesian(BayesNet, NaiveBayes, and NaiveBayesUpdateable), Function(Logistic, Multilayer Perceptron, Simple Logistic, and SMO), Lazy(IBK, KStar, and LWL), Meta(Bagging, Regression, LogitBoost, MultiClassClassifier, and MultiClassClassifierUpdateable) and Rule-based (DecisionTable , JRip , OneR, and PART) classifiers provided in the Weka implementation by considering the average values of each classification method category.
The performance of classifiers is evaluated in terms of TP Rate, FP Rate, Precision, Recall, FMeasure, MCC, ROC Area, PRC Area, and Kappa statistic. The empirical results analysis shows that the performance of NaiveBayes and NaiveBayesUpdateable classifiers remain the same within the Bayesian classifiers' completion. In the group of Function classifiers, the performance of MultilayerPerceptron is promising to others. For Lazy classifiers, KStar is a winner in the context of most of the evaluation metrics. In the case of rule-based classifiers, PART is leading in terms of evaluation metrics. The Regression classifier outperformed the class of Meta classifiers. With the consideration of average evaluation metrics values of each category of classifiers, the Function category significantly better performed concerning other competitive classification categories.
In future work, we consider the larger number of cultural heritage data sets to evaluate the performance of the state of art classification algorithms.
I am thankful to Higher Education Commission (HEC) of Pakistan for supporting and funding for higher education in the form PhD Scholarship. [1] [2]
W. Ali, J. Kumar, and J. J. A. I. Shao, "Inferring context with reliable collaborators: a novel similarity estimation method for recommender systems," pp. 1-19, 2022.
Z. Deng, C. Li, S. Liu, W. Ali, and J. Shao, "Knowledge-aware group representation learning for group recommendation," in 2021 IEEE 37th International Conference on Data Engineering (ICDE), 2021, pp. 1571-1582: IEEE.
W. Ali, R. Kumar, Z. Deng, Y. Wang, and J. J. T. C. J. Shao, "A federated learning approach for privacy protection in context-aware recommender systems," vol. 64, no. 7, pp. 1016-1027, 2021. N. Friedman, D. Geiger, and M. Goldszmidt, "Bayesian network classifiers," Machine learning, vol. 29, no. 2, pp. 131-163, 1997.
H. Zhang, "The optimality of naive Bayes," Aa, vol. 1, no. 2, p. 3, 2004.
G. H. John and P. Langley, "Estimating continuous distributions in Bayesian classifiers," arXiv preprint arXiv:1302.4964, 2013.
S. Le Cessie and J. C. Van Houwelingen, "Ridge estimators in logistic regression," Journal of the Royal Statistical Society: Series C (Applied Statistics), vol. 41, no. 1, pp. 191-201, 1992.
Y. Liu, Z. You, and L. Cao, "A novel and quick SVM-based multi-class classifier," Pattern Recognition, vol. 39, no. 11, pp. 2258-2264, 2006.
D. R. Cox, "The regression analysis of binary sequences," Journal of the Royal Statistical Society: Series B (Methodological), vol. 20, no. 2, pp. 215-232, 1958.
J. Platt, "Sequential minimal optimization: A fast algorithm for training support vector machines," 1998. J. Tighe and S. Lazebnik, "Superparsing," International Journal of Computer Vision, vol. 101, no. 2, pp. 329-349, 2013.
D. W. Aha, D. Kibler, and M. K. Albert, "Instance-based learning algorithms," Machine learning, vol. 6, no. 1, pp. 37-66, 1991.
J. G. Cleary and L. E. Trigg, "K*: An instance-based learner using an entropic distance measure," in Machine Learning Proceedings 1995: Elsevier, 1995, pp. 108-114.
C. G. Atkeson, A. W. Moore, and S. Schaal, "Locally weighted learning," Lazy learning, pp. 11-73, 1997.
B. Qin, Y. Xia, S. Prabhakar, and Y. Tu, "A rule-based classification algorithm for uncertain data," in 2009 IEEE 25th international conference on data engineering, 2009, pp. 1633-1640: IEEE. R. Kohavi, "The power of decision tables," in European conference on machine learning, 1995, pp. 174189: Springer.
W. W. Cohen, "Fast effective rule induction," in Machine learning proceedings 1995: Elsevier, 1995, pp. 115-123.
R. C. Holte, "Very simple classification rules perform well on most commonly used datasets," Machine learning, vol. 11, no. 1, pp. 63-90, 1993.
E. Frank and I. H. Witten, "Generating accurate rule sets without global optimization," 1998. L. Breiman, "Bagging predictors," Machine learning, vol. 24, no. 2, pp. 123-140, 1996. J. Friedman, T. Hastie, and R. Tibshirani, "Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)," The annals of statistics, vol. 28, no. 2, pp. 337-407, 2000.