The preservation and classification of cultural heritage images play a crucial role in understanding and appreciating the rich history and diverse cultures of our world. However, accurately categorizing and identifying these images can be challenging due to their varying characteristics, such as different artistic styles, degradation, and aging effects. In this paper, we propose a novel approach for classifying augmented cultural heritage images using Convolutional Neural Networks (CNNs). Our approach leverages the power of CNNs, a class of deep learning models specifically designed for image analysis, to automatically learn and extract discriminative features from augmented cultural heritage images. To address the challenges posed by the variability and uniqueness of these images, we employ data augmentation techniques during the training phase. This includes random transformations such as rotation, scaling, and flipping, which help to increase the diversity and robustness of the training data. We evaluate our proposed method on a large dataset of augmented cultural heritage images. Experimental results demonstrate the effectiveness of our approach, achieving state-of-the-art classification accuracy. The CNN-based model effectively learns intricate patterns and features from the augmented images, enabling accurate identification and categorization of different cultural artifacts, artworks, and historical scenes. Furthermore, we conduct a comparative analysis with traditional models and demonstrate the superiority of CNNs in terms of various performance metrics such as precision, recall, F1 score and ROC.
The preservation and understanding of cultural heritage are crucial for maintaining and
celebrating our diverse human history. Cultural heritage images, which include historical
artifacts, archaeological sites, and artwork, play a significant role in documenting and
communicating our cultural legacy. With the advent of digital technologies, an unprecedented
volume of cultural heritage images has become available, posing new challenges and
opportunities for their analysis and classification [
Traditional methods for classifying cultural heritage images often rely on handcrafted
features and shallow learning algorithms. However, these approaches are limited in their ability
to capture complex visual patterns and variations present in such images. In recent years, deep
learning techniques, especially Convolutional Neural Networks (CNNs), have demonstrated
remarkable success in various computer vision tasks, including image classification, object
detection, and image synthesis [
This paper focuses on harnessing the power of CNNs for the classification of augmented
cultural heritage images. Image augmentation techniques have proven to be effective in
increasing the diversity and size of training datasets, thereby enhancing the generalization
capabilities of deep learning models. By integrating augmented data into the training process,
we aim to improve the accuracy, robustness, and adaptability of CNN-based classifiers for
cultural heritage images [
Augmented cultural heritage images refer to digitally enhanced or modified versions of
original images. Augmentation techniques can include geometric transformations (such as
rotation, scaling, and translation), photometric variations (such as brightness, contrast, and
color adjustments), and noise injection [
In this paper, we propose a novel approach that combines a state-of-the-art CNN architecture with augmented cultural heritage images to achieve improved classification performance. We investigate the impact of different augmentation techniques on the network's ability to learn discriminative features and adapt to challenging variations in cultural heritage images. Furthermore, we explore the effectiveness of transfer learning and fine-tuning strategies to leverage pre-trained CNN models trained on large-scale datasets for enhanced classification accuracy.
The main contributions of this work can be summarized as follows: • We devise a novel CNN architecture tailored to the specific demands of classifying augmented cultural heritage images. The architecture integrates layers optimized for handling the unique features and challenges present in these images. • We propose a data augmentation technique based on stain normalization, image patches generation and affine transformation. • Investigation of transfer learning and fine-tuning strategies for leveraging pre-trained
CNN models and improving classification accuracy on limited cultural heritage datasets. • Experimental validation and comparative analysis of the proposed approach using a diverse collection of cultural heritage images, highlighting its advantages over traditional classification methods.
The rest of the paper is organized as: Section 2 provides an overview of related works in the field of cultural heritage image analysis and classification. Section 3 addresses the problem definition. Section 4 presents the methodology and architecture employed for CNN-based classification of augmented cultural heritage images. Section 5 presents the experimental setup and evaluation results. Section 6 discusses the conclusion and future directions.
In recent years, several studies have explored the application of convolutional neural networks (CNNs) for the classification of augmented cultural heritage images. These studies have focused on various domains, including archaeological artifacts, historical paintings, and ancient manuscripts. This section presents a review of the related work in this area, discussing the methodologies, datasets, and evaluation metrics used in existing research.
One prominent study by Lu et al. [
The literature contains a plethora of studies showcasing diverse applications of deep
learning in image classification. These applications encompass a wide range, including both
generic domains [
Likewise, in the realm of historical paintings, Karthic et al. [
In conclusion, previous research has demonstrated the effectiveness of CNNs for classifying cultural heritage images in domains such as archaeological artifacts, historical paintings, and ancient manuscripts. However, challenges related to dataset availability, variations, size, and evaluation protocols need to be addressed to further advance the field. Also most of the models proposed in the literature classifying the images are lacking in prediction accuracy. Therefore, we integrated the concept of image augmentation in our model to overcome these challenges and enhance the classification accuracy of cultural heritage images. Future research should focus on transfer learning and synthetic data generation techniques to enhance classification performance and facilitate wider adoption in cultural heritage preservation efforts.
Given a dataset of cultural heritage images, denoted as = {%, ', ( , … . +} where represents the number of images, and a set of augmentation techniques, denoted as = {%, , , … . 0 } where is the number of augmentation techniques, the goal is to build a CNN model denoted as (. )that can accurately classify these augmented images.
The augmentation process can be mathematically represented as follows: 678 = 8(7) for = 1,2,3, … , and = 1,2,3, … (1)
Where 678 represents the augmented image obtained by applying augmentation technique to the original image 7. The dataset of augmented images can be represented as: {D = 6%%, 6%', 6%( , … , 6+0 } (2)
Given = {%, ', ( , … , +} (Original input images), = {%, ', ( , … , +} (Augmentation techniques) and (678 ; , ) (Classification prediction for each augmented image).
Subject to optimize the CNN model’s parameters, and , to minimize the classification error.
Where (678 ; , ) represents the output of the CNN model for augmented image 678 with the learned parameters, and to optimize the classification of images.
The problem aims to develop a CNN model capable of accurately classifying augmented cultural heritage images by effectively utilizing the augmented features introduced by the set of augmentation techniques. The limited labeled data challenge is addressed by optimizing the model's performance and generalization capabilities. The goal is to enhance the preservation and understanding of cultural heritage artifacts through advanced image analysis techniques enabled by CNNs.
By mathematically addressing the augmentation process and formulating the problem, we aim to provide an accurate and efficient solution for the classification of augmented cultural heritage images, contributing to the preservation efforts and furthering the understanding of cultural heritage artifacts.
The Convolutional Neural Network [23] is well-known discriminative learning paradigm that does not require human feature extraction and directly learns from the input. Like regularized MLP networks, CNN improves working of conventional ANN. Every layer in CNNs considers most appropriate parameters for a reliable output and also reduces the complexity of the model. Key application areas of CNNs include computer vision, medical image processing, recommender systems, natural language processing, blockchain etc. Also CNNs have specific intention towards dealing with variety of 2D shapes [24, 25]. CNNs have the potential to automatically discover key features from the input while ignoring human intervention which gives them more power.
The input layer represents the image data that is fed into the neural network. Each image is typically represented as a multi-dimensional array, where each element corresponds to a pixel value. Convolutional layers are responsible for extracting visual features from the input images. Each convolutional layer consists of a set of filters (also known as kernels) that perform convolution operations on the input. The output of a convolutional layer is a feature map that captures different visual patterns and structures in the image as shown in equation 1. ℎI = J I ∗ ℎIM%N + I (3)
After each convolutional layer, an activation function is applied element-wise to introduce non-linearity into the model. Popular choices include the Rectified Linear Unit (ReLU) function, which sets negative values to zero and keeps positive values unchanged.
() = max (0, )
I = (hX)
Pooling layers are used to down sample the feature maps and reduce the spatial dimensions. Common pooling operations include max pooling, which takes the maximum value within a specified window, and average pooling, which computes the average value within the window.
Where , I represents the pooled feature map obtained by down sampling the feature map ℎI using a pooling operation such as max pooling or average pooling.
After the convolutional and pooling layers, the output is flattened and passed through one or more fully connected layers. These layers are fully connected, meaning each neuron is connected to every neuron in the previous layer. Fully connected layers learn higher-level representations and capture complex relationships in the data.
The output of the fully connected layer (with weight matrix denoted as YZ and bias vector denoted as YZ ) can be represented mathematically as follows:
We used Architectural Heritage Elements Dataset (AHE) for the training and testing of our model. AHE is an image dataset for developing deep learning algorithms and specific techniques in the classification of architectural heritage images. This dataset consists of 10235 images classified in 10 categories: Altar, Apse, Bell tower, Column, Dome (inner), Dome (outer), Flying buttress, Gargoyle, Stained glass, Vault. We have used 80% of the data for training and 20% for validation purpose. 5.2. Assessment Metrics = YZ . ℎ\M% + YZ (6)
Where, represents the input to the fully connected layer, ℎ\M% is the feature vector from previous layer, (. ) denotes matrix multiplication, and YZ is the bias vector.
The output layer is typically a fully connected layer with the number of neurons equal to the number of classes in the classification problem. The activation function used in the output layer depends on the task; for multi-class image classification, a softmax function is commonly used to produce class probabilities. Following equation define soft max function.
]^_ ( 7) = ∑a ]^a = ()
The output layer (typically a fully connected layer) with softmax activation produces the predicted class probabilities, denoted as .
Where, represents the activation values of the output layer before applying the function. Training of the model involves optimizing the model's parameters (weights and biases) to minimize a loss function, such as cross-entropy loss, using gradient-based optimization algorithms like stochastic gradient descent (SGD) or its variants. (5)
(4) (7) (8)
To evaluate the effectiveness and reliability of our classification model we use precision, recall F1-measure and ROC. 5.2.1 F1-measure
The F1 measure is a metric used to assess the performance of a classification model, particularly when the data is imbalanced. It combines both precision and recall into a single score. It is the harmonic mean of precision and recall, calculated as follows: F1 = 2 * (precision * recall) / (precision + recall) (9)
The F1 measure ranges from 0 to 1, where 1 indicates perfect precision and recall, and 0 indicates poor performance. It is especially useful when the goal is to strike a balance between precision and recall, rather than favoring one over the other.
Precision is a metric that measures the accuracy of positive predictions made by a model. It quantifies the proportion of true positives (correctly predicted positive samples) out of all positive predictions, including both true positives and false positives. Precision is computed using the following formula:
Precision = true positives / (true positives + false positives) (10)
Precision ranges from 0 to 1, where 1 signifies perfect precision (no false positives), while 0 indicates poor precision. 5.2.3 Recall
Recall, also known as sensitivity or true positive rate, measures the ability of a model to correctly identify positive samples. It calculates the proportion of true positives out of all actual positive samples, including both true positives and false negatives. Recall is computed using the following formula:
Recall = true positives / (true positives + false negatives) (11)
Similar to precision, recall also ranges from 0 to 1, where 1 represents perfect recall (no false negatives), and 0 denotes poor recall.
ROC is a graphical representation of a classification model's performance across various discrimination thresholds. It plots the true positive rate (recall/sensitivity) on the y-axis against the false positive rate (1 - specificity) on the x-axis, as the discrimination threshold is varied. ROC curves provide insights into the trade-off between true positive and false positive rates, allowing the selection of an appropriate threshold based on the desired balance.
The AODE [
The Forest PA [
The RSeslibKnn [28] is a k-nearest neighbor classifier designed for efficient handling of large datasets by employing optimized k selection and fast neighbor search techniques. Classification is performed by identifying the k nearest neighbors in the dataset and determining the decision through voting. The algorithm supports three voting methods: equally weighted, inverse distance weights, and inverse square distance weights.
In this section, we empirically evaluate whether our proposed model can achieve better results compared with other state-of-the-art models presented in section 5.3 by discussing the accuracy, precision, recall, F1-measure and ROC of our model in comparison with the state-ofthe-art models as shown in table 5.1. The Random Forest model achieved an accuracy of 0.786, indicating that around 78.6% of predictions were correct. Precision and recall are both 0.787, suggesting that the model is equally effective in correctly identifying positive instances and avoiding false positives. The F-measure, at 0.797, suggests a harmonic balance between precision and recall. The ROC score of 0.959 indicates that the model has a strong ability to discriminate between positive and negative instances. AODE model achieved an accuracy of 0.808, suggesting a relatively higher percentage of correct predictions. The precision of 0.760 implies that of all the instances predicted as positive, only 76% are truly positive. The recall of 0.808 indicates the model's effectiveness in identifying actual positive instances. The F-measure is 0.808, which is the harmonic mean of precision and recall. The high ROC score of 0.965 signifies strong discrimination capabilities. Similarly, RSeslib Knn model achieved an accuracy of 0.806, indicating that it correctly predicted around 80.6% of instances. The precision of 0.811 indicates that the model's positive predictions were correct 81.1% of the time. The recall of 0.807 suggests its ability to find 80.7% of actual positive instances. The F-measure, at 0.805, suggests a balance between precision and recall. The ROC score of 0.879 indicates a moderate level of discrimination.
In conclusion, the proposed model outperformed the others with an accuracy of 0.937, indicating a high level of correct predictions. Precision of 0.946 suggests that a large majority of instances predicted as positive were indeed positive. Recall of 0.944 indicates the model's ability to capture a high proportion of actual positive instances. The F-measure, also at 0.944, confirms the balance between precision and recall. The ROC score of 0.970 suggests excellent discrimination. Figure 2 describes the graphical representation of all the performance measures of proposed model in comparison with the state-of-the-art models.
This research work delved into the impactful fusion of Convolutional Neural Networks (CNNs) and augmented reality techniques for the enhanced classification of cultural heritage images. Through a comprehensive exploration of both CNN-based image analysis and the integration of augmented reality overlays, we have demonstrated the potential of this approach to revolutionize the field of cultural heritage preservation and documentation. By harnessing the capabilities of CNNs, we have showcased the effectiveness of deep learning in automatically extracting intricate features from cultural heritage images, enabling accurate and efficient classification. The results obtained substantiate the notion that CNNs can significantly augment 1 0,8 0,6 0,4 0,2 0 Forest PA AODE RSeslib Knn Proposed Model the traditional methods of image classification, offering a robust framework for managing and understanding the diverse visual aspects of cultural heritage. Furthermore, the incorporation of augmented reality overlays into the classification process has opened up new dimensions in the way we interact with and interpret cultural artifacts. The visual enhancements provided by augmented reality not only offer an engaging and immersive experience for viewers but also aid researchers, historians, and enthusiasts in gaining deeper insights into the historical and cultural significance of these artifacts.
Future research directions should address these challenges and advance the CNNempowered classification of augmented cultural heritage images. One potential direction is the development of transfer learning techniques that can leverage pre-trained models on largescale general image datasets to improve classification performance with limited annotated cultural heritage images. Additionally, the generation of synthetic training data could help augment the existing datasets and alleviate the data scarcity issue.
We acknowledge to Higher Education Commission (HEC) of Pakistan to support this research work in the form of an award of a Ph.D. scholarship. Furthermore, we thank Islamia College University Peshawar and the National University of Technology (NUTECH) Islamabad for technical support. 132: p. 679-688. [25] Jabbar, M. and R. Aluvalu, RFAODE: A novel ensemble intrusion detection system. Procedia based on attribute selection. Information, 2019. 11(1): p. 12.