The term cultural heritage according to UNESCO encompasses two main
categories [
11
]: (i) tangible, and (ii) intangible cultural heritage. Tangible cultural
heritage can be categorised into: (i) movable cultural heritage (paintings,
sculptures, coins, manuscripts), (ii) immovable cultural heritage (monuments,
archaeological sites, historical buildings), and (iii) underwater cultural heritage
(shipwrecks, underwater ruins and cities). Intangible cultural heritage has been one
of UNESCO’s priorities in the cultural domain recently, as it promotes cultural
diversity. Through preservation of oral traditions and expressions, ways of life,
traditional crafts and festivals, amongst other activities, humans are
safeguarding their cultural identities.
The classification methods applied to the cultural heritage context can fit to
this categorisation. Figure 1 shows the cultural heritage categorisation.
2.1
Tangible Cultural Heritage
Movable Heritage. In the document analysis context, Esposito et al. [
8
]
proposed the use of an intelligent system for document processing based on machine
Cultural Heritage
Tangible Cultural
Heritage
Intangible Cultural
Heritage
Movable Heritage
Immovable Heritage
• Paintings
• Sculptures
• Furnitures
• Wall paintings
• Documents
• Historical buildings
• Monuments
• Archeological sites
• Oral traditions and
expressions
• Science & habits related
to nature and world
• Traditional skills
learning techniques and adapted it to the problem of automatically processing
documents in film archives for the COLLATE project. The WISDOM++
system previously developed by users [
8
] provides a document transformation into
a web-accessible form such as XML format. It is designed with high
adaptivity, real-time user interaction and multi-page document management as main
features. Throughout various steps of document processing, a rule base is built
from a training set using machine learning techniques, hence a highly adaptive
system. The authors use inductive learning techniques throughout the whole
course of the document processing, consisting of document analysis,
classification and understanding, and transformation into a web-accessible format. Each
of these steps uses machine learning as a viable solution to various problems
arising in image processing such as: classifying document components with
respect to the content (separation of text from graphics), finding logical structure
of the document based on layout structure extraction, text extraction from the
relevant part of the document using Optical Character Recognition (OCR), and
the transformation of the page into HTML/XML format. Preliminary results
produced by the research are with limitations relating to the WISDOM++
system upgrade for facilitating color images, as well as images of mixed content,
and further experiments are needed. Also, need for a better integration of OCR
with WISDOM++ is reported to improve the usability by the end user.
Also, INTHELEX (INcremental THEory Learner from EXamples) is a
learning system based on object identity assumption that learns theories from positive
and negative examples and can learn multiple concepts simultaneously. It is a
fully incremental learning system exploiting the induction of hierarchical logic
theories examples. Also, COLLATE is a project for annotation, indexing and
retrieval of digitalized historical archive material with the need to address the
automatic processing of multi-format cultural heritage documents from film archives.
Considering that the authors successfully applied INTHELEX in the paper
document processing domain [
2
], it was natural to incorporate it to COLLATE to
learn rules for the automatic identification of a wide range of COLLATE
document classes. The identified classes were further used for indexing and retrieval
as well as annotation by users. There are 119 considered COLLATE documents,
belonging to five di↵erent classes (four classes of positive and one class of negative
examples). An experiment was conducted as follows: the first-order descriptions
were generated by the WISDOM++ and are used to run the learning system.
Scanned images were used to identify layout blocks (type and relative position)
hence each document belonging to positive classes was described, on average,
with 144, 215, 269, and 260 literals respectively. The features that are contained
in the document descriptions are: height and width of the layout blocks,
horizontal and vertical position, type of the layout and relative position. Each document
is, at the same time, considered a positive example of the class it belongs to and
a negative example for other classes. Definitions for each class were learnt from
the initial theory that was empty. In order to compare their solution with the
state-of-the-art system, the authors conducted experiments using Progol and
observed accuracy and runtime of both systems confirming there are insignificant
di↵erences between the two in terms of accuracy. The runtime was significantly
shorter using the INTHELEX system contributing to the incremental approach
vs. batch approach in the case of Progol. Hence, the obtained results imply that
the proposed solution is also a feasible solution for the automatic processing of
multi-format cultural heritage documents.
In the same context, Brodi´c et al. [
6
] proposed a new document classification
method for discrimination based on the language. The first phase consisted in
extracting a feature representation from the document, where each letter was
codified into a numerical code according to its extension in the text-line area. A
total of four numerical codes were considered, corresponding to grayscale levels of
an image. In this way, the document was represented by a 1-D image from which
vectors of co-occurrence texture analysis features were extracted. In the second
phase, the feature vectors were subjected to a genetic-based classification
approach for discrimination of the corresponding documents in di↵erent languages.
The algorithm represented the documents as a weighted undirected graph, where
nodes were the documents and edges linked each node with its spatially close
and k nearest neighbour nodes in terms of L1 distance among the corresponding
vectors. The weight on each edge corresponded to a similarity value computed
from the distance between the involved nodes. Then, a genetic algorithm was
applied on the graph for detecting the connected components which corresponded
to groups of documents in the same language. The genetic algorithm started
with a population of individuals, each randomly initialised and representing a
possible division of the graph in connected components. Then, variation
operators of uniform crossover and mutation were applied on each individual and
the fitness function of weighted modularity was computed. This procedure was
iterated for a given generation number. At the end, the individual with the
best fitness function was selected as the final solution. Finally, a complete link
clustering strategy for correcting the local optima was employed on the genetic
solution. The experiment was performed on a database of 85 documents given in
French, English, Serbian and Slovenian contemporary languages. The obtained
results showed the discriminatory ability of the introduced document feature
representation as well as the good performances of the genetic-based classifier
versus other competing methods.
An extension of the genetic-based classifier was proposed by Brodi´c et al. [
5
]
for dealing with a more complex task of discrimination of languages evolved over
time. The first modification from the baseline algorithm was the introduction of
a parameter in the similarity computation for reducing the sparsity of the
document graph which may occur when representing close languages. The second
modification consisted in managing the presence of isolated nodes of the graph
inside the genetic algorithm, which can be caused by the procedure of graph
construction. The same document image coding was adopted for discrimination
of documents given in languages evolved one into another over time. The
difference was that run-length and local binary patterns instead of co-occurrence
textural features were extracted from the 1-D grayscale image of the document
and combined to create the feature vector. The experiment was carried out on
a database of 50 documents given in Italian Vulgar (1260-1374 AC) and
modern Italian languages. Comparison results with other well-known discrimination
algorithms revealed that the proposed approach overcomes the other methods
as well as the baseline genetic algorithm in discriminating the di↵erent evolved
languages.
In [
4
], the same feature coding was used with the Naive-Bayes method for
identification of the orthography style of documents based on the evolution of
script or language characteristics over time. In this case, the documents were
stored as images, and a preprocessing phase of bounding box detection and
filling was carried out for finding the position of the letters in the text-line area.
This allowed the coding of the document in a sequence of numerical codes from
which the grayscale 1-D image was produced. Then, run-length and adjacent
local binary patterns were used on the obtained image for the extraction of the
document feature vector. Finally, the Naive-Bayes classifier was employed on
the vector for identification of the orthography of the corresponding document.
The proposed approach was tested on two di↵erent contexts of: (i)
languagebased orthography identification from Serbian historical documents
(classification of Slavonic-Serbian and Serbian languages), and (ii) script-based
orthography identification from Croatian historical documents (classification of old and
new Glagolitic scripts). Results obtained on two databases of digitised
documents showed that the proposed approach is robust to data noise and overcomes
other competing methods in orthography style identification.
Also, Naive-Bayes and support vector machine were adopted in [
3
] on the
same feature coding for identification of documents as given in di↵erent Serbian
pronunciations from the Shtokavian dialect. After the document image coding,
di↵erent textural features were extracted from the 1-D image and combined to
create the feature vector for the document. They included: (i) local binary
patterns, (ii) neighbour binary patterns, and (iii) the newly introduced adjacent
neighbour binary patterns. The texture operators were considered with
di↵erent parameter values and tested. Then, both classifiers were used on the feature
vectors for identification of the Serbian pronunciation of the corresponding
documents. The experiment was performed on 50 documents written in ijekavian and
ekavian pronunciations of the Shtokavian dialect in Serbian language. The
obtained results showed that the proposed method overcomes the n-gram approach
in the classification task.
In the furniture context, a new technique was proposed in [
16
] for
automatic classification of archaeological pottery sherds image views contained in
two databases, using features based on colour and texture. The classification
task is challenging considering the pottery sherds belonging to di↵erent classes
have hardly no visible di↵erences. Ground truth images indicated by specialists
are used as representatives of each class. Local features based mainly on color
properties (sherds hue, chromaticity, saturation) and local texture variance of
the pottery sherds are extracted for each pixel from both front and back views
of the individual items. Histograms of local features are created in order to focus
on prevailing information and combined with a novel bag-of-words model based
on Reddi multithresholding. In this way, a global sherd descriptor for each sherd
image is created. In order to reduce the global descriptors’ dimensionality as well
as preserve classification accuracy, the authors resorted to the Principal
Component Analysis (PCA) feature selection statistical method as the most balanced
amongst eight considered feature selection methods. This process o↵ers a
tradeo↵ between the computational complexity (size of the feature matrix) and loss of
information on the other side resulting in potential misclassification. The authors
use several machine learning algorithms, namely K-Nearest Neighbour (KNN),
support vector machine, Naive-Bayes, SMO, and Simple Logic for classification
of the reduced global descriptor and have shown that the KNN algorithm is the
best selection for a given method. The classification rate when using ground truth
images as training example for di↵erent classes, or when splitting the dataset
into training and testing parts (40% training, 60% testing) provides comparable
results of about 70%. Additionally, the authors have tested their method on the
ceramic sherd database as well as compared their method in both databases
with four state-of-the-art feature extraction techniques (self-similarity, pyramid
histogram of words–color/gray–geometric blur and a weighted combination of
them) that have demonstrated an improved performance when tested in a generic
image database. In the case of both tests satisfactory results were obtained.
Finally, Mensink and van Gemert [
17
] introduced a new large dataset
composed of 112,039 artistic items from the collection of Rijksmuseum in
Amsterdam, the Netherlands. The items consisted of images and associated metadata
in XML format from portraits, furniture, pictures, miniatures, etc. of ancient
and medieval period, and late 19th century. The dataset is open and freely
available for tasks of art image classification and content-based image retrieval. Also,
di↵erently from previous datasets of specific artistic objects, such as paintings
or vases, the proposed dataset is broad and variegated enough for capturing a
museum-centered view. Starting from the dataset, four open challenges were
proposed, including: (i) prediction of the artist, (ii) prediction of the type of artistic
work, (iii) prediction of the material by which the artistic work is created, and
(iv) prediction of the year of creation. For each challenge, Fisher vectors were
adopted as image feature representation and support vector machine was used
for classification of the artistic images. All material retrieved from the
experiments is available for download, which includes dataset, experimental settings
and image features.
Immovable Heritage. To protect, keep and eventually restore tangible
cultural heritage buildings, systematic image collection and classification in order
to correctly interpret and manage the vast information is of utmost importance.
Guiding principles for recording, documentation and information management
for the conservation of heritage buildings were used in [
14
], such that, in their
works the authors created datasets comprising of ten di↵erent categories and
performed classification of the images. A motivation for creating datasets of the
architectural cultural heritage images was the lack of available datasets within
the research community. The dataset encompasses ten di↵erent categories of
Cultural Heritage images, namely Altar, Apse, Bello tower, Column, Dome
(inner), Dome (outer), Flying buttress, Gargoyle (and Chimera), Stained glass, and
Vault, totalling to over 10,000 images. Each image has a main element that is
reflected in the labelling process. One of the authors’ contributions is that the
labeled images can be used as a starting point to establish superior
classification techniques. They claim to have used deep learning techniques based on
Convolutional Neural Network (CNN) that previously have not been used for
classification of cultural heritage images and reported significant accuracy.
In [
19
], the authors’ main motivation was the classification of three di↵erent
architectural styles of buildings from images of the Mexican cultural heritage.
Raw video content was used for the production of cultural heritage images, hence
they contain undesirable yet not obvious elements besides the object in the focus
that could impair the classification process. Machine learning techniques can
address issues of identification of complex patterns in large sets of data where
the objects in the focus are positioned within a specific scene (amongst other
objects in the image), perhaps given in a di↵erent perspective, etc. The authors
use visual attention predictors (saliency driven content selection approaches) and
a traditional cropped image (centered-content) for training of a CNN in order
to classify di↵erent styles of architecture. Graph based visual saliency is the
model used for saliency prediction. The results show that using visual attention
predictors increases the quality of the cropped data, hence better classification
results are obtained based on saliency driven content selection in comparison to
a traditional cropped image selection. The obtained classification rates and the
authors’ future work course imply that the system for batch processing of raw
video footage could be devised with the contribution of creating a large number
of cultural heritage images.
Also, Grilli et al. [
10
] aim to improve on traditional preservation and
restoration methods of cultural heritage monuments. Such complex tasks require, at
least, documenting and archiving architectural/historical heritage content,
differentiating between various techniques of constructing buildings, and
recognition of previous restoration evidence, if any. Point clouds, amongst other
purposes, are used for 3D modelling of the cultural heritage monuments. Present
challenges in automatic model analysis are in the domain of segmentation and
classification of point clouds. Authors use 2D segmentation of the texture of 3D
models generated based on three archeological case studies conducted in Italy
(Villa Adriana in Tivoli; Cavea walls of the Circus Maximus in Rome; Portico
in Bologna). Downsides of using this technique come from the large amount of
data drawn from di↵erent types of constructing buildings as well as decorative
elements used on the facades. In addition, the cultural heritage sites are from
di↵erent time periods and in di↵erent stages of deterioration, hence the
classification is less ecient. Supervised machine learning (based on decision tree
algorithms) on UV maps, generated from the unwrapped textured 3D models,
is used for classification of the 3D cultural heritage models. Accuracy and
algorithm execution time were used as performance metrics. A training stage is
required for supervised learning and a sucient dataset of manually labeled
images has been used. The authors propose using deep learning to get a better
classification accuracy and hence propose to adequately extend the training set.
In the context of archeological sites, ICARE is an emerging digital heritage
platform presently at the early stages of development [
20
]. It is envisioned as
multimodal archiving system intended for archiving digital heritage content from
potential donors and performing semantic queries for searching data archives
from potential requestors. Both donors and requestors are interacting through
the system that could in the future, by means of multimedia, metadata and text
archives, provide believable digital cultural experiences in cultural heritage sites
that are forever lost to humankind. So far the authors have collected images from
cultural heritage sites in the Palmyra region and performed tests of several
machine learning models for visual categorization. The bag of features framework is
a novel technique for visual categorization by the support vector machine
classifier that used the largest number of categories in a simultaneous experiment.
Deep learning using CNN is also used by the authors for visual categorization.
An episodic memory technique was adopted to achieve better classification
results considering the limited image dataset. CNN using the transfer learning
approach provided the best classification accuracy.
2.2
Intangible Cultural Heritage
As oral traditions, Michon et al. [
18
] solved the problem of identification of
Arabic dialects using neural network architectures which allowed to achive
satisfactory results with also a reduced number of training data for learning. In
particular, the task required the discrimination of Modern Standard Arabic
language and four Arabian dialects: (i) Egyptian, (ii) Gulf, (iii) Levantine, and
(iv) North African, which are pretty similar. Three di↵erent runs were
generated for the challenge. The first one was a neural network model characterised
by a Multi-Input CNN, which separately learned lexical, acoustic and phonetic
characteristics of the languages. The second run was characterised by a neural
network CNN-biLSTM where the inputs were speech spectrograms, from which
spatial and sequential characteristics were extracted. The last run included a
binary CNN-biLSTMs. Results obtained by the di↵erent runs were compared with
results obtained by a support vector machine. It was visible that Multi-Input
CNN (first run) obtained the best performances versus the other methods and
architectures.
Also, Ma et al. [
15
] explored the properties of their structural pronunciation
representation for the extraction of linguistic features useful to identify the
dialect or subdialect of Chinese speakers. In the first stage, the experiment involved
a dialect-based speaker classification on data from 19 speakers with di↵erent
dialects and sub-dialects. In the second phase, a new dataset of recordings in the
di↵erent dialects and sub-dialects from an expert dialectologist was created with
minimum speaker di↵erences. Another classification task was performed on the
new dataset which obtained close results than the previous experiment. Finally,
a last dataset of recordings with maximum speaker di↵erences was built.
Different classification experiments were carried out on the original and simulated
datasets, based on both structural and spectral comparison. Results showed that
the structural comparison is able to capture the linguistic features and
classification performances do not depend on the speaker characteristics.
In the traditional skills and habits context, Liu et al. [
13
] proposed a
classification approach for identification of music pieces as belonging to cultural
styles. They were represented by four di↵erent types of features: (i) timbre, (ii)
wavelet coecients, (iii) rhythm, and (iv) characteristics based on musicology.
In particular, the timbral texture was adopted, which was composed of spectral
shape and contrast. Also, the rhytm was characterised by strength, regularity
and tempo. As musicology-based features, the chromogram, chord distribution,
contrast, and pitch interval histogram were adopted. Finally, the first three
moments (mean, variance and skewness) of the normalised histogram of wavelet
coecients in each level were extracted. The experiment was carried out on a
dataset of 1300 music pieces, each represented by the aforementioned feature
sets. Three classifiers were then employed on the dataset: (i) decision tree, (ii)
KNN and (iii) multi-class support vector machine. The obtained results showed
that support vector machine and KNN overcome the decision tree in
classification of the music pieces in six cultural styles: (i) Western classical music, (ii)
Chinese traditional music, (iii) Japanese traditional music, (iv) Indian classical
music, (v) Arabic folk music, and (vi) African folk music. In particular,
support vector machine obtained the highest accuracy of more than 86% when all
features were used.
Also, Dimitropoulos et al. [
7
] presented the i-Treasures Project for
management, extraction and analysis of intangible cultural heritage, which included
traditional songs, dance, pottery and contemporary music pieces. In particular,
the project aimed to build a platform for open access to intangible cultural
heritage, transmission and exchange of knowledge. This was not only focused on
digitization of cultural resources, but also on generation of new knowledge by
exploiting new multisensory-based techniques for exploring the digital content.
The first step in the project was the semantic analysis of the cultural content in
order to create a unified knowledge platform. It was accomplished by extracting
semantics for capturing hidden patterns and relationships among elements of
intangible cultural heritage, which are useful for tracking its evolution over the
di↵erent generations. Image and signal processing methods were employed on
the di↵erent signals for the feature extraction. Finally, a learning environment
was developed using 3D technology of web-based game engines.
Finally, Li et al. [
12
] proposed a method for classification of traditional
Chinese folk songs according to the di↵erent regional styles. A temporal model was
adopted for capturing the temporal structures of the folk songs. In particular,
Conditional Random Fields (CRF) were used for creating the model, whose state
and transfer functions were computed by the Gaussian Mixture Model (GMM)
approach. The experiment involved a dataset of 334 folk songs from the Chinese
regions of Shaanxi, Jiangsu and Hunan. For each song category, the CRF model
was learned from the training set. Then, the GMM was employed for fitting the
audio frame features and estimating the corresponding label sequence for each
CRF. In order to predict the category of a folk song in the test set, for each CRF
corresponding to a specific category (regional style), the posterior probability of
the folk song to belong to the given category was computed. The regional style
class of the folk song was that corresponding to the highest computed posterior
probability. The obtained results showed that the proposed classification method
achieved an improvement of 4.6%-18.13% versus the other competing methods,
i.e. support vector machine, KNN, etc.
3