work in the literature of art painting categorisation in explainable AI. In this way, the three main features of the C-LAD
algorithm represent a significant advance in the field: (i) the classifier uses logic aggregators and integrates the subjective
aggregation theory developed by Dujmovic´ [
This section contains the preliminaries of the paper. First, the two datasets used in this paper, the QArt-Dataset
and Painting-91-BIP datasets, are presented. Then, we briefly sketch the computational model for Qualitative Colour
Description (QCD model) introduced by Falomir et al. [
The QArt-Dataset contains 90 images (30 paintings for each style) and the Painting-91-BIP dataset includes 247 images (74 for the Baroque style, 82 for the Impressionism style and 91 for the Post-Impressionism style). For each art style, both datasets consider two representative authors: Velázquez and Vermeer for the Baroque style, Monet and Renoir for the Impressionism style, and Gauguin and van Gogh for the Post-Impressionism style (see Fig. 1 for some examples). B I PI
Dujmovic´ introduced logic aggregators, i.e. functions that aggregate two or more degrees of truth and return a degree
of truth in a way similar to observable patterns of human reasoning [
The QCD model [
An aggregator A : [0; 1]n ! [0; 1] [7, Definition 1] is defined as a nondecreasing function in each argument (where n > 1 and [0; 1] is the unit interval of the real numbers) and such that satisfies boundary conditions of idempotency in extreme points 0 and 1. An aggregator A : [0; 1]n ! [0; 1] is a logic aggregator [7, Definition 2] if A is a continuous function that satisfies the following sensitivity conditions: for every x1; : : : ; xn 2 [0; 1] and 1 i n, A(x1; : : : ; xn) > 0 if xi > 0; and A(x1; : : : ; xn) < 1 if xi < 1. From now on, let A : [0; 1]n ! [0; 1] be a logic aggregator.
In the process of classifying a picture into a painting style, humans may have input percepts of the degrees of adequacy
of the picture with respect to the different distinctive traits of the painting styles. The distinctive colour features can
be regarded in a natural way as fuzzy notions [
Pattern 1. Idempotent Aggregation. This aggregation pattern is based on the assumption that the membership degree to an art style must be between the lowest and the highest value of the colour traits of this style. Formally, it is said that A is idempotent [7, Definition 3] if for every x1; : : : ; xn 2 [0; 1], min(x1; : : : ; xn) A(x1; : : : ; xn) max(x1; : : : ; xn). Pattern 2. Noncommutativity. Observe that for each painting style, each colour trait may have its degree of importance. Hence it is essential to model asymmetry, which can be represented using different weights of the arguments. Formally, it is said that A is asymmetric [7, Definition 5] if for every x1; : : : ; xn 2 [0; 1], i 6= j, xi 6= xj , we have that A(x1; : : : ; xi; : : : ; xj ; : : : ; xn) 6= A(x1; : : : ; xj ; : : : ; xi; : : : ; xn). Asymmetry is usually realised using different weights of arguments. Weights are assumed to be positive and normalized: 0 < wi < 1, Pi wi = 1.
Pattern 3. The use of Annihilators. An annihilator is an extreme value of suitability (either 0 or 1) of a feature which is
sufficient to decide the result of aggregation regardless of the values of other inputs. In the case of necessary conditions
the annihilator is 0: if some mandatory requirement for a painting style is not satisfied, the painting is not classified in
this style. In a dual case of sufficient conditions the annihilator is 1. The aggregators that support annihilators 0 and 1 are
called hard, and aggregators that do not support annihilators are called soft. Although many examples of the use of hard
aggregators in human aggregation can be observed [
In this section we propose distinctive colour traits for the painting styles under study. Note that in this work we expand
the set of colour traits defined in [
With the aim of defining the different colour traits for the three styles, the QCD model was extended in [4, Def.1] as follows: dark_colours = fblack, dark_(red,orange,yellow,green,turquoise,blue,purple,pink,grey)g, pale_colours = fpale_(red,orange,yellow,green,turquoise,blue,purple,pink,grey)g, light_colours = fwhite,light_(red,orange,yellow,green,turquoise,blue,purple,pink)g, grey_hue = fgrey; pale_grey; light_grey; dark_greyg (analogously for red_hue, orange_hue, yellow_hue, green_hue; turquoise_hue, blue_hue, purple_hue, pink_hue) warm_hue = fred_hue,orange_hue,yellow_hueg, and vivid_colours = fred, orange, yellow, green, turquoise, blue, purple, pinkg.
In addition to this extension of the QCD model, let us add the following: red_colours={red, orange, pale_red, light_red, dark_red, pale_orange, dark_orange, light_orange}. Now we proceed with the categorisations of the styles.
Mainly, the Baroque painting shows a high level of darkness provided by the use of dark colours (black colour and dark
tones of other colours) [
W4 red_colours(p).
Regarding colour features in the Impressionism style, the literature [
W10 diversityofHues(p)
W20 diversityofQCDs(p)
W30 bluish_level(p)
W40 greyish_level(p).
The Post-Impressionist style breaks the tendency of representing colours as appearing in reality [
W200 warm_colours_level(p)
W300 contrast_blue_yellow_level(p)
W400 contrast_red_green_level(p).
In order to calibrate the distinctive colour traits of each art style, we have considered the mean of each of these features in all the images of the QArt-Dataset: 1C. Tate. Post-Impressionism. http://www.tate.org.uk/art/art-terms/p/post-impressionism, accessed 14 April 2018.
Notice that, although all the values of the colour traits are normalised, their means are quite dispar in such a way that a priori some features are more significant than others. For instance, a painting might show 0:75 of no_paleness_level and 0:25 of bluish_level. Hence the model should give more importance to the bluish_level feature of the painting than to the no_paleness_level trait, since in this case the bluish_level feature is specially relevant to determine the art style. Thus we need to calibrate the values of the colour traits. In order to do it, we consider the mean of all the colour traits, which is 0:4098. Then, the calibrated colour feature, indicated by , is obtained as follows: darkness_level(p) = maxfdarkness_level(p) (0:5004 0:4098); 0g; : : : ; contrast_red_green_level(p) = vividness_level(p) + (0:4098 0:0558), etc. That is, the difference between the general mean of the all the colour features and the mean of the colour trait considered is added or substracted (depending on the mean of the colour trait considered) to the value of the colour trait for p.
Finally, in order to obtain the weights corresponding to the colour traits distinctive of the Baroque style (W1; : : : ; W4), we have considered the mean of each of these features. We explain the procedure for the colour trait darkness_level. The mean for darkness_level is denoted by xdly, where dl denotes darkness_level and y is substituted by B; I; P I, depending on the art style selected. For instance, xdlB denotes the mean of the darkness_level of the 30 Baroque paintings in the QArt-Dataset. The other colour traits of the Baroque are denoted by npl; cl and rc. The results obtained are shown next. art painting style
Baroque (B)
Impressionism (I) Post-Impressionism (PI) xdly 0:67 0:33 0:24 xnply 0:50 0:36 0:37 xcly 0:59 0:38 0:28 xrcy 0:49 0:38 0:36
From these data, we obtain the differences between the mean of the colour trait for the Baroque style and the mean of this feature for the other styles: difdl = (xdlB xdlI )+2(xdlB xdlP I ) = 0:38, difnpl = 0:15, difcl = 0:26, and difrc = 0:12. The highest difference must correspond to the highest weight, because a higher difference indicates that the colour trait is more distinctive for the art style selected. In this case, it corresponds to the darkness_level (i.e. dl). The weights are thus obtained by solving the system of linear equations fW1 + W2 + W3 + W4 = 1; W1 = 2:61W2; W1 = 1:48W3; W1 = 3:10W4g, whose solutions are W1 = 0:42; W2 = 0:16; W3 = 0:28, and W4 = 0:14. With respect to the other styles, we proceed analogously and obtain that W10; W20; W30; W40; W100; W200; W300; W400. Therefore we have parametrised the three formulas that categorise the Baroque, the Impressionism and the Post-Impressionism styles. 4
In this section we define the art painting style Classifier based on Logic Aggregators and qualitative colour Descriptors (C-LAD) and show some examples of responses produced by the C-LAD categorisation for paintings of the QArt-Dataset.
We define first the evaluation degree for p to belong to an art style as: ebAS(p) = 8(Bst; B(p)) > > ><(Ist; I(p)) if maxfB(p); I(p); P I(p)g = B(p) and P I(p) 6= B(p) 6= I(p); if maxfB(p); I(p); P I(p)g = I(p) andB(p) 6= I(p) 6= P I(p); >(P Ist; P I(p)) if maxfB(p); I(p); P I(p)g = P I(p) andB(p) 6= P I(p) 6= I(p); > :>(U nkst; I(p)) otherwise:
Since the C-LAD algorithm has to give a second option in difficult cases, a function between evaluation degrees, Sim, is defined: SimB;I (p) = jB(p) I(p)j, SimB;P I (p) = jB(p) P I(p)j, and SimI;P I (p) = jI(p) P I(p)j, where SimB;I (p) stands for the closeness between the Baroque and the Impressionism evaluation degrees of p, and SimB;P I (p); SimI;P I (p) are described analogously. The C-LAD algorithm categorises paintings as follows: 1. If edAS = (Bst; B(p)), then “p is a Baroque painting.” & explanationC-LAD(B; p).
– If SimB(p);I(p) 0:15, then “Although p is categorised in the Baroque style, there are reasons to believe that it may belong to the Impressionism.” & explanationC-LAD(I; p). – If SimB(p);P I(p) 0:15, then “Although p is categorised in the Baroque style, there are reasons to believe that it may belong to the Post-Impressionist.” & explanationC-LAD(P I; p). 2. If edAS = (Ist; I(p)), then “p is an Impressionist painting.” & explanationC-LAD(I; p).
– If SimB(p);I(p) 0:15, then “Although p is categorised in the Impressionist style, there are reasons to believe that it may belong to the Baroque.” & explanationC-LAD(B; p). – If SimI(p);P I(p) 0:15, then “Although p is categorised in the Impressionist style, there are reasons to believe that it may belong to the Post-Impressionism.” & explanationC-LAD(P I ; p). – If SimB(p);P I(p) 0:15, then “Although p is categorised in the Post-Impressionist style, there are reasons to believe that it may belong to the Baroque.” & explanationC-LAD(B; p). – If SimI(p);P I(p) 0:15, then “Although p is categorised in the Post-Impressionist style, there are reasons to believe that it may belong to the Impressionism.” & explanationC-LAD(I ; p). 4. If edAS = (U nkst; I (p)), then “p could not be classified.”
In addition, explanations for specific characteristics in each art style can also be provided. For instance, let us consider the colour trait darkness_level as significant for classifying p into the Baroque style, whenever darkness_level(p) is higher than the mean of the darkness_level of the 30 Baroque paintings in the QArt-Dataset, say darkness_level (and analogously for the rest of the distinctive colour traits). Hence this mean plays the role of a threshold. That is, if this is the case, the presence of this feature must appear as an explanation/evidence for p classified into this style. For the Baroque style, explanationC-LAD(B; p) provided by C-LAD are the following (the rest of the explanations are analogous): If darkness_level(p) darkness_level, then “The darkness evidences the Baroque style.” If no_paleness_level(p) no_paleness_level, then “The contrast of dark and pale colours evidences this style.” If contrast_level(p) contrast_level, then “The lack of pale colours evidences this style.”
If red_colours(p) red_colours, then “The high proportion of peroxide-based yellows, oranges and reds evidences this style.”
The C-LAD algorithm has been implemented in the program language R2, but let us recall that the values of the colour traits were obtained using Swi-Prolog, as explained in [4, Section 6]. A random image in the QArt-Dataset of each style has been selected as a running example (see Figure 3), and the categorisation and the explanations provided by C-LAD for these paintings are: v4 is a Baroque painting. The darkness evidences the Baroque style.
The contrast of dark and pale colours evidences this style. rn5 is an Impressionist painting. The variety of hues evidences the Impressionism style.The diversity of qualitative colours evidences the Impressionism style. The amount of bluish evidences this style. The amount of grey colour evidences this style.
rn5 is an Impressionist painting. The variety of hues evidences the Impressionism style.The diversity of qualitative colours evidences the Impressionism style. The amount of bluish evidences this style. The amount of grey colour evidences this style.
V4 rn5 vg10 v3, outlier for C-LAD rn3, outlier for C-LAD vg3, outlier for C-LAD This section shows and discusses the results obtained when classifying the 90 images in the QArt-Dataset using the C-LAD algorithm. The results for the 247 images in the Painting-91-BIP dataset are only highlighted at the end of the section.
Table 2 shows the confusion matrix obtained for the three art styles in the QArt-Dataset. The blue cells correspond to the correct classifications: on the left, correct classifications where C-LAD is sure (X); on the right, correct classifications where C-LAD is not sure and yields an alternative style as a second opinion (?). The rest of the cells correspond to the outliers: in each column, the cell on the left indicates the outliers in that C-LAD does not give a second opinion (X); and the cell on the right shows the outliers in that the second opinion given classifies correctly the painting (?). Let us indicate that, when the algorithm is doubting, it provides two possible styles as a result. The first option (highest certainty) is the one considered as a correct classification. If the second opinion (lowest certainty) is the correct one, it is not counted as a correct classification and it appears in a column corresponding to a different style.
The highest accuracy is obtained for the Baroque style (93:7% – 28/30), and 82.1% of correct classifications (23/28) C-LAD certainly categorises the painting, without yielding a second opinion. An outlier of the Baroque style is the painting The Surrender of Breda (v3, Figure 3), from Velázquez, which is categorised as an Impressionist painting, but in fact C-LAD warns that there is evidence to belong to the Baroque style. The Impressionism style obtains an accuracy of 70:0% (21/30). In 44.4% of the misclassifications obtained, C-LAD warns that there is evidence to belong to the Impressionist style. An example of an outlier in this style is Composition, Five Bathers (rn3, Figure 3) by Renoir, for which B(rn3) = 0:42, I(rn3) = 0:31, and P I(rn3) = 0:58. The Post-Impressionism style gets 76.7% of accuracy rate (23/30). Observe that 71.4% of the outliers are categorised in the Impressionism style. In addition, in 57.1% of the misclassifications C-LAD warns that there is evidence to believe that a painting belongs to the Post-Impressionist style. An example of an outlier in the Post-Impressionist style is Paysage de Bretagne. Le moulin David (gg11, Figure 3) by Gauguin, for which B(gg11) = 0:33, I(gg11) = 0:55, and P I(gg11) = 0:44 (in this case C-LAD warns that there is evidence gg11 to belong to the Post-Impressionism style). The general accuracy rate got by C-LAD in the QArt-Dataset is 80.0%. Table 3 shows the confusion matrix obtained for the three art styles in the 247 images of the Painting-91-BIP dataset. Note that the total accuracy rate got by C-LAD in the Painting-91-BIP dataset is 64.0%.
The C-LAD algorithm has been presented and tested on the QArt-Dataset and the Painting-91-BIP datasets. The results
obtained are similar to other research works as [
Regarding future work, we intend to improve the function ebAS in order to solve any event of a tied evaluation degree that may appear in future testings of the algorithm. Other future work includes a detailed comparison of the different approaches for art painting style classification. We also plan to design a classifier that mixes qualitative colour descriptors, logic aggregators and art genres data. Finally, future research should aim to add new art styles to the dataset in order to define categorisations of a larger number of styles.
Acknowledgements
The author would like to thank the reviewer for the constructive feedback. The research leading to the results presented
has received funding from the grant FI-2017 (Generalitat de Catalunya and the European Social Fund) and from the DAAD
Short-Term Grant (2019, 57442045). The author of this paper would like to thank Prof. Pilar Dellunde and Dr.-Ing. Zoe
Falomir for their helpful comments, and also to thank Prof. Dujmovic´ for his constructive comments on the topic of this
article, which will serve to improve the extended future versions of this paper. Finally, the author also acknowledges the
datasets provided by [