The ability to evaluate both explicit facial expressions and intermediate expressions is helpful for human monitoring. Since intermediate facial expression is out of the scope of traditional studies, evaluation scores obtained from traditional facial expression recognition techniques are unreliable. In this paper, we propose an ordinal scale-based evaluation scheme for facial expression based on a comparison. The proposed framework is based on an ordinal scale; it is challenging to construct a standard scale that can be applied to multiple individuals. However, it is expected to be efective enough to track changes in the facial expressions of the specific individual, including intermediate expressions. We also propose an algorithm for selecting reference images from the data by taking into account the consistencies of the strong-weak relationships between reference images because the reference image selection significantly impacts the ordinal evaluation. Our approach is evaluated by conducting experiments with human annotators.
Transitions of facial expressions Monitoring an individual’s Quality of Life (QOL) is becoming increasingly important to maintain good mental ity cBoencdauitsioendsiraencdt QdeOteLcitneqaurilryestraernedbsoi nthhearirnmgfualncdoint disitdioifi-ns. iiltsnneem cult to accurately represent one’s internal state, estimat- S ing internal state from external nonverbal information time is desired. Facial expression is one of the modalities that reflects an individual’s internal state and is expressed Time with being influenced by mental condition. For example, when an individual is not feeling well, the same smile Figure 1: An example of transition curve of smiling intensity may appear weaker than usual. Therefore, monitoring fa- in daily life cial expressions in daily life is a crucial clue to estimating an individual’s QOL.
The research field of facial expression recognition Though the traditional algorithm of FER seems able
(FER) has a long history, and it has already been put to evaluate intermediate facial expressions as a
probinto practical use as a technology, such as smiling shut- ability that a specific facial expression is represented,
ters. While traditional FER mainly focuses on recognizing the probability values are not so reliable, especially for
whether a clear facial expression is represented or not, evaluating intermediate expressions. This is because the
from the viewpoint of monitoring in daily life, evaluating intermediate facial expression was out of the scope of
the degree of expression for the individual is rather cru- the traditional studies; learning is likely to output a value
cial, especially for patients with dementia who have little close to the binary value of either no expression (0) or
or no facial expressions. Based on this point of view, this an expression (1). As a result, for example, when the
research aims to draw a curve of transitions of the indi- degree of smile expression is estimated for a series of
vidual’s degree of facial expressions, particularly smiling facial expressions, the value may change abruptly over a
intensity, as shown in Figure 1. series of times as shown in Figure 2. In addition, it is also
dificult for the machine learning algorithm to directly
(MMaLc4hCinMeHL),eAarAnAinIg20f2o4r, VCaongcnoiutviveer, BaCn,dCManeandtaal Health Workshop learn the intermediate facial expressions since it is
difi* Corresponding author. cult for even humans to give appropriate absolute values
$ shimonishi@i.kyoto-u.ac.jp (K. Shimonishi); for intermediate facial expressions.
kondo@ccm.media.kyoto-u.ac.jp (K. Kondo); Kondo et al. [
Time is based on the assumption that the problem of relatively evaluating which of two images represents more smiles by comparing two images is easier than absolutely evaluating a degree of smiling from only one image.
By borrowing this comparison-based idea to evaluate facial expressions, we propose an approach to evaluate smiling intensity with an ordinal scale. The basic idea of this approach is that if we have multiple reference face images for a specific individual and a method for comparing facial expressions, we can evaluate the smiling intensity of a new image of the individual through pairwise comparison with the reference images, as shown in Figure 3.
Since the expression ratings in this method are based on an ordinal scale, the degree of each rating is not the same for multiple individuals. However, this ordinal scale-based approach may satisfy our need to capture changes in facial expressions for each individual.
In addition, reference image selection is crucial for this ordinal-based evaluation because they are considered an evaluation space for facial expressions. Therefore, we also propose an algorithm of reference image selection from a large number of face image data of each individual based on consistencies of comparison results within images.
In summary, the contributions of this paper are as follows: • We propose an approach to evaluating intermediate smiling intensity by ordinal scales based on comparisons. • We propose an algorithm for selecting appropriate reference images to construct appropriate evaluation space.
Then, we introduce an approach to evaluate facial expressions by ordinal scales and an algorithm of reference image selection. We evaluate our approach and algorithm with human annotators, and finally, we conclude our research.
2.1.1. Facial Action Coding Systems
Facial Action Coding Systems (FACS) [
Siamese network [
a NN (sh C Forward comparison stream Inverse comparison stream rse )ed lya ra FC (sh
In the inverse comparison ‘Ascent’ ‘Descent’ ‘Ascent’ ‘Descent’ In the forward comparison ‘Ascent’ ‘Descent’
Ground truth
‘Ascent’
‘Descent’
Categorical cross
entropy loss
recognition [
Kondo et al. [
evaluation by ordinal scales
two-category classification problem (i.e., determining
which of two input images represents the greater degree
of smiling) and construct a Siamese-based network to
recognize smiling similar to the network Kondo et al.
have developed [
Figure 4 shows the structure of the proposed network
that accepts two input images and returns two likelihood
values corresponding to ascension and descension labels
relative to the degree of smiling. We employed the CNN
component of VGG16 [
Simply thinking, the degree of smiling in the reference images can be determined by searching the position whose scores are maximum. Here, the position can be derived as: 3.3. Voting-based evaluation where (, ) and (, ) represent probabilities that the degree of smiling of image is larger or smaller than that of image , respectively. In other words, (() > ()) and (() < ()), where () represents a degree of smiling of image . Also, and represent the likelihoods of the forward comparison stream and inverse comparison stream, respectively.
In total, our network is trained to decrease the following loss function: To apply voting-based ordinal-scale evaluation as described above, we first need to construct an evaluation space with several reference images. Since the proposed approach utilizes ordinal scales, the construction of the evaluation space is crucial for the capability of the approach. Although a straightforward way is to utilize all the face data as reference images, the evaluation space = + + . (3) constructed by very similar or subtle diferent images is unreliable due to the ambiguity of these images.
Here, we expected that the CNN and the fully con- In this paper, we first consider all the data as baseline nected components would be trained to compare ex- images and take pair-wise comparisons to sort all the tracted features in order to project the results onto the data in a dataset and construct a baseline ranking. Then, likelihood values of the ascension and descension labels, we select several images from the baseline ranking as respectively. reference images and quantize the evaluation space by taking into account consistency to address the issues due to ambiguity.
Since the reference images may include some ambiguity 4.1. Baseline ranking construction between neighboring images, it is dificult to directly determine the degree of smiling of the new target image Figure 6 (a) shows a comparison table of the result of all in a reference image set. Therefore, we apply a voting pair-wise comparisons in baseline images. Each color technique to determine the final rank of the image. The shows a probability of how a target image has a stronger algorithm votes to possible ranks using the result of each smile than a reference image, i.e., (, ). The comparison of reference images and a target image. As blue shows a pair whose target image has a stronger a result, the most likely rank should have a maximum smile than the reference image, i.e., (, ) > number of votes. (, ). In contrast, the red area shows a pair
In particular, the procedure is as follows. Suppose that whose reference image has a stronger smile than a target we have reference images with its order of degree of image, i.e., (, ) < (, ). The white smiling, i.e., () > ( ), ∀ < . A new target area represents that the target and the reference images image is compared to all reference images, and like- represent similar facial expressions. lihood values that the degree of smiling of target image is By sorting the baseline images based on the sum of the larger than that of a reference image (, ) and probability values in each column of this table, a baseline likelihood that the degree of smiling of target image is ranking considering the consistency of the strong-weak lower than that of a reference image (, ) for relationship can be constructed (Figure 6 (b)). In particall reference images ( ∈ {1, . . . , }) are obtained. Be- ular, suppose we have baseline images {1, . . . , } cause if () < () is estimated, the smile rank in total, and denote images sorted in descending order of is estimated as larger than , large values are by smiling intensity as {1, . . . , }. Since the gae m i frceneeeR
Consistent e g a m i cne ree Inconsistent f e R
(a) Aconsistency of strong-weak relations in the baseline ranking images (b) Aconsistency square of
neighboring images calculated the same as the bottom-right figure of Figure 6, the less red and white colors area is a better sign of reference image selection. strong-weak relationship between each image to other To realize that, we focus on a square region of neighbor images ^, ˆ ∈ {1, . . . , } are calculated as probabil- images as shown in Figure 7, and call this square consisity values and , the total consistency values in tency square. Suppose the diferences between images the baseline ranking is derived as are significant, and a strong-weak relationship is evident ⎧ in the images. In that case, the consistency values in the = ∑︁ ⎨⎪ ∑︁ (, ˆ) icnontshiestesqnucyarsequbaerceoamree ablsluoee.xpInecctoednttroabste, lwarhgeen,i.iem., acgelelss ⎪⎩{^|^∈{1,...,−1}} are similar, and therefore the diference between images ⎫ is ambiguous, the consistency values in the consistency + ∑︁ (, ˆ)⎪⎬ (5) saqnudarreedb.ecome low, i.e., cells in the square become white {^|^∈{+1,...,}} ⎪⎭ The basic idea of building a consistent evaluation space is to quantize images with low consistency values in the By maximizing this total consistency, base- consistency square into a single class. As a result, the line ranking images (1, . . . , ) = ambiguity between these images becomes “don’t care” in arg max 1, . . . , can be obtained. From now the evaluation space, and the consistency of the evaluated on, the subscript will be used to sort the images in value becomes significant. That is, it is good to select descending order of smiling degree. images where the total consistency values in the sum of
An example of the consistency of the strong-weak consistency squares, as shown in the right of Figure 7, relations in this rearranged table is shown in Figure becomes low. 6 (c) by replacing (, ) into (, ) In addition, neighbor images in the baseline ranking when () < ( ). We here denote probabil- should not be selected as reference images. In other ities , to indicate this consistency as follows: words, to select good reference images, the evaluation space is better to be divided evenly. To realize that, select , = {︃ ((,,) ) iiff (()) >< (( )),. aofgcroounpsisotfernecfyersepnacceeimbeacgoemsseos tshmaatltlh.eTsoumsumof tuhpe, aitreias (6) better to choose a group of images for which both the
Ideally, all cells would be blue, i.e., consistency is nearly sum of consistency values and the sum of areas within equal to 1. However, due to the ambiguity of compar- the consistency square is small. Here, selecting a group ison results for similar facial expressions, there is also of images with high consistency values within the consisambiguity in the consistency between the neighboring tency square is equivalent to selecting a group with low baseline ranking. Therefore, reference image selection is inconsistency. In summary, a group of reference images important to construct appropriate evaluation space. should be selected to maximize the total values of inconsistencies in consistency squares divided by the sum of 4.2. Reference image selection the areas of consistency squares.
In practice, a procedure of reference image selection An important factor in selecting reference images is the is the following. Suppose there are images in total, consistency of the strong-weak relationships within ref- and we want to select one image as a reference image. erence images. That is, when the consistency table is (2) = At first, baseline ranking is constructed as introduced in the previous subsection and obtains consistency values , (, < ) for all pairs in the ranking. When evaluation space is divided into two with image ( < ), the sum of inconsistency values of the two spaces divided by the sum of the area is calculated as:
(7) By searching the position whose (2) are maximum, the best reference image that divides evaluation space into two can be obtained. Similar to this, by calculating the sum of inconsistency in the consistency squares divided by the sum of the area with diferent division numbers + 1, reference images can be obtained.
When it comes to calculating these values, we apply a scheme of dynamic programming to reduce calculation costs.
training data in this paper. In particular, we manually an• How a proposed network can evaluate image pair. notated segments that we thought the degree of smiling • How appropriately reference images can be se- ascended or descended monotonically. We then picked lected regarding consistency of both network and each segment’s start and end frame to construct one pair human annotators’ evaluations. with its label. The number of image pairs of each dataset • How the selected reference images evaluate face were 216, 174, and 123, respectively. Also, all face images images. in these pairs were utilized as baseline images. That is, the size of the dataset is twice the size of the image pairs; 432, 348, 246, respectively.
At first, the face image dataset is constructed by captur- 5.2. Evaluation scheme ing participants’ face images. We conducted two types of experiments to construct datasets with diferent situa- The procedure of evaluation was constructed the followtions. In the first type of dataset, we asked a participant ing four steps; (1) the proposed network was trained for to sit in front of the camera and to listen to funny radio. each individual by collected data and was evaluated by In the second type of dataset, we asked a participant to the cross-validation scheme; (2) we constructed the basesit in front of the laptop PC and play a simple game. We line ranking and evaluated the voting-based algorithm by captured facial images of these participants during the determining the rank within the images in the baseline experiment. The second experiment was still experimen- ranking; (3) reference images were selected from baseline tal but closer to a natural scene than the first one. Each images as proposed in the previous section and evaluated dataset was constructed only by one participant because by human annotators on how they were consistent; (4) our focus was to build a model to evaluate each individ- we confirmed the smiling intensity of face images in the ual. We collected two datasets of the first type, and one evaluation space constructed by reference images. dataset of the second type. As for training our network, we utilized pre-trained
Then, we added labels between image pairs that feature extraction layers of VGG-Face [
As for evaluating selected reference images, nine reference images were selected from the baseline images.
Then, human annotators were asked to evaluate which of the two images represented more smiling for the pair of neighboring images in reference images. The images up to the third nearest neighbor images were considered a pair, and all image pairs were annotated twice by swap- Figure 9: Selected reference images of dataset 1. More smile ping the left and right sides of the comparison image. images are located on the left side.
After comparing the image pairs within each dataset, the participants moved on to the next dataset. The order of evaluated image pairs was randomized for each dataset, but the order of the datasets was constant. We evaluated the consistency of the reference images by how accurate and consistent annotators evaluated the image pair. A group of 9 images regularly extracted every /10 from the baseline images {1, . . . , 9} was used as the reference image for comparison. Seven participants be- Figure 10: Consistency of estimated rank and original rank tween the ages of 21 and 27 (6 male and 1 female) were in baseline ranking recruited as annotators.
5.3.3. Selected reference images
5.3.1. Prediction accuracy Figure 11 shows selected reference images by the
proposed algorithm and equally picked up from the baseline
We first show the evaluation results of the trained ranking. The consistency table correlated to this result of
comparison network in terms of accuracy. In dataset 1 is shown in Figure 12. In this figure, the green
this evaluation, five-fold cross-validation was applied, line shows where the algorithm divides baseline ranking.
and prediction accuracies of three datasets were These results show that there still appears to be some
am99.5% (215/216), 100% (174/174), 98.3% (121/123), biguity between adjacent images, even with the proposed
respectively. Figure 8 shows examples of prediction re- approach. However, it appears to be reduced compared
sults with Gradient-weighted Class Activation Mapping to a group of images acquired at regular intervals.
(Grad CAM) [
Weakest smile (a) Selected reference images by proposed algorithm
Weakest smile (b) Selected reference images by regularly
picked up from baseline ranking
images and the proposed network are shown in Figure 16.
Figure 12: Consistency table of dataset 1. The green line Since it is sometimes hard to qualitatively evaluate two shows where the algorithm divides baseline ranking. adjacent images in a row, the four reference images skip one rank at a time. The images with a smile level one class lower than the reference image are listed, and each row shows the same evaluation value. The images on the images selected by the proposed algorithm obtain higher left side of the figure are recognized as having a higher accuracies and higher consistencies. Since the smiles ex- degree of smiling. This result confirms that the proposed pressed in the experimental time were quantized into ten method efectively evaluates the degree of smiling within levels and the maximum value of the smile was not very the ordinal scale. high, both methods have a certain degree of similarity be- Finally, a part of the transition of the smiling intensity tween the neighboring reference image pairs. Therefore, during the experiment is shown in Figure 17. In this evaluation by humans may be somewhat dificult even result, an evaluation score was smoothed by the median with the proposed method. However, even in such a situ- iflter to trace the trend of transitions. We can see that ation, we can confirm that the proposed method selects Evaluated images
Strongest smile
Weakest smile the participant smiles several times in this period. It can be seen that smiles of slightly stronger intensity than the middle level occurred several times in succession in the first half of this period. In comparison, smiles of considerably stronger intensity occurred with a short interval in the second half of this period.
In this paper, we propose an approach to evaluate the degree of smiling of individuals by ordinal scales based on multiple comparisons for the purpose of monitoring individuals. Suppose that we have enough data from individual face images; we also propose an algorithm for selecting appropriate reference images for the ordinal evaluation.
Experimental results show that our ordinal scale-based evaluation can successfully give the degree of not only clear smiling but also intermediate facial expressions. In addition, we can see that an evaluation space constructed by selected reference images by our algorithm is more consistent and, therefore, considered to be reasonable.
One of the future works is to map the proposed and constructed ordinal scale to some physical index. Although this paper proposed a method of selecting reference images that are somewhat reasonable when evaluated by humans, the validity of the scale would be improved if it could be mapped to some physical index. For example, by measuring the myoelectricity of facial muscles, the degree of muscle activity could be used as an index. In addition, the other future work is to apply this technique to people whose facial expressions do not change much, e.g., dementia patients, as we described in the introduction section.
Our method aims to monitor the daily health conditions of a specific individual by evaluating the smiling intensity using a model trained specifically for the individual’s facial images. Since data for model training and smiling intensity evaluation can be collected and processed at terminals installed in each individual’s environment, it is expected to reduce the risk of leakage of particularly strong personal information such as facial images being stored in the cloud in practical applications.