Understanding how the data used to train intelligent systems afects their behaviour is a critical task in the Artificial Intelligence field. It is also known that making artificial agents capable of adapting their actions according to the culture improves their interaction with humans. For this reason, it may be crucial to know how the cultural component inside data afects the prediction of intelligent systems. In this paper, we propose a method to acknowledge the cultural factor inside data, and we show some preliminary results obtained by using Random Forest and Support Vector Machine models on two publicly available datasets.
Intelligent systems, e.g., social robots, are a technology that has grown in the last few years.
According to a recent definition of [
On the other side, it is well known that people’s cultures play a crucial role in human-human
interaction: if we consider, for example, the thumb-up gesture, we notice that this is a widely
recognized sign of approval in several countries, but it is also used as an insulting gesture in
Bangladesh. Hence, given the importance of culture in human-human interactions, it is natural
to ask oneself if culture may play the same important role also in human-robot interactions.
Answers to this question have been given in some recent works [
For this reason, the problem of making artificial agents able to interact with humans by considering their culture has also been recently addressed. To embed culture in intelligent systems, researchers might use two approaches: • Theory-based: design a cultural representation that is embedded inside computational models such that they will produce culturally-aware predictions. • Data-driven: exploit culturally-dependent data in the learning phase of models so that they can produce culturally-aware predictions.
Among the most relevant works in this context, a theory-based model for automatically
adapting HCI systems to users by using a culturally-aware adaptive procedure has been presented
in [
Following the example at the beginning of this article, we may consider a classification
problem with a culturally-dependent gestures dataset where the goal is to predict whether the
gestures are ofensive. By analysis, we may find that the accuracy of a model trained with
data depending on one culture and tested with data of the same culture is significantly higher
than the accuracy of the same model tested with data depending on another culture. However,
building an accurate model for every diferent culture may not be possible since the labeled data
are insuficient. For this reason, we may exploit an instance weighting technique [
By combining the two concepts of intelligent systems and culture, we propose in this work a methodology based on classification techniques for understanding the role of culture when intelligent systems need to be trained to perform a given task, for example, activity recognition. Our method starts from the conjecture that, in a classification problem, the data to be classified are necessarily afected by cultural aspects. This method can be, in principle, used with any dataset containing a feature that clearly identifies the cultural context where data have been collected.
In the following chapter, we show some preliminary results obtained by relying on custom Random Forest and Support Vector Machines models with two diferent culturally-dependent datasets.
The proposed work aims to analyze how the cultural component afects the data by evaluating a
learning model trained on diferent datasets. More in detail, we picked two public datasets [
At the end of this process, we considered the following datasets:
• The CROCUFID food dataset [
Both datasets would be helpful to predict whether or not a food contains a critical ingredient for a person’s health. For instance, a social robot that suggests dishes to a person will avoid suggesting those that contain sugar if the person is overweight or diabetic.
We trained and optimized four Random Forest and Support Vector Machines models: • 1) Trained by using only the data that have = and by removing the cultural identifier. • 2) Trained by using only the data that have = and by removing the cultural identifier. • 3) Trained by using all the data and by removing the cultural identifier.
• 4) Trained by using all the data.
To choose the best Random Forest model, we used 1000 trees, and we optimized only the hyperparameter that controls the cardinality of the random subset of features that each node of the trees can check. To choose the best Support Vector Machine model, we used the radial kernel type, and we optimized the C hyperparameter that controls how much we want to avoid misclassifying each training example and the gamma hyperparameter that controls the kernel size. We used 10-fold cross-validation and Balanced Accuracy as well as AUC value as metrics for both models.
After the training, we tested each model by using two diferent test sets: one contains only data that have = while the other contains only data that have = . To better compare the models, we maintained constant the number of training and validation samples that we used.
The ROC curves with AUC values achieved by predicting if there is sugar or not in the whatscooking dataset are shown in Figure 1: it can be noticed that models trained with data related to a specific culture fail to predict the right class if tested with the data related to the other culture. It can also be noticed that adding the cuisine feature in the model trained with all the data slightly improves the result.
These results are coherent with the confusion matrices in Figure 2 obtained by optimizing the Random Forest models as described before. The only diference is that in this case, the metric chosen for the optimization is the Balanced Accuracy, which considers possible unbalances of classes during the training. It may be noticed from Figure 2 that the model trained with Thai cuisine data well predicts test data related to Thai cuisine, but does not perform well with the test data related to Cajun-Creole cuisine. The same also occurs in the opposite situation (i.e., model trained with Cajun-Creole cuisine data and tested with data related to Thai cuisine). Finally, the model trained with data related to both cuisine types performs well with both test data, but here an interesting fact can be noticed: the model fails at predicting if the sugar is not present in the test data related to Thai cuisine. This may be due to the fact that both datasets are used in the training process, and the model cannot completely identify them in the classification process.
By using the CROCUFID dataset with Random Forest models, we have obtained results similar to the ones represented in Figure 1 but less noticeable: the model trained with Western data had only an improvement of 0.13 in the AUC value when tested with Western data with respect to Asian. Similarly, the model trained with Asian data had only an improvement of 0.10 in the AUC value. By using the Support Vector Machine models, as shown in Figure 3, the diferences become more pronounced, but the behaviour remains coherent with the one obtained using Random Forest models.
Even if these preliminary results are encouraging, it must be noticed that considering diferent couples of cultures may produce diferent outcomes. Sometimes diferences between models cannot be noticed, e.g., in the comparison of Indian vs. Italian recipes. The same occurs with other critical predicted ingredients, such as peanuts. Further analyses are needed to better assess the role of cultural factors inside data and their possible exploitation in classification problems. To this aim, a custom dataset related to human gestures has been developed, and it is currently under analysis. The dataset is composed of more than 2000 images, related to two gestures that are similar to each other: the Namaste gesture, which is used in the Indian culture, and the Pray gesture, typical in diferent countries around the world.
Overall, this preliminary work lays the basis for a deeper investigation to understand the role of culture in classification problems. We finally note that the authors who created the culturallydependent datasets may have introduced their stereotypes during the creation process: Italian food is not the one you find in Italian restaurants around the world, and the same is true for other cuisines. We did not consider the possibility that datasets have been compiled with the version "for foreigners" of popular dishes, but this can be explored in the future.