This report documents the construction and output of extreme gradient boosted algorithms that were trained using the phonemes that make up American English words to identify how different sounds express emotion and sentiment. The data comprised of two corpora that consist of words that have been assigned scores according to how they reflect certain emotions and sentiments. The models are trained only on the phonemes that make up each word. This is a unique approach to automatic emotion recognition and sentiment analysis which typically does not consider individual phonemes. In addition to the boosted algorithms, linear regression is used to examine the relationships between word length, and emotions and sentiments.
The principle of the arbitrariness of the sign [1] is a foundational concept in linguistics. It posits that there is no inherent or logical connection between the sound of a word and its meaning. In other words, linguistic signs are considered to be arbitrary, with their meanings assigned by convention rather than by any intrinsic relationship between sound and sense. In recent years, there has been a growing interest in exploring sound symbolism, challenging the notion of the arbitrariness of the sign (e.g., [2], [3], [4], [5]). These studies have revealed that concepts such as size and shape exhibit stochastic relationships with sounds. Furthermore, many of these relationships are found to be consistent across different languages. For instance, the mil/mal effect, which observes that vowels like /i/ and /a/ are more frequently used in the names of small and large referents respectively, and the kiki/bouba effect, which notes that spiky shapes are often associated with sounds resembling kiki, while rounded shapes are often associated with sounds resembling bouba, have been found to hold true in numerous languages around the world (e.g., [2], [4]). Emotional sound symbolism refers to the phenomenon where the sounds of words are associated with, and convey, specific emotional or affective qualities. In this linguistic concept, certain phonemes or combinations of sounds are thought to evoke or symbolize particular emotional states or feelings. In a cross-linguistic study of five languages [6], researchers observed a pattern whereby phonemes at the beginnings of words predict emotional valence most strongly and that phonemes associated with negative valence were uttered more quickly, drawing parallels between emotional sound symbolism and alarm calls in the animal kingdom. Important to the present study, they observed that in English, phonemes like /ʌ/, /d/, /ɪ/, and /oʊ/ were associated with negative valence while phonemes like /tʃ/, /ɛ/, and /p/ were associated with positive valence.
Automatic emotion recognition and sentiment analysis is a subfield of natural language processing which endeavors to construct algorithms that understand human emotion and sentiment in language. Sound symbolism has been largely overlooked in automatic emotion recognition and sentiment analysis although there are a few studies that explore sound symbolism through the lens of machine learning. For example, Winter and Perlman [7] constructed random forest algorithms to show a systematic sizesound relationship in English size adjectives. As with the present study, the data was engineered in a manner so that each sample returned a count of the number of times each phoneme occurs in each word. The outcome of this is a dataset comprised of mostly null values. Following on this method researchers constructed algorithms to classify Pokémon names according to their evolution level using sound symbolism [8]. They showed that the random forest algorithms were able to classify novel Pokémon names more accurately than Japanese university students assigned to an identical task. An issue of overfitting due to the high number of null values in the dataset was uncovered and resolved using crossvalidation. In the present study, word length is found to be a significant predictor of several emotions and sentiments and this effect is taken into consideration in the design and analysis of the algorithms. Potentially related, Li et al. [9] examined the relationship between utterance length and word error rates in automatic speech recognition and speech emotion recognition. They found that shorter utterances tended to have higher word error rates likely due to a lack of contextual information.
The present report outlines the construction and output of algorithms designed to combine sound symbolism with automatic emotion recognition and sentiment analysis. Two corpora, with a combined total of almost 20,000 words, are used to train 19 algorithms that are each designed to classify samples according to specific emotions and sentiments. All algorithms return significant accuracy estimates.
All data, files, codes, and links to a YouTube series documenting this project can be found in the following online repository: https://osf.io/brus3/?view_only=63412.
The present study uses two separate corpora to train machine learning algorithms. The first is the Glasgow Norms [10], a list of 5,500 words that have been assigned Likert scores for 9 sentiments. A full list of the sentiments in the Glasgow Norms is provided in Table 1. The second corpus is the NRC Word-Emotion Association Lexicon [11], a list of 14,000 words that have been assigned a binary score as to whether each word is associated with 10 emotions and sentiments. A full list of the emotions and sentiments in the NRC Lexicon can be found in Table 2. Words from both corpora were cross referenced with the Carnegie Mellon University Pronouncing Dictionary (CMU [12]) to obtain American English phonemes for each word. Words that did not find a match in the CMU were manually checked. Instances of mismatched spelling were corrected. All other unmatched samples were discarded.
All analyses were conducted in the R environment [13]. Word length was calculated by summing the number of phonemes in each word. No additional considerations were made for diphthongs or long vowels which were counted as single phonemes. The relationships between word length, and emotions and sentiments were analyzed using a series of regression equations, dependent variables being the average Likert scores in the Glasgow Norms and the binary classification in the NRC Lexicon; independent variables being word length. The XGBoost algorithms were constructed using the XGBoost [14] and caret [15] packages. K-fold cross-validation (K = 28) was used to avoid the overfitting issue reported in [8]. The data was split into 8 subsets (A-H) and recombined using a Latin square resulting in 28 subsets with a 3:1 training to testing split. For example, the first iteration of each model is trained using subsets A through F and tested on subsets G and H. The following results report on the aggregate of each series of 28 iterations. Combined significance was calculated using Stouffer's [16] and Fisher's [17] methods; however only Fisher's method is reported as it returned more conservative significance estimates. The algorithms for both corpora were designed to classify samples so the Likert scale values in the Glasgow Norms were assigned to binary categories using a median split. The XGBoost algorithm was found to be susceptible to distribution skew, so categories were balanced by randomly removing samples from the majority category. This had little effect on the Glasgow Norms dataset due to the median split but removed around 80% of samples in the NRC dataset because only around 10% of samples in that dataset have a value of 1 in the binary dependent variable. To increase variability, balancing was conducted after cross-validation sub-setting. To limit the influence of word length in the XGBoost models, phoneme counts were divided by word length so that features were a percentage how much a phoneme makes up each word. This resulted in a convergence issue during tuning, so α was manually adjusted and the same learning rate was applied to all models (α = 0.1). All other hyperparameters were automatically tuned by inputting diverse hyperparameter settings into a tuning grid.
A series of linear regression models were calculated to test the relationship between word length, and the emotions and sentiments in the Glasgow Norms (Likert score) and the NRC Lexicon (binary). Table 1 reports on the findings of the analyses conducted on the Glasgow Norms. Increased Age of Acquisition, Arousal, Size, and Valence had a significant positive correlation with word length while Concreteness, Familiarity, and Imaginability had a negative one. All significant relationships observed in the analyses conducted on the NRC Lexicon (Table 2) showed a positive correlation. These include Anger, Sadness, and Trust emotions while the Negative and Positive sentiments were also significant.
All models constructed and tested using the Glasgow Norms achieved an accuracy greater than chance and a Fisher's combined p-value < 0.001. Table 3 reports on the aggregated accuracy and standard deviation of these models. A similar result was found in the NRC models except in the case of the Surprise algorithm (p = 0.022 using Fisher's method and p = 0.021 using Stouffer's method). The NRC models are presented in Table 4. The Glasgow Norms models did report a greater accuracy than the NRC models; however, it is important to note that these were constructed and tested on larger datasets due to the balancing outlined in 2. The accuracy was, on average, higher and variability was lower in the models constructed using the Glasgow Norms models compared to the NRC models.
Tables 5 and 6 report the 15 most important features for the Glasgow Norms and NRC models respectively. Certain features are consistently important across models despite measuring different emotions. Features with high feature importance across models include voiceless plosives (/t/ and /k/), the alveolar nasal (/n/), approximant consonants (/ɹ/ and /l/), the alveolar fricative (/s/), and the openmid back vowel (/ʌ/) which appears particularly important, but it should be noted that this is also the most common phoneme in American English according to the CMU.
The most important phonemes (IPA) and their feature importance (IMP) in the Glasgow Norms models.
The most important phonemes (IPA) and their feature importance (IMP) in the NRC models.
All models achieved accuracy significantly greater than chance (p < 0.001 in all cases but one).
Although further investigation is recommended, the results suggest that it is unlikely that sound symbolism in American English expresses fine-grained emotions and sentiments because the feature importance scores suggest many models are using the same features to make decisions. Rather, it seems that sound symbolism communicates emotional and sentimental weight. Consider that the Valence model in the Glasgow Norms-where high valence is positive and low valence is negative-and the Positive and Negative models in the NRC Lexicon all showed a positive correlation with word length. Positivity and negativity are sound symbolically expressed through longer words, although this is slightly stronger for positive sentiments as shown by the NRC lexicon models and the significant, but relatively weak, positive correlation in the Valence regression model. Those sounds that have high feature importance scores across models include voiceless plosives (/t/ and /k/), the alveolar nasal (/n/), approximant consonants (/ɹ/ and /l/), the alveolar fricative (/s/) and the open-mid back vowel (/ʌ/). Most of the consistently important consonants are produced at the alveolar ridge. /ʌ/ appears to be an especially important feature across models. This observation falls in line with [6] who showed that /ʌ/ was associated with negative valence; however, few other patterns can be drawn between the that study and the present report. The high importance of /ʌ/ might also be due to a combination of its high occurrence frequency, being the most common phoneme in the CMU, and the distribution of null values in independent variables (NRC = 85%; Glasgow Norms = 88%). XGBoost algorithms are constructed using decision trees which base their decisions upon the outcomes of nodes. At each node a certain number of features are tested. /ʌ/ is the most commonly occurring phoneme in English and it will often be tested against low frequency features with null values. This issue was somewhat mitigated by dividing phoneme counts against word length, but it doesn't solve the problem entirely. Take for example Age of Acquisition Likert scores which were shown to have the strongest association with increased word length across all models, this is an unsurprising finding. However, Age of Acquisition XGBoost model feature importance scores revealed that /ʌ/ was the most important feature in that model, to a much greater degree than other models. This suggests that word length is still contributing to the models despite attempts to mitigate its influence through model tuning and data engineering. Word length could be included in the XGBoost models; however, given that length has a greater range than phoneme counts and no null values, this would likely mask weaker features [8].
That said, all models reported significant accuracy. Given that most automatic emotion recognition and sentiment analysis systems rely heavily on lexical and syntactic features, this study underscores the potential of phonemic information as an additional valuable resource for improving the accuracy of such systems, especially when dealing with emotional and sentimental aspects of language. While the current study provides valuable insights into the role of sound symbolism in sentiment analysis, future research could delve further into the interplay between phonemic features and linguistic and contextual factors to enhance the robustness and generalizability of sentiment analysis models across different languages and domains.