Chatbots are said to soon become the most frequently used consumer channel service in the world. Users' expectations are growing, especially regarding the usage of the natural language. In this study, we present our attempts of training NLP models for Japanese dialect variation with the aim of creating Japanese dialect-aware chatbot in the future. Firstly, we describe the current situation on chat-bot market. Secondly, we focus on describing attempts that have already been made to create chatbots sensitive to dialects and low-resourced languages. Furthermore, the process of collecting data needed for our research is presented. Finally, evaluation results after model training are shown and described. The results reveal key insights into the models' strengths and limitations in understanding Japanese dialects. We discuss noted occurrences and finally outline directions for our future work.
the correctness of the answers given and the understanding of the input text were identified as the biggest problems. It led company carrying out the survey to conclusion that technical aspects directly related to response content, such as natural language processing and chatbot response accuracy need to be improved.
The outline of the paper is as follows. Firstly, we discuss the current state of chatbot technology and actions taken so far to build chatbots sensitive to dialects. Then, we shortly describe collecting data process and how we utilized it. What is more, we explore one of the fundamental steps essential for creating Japanese dialect-aware chatbot, meaning training models assumed to handle Japanese dialects. Results of experiments are presented and discussion points are highlighted.
Around 2027, chatbots are supposed to become the dominant customer service channel for approximately
25% of organizations 3. Moreover, the global chatbot market size was calculated at USD 5,132.8 million
in 2022 and is predicted to achieve a compound annual growth rate (CAGR) of 23.3% from 2023 to 20304.
Such a scale indicates the improvements taking place in the world of Artificial Intelligence (AI) as well
as Natural Language Processing (NLP)‘s realm. Technologies that enhance the knowledge of chatbots
encompass, among others, deep learning, machine learning and NLP [
Researchers have been working on developing chatbots that are dialect-aware and capable of handling
under-resourced languages to improve their performance and usability across diverse linguistic
communities. We cannot yet speak of a significant number of them, however, on the basis of those available
we can observe certain trends and draw conclusions. Chatbots created to understand language varieties
in linguistically rich countries or communities are coming to the fore. Among 21 countries in the Arab
world, each has its own dialect. In the case of Arabic, nine dialectal categories are distinguished and
each of them has more varieties depending on the particular city or town [
Boulesnane et al. [
In India, despite the common perception that English is mainly spoken there, other so-called 21
modern Indian languages are used and English is spoken by only ten percent of India population [
As the interest in language varieties is constantly growing [
In this study, a comprehensive data collection approach was employed, utilizing several key datasets, including: • Japanese Dialect Conversations Data6: includes transcripts of short dialogues in diferent
Japanese dialects in diverse formality of language.
• Corpus of Japanese Dialects: COJADS7: a parallel corpus consisting of standard Japanese,
(hyo¯jungo) text and dialect texts as well as audio. It has been sorted according to 47 Japanese
prefectures. Among those corpuses and datasets that we decided to incorporate in our research,
COJADS has the the widest variety of dialectal data.
6Ho¯gen ro¯ru purei kaiwa deetabeesu [Japanese Dialect Conversations Data] http://hougen-db.sakuraweb.com/[access:
2024/06/27].
7Corpus of Japanese Dialects (COJADS). NINJAL. We used version published in March 2023.
https://www2.ninjal.ac.jp/cojads/index.html [access: 2024/06/27].
• The Corpus of Kansai Vernacular Japanese [
By incorporating those, it was necessary to pay heed to the Japanese writing system. Japanese is written as a combination of three types of symbols as well as roman letters and Arabic numerals. The oldest symbols, kanji, came to Japan from China and were adapted to the Japanese language. These logographic characters are used for conceptual words and names. Two syllabaries, hiragana and katakana, were developed later. Hiragana‘s main function is recording words and grammatical endings that are not written in kanji, while katakana serves as a method of writing words of foreign origin [36]. In COJADS, mixed kanji and hiragana script marks text in standard form of Japanese. On the contrary, dialect texts are written in katakana. 8Shinnosuke Takamichi, "JMD: Japanese multi-dialect corpus," https://sites.google.com/site/shinnosuketakamichi/publication/ research-topics/jmd_corpus [access: 2024/08/08].
We decided to convert dialectal texts from the other four corpuses into katakana. Before that, we adjusted data for our needs: Firstly, we decided not to include tagged parts of each sentence. For instance, we cleaned the text from non-verbal sounds, like cough or laugh, that were originally indicated by curly brackets as we focus on the text, not the audio version. The same was done to the hyo¯jungo version. Consequently, we did the same with other corpus that treated non-verbal sounds in the same way in the text, namely Japanese Dialect Conversations Data. For the same reason, we did not take into consideration symbol /// that was supposed to highlight unknown meanings of words. Moreover, in Japanese Dialect Conversations Data, one could find the arrows that indicated up or down intonation. Those were also deleted, together with signs // which meant that the immediately following utterance started at the same time as the next speaker’s utterance [37]. Some of the talks were held in standard Japanese, however, it needs to be emphasized that each dialogue difers and we do not possess parallel data. When it comes to the Corpus of Kansai Vernacular Japanese, we used a morphologically untagged version of the texts. In JMD: Japanese multi-dialect corpus for speech synthesis neither tags nor special signs were used. We only had to clear the text of some kanji readings (probably those considered as hard), which were taken in brackets and written in the hiragana syllabary, and transform the text into katakana.
For katakana conversion, we used MeCab [38], a highly optimized Japanese morphological analyzer. It efectively processes mixed kanji and kana sentences, converting them into uniform katakana or hiragana versions. It is believed to achieve a high eficiency at breaking down Japanese text into individual morphemes, which we find essential for accurate katakana transcription. We started from converting each character into katakana (e.g., お父さん became オトウサン, read: oto¯san). Then we focused on transcription that can provide a standardized phonetic nuance representation (so here お父さん became オトーサン), which is believed to be useful when dealing with various Japanese dialects. We hope that this method will benefit in capturing the subtle diferences between accents and pronunciation among dialects.
In the next stages of our study, removing empty lines was needed. Possessing all data in their katakana version, we were able to incorporate them into relevant classes. We have adhered to the division used in COJADS to a large extent as grouping dialects by prefectures can provide a convenient framework for study. However, due to the apparent dominance of hyo¯jungo and the Kansai dialect, we decided to combine Nara, Kyoto, Osaka and Hyo¯go based texts into one data, as Kansai-ben. These dialects share many features, which can make them more compatible for grouping. Due to scarcity of data, dialect imbalances in our dataset were, for the time being, unavoidable. On the other hand, dataset constraints made it impossible to capture dialects spoken in territories more extensive or narrower than prefectures and between diferent social groups. For instance, Tsugaru dialect (hereafter, Tsugaru-ben), spoken on the west side of Aomori prefecture 9, was not mentioned as specified group, but still covers some of the data from class called Aomori. This choice, with all of its advantages and disadvantages, enabled us to start testing. We also made a decision to include hyo¯jungo in our research. It was developed from the educated middle-upper-class Tokyo dialect for the purpose of spreading a sense of a nation unity within the country in the Meiji era (1868-1912) [39]. This denotes that hyo¯jungo is also a variation of a language. It is believed that recognizing the standard language as one of many varieties, among that each with separate rules and functions, enables scholars to perceive it within the broader context of language variations, meaning dialects and social variants [40]. We also felt an obligation to discriminate between hyo¯jungo and the rest of the data operated under the name of Tokyo. In COJADS, Tokyo was treated purely territorially, but authors provided users information about particular places (namely: district, villages, towns etc.). It could be easily observed that language used within a metropolis was standardized, while language spoken in rural areas near Tokyo (like, for example, Hinohara village) possessed noticeable diferences from it. That is why we draw a line between those two.
Next stage consisted in making dialect distribution. We checked the number of lines (sentences/turns) per each dialect area. The greatest number was observed in the Kansai dialect (more than 300,000 lines), 9See more here: https://tsugaruben.net [access: 2024/10/11]. followed by hyo¯jungo that noted only 175,849 lines due to the division we have applied. There is a significant gap between those two classes and the rest of the groups. Remaining groups did not even reach 100,000 lines or at least 50,000 lines. Third place went to Gunma, with 18,165 lines. The lowest number of lines was recorded in Yamanashi, Saga, Nagasaki and Tottori, respectively. Tottori had less than 200 lines. Next step involved an investigation of the average length of each line, calculating the average length of sentence for each dialect and finally checking their standard deviation as it is known as a universal and functional measure as it shows the average distance of each score from the mean.
First of all, in order to create a Japanese dialect-aware chatbot, we need a dialect identification model that will be able to detect dialectal text written in katakana quickly and successfully. Texts are categorized by dialect/area, and the neural network is trained on katakana representations for each dialect category. In the experiments, for training we used 525,569 out of 656,826 sentences, while the rest of them was left for testing. All four models were trained using learning rate 5e-05 and a batch size of 32. We set 3 as a number of epochs. In the first place, we trained four models with preprocessed data: • tohoku-nlp/bert-base-japanese10: A BERT model pre-trained on a Japanese corpus and trained on Japanese Wikipedia. • tohoku-nlp/bert-large-japanese11: A bigger BERT model that contains around 30M sentences. • tohoku-nlp/bert-large-japanese-char12: A character-level version of the larger BERT model, trained with the whole word masking enabled for the masked language modeling (MLM) objective. • tohoku-nlp/bert-large-japanese-v213: A word-level variant of the large BERT model, enriched by the Japanese portion of CC-100 dataset that incorporates approximately 392M sentences alone.
Our evaluation measures, namely Precision, Recall and F1 Score, are widely used for Text Classification [41]. Their formulas demonstrate the significance of retrieval of positive examples in text classification. Table 1 presents evaluation results for models mentioned above.
All four models were based on Bidirectional Encoder Representations from Transformers (BERT) [42] - language framework that deals with pretraining techniques such as Mask Language Modeling (MLM) and Next Sentence Prediction (NSP) [43]. The tohoku-nlp/bert-base-japanese model entirely outperforms others, reaching the highest overall F1 score. The tohoku-nlp/bert-large-japanese model achieved, surprisingly, slightly lower scores in comparison to the base version, despite possessing more parameters. Moreover, the tohoku-nlp/bert-large-japanese-char model performance is poor across almost all dialects. It may suggest that the character-level approach is not efective for this task. The tohoku-nlp/bert-large-japanese-v2 model showed some improvement over the character-level model. However, it still noticeably shows small efectiveness compared to the base and large models.
After getting unfavorable results, we decided to compare the performance of these with two other models: • google-bert_bert-base-multilingual-uncased14: trained on multilingual data, does not make distinction between capital and lowercase letters. It was described in detail by Jacob Delvin et al. [42] • google-bert_bert-base-multilingual-cased15: almost identical model, but prepared to detect diferences between uppercase and lowercase letters.
Letter case might can heavily impact meaning in some languages. Texts in languages like Japanese, Chinese or Korean usually do not have any spaces between words. This fact was considered during the development process.
The results are summarized in Table 2. Both models performed very well dealing with hyo¯jungo and Kansai dialect. Strong performance could be also observed for Okinawian dialect, especially high F1 score for cased model. A score of over 0.50 was achieved by Aomori and Oita for both models. Both for tohoku-nlp models as well as google-bert models, poor performance across all metrics can be noted among Tottori, Tochigi, Shiga and Hiroshima regions. Most of the dialects can be found in the group of dialects with visible moderate performance, rating F1 score between 0.20 and 0.40.
Significant disparity between Precision and Recall might be observed within some dialects, like Fukuoka (for models 1, 2, 5 and 6). Same thing applies to Gifu. Discrepancy is highly noticeable among dialects with overall bad performance that mark very low precision and even lower recall, like Aichi, Niigata, Kanagawa or Mie.
Based on these results, we can observe a significantly strong correlation between results and amount of data. Kansai dialect, with the largest amount of sentences, reaches the best results whereas low-sourced dialects note scores close to 0. It indicates the need for more extensive data collection and consequently searching for more training data in general. The google-bert_bert-base-multilingual-uncased model generally performs better than the cased model, especially in dialects with lower performance. This leads to an assumption that case insensitivity might be helpful in efectively handling diverse language forms. tohoku-nlp/bert-large-japanese-char-v2. Model 4: tohoku-nlp/bert-large-japanese-v2. n 2 4 0 0 9 6 5 0 3 6 9 3 4 6 2 1 1 2 7 5 9 3 4 2 6 4 9 6 4 1 9 7 8 8 3 0 5 3 1 9 6 7 7 iso .70 .10 .50 .70 .20 .10 .20 .50 .50 .40 .50 .00 .20 .30 .50 .50 .10 .40 .80 .10 .30 .20 .30 .30 .30 .60 .50 .60 .50 .10 .10 .30 .50 .00 .10 .00 .20 .02 .05 .03 .02 .02 .03 /trceaA ¯jgooyun iichA itakA irooAm iabhC ieEhm ikFuu aokkFuu isakuhm ifuG aunGm iirsaohm iaookkd iIraakb iIsaakhw aaagKw isaagohm aaaagnw isaanK icohK iM izaayM aagN saagN iigN O aakyOm iaknwO agS itaaSm iSh iaShm izoSuh coTh isokuhm ttooT aoyTm aaakym caagum saaanm aoumm okT e ik on iak taa ita a a a a a e a i h h t y g n k ig a ir a a i i o o le h Fu H H K K T W Y Y K
O s 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 C s
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 S a l b en 28 5 4 5 5 82 4 0 4 3 6 9 3 9 3 0 9 8 8 8 3 m t 6 1 5 7 7 1 6 4 3 2 3 5 9 9 u n 0 3 0 1 8 7 0 2 3 9 5 6 4 5 3 9 9 6 9 5 0 5 0 8 8 6 0 0 4 7 2 3 2 7 3 4 7 1 7 6 2 1 e 4 4 8 2 1 1 9 8 9 8 4 2 3 1 1 0 9 4 5 1 8 0 6 3 0 2 6 2 8 0 1 0 9 6 1 4 2 0 6 4 4 7 N s 1 9 1 6 7 1 3 5 1 2 1 2 1 7 8 1 1 3 2 1 2 1 4 5 1 9 9 2 5 1 1 1 9 8 1 1 2 8 9 7 9 7 multilingual- cased.
Evaluation results. Model 5: google-bert_bert-base-multilingual-uncased, Model 6: google-bert_bert-baseNumber of
Class
Dialect/Area
Model 5
Model 6
An interesting case in an Okinawa region that reaches a good performance, despite the small amount of data. The reasons for this may be found in many diferences between Japanese and Ryukyuan languages. It is essential to notice that Ryukyuan languages have been downgraded to the status of Japanese dialects [44] despite being mutually unintelligible in many aspects with standard Japanese. Aleksandra Jarosz [45] as major features of Ryukyuan languages points out, among others, the raising of short mid-close vowels and a phonological appearance of the glottal stop. The question is, whether other models will achieve similar ratings for Okinawa or is it only used models specificity.
Striking variability between Precision and Recall results was noticeable in some classes. Recall is the proportion of Real Positive cases that are accurately Predicted Positive while Precision represents the proportion of predicted positive cases that are correctly identified as true positives [ 46]. Imbalance between them might suggest that while some models are fairly efective at recognizing instances they can detect, they still fail to capture a significant amount of relevant dialect data. This situation might a a consequence of a lack of suficient feature representation. Moreover, results achieved by some classes like Aichi, Niigata or Mie indicate that models are often incorrect with their predictions. Their low recall reflects inefectiveness in identifying the majority of actual instances.
For certain dialects, specialization might be necessary since specialized models can be fine-tuned with dialect-specific data, resulting in higher Precision, Recall, and F1 scores for dialects with poor performance. This poses a dilemma of whether to create several specialized models for various dialects or a single, highly generalized model.
In our study we monitored the performance of some models working on Japanese dialects. We presented the results and outlined conclusions. We also introduced some already existing dialect-aware chatbots and chatbots that were made with the thought of low-rescourced languages. The data collection process was also explained in detail.
Models successfully dealt with standard Japanese and Kansai dialect, but showed an inability in detecting less-resourced dialects. This contrast suggests that standard NLP models might have problems in generalizing across dialectal features without adequate data representation. This stresses also the need for more targeted training or extensive data collection. Moreover, it gave us a hint to try implementing other learning rates and scrutinize what impact they will have on the model performance.
Our study highlights the challenges in dialect detection due to data scarcity, emphasizing the complexity of dialect detection within Japanese language. Facing mentioned challenges is crucial for advancing NLP systems that can accurately reflect the rich linguistic landscape of Japan.
In the future, besides learning rates implementation, we plan to check the performance of some other models in order to have a comparison and choose the best performing model as a way to focus on its application, namely Japanese dialect-aware chatbot. This paper revolved around dialect recognition, but it cannot be excluded that in the future we will focus on style shifts within dialects. Style adaptation might greatly influence the chatbot performance and users‘ impression. With the increasing need of personalization [47], incorporating both dialect recognition and the ability to adjust for conversational style can provide a more customized and responsive user experience.
In the case of Japanese, formality’s level of the language plays an important role in customer service and everyday relationships. Speaker‘s awareness of his position in particular situation in Japan is reflected by honorific language, keigo [48]. However, among dialects keigo expressions can also difer. On the other hand, there are also regions, like southern part of the Tohoku Region, that do not have the linguistic forms corresponding to the honorific and humble forms of the national lingua franca, and where honorific expressions are considered to be uncommon 16. Hence, profound research on honorific forms among dialects will be needed. Taking into consideration the scarcity of dialectal data, for the time being successful distinguishing between keigo expressions within dialects might be impossible. Nonetheless, collecting dialectal data focused on honorific forms might be another goal to be set. B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan, 2018, pp. 434–437. URL: https://aclanthology.org/L18-1067. [36] W. Hadamitzky, M. Spahn, Japanese Kanji & Kana:(JLPT All Levels) A Complete Guide to the
Japanese Writing System (2,136 Kanji and 92 Kana), Tuttle Publishing, 2013. [37] I. Fumiko, A Study of Regional and Generation Diferences in Discourse Pattern, Technical Report, The National Institute for Japanese Language and Linguistics, 2014. Application/pdf, NINJAL Collaborative Research Project Reports ; 13-04. [38] Y. Sim, A morphological analyzer for japanese nouns, verbs and adjectives, ArXiv abs/1410.0291 (2014). URL: https://api.semanticscholar.org/CorpusID:15195083. [39] S. Saito, M. Turner, Language diversity in “monolingual” japan: Language awareness among high school teachers of english, Journal of Language, Identity & Education (2024) 1–15. URL: http://dx.doi.org/10.1080/15348458.2024.2385837. doi:10.1080/15348458.2024.2385837. [40] R. Quirk, Language varieties and standard language, English Today 6 (1990) 3–10. doi:10.1017/
S0266078400004454. [41] M. Sokolova, G. Lapalme, A systematic analysis of performance measures for classification tasks,
Information Processing & Management 45 (2009) 427–437. doi:10.1016/j.ipm.2009.03.002. [42] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/N19-1423. doi:10.18653/v1/N19-1423. [43] C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al., A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, arXiv preprint arXiv:2302.09419 (2023). [44] P. Heinrich, S. Miyara, M. Shimoji, Handbook of the Ryukyuan Languages: History, Structure, and
Use, 2015. doi:10.1515/9781614511151. [45] A. Jarosz, Japonic languages: an overview, Silva Iaponicarum (2017). doi:10.14746/sijp.2017.
41/42.6. [46] D. M. Powers, Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation, arXiv preprint arXiv:2010.16061 (2020). [47] T. Ait Baha, M. El Hajji, Y. Es-Saady, H. Fadili, The power of personalization: A systematic review of personality-adaptive chatbots, SN Computer Science 4 (2023). URL: http://dx.doi.org/10.1007/ s42979-023-02092-6. doi:10.1007/s42979-023-02092-6. [48] E. T. Rahayu, Japanese honorific language in various domains, in: Proceedings of the Fourth Prasasti International Seminar on Linguistics (Prasasti 2018), Atlantis Press, 2018/08, pp. 25–34. URL: https://doi.org/10.2991/prasasti-18.2018.5. doi:10.2991/prasasti-18.2018.5.