This study investigates the language utilised in dialogues in abridged literary texts in Japanese that are commonly appreciated by language learners. The focus on dialogue is essential, as it uniquely illustrates the dynamics between interlocutors whilst highlighting cultural and linguistic norms. The study explores phenomena typical of the Japanese language, such as registers of formality, the use of end-of-sentence particles and explicit subject markers, as well as universal linguistic features commonly used in readability formulas and considered a proxy for textual complexity. Three world-famous literary texts as abridged by popular publishers (e.g. 'Aoitori Bunko') are considered, and insight is sought concerning how the dialogues within them compare to the texts in their entirety as well as to the dialogues in unabridged versions of the same texts. Ultimately, an experiment is proposed in order to test the overall validity of the identified linguistic patterns, in which the state-of-the-art GPT-4o model is provided with relevant guidelines and asked to simplify literary dialogue.
Advanced dialogue systems such as OpenAI’s popular chatbot ChatGPT and a growing fascination for
Japanese-language cultural artefacts like animations and comic books coincidentally mark the current
global landscape, particularly in relation to a young audience. A natural intersection point of the two
phenomena is the educational sphere, where AI-based tools are more and more commonly employed
as learning support, and the number of learners of Japanese as a foreign language is demonstrating
a perceivable increase [
The current work aims to provide quantifiable insights into the characteristics of abridged texts in the Japanese language for future use in the context of automatic language generation. Specifically, the rendition of dialogues is initially focused on as they saturate an array of specificities of Japanese culture, including interpersonal dynamics and levels of politeness, which in turn are expressed through language in standard ways, such as the chosen register (long vs short verb forms). The dialogues in three adapted literary texts (Alice’s Adventures in Wonderland, Les Miserables and Tom Sawyer) are examined with a focus on typical features of the Japanese language pertaining to dialogue (e.g. the use of end-of-sentence particles) as well as readability-based characteristics (e.g. type-to-token ratio). Comparisons are made, on the one hand, between the characteristics of the dialogue portions of the texts and the texts taken as a whole and, on the other hand, between dialogue in the abridged texts and in their unabridged counterparts. As a result of the comparison, guidelines for simplification of literary dialogue in the language are established and, by extension, conclusions can be drawn about the features of such dialogues that render them easy or dificult for readers. In a related experiment, the GPT-4o model (as per ChatGPT) is asked to provide a simplified version of a literary dialogue in Japanese in three discrete settings: zero-shot, one-shot (in which a reference pair of unabridged and abridged versions of a dialogue-filled text is provided) and through chain-of-thought prompt engineering that incorporates the quantified norms established by the current work.
Automatic dialogue generation diferentiates between task-oriented and open-domain systems. The former are associated with a specific domain and, typically, specific knowledge and databases. In contrast, open-domain systems are broader in scope and may be generative, retrieval-based or ensemble. Dialogue systems, like the majority of NLP-related technology, have seen development over the past few decades from rule-based through machine-learning-driven to neural approaches. In the last couple of years, Large Language Models (LLMs) such as GPT have been particularly revolutionary in the field.
The evaluation of automatic dialogue is a challenging task due to dialogue’s multiple and not readily
quantifiable characteristics. Whilst task-oriented or retrieval-based open-domain dialogue systems
might benefit from the application of BLEU scores, which compare their responses to human-made
ones, the same cannot be said of systems that support a freer textual format. In their survey on the topic,
[
Dialogue systems have been applied in foreign language education since the 1960s [
A dialogue is an exchange between two or more parties that is associated with a specific setting and
semantics. It is important to underline that the dialogues present in literary texts are by no means
identical to those that occur in real life. [
Japan is known for its specific (and often perplexing for foreigners) social norms, especially when it comes to formal communication. Naturally, these norms are to a great extent reflected verbally in interpersonal dialogue in the Japanese language.
To start with, several registers, associated with specific verb conjugation, occur in Japanese in
function of the interlocutor’s perceived ’status’. A very simplified rule of thumb would require that one
utilise short forms (e.g. da1) with close friends and family, -masu forms (e.g. desu) in the majority of
circumstances, ’humble’ forms (e.g. de gozaimasu) when referring to themselves in front of a superior
and ’honorific’ forms (e.g. de irasshaimasu) when addressing or referring to a superior interlocutor.
However, as [
Examples of constructions that exert significant influence on their context are end-of-sentence
particles. They are specific to and almost indispensable in Japanese dialogue. The particles zo, ze, sa and yo
are associated with insistence, whilst others like ne and na denote a request for confirmation. However,
practice once again shows that the employment of the phenomenon is not this easily generalisable. In
fact, multiple and sometimes conflicting theories have been brought forward considering the nature
of discrete particles and the diferentiation between them. For instance, the speakers’ ’territory of
information’ is often invoked [
Unlike English and other Western languages, Japanese introduces subject and object roles within a
sentence via specific case markers that precede the associated verb. In addition, a ’topic’ marker (wa )
is also commonly used, whose function, albeit similar to that of the subject marker, is diferent in that
it is not directly associated with syntactic roles. The absence of explicit particles, commonly employed
in dialogue, is a case of anaphora referred to as ’zero pronoun’. Its function is seen as comparable to
that of personal pronouns in English [
3.1. Texts
The primary literary texts investigated are abridged versions of Mark Twain’s The Adventures of Tom
Sawyer2, Victor Hugo’s Les Miserables3 and Lewis Carroll’s Alice’s Adventures in Wonderland4. The
ifrst two books belong to the series ’Aoitori Bunko’ by the publisher Kodan [
Both series as well as others like ’Tsubasa Bunko’ [
In order for conclusions to be drawn that are specifically valid for and set apart abridged Japanese texts of an afirmed quality, each of the abridged texts is coupled with a non-abridged Japanese trans1to be 2Twain, M., Nishi, K., Iijima, A.: Tomy Souyaa no bouken. Kodan: Aoitori Bunko (2012). 3Hugo, V., Katayama, W., Tsukahara, R.: Re mizeraburu. Kodan: Aoitori Bunko (2012). 4Carroll, L., Nishida, Y.: Fushigi no kuni no arisu. Popura: Poketto Bunko (2015). 5logographic Chinese-based characters 6corresponding, respectively, to the B1 and B2 levels of the CEFR framework lation of the same original text.7 8 9
All texts are converted to plain text format10, any metadata (including titles, character information and prefaces) is removed and basic textual cleaning is applied, including the removal of non-alphanumeric symbols resulting from textual conversion. Tabs and new lines are removed to allow for further processing of the texts. For tokenisation of Japanese sentences into words, python’s janome11 library is utilised. For conversion of kanji into kana12, the pykakasi13 library is opted for. Dialogue is defined as all text closed between ’’ and ’ ’ signs that ends in punctuation.
Two sets of linguistic characteristics are measured in relation to dialogue instances in the selected texts. When applicable, the dialogue in the texts is compared to the texts taken as a whole and/or with the dialogue in the abridged texts’ non-abridged counterparts. The first set is conceived to represent the specificity of Japanese dialogue and consists of the percentage of sentences: 1. with zero pronouns i.e. not containing the topic (wa) or subject marker (ga); 2. ending with the short verb form da and the related forms datta, dewa/ja nai and dewa/ja nakatta 3. ending with the full verb form desu and the related forms deshita and dewa/ja arimasen 4. ending with the particles yo, ne, na, sa, zo, ze and wa.
The second set consists of atomic readability characteristics that are not specific to the Japanese
language and are commonly considered as representative of textual complexity: (i) the text’s total length
(in characters), (ii) the percentage of dialogue in the full text, (iii) the average number of words per
sentence, (iv) the average number of letters per word and (v) the type-to-token ratio (TTR) (vi) the
percentage of words that are not found in a frequency list14 [
Following the measurement of the characteristics in abridged and unabridged versions of Japanese literary texts as detailed in section 3.3, an experiment in automatic generation of abridged text is conducted. For this purpose, ChatGPT is employed in its recent GPT-4o version. The model is asked to provide an abridged version of an extract from Charles Dickens’ A Christmas Carol15 as translated into Japanese, which contains a large amount of dialogue. Three discrete generation scenarios are experimented with based on the nature of the provided prompts: zero-shot (in which only instructions are provided), one-shot (in which the model is given an example pair of unabridged and abridged extracts, taken from the respective professional versions of Alice in Wonderland) and chain-of-thought (which includes additional prompting following the model’s generation). In the chain-of-thought setting, the model is asked to explicitly alter the text output in the zero-shot setting based on the trends that are 7Twain, M., Yoshida, K.: Tomy Souyaa no bouken. Kaisei (1975). 8Hugo, V., Yoshio, T.: Re mizeraburu. Kodan: Aozora Bunko (1987). 9Carroll, L., Yamagata, H.: Fushigi no kuni no arisu. Asahi (2000). 10Where OCR is required, it is achieved with the tool https://www.pen-to-print.com/ 11https://pypi.org/project/Janome/ 12syllabic characters 13https://pypi.org/project/pykakasi/ 14The utilised frequency list is based on the top 30k words of the Open Subtitles multilingual corpus, accessible here: https://github.com/hermitdave/FrequencyWords). Words are considered as types rather than tokens. Due to the nature of the frequency list, words consisting entirely of hiragana characters are not considered. 15Dickens, C., Katokt (trans.): Kurisumasu Kyaroru. Tatsu-Zine (2003). discovered in relation to the professional texts but not exhibited by the current output. For the full generated texts, please refer to Appendix A; and for the prompts employed in the three scenarios, please refer to Appendix C.
The generated texts are then evaluated qualitatively via a survey administered to seven speakers of
Japanese as a foreign language. There are two versions of the survey, each of which includes two of the
automatically generated texts16. The two versions share a common text to facilitate the calculation of
agreement between participants; the one-shot text is opted for as the other two texts are very similar
to each other and their co-occurrence may cause confusion. The evaluation protocol is adapted from
[
The detailed analysis of our experiments are shown in Table 1 (Alice in Wonderland), Table 2 (The Adventures of Tom Sawyer) and Table 3 (Les Miserables) . The lengths of each abridged and unabridged text were calculated and so were the ratios between each pair, revealing variation between 0.07 (for the longest original text, Les Miserables) and 0.87 (for Alice in Wonderland).
The percentage of dialogue was discovered to be consistently higher in the abridged texts, ratios ranging from 1.14 for Tom Sawyer to 1.38 for Les Miserables. The number of words per sentence is lower for dialogue than for the text taken as a whole in all six texts. Due to the presence of a clear trend, the ratios between the values of this feature for abridged and unabridged texts were also calculated in an attempt to discover a further trend that characterises abridged texts. Indeed, it was discovered that the ratios between the number of words per sentence in dialogue and in the full texts are generally higher for the abridged versions (or identical in the case of Alice in Wonderland)17.
No clear trends were discovered in relation to the number of letters per word (e.g. it is higher for dialogue in both of Alice in Wonderland’s versions but this is the case only for Tom Sawyer’s abridged version). In contrast, TTR does exhibit a trend, as it is higher in the case of dialogues in all texts (thereby speaking of higher lexical variety). Therefore, the ratios of the values in relation to the three abridgedunabridged pairs were also calculated; however, no further trends pertaining to abridged texts were discovered18. The percentages of words that fall outside the utilised frequency list were higher for the texts when taken as a whole as compared to the dialogue within them (the sole exception being Tom Sawyer’s unabridged version). No further trends are noticeable in relation to a text’s ’unabridged’ versus ’abridged’ status19.
The Japanese-specific feature ’percentage of sentences without topic or subject marker’ reveals a general trend pointing to higher values in dialogue parts20. Calculation of the values’ ratios within a textual pair shows a further trend of higher ratios in unabridged as compared to abridged texts21. 1 and the related forms datta, dewa/ja nai, and dewa/ja nakatta 2 and the related forms deshita and dewa/ja arimasen
3 yo, ne, na, sa, zo, ze, and wa 4 the proportion of number of characters in dialogue over the number of characters in the whole text 5 based on the multilingual Open Subtitles corpus and available at:
https://github.com/hermitdave/FrequencyWord
Although the percentages of short and long verb forms within the diferent texts vary significantly as a consequence of the nature of the characters and their interactions (e.g. there is much higher formality in Les Miserables than in Alice in Wonderland), there is overall a higher number of short forms in all six texts. A calculation of ratios between the two types of verb forms in textual pairs was conducted; however, revealing no further trend22.
With the exception of Les Miserables, the abridged texts are associated with a lower percentage of end-of-sentence particles. Understandably, the choice of particles used is highly text-specific; for instance,Alice in Wonderland contains many instances of the particle wa, which is used almost exclusively by female speakers. The most common particles in both types of texts are yo, ne, and na; whilst the particle ze is generally avoided, in particular in abridged texts (even when used abundantly in the unabridged counterpart, like in the case of Tom Sawyer).
ChatGPT Output Table 4 ofers detailed juxtaposition of the investigated features within the textual extract provided to ChatGPT versus its zero-shot output. The overall length of the two texts is, respec1.68 vs 2.12 for Les Miserables 2211.8 vs 3.19 for Alice in Wonderland 6.46 vs 21.96 for Tom Sawyer 1.21 vs 1.76 for Les Miserables tively, 2331 and 1457 characters, making the abridged version 0.63 of the length of the original one. This proportion falls neatly within the range exhibited by the previously examined professional texts. The percentage of dialogue is, naturally, high as the extract has been specifically selected to include extensive dialogue: 0.7 for the abridged vs 0.57 for the unabridged text. The abridged text contains a higher percentage of dialogue, thus matching the trend discovered in relation to the professional texts. TTR is higher for the automatically abridged text, which also keeps in line with the discovered trend. Similarly, the number of words outside the frequency list is indeed higher in the full text than in its dialogue portions. However, when it comes to the number of words per sentence, the trend of there being fewer words in dialogue parts is not exhibited by ChatGPT’s output.
The percentage of sentences without subject or topic marker is higher in dialogue parts than in the full text both within the original extract and ChatGPT’s output. To go further, the ratio between the two values is higher for the unabridged text, thus keeping up with both trends related to the textual feature. Interestingly, even though the percentages of sentences ending in da vs desu go against the discovered trend in the originally provided extract (the latter being more common), ChatGPT’s output does in fact match the trend. Finally, a higher percentage of sentences end in a particle in the abridged text, going against the relevant trend. More importantly, there is very little variety in the included particles, yo single-handedly accounting for their vast majority.
Based on this conducted evaluation, ChatGPT was asked to make two modifications in the chain-ofthought setting: (i) Reduce the number of words in dialogue sentences and (ii) Add a larger variety of end-of-sentence particles, such as the commonly used ne and na. Whilst the requested particles were indeed added23, the overall length of sentences in dialogue was not decreased24. Still, upon reflection, no further prompts were provided at this point in order for the model’s unedited attempt at meeting the requirements to be evaluated in the ensuing survey.
Human Evaluation The survey’s ’version 1’ (which consisted of the zero-shot and one-shot texts) was answered by three respondents and ’version 2’ (one-shot and chain-of-thought) by five. 23ne: 11.86%; na: 8.47%; yo: 6.78%; zo: 1.69% 24in fact, it was slightly increased to 14.82 due to the particles that were added simultaneously
Inter-rater agreement as per Fleiss’s Kappa was low for all three texts25. Highest agreement was associated with ’correctness’ in relation to the one-shot text and with ’textual coherence’ in relation to the chain-of-thought one. Agreement was stable between categories for the zero-shot text.
The chain-of-thought text received the highest percentage of ’yes’ answers, 76%; followed by 24% ’mostly yes’. The one-shot text received 70% ’yes’, 20% ’mostly yes’ and 10% ’mostly no’. Finally, the zero-shot text received 66.7% ’yes’, 26.7% ’mostly yes’ and 6.7% ’mostly no’ answers. No text was given less than 50% of ’yes’ answers for a given category.
The one-shot text was met with a higher number of negative free-text comments. Grammar was said to be complex, and the lack of kanji representations to be confusing. It was sometimes dificult to tell who was speaking, and the lack of explicit subject or owner rendered some sentences hard to understand. On the positive side, the text was said to be ’very flowing and natural’. Furthermore, the dialogue attributed to the characters was perceived as vivid and judged to match their personal traits (”Scrooge’s dismissive attitude and the cheerful responses of his nephew”).
The zero-shot text was said to be written in standard and fitting ’conversational Japanese’, and it was pointed out that repetitions within dialogue were efective. The chain-of-thought text, in which ChatGPT was explicitly asked to replicate the trends discovered in this study, received mostly positive comments. Anaphora was said to be more readily resolvable (e.g. the pronoun kare26 was employed several times). Compared with its one-shot counterpart, the text was said to be ’better and smoother’ and to contain simpler grammar. A specific sentence was said to be more ’aesthetically appealing’ than the parallel one used in the one-shot text. According to the negative comments, complex kanji were sometimes used, verb tense was inconsistent on one occasion and, importantly, there was said to be less of a striking diference in the dialogue styles attributable to the two main characters. 25the possible answers (’yes’, ’mostly yes’, ’mostly no’, and ’no’) were taken as categorical at the calculation of agreement 26he
An automatic evaluation of both universal and language-specific features as present in dialogue in abridged (and related unabridged) literary texts in Japanese led to the discovery of several distinct trends. Some of them simply pertain to dialogue in Japanese literature, whilst others go further to reveal specific traits of the dialogue present in abridged texts. The latter trends help shed light on these texts’ features that render them suitable for an audience of language learners and/or younger readers.
Abridged texts are associated with a higher percentage of dialogue and a lower percentage of endof-sentence particles. There tend to be a higher number of infrequent words in the text when taken as a whole than in its dialogue portions. TTR is higher in dialogue compared to the literary texts taken as a whole, and there tend to be more short than long verb forms in dialogue. The number of words per sentence is higher in dialogue than in full texts, and the ratio between the two values is higher for abridged texts. Sentences without marked topic or subject are more frequent in dialogue, and the associated ratio is higher for unabridged texts.
The chain-of-thought setting of automatic abridgement, in which ChatGPT was asked to apply modifications for its initial output text to inscribe within the mentioned trends, was met with significant encouraging feedback, such as the clear use of anaphora, which also characterises the zero-shot text. However, an important piece of criticism was the perceived lack of distinguishing features in the characters’ language. There are indeed instances of dialogue in the chain-of-thought text that go against the grain of Scrooge being rather informal and patronising in contrast with his nephew being cheerful and always respectful. An example is the nephew informally skipping the question marker ka; e.g. ” ” 27. However, upon verification, these instances are identical in the zero-shot text. Therefore, one can safely conclude that it was not the fact of asking the model to make a couple of specific adjustments that caused a compromise in terms of creativity and charac27”Why would you say this?” terisation. Conversely, it was the one-shot setting that helped enhance creativity. For instance, there, the mentioned question is rendered in a very formal manner and does end in a question particle: ” ”
Interestingly, increased use of kanji was seen as an advantage and drawback by diferent participants in the survey, leading to the conclusion that generalisation of the texts’ qualities in relation to diferent levels of reader proficiency is not optimal. Based on the respondents’ feedback, the one-shot text seems to be suitable for more experienced readers of Japanese compared to the other two. chain-ofthought ? ?
! 28Dickens, C.: A Christmas Carol. Chapman & Hall (1843).
It is likely that the one-shot text’s higher overall complexity is dependent upon the sample extract that was provided to the model and not an intrinsic feature of the generation scenario - a hypothesis that should be tested through the inclusion of multiple texts of diferent proficiency levels. High potential can therefore be expected from a future combination of the one-shot technique’s natural tendency to provide variety and lexical simplification and the chain-of-thought addition’s incorporation of specific instructions.
Importantly, the current study’s observations are general in nature as they are based on a very limited number of texts. A deeper examination of the discovered trends is planned to involve a substantial increase in corpus size. At best, works of a variety of genres, years, and original languages of writing will be added. A large corpus size could allow for not only trends but also reliable value ranges to be defined concerning the examined textual features (with the caveat that some features, such as TTR, are highly dependent on textual length).
Based on the conducted evaluation, it can be concluded that automatic generation may benefit from longer chains of thought and from a combination of the one-shot and chain-of-thought generation methods. Subsequently, the newly derived texts should ideally be evaluated by a larger sample of readers of defined Japanese proficiency levels.
Comparison of the current experiment with similar ones that focus on diferent languages may provide interesting insight about the acquisition processes as associated with the languages at hand; in particular, in relation to literary dialogue. For instance, the use of anaphora in the face of the Japanese zero pronoun may be compared to the use of personal pronouns in English or other European languages. 29”the nephew answered”
The limited size of the literary corpus, caused by restrictions of availability and of a financial nature, makes observations less generalisable as the author’s or translator’s choices play a significant role in the texts’ characteristics.
Some of the examined automatic characteristics are not exhaustive in nature and are meant to help discover overall tendencies rather than specific values or frequencies. For instance, solely the verb desu is used to diferentiate between the use of short and -masu forms, and the cases when the verb may not be positioned at the end of the sentence are not accounted for.
Finally, as the number of participants in the conducted survey is limited and heterogeneous in terms of proficiency levels, it is challenging to conclude on the quality of the three generated texts solely on its basis.
Original Extract from A Christmas Carol –
– – ! – – Extract from Alice in Wonderland (unabridged), Used in the One-Shot Setting ―― ――
Extract from Alice in Wonderland (abridged), Used in the One-Shot Setting ?? –– – ” – ” ! ! ! ! ! Text Generated in the Zero-Shot Setting –– –
– –
Thank you for taking part in this evaluation, which is inscribed within the projecDtialogue in Abridged Literary Texts in Japanese: Observations and Automatic Reproducibility.
You have been assigned two texts to evaluate based on the provided criteria. The texts are automatically abridged versions of the same extract from Charles Dickens novel A Christmas Carol. Please fill in the two tables independently by marking your answers with a tick or cross (or by circling the relevant answer). The section Aesthetic Appeal is highly subjective; please provide your personal opinion. You are welcome to write additional comments in the rightmost field.
This protocol is adapted from Mousavi et al. (2022). If you have any questions, please address them to iglika.nikolova.stoupak@gmail.com
Dimension/Question tAionnswser Op- Definition tCioonmaml)ent (op
Is the text generally un- Yes derstandable (in terms Mostly Yes of storyline, vocabulary, Mostly No and syntax)? No
Does the text exhibit Yes correct vocabulary, Mostly Yes grammar, punctuation, Mostly No and structure? No
Is the text consistent in terms of register, writing style, and the information presented? Yes Mostly Yes Mostly No No
aapnAbnrereleaeesdp?elhtneeoIstxsrstaaturnhecadeplleentatterarixatltyitunorfsrnarie?telsi?eoonloIvssf- YNMMeooosssttllyy YNeos
Does the text have ap- Yes propriate length, pac- Mostly Yes ing, style, and register? Mostly No Is the text engaging? No
The text is easy to understand. It is free from overly complex words and sentence constructions.
There are no non-existent or irrelevant words, grammatical mistakes, mistakes in punctuation, or non-optimal paragraph/sentence structure.
The text maintains the same genre/level of formality, verb tenses, character traits, and knowledge. There are no contradictions in the presented information.
The text flows fluently between sentences (paragraphs). There are no unclear anaphora referents. Information (incl. separate words) is not overly repeated.
The text feels neither too long nor too short. Appropriate textual length is accorded to plot segments. Formality levels are natural.
Here is an extract from the novel A Christmas Carol by Charles Dickens, as translated into Japanese. Please generate an abridged version of this extract that is suitable for an audience of {the prompt for the zero-shot setting} Please see below an example of an original and a well-abridged version of an extract from a novel. Here is an extract from the novel A Christmas Carol by Charles Dickens, as translated into Japanese. {text} You previously provided me with the following abridged version of it: {zero-shot output} Could you please make the following small modifications in the text, whilst trying to alter it as little as possible: - Make the sentences in the dialogue parts a little shorter (in terms of number of words). - Include a larger variety of end-of-sentence particles in the dialogue, such as ne and na.