From Tale to Speech: Ontology-based Emotion and Dialogue Annotation of Fairy Tales with a TTS Output

Abstract

In this demo and poster paper, we describe the concept and implementation of an ontology-based storyteller for fairy tales. Its main functions are (i) annotating the tales by extracting timeline information, characters and dialogues with corresponding emotions expressed in the utterances, (ii) populating an existing ontology for fairy tales with the previously extracted information and (iii) using this ontology to generate a spoken version of the tales. Common natural language processing technologies and resources, such as part-of-speech tagging, chunking and semantic networks have been successfully used for the implementation of the three tasks mentioned just above, including the integration of an open source text-to-speech system. The code of the system is publicly available.

Keywords

• ontology
• natural language processing
• text-to-speech
• semanticnetwork
• fairy tale
• storytelling

-

The idea of developing an ontology-based storyteller for fairy tales was based on the consideration of two previous works in the field of narrative text processing. The first work is described in (Scheidel & Declerck, 2010) , which is about an augmented Proppian1 fairy tale markup language, called Apftml, which we extended according to the needs of our current work.

Our second starting point is described in (Declerck et al., 2012) , which presents an ontology-based system that is able to detect and recognize the characters (participants) playing a role in a folktale. Our system combines and extends the results of those studies, adding the detection of dialogues and emotions in the tales and an ontology-driven Text-To-Speech (TTS) component that “reads” the tales, with individ ual voices for every character, including also a voice for the narrator, and taking into account the types of emotions detected during the textual processing of the tales.

To summarize: Our system first parses the input tale (in English or German) and extracts as much relevant information as possible on the characters – including their emotions -- and the events they are involved in. This provides us with an annotated version of the tale that is used for populating the ontology. The system finally uses the ontology and a robust and parameterizable TTS system to generate the speech output.

All the data of the system have been made available in a bitbucket repository (https://bitbucket.org/ceisen/apftml2repo), including documentation and related information2. 2

Architecture of the System

Firstly, we use the Python NLTK3 and Pattern API4 to annotate the tale. Then we use the Java OWL-API5 to populate the ontology. And finally the Mary Text-To-Speech system6 is used to generate the speech output. Mary is an open-source, multilingual Text-to-Speech Synthesis platform, which is robust, easy to configure and allows us to extend our storyteller to more languages. The general architecture of the system is displayed below in Fig. 1.

The Ontology Population

The ontology we use is an extension of the one presented in (Declerck et al., 2012) , which describes basically family structures among human beings, but also a small list of extra-natural beings. In the extended version of the ontology we include also temporal information (basically for representing the mostly linear structure of the narrative) as well as dialogue structures, including the participants involved in the dialogues (sender(s) and receivers(s)), whereas we give special attention also to the narrator  of  the  tale,  since  this  “character”  is  also  giving  relevant  informati on about the status of the characters in the tales, including their emotional state. Dialogues are synchronized with the linear narrative structure. Detected emotions are also included in the populated ontology, and are attached for the time being to utterances, and will be attached in the future to the characters directly. The Mary TTS system is accessing all this information in order to parameterize the voices that are attached to each detected characters.

A Gold Standard

In order to support evaluation of the automated annotation of fairy tales with our integrated set of tools 5 fairy tales have been manually annotated7. The tales are “The Frog Prince”, “The Town Musicians of Bremen”, “Die  Bremer  Stadt  Musika nten”  (the German original version), “The Magic Swan Geese ” and  “Rumpelstiltskin”.

The annotation examples show the different steps involved in the system: the text analysis, the temporal segmentation, the recognition of the characters and the dialogues they are involved in, the emotions that are attached to the utterances and delivered during speech the story in near real time.

Summary and Outlook

We have designed and implemented in the field of fairy tales an ontology-based emotion- and dialogue annotation system with speech output. The system provides robust results for the tested fairy tales. While the annotation and ontology population processes are working for both English and German texts, the TTS output is for the time being optimized for the English language.

Future work can deal with adding a graphical user interface, extending the parsing process for annotating tales in other languages and populating the ontology with more information, like the Proppian functions. 7

The manually annotated tales, together with the annotation schema, are available at https://bitbucket.org/ceisen/apftml2repo/src/763c5eb533f09997e757ec61652310c74223838 4/soproworkspace/SoPro13Java/gold/?at=master

References

1. Horridge Matthew and Bechhofer Sean ( 2011 ). The owl api: A java api for owl ontologies IOS Press, IOS Press volume 2 number 1 , 11 -- 12

2. Schröder Marc and Trouvain Jürgen ( 2003 ). The German text-to-speech synthesis system MARY: A tool for research, development and teaching . Springer: International Journal of Speech Technology , volume 6 number 4 , 365 - 377 .

3. Marcela Charfuelan and Ingmar Steiner ( 2013 ). Expressive speech synthesis in MARY TTS using audiobook data and EmotionML . ISCA: Proceedings of Interspeech 2013

4. Steven Bird , Ewan Klein, and Edward Loper ( 2009 ). Natural Language Processing with Python--- Analyzing Text with the Natural Language Toolkit .. O'Reilly Media , (http://www.nltk.org/book/)

5. Ekman Paul ( 1999 ). Emotions In T. Dalgleish and T. Power (Eds.) The Handbook of Cognition and Emotion Pp . 45 - 60 . Sussex, UK: John Wiley \& Sons, Ltd.

6. De Smedt , Tom and Daelemans, Walter ( 2012 ). Pattern for python . The Journal of Machine Learning Research , volume{ 13 } nr. 1 2063 -- 2067

7. Scheidel Antonia and Declerck Thierry ( 2010 ). Apftml-augmented proppian fairy tale markup language . First International AMICUS Workshop on Automated Motif Discovery in Cultural Heritage and Scientific Communication Texts .. Szeged University, volume 10

8. Declerck Thierry , Koleva Nikolina and Krieger Hans-Ulrich ( 2012 ). Ontology-based incremental annotation of characters in folktales Association for Computational Linguistics . Proceedings of the 6th Workshop on Language Technology for Cultural Heritage , Social Sciences, and Humanities , 30 -- 34

9. Propp V.Y. Morphology of the Folktale . Leningrad, 1928 ; English: The Hague: Mouton, 1958 ; Austin: University of Texas Press, 1968 .

10. Inderjeet Mani: Computational Modeling of Narrative. Synthesis Lectures on Human Language Technologies , Morgan & Claypool Publishers 2012.