A Multimedia Scripting System for Endangered Language Learning Min Chen1 , Mizuki Miyashita2 , Beau Ufen1 , Bradley Bahls1 , and Tyler Julian1 1 Department of Computer Science University of Montana Missoula, MT 59801, USA 2 Linguistics Program, Department of Anthropology Unviersity of Montana Missoula, MT 59801, USA Abstract. In this paper, we propose a Multimedia Scripting System (MSS) to support endangered language learning. With the use of ad- vanced multimedia techniques and web 2.0 technology, MSS provides a user friendly environment to ensure that general users (without strong technical backgrounds) in an on-line community can effectively and col- lectively create multimedia stories and game scripts for a common cause. It is an essential component of our on-going endangered language revi- talization project. Key words: multimedia scripting system, story-telling, educational games, endangered language revitalization 1 Introduction As stated in [10], games can be very effective tools for learning. However, the complexities of educational game design is considerable [1]. Recently some re- searchers proposed to take non-linear digital stories as a starting point for gen- erating game scripts. With the advance of multimedia techniques and Web 2.0 technology, collaborative multimedia story-telling has been proposed [4]. How- ever, the research is still at an early stage as how to enable general users (without strong technical backgrounds) to manipulate various multimedia data in order to collaboratively create multimedia stories and game scripts. In this paper, we present some on-going work and ideas on revitalizing en- dangered languages using a multimedia scripting system (MSS). MSS promotes the use of advanced multimedia techniques and web 2.0 technology to ensure that new evolving multimedia presentations (i.e., multimedia stories or game scripts) can be dynamically created, edited, aggregated, syndicated and shared via web by both experts and general audiences. The game scripts and multime- dia contents collected and created in MSS can later be used for the development of a gaming environment. The remainder of the paper is organized as follows. Section 2 gives a brief background introduction and literature review. Section 3 details the MSS system. Finally, Section 4 concludes the paper and discusses some future works. 2 Min Chen et al. 2 Background and Literature Review Languages are an essential part of the living heritage of humanity but more than half of the 6000 world languages will disappear in this century [9]. A so- lution to this linguistic and cultural crisis is to revitalize languages by teaching them to young generations. As proven by researches in second language teaching methodology, people learn expressions more effectively with appropriate situa- tions without being provided with grammatical explanations [11] and through physical actions without learners first or native language [3]. Therefore, edu- cational game is a promising learning tool where users acquire their heritage language through simulated real life social communications and activities. It is commonly agreed that game development is an interdisciplinary field [2] requiring appreciations for the background of game subjects. However, most en- dangered languages are not fully described nor digitally documented. Non-linear storytelling, which has been an effective means of knowledge sharing and learn- ing in organizations and societies for a long time, therefore becomes a valuable input to the game development. [4] proposed to combine classic multimedia pro- duction environments for non-linear stories with Web 2.0 environments for user generated contents. This is a good approach for collective story-telling. However, more work needs to be done to create a user friendly environment where general users can easily participate in the on-line community, readily manipulate the multimedia data, and proactively contribute to the generation and optimization of multimedia stories and game scripts. 3 Multimedia Scripting System Our project is envisioned to accommodate various language communities. Cur- rently, we are working on revitalizing Blackfoot, a Native American language with very few documents and only about 5000 speakers in the world [7]. With such limited resources, we want to maximally share the knowledge and ideas among the community. In addition, it is required that the learning materials used in the project are socially relevant and culturally correct. Therefore, a mul- timedia scripting system (MSS) is proposed to enable the community to share resources and compose multimedia stories and game scripts collaboratively. A group of activities are supported in MSS, which can be roughly catego- rized into Web-based and multimedia-based functions. The first category in- cludes some common Web functions, such as Upload (i.e., adding data to the system server), Post (posting a subject to the webpage), Describe (adding de- scriptions, which can be used for searching, about a subject), Search (finding interested data), Recommend (sending the link of a subject to others via emails), Comment (posting comments about a subject), Share (defining the read/write permission about a subject among the community), etc. Currently, an interface to YouTube [12] has been developed in MSS so the users are able to search for data in MSS and YouTube. A future improvement is to provide interfaces to popular on-line media searching engines such as Google Image Search [8]. A Multimedia Scripting System for Endangered Language Learning 3 The multimedia-based functions are the focus and main contributions of MSS, which are used to generate and update multimedia presentations. In MSS, a multimedia presentation is broadly defined to be either a single media object (e.g., a text document, image, audio track, video clip, etc.) or a composition of multiple media objects that are related temporally and/or spatially. Therefore, a multimedia story or a game script is essentially a multimedia presentation. Two functions, Multimedia Presentation Authoring and Layered Augmenting, have been developed to enable the collaborative creation and manipulation of multimedia presentations by general users without strong technical backgrounds. 3.1 Multimedia Presentation Authoring Multimedia presentation usually contains various media objects with spatial and/or temporal relationships. In our current prototype, we have adopted the Multimedia Augmented Transition Network (MATN) [6] to model the temporal relationships between any two media objects and to support the design and ren- dering of a multimedia presentation. Fig. 1. (a) The MATN-based authoring environment for multimedia presentations and (b) The rendered multimedia presentation played by the web browser As shown in Fig. 1(a), to add a media object into a presentation, users can first issue a query using the Search function (defined earlier) and see the results in the upper-left window. A preview of the selected object will be given in the bottom-left window. By using the “Add an arc” function, a window pops up where the selected file can be added. The users can continue adding more files if they are to be synchronized. By clicking the “Enter” button, a corresponding arc will be generated. The main window in Fig. 1(a) shows a modeled presentation, where the labels on the arcs represent different media objects to be included in the presentation and the circles represent their starting and ending points (reading from left to right). A multimedia presentation can be easily created, edited, saved, and opened through this MATN-based authoring environment. We have also implemented an interpreter to translate the MATN structure automat- 4 Min Chen et al. ically to an HTML+SMIL document for Web-based presentation rendering (see Fig. 1(b) for a snapshot of the resulting presentation). 3.2 Layered Augmenting To facilitate collaborations among the community, a system should have the capability to support media manipulation and to enable shared but differentiated social viewing among users [5]. They are realized by an annotation creation tool called AnnoTube in MSS. Currently this tool provides a graphical and intuitive interface for users to apply enrichments to a video. Fig. 2. One example enriched video in AnnoTube. As shown in Fig. 2, it has four main sections, a main window on the top, an editable information panel on the bottom-left, a toolbar on the bottom-right (with “Add Text,” “Add Image,” and “Add Audio” buttons and a “Submit” command) and a layer manager box (above the toolbar). After selecting a video (from YouTube in this example), users can apply enrichments by click on “Add Text,” “Add Image,” or “Add Audio” button. A layer is created and its name is placed in the layer manager (“Layer 1,” “Layer 2,” etc.). Editable informa- tion panel shows its related parameters, where users can enter texts or upload image/audio depending on the type of the enrichment. Such enrichment is auto- matically placed at the upper-left corner of the main window but can be dragged around using the mouse (the X and Y coordinates in the editable information field will be changed accordingly). Users can also define when to show such an- notations by specifying the On and Off time. More layers can be added following the same process. Each layer can be edited by selecting its name in the layer manager and changing values in the editable information panel. In the example, three layers (one text box and two images) are placed on the black margin of the video. If an audio layer is added, it will replace the original sound track of A Multimedia Scripting System for Endangered Language Learning 5 the video during the specified On and Off time. These are dynamically created and inserted to the page through Javascript, utilizing the DOM structure. Once the user has finished layer creation, the video can be saved and up- loaded back to the MSS server via AJAX to be used in multimedia presentation. The advantage of layer augmentation is that users are allowed to create alter- native views by selecting or unselecting the layers in the layer manager box. For example, by unselecting layers 2 and 3, the video becomes the one showing inside the top right box of Fig. 1(b). 4 Conclusions In this paper, a multimedia scripting system (MSS) is presented for endangered language learning. In the system, various web-based functions and multimedia functions are proposed to ensure an on-line participating environment where general users can share their opinions and actively contribute to the creation and improvements of multimedia stories and game scripts. The system can be further improved in many aspects. For instance, the Multimedia Presentation Authoring function can be extended to model the spatial relationships among media objects and the Layered Augmenting function needs to support the enrich- ment on other media objects besides videos. The game scripts and multimedia contents collected and created in MSS can be used for the game development. References 1. Akilli, G.K.: Games and Simulations: A New Approach in Education? In: Gibson, D., Aldrich, C., Prensky, M. (eds.) Games and Simulations in Online Learning: Research and Development Frameworks, pp. 1–20. Information Science Pub (2007) 2. Argent, L., Depper, B., Fajardo, R., Gjertson, S., Leutenegger, S.T., Lopez, M.A., Rutenbeck, J.:Building a Game Development Program. In: Computer, vol. 39, no. 6, pp. 52–60 (2006) 3. Asher, J.J., Adamski, C.: Learning Another Language Through Actions: The Com- plete Teacher’s Guidebook. Sky Oaks Productions (2003) 4. Cao, Y., Klamma, R., Martini, A.: Collaborative Storytelling in the Web 2.0. In: First Workshop on Story-Telling and Educational Games (2008) 5. Cesar, P., Bulterman, D.C.A., Geerts, D., Jansen, J., Knoche, H., Seager, W.: En- hancing Social Sharing of Videos: Fragment, Annotate, Enrich, and Share. In: Pro- ceeding of the 16th ACM international conference on Multimedia, pp. 11–20 (2008) 6. Chen, S.-C., Shyu, M.-L., Zhao, N.: MediaManager: A Distributed Multimedia Man- agement System for Content-Based Retrieval, Authoring and Presentation. In: Pro- ceedings of the 9th International Conference on Distributed Multimedia Systems, pp. 17–22 (2003) 7. Ethnologue. http://www.ethnologue.org 8. Google Image Search. http://images.google.com/imghp?hl=en&tab=wi 9. Harrison, K. D.: When Languages Die: The Extinction of the Worlds Languages and the Erosion of Human Knowledge. OUP, Oxford (2007) 10. Horizon Report, http://www.nmc.org/pdf/2006\_Horizon\_Report.pdf 11. Omaggio Hedley, A.: Teaching Language in Context. Heinle, Boston (2000) 12. YouTube. http://www.youtube.com/