"Nakari ridinja nungka rikarrangu nya-mangaji ni-maliji, kilha-nyngkarri nyinku wuka ngala yinda wukanyinjawu, yinbayawu ji-awarawu, barra bawuji wakara ki-yabarri ni-wurdu, ngabaya jibiya baji ki-yibarra ni-maliji ajinjala yinku rdiyangu kurda kurdardi ji-wankalawu. " "From yesterday perhaps, or maybe early this morning those handprints, the spirit beings heard your words, when you spoke and sang for this country, so that was what they did, they felt good, so the spirit beings from this place put their handprints in the cave for you, they are newly made, they are not from a long time ago. "
This opening quote comes from Mavis Timothy a-Muluwamara, a senior Yanyuwa woman, the Aboriginal owner of sections of land and waters throughout the southwest Gulf of Carpentaria, northern Australia. Like many indigenous languages, Yanyuwa demonstrate an incredibly intimate knowledge of their country. At the time of writing, Yanyuwa is one of the most critically endangered indigenous languages, with only two fluent speakers in the elderly group. Unlike the lost land, when you lose a language, a whole way of being, a whole cultural universe, is lost forever [1,2]. The prior revitalisation effort has shown that the establishment of a knowledge repository (i.e., a data pool or language archive which contains critical indigenous knowledge including encyclopedia, books, etc) is a crucial step toward language revitalisation programme success [3,4,5,6]. Since a sizeable knowledge repository effectively connects language with critical knowledge, enriching linguistic knowledge (e.g., prefix to words, grammar rules) with oral traditions (e.g., song, stories) and other spiritual / healing practice. Such cultural contextualisation of a language can contribute to strengthening cultural identity and enhancing the overall quality of language education [7,5,6].
While the establishment of these knowledge repositories represents a meaningful effort to preserve the critical knowledge about these languages, these repositories have rarely been tailored for practical utilisation, presenting a significant barrier for usage by the non-technical community members to adopt them for language revitalisation (e.g., designing language educational material) [8,9,10]. To tackle this, we aim to investigate the Research Question:
How can indigenous knowledge be represented and shared to foster knowledge sharing between technical (e.g., computational researchers) and non-technical stakeholders (e.g., indigenous educators, community leaders, and traditional knowledge and land owners)?
Inspired by data storytelling, this paper employed an innovative knowledge graph (KG) technology to construct such language repository. Different from prior approach [11,12], the knowledge graph models the relations of different pieces of information in a graphical format, thereby providing semantically rich and deeply contextual knowledge about the Yanyuwa language. We argue that such "contextualisation" paves way for narrating more meaningful and engaging knowledge stories, which may be further polished with a "comic" layer to display visual knowledge stories to support non-technical stakeholders' sense making of knowledge encoded in the KG. In doing so, we seek to bring new insights to the evolving discourse on how to cultivate effective and inclusive language revitalisation practices for endangered indigenous languages 1 .
Scholars have long recognized the urgency of preserving and reviving endangered indigenous languages, which are currently disappearing at a accelerated rate [13,14]. Central to the revitalisation efforts has been the exploration of innovative methods that go beyond traditional linguistic documentation [8,9,15,10]. Recent literature has increasingly emphasized the growing potential of advanced computational models, particularly in the natural language processing (NLP) domain [16]. For instance, in [17], the authors have systematically explored the potential use of state-of-the-art NLP tools to assist language education to revitalise the Cherokee language, including the use of automated question generation and language assessment material creation. In a similar vein, Teodorescu et al. [18] explore the use of various language models in the Cree context (an indigenous language spoken in North America), the authors concluded that NLP tools (e.g., text classifier) may be potentially adapted for community preferences and help with language revitalisation efforts. Despite these advances, researchers caution about the challenges of translating nuanced cultural contexts of indigenous languages into such digital formats [19,20]. These insights from recent literature illustrate a growing recognition of the need for innovative, culturally sensitive methods in the documentation and revitalisation of endangered languages like Yanyuwa.
Recent development in computational techniques, such as knowledge graphs, open up new possibilities in reviving the indigenous language [21,22]. Different from prior language revitalisation efforts which typically focuses on the documentation and archiving of linguistic knowledge [23], the knowledge graph is capable of modelling the interconnectedness between language and its cultural significance as a graphical representation. Conceptually, a knowledge graph G is composed of the ontology O, the data graph D and the corresponding relation R between them: G = (O, D, R), so the knowledge could be organised as the triples ("entityrelation-entity" or "entity-property-value") which could be scaled to a large graph network [24]. The adoption of knowledge graphs as a means to encode knowledge into practical knowledge bases has shown considerable promise, especially in the field of linguistics. In fact, the early establishment of knowledge graphs like WordNet [25] and ConceptNet [26] demonstrated a way to model and analyse English language by connecting words and concepts into an intricate web of semantic relationships, these graphs had subsequently been widely used to perform natural language understanding tasks [27,28,29]. In the context of language learning and education, the knowledge graph may also serve not just as repositories of linguistic knowledge (e.g., similar to a dictionary) but also as an intelligent tool to encapsulate and disseminate the cultural and contextual elements that are central to languages [30,31,32]. Although KG has potential to innovate the field of language revitalisation, such technical solutions are typically constructed by data scientists or technical personnel, and may be difficult to understand and utilise by non-technical stakeholders, such as indigenous educators, school policy makers or parents, whose roles are critically important to the success of a language revitalisation programme [33].
To include and engage non-technical stakeholders in such KG-based language revitalisation initiative, we resorted to the growing body of research in human-computer interaction and data storytelling. Data storytelling is the combination of data, visuals and narrative to support casual users or users with less experience in complex data analysis scenarios [34,35]. Data comics have emerged as an important visual storytelling branch to data storytelling by using sequential images, constructed from data-driven visualisations [36]. Fundamentally, a data comic has two dimensions: content relation and layout. Content relation describes a set of transitions between comic panels to construct a narrative. Comic layout is the organisation of comic panels. There are six types of content relations (i.e., narrative, temporal, faceting, visual encoding, granular, and spatial) and nine content layouts (i.e., large panel, annotated, grouped, tiled, parallel, grid, network, branching, and linear). These narrative components of data comics allow for a nuanced presentation of complex information, making them particularly useful in scenarios where understanding and participation of non-technical stakeholders are key [37]. Additionally, their visual nature supports the integration of diverse data types, from quantitative graphs (e.g., data points) to qualitative anecdotes (e.g., narratives), offering a comprehensive approach to data representation. In the work of [38] data comics was successfully used to explain diabetes in a complex and multi-modal healthcare context. In a similar vein, the work presented by [39] illustrated the potential of comics to explain complex networking data with temporal changes. Our research builds on the growing interest in data comics and their capabilities for effective multi-modal data communication in a language revitalisation context.
This study is based on the comprehensive Yanyuwa archive collected by John Bradley (one of the co-authors) over a 43 year period, which includes two volume dictionary, encyclopedia of the Yanyuwa language and digitised files of stories and songs composed and sung by contemporary Yanyuwa men and women, image/photograph/map collections about the lives of Yanyuwa people, narrative animations, diagrams and illustrations of material cultures. Table 1 provides a detailed overview of the source data in this collection.
To fully harness the capabilities of knowledge graphs for storytelling, we strategically encode knowledge at various levels of granularity and across multiple modalities. We argue that such diverse array of knowledge representation may open up possibilities for more compelling blend of visual elements that subsequently leads to better narration for engaging data comic stories.
A team of three data science researchers (with no Yanyuwa expertise) worked collaboratively to systematically extracted knowledge entities of three granularity: document (i.e., a collection of text, audio file, or videos about the same topic, such as storybook, songbook, or a dictionary), sections (a meaningful segment of a document, such as a story within a storybook), and atomic knowledge (this may be a sentence, a phrase, or a video frame which cannot be meaningfully reduced further). Conceptually, these knowledge entities are "nodes" within a knowledge graph. Given the structured nature of our Yanyuwa dataset (e.g., a storybook containing chapters of stories), we can easily extract document and section entities via a manual segmentation (e.g., a single story within a storybook is extracted as a section knowledge entity). For audio and video data, text transcripts were used, and for images, captions were used. In section and atomic granularity, we excluded audio/video without corresponding transcripts (around 62% of all audio/video data), since we cannot reliably extract meaningful textual representation (our exploration of using Whisper tool, which is a state-of-the-art speech recognition tool only reaches about 30% accuracy), we plan to include these data in the next phase of this study where we conduct in-depth interview sessions with Yanyuwa community knowledge owners and educators.
Extracting atomic entities and relations. For atomic entities, we resort to automated natural language processing techniques as each atomic entities may be of variable length, and cannot be easily segmented and extracted manually. The process is threefold. First, we conducted frequency analysis using Term Frequency-Inverse Document Frequency (TF-IDF) [40], which is a widely adopted text analytics approach to identify words / terms that are of high-importance in a document. Second, for all entities, we conducted dependency parsing of their "title" property (e.g., the name of a song) in the Yanyuwa dataset; this will extract the appearance of these entities title along with their most likely dependencies (e.g., a verb which connects two entities); this enables us to extract a large number of entity-relations in the graph. Third, to further establish missing relations, we conducted a co-occurrence analysis involving identifying the frequency with which entities appear together in a given dataset (not necessarily with a dependency relation). We measured the strength of association and, for top-100 strong association, we established the relation via manually analysing the surrounding text. A key motivation behind these entity-relation extraction approaches is so that all the encoded entities and relations are clearly traceable back to the source data. Alternatively, while other inference-based approaches (e.g., GPT) may be more powerful, we opt for not using them given that they may produce novel relations / entity names that are not traceable to the original Yanyuwa dataset and hence compromise the authenticity of the knowledge graph. Once all the entities and relations were established, we used Neo4j tool 2 to import them into a graphical format. A visual summative view of the KG can be seen in Figure 1.
We invited the three researchers who developed the KG in Section 3.2 to participate in the following design workshops. All three researchers are male, and have background in data science. Two of the three researchers have prior knowledge about KG ontology and constructions before participating this project. The team conducted six design workshop sessions in total.
Session 1: introduction to data comics. Given that KG researchers do not have prior expertise in data comics, we conducted an introductory session to introduce key concepts. In the first half-session, data storytelling fundamentals were introduced. Then, for the second half session, data comics were introduced, including two important comic dimensions: content relation and layout [41]. Details regarding six types of content relations and nine content layouts were introduced (see Section 2.3). An example data comic for each layout was presented to the participants.
Session 2: design objective. To help clarify the aim of the comic, prior to the comic design sessions, we had an in-depth discussion about how to communicate the knowledge encoded in KG with the non-technical stakeholders (e.g., indigenous educators), and how can data comic help address it. All three participants agreed that this is a complex question that may be more meaningfully tackled by first deconstruct it into smaller sub-question: i) How many nodes should be involved in a single data comic? ii) What are the key visual techniques / data storytelling principles in data comics that can help "narrate" a selection of nodes? iii) how should we mitigate the risk of compromising the authenticity of the original knowledge, given that each person may narrate the story differently and none of the participants were of Yanyuwa background? Each participant was given the opportunity to contribute to these questions via filling a sticky note, followed by a group discussion to resolve conflicts. The discussion resulted in following agreement and conflicts. We reached conclusion about sub-question i and ii. For sub-question i) although presenting KG nodes as a visual data story is context-specific, there should be a minimum number of node involved for a sufficiently informative story, all the node should be connected with at least one other node; sub-question ii) in data comics, the key visual knowledge is with regard to content relation, which can be mapped from KG entityrelation, after which, the comic layout could be determined. For sub-question 3, conflicting view were raised, on the one hand, one participant believe that the knowledge story narration should be based on KG-relations, therefore the visual stories should have a "ground truth" of a set of optimal narration. On the other hand, other participants believe that, despite being guided by the KG relation, comic design are still largely subjective given that each comic design decision (e.g., which knowledge to focus, and which connecting node to feature, which layout to adopt) will affect the final comic narration. As a compromise, the discussion moderator (an experienced data storytelling researcher) decided to alternatively discuss around a set of guidelines for designing a narrative structure based on perception theory [42,43,44,45], which are recorded in a form of design Guidelines:
• G1 [Visual perception]: Visually explain and engage the audience about the multi-modal knowledge stories in the KG.
• G2 [Gestalt Principles]: Support recognition and sense-making via coherent and intuitive designs principles based on proximity and continuity of knowledge.
• G3 [Affordances]: Informing KG's use cases in language revitalisation and potentially guide the decision making about material design process.
• G4 [Cultural context]: Produce an accurate representation of the knowledge encoded by KG with traceability.
• G5 [Cultural context]: Explore the data comic's potential for designing engaging material for cultural revitalisation purposes.
Session 3: Sketching. To practice data comic design skills, a data comic researcher selected an example KG segment consisting of 8 KG entities about an animated Yanyuwa story "Mud crab", consisting of 3 atomic vocabulary entities, 3 atomic image entities, and 2 story text sections. The reason for selecting this example segment was due to that: it included a representative set of nodes from different type of entities and modalities. KG researchers were instructed to first think about the content relations of the entities for 20 minutes, then they selected comic layout to sketch a comic that can best illustrate the KG segment based on design guidelines G1-5. Feedback was given by a senior data comic researcher, and a 30-min revision session were carried out to alter / polish the comic regarding: (i) if too little / too much information are presented in a single layout; (ii) if the comic simply does not make sense; (iii) if misleading visual cues were used which may cause confusion. We ensure that participants can design data comics independently at the end of this session.
Session 4: Selecting and creating KG stories. The three KG researchers were instructed to independently select a KG segment based on their judgement of content importance. The segment should contain at least 3 entities and should be interconnected.
Session 5: storyboard prototyping. For the selected KG segment, participants were instructed to independently design and construct comics using miro tool 3 . After all the participants finished prototyping. Participants were allowed to polish / or create an alternative comic about the same KG segment; however, only one comic was submitted per person for presentation during session 6.
To reflect on the design, we organised a discussion meeting for each participant to present their comic design and explain their design decisions.
During presentations, in addition to explaining design decisions according to guidelines G1-5, more in-depth questions were asked to further our understanding about the KG comic design process: (i) rationale behind the selection of KG segment with respect to narration and whether entities should be included / removed to improve the narration; (ii) whether it is possible to automatically translate KG relation into content relation and comic layout (e.g., 'a Yanyuwa vocabulary from a song' may be translated to 'space-annotations' content relation, and be visually represented using Annotated layout with vocabulary in the peripheral and song in the middle panel); and (iii) the benefit / limitations of the use of multiple modalities (e.g., are certain modalities preferred over the other).
We present a summative view of KG containing all the representative entities of different types in Figure 1. The descriptive details of all entities (i.e., "Node (Granularity)" column), their properties ("Properties of Node"), possible relations with adjacent nodes ("Possible relations of a Node") and semantic category ("Possible Cattegory") extracted using TF-IDF (see Section 3.2), note that since atomic node (i.e., Granularity = 2) is a single atomic unit, they are automatically assigned a category based on their connected section node. The central of Figure 1 are all the document type node (Granularity = 0), their immediately connected nodes are section type node (Granularity = 1), the node at the outer circle are atomic nodes (Granularity = 2). Logically, a document node can have many section node, and a section node can have many atomic node. Note that we have not exhaustively listed all the nodes and all the connections as it would be too much information to display / understand clearly, and instead, we opt for displaying the key representative node and their main structural information (i.e., the document-section-atomic structure). Upon analysis, we observed following advantages of this graphical way of organising knowledge: (i) Traceability. The hierarchical structure of document-section-atomic enables encoded knowledge to be efficiently traceable to the original source, e.g., when we aim to know about the events that occurred at a particular location, we begin by identifying the location node and then trace back through the 'Located at' or 'Happened at' relations to explore all the events linked to that place. Subsequently, we can further trace back through the event relations to uncover additional knowledge about these events and the source documents, thereby facilitating decision-making about the use case of the knowledge, for instance, the community may prefer including stories originated / authored by people of Yanyuwa background. Besides, any extensive use of the KG knowledge (e.g., in a language learning material) can be easily referred back to the KG entity via retrieval from KG (i.e., querying the knowledge name), thereby ensuring the authenticity of knowledge and enhancing accountability. (ii) Alignment between different modalities. We found that, by organising knowledge of different modality in an inter-connected manner, the knowledge of different modalities (i.e., audio, video, images, text) may be aligned semantically. For example, when identifying key atomic entities from video animations of background stories of 'Gulf Country Song', similar atomic entities can often be extracted from the original song text appears in the book, thereby established a one-to-one connection between song lyric and frames of video. In more complicated cases, an image about Hammerhead shark appear in encyclopedia book can also be associated with a story in the storybook about the shark with key vocabulary terms explained in the dictionary. (iii) Knowledge association and connection. By qualitatively analysing nodes, we are able to infer latent relations between nodes which further enhance the semantics of the encoded knowledge. For example, when we examine the cooking Turtle series, we can deduce that some of the hunting or cooking tools might be potentially associated with this particular event and then we can try to explore these potential connections to broadening the scope of knowledge encompassed by it. This is further exemplified and detailed in our data comic result, where the multi-modal comic elements may appear from multiple sources, see Figure 4. While KG constructed in this way has clearly demonstrated advantages in terms of content narration, we note that the current version of KG are constructed by data scientist and have not yet involved Yanyuwa community member to validate its correctness and comprehensiveness, which is planned to be the next step of our study. Additionally, we also find that as KG scales (i.e., by adding more entity-relation-entity triple), the validation of KG quality (e.g., certain connection such as "related" may not be specific enough) becomes more difficult. Indicating a need for exploring automated KG validation approaches.
In preparation to prototyping data comics, we have conducted storytelling workshop session 1-3, from which the three KG researchers gained: (i) introductory knowledge about comics [session 1]; (ii) the goals of what and how comic should convey key KG insights to the audience based on a set of explicit guidelines, see Section 3.3 [session 2]; (iii) practical experience in comic design and drawing via sketching [session 3]. We hereby continue to reporting the results in session 4-5 about the selection and narration decision of a selected segment of KG, and how they mapped to comic design based on the pre-determined guidelines. See Table 3, the KG segments selected by the three participants are reported in column "Selected KG segments", the narration process is reported in column "Narration rationale" and the chosen data comic relation (a.k.a "content relation" as termed in the original data comic paper) is reported in "Comic relation".
An overview of the decision justification in this study includes the relations that can be translated from content to comic, an overview of the selected KG segment, the narration process, the use of multiple modalities and the corresponding participant Overall, we observed that participants generally selected different contexts for narration including: turtle cooking [P1], song [P2], and shark encyclopedia [P3] respectively. The contexts selection was mainly based on: (i) the availability of imagery [P1, P2, P3] and video / audio [P2] modalities for engagement purposes (G1); and (ii) perceived environmental and social significance (G4). Though not explicitly, all participants followed the rule of proximity and continuity of knowledge (G2) and opted for knowledge within a single context. We particularly draw attention to P3, where the map node were extracted from Encyclopedia to complement the Hammerhead shark story in Ancestral dreaming text, which enriched the story with a dynamic location context (i.e., the travels over tag in the comic Figure 4). All participants emphasized the importance of linguistic knowledge in their respective narrative (G3), and subsequently selected atomic / section nodes from Yanyuwa dictionary to highlight the: (i) part of speech (e.g., noun, verb) [P1, P2, P3]; (ii) English definition [P1, P2, P3]; and (iii) Synonym [P2];
In this section, we report three final comic design results obtained at Session 6 for three selected KG segment in Table 3, respectively, as shown in Figure 2, Figure 3 and Figure 4. Note that the comic is displayed in the left, while a visualised KG selection is displayed to the right for traceability purposes (G4). On average, each comic displays 4-8 panels. We highlight key observations in following aspects: (i) Narrative complexity. We observed a clear difference in level of complexities in narrating on different selections of KG segment, which have been highlighted during the comic presentation time. Specifically, all participants agreed that including multiple modalities does not necessarily lead to narration complexity (as in Figure 1), and, P1 (Figure 2) is the least complex while P3 (Figure 4) is the most complex. We posit that this is related to the KG segment selection as guided by the proximity and continuity of knowledge (G2). P2, for example noted that "if removing song translations and vocabulary panels, the comic would be cleaner and easier to understand" when questioned about the inclusion / exclusion of the nodes. (ii) Content relation and translation. Generally, all participants agreed that it is possible to extract a set of generalisable rules for translating KG relation into content relation, for instance, KG nodes that are tagged with sequential KG relations (e.g., "followed by") could be mapped to temporal or spatial content relations i.e., Figure 3. Similarly, relations such as authored by, composed by may be mapped to narrative. However, given that 1 KG relation may be mapped to multiple content relation, and 1 content relation may be mapped to multiple comic relation, this indicated that more constraints needs to be placed to facilitate automated KG to comic translation. (iii) Composite layouts. Apart from P1, both P2 and P3 contains multiple layouts, i.e., P2 includes "grouped" (overall) and "tiled" layouts, while P3 includes "Large panel" and "tiled". However, a trade-off between clarity in overall visual stories (G1) and the sense-making of individual knowledge panels (G2) may need to be made. As P2 noted that "I wish to add important Yanyuwa English translations as annotations, and potentially the more detailed definition as a new branch using branching layout, but it becomes overly complicated." So, while adding translations and additional vocabulary may help with understanding the details of the song, the attention may be drifting away from the song to the branching panels, it is for this reason that the vocabularies nodes are ultimately added as a small tile so that the main attention is within the song content. A possible solution to mitigate this challenge is via interactive data comics [46], i.e., the element may dynamically appear only if user's mouse hovers over.
In this paper, we have pioneered a study combining the power of KG with storytelling, and situated this study in the important domain of indigenous language revitalisation. We have investigated the use of data comic to generate visual insight about a Yanyuwa knowledge graph. We conducted design sessions and reported our design process and final comic design.
The main implications of this study is threefold. First, given the unique advantages of knowledge association in the KG, this may enable rapid processing of narration which translate the lowlevel KG relation into more meaningful high-level content relation (as demonstrated in Table 3). For instance, we may extract all the KG segments that are relevant to key knowledge priorities as determined by Yanyuwa community, and use that as a basis to co-design language learning materials with Yanyuwa language educators. The traceability of KG means that we may adopt an iterative approach which may alter the narration past the initial stages (e.g., using a different node in the KG instead), and revise based on feedback obtained from different community members / knowledge owners. Second, we also find that all the KG relations are translatable into content relations (i.e., temporal and narrative). Although, such translation may not be one-to-one, e.g., the hammerhead shark series [P3] may also be identified as granular content relation between the map overview, detailed map location, and appearance of hammerhead shark. This implies that, we could potentially adopt a new approach to construct KG, by designing a subset of more fine-grained content relations to connect entities in KG. Therefore, KG is connected using content relation directly and may be more directly mapped into a comic layout to support rapid and automated prototyping of comics. As part of the future work, we may, for example, display automatically-generated comics that are highly ranked to be of high quality to other non-technical stakeholders, while revising the low-quality comics, rather than creating comics from scratch. Third, we noted a potential trade-off between understanding global (the main knowledge conveyed by the comic, such as the "Terns at Yinijini" song content) and local knowledge (the explanatory knowledge to facilitate the understanding of the main knowledge, such as the vocabulary explanation), while the more information added can enhance local information understanding, the global attention may be shifted to undesirable panels. This implies that, comics displayed may need to consider reader's prior knowledge, so as to avoid display excessive panels, e.g., displaying related knowledge panels for all the vocabularies which may lead to information overload.
We acknowledge following limitations. First, our KG, though being carefully constructed and validated within the internal team, has not undergone validation by the Yanyuwa community. The main reasoning is due to the critically endangered status of the language, and of the few people who can still speak and understand the language (including John Bradley who is the paper co-author, and two elderly Yanyuwa women), they lack the technical expertise in knowledge graph. We hope the adoption of data comics can facilitate this process, and help us communicate the KG with the non-technical stakeholders to establish a meaningful collaboration. Second, due to copyright concerns, we have only included in our KG the data collected by John Bradley (coauthor of the paper) over a 43 period of time, we acknowledge that there are other sources which may contain meaningful Yanyuwa knowledge data such as governmental digital repositories including The Australian Institute of Aboriginal and Torres Strait Islander Studies. Our KG is evolving and we will add new knowledge entities on a continual basis.
This research was supported by Google 2023 Award for Inclusion funding and Monash Data Future Institute (MDFI) 2023 seed funding grant.