This paper presents WarMemoirSampo, a portal that provides semantic search and navigation of video interviews with Finnish World War II veterans. The portal associates video fragments with contextual data extracted from the video transcriptions, enabling users to find suitable video segments via faceted search and highlighting relevant content in the video being watched. This is carried out by processing natural language texts in order to extract named entities, keywords and lemmas. The result is a Linked Data Knowledge Graph that underpins the portal. We describe the collaboration between Natural Language Processing and Semantic Web technologies used in order to produce these results.
in order to produce a harmonized knowledge graph (KG) – including named entities, keywords and lemmas – with timestamp information. This enables the videos to be accessed at diferent points according to their semantic content, and additional contextual information to be provided while the video is being watched. In order to achieve this, we created a KG containing enriched data for the interviews as well as information about the interviewees. The data enrichment and lemmatization is carried out using Natural Language Processing (NLP) techniques and knowledge extraction, resulting in new metadata about keywords and mentioned named entities – e.g., people, places, and events. These are integrated into the KG as entities and properties. The data is then stored in a SPARQL endpoint for open access.
The interview videos were edited by adding titles and copyright notes in the beginning and by merging multi-part interviews. They were then published on the YouTube platform for streaming, as well as archived in a repository at the National Archives for later use. The WarMemoirSampo Portal2 was published on December 3, 2021, at the National Archives of Finland.
This system is related to previously published works of knowledge extraction from text to
linked data [
The paper is organized as follows: first, the knowledge extraction and transformation pipeline for the underlying data service is explained. After this, functionalities of the portal application on top of the data service are briefly delineated. In conclusion, the main results of the work are discussed and directions for future research outlined.
The interviews were carried out in Finnish, which presents additional dificulties for natural language processing in comparison to better-researched European languages: its rich morphology4 complicates lemmatization and Named Entity Recognition (NER). Moreover, Finnish NLP models are not as varied as for example those for English or French, and since the interviews were realized in colloquial, dialectal Finnish, not even state-of-the-art speech-to-text algorithms proved robust enough to handle them: at this point in time, only the standard variant of the language (known as yleiskieli) leads to acceptable results in speech-to-text tasks.
As an alternative, rough transcriptions made post-hoc by the interviewers were used for each of the 159 interviews. These vary widely in length and level of detail, with an average character count of 5599 ± 2458, which corresponds to a correlation of 0.718 with the interview lengths. Due to their nature, these notes are ultimately not meant to be used as a source of information 2The portal is in use at: https://sotamuistot.arkisto.fi . 3www.media.mit.edu/hypersoap/ 4Finnish contains around 15 cases and various sufixes, which result in a number of surface forms that may reach well over 2000 for a single word. For example, the webpage http://www.ling.helsinki.fi/~fkarlsso/genkau2.html lists 2253 automatically generated forms for the word kauppa ’shop’ for others than the interviewers themselves: they do not provide full-fledged texts – or even sentences necessarily – and contain abbreviations as well as orthographic and grammatical errors, which compounds the challenge posed by the Finnish language. Nevertheless, we found the notes mostly adequate for the purposes of this project, although we have not thoroughly assessed their accuracy.
The interviewers’ notes were provided as a spreadsheet document containing also other types of metadata, such as interviewees’ name, date and place of birth, length and place of the interviews, as well as links for the interview videos uploaded to the YouTube platform. These notes are divided into rows, containing each one roughly a sentence. Each row also has a timestamp related to their location in the YouTube video. However, these timestamps are not unique: usually several consecutive rows have the same timestamp. These rows were grouped together, resulting in video segments of variable length but typically several minutes long, which became our data unit for this project.
An overview of the subsequent data processing and transformation process is shown in Figure 1. The original spreadsheet – containing one tab per interview, as well as metadata for all of them – is initially split into multiple CSV files. Each interview is then divided into shorter segments, as explained above. Three diferent procedures are carried out: the notes related to these segments are lemmatized in order to facilitate searching and other tasks; their keywords are generated; and their named entities are extracted and linked. The result of the data transformation process5 are RDF files, which contain the KG that portrays all the diferent entities found in the data, their relations and properties.
The lemmatization of the texts is carried out by our Secompling library6, while keywords are
obtained via Annif [
Lemmatization and NER with Secompling. Secompling is a library that aims at
combining various Finnish NLP tools, including third-party ones, in order to provide an easy and
integrated interface for methods such as lemmatization, NER, relevance feedback and
unsupervised classification [
The results of the NER tool and the parser are then aligned. Since the former does not provide
lemmatization, this step provides the means for obtaining basic word forms for the named
entities. Moreover, this procedure helps to fix errors in both tools, such as combining tokens
that should be split, or assigning diferent named entity classes to parts of the same entity. The
results of a brief examination with 100 random entities indicate that the Neural parser achieved
a lemmatization accuracy of 0.84, while Secompling raised it to 0.96 by also applying named
entity-specific heuristics [
Only a part of the eighteen entity categories recognized by the NER tool were judged to be of interest in WarMemoirSampo: Person, Product, Organization, Event, and a combination of Location (LOC), Facilities (FAC) and Geopolitical entities (GPE) as Place. Warsa-linkers adds Military Units to this list.
Since at this point Secompling-NER does not perform entity linking, it does not have the means to tell an accurate named entity from an error in either recognition or lemmatization. As a temporary solution, manual correction is applied to the candidates, so that it is possible to remove an entity, or edit its category or lemma. As a result, a total of 336 entities were ignored and around 270 had their category and/or lemma changed.
NEL Using Nelli. The named entity linking tool Nelli [
In the interviews, the veterans mentioned some of their comrades in arms and superiors. Correctly identifying the most notable oficers is often easy; disambiguating regular soldiers is not. The latter were often mentioned only by first name and/or surname, so that more information would have been necessary to diferentiate them. Hence, the linking strategy centered around notable figures present in Wikidata.
Regarding place linkage, several Wikidata ARPA configurations are used to link places such as continents, countries, cities, as well as smaller places such as towns and villages. It can be challenging to match former Finnish place names to Wikidata due to varying practices in classifying place entities (e.g., towns or urban settlements) or missing labels, e.g., Enso or Viipuri. Moreover, places that were annexed to Russia often changed their names from Finnish to Russian, and some place names in Wikidata were lacking their original Finnish names mentioned in the interviews.
Warsa-linkers. WarSampo contains a data and ontology infrastructure for Finland in World War II. The WarSampo KG consists of around 100 000 persons, 51 000 places, 16 000 military units, 166 000 photographs, 26 000 war diaries, and 3000 war veteran memoir articles, among others, with large amounts of links between entities.
The data processing infrastructure of WarSampo contains a NEL process for linking
descriptions of events and photographs to persons, places, and military units mentioned in texts [
Entity Reconciliation. The generated named entities from diferent steps are disambiguated between the diferent data sources based on 1) matching the sets of URIs received for them from LOD data sources, and 2) matching entity types and names. WarSampo and Wikidata often contain alternative labels which are helpful in entity matching. In Wikidata, it is also possible to manually add missing labels in order to improve entity matching. After matching, the entities are reconciled by merging the duplicates found. The reconciled named entities consist of 310 persons, 1840 places, 66 events, 51 products, 117 military units, and 610 other organizations. These include linked entities as well as those unlinked which were deemed important, which also receive their own reference page (as explained in section 4).
We evaluated the outcome of the entity recognition pipeline by inspecting twenty random
interview segments containing a total of 99 recognized entities [
The WarMemoirSampo KG, which is the result of the data transformation pipeline, is contained in RDF files and served via a SPARQL endpoint 9. It is the only point of contact between the system’s NLP backend and its user interface. It contains 323 371 triples, each one describing an entity or linking it to another. The main classes are (with the : namespace referring to http://ldf.fi/schema/warmemoirsampo/ ).
• :Interview (159 instances), corresponding to one video interview; • :PersonRecord (159 instances), which collect the interviewee’s personal information, such as given name, family name, gender and date of birth; • :TimeSlice (2417): a video segment from start timestamp to end timestamp. An :Interview is divided into :TimeSlices, as explained in section 2; • :NamedEntity (2994): Named Entity instances. Each named entity type also receives its own subclass; • skos:Concept (3127): keywords as provided by Annif. They are instances of another KG, the General Finnish Ontology (YSO);
Both Interviews and TimeSlices are indexed with named entities and keywords. The named entities and keywords contain links to the same resources in other information sources.
In order to provide an intuitive and user-friendly access to the WarMemoirSampo KG, a new
web-based user interface had to be created. To avoid starting from scratch, the Sampo-UI
JavaScript framework [
The structure of the WarMemoirSampo Portal11 is based on the three main classes in the
KG: :Interview, :TimeSlice, and :NamedEntity. Guided by principle 4. of the Sampo model [
Word cloud Embedded YouTube
video player General interview
metadata All automatically
recognized named entities 3. The Index perspective includes faceted search and browsing functionalities for the nearly 3000 named entities mentioned in the summary notes of the interviews. For each named entity, the search results include links to specific part(s) of the interview where this entity is mentioned.
Each search result in the aforementioned faceted search perspectives acts as link to an instance page, which aggregates information about the particular instance (e.g. an interview or a place) and ofers links to related instances in WarSampo and Wikidata. These pages ofer the end-user the possibility to learn more about e.g. the military units, persons or locations mentioned.
For the purposes of interactive viewing of interview videos, the Sampo-UI framework’s instance page component was extended for embedded videos with dynamic table of contents functionalities using the YouTube IFrame player API12. Figure 2 shows an instance page of an interview. On the top left there is an embedded YouTube video player, while on the bottom left there is metadata related to the interview. On the right there is a dynamic table of contents (TOC) of the interview, which can be used to navigate to a specific segment of the interview video. The time position of the YouTube video is synchronized with the TOC, so that the active segment is expanded automatically. The expanded TOC part contains the interviewer notes as well as the recognized named entities related to the current part of the interview, which are also links to instance pages.
12https://developers.google.com/youtube/iframe_api_reference
WarMemoirSampo is the outcome of a collaboration between Linked Open Data and NLP techniques. Its backend capitalizes on various language-processing tools in order to handle texts and extract information from them, while the result of this process, a linked data knowledge graph, is able to store the information and swiftly deliver relevant parts to a faceted search-based user interface. The resulting WarMemoirSampo Portal ofers the general public as well as specialists a way to grasp the contents of the interviews through keywords and named entities and the means to easily search and access relevant parts of the videos.
The portal is also provided with a feedback button so that the developers can gain insights based on user experience, since the public did not have access to it during development. Nevertheless, more features are planned for the future: the links to the entities will be integrated into the text instead of being listed separately. An event detection tool is to be developed which extracts event entities based on times and places mentioned in the interviews. Moreover, when Finnish speech-to-text technology advances to the point that everyday dialectal speech can be reliably transcribed, the same tools could be used on these new transcriptions.
Acknowledgements Ilpo Murtovaara provided the original transcripts of the videos and was influential in filming the videos. Kare Salonvaara edited the videos. Tammenlehvän Perinneliitto ry funded our work. CSC – IT Center for Science provided computational resources.