The Virtual Record Treasury of Ireland (VRTI) 1 [1][2][3][4] is the result of a seven-year programme of State-funded research hosted at Trinity College Dublin. The VRTI digitally recreates the archival collections which were lost in the destruction of the Public Record Office of Ireland (PROI) at the opening of the Irish Civil War in June 1922 [1][2][3]. The PROI was Ireland's state archive, which was located within the Four Courts complex in Dublin, on the This paper presents an initial user evaluation of the VRTI-KG Editor, a web application designed to enable subject-matter experts, to access and refine pertinent data (including geospatial information) within a KG. The editor was designed as a configurable framework allowing the synchronization of the editing views with changes in the data model. In addition, it is hoped the configurability of the editor will allow it to be used within other KG projects. The participants in the evaluation were asked to complete a number of tasks which mimic expected user interaction by subject matter experts. A number of metrics were used to measure the user satisfaction with and user understanding of the application, together with metrics to measure accuracy and efficiency in use of the editor. The qualitative data collected through open comments was analysed to identify common patterns which influence usability. The findings of the evaluation will be used in the next phase to inform an iteration of refinement which is hoped to further improve usability of the editor.
This paper is structured as follows: Section 2 presents background information and related work relevant to the design and implementation of the VRTI-KG Editor. Section 3 provides an overview of the VRTI-KG Editor itself. Section 4 presents the structure of the user evaluation completed on the editor and findings discovered from the results. Section 5 outlines future work and concludes the paper.
The related work discussed in this section includes approaches designed to support users in exploring and investigating RDF data using a user interface, in particular visualisations.
A survey [10] in 2023 was completed of 28 tools which were designed for the visualization and exploration of KGs. The tools were tagged for each respective reference context: digital libraries or generic databases and were classified according to the type of interaction paradigm used, the type of information displayed and the strategy used to reduce the displayed information. The interfaces were categorised using four categories which include 1) relates to the interaction paradigm used, which can be a tabular visualization, node-link visualization or visual query composition 2) relates to the type of information to be displayed 3) category relates to the strategies used to reduce the displayed information: navigational visualization, incremental visualization or summarized visualization and 4) relates to the interfaces that deals with digital library contents. The survey concludes that few of the existing interfaces combine different interaction paradigms. Information reduction strategies are present in most tools and are crucial for exploring complex KGs. However, the challenge of support for users with limited relevant technical background still exists.
Sampo-UI [11] is a tool designed to allow users to create configurable GUIs for respective KGs. The tool is based on the "SAMPO" model which aims to provide diverse perspectives of RDF data stored in respective graphs. The tool includes faceted search which allows searching through data analytics represented in tabular format and visualisations including geospatial. The tool has been adopted most often by the cultural heritage domain. While the tool is used by a large number of users, several limitations were identified. For example, Sampo-UI has limited support for the geospatial mapping of polygons required to represent area boundaries. In addition, named graphs are currently not supported, which is a requirement for certain KGs, such as the VRTI-KG.
VisKonnect [12] is an approach designed to present visual connections between historical figures and potential interlinks between them. The interlinking of these figures focuses on related events such as sport tournaments, award ceremonies, or summit meetings. VisKonnect models data using the EventKG, an event-centric KG that covers relations between events and persons and allows inferring of required information. The approach provides three visualisations which include event timeline, an event map, and a relationship graph. In addition, a chat interface is used to query the data using natural language to allow users to easily answer questions. The multiple visualisations demonstrate how the can aid searching by users. In particular, the event map inspired the navigation on the geospatial maps in the VRTI-KG editor.
LodView [13] is a tool designed to allow browsing of the RDF resources stored in a dataset using SPARQL [7] queries. A HTML representation of the links of each resource is shown to users when a URI is selected. The representation includes a table which lists the properties and objects associated with the selected resource. The table allows users to follow links in the table to other related resources. The tool includes a configuration file to customize elements such as the SPARQL endpoint, namespace of resources and styling of the interface. The configuration file provided inspiration for use of one in the editor. However, LodView does not support searching for resources other than directly inputting a specific URI and editing of data.
WissKI4 (Wissenschaftliche Kommunikationsinfrastruktur) is an interface tool designed for managing scholarly KGs. The tool is integrated with an existing tool named Drupal5 which is a web content management system. It leverages the structure of scholarly ontologies, such as CIDOC CRM [6] to create the data model used by the tool to facilitate editing. WissKI's Pathbuilder supports the configuration of the tool for a specific KG. However, WissKI does not include a data creation process that completes data validation steps to prevent poor quality data being inserted into the KG. In addition, the tool requires a high-level of knowledge of the RDF ontology terms, which inhibits non-technical users. The editor we propose includes a data creation pipeline (Section 3.2) to facilitate straightforward detection of potential quality issues. In addition, the information presented on the interface (Section 3.1) is simplified to hide the underlying complexities of the RDF terms to support non-technical user's.
Approaches surveyed provided useful insights which helped to inform design decisions for the VRTI-KG editor. In particular, Sampo-UI [9] provided inspiration as it uses different search paradigms: free-text search, faceted, geospatial, and temporal similar to the proposed editor. However, it was discovered that few undertook a formal user evaluation.
The VRTI-KG Editor6 [4] was designed to allow subject matter experts with no relevant KG technical background to edit and insert data into the VRTI-KG. The user requirements for the editor were gathered by conducting several interviews with digital humanities researchers. An iterative process of prototyping was completed where the requirements were defined, validated and refined based on the feedback of the interviews. The web-based editor allows users to search the graph to find desired resources and displays the current data related to the selected resource.
The graph editor was implemented using several Python libraries 7 . Flask was used to create the web application and provides a comprehensive and customizable library for constructing web applications and allows diverse Python libraries to be used. In the editor, a number of such Python libraries are used. The SPARQLWrapper library facilitates the execution of SPARQL [7] queries on remote endpoints. Folium is used to visualize geospatial data on geographical maps. Queries are manipulated using string formatting methods available in Python. PySHACL is used to execute Shapes Constraint Language (SHACL) [14] constraints. SHACL is a W3C recommendation which defines the SHACL Shapes Constraint Language, a language for validating RDF graphs against a set of conditions. These conditions are provided as shapes and other constructs expressed in the form of an RDF graph. In this context, RDF graphs are named "shapes graphs" in SHACL and the RDF graphs that are validated against specified shapes are called "data graphs".
This section illustrates the use of the VRTI-KG editor web application through the searching and editing of a person entity resource. The selected person used for illustration is a prominent Irish writer named "Samuel Beckett"8 . The SPARQL queries created by the editor during the use case are available 9 . First, the person editing option is selected. A person can be retrieved by name, gender, ID, birth/death date, birth/death place, among other using the page shown in Figure 1.
In our example, the string "Samuel Beckett" was entered into the text area associated with the full name search filter. Thereafter, the string is inserted into a FILTER keyword condition in a SPARQL [7] query (FILTER (CONTAINS(LCASE(STR(?Name)), LCASE('Samuel')) && CONTAINS(LCASE(STR(?Name)), LCASE('Beckett'))). The name is split into multiple words to ensure that matches are found regardless of the ordering of the input. The results of the query are shown in a tabular format which allows users to easily retrieve the respective resource. In addition, the table can be searched (A -Figure 1) when there are larger query results retrieved by the initial search. As can be seen in Figure 1, only one resource is returned for the completed search in our example. Thus, the user then selects the edit button (B) for Beckett which results in redirection to the page detailing associated data related to Beckett. Another more specific query is used to retrieve all information in the graph related to Beckett including direct and inverse relations. All URIs returned from the query results are formatted in a more human readable format which involves stripping to the end of the URI. In addition, they are converted into internal links allowing users to further traverse the graph using the application itself. "Filled Values" (A -Figure 2) relate to attributes which have values already associated, while "Empty Values" (B) relate to attributes which do not contain values as yet. The "View Hidden Data" button (C) allows users to view data stored in a private experimental graph. The hidden data includes information which is currently being curated in order to improve quality for publication. The data is retrieved using a SPARQL query executed on a separate namespace graph and displayed in tabular format. The "Show Links" button relates to other associated links, such as those to other data stored in the VRTI. The links are categorised as internal (VRTI link) and external (outside VRTI). The "Copy Link" button allows users to easily copy and paste the resource URI into their desired location. The "History of Edits" button redirects to a page which displays provenance of changes which have been executed on the respective resource. The provenance includes the name of the user, time of edit, number of edits and details of the edited values. The provenance data is initially stored in a database which is uplifted using R2RML (RDB to RDF Mapping Language) [15] into a private provenance sub-graph in the VRTI-KG. Thereafter, users can select to edit the resource which results in redirection to the view presented in Figure 3. The headings shown (A -Figure 3) at the top of the page are designed to group attributes based on similarities in order to improve navigation for the user. For instance, "Places" relate to the birth and death of a person, while "Relationships" relate to family and marriages. As can be seen in the example, Beckett has a basic level of completeness (represented in blue) and additional information could be added. Once the necessary changes are completed, validated (see earlier) and submitted (B), the updates will be propagated into the VRTI-KG and in the view presented in Figure 2.
The editor includes several configuration files represented in JSON format, making it versatile for diverse KG deployments. These files are responsible for customising the editing views (see Figure 3) including the dropdowns, styling and SPARQL query creation. Listing 1 presents an extract of a configuration file for the occupation of a person in the VRTI-KG. The extract shows the occupation of a person ("Occupation"). The property ("property_uri") used to link an occupation resource to a person. The column ("column_names") of the CSV file where an occupation name is inserted when a new resource is created. The type ("type_uri") of the resource to be shown in the dropdown options when an occupation is being edited. The name of the tab ("tab_name") where occupations are shown on the editing page. The structure of the SPARQL query ("inverse_relation") and whether an occupation can be removed ("removable") from a person. Some attributes such as a gender can only be changed and not removed.
The file contains similar configurations for the attributes related to people, places, organisations and offices in the VRTI-KG. The configuration can be easily changed when the data model changes, which will automatically propagate the changes into the interface of the editor. Styling can also be changed from the configuration files. Bootstrap 10 CSS classes are used for configurable styling. For instance, the maintainer wants to change the colour of the blue buttons (Figure 1) to another colour, such as red. They would find the class for the colour red which is "danger" and replace it in the configuration which will automatically update the interface.
The previous method to ingest data into the VRTI-KG described in [3] involves the creation of resources using only CSV file upload. The graph editor method is designed to supplement that process by providing an intuitive interface to subject-matter experts, and which completes quality checks to ensure that incomplete data is not inserted into the VRTI-KG. New resources can be created and inserted into the KG using the editor interface alone. The process involves users interacting with a form on the interface. The form allows users to input respective values using various types (e.g. dropdown and free text) of input. The form includes validation which helps to prevent incorrect data being created. For instance, regular expressions are used to ensure that Wikidata links match the expected format (e.g. URI starts with respective host name). The resulting RDF graphs are compared against a set of conditions to capture any other quality issues. In addition, attributes of created resources are compared with existing resources to ensure that no duplicates exist. The steps in the creation pipeline 11 can be summarized as follows:
User Input. First, users input information into a form displayed on the application 12 . The form includes a combination of free text input and selection from restricted dropdown menus. Validation is completed on the input to help prevent incorrect data being inserted. The validation provides visual cues and popups to indicate incorrect values. The form once valid and submitted will result in the generation of temporary source data represented as relational data. The relational data is uplifted into RDF format using R2RML. The respective R2RML mapping is retrieved and executed which results in an RDF file containing mapped data from the input of users.
SHACL Validation. SHACL [14] shapes have been created for the entities created by the graph editor in order to ensure that resulting resources are sufficient quality. Graphs which do not satisfy these conditions result in debugging information being shown to users and the creation discontinued. The information shown is derived by querying (using SPARQL [7]) the SHACL quality report defined in RDF format using the SHACL Validation Report Vocabulary.
Duplicate Detection. Duplicate detection involves querying existing data in the graph to ensure that no other resources exist which represent the same entity. For instance, attributes such as first name, surname and century are compared to existing data when a person entity is created. The comparison is completed using SPARQL [7] query templates where respective values from the user's input are inserted. A listing of potential duplicates (if any found) is presented to the users. The listing includes other details (such as the associated dates) related to each duplicate to allow them to filter the results. In addition, each duplicate contains an internal link to allow them to instead edit a discovered duplicate.
Finally, the generated RDF data is inserted into the VRTI-KG when the SHACL shapes conform to the data graph and the users confirm that no duplicates exist. A SPARQL Load query is used to insert the generated data into the graph. Thereafter, the updates are propagated into the view of the interface to allow users to search and edit the created resource(s).
The hypothesis for this study was: The application provides a sufficient level of understanding, accuracy, efficiency and satisfaction during the experiment.
A user evaluation 13 was conducted to investigate the hypothesis. The participants were provided with a list of tasks to complete using the editor. Then, they were asked to fill out questionnaires detailing their user experience during the experiment. In addition, separate namespaces were setup in the Blazegraph triplestore 14 to allow each participant to independently edit a copy of the VRTI-KG. Each namespace was analyzed after the study to calculate the accuracy of each participant's edits. The following four instruments were used during the experiment to measure user's interaction with the editor.
Post Study Usability Questionnaire (PSSUQ). The PSSUQ [9] is a standardized questionnaire, which was designed by IBM in 1995 to determine the level of satisfaction provided by a software system. The questionnaire includes a Likert scale from 1 (best case) to 7 (worst case) to rate the level of satisfaction. In addition, an open-comment section is included with each question. The questionnaire includes four sub-scales which relate to different aspects of the interaction. The four sub-scales are system usefulness, information quality, interface quality and overall. It was decided to use the PSSUQ rather than the System Usability Scale (SUS) [16] due to the extensive psychometric evaluation that has completed with it. In addition, it includes open-comment sections to capture diverse feedback, which allows participants to elaborate on their user experience.
Understanding Questionnaire. The understanding questionnaire15 was designed to measure the user's understanding of information provided by the editor during the course of the experiment. The questions included a combination of free text and multiple choice. Each question was accompanied by an optional open comment section to allow participants to elaborate further on their answer. The questions asked for information related to different search methods which included entering the URI of a resource (Q1-2) and selecting a search filter and entering an accompanying value (Q3-7) For instance, task 1 asked participants to complete a search using the URI of a person and to examine the search results. Then, they were asked to provide the birth date of the person in Q1.
Time of Completion. The participants were asked to self-report the total time it took them to complete the experiment. The recorded times were used to measure the efficiency of the interaction.
Accuracy of Edits made. SPARQL [7] ASK queries16 were posed to each participant's associated namespace graph. Each query embodied the change to the graph that would be expected as a result of the correct execution of a task. The ASK queries were used to return a Boolean (true or false) indicating if each edit was completed successfully or not.
The experiment was approved by the TCD Research Ethics Committee. This section discusses participants and tasks they were asked to complete.
Participants. The participants consisted of 9 subject-matter expert historians. The participants were recruited based on an internal meeting with the lead VRTI historian which discussed who satisfied the inclusion/exclusion criteria. Participants signed an informed consent form. The experiment was completed asynchronously with each participant provided with login details and a link to the task sheet and questionnaires. Thereafter, participants could access the editor online whenever they decided to complete the experiment. Participants were not provided with any assistance during the completion of the experiment. In addition, they were not provided with any training for the application prior to the experiment.
Task Sheet. A focus group was conducted with researchers in computer science and digital humanities to determine what tasks should be completed during the experiment. The resulting task sheet17 included 8 tasks with the final task consisting of 6 sub-tasks. The tasks were designed to mimic expected user interaction of historians with the editor, including search, edit and creation of resources in the VRTI-KG. Tasks 1-2 involved searching for people using a provided URI. Tasks 3-5 involved searching for people using different search filters, such as their area of interest. Tasks 6 and 7 involved changing and adding attributes to an existing person. Task 8 included sub-tasks which involved creating a new person with several attributes, including name, occupation and associated dates and places.
Figure 4 presents a boxplot of the results of the PSSUQ questions and four sub-scales (Section 4.1), which was used to measure user satisfaction with the editor.
The rectangle of a boxplot represents 50% of the data points. The position of the line in the rectangle represents the distribution of the data points. The line indicates if the data is normally distributed (centre), positive skew (left) and negative skew (right). Outliers are represented by the points outside of the rectangle. All sub-scales scored between 11.3% and 25.2% better than their respective "acceptable thresholds" [9], which indicates overall sufficient satisfaction with the application by users. System usefulness (SysUse) scored best with 25.2% better than the normal PSSUQ threshold. Interface quality (IntQual) an Overall scored similar to their thresholds with 14% and 17.9%, respectively. Information quality (InfoQual) scored worst with only 11.3% better than its threshold, which indicates the information provided by the editor should be improved. An example comment which relate to information quality: "Maybe more hover info would be useful for people less familiar with the KG.". Adding more descriptive names to buttons and hover text, and softening technical terminology are hoped to improve the information quality score in the next iteration of the application. In addition, limiting the initial information to prevent overwhelming the users could improve the information quality score, as a participant stated "I think the page for editing an entity has too much when you first land on it". However, comments such as "clean, clear, uncluttered, and no ambiguity", "I found the system to be very easy to use and navigate" and "Yes, the design is clean and consistent" are supportive of an overall conclusion of sufficient user satisfaction.
Figure 5 presents a bar chart of the scores of how many participants answered correctly each question in the understanding questionnaire, used to measure the understanding of each user of the information being presented on the editor interface. 7 questions were included in this questionnaire.
Mean score for all questions was 94%. 4 out of 7 questions scored 100% correct, which indicates overall sufficient understanding of the information provided by the editor to users. In addition, comments such as "Navigation makes sense and I could find what I needed", "Yes -the design and text fields provided made it easy to navigate and find the information I needed" and "very clear in the novel search" support this observation. The worst scoring questions (Q2 and Q5) related to the occupation of a person shown in the search results table. 2 participants provided the area of interest rather than occupation, which is the column beside in the results table. In addition, 2 participants when answering Q5 provided the URI for the person rather than the named graph as stated in the question. Both are located near each other, which may have caused confusion.
Figure 6 presents a bar chart of the accuracy (number of correct edits) determined by executing the ASK queries (see section 4.2) upon each participant's namespace graph which resulted from the edits. Task 6 and 7 asked the participants to edit an existing person, while Task 8 asked them to create a new person with distinct attributes (subtasks a-h).
The mean score for all of the edit tasks was 7 (77%) correct edits, indicating that the accuracy of edits by participants was overall sufficient. The edits of existing people (T6 and T7) scored best with a mean of 8 (88%), while the creation of a new person scored worse. However, T8a-T8e scored similar with a mean of 8 (88%) correct edits. These sub-tasks asked participants to add associated names, dates and gender of a person. T8(f)-T8(h) scored worse with a mean of 6 (66%). These sub-tasks asked them to add the occupation, birth and death place. A participant stated that there was no option to insert these attributes and included a comment, "During the last task there was no option to enter the final three pieces of information given.". The inputs for these attributes were only visible when the user selected the respective tab on the page. The task sheet stated, "by navigating through the tabs at the top of the page", however, the task could have explicitly stated which tabs to use for each attribute. 2 participants submitted their edits to the graph before adding all attributes and the second attempt was blocked as the editor does not allow a URI to be created twice (unless deleted first using the editor). A participant stated "I accidentally created the record before I had completed the instructions, and then could not add the final piece of data as I got a 'Duplicate entity' error message.". This issue could be resolved in future iterations of the editor by adding an explicit warning message to state the URI already exists in the KG and allowing them to recreate in exceptional cases.
Figure 7 presents a boxplot of the overall times for completion of the experiment (self reported by each participant), used as a proxy measure efficiency of user interaction with the editor. The mean time for completion of the experiment was 18 minutes. The minimum time was 10 minutes and maximum time was 25 minutes. The standard deviation was 4.8 which indicates the times were spread around the mean and that efficiency of interaction was not equal for all participants. Some of the Comments which supported sufficient efficiency include, "It was straightforward to learn" and "It was pretty efficient". The outliers could be as a result of certain participants investigating other aspects of the interface which were not directly involved in the experiment. Furthermore, some participants may have had previous experience using semantic web technologies which resulted in improved efficiency. Moreover, the amount of buttons could have decreased efficiency for certain participants with a participant stating, "The pages are a bit overwhelming with button options". The Spearman's Rank-Order coefficient test [17] was used to measure the correlation between efficiency and the other metrics. Spearman's is a nonparametric test designed to measure correlation between variables and is less sensitive to outliers compared to similar tests. A p-value of 0.05 was applied to the test to indicate a statistically signification correlation. The p-value for efficiency and PSSUQ is -0.033 which is not statistically significant. However, the p-value for time and accuracy is -0.757, which is statistically significant. This value indicates as time decreases, accuracy increases. Thus, participants who completed the experiment faster had better accuracy. This correlation indicates that participants who struggled with the interaction of the application completed worse edits.
The following key findings have been drawn from the analysis of the experiment results.
Too many buttons. 2 of the participants stated that the information on pages was overwhelming. Comments such as "The pages are a bit overwhelming with button options" and "I think the page for editing an entity has too much when you first land on it" related to this aspect. While the inclusion of additional buttons was hoped to improve functionality, it resulted in confusion during general interaction. It was decided to review buttons and add groupings of existing buttons into dropdown menus in hopes of not overwhelming users.
Re-creation of Resources. 2 participants submitted their created person to the graph before adding all stated attributes with one stating "I accidentally created the record before I had completed the instructions, and then could not add the final piece of data as I got a 'Duplicate entity' error message.". 1 of these participants was the same as mentioned in the last point, stating the page was overwhelming. As this could happen in future, we will consider providing an explicit warning and allowing them to recreate resources in exceptional cases. However, users with more experience would be likely to know that they can access the edit page for the created resource and make changes on that page.
Improvement on existing approach. The previous method to view resources in the VRTI-KG involved dereferencing specific URIs using LodView [13] (Section 2). A participant stated "A big improvement on the LodView" which indicates the editor is possibly an improvement upon LodView for viewing data for the VRTI use case.
Improvement with Experience. 5 of the participants stated that they feel with more experience and guidance the application would become straightforward to use with them stating "Lots of options, easy to navigate, but will be more comfortable once used more frequently.", "I think that some direction would be needed to train users initially but I can easily see myself becoming proficient using this system", "More hand-holding might be useful for initial use" and "The editor looks good and will be straightforward to use with guidance and greater familiarity.".
In summary, sufficient satisfaction was observed as the scores of each sub-scale of the PSSUQ scored between 11.3% and 25.2% better than their acceptable research thresholds [9]. Sufficient user understanding was supported with a mean of 94% correct answers for all questions. Sufficient efficiency of interaction with the editor was supported as the mean time was 18 minutes and all participants completed all tasks between 10 and 25 minutes, which involved multiple searches, edits and creation of a new person. Sufficient accuracy of the edits was supported as the mean was 77% correct edits. In addition, Spearman's correlation test [17] indicates that as efficiency improves so does accuracy. It is expected that the improvements outlined in Section 4.4.1 could improve these measurements in future versions of the editor. Publishing the implementation will allow other research projects with KGs to configure and use it to search and edit their data. Finally, it is hoped that the evaluation methodology can provide useful insights to researchers when validating similar approaches.
Virtual Record Treasury of Ireland (VRTI) is funded by the Government of Ireland, through the Department of Tourism, Culture, Arts, Gaeltacht, Sport and Media, under the Project Ireland 2040 framework. The project is also partially supported by the ADAPT Centre for Digital Content Technology under the SFI Research Centres Programme (Grant 13/RC/2106_P2).