In the past two decades, the trend towards evidence-based public policy has catalysed a significant shift in the social sciences, emphasising impact evaluation. Drawing on methodologies from Randomised Controlled Trials (RCTs) in medical research, social scientists and policymakers have embedded evaluation mechanisms into interventionist policies to assess their effectiveness. This research approach has yielded important insights, particularly for public policy in lower-income countries. For instance, studies have shown that childhood exposure to cash transfer programs with conditions tied to health and education can lead to improved educational, mobility, and labour market outcomes in adulthood [1]. Additionally, the duration of exposure to these programs has been linked to increased long-term consumption [2].
Recently, there has been a growing emphasis on meta-analysis, with researchers seeking to extract broader policy lessons from a deepening pool of evidence [3], [4]. Efforts have been made to create systematic review frameworks to support specific policy areas and address external validity concerns that may arise from conclusions based on single evaluations. Traditional meta-analyses have included both qualitative desk studies that synthesise findings from multiple studies [5], [6] and quantitative approaches that aggregate treatment effects to evaluate the effectiveness of interventions in a particular domain [3], [7].
InterDev, an online knowledge discovery platform, was developed to support this growing need for systematic review frameworks by enabling users to find, discover, and reuse data from evaluations, to make accessing such data Findable, Accessible, Interoperable and Reusable (FAIR) [8]. It builds on previous work on ontology development by providing an interface that allows for the curation of data according to the ERCT ontology framework [9], without the need for knowledge graph expertise that may not be within the remit of non-technical researchers [10]. This follows research which has similarly adapted knowledge graph data for non-technical researchers in health research [11] In this paper, we focus on the user evaluation of InterDev to understand its effectiveness and usability, presenting the first set of results from a planned three round evaluation. Participants were assigned 10 tasks and their task completion times, the number of interventions required, and verbal processes via the think-aloud protocol were recorded by the author and later transcribed. They also completed the Post-Study System Usability Questionnaire (PSSUQ). Through thematic analysis of open-ended responses and recordings, and quantitative analysis of the PSSUQ, we found that users generally navigate the platform well but highlighted the need for additional functionalities, such as enhanced features and improved search capabilities, to maximise its utility.
This paper is structured as follows: Section 2 describes the implementation and methodology of InterDev, detailing data collection, semantic uplift, data presentation, and usability evaluation. Section 3 describes the technical architecture and data integration processes. Section 4 presents the evaluation, including both quantitative results, such as task completion times and PSSUQ scores, and qualitative results from thematic analysis of user feedback. Finally, Section 5 concludes with a summary of findings, discussing strengths and areas for improvement, and outlining future development directions for InterDev.
The methodology of InterDev can be defined in five key stages, illustrated in figure 1. Step 1: Data Collection. In the first phase, evaluation data from various development data sources are gathered along with contextual data from multiple repositories. This comprehensive data collection provides a rich dataset that enables the platform's functionality. Any source of development data where data is structured as evaluations can be integrated.
Step 2: Semantic Uplift. The second phase involves the semantic uplift of collected data, where the data is structured according to the ERCT Ontology [9] using tools such as RDFLib [12], allowing for the expression and combination of the underlying data as RDF [13]. This process converts CSV data into RDF (Resource Description Framework) format, facilitating the creation of the InterDev Knowledge Graph (KG). The semantic uplift ensures that data is not only standardized but also enriched with semantic meaning, enhancing the platform's ability to support sophisticated queries and data integration, improving the discoverability and usability of the information.
Step 3: Data Presentation and Curation. The third phase focuses on the presentation and curation of data within the InterDev user interface (UI). The platform offers various views, such as Evidence View, Collection View, Submission View, and Evaluation Filters, to help users navigate and interact with the data effectively. This phase is important for transforming raw data into a user-friendly format, enabling users to access, explore, and curate the information they need efficiently, particularly for non-technical researchers unfamiliar with semantic web technology [14].
Step 4: Data Export. In the fourth phase, users can export curated data collections in .ttl (Turtle) format. This capability allows users to download and utilize the data outside the platform, facilitating broader dissemination and application of the knowledge discovered through InterDev. Data export is a vital feature for researchers who need to incorporate the data into their analyses or share it with collaborators.
Phase 5: Usability Evaluation. The final phase involves a thorough usability evaluation, consisting of user experiments, refinements based on feedback, re-evaluations, and eventual delivery of the improved platform. These evaluations consist of multiple metrics and formats, such as PSSUQ, user interviews and thematic analysis. This phase ensures that the platform meets user needs and expectations, leading to iterative refinements and enhancements based on real user experiences.
Overall, this approach allows for the platform to grow and evolve in response to user feedback to develop a KG powered platform that is shaped by user needs. The KG backend allows for diverse types of data to be integrated into the system, while the incorporation of the ERCT ontology allows the mapping of this data to move towards standardisation. Meanwhile, the development of the InterDev dashboard and frontend allows users who are familiar with international development but do not have technical skills in semantic web technology to take advantage of linked data. As this research develops, the platform is likely to change and adapt in response to each evaluation round.
Existing portals, such as the 3IE Evidence Portal [15] and the American Economic Association's repository of randomized controlled trials [16], primarily provide high-level overviews and repository functions. InterDev, in contrast, focuses on international development and employs a decentralized, knowledge graph-based approach. This method ensures data consistency and interoperability across diverse datasets. By adopting a single standard for organizing and linking data, InterDev aims to enhance the accessibility and effectiveness of data for policymakers and researchers in the international development sector. InterDev is designed to provide a knowledge discovery platform aimed at facilitating the integration, curation, and analysis of impact evaluation data within the realm of international development. The architecture of InterDev, shown in Figure 1, is centered around a knowledge graph and an interface developed using React 18.2, with a backend infrastructure supported by Flask 3.0. This setup ensures efficient data discovery and interaction.
The data for this study was collected from multiple sources. Data from the International Initiative for Impact Evaluation (3ie) was scraped from their evidence portal, providing extensive information on the effectiveness of various development interventions. The American Economic Association (AEA) Registry data was obtained through downloadable CSV files, offering detailed records of randomized controlled trials. Additionally, contextual data from the World Bank was sourced from their databank, encompassing a wide range of global development indicators. This multi-source data collection approach underpins the robust knowledge base of InterDev, facilitating thorough analysis and evaluation of development initiatives.
The collected data was uplifted using RDFLib to convert it into RDF (Resource Description Framework) format. This process involved structuring the data according to the ERCT ontology, ensuring consistency and interoperability across different datasets. RDFLib facilitated the transformation of raw data into a standardized format, enabling integration within the knowledge graph.
The main dashboard is the central area for accessing the features of InterDev. It is divided into three sections:
Navigation Menu: Located on the left side, this menu provides quick access to key functionalities such as filtering by sector or country.
Primary Views: Users can switch between different views (Evidence View, Collection View, Submission View) using buttons at the top.
Content Area: The central part displays results of user interactions, such as evidence summaries, collections, or submission forms.
The Evidence View is designed for exploring and searching impact evaluations. Users can refine their searches by filtering results based on criteria such as sector or country. Results are displayed in a grid format, with each tile representing an evaluation. Tiles provide snapshots including the title, authors, and a brief description. Clicking on a tile gives access to detailed information about the evaluation, including methodology, findings, and related data.
The Collection View allows users to create, manage, and share collections of evaluations for projects or policy decisions. Users can add evaluations from the Evidence View into their collections, view contents, and share or download these collections directly from the platform. This feature facilitates collaboration and the effective utilization of relevant studies.
The Submission View provides an interface for submitting new evaluation data. It guides users through the process to ensure comprehensive and standardized data collection, capturing essential information such as the abstract, authors, title, project details, and evaluation design. This approach adheres to ERCT ontology standards, ensuring submitted data is integrated into the knowledge graph and accessible for future searches and analysis.
The first iteration of the InterDev evaluation is described below.
The evaluation methodology integrates both qualitative and quantitative approaches. Participants completed ten specific tasks using the InterDev platform, such as searching for evaluations, creating collections, and submitting new data. The study was conducted with five participants, including a mix of PhD researchers and social science researchers, none of whom had prior experience with semantic web technology. The evaluation aimed to assess how effectively users could navigate, interact with, and utilize the platform for their research needs without the requirement of experience in the semantic web. Task completion times were recorded by the observer to measure efficiency. During these tasks, the think-aloud protocol was employed, where participants verbalized their thoughts and actions, providing real-time feedback on their experiences and any difficulties encountered [17].
The tasks involved in the evaluation were as follows: selecting "Evidence View" from the navigation bar and waiting for the information to appear (T1), selecting any trial from the evidence view and viewing its associated information (T2), noting the sector of the selected trial (T3), filtering the trials in the evidence view by the noted sector until only trials from that sector appear (T4), adding four trials from this selection to the collection and confirming their presence in the "Collection View" (T5), returning to the evidence view and filtering for both a country and a sector, adding at most four more trials to the collection, and confirming their presence in the "Collection View" (T6), going to the "Collection View," filtering the collection by any property, and downloading the collection (T7), submitting a new trial with any data in the "Trial Submission" view (T8), finding the submitted evaluation data in the "Evidence View" (T9), and finally, downloading the evaluation data in the .ttl format (T10).
Additionally, instances where participants encountered an issue and required assistance were recorded by the observer to identify potential areas for improvement within the platform. The number of instances of these were recorded by the observer for each task. After completing the tasks, participants filled out the Post-Study System Usability Questionnaire (PSSUQ), which provided quantitative data on their overall satisfaction and the usability of the platform. This is a standardised survey that assessed the evolution of the usability during the development of the system and comprises of 19 questions [18].
To analyze the data, thematic analysis was conducted on the open-ended responses within the PSSUQ and recordings from the think-aloud protocol, identifying common themes and user feedback. The thematic analysis followed a standardised 6 step process: familiarisation with the data, generation of initial codes, a search for themes, a review of themes, definition and naming of themes and then reporting on findings [19]. Instances of themes were tagged in-text and a script written in Python to count and summarise the instances of these themes across the evaluation data. The PSSUQ results were quantitatively analyzed to assess various aspects of usability, such as ease of use, efficiency, and error handling. This methodology ensures a thorough evaluation of the InterDev platform, combining both user experiences and measurable data to inform future improvements and enhance the platform's usability and effectiveness for researchers and policymakers in international development.
Figure 3 illustrates the box plot of time spent to complete each task. Tasks such as selecting the "Evidence View" from the navigation bar (Task 1), selecting any trial from the evidence view (Task 2), and noting the sector of the trial (Task 3) have low median completion times and minimal variability, indicating that users found these tasks straightforward and easy to complete. However, tasks involving filtering and managing collections presented more challenges. For instance, Task 4, which requires filtering trials for the noted sector, shows moderate median completion time with some variability, suggesting users found the filtering function somewhat challenging. Task 5, which involves adding four trials to the collection, and Task 6, which includes filtering for both a country and a sector, both exhibit higher median completion times and significant variability, indicating these tasks were particularly difficult for users. Other tasks, such as submitting a new trial (Task 8) and finding the submitted evaluation in the evidence view (Task 9), also show higher median completion times and some outliers, reflecting challenges in the submission process and locating submitted evaluations. Figure 4 shows the intervention count for each task, providing further insights into task difficulty. Task 7, which involves filtering the collection by any property and downloading it, had the highest number of interventions, suggesting it was particularly challenging for users. Tasks 2, 3, and 5 had moderate intervention counts, indicating these tasks presented some challenges but were generally manageable. Tasks 1, 4, and 8 had lower intervention counts, suggesting these tasks were relatively straightforward for users. Tasks 6, 9, and 10 had no recorded interventions, indicating that these tasks were the easiest for users to complete independently. The analysis of the PSSUQ data seen in figure 5 indicates that users generally find the system functional, with lower scores reflecting better usability and satisfaction. However, significant variability in satisfaction levels was observed. Notably, questions related to error messages (Q9) and overall satisfaction (Q19) exhibit higher scores and outliers, suggesting inconsistent user experiences in these areas. This inconsistency underscores the need for targeted improvements in error message clarity and overall system responsiveness. Additionally, the higher median scores for some questions indicate areas where users are less satisfied, highlighting the necessity for comprehensive enhancements in interface design and functionality.
The implications of these findings suggest that while the InterDev platform serves its primary purpose, there is substantial room for improvement. Enhancing error message clarity can significantly reduce user frustration and improve task efficiency, allowing more intuitive interaction with the system.
Table 1 summarizes the thematic analysis for the first iteration of InterDev user testing, providing further insights into user feedback. Usability and Interface Design (UID), which encompasses overall design, intuitiveness, and ease of use, had the highest frequency with 19 mentions, indicating that users frequently commented on the visual layout, ease of finding information, and general user experience. Guidance and Navigation (GN) had 13 mentions, highlighting user comments on the clarity of instructions, ease of navigation, and suggestions for improving user guidance, such as better task prompts and visual cues. Functionality and Features (FF) was mentioned 12 times, reflecting feedback related to the platform's functionalities, including search capabilities, filtering options, and specific features like collection management and submission forms. Efficiency and Performance (EP), with 9 mentions, included observations related to the speed and efficiency of completing tasks, as well as any technical issues or bugs encountered during use.
Thematic analysis summary for the first iteration of InterDev user testing.
Code Description Frequency Usability and Interface Design (UID)
Overall design, intuitiveness, and ease of use of the platform's interface. This includes feedback on visual layout, ease of finding information, and general user experience.
Guidance and Navigation (GN)
Comments on the clarity of instructions, ease of navigation, and suggestions for improving user guidance, such as better task prompts and visual cues.
Functionality and Features (FF)
Feedback related to the platform's functionalities, such as search capabilities, filtering options, and specific features like collection management and submission forms.
12 Efficiency and Performance (EP) Observations related to the speed and efficiency of completing tasks, as well as any technical issues or bugs encountered during use. 9
The initial evaluation of InterDev demonstrates its potential to enhance data discovery and usability for researchers and policymakers in international development. While users found the platform generally functional, significant improvements are needed, particularly in filtering, error message clarity, search capabilities, and overall interface design. Quantitative and qualitative feedback from our user study highlighted key areas for enhancement, such as better guidance, improved navigation, and more intuitive features. These insights will guide the iterative refinement of InterDev to better meet user needs. While InterDev shows promise, further continuous user-centered development is required. Future iterations will address identified challenges, further refine user needs, and aim to improve the user experience and maximize the platform's utility in making international development data more accessible and actionable.
This research was conducted with the financial support of Science Foundation Ireland under Grant Agreement No. 13/RC/2106_P2 at the ADAPT SFI Research Centre at Trinity College Dublin. ADAPT, the SFI Research Centre for AI-Driven Digital Content Technology, is funded by Science Foundation Ireland through the SFI Research Centres Programme.