In the social sciences, researchers formulate a hypothesis, conduct background research and run a meta-analysis to test that hypothesis synthesizing a large body of evidence. Results of the meta-analysis are published in a meta-review. Yet this process is manual, demanding significant efort and expert knowledge, and results in static PDF formats. Furthermore, social scientists struggle in keeping the reviews up to date due to the increasing number of empirical studies. A living, automated meta-review generation system would be the solution to ease this process. We present an interface to select starting hypotheses and to create living meta-reviews in HTML format automatically, with a case study on human cooperation. This will enable to assist social scientists in their daily tasks, and we believe that the benefits of this work will be broadly relevant across the social sciences.
A
Case Study on
Let us imagine social scientists investigating the impact of gender on cooperative behaviors. This
example follows one from the literature [
Recent work [
Such datasets, and the research platforms built upon them [
CEUR
ceur-ws.org
formats.1 In the R Shiny2 interface3 [
A living, automated meta-review generation system summarizing meta-analyses would be the solution to accommodate this challenge. By enabling researchers to interact with the data, such a tool would enhance the utility of meta-analytic knowledge. The meta-analyses would be easily findable in an open, standard format, fostering reproducibility and credibility. Additionally, as the living meta-analyses are dynamically built on structured datasets and automatically updated with data changes, they would provide the most current information to guide future research.
We present an interface to create living meta-reviews automatically built from data queried on demand. The meta-review in HTML format becomes dynamic with the regeneration of the HTML each time a hypothesis is modified. We present a use case in human cooperation. This will aid social scientists in summarizing empirical evidence, reducing human eforts at all levels of the data management process like publication or querying. We believe the benefits of this work will extend beyond our case study and will be broadly relevant across the social sciences. To the best of our knowledge, we are the first ones to present such a novel interface. We make our code to build the interface openly available.4
Overview. Our interface allows users to select a hypothesis and to generate a meta-review in HTML format automatically. Studies comparing two groups are retrieved from the hypothesis and refined from inclusion criteria. The analytic strategy defines the statistical model to run the meta-analysis, and control variables are displayed in the meta-review. Lastly, custom content can be added before generating and displaying the meta-review. We propose a step-by-step overview of the interface’s functionalities, and we make a demonstration video of the interface publicly available.5 1. Select a hypothesis. This is the first step where the user can choose a hypothesis based on the comparison of two diferent groups. This enables fetching studies for the meta-analysis.
We define terms used by social scientists [ 12]. An efect size ( es) [13] measures the relationship strength between two variables in a study. A dependent variable (dv) is the outcome that is measured in a study to assess the efect of changes in the independent variable(s). Our case study investigates human cooperation, using CoDa as the KG backend resource. It contains around 3,000 studies from the social and behavioural sciences, annotated with cooperation-related features. Generic independent variables (givs) are broad factors categorized to observe their efect on a dv. Specific independent variables (siv) are well-defined variables within a study, categorized to determine their efect on a dv. Specific independent variable values ( sivvs) are the value assigned to the siv. es, dv, giv, siv and sivv are assumed from CoDa. The left part of Figure 1 provides an example of the structure of CoDa: the paper id:ENG00006 includes the study id:ENG00006_1 that reports the efect id:ENG00006_1.1.1.2.d comparing the treatments id:00006_1.1.2 and id:ENG00006_1.1.1, hereafter denoted as T1 and T2. T1 and T2 have diferent characteristics for the cp:gender property, female and male.
We consulted with two experts to design meaningful hypotheses. While we lack formal evaluation, the experts agreed that comparing only identical or related independent variables is meaningful. Furthermore, user studies we conducted with 5 domain experts on the same templated hypotheses did not raise any objection. The main technical constraint we impose is thus that the sivvs to be compared be instances of the same siv, which is in the ontology. The right part of Figure 1 shows the 1Static examples of such meta-reviews can be found here: https://cooperationdatabank.org/living-reviews/ 2https://shiny.posit.co 3https://app.cooperationdatabank.org 4https://github.com/InesBlin/coda_meta_review/tree/main 5https://bit.ly/ekaw-pd-2024-meta-review-generation hypothesis we chose as a running example for this paper. CoDa contains three dvs: cooperation, its subcategory contributions, and withdrawals, with 9,456, 9,948, and 1,774 reported efects, respectively. Since withdrawals had fewer studies, we grouped contributions with cooperation, considering cooperation as the sole dv. The user can choose giv, siv and sivv in this step, and es in Step 4.
CoDa Overview id:dependentvariable/cooperation dv
es id:esmeasure/d cp:dependentVariable cp:eSmeasure id: ENG00006_1 cp:reportsEffect id:ENG00006_1.1.1.2.d
cp:treatment cp:study id: ENG00006 id:ENG00006_1.1.2
cp:gender sivv id:gender/female siv cc:Gender id:ENG00006_1.1.1 cp:gender sivv id:gender/male
giv cc: GenderVariable cc: <https://data.cooperationdatabank.org/vocab/class/> cp: <https://data.cooperationdatabank.org/vocab/prop/> cc:Paper cc:Observation id: <https://data.cooperationdatabank.org/id/> rdf: <rdhft:ttpy:p//ewww.w3r.dofrsg:/s1u9b9C9l/a0s2s/2O2f-rdf-syntax-ns#> cc:Study cc:Treatment Hypothesis
Variables comparative, concept, concept_val_1, concept_val_2
Template Cooperation is significantly {comparative} when {concept} is {concept_val_1} compared to when {concept} is
{concept_val_2}. cp:treatment :Study1.1 cp:gender
Example cp:dependentVariable :Study1 :Study1.2
cp:gender cdp:ES :positive id:gender/female :cooperation
id:gender/male Cooperation is significantly higher when gender is female compared to when gender is male.
The options dynamically update based on the current selection, ensuring they remain in sync with CoDa. Figure 2 shows the interface with our running example, from which the hypothesis is derived.
2. Inclusion criteria. This step refines study filters for the meta-analysis. There are four types of criteria: (1) metadata, (2) quantitative (3) sample based on participants characteristics, and (4) study. The left part of Figure 3 shows an example, where the user only keeps studies from published articles. 3. Control variables. This step enables to choose additional variables to be described and displayed in the meta-review, such as maleProportion and discussion on the right part of Figure 3 in the interface. 4. Analytic strategy. This step concerns the statistical model to run on the selected studies to test the hypothesis for the meta-analysis, like a simple restricted maximum likelihood (REML) model [14] with Cohen’s standardized mean diference [ 15] as efect size on the left part of Figure 4 in the interface.
5. Custom text. This step is the possibility for experts to provide more descriptive information in the form of custom text, as shown on the right part of Figure 4, and to be added to the meta-review. 6. Generate and display the meta-review. The meta-review is based on an HTML template that was carefully designed with domain experts, and is generated based on previous criteria. One example of output can be seen in Figure 5.
Technical details. Regarding data storage and management, CoDa is hosted by Triply6 in the form of 7 Named KGs. To make the interface more eficient, we used SPARQL queries, available in the code, to create .csv backend files. For the frontend/user interface, we used Streamlit,7 and plotly8 for most visualisations. The interface is in the app folder of the repository. The core functions that are called in the interface are in the src folder. For the backend, we used Python,9 and called the R package metafor10 for the meta-analyses. The interface is not yet deployed live. 6https://odissei.triply.cc/coda/databank 7https://streamlit.io 8https://plotly.com 9https://www.python.org 10https://cran.r-project.org/web/packages/metafor/index.html
We present an interface to create living meta-reviews based on hypotheses comparing two groups with diferent characteristics. This will aid social scientists in the process of summarizing empirical evidence on their research questions, reducing the human eforts at all levels of the data management process, such as publication or querying. Ultimately, the goal is to index, aggregate and centralize meta-reviews to avoid duplication. Quality will be ensured through peer review or expert validation. The interface is now limited to one type of hypothesis only, and there is currently no user evaluation. We plan to integrate more complex hypotheses and to extend meta-reviews templates to improve the tool for social scientists. We also plan to deploy the interface online, and to conduct user studies to better assess the usefulness of the tool. We are also working on automatically suggesting hypotheses with various AI methods, as well as user studies to evaluate the hypotheses.
This work was partly funded by the European MUHAI project, grant no. 951846, and by the XS NWO Project, grant no. 406.XS.04.118. We thank our reviewers for constructive comments. case-study, in: The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31–June 4, 2020, Proceedings 17, Springer, 2020, pp. 287–303. [12] A. Bryman, Social research methods, Oxford university press, 2016. [13] K. Kelley, K. J. Preacher, On efect size., Psychological methods 17 (2012) 137. [14] Y. Dodge, The Oxford dictionary of statistical terms, Oxford University Press, USA, 2003. [15] J. Cohen, Statistical power analysis for the behavioral sciences, routledge, 2013.