RDF [1,2] is a powerful tool for expressing statements as absolute, asserted relationships between entities. Yet, it falls short when it comes to representing nuances about such relationships, e.g., to enrich them with contextual information, or to represent in full how they relate to each other. While some statements may never need to be questioned (or maybe there is no interest in questioning them), and therefore can be represented adequately with plain RDF triples, frequently on the other hand scientific and critical discourse is filled with concurrent opinions and interpretations. This knowledge cannot be simply expressed as plain RDF triples but requires the specification of contextual information (e.g. who is the author of such claims), and possibly of multiple and competing statements, which the community may settle or discard over time.
Woodstock'21: Symposium on the irreproducible science, June 07-11, 2021, Woodstock, NY Envelope valentina.pasqual2@unibo.it (V. Pasqual); gerald.manzano@studio.unibo.it (G. Manzano); eduart.uzeir@studio.unibo.it (E. Uzeir); francesca.tomasi@unibo.it (F. Tomasi); fabio.vitali@unibo.it (F. Vitali) GLOBE https://valentinapasqual.github.io/ (V. Pasqual) Orcid 0000-0001-5931-5187 (V. Pasqual); 0000-0002-6631-8607 (F. Tomasi); 0000-0002-7562-5203 (F. Vitali) This is for instance the case of the hypotheses made by scholars about the many (still undetermined) locations of the Mona Lisa painting when it was stolen in 1911 [3], or of the attribution of the painting Salvator Mundi, possibly by Leonardo da Vinci and world-famous for having been the most expensive painting ever sold at public auction. Its attribution is now under discussion by many scholars [4]. Our guiding example in this paper, on the other hand, is the debated attributions of the painting Napoleon crossing the Alps 1 , currently attributed to Jacques-Louis David, but previously attributed to Jérôme-Martin Langlois and the workshop of Jacques-Louis David.
RDF Reification2 methods [5] have been used to make statements about statements in RDF [6] and even allowing the coexistence of multiple opinions in RDF. For instance, reification enables to express sentences like "JocondeLab3 reports that Napoleon crossing the alps author is Jacques-Louis David". Usually, reification methods typically involve the use of extra triples to provide information about the original triple being reified. Listing 1 exemplifies the statement above presented using traditional reification. In the example, a new entity is introduced as an instance of the class rdf:Statement to introduce "JocondeLab reports" and the original triple (Napoleon crossing the alps author is Jacques-Louis David) is expressed by the means of three additional triples whose predicates are strictly rdf:subject, rdf:predicate, rdf:object. @ p r e f i x r d f : < h t t p : / / www. w3 . o r g / 1 9 9 9 / 0 2 / 2 2 − r d f − s y n t a x − ns #> . Listing 1: Representation of the claims "XXX states that Napoleon Crossing the Alps" author is Jacques-Louis David using standard reification Accepted statements can be therefore characterised by a plain triple to assert the statement, and the same triple under reification to provide additional information about the claim itself. This would be the case of the settled attribution of Napoleon crossing the alps currently attributed to Jacques-Louis David. Statements that represent alternative or historical information would then be represented with just the claim in reified form and no assertion through a plain RDF triple. This would be case of the earlier attributions of Napoleon crossing the alps to Jérôme-Martin Langlois and the workshop of Jacques-Louis David. We call this approach Expressing Without Asserting (EWA) [7] and we consider it as a powerful tool to express claims with different degrees of validity and even critical debate.
In addition to analysing the expressivity and effectiveness of the many existing reification approaches, their efficiency is also an open issue. To the best of our knowledge, no study has been proposed on the evaluation of the efficiency of EWA mechanisms with ontologyindependent approaches. This paper evaluates existing reification methods for expressing without asserting in RDF with the related contextual information and compares them with a new approach, called Conjectures.
The paper is structured as follows: in section 2 we discuss of a number of syntactical approaches to EWA and benchmarks adopted in the literature for their evaluation. In section 3 we briefly introduce and compare conjectures as an alternative method to represent EWA statements in addition to existing proposals such as RDF reification, RDF star, and others. In section 4 we discuss the dataset and the metrics used for our comparison of the various EWA approaches, and in section 5 we analyze our findings. In section 6 we discuss the results before drawing some conclusions in section 7.
Many Reification methods have been implemented [8,9,10,11,12,13] The scientific community has suggested the following major reification strategies to be able to encapsulate all this variety of contextual information about RDF statements.
Among existing Knowledge Graphs, Wikidata provides RDF data with its reification method [11], and a long list of qualifiers about provenance, time, and place to represent complex and competing information. Additionally, each claim validity is provided by a customised approach to express without asserting [7] such claims (via ranking mechanism4 ) which truth state cannot be given for granted. Consider for example the case of concurring attributions of Napoleon crossing the Alps, as shown in figure 1, is represented in Wikidata with three different claims: the currently accepted attribution to Jacques-Luois David (marked with a preferred rank and therefore asserted), the former attribution to the his workshop (marked with a normal rank and non-asserted) and the former attribution to Jérôme-Martin Langlois (marked with a deprecated rank and therefore non-asserted). The two former attributions are therefore marked with contextual information about the claim (e.g. reason for deprecated rank, former attribution) to express additional information on the claim itself.
To be widely accepted, every data representation method must be both effective and efficient. While a method's effectiveness can be formally demonstrated, we must rely on empirical observations to determine its efficiency. Every new method that is suggested typically includes performance data as well. This is accomplished by carrying out the necessary tests to see how the approach performs on performance indicators like the number of triples, query execution time, query complexity, dataset storage consumption, support by existing tools and implementations and other pertinent metrics. There is a vast number of academic papers that suggest this kind of experimental setup to benchmark the aforementioned reification methods [16,12,6,14,17,18,19]. Additionally, all the benchmarks mentioned above rely on the same four components, namely datasets, queries, triplestores and reification methods.
Conjectures is an approach intended to express RDF statements with three main states of knowledge: undisputed claims, disputed claims or evolving situations, and settled claims. Conjectures is a RDF 1.1 compliant characterization of Named graphs (the weak form), and an extension of RDF Named graphs semantics to distinguish plain Named graphs from disputed (conjectural) and settled (the strong form) 5 .
Undisputed claims Non-disputed claims are expressed as asserted Named graphs (and therefore introduced by the keyword GRAPH). For example, the main subject of Napoleon Crossing the Alps has been recognised as Napoleon itself without any doubt as shown in listing2.
Conjectures is a prototypical extension of the syntax of Trig, where the keyword GRAPH is replaced with CONJECTURE in front of a graph whose contents is expressed but not asserted and expressing those statement which is disputes or conveys evolving situations. For example, Napoleon crossing the Alps former attribution to Jérôme-Martin Langlois can be represented via a Conjecture as in listing 2. Naturally, graphs that are not marked as conjectures maintain the same (locally-decided) semantics as before, and therefore they may or may not contribute to the truth value of the entire dataset depending on such choice. A key aspect is that Conjectures do not use reification, 𝑛-ary relationships or ad hoc classes For example, the rdf:Statement class, as employed in standard reification and illustrated in Listing 1, and therefore they are orthogonal to, and fully compatible with, most of the other approaches.
# The p a i n t i n g main s u b j e c t i s Napoleon GRAPH s : c l a i m 1 { wd : Q19801150 wdt : P921 wd : Q517 } # t h e p a i n t i n g ( wd : Q19801150 ) c r e a t o r ( wdt : P170 ) was s a i d t o be Jérôme − M a r t i n L a n g l o i s ( wd : Q672158 ) CONJECTURE s : c l a i m 2 { wd : Q19801150 wdt : P170 wd : Q672158 } # t h e p a i n t i n g c r e a t o r ( wdt : P170 ) was s a i d t o be from t h e s c h o o l o f J a c q u e s − L o u i s David CONJECTURE s : c l a i m 3 { wd : Q19801150 wdt : P170 _ : b l a n k } # A f t e r d e b a t e , t h e p a i n t i n g ( wd : Q19801150 ) c r e a t o r ( wdt : P170 ) i s J a c q u e s − L o u i s David ( wd : Q83155 ) SETTLED s : c l a i m 4 { wd : Q19801150 wdt : P170 wd : Q83155 } Listing 2: Representation of the disputed claim about "Napoleon crossing the Alps" creator with Conjectures
Settled claims Settled claims (introduced by the keyword SETTLED) record both the dispute, as well as its subsequent resolution. This is specifically and intentionally different from a trivial re-assertion of the disputed claims, in which we do not acknowledge or mention the dispute at all (the case of GRAPH in 2). To handle settled disputes we introduce Settled Conjectures, a third type of named graph that is at the same time conjectured and asserted. The collapse graphs allow us to both represent the conjectural triples (inside the usual conjectural graph) as well as the same triples but completely asserted (inside the collapse graph). In addition, the 𝑐𝑜𝑛𝑗 ∶ 𝑠𝑒𝑡𝑡𝑙𝑒𝑠 relation connects the conjecture and its settlement, simplifying the task of exploring the relationships between disputes and their settlements. The rationale behind Settled Conjectures is two-fold: on the one hand, to stress the difference between claims that have not been challenged and claims that emerged as winning among competing and incompatible hypotheses and on the other to represent the dual nature of settled claims as both conjectures and assertions. Consider for example "Napoleon Crossing the Alps" current attribution to Jacques-Louis David can be represented via a Settled conjecture as in listing2.
The methodology adopted to run the tests outlined in this study can be summarized as follows: first, a series of datasets were generated, scaled, and converted in RDF to mimic a set of EWA approaches (see Section 4.1). Next, the hardware and software employed to run our experiments on the collected data has been setup (see Section 4.2). A set of metrics has been defined to evaluate the efficiency of EWA approaches (see Section 4.3). Lastly, the outcomes of the tests are presented in the next section (see Section 5).
The dataset on which these experiments have been run is composed as follows and has been named D3:
• Art: A thematic set of claims about 300k artwork entities in Wikidata (i.e., painting, manuscripts, books). This corresponds to about 10% of all artwork entities currently present inside Wikidata. • Random: after considerable deliberation, we concluded that adding some kind of entropy to the dataset would make it more representative. This dataset contains the claims of 300k Wikidata random entities. • Dummy: a selection of dummy statements regarding the artwork attributions (represented by the property wdt:P50 and wdt:P170 and including from 1 to 4 authors in each claim and the source of the claim) and artworks locations (represented by the property wdt:P276, including 1 possible location, time constraints and source) has been created 6 .
Those new statements contain dummy arbitrary information ranked as deprecated and therefore non-asserted to represent alternative or historical claims to those contained in Art dataset. This design choice was made to increase the number of conjectural statements in the final dataset.
An excellent way to evaluate an algorithm's performance is to observe how it responds to variations in input size [6]. We started by downloading the whole subset of artwork entities, related individuals (basically, attributed authors) and locations. This dataset, called D4, is composed of about 3,5 million artwork entities and 188 thousand related entities (humans and locations). We have not used this dataset for our comparison due to the excessive number of timeouts in many of the queries and methods we used. Thus we scaled the dataset logarithmically in three further sizes:
• Dataset D3 : D3 is obtained by extracting one tenth of the data in D4 (D3 = D4/10).
• Dataset D2 : D2 is obtained by extracting one tenth of the data in D3 (D2 = D3/10).
• Dataset D1 : D1 is obtained by extracting one tenth of the data in D2 (D1 = D2/10).
We then surveyed the state of the art regarding reification methods to express without asserting and selected a set of methods for our analysis: Singleton properties [12], Named graphs [10] (using Wikidata rankings to decide whether a triple is asserted or not), Wikidata [11] and the recent RDF-star [13] approach. We converted Wikidata JSON files into the six selected reification methods through automatic scripts. In table 1 we provide some data about our datasets. At the end of this process, we obtained 18 new method-specific datasets. In other words, for each dataset 𝐷𝑛, 𝑛 ∈ [1,3], we constructed the following datasets:
Consider the case of Napoleon crossing the alps and its concurring attributions. In addition to the statements present in Wikidata, the listings below present an additional dummy statement reporting the historical attribution of the painting to Sophie Chéradame (Q60804575), a statement claimed by the source ContrivedAttributionsInArtHistory-VP and never adopted by the scholars (and therefore marked with deprecated rank). An example of these concurring claims is represented in the listing below with a different reification method: Wikidata statements (listing 3), RDF-star (listing 4), Conjectures in strong form (listing 5), Named graphs with the (listing 6), Conjectures in weak form (listing 7) and Singleton properties (listing 8) 7 .
wd : Q19801150 wdt : P170 wd : Q83155 .
Tests have been run on a computer with processor Intel Core i5-8259U CPU @ 2.30GHz 2.30 GHz, RAM 32,0 GB, Windows 10 pro 64 bits, 1T hard disk. The TriG and SparQL parsers of our GraphDB engine were modified to parse Conjectures in strong form. Our GraphDB configuration uses 28G Ram allocated to the application, 89 , and 10G cache size. A repository has been created for each dataset with inferences off, no rule set assigned, predicates list index enabled and (when possible) contexts enabled. All other parameters are left in their default values. Repositories are already running before their performance tests are executed.
We decided to base the comparison of our reification methods on four major metrics. These metrics are well-established when it comes to RDF quantitative analysis. The performancerelated features of the reification methods under consideration should all be covered by those criteria, which should also give us a clear picture of the benefits and drawbacks of each method.
• Total number of triples in endpoint: This value is particularly interesting since it makes it possible to assess the verbosity of each method. • Loading time: Time consumed by each dataset to be uploaded in the SPARQL endpoint.
• Dataset weight in triplestore: The storage size of the dataset after it has been uploaded and stored in the triplestore.
• Query execution time: Response time on a selected set of queries. Each query is executed automatically ten times. The average value is then computed.
Two sets of SPARQL queries (GQn, FQn) have been designed. While GQn queries do not include any filter, FQn queries restrict the results only to paintings (Q3305213). Each query set is composed of 6 queries assessing all possible statuses of claims validity. In particular, the queries retrieve the following topics: valid claims (Q1), debated claims (Q2), debated claims with their provenance/time (Q3), currently disputed claims (Q4), accepted claims after being debated (Q5), undisputed claims (Q6). Considering that authors' and locations' attributions provide a simple, yet effective use case to test RDF representation of EWA over our dataset, GQn and FQn have been then customised on retrieving authorship attributions (GQn-P170 and FQn-P170) and artworks' locations (GQn-P276 and FQn-P276) respectively by the use of Wikidata properties P170 and P276. All queries return the same data and are all correct in their respective datasets. Each query set has been automatically run 10 times and the average times have been calculated.
All reification methods add additional triples to the already existing ones to represent the necessary metadata (e.g. Singleton properties) or extend RDF 1.1 syntax (e.g. RDF-star). As shown in table 2, Named graphs are the method which uses reification with the lower number of triples, but with no explicit distinction between asserted and non-asserted graphs. While other surveyed methods (in particular, RDF-star, Wikidata statements and Singleton properties) use reification methods and assert each claim with an additional triple, Conjectures uses Named graphs structure to express both statements and reification without adding additional triples resulting the method to express without asserting with the lowest addition of triples.
In the context of dataset D1 and D2, Conjectures in the strong form remain competitive with the most efficient methods, notably RDF-star, and outperform Wikidata statements and Singleton properties as shown in 2. However, the loading times increase in D3. This performance discrepancy is attributed to the triplestore's parser method for recognizing conjectural data. Specifically, the process of checking each resource's presence in a collection during loading contributes to the observed delays. In essence, the loading time of the dataset increases proportionally with the quantity of non-asserted triples (conjectures).
The Singleton method exhibits a storage size tenfold greater than alternative approaches, with Conjectures in their weak form and Wikidata occupying intermediate positions. RDF-star, Conjectures in their strong form, and Named graphs demonstrate similar sizes as shown in figure 2.
The time response average for each dataset seems to increase linearly for each surveyed dataset (D1, D2, D3). For this reason, figure 3 provides the snapshot of the execution time of queries GQn and FQn on attributions and locations only on dataset D3. As illustrated in figure3, the response times obtained from the execution of general queries (GQn) on dataset D3 show that the strong form of Conjectures is less efficient than other methods when retrieving asserted data, particularly in retrieving valid claims, currently disputed claims and undisputed claims (queries GQ1, GQ4 and GQ6) for both creators and locations. However, Conjectures perform better with disputed claims. In particular, Weak and Strong Conjectures outperform other surveyed methods in the retrieval of debated statements with and without provenance information (queries GQ2 and GQ3) and accepted claims after being debated (GQ5) for both locations and creators.
Similar to what was observed in GQn, Conjectures are less efficient in retrieving valid claims (FQ1). On the contrary, Conjectures strong form is the most efficient method regarding the retrieval of debated claims with and without contextual information (queries FQ2 and FQ3) and accepted claims after debate (FQ5). In the remaining queries, currently disputed claims (FQ4) and statements that have never been subject to debate (FQ6), Conjectures still maintain competitive times with the rest of the methods. Strong Conjectures, in particular, address the significant increase in response times for weak-form queries FQn [3:8]. Essentially, a notable improvement in the performance of the strong form has been detected, proving to be the most efficient method in half of the selected queries and, in the remaining ones, a valid competitor.
In query performance assessment, we registered some slightly different values for the number of results of some queries, such differences are explained below highlighting some intrinsic differences in the models. Typically, a SPARQL query can still retrieve claims that won a certain debate by accessing concurring ones stored in the Knowledge Graph (KG). In cases where debates are not recorded, and only accepted statements are reported, reification approaches fall short. Reification methods do not distinguish between claims that have never been questioned and For instance, the painting Portrait of Dona Isabel de Requesens (Q29651096) has been attributed to Giulio Romano and marked as the settled claim, but no concurrent claim is reported. The concept of SETTLED in Conjectures can also capture this nuanced distinction in an ontologyindependent fashion which is transversal to be used on whichever KG. Another instance is when two claims express the same triple but with different qualifiers. Consider a scenario where a historical painting X was initially attributed to author Y (marked as an attribution) and afterwards this attribution has been considered settled by the community. The historical attribution cannot be retrieved in a SPARQL query since its content is also asserted. But, while Wikidata and Singleton provide unique IDs to distinguish such claims which can be involved in the query to retrieve such claim, RDF-star associates all contextual triples with the same quoted triple. This becomes more complex in multi-triple claims, where expressing them as individual Wikidata statements is not feasible. For instance, when dealing with paintings attributed to the collaboration of multiple individuals, expressing this complexity becomes particularly interesting. Conjectures use Named graphs to group statements. which allows to express and retrieve such complex statements with simple SPARQL queries. Other methods would require to adopt other types of grouping methods (e.g. a n-ary relationship and/or a blank node) with additional complexity and execution times.
Overall, we can immediately see several trends concerning the surveyed methods' efficiency: Singleton properties are systematically slower than the others, while Named graphs and Conjectures in weak form performs at an intermediate level about the fastest methods, Wikidata, Conjectures in strong form and RDF-star. Strong form also outperforms RDF-star in many queries where the specifics of debated attributions and past locations become meaningful. Strong form is the quickest method for expressing debates (disputed claims, GQ2 and FQ2, disputed claims with provenance GQ3 and FQ3, settled claims GQ5 and FQ5), with a small loss in term of performance for what concern asserted claims (valid claims GQn and FQn, undisputed claims GQ6 and FQ6) and currently disputed claims (GQ5 and FQ5).
Conjectures in strong form are also competitive for what concerns number of triples and overall weight in the triplestore. It is competitive on loading time for, but loading times show an interesting loss of performance for large datasets, that need to be investigated further.
This work evaluates the efficiency of EWA mechanisms by comparing several reification methods (Wikidata, RDF-star, Named graphs, Singleton properties) and the novel Conjectures approach (weak and strong form) on four major metrics (number of triples, loading time, dataset weight and query execution time). Among the most efficient methods as RDF-star and Wikidata statements, the strong form of Conjectures exhibits notable performance gains, particularly in retrieving claims about debates (e.g., disputed claims with and without provenance information and settled claims). In the future, we aim to optimize some aspects concerning the efficiency of the strong form of Conjectures. In particular, we will prioritize the optimization of the loading process, aiming to reduce the loading time and enhance its overall performance.