This demo paper presents UnScientify (https://bit.ly/unscientify-demo), an interactive system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique that employs a fine-grained annotation scheme to identify verbally formulated uncertainty at the sentence level in scientific texts. The pipeline for the system includes a combination of pattern matching, complex sentence checking, and authorial reference checking. Our approach automates labeling and annotation tasks for scientific uncertainty identification, taking into account diferent types of scientific uncertainty, that can serve various applications such as information retrieval, text mining, and scholarly document processing. Additionally, UnScientify provides interpretable results, aiding in the comprehension of identified instances of scientific uncertainty in text.
Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents and the 3rd AI + Informetrics (EEKEAII2023), June 26, 2023, Santa Fe, New Mexico, USA and Online * Corresponding author. $ panggih_kusuma.ningrum@univ-fcomte.fr (P. K. Ningrum); philipp.mayr@gesis.org (P. Mayr); iana.atanassova@univ-fcomte.fr (I. Atanassova)
0000-0002-8630-6603 (P. K. Ningrum); 0000-0002-6656-1658 (P. Mayr); 0000-0003-3571-4006 (I. Atanassova)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License biological events with negation [
Secondly, identifying scientific uncertainty in text in- These sentences were annotated to identify uncertainty
volves complex linguistic features as it is often conveyed expressions and authorial references. By utilizing
multhrough a combination of linguistic cues, including the tiple corpora from diferent disciplines, this study aims
use of modal verbs (e.g. may, could, might), hedging to capture a diverse range of uncertainty expressions
devices (e.g. seems, appears, suggests), and epistemic and improve the generalizability of the results. Table 1
adverbs (e.g. possibly, probably, perhaps) [
Another challenge concerns scientists’ discourse in
scientific writing. A typical scientific text contains vari- Identifying scientific uncertainty in academic texts is a
ous statements and information which not only discuss complex task due to various reasons. Previous research
the current or present study but also the former studies indicates that relying solely on cues or markers such as
[
To overcome these challenges, we propose a weakly and patterns formulation supervised technique that employs a fine-grained annotation scheme to construct a system for scientific uncertainty identification from scientific text focusing on the sentence level. Our approach can be used to automate labeling or annotating tasks for scientific uncertainty identification. Moreover, our annotation scheme provides interpretable results, which can aid in the understanding of the identified instances of scientific uncertainty in text. We anticipate that our approach will contribute to the development of more accurate and eficient scientific uncertainty identification systems, and facilitate the analysis and interpretation of scholarly documents in NLP.
The present study adopts a span-based approach for identifying scientific uncertainty in academic text. Rather than relying solely on linguistic cues, the scheme classiifes spans of text into several groups based on their linguistic features, including Part of Speech (POS) tags, morphology, and dependency. The scheme is also informed by a comprehensive analysis of scientific language, allowing for a more nuanced and accurate understanding of uncertainty expression.
During the annotation process, a list of annotated spans was created and classified into twelve groups of scientific uncertainty (SU) patterns based on their semantic meaning and characteristics. The groups include conditional expressions, hypotheses, predictions, and subjectivity, among others. In other words, the classification is based on the types of expressions used to convey uncertainty and the context in which they are used. Additionally, the scheme considers spans of text that signal disagreement statements as one of SU groups, despite ongoing debate regarding whether disagreement expressions should be considered as such. The justification for this approach is rooted in the idea that uncertainty in
the training set. These corpora consist of 59 journals from four diferent disciplines: Medicine, Biochemistry, Genetics & Molecular Biology, Multidisciplinary, and Empirical Social Science1 which represent Science, Technology, and
we selected articles from 53 social science journals indexed in
BMC Med
No
Yes
Yes
research can stem from conflicting information or data, tiple labels assigned to diferent SU pattern groups, as
where multiple sources provide contradictory knowledge seen in the second example, where labels for both
condi[
3.2. Authorial Reference Checking group. It should be noted that a sentence can have mul- 2Demo’s documentation: https://bit.ly/unscientify-demo 1
Explicit SU 2 3 4 5 6 7 8 9 10 11 12
Nongeneralizable statement
obvious scientific uncertainty keywords, unclear. indicating direct and explicit uncertainty 2) ... the functional relevance of G4 in vivo in expression mammalian cells remains controversial.
indicate uncertainty through the use of modal pretations about ... language 2) There may also be behavioral efects.
expressions that indicate uncertainty by is probable that the mere-exposure efect is presenting a condition or circumstance that weakened thus we hypothesize... must be met for a certain outcome to occur 2) If there are any violations, subsequent inferential procedures may be invalid, and if so, the conclusions would be faulty.
expressions that indicate uncertainty by for markets should be stronger... proposing a tentative explanation or 2) We assume that post-materialistic individassumption that requires further testing and uals may have difering attitudes towards docverification to be confirmed tors than those...
that indicate uncertainty by proposing a ergy Scenarios projected that the UK governforecast or projection that may or may not ment... come to fruition, thereby introducing an 2) Since aging leads to decreased Sir2, we preelement of uncertainty dicted that, in young cells...
expressions that indicate uncertainty by posing observed results can be replicated across difa question or series of questions, which may ferent populations. suggest doubt or uncertainty about a 2) ...this research literature has also contested particular concept or phenomenon whether or not citizens’ knowledge about these issues is accurate...
expresses uncertainty with limited scope or eralized to low-income nations nor extrapoapplicability, which may not represent a lated into the long-term future. broader context or population 2) ...estimates may not be generalisable to women in other to women in other ancestry groups...
adverbs that modify or shift the sentence’s transition from search to recognition mode meaning, introducing uncertainty is poorly understood.
the subpopulation of gender heterogenous...
indicate uncertainty through the use of clear. negation which may alter the meaning of the 2) There was no consistent evidence for a sentence and introduce an element of causal relationship between age at menarche uncertainty and lifetime number of sexual partners...
indicating uncertainty through subjective voters to care about... language like opinions, beliefs, or personal 2) To our knowledge, this is the first study experiences to provide global...
through conjecture or speculation, using religiosity. guessing or suppositions without concrete 2) Better performance seems to be linked to evidence life satisfaction...
that express uncertainty through disagreement sults did not show a significant efect... or contradiction, often indicating opposing 2) On the one hand, some researchers argue viewpoints or conflicting evidence that the use of technology in the classroom can enhance... sentence was annotated based on the citation & co- as ’Non-SU expression’. To optimize the matching procitation patterns, and the use of personal & impersonal au- cess, we customized a rule-based matcher from Spacy, thorial references. Furthermore, sentences were labeled which considers both keyword matches and patterns and into three groups including 1) author(s) of the present ar- linguistic features. ticle, or 2) author(s) of previous research. The last group, The second step, Complex Sentence Checking, deter3) both, is intended to accommodate complex sentences mines whether there are any rebuttal or confirmation that may refer to both the author(s) and the previous statements that can cancel the uncertainty expressed in study(s). Here, we present some examples of typical au- the sentence. If no such statements are detected, the thorial reference mentions in context: system labels the sentence as ’SU Expression’ and provides a list of final SU spans that provide information on 1. <I/We/Our study...> <text> the reason why a particular sentence is considered a ’SU 2. <Author/The former study...> <text> expression’. 3. (Author) (Year) <Text> The third step, Authorial Reference Checking, iden4. <Text> (Author1, Year1; Author2, Year2 tifies the authorship of the uncertainty expression, . . .) whether it belongs to the authors, to a previous study, or 5. <Text> [Ref-No1, Ref-No2 . . . ] both. The output of this step is the authorial reference of the sentence. 4. Demo System Figure 3 provides an overview of the functioning of
The demo system3 for identifying SU expressions op- pression, matching the ’Hypothesis group’ pattern. This erates at the sentence level and consists of three main demonstrates that UnScientify not only detects uncercomponents: 1) Pattern Matching, 2) Complex Sen- tainty expressions in sentences but also provides informatence Checking, and 3) Authorial Reference Checking, as tion about which sentence elements support the outcome shown in Figure 2. as well as descriptive information about why the sen
The first step, Pattern Matching, employs a list of pat- tence is considered an SU expression. In this case, the terns derived from 12 SU pattern groups (see Table 3). output identifies the sentence as an SU expression due The input sentence is matched against these patterns, to the occurrence of the "Hypothesis group’ pattern in and if a match is found, a list of SU span candidates is the sentence, indicating a tentative explanation or asgenerated. If there is no match, the sentence is labeled sumption that requires further testing for confirmation.
labeling this instance as ’Author(s)’, suggesting that the sentence originates from the author rather than being cited from other sources or previous studies. As a result, it provides more contextual and interpretable results.
Our demonstration system ofers a comprehensive approach to identifying uncertainty expressions in scientific text. By utilizing pattern matching, complex sentence checking, and authorial reference checking, we provide clear and interpretable output that explains why a sentence is flagged as expressing uncertainty, addresses the element of SU expression, and verifies authorship reference.
We firmly believe that our approach holds great potential for enhancing information retrieval, text mining, and broader scientific article processing. Moreover, it lays the groundwork for further research on scientific uncertainty and epistemology. While our system currently operates at the sentence level, it can be expanded to process text at the document level.
To further enhance the UnScientify system, we acknowledge the need for improvements to identify additional dimensions of scientific uncertainty, including its nature, context, timeline, and communication characteristics. Nonetheless, we are confident that our scheme serves as a promising starting point for an in-depth exploration of how scientific knowledge is constructed and communicated.
0003-01, and the Chrysalide Mobilité Internationale des
extends to the GESIS – Leibniz Institute for the Social
(a) Demo 1: Detecting Explicit SU with multiple SU spans (b) Demo 2: Detecting a sentence containing an interrogative expression (c) Demo 3: A sentence showing a Non-SU expression