CCS CONCEPTS • General and reference → Computing standards, RFCs and guidelines; • Information systems → Document structure; Ontologies; Data encoding and canonicalization.
We describe an annotation initiative to capture the scholarly contributions in natural language processing (NLP) articles, particularly, for the articles that discuss machine learning (ML) approaches for various information extraction tasks. We develop the annotation task based on a pilot annotation exercise on 50 NLP-ML scholarly articles presenting contributions to five information extraction tasks 1. machine translation, 2. named entity recognition, 3. question answering, 4. relation classification, and 5. text classification. In this article, we describe the outcomes of this pilot annotation phase. Through the exercise we have obtained an annotation methodology; and found ten core information units that reflect the contribution of the NLP-ML scholarly investigations. The resulting annotation scheme we developed based on these information units is called NLPContributions.
The overarching goal of our endeavor is four-fold: 1) to find a
systematic set of patterns of subject-predicate-object statements for
the semantic structuring of scholarly contributions that are more
or less generically applicable for NLP-ML research articles; 2) to
apply the discovered patterns in the creation of a larger annotated
dataset for training machine readers [
As the rate of research publications increases [
In this regard, several efforts exist or are currently underway for
semantifying scholarly articles for their improved machine
interpretability and ease in comprehension [
The automated semantic extraction from scholarly publications
using text mining has seen early initiatives based on sentences as
the basic unit of analysis. To this end, ontologies and vocabularies
were created [
Over which, many recent data collection and annotation
efforts [
Since scientific literature is growing at a rapid rate and researchers
today are faced with this publications deluge [
At https://www.orkg.org/ one can view the contribution knowledge graph of a single paper as a summary over its key contribution properties and values; or compare the contribution knowledge graphs over common properties across several papers in a tabulated survey.
Practical examples of the latter can be found accessible online at https://www.orkg.org/orkg/featured-comparisons. This practically addresses the knowledge ingestion problem for researchers. How? With the ORKG comparisons feature, researchers are no longer faced with the daunting cognitive ingestion obstacle from manually scouring through dozens of papers of unstructured content in their ifeld. Where this process traditionally would take several days or months, using the ORKG contributions comparison tabulated view, the task is reduced to just a few minutes. Assuming the individual paper contributions are structured in the ORKG, they can then simply deconstruct the graph, tap into the aspects they are interested in, and can enhance it for their purposes. Further, they can select multiple such paper graphs and with the click of a button generate their tabulated comparison. For additional details on systems and methods beyond just the contribution highlights, they can still choose to read the original articles, but this time around equipped with a better selective understanding of which articles they should read in depth. Of-course scholarly article abstracts are intended
D’Souza and Auer
for this purpose, but they are not machine interpretable, in other
words, they cannot be comparatively organized. Further, the
unstructured abstracts representation still treats research as data silos,
thus with this model, research endeavors, in general, continue to
be susceptible to redundancy [
In this paper, we propose a surface semantically structured dataset
of 50 scholarly articles for their research contributions in the field
of natural language processing focused on machine learning
applications (the NLP-ML domain) across five diferent information
extraction tasks to be integrable within the ORKG. To this end,
we (1) identify sentences in scholarly articles that reflect research
contributions; (2) create structured (subject,predicate,object)
annotations from these sentences by identifying mentions of the
contribution candidate term phrases and their relations; and (3) group
collections of such triples, that arise from either consecutive or
non-consecutive sentences, under one of ten core information units
that capture an aspect of the contribution of NLP-ML scholarly
articles. These core information units are conceptually posited as
thematic scripts [
It has the following characteristics: (1) via a contribution-centered model, it makes realistic the otherwise forbidding task of semantically structuring full-text scholarly articles—our task only needs a surface structuring of the highlights of the approach which often can be found in the Title, the Abstract, one or two paragraphs in the Introduction, and in the Results section; (2) it ofers guidance for a structuring methodology, albeit still encompassing subjective decisions to a certain degree, but overall presenting a uniform model for identifying and structuring contributions—note that without a model, such structuring decisions may not end up being comparable across users and their modeled papers (see Figure 6); (3) the dataset is annotated in JSON format since it preserves relation hierarchies; (4) the annotated data we produce can be practically leveraged within frameworks such as the ORKG that support structured scholarly content-based knowledge ingestion. With the integration of our semantically structured scholarly contributions data in the ORKG, we aim to address the tedious and time-consuming scholarly knowledge ingestion problem via its contributions comparison feature. And further, by using the graph-based model, we also address the problem of scholarly information produced as data silos, as the ORKG connects the structured information across papers. 2
Sentence-based Annotations of Scholarly Publications. Early
initiatives in semantically structuring scholarly publications
focused on sentences as the basic unit of analysis. In these
sentencebased annotation schemes, all annotation methodologies [
Phrase-based Annotations of Scholarly Publications. The
trend towards scientific terminology mining methods in NLP steered
the release of phrase-based annotated datasets in various domains.
An early dataset in this line of work was the ACL RD-TEC
corpus [
Next, we discuss related works that semantically model instructional scientific content. In these works, the overarching scientific knowledge capture theme is the end-to-end semantification of an experimental process.
Shallow Semantic Structural Annotations of Instructional
Content in Scholarly Publications. Increasingly, text mining
initiatives are seeking out recipes or formulaic semantic patterns to
automatically mine machine-actionable information from scholarly
articles [
In [
Along similar lines, inorganic materials synthesis reactions and
procedures continue to reside as natural language descriptions in
the text of journal articles. There is a growing impetus in such
ifelds to find ways to systematically reduce the time and efort
required to synthesize novel materials that presently remains one
of the grand challenges in the field. In [
The NLPContributions annotation methodology proposed in this paper draws on each of the earlier categorizations of related work. First, the full-text of scholarly articles including the Title and the Abstract are annotated in a sentence-wise granularity with the aim of the annotated sentences being only those restricted to the contributions of the investigation. We selectively consider the full-text of the article by focusing only on specific sections of the article such as the Abstract, Introduction, and the Results sections. Sometimes we also model the contribution highlights from the Approach/System description in case if the Introduction does not contain such pertinent information of the proposed model. We skip the Background, Related Work, and Conclusion sections altogether. These sentences are then grouped under one of ten main information units, viz. ResearchProblem, Objective, Approach, Tasks, ExperimentalSetup, Hyperparameters, Baselines, Results, and AblationAnalysis. Each of these units are defined in detail in the next section. Second, from the grouped contribution-centered sentences, we perform phrase-based annotations for (subject, predicate, object) triples to model in a knowledge graph. And Third, the resulting dataset has an overarching knowledge capture objective: capturing the contribution of the scholarly article and, in particular, to facilitate the training of machine readers for the purpose along the lines of the machine-interpretable wet-lab protocols. 3 3.1
The development of the NLPContributions annotation model was
backed by four primary goals:
(1) We aim to produce a semantic representation based on
existing work, that can be well motivated as an annotation
scheme for the application domain of NLP-ML scholarly
articles, and is specifically aimed at the knowledge capture of
the contributions in scholarly articles;
D’Souza and Auer
(2) The annotated scholarly contributions based on
NLPContributions should be integrable in the Open Research
Knowledge Graph (ORKG)1–the state-of-the-art content-based
knowledge capturing platform of scholarly articles’ contributions.
(3) The NLPContributions model should be useful to produce
data for the development of machine learning models in
the form of machine readers [
The NLPContributions annotation model is designed for building a knowledge graph. It is not ontologized, therefore, we assume a bottom-up data-driven design toward ontology discovery as more annotated contributions data is available. Nonetheless, we do propose a core skeleton model for organizing the information at the top-level KG nodes. This involves a root node called Contribution, following which, at the first level of the knowledge graph, are ten nodes representing core information units under which the scholarly contributions data is organized. 3.2
In this section, we describe the ten information units in our model.
ResearchProblem. Determines the research challenge addressed by a contribution using the predicate hasResearchProblem. By definition, it is the focus of the research investigation, in other words, the issue for which the solution must be obtained.
The task entails identifying only the research problem addressed
in the paper and not research problems in general. For instance,
in the paper about the BioBERT word embeddings [
The ResearchProblem is typically found in an article’s Title, Abstract and first few paragraphs of the Introduction. The task involves annotating one or more sentences and precisely the research problem phrase boundaries in the sentences.
The subsequent seven information objects are connected to Contribution via the generic predicate has.
Approach. Depending on the paper’s content, is referred to as Model or Method or Architecture or System or Application. Essentially, this is the contribution of the paper as the solution proposed for the research problem.
The annotations are made only for the high-level overview of the
approach without going into system details. Therefore, the
equations associated with the model and all the system architecture
ifgures are not part of the annotations. While annotating the earlier
1https://www.orkg.org/orkg/
2In future work, we will expand our current pilot annotated dataset of 50 articles with
at least 400 additional similarly annotated articles to facilitate machine learning.
ResearchProblem did not involve semantic annotation granularity
beyond one level, annotating the Approach can. Sometimes the
annotations (one or multi-layered) are created using the elements
within a single sentence itself (see Figure 1); at other times, if they
are multi-layered semantic annotations, they are formed by
bridging two or more sentences based on their coreference relations.
For the annotation element content itself, while, in general, the
subject, predicate, and object phrases are obtained directly from the
sentence text, at times the predicate phrases have to be introduced
as generic terms such as “has” or “on” or “has description” wherein
the latter predicate is used for including, as objects, longer text
fragments within a finer annotation granularity to describe the
top-level node. The actual type of approach is restricted to those
sub-types stated in the beginning of the paragraph and is decided
based on the the reference to the solution used by the authors or
the solution description section name itself. If the reference to the
solution or its section name is specific to the paper, such as ‘Joint
model,’ then we rename it to just ‘Model.’ In general, any alternate
namings of the solution, other than those mentioned earlier,
including “idea”, are normalized to “Model.” Finally, as machine learning
solutions, they are often given names. E.g., the model BioBERT [
The Approach is found in the article’s Introduction section in the context of cue phrases such as “we take the approach,” “we propose the model,” “our system architecture,” or “the method proposed in this paper.” However, there are exceptions when the Introduction does not present an overview of the system, in which case we analyze the first few lines within the main system description content in the article. Also, if the paper refers to their system by “method” or “application,” this is normalized to Approach information unit. System or Architecture is Model information unit.
Objective. This is the defined function for the machine learning algorithm to optimize over.
In some cases, the Approach objective is a complex function. In such cases, it is isolated as a separate information object connected directly to the Contribution.
ExperimentalSetup. Has the alternate name Hyperparameters. It includes details about the platform including both hardware (e.g., GPU) and software (e.g., Tensorflow library) for implementing the machine learning solution; and of variables, that determine the network structure (e.g., number of hidden units) and how the network is trained (e.g., learning rate), for tuning the software to the task objective.
Recent machine learning models are all neural based and such models have several associated variables such as hidden units, model regularization parameters, learning rate, word embedding dimensions, etc. Thus to ofer users a glance at the contributed system, this aspect is included in NLPContributions. We only model the experimental setup that are expressed in a few sentences or that are concisely tabulated. There are cases when the experimental setup is not modeled at all within NLPContributions. E.g., for the complex “machine translation” models that involve many parameters. Thus, whether the experimental setup should be modeled or not, may appear as a subjective decision, however, over the course of several annotated articles becomes apparent especially when the annotator begins to recognize the simple sentences that describe the experimental setup.
The ExperimentalSetup unit is found in the sections called Experiment, Experimental Setup, Implementation, Hyperparameters, or Training.
Results. Are the main findings or outcomes reported in the article for the ResearchProblem.
Each Result unit involves some of the following elements: {dataset, metric, task, performance score}. Regardless of how the sentence(s) are written involving these elements, we assume the following precedence order: [dataset -> task -> metric -> score] or [task -> dataset -> metric -> score], as far as it can be applied without significantly changing the information in the sentence. Consider this illustrated in Figure 2. In the figure, the JSON is arranged starting at the dataset, followed by the task, then the metric, and finally the actual reported result. While this information unit is named per those stated in the earlier paragraph, if in a paper the section name is non-generic, e.g., “Main results,” “End-to-end results,” it is normalized to a default name “Results.”
The Results unit is found in the Results, Experiments, or Tasks sections. While the results are often highlighted in the Introduction, unlike the Approach unit, in this case, we annotate the dedicated, detailed section on Results because results constitute a primary aspect of the contribution. Next we discuss the Tasks information unit, and note that Results can include Tasks and vice versa as we describe next.
Tasks. : The Approach or Model, particularly in multi-task settings, are tested on more than one task, in which case, we list all the experimental tasks. The experimental tasks are often synonymous with the experimental datasets since it is common in NLP for tasks to be defined over datasets. Where lists of Tasks are concerned, the Tasks can include one or more of the ExperimentalSetup, Hyperparameters, and Results as sub information units.
Experiments. Are an encompassing information unit that includes one or more of the earlier discussed units. Can include a combination of ExperimentalSetup and Results, or it can be combination of lists of Tasks and their Results, or a combination of Approach, ExperimentalSetup and Results.
Recently, more and more multitask systems are being developed.
Consider, the BERT model [
AblationAnalysis. Is a form of Results that describes the performance of components in systems.
Unlike Results, AblationAnalysis is not performed in all papers. Further, in papers that have them, we only model these results if they are expressed in a few sentences, similar to our modeling decision for Hyperparameters.
The AblationAnalysis information unit is found in the sections
that have Ablation in their title. Otherwise, it can also be found
in the written text without having a dedicated section for it. For
instance, in the paper “End-to-End Relation Extraction using LSTMs
on Sequences and Tree Structures” [
Baselines. are those listed systems that a proposed approach is compared against.
The Baselines information unit is found in sections that have
Baseline in their title. Otherwise, it can also be found in sections that
are not directly titled Baseline, but require annotator judgement
to infer that baseline systems are being discussed. For instance,
in the paper “Extracting Multiple-Relations in One-Pass with
PreTrained Transformers,” [
Of these ten information units, only three are mandatory. They are ResearchProblem, Approach, and Results; the other seven may or may not be present depending on the content of the article.
Code. is a link to the software on Github or on other similar open source platforms, or even on author’s website. 3.3
Except for ResearchProblem, each of the remaining nine information units encapsulate diferent aspects of the contributions of scholarly investigations in the NLP-ML domain; with the ResearchProblem ofering the primary contribution context. Within the seven diferent aspects, there are what we call Contribution Sequences.
Here, with the help of an example depicted in Figure 3 we illustrate the notion of contribution sequences. In this example, we model contribution sequences in the context of the ExperimentalSetup information unit. In the figure, this information unit has two contribution sequences. The first connected by predicate ‘used’ to the object ‘BERTBase model,’ and the second, also connected by predicate ‘used’ to the object ‘NVIDIA V100 (32GB) GPUs.’ The ‘BERTBase model’ contribution sequence includes a second level of detail expressed via two diferent predicates ‘pre-trained for’ and ‘pre-trained on.’ As a model of scientific knowledge, the triple with the entities connected by the first predicate, i.e. (BERTBase model, pre-trained for, 1M steps) reflects that the ‘BertBase model’ was pretrained for 1 million steps. The second predicate produces two triples: (BERTBase model, pre-trained on, English Wikipedia) and (BERTBase model, pre-trained on, BooksCorpus). In each case, the scientific knowledge captured by these two triples is that BERTBase was pretrained on {Wikipedia, BooksCorpus}. Note in the JSON data structure, the predicate connects the two objects as an array. Next, the second contribution sequence, hinged at ‘NVIDIA V100 (32GB) GPUs’ as the subject has two levels of granularity. Consider the following three triples: (NVIDIA V100 (32GB) GPUs, used, ten) and (ten, for, pre-training). Note, in this nesting pattern, except for ‘NVIDIA V100 (32 GB) GPUs,’ the predicates {used, for} and remaining entities {ten, pre-training} are nested according to their order of appearance in the written text. Therefore, in conclusion, an information unit can have several contribution sequences, and the contribution sequences need not be identically modeled. For instance, our second contribution sequence is modeled in a fine grained manner, i.e. in multiple levels. And when fine-grained modeling is employed, it is relatively straightforward to spot in the sentence(s) being modeled. 4
The pilot annotation task was performed by a postdoctoral researcher with a background in natural language processing. The NLPContributions model or scheme just described, were developed over the course of the pilot task. At a high-level, the annotations were performed in three main steps. They are presented next, after which we describe the annotation guidelines. (a) Contribution-Focused Sentence Annotations. In this stage, sentences from scholarly articles were selected as candidate contribution sentences under each of the aforementioned mandatory three information units (viz., ResearchProblem, Approach, and Results) and, if applicable to the article, for one or more of the remaining seven information units as well.
To identify the contribution sentences in the article, the full-text of the article is searched. However, as discussed at the end of Section 2, the Background, Related Work, and Conclusions sections are entirely omitted from the search. Further, the section discussing the Approach or the System is only referred to when the Introduction section does not ofer suficient highlights of this information unit. In addition, except for tabulated hyperparameters, we do not consider other tables for annotation within the NLPContributions model.
To better clarify the pilot task process, in this subsection, we use
Figure 2 as the running example. From the example, at this stage,
the sentence “For NER (Table 7), S-LSTM gives an F1-score of 91.57%
on the CoNLL test set, which is significantly better compared with
BiLSTMs.” is selected as one of the contribution sentence candidates
as part of the Results information unit. This sentence is selected
from a Results subsection in [
Resorting to our running example, for the selected sentence, this stage involves annotating the phrases “For,” “NER,” “F1-score,” “91.57%,” and “CoNLL test set,” with the annotator cognizant of the fact that they will use the [dataset -> task -> metric -> score] scientific entity precedence in the next step. (c) Creating contribution sequences. This involves relating the subjects and objects within triples, which as illustrated in Section 3.3, the object in one triple can be a subject in another triple if the annotation is performed at a fine-grained level of detail. For the most part, the nesting is done per order of appearance of the entities in the text, except for those involving the scientific entities {dataset, task, metric, score} under the Results information unit.
In the context of our running example, given the early annotated scientific entities, in this stage, the annotator will form the following two triples: (CoNLL test set, For, NER), (NER, F1-score, 91.57%) as a single contribution sequence. What is not depicted in Figure 1 are the top-level annotations including the root node and one of the ten information unit nodes. This is modeled as follows: (Contribution, has, Results), and (Results, has, CoNLL test set). 4.2
In this section, we elicit a set of general guidelines that inform the annotation task.
How are information unit names selected? For information units such as Approach, ExperimentalSetup, and Results that each have a set of candidate names, the applied name is the one selected based on the closest section title or cue phrase.
Which of the ten information units does the sentence belong to? Conversely to the above, if a sentence is first identified as a contribution sentence candidate, it is placed within the information unit category that is identified directly based on the section header for the sentence in the paper or inferred from cue phrases from the ifrst few sentences in its section.
Inferring Predicates. In ideal settings, the constraint on the text used for subjects, objects, and predicates in contribution sequences is that they should be found in their corresponding sentence. However, for predicates this is not always possible. Since predicate information may not always be found in the text, it is sometimes annotated additionally based on the annotator judgment. However, even this open-ended choice remains restricted to a predefined set of candidates. It includes {“has”, “on”, “by”, “for”, “has value”, “has description”, “based on”, “called”}.
How are the supporting sentences linked to their corresponding contribution sequence within the overall JSON object? The sentence(s) is stored in a dictionary with a “from sentence” key, which is then attached to either the first element or, if it is a nested triples hierarchy, sometimes even to the second element of a contribution sequence. The dictionary data-type containing the evidence sentence is either put as an array element, or as a nested dictionary element.
Are the nested contribution sequences always obtained from a single sentence? The triples can be nested based on information from one or more sentences in the article. Further, the sentences need not be consecutive in the running text. As mentioned earlier, the evidence sentences are attached to the first element or the second element by the predicate “from sentence.” If a contribution sequence is generated from a table then the table number in the original paper is referenced.
When is the Approach actually modeled from the dedicated section as opposed to the Introduction? In general, we avoid annotating the Approach or Model sections for their contribution sentences as they tend to delve deeply into the approach or model details, and involve complicated elements such as equations, etc. Instead, we restrict ourselves to the system higlights in the Introduction. However, in some articles the Introduction doesn’t ofer system highlights which is when we resort to using the dedicated section for the contribution highlights in this mandatory information unit.
Do we explore details about hardware used as part of the contribution? Yes, if it is explicitly part of the hyperparameters. Are predicates always verbs? Predicates are not always verbs. They can also be nouns especially in the hyperparameters section. Creating contribution sequences from tabulated hyperparameters. Only for hyperparameters, we model their tabulated version if given. This is done as follows: 1) for the predicate, we use the name of the parameter; and 2) for the object, the value against the name. Sometimes, however, if there are two-level hierarchical parameters, then the predicate is the first name, object is the value, and the value is qualified by the parameter name lower in the hierarchy. Qualifying the second name involves introducing the “for” predicate.
How are lists modeled within contribution sequences? As part of the contribution sentence candidates, are also included sentences with lists. Such sentences are predominantly found for the ExperimentalSetup or Result information units. This is modeled as depicted in Figure 4 for the first two list elements. Here, the Model information unit has two contribution sequences, each pertaining to a specific list item in the sentence. Further, the predicate “has description” is introduced for linking text descriptions. Which JSON structures are used to represent the data? Flexibly, they include dictionaries, or nested dictionaries, or arrays of items, where the items can be strings, dictionaries, nested dictionaries, or arrays themselves.
How are appositives handled? We introduce a new predicate “name” to handle appositives. 5 5.1
A collection of scholarly articles is downloaded based on the ones in the publicly available leaderboard of tasks in artificial intelligence called https://paperswithcode.com/. It predominantly represents papers in the Natural Language Processing and Computer Vision ifelds. For the purposes of our NLPContributions model, we restrict ourselves just to the NLP papers. From the set, we randomly D’Souza and Auer select 10 papers in five diferent NLP-ML research tasks: 1. machine translation, 2. named entity recognition, 3. question answering, 4. relation classification, and 5. text classification. 5.2
JSON was the chosen data format for storing the semantified parts of the scholarly articles contributions. To avoid syntax errors in creating the JSON objects, the annotations were made via https: //jsoneditoronline.org which imposes valid JSON syntax checks. Finally, in the early stages of the annotation task, some of the annotations were made manually in the ORKG infrastructure https: //www.orkg.org/orkg/ to test their practical suitability in a knowledge graph; three of such annotated papers are depicted in Figure 6. The links in the Figure captions can be visited to explore the annotations at their finer granularity of detail. 5.3
Overall, the annotated corpus contains a total of 2631 triples (avg. of 52 triples per article). Its data elements comprise 1033 unique subjects, 843 unique predicates, and 2182 unique objects. In Table 1 below, we show the per-task distribution of triples and their elements. Of all tasks, relation classification has the highest number of unique triples (544) and named entity recognition the least (473).
Generally, in the context of triples formation, predicates are often selected from a closed set and hence comprise a smaller group of items. In the NLPContribution model, however, predicates are extracted from the text if present. This leads to a much larger set of predicates that would require the application of predicate normalization functions to find the smaller core semantic set. In Figure 5, to ofer some insights to this end, we show the predicates that appear more than 15 times over all the triples. We find the predicate has appears most frequently since its function often serves as a filler predicate. A complete list of the predicates is released in our dataset repository online https://doi.org/10.25835/0019761.
MT NER QA RC TC Subject 259 209 203 228 221 Predicate 243 220 187 201 252 Object 471 434 515 455 459
Total 502 473 497 544 504 Table 1: Per-task (machine translation (MT), named entity recognition (NER), question answering (QA), relation classiifcation (RC), text classification (TC)) triples distribution in terms of unique subject, predicate, object, and overall. 6
As a use case of the ORKG infrastructure, instead of presenting just
the annotations obtained from NLPContributions, we present a
further enriched showcase. Specifically, we model the evolution of
the annotation scheme at three diferent attempts with the third one
arriving at NLPContributions. This is depicted in Figure 6. Our
use case is an enriched one for two reasons: 1) it depicts the ORKG
infrastructure flexibility for data-driven ontology discovery that
makes allowances for diferent design decisions; and 2) it also shows
how within flexible infrastructures the possibilities can be too wide
that arriving at a consensus can potentially prove a challenge if it
isn’t mandated at a critical point in the data accumulation.
(a) Research paper [
Figure 6(a) depicts the first modeling attempts of an NLP-ML contribution. For predicates, the model restricts itself to use only those found in the text. The limitation of such a model is that not normalizing linguistic variations very rarely creates comparable models across investigations even if they imply the same thing. Hence, we found that for comparability a common predicate vocabulary at the top-level in the model minimally needs to be in place. Figure 6(b) is the second attempt of modeling a diferent NLP-ML contribution. In this attempt, the predicates at the top-level are mostly normalized to a generic “has,” however, “has” is connected to various information items again lexically based on the text of the scholarly articles, one or more of which can be grouped under a common category. Via such observations, we systematized the knowledge organization at the top-level of the graph by introducing the ten information unit nodes. Figure 6(c) is the resulting NLPContributions annotations model. Within this model, scholarly contributions with one or more of the information units in common, viz. “Ablation study,” “Baseline Models,” “Model,” and “Results,” can be uniformly compared. 7
Obtaining disjoint (subject, predicate, object) triples as
contribution sequences. It was not possible to extract disjoint triples
from all sentences. In many cases, we extract the main predicate and
use as object the relevant full sentence or its clausal part. From [
Tabulated results are not incorporated within NLPContributions. Unlike tabulated hyperparameters which have a standard format, tabulated results have significantly varying formats. Thus their automated table parsing is a challenging task in itself. Nonetheless, by considering the textual results, we relegate ourselves to their summarized description, which often serves suficient for highlighting the contribution.
Can all NLP-ML papers be modeled by NLPContributions? While we can conclude that some papers are easier to model than others (e.g., articles addressing ‘relation extraction’ vs. ‘machine translation’ which are harder), it is possible that all papers can be modelled by at least some if not all the information units of the model we propose. 8
From the pilot dataset annotation exercise, we note the following regarding task practically. Knowledge modeled under some information units are more amenable to systematic structuring than D’Souza and Auer others. E.g., information units such as ResearchProblem, ExperimentalSetup, Results, and Baselines are readily amenable for systematic templates discovery toward their structured modeling within the ORKG; whereas the remaining information units, especially Approach or Model, will require additional normalization steps toward the search for their better structuring. 9
The Open Research Knowledge Graph [
Finally, aligned with the initiatives within research communities
to build the Internet of FAIR Data and Services (IFDS) [
Nevertheless, the NLPContribution model is a surface
semantic structuring scheme for the contributions in unstructured text. To
realize a full-fledged machine-actionable and inferenceable
knowledge graph of scholarly contributions, as future directions, there are
a few IE modules that would need to be improved or added. They
are (1) improving the PDF parser to produce less noisy output; (2)
incorporating an entity and relation linking and normalization
module; (3) merging phrases from the unstructured text with known
ontologies (e.g., the MEX vocabulary [
In general, even if the annotations are performed by a single annotator, there will be an annotation discrepancy. Compare the same information unit “Experimental Setup” modeled in Figure 7 below versus that modeled in Figure 3. Fig. 7 was the first annotation attempt and includes the second attempted model, done on a diferent day and blind from the the first. While neither are incorrect, the second has taken the least annotated information route possibly due to annotator fatigue, hence a two-pass methodology is recommended.