NaïveRole: Author-Contribution Extraction and Parsing from Biomedical Manuscripts Dominika Tkaczyk 1,a, Andrew Collins1,b, Joeran Beel1,c 1 Trinity College Dublin, ADAPT Centre, School of Computer Science and Statistics, Ireland ad.tkaczyk@gmail.com, bancollin@tcd.ie, cjoeran.beel@scss.tcd.ie Abstract. Information about the contributions of individual authors to scientific publications is important for assessing authors’ achievements. Some biomedical publications have a short section that describes authors’ roles and contributions. It is usually written in natural language and hence author contributions cannot be trivially extracted in machine readable format. In this paper, we present 1) A sta- tistical analysis of roles in author contributions sections, and 2) NaïveRole, a novel approach to extract structured authors’ roles from author contribution sec- tions. For the first part, we used co-clustering techniques, as well as Open Infor- mation Extraction, to semi-automatically discover the popular roles within a cor- pus of 2,000 contributions sections from PubMed Central. The discovered roles were used to automatically build a training set for NaïveRole, our role extractor approach, based on Naïve Bayes. NaïveRole extracts roles with a micro-averaged precision of 0.68, recall of 0.48 and F1 of 0.57. It is, to the best of our knowledge, the first attempt to automatically extract author roles from research papers. This paper is an extended version of a previous poster published at JCDL 2018. Keywords: document analysis, author contributions, semantic publishing 1 Introduction Authorship is an important concept in scholarly communication. It allows people to properly credit those who contributed to scientific discoveries and is widely used to assess people’s scientific achievements. However, to fully evaluate researcher’s achievements, it is useful to know the precise nature of their contributions to authored publications. In some biomedical journals, a submitting author must provide infor- mation about each author’s individual contributions. This information is then attached to the manuscript as a short section entitled e.g. “Authors’ Contributions” (Fig. 1). Ex- amples of contributor roles include the preparation of data, designing experiments, pro- gramming software, or writing and editing the manuscript. These sections are usually written in natural language, are unstructured, and are in- tended for humans to read rather than machines. Contribution taxonomies and machine- readable formats are being introduced slowly, however, digital libraries contain docu- ments that have already been published in previous decades. Contribution information in such documents will not conform to new standards and will remain in an unstructured format. Consequently, analyses of author contribution information requires time- Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). consuming manual work, which makes processing large collections of documents in digital libraries impractical. We address these issues by proposing: 1. a method for semi-automatically discovering what roles are common in a corpus of sections of interest 2. a scalable approach for annotating a ground truth role dataset 3. a supervised algorithm for automatic extraction of the roles from unstructured text Fig. 1. Example of “Authors’ contributions” section with abbreviated author names. This paper is an extended version of a poster published at the Joint Conference on Dig- ital Libraries [1]. This extended version contains further descriptions of our study and proposed approaches, a comparison of our results to an existing contributor role taxon- omy, and an error analysis of the proposed automatic role extractor. 2 Related Work General information extraction from scientific literature is a popular research area, re- sulting over the years in many approaches and tools, including CERMINE [2], GROBID [3], PDFX [4], ParsCit [5], Science Parse1 and Docear PDF Inspector [6]. However, none of these systems extracts information related to the contributions of individual authors directly from the content of the paper. The scientific community has thus far not agreed on standard author contributions or even standard criteria for authorship. Nevertheless, some initiatives have been un- dertaken to increase the level of consistency between journals. For example, the Inter- national Committee of Medical Journal Editors published guidelines that suggest min- imum requirements for authorship, and the use of these guidelines is now encouraged by some medical journals. CRediT 2 is an example of a contribution taxonomy that defines the standard for contributors’ roles. CRediT is composed of 14 roles and was created based on free- form contributions and acknowledgements sections. Journals are increasingly adopting taxonomies like CRediT to consistently describe author contributions [8]. Our study does not assume any input taxonomy but aims at discovering popular roles within a corpus of contribution descriptions in an unsupervised way. Some journals, such as PLOS One or Annals of Internal Medicine, publish author contribution information in a machine-readable form. Several studies have examined author contributions using this data, for example, comparing author orderings to 1 https://github.com/allenai/science-parse (we used version 1 as at the time of our analysis the currently re- leased version 2 was not available, or we were at least not aware of it) 2 Contributor Roles Taxonomy: https://casrai.org/credit/ contributions [9, 10]. Typically, however, author contribution information has an un- structured, natural language form, and cannot be trivially examined in this fashion. 3 Methodology 3.1 Roles Discovery The first stage of our workflow (Fig. 2) is to discover common roles appearing in the corpus. Our analysis was composed of the following steps:  Data preparation, where we gathered a corpus of contributions sections.  Data preprocessing, where role mentions were extracted and cleaned.  Clustering, where abstract role concepts were discovered. Fig. 2. The workflow of our study. First, role mentions from the corpus were clustered to discover abstract role concepts (1). Then, resulting clusters were manually inspected and corrected (2). Next, cleaned clusters were used to generate the training set (3). Finally, a supervised machine learning model able to classify role mentions was trained (4). Data Preparation We use the PubMed Central Open Access Subset as data for our work. This is a subset of the total collection of articles in PMC, published under open licenses. We down- loaded the corpus of 1.6 million documents in machine-readable JATS format3. From each document we extracted any section whose normalized (lowercased and with all non-letters removed) title equals “authorscontributions”. We found these sections in 186,874 documents, constituting ~12% of the corpus. For performance reasons, we use a random subset of 2,000 sections only. All sections are written in English. Preprocessing Authors’ contribution sections typically mention the roles of several individual authors. We refer here to a natural language expression of the role an author plays as a role mention. A same role (e.g. data analysis) can be expressed by many forms of role men- tion (e.g. “X analyzed microarray sequences”, “X was involved in data analysis”). 3 https://jats.nlm.nih.gov/ We represent a role mention as a 3-element tuple containing: 1) subject: “who”, usually author name or initials, 2) action: activity, often a verb phrase, 3) object: “what the action was applied to”, typically a noun phrase (Fig. 3). Fig. 3. The decomposition of a single role mention into three parts: subject, action, and object. We use the Stanford Open Information Extraction tool4 to extract role mentions from the text. OpenIE [11] is an information extraction paradigm, in which it is possible to extract relations in the form of tuples (relation plus its two arguments) from the text, in an unsupervised way. The output corresponds to 3-element role mentions, where action is the relation expression and subject and object are its two arguments. As a result of applying OpenIE to our sections corpus, for every section we obtained a bag of role mentions, where a mention is a tuple of three text fragments from the original text. For example, from the sentence “AWL did the literature search and par- ticipated in the writing of the manuscript.” we got the following tuples: (“AWL”, “did”, “literature search”) and (“AWL”, “participated in”, “writing of manuscript”). OpenIE tools tend to output tuples that are redundant. For example, from the same sentence we might get both (“authors”, “read”, “final manuscript”) and (“authors”, “read”, “manuscript”) tuples. We analyze all pairs of tuples and consider one tuple in a pair redundant if the following conditions were met: 1) their subjects are exactly the same, 2) the action of one tuple contains all the words of the other action in the same order, and 3) the object of one tuple contains all the words of the other object in the same order. We remove such redundant tuples. The roles in role mentions are expressed by action-object pairs, and the subject re- fers only to the author. At the beginning, our corpus of 2,000 sections contained 6,924 distinct action-object pairs, many of which expressed the same roles. To merge some mentions and reduce the number of distinct action-object pairs, we applied cleaning and normalizing to actions and objects of role mentions. First, we stemmed words within actions and objects, and removed stopwords. For stemming we used R’s SnowballC library, and the stopwords list was downloaded from an online source5. This reduced the number of distinct roles to 6,289. We also remove rare role mentions, that is, mentions appearing less than five times in the corpus. This leaves 434 distinct action-object pairs while keeping 55% of role mentions. Finally, we observed that due to splitting role mentions into action and object, we still have distinct mentions that obviously refer to the same role, such as (“analys”, “data”) and (“perform”, “the analys of the data”). We wanted to normalize this, at the 4 https://nlp.stanford.edu/software/openie.html 5 http://www.ranks.nl/stopwords same time keeping the tuple-based structure of the mentions. To achieve this, we ex- tracted a number of most common terms from both actions and objects of the mentions (terms appearing at least 20 times in the corpus), and then each term was labeled as “action keyword” or “object keyword”, based on whether it is more common among actions or objects. Table 1 lists extracted action and object keywords. Each role mention in the corpus was then transformed in the following way: 1) the subject was left intact, 2) all action keywords found in the entire original mention formed the new action, and 3) all object keywords found in the entire original mention formed the new object. In addition, if the new action turned out to be empty, we added a single “perform” keyword to it. Table 1. Action and object keywords appearing in the corpus. The words are stemmed. Action keywords Object keywords read, particip, draft, contribut, conceiv, perform, manuscript, studi, data, final, design, analys, ex- write, revis, carri, critic, approv, made, prepar, peri, collect, interpret, statist, respons, involv, pa- conduct, provid, review, supervis, equal, de- per, concept, result, version, substanti, acquisit, velop, edit, plan, initi, acquir, assist, coordin, project, patient, research, work, content, intellectu, help, took, undertook, gave, comment, take, re- import, articl, discuss, first, protocol, molecular, cruit investig, sequenc, literatur, idea, part, princip, clinic, trial, sampl, genet, laboratori, advic, tool This operation moved words between actions and objects so that action keywords are always in the actions of the mentions and object keywords are in their objects. For example, since “perform” is an action keyword, and “analys” and “data” are object keywords, both mentions (“analys”, “data”) and (“perform”, “the analys of the data”) became (“perform”, “data analys”). This process left us with 285 distinct role mentions. Finding Roles In this phase, we detect roles in our collection of role mentions. We adopted an unsu- pervised machine learning technique (clustering) for this task. This is similar to a stand- ard ontology learning approach [12]. At the end of clustering, all mentions that refer to the same role should belong to the same cluster. For example, (“performed”, “data anal- ysis”) and (“was involved in”, “analyzing data”) should be clustered together. After preprocessing, our set contained 9,709 role mentions represented by cleaned subject- action-object tuples. We were interested in co-clustering the actions and the objects separately yet simultaneously, which in turn would define a third clustering based on the combinations of actions and objects. More formally, let 𝑀 = {𝑚 , … , 𝑚 } be the input mention set, and 𝐴 and 𝑂 the set of action clusters and the set of object clusters, respectively. We define an action clus- tering as a function 𝑓 : 𝑀 → 𝐴, which maps mentions to their action clusters. Similarly, let 𝑓 : 𝑀 → 𝑂 be the mapping function which defines object-based clustering. This lets us define a role set 𝑅 as the set containing all combinations of action and object con- cepts that share some mentions: 𝑅 = {(𝑎, 𝑜) ∈ 𝐴 × 𝑂 | 𝑓 (𝑎) ∩ 𝑓 (𝑜) ≠ ∅}. The fi- nal combined clustering is 𝑓 : 𝑀 → 𝑅 such that ∀ ∈ 𝑓 (𝑚) = 𝑓 (𝑚), 𝑓 (𝑚) . Set 𝑅 defines a binary relation between action and object clusters. We can define the weight of this relation as the number of the mentions that the clusters share: ∀ ∈ , ∈ 𝑟(𝑎, 𝑜) = |{𝑚 ∈ 𝑀 | 𝑓 (𝑚) = (𝑎, 𝑜)}| = |𝑓 (𝑎, 𝑜)|. Intuitively, if an action concept and an object concept appear in many role mentions together, they form a com- mon role, and the weight of the role is large. This defines a graph structure among the clusters, with action and object concepts as nodes and weighted edges representing re- lation strength. Finally, during our analysis we used the idea of a cluster label, defined as a bag of terms of the most numerous member of the cluster. We use bottom-up clustering, where we start with initial action and object clusters, and in several phases we merge clusters together. Initially, the clusters are defined as distinct normalized actions and objects. In other words, two mentions are in the same action/object cluster if their normalized actions/objects are identical. Each round of clustering is composed of two stages. The first one is based purely on cluster term la- bels. The second one uses the graph structure defined previously. Algorithm 1 presents the pseudocode of the role mentions clustering. The first stage of the clustering is based on the action/object label terms of the cur- rent role clusters. We examine pairs of role clusters and merge them if action and object terms of one of them contain the other cluster’s terms. The new cluster is always given a label equal to the label of the bigger cluster from the examined pair. The main clustering stage is based on the weighted graph relations between action and object clusters. First, we identify an action or object cluster pair that is most similar to each other, then their clusters are merged. When the highest similarity drops below a predefined threshold, the clustering procedure terminates. We will only explain how the similarity between two action clusters is defined. The similarity between object clusters is defined analogously. The main observation used for calculating the similarity between two action clusters is that two actions related to a lot of common objects will be more similar to each other. However, this assumption is trivially violated in cases where there simply are different ways we can affect the same object (for example the manuscript can be read, written, reviewed, etc.). In such cases we would like the overall similarity to be lower. Algorithm 1: Role mentions clustering action_clusters  grouping of actions by their normalized value object_clusters  grouping of objects by their normalized value similarity  ∞ while similarity > threshold do for each role cluster pair do if one element contains all terms of the other then merge clusters & relabel the smaller cluster end end pair  action or object cluster pair with the highest similarity similarity  the highest similarity merge clusters from pair & relabel the smaller cluster end To reflect these observations, we introduce an object weight which is the reciprocal of the number of distinct actions it is related to: ∀ ∈ 𝑤(𝑜) = |{𝑎 ∈ 𝐴 | (𝑎, 𝑜) ∈ 𝑅}| . Intuitively, an object with a small weight (such as “manuscript”) interacts with many different actions, in other words there are many actions that can be applied to it. We define the similarity between two actions as the sum of the weights of all the objects they share: ∀ , ∈ 𝑠 𝑎 , 𝑎 = ∑ ∈ ,( , ) ∈ ,( , ) ∈ 𝑤(𝑜). Intuitively, two actions will have high similarity if: 1) they share a lot of objects, and 2) the objects they share are “specific” (few distinct actions apply to them). An object that interacts with many actions will not contribute much to the action similarity. Examples of merged clusters include: “particip” and “perform”, “contribut” and “perform”, “assist” and “perform”, “manuscript” and “paper”, “carri” and “perform”, “experi” and “study”, “perform” and “undertook”, “manuscript” and “articl”. Fig. 4. The role graph resulting from automated clustering. The nodes represent action and object clusters (their labels are bags of stemmed terms). The width of edges represents the strength of the relation between action and object nodes. Less common roles were removed. The clustering procedure resulted in reducing the number of role clusters from 285 to 63. The following clusters were merged: 1. “particip” and “perform” 9. “manuscript” and “articl” 2. “contribut” and “perform” 10. “approv” and “read” 3. “assist” and “perform” 11. “made” and “perform” 4. “manuscript” and “paper” 12. “conduct” and “perform” 5. “project” and “study” 13. “perform” and “supervis” 6. “carri” and “perform” 14. “help” and “perform” 7. “experi” and “study” 15. “perform” and “plan” 8. “perform” and “undertook” The procedure made a few errors, merging for example: “approv” and “read”, “per- form” and “supervis”. The final graph is shown in Fig. 4. 3.2 Manual Correction To reduce the number of errors from automatic clustering, we manually inspected 63 clusters. This included removing some clusters and merging others. We also assigned role names to the clusters. The entire procedure resulted in 13 roles. The final set of 13 roles, as well as the fractions of mentions for every role, are presented in Fig. 5. Fig. 5: The final set of roles, showing the counts and fractions of the entire role mention set. 3.3 Annotating the Dataset We annotated the dataset of role mentions. More specifically, the dataset contains role mentions labelled with abstract roles. For example, the dataset might contain the entry: (“participated in, the analysis of microarray data”, data analysis). The resulting dataset is composed of the role mentions from the clusters, and the label for each role mention is the role name assigned to the mention’s cluster. This annotation approach differs from the typical approach, in which we would manually label each role mention in the dataset. Even though our approach still requires manual work, it was performed on the clusters, not each individual role mention. Since the clusters are much less numerous than the role mentions, our proposed approach is less labor intensive. 3.4 Roles Extraction from the Text This section describes our prototype of an automated extractor of authors’ roles from text. The extractor takes a contributions section as input and outputs a set of extracted roles. We used the previously developed preprocessing pipeline and discovered roles for this task. The extraction algorithm is composed of the following steps:  First, a set of role mentions is extracted from the text of the section. If the section is written in a natural language, this is done using OpenIE. In some rare cases we came across, the contributions section was not written in natural language, but rather con- tained a list of contributions in the following format (or a variation of it): “author1: role1, role2; author2: role3; ...”. In such cases we extract role mentions using regular expressions. Redundant mentions are then removed.  Next, each mention is represented as a feature vector. We use a binary bag-of-words representation, with 64 words corresponding to the object/action keywords (Table 1). Only the keywords that remained after manual cluster removal are used.  Finally, each mention is classified by a supervised Naïve Bayes model trained on the mention set generated previously. The final output is a set of author-role pairs. 4 Results 4.1 Roles Discovery Fig. 4 shows the graph resulting from the automated clustering procedure (before man- ual correction). The final corrected roles resulting from our study are: experimenting (1,743 instances, 17% of the entire role set), analysis (1,343, 16%), study design (1,132, 13%), interpretation (879, 10%), conceptualization (865, 10%), paper reading (823, 10%), paper writing (724, 8%), paper review (501, 6%), paper drafting (351, 4%), coordination (319, 4%), data collection (76, 1%), paper revision (41, 0.5%) and liter- ature review (41, 0.5%). Our final role set was manually compared to the existing taxonomy CRediT. It is important to note that our study was based on biomedical data only, while CRediT is a general-purpose taxonomy. As a result, some differences are to be expected. Table 2. Comparison of the roles discovered by our study and existing taxonomy CRediT. Similarities Differences Our study CRediT Our study CRediT Analysis Formal analysis Paper reading - Conceptualization Conceptualization Literature review - Experimenting Investigation Interpretation - Study design Methodology - Software Coordination Project administration - Validation Data collection Resources - Funding acquisi- tion Paper drafting/ Writing – original draft - Supervision Paper writing Paper review/ Writing - review & edit- Paper revision ing In general, the results are similar (Table 2). Five roles appear in both our clusters and CRediT. Our study resulted in four roles related to preparing the manuscript itself, while CRediT has only two such roles. Three roles discovered in our study (paper read- ing, literature review and interpretation) are not included in CRediT. 4.2 Roles Extraction To evaluate our role extractor, we manually annotated a test set of 100 contributions sections. At this point, we observed three new roles that were not discovered in our study: paper approving, supervision and funding acquisition. Since the classifier does not have any training data for these roles, they are never assigned. During the evaluation, for every document we compared the extracted author-role pairs to the ground truth pairs. A pair was marked as correctly extracted if identical to any pair in the ground truth. We obtained the following micro-averaged results: preci- sion 0.68, recall 0.48, F1 0.57. Table 3 presents the results for individual roles. Table 3. Precision, recall and F1 for individual roles. Role Precision Recall F1 Analysis .91 .53 .67 Conceptualization .75 .50 .60 Experimenting .22 .80 .34 Study design .77 .60 .67 Coordination 1.0 .35 .52 Data collection .58 .56 .57 Paper drafting .87 .54 .66 Paper writing .61 .41 .49 Paper review .95 .50 .66 Paper revision .93 .31 .46 Paper reading .81 .85 .83 Literature review .91 .83 .87 Interpretation .90 .51 .65 4.3 Error Analysis We manually analyzed mistakes made by the extractor in the test set, and found two types: false positives that lower precision (a subject-role pair incorrectly present in the extracted output), and false negatives that that lower the recall (a correct subject-role pair missing from the extracted output). We identified three sources of errors (Fig. 6):  Errors related to mention extraction from the text. That is, an incorrect mention is extracted, or a certain role mention is missing. These errors are responsible for 26% of false positives and 73% of false negatives.  Errors appearing during role discovery, related to incorrect cluster merging. These errors result in the lack of roles paper approving, supervision and funding acquisi- tion in the extractor’s output and are responsible for 21% of false negatives. Classification errors, resulting in assigning an incorrect role to the tuple. These errors are responsible for 74% of false positives and 6% of false negatives. The quality of the mention extraction has the biggest impact on the overall results, in particular recall. In a typical scenario, some mentions are missing from OpenIE output, which makes it impossible to extract specific subject-role pairs. Incorrect tuples also affect the second cause of errors. For example, we observed that in many cases, Stanford’s OpenIE tool extracts only one tuple from typical sen- tences similar to “All authors read and approve the final manuscript”: (“all authors”, “read”, “the final manuscript”). In this case, the missing mention related to approving the manuscript resulted in the failure to discover this role in the corpus. Finally, we observed that in some cases the classifier made the decision based on a single term such as “make”, which does not carry enough information for a correct classification decision. Additional feature selection procedures for the classifier might result in better classification performance. Fig. 6. The fraction of three error causes in types of errors (precision and recall errors). 5 Summary and Future Plans In this paper, we presented a study of author contributions sections obtained from pub- lications in biomedical disciplines. The results of our study include: 1) a set of roles discovered in the data in an unsupervised manner, and 2) a first prototype of a tool able to automatically extract the roles from the contributions section. We semi-automatically discovered the following roles: experimenting, analysis, study design, interpretation, conceptualization, paper reading, paper writing, paper review, paper drafting, coordination, data collection, paper revision and literature re- view. Three discovered roles (paper reading, literature review and interpretation) are not included in the existing contributor roles taxonomy CRediT. The proposed auto- mated role extractor is able to extract roles directly from the text with micro-averaged precision 0.68, recall 0.48 and F1 0.57. Our plans for future work include: testing alternative mention extraction approaches and tools; testing alternative classification algorithms; and examining the relationships between author orderings, H-index and the nature of contributions in a larger corpus than used in previous analyses [9, 10]. Acknowledgements This research was conducted with the financial support of Enterprise Ireland and the European Regional Development Fund (ERDF) under Ireland’s European Structural and Investment Funds Programme 2014-2020 under Grant Agreement No. CF/2017/0808-I at the ADAPT SFI Research Centre at Trinity College Dublin. The ADAPT SFI Centre for Digital Media Technology is funded by Science Foundation Ireland through the SFI Research Centres Programme and is co-funded under the Eu- ropean Regional Development Fund (ERDF) through Grant # 13/RC/2106. References [1] D. Tkaczyk, A. Collins and J. Beel, "Who Did What? Identifying Author Contributions in Biomedical Publications using Naïve Bayes," in JCDL, 2018. [2] D. Tkaczyk, P. Szostek, M. Fedoryszak, P. Dendek and L. Bolikowski, "CERMINE: automatic extraction of structured metadata from scientific literature," International Journal on Document Analysis and Recognition, vol. 18, no. 4, pp. 317-335, 2015. [3] P. Lopez, "GROBID: combining automatic bibliographic data recognition and term extraction for scholarship publications," Research and Advanced Technology for Digital Libraries, 2009. [4] A. Constantin, S. Pettifer and A. Voronkov, "PDFX: fully-automated pdf-to-xml conversion of scientific literature," DocEng, pp. 177-180, 2013. [5] I. Councill, C. Giles and M.-Y. Kan, "ParsCit: an open-source CRF reference string parsing package," in International Conference on Language Resources and Evaluation, 2008. [6] J. Beel, S. Langer, M. Genzmehr and C. Mueller, "Docear's PDF Inspector: Title Extraction from PDF Files," in Joint Conference on Digital Libraries, 2013. [7] V. Ilakovac, K. Fister, M. Marusic and A. Marusic, "Reliability of disclosure forms of authors' contributions," Canadian Medical Association Journal, vol. 176, no. 1, pp. 41-46, 2007. [8] M. K. McNutt, M. Bradford, J. M. Drazen, B. Hanson, B. Howard, K. H. Jamieson, V. Kiermer, E. Marcus, B. K. Pope, R. Schekman and others, "Transparency in authors’ contributions and responsibilities to promote integrity in scientific publication," Proceedings of the National Academy of Sciences, vol. 115, no. 11, pp. 2557-2560, 2018. [9] E. A. Corrêa Jr, F. N. Silva, L. d. F. Costa and D. R. Amancio, "Patterns of authors contribution in scientific manuscripts," Journal of Informetrics, vol. 11, no. 2, pp. 498-510, 2017. [10] T. V. Perneger, A. Poncet, M. Carpentier, T. Agoritsas, C. Combescure and A. Gayet-Ageron, "Thinker, Soldier, Scribe: cross-sectional study of researchers' roles and author order in the Annals of Internal Medicine," BMJ open, vol. 7, no. 6, p. e013898, 2017. [11] G. Angeli, M. J. J. Premkumar and C. D. Manning, "Leveraging Linguistic Structure For Open Domain Information Extraction," in ACL, 2015. [12] W. Wong, W. Liu and M. Bennamoun, "Ontology learning from text: A look back and into the future," ACM Comput. Surv, vol. 44, no. 4, pp. 20:1-20:36, 2012.