Optimizing Automated Term Extraction for Terminological Saturation Measurement

Abstract

Assessing the completeness of a document collection, within a domain of interest, is a complicated task that requires substantial effort. Even if an automated technique is used, for example, terminology saturation measurement based on automated term extraction, run times grow quite quickly with the size of the input text. In this paper, we address this issue and propose an optimized approach based on partitioning the collection of documents in disjoint constituents and computing the required term candidate ranks (using the c-value method) independently with subsequent merge of the partial bags of extracted terms. It is proven in the paper that such an approach is formally correct - the total c-values can be represented as the sums of the partial c-values. The approach is also validated experimentally and yields encouraging results in terms of the decrease of the necessary run time and straightforward parallelization without any loss in quality.

Keywords

• Automated term extraction
• terminological saturation
• partial c-value
• merged-partial c-value
• optimization

-

Ontology learning from texts is a developing research field that aims to extract domain description theories from text corpora. It is increasingly acknowledged as a plausible alternative to ontology development based on the interviews of domain knowledge stakeholders. One shortcoming of learning an ontology from texts is that the input corpus has to be quite big for being representative for the subject domain. Another shortcoming is that learning ontologies from text is expensive, in terms of taken time, as it involves the use of several algorithms, in a pipeline [ 1 ], that are computationally hard.

Automated term extraction (ATE) is an essential step at the beginning of the pipeline for ontology learning [ 1, 2 ], that is known to be bulky in terms of the increase of the run time with the growth of the input text corpus. Therefore, finding a way to reduce: (i) either the size of the processed text; or (ii) the time spent for term extraction; or (iii) both is of importance.

In our prior work [ 2, 3, 4, 5 ], we developed the ATE-based approach (OntoElect) that helps circumscribe the minimal possible representative part of a documents collection, which forms the corpus for further ontology learning. This technique is based on measuring terminological saturation in the collection of documents, which is computationally quite expensive in the terms of the run time.

In this paper, we present the approach, based on the partitioning of a document collection, which allows substantially reducing ATE run time in the OntoElect processing pipeline.

The remainder of the paper is structured as follows. In Sect. 2, we outline our OntoElect approach to detect terminological saturation in document collections describing a subject domain. In Sect. 3, we review the related work in ATE and argue for the choice of the c-value method as the best appropriate for measuring terminological saturation. In Sect. 4, we explain our motives to optimize the c-value method based on partitioning a document collection and present a formal framework for that. Section 5 reports on the setup and results of our experimental evaluation of the proposed optimization approach. Finally, we draw the conclusions and outline our plans for the future work in Sect. 6.

Background and Research Problem OntoElect is the methodology for learning a domain ontology from a statistically representative sub-collection ( ) of the complete collection of documents ( = { })1 describing this subject domain. The representativeness of a sub-collection is decided using a successive approximation method, based on measuring terminological saturation. In this method, sub-collections are incrementally extended by adding several ) documents to the previous sub-collection in the sequence.

Let

1, sub-collections, , = 1, … , 2, … , such , are chosen from the remainder of the that 0 = ∅ and

= , … be the sequence of incrementally extended document −1 ∪ { } ,

where using one of the possible ordering criteria [ 6 ]: chronological, reversed-chronological, bi-directional, random, or descending citation frequency. Let also 1, 2, … , , … be the bags of retained significant terms extracted from 1, 2, … , , …. In OntoElect, the measure of terminological difference ( ℎ ) is used for comparing the bags of terms , +1 retained from the successive ,

+1. It returns the difference as a real positive value. If, at some : (i) ℎ goes below the threshold of the statistical error ; and (ii) there is a convincing 2 ( 1 In OntoElect, we do not require the availability of this complete collection. Instead, we require that a substantial part of it is available, which presumably contains all the significant terms describing the subject domain. If so, it is further revealed that ⊂ .

This representative set of terms is denoted as the terminological basis ( evidence that it will never go above this threshold; then the difference (distance) between and hypothetical

is not higher than . Such a could be used as an -approximation of the representative set of significant terms describing the domain. ject domain. This , labelled further as , is denoted as the saturated term set, and ) of the sub. The differ , labelled further as

, is the saturated and any successive , including , is within the statistical the corresponding ence ( ℎ ) between error: ℎ ( ,

) < .

In our prior work, it has been demonstrated that ℎ is the measure, which can be effectively used for comparing terminological sets as vector representations of the semantic similarity/distance between document collections. However, one substantial shortcoming of this approach is that it is computationally expensive. Indeed, given an approximately fixed length of an increment { } and the increasing size of , the method processes more and more the same part of the collection with the growth of .

Therefore, the computational cost (run time) for measuring ℎ would have been substantially lowered if there was a way to process only the increments of the successive sub-collections. This processing is, essentially the ATE pipeline. Hence, the research problem is to prove that modifying the ATE processing pipeline for measuring terminological saturation to process: • Only the disjoint parts { } of a document collection • Instead of sub-collections

yields the same result and takes substantially less execution time. 3

Related Work in ATE

In the majority of approaches to ATE [ 7, 8 ] processing is done in two consecutive phases: linguistic and statistical. Linguistic processors, like POS taggers or phrase chunkers, filter out stop words and restrict candidate terms to n-gram sequences: nouns or noun phrases, adjective-noun and noun-preposition-noun combinations. Statistical processing is then applied to measure the ranks of the candidate terms. These measures are [ 9 ]: either the measures of unithood, which focus on the collocation strength of units that comprise a single term; or the measures of termhood, which point to the association strength of a term to domain concepts.

For unithood, the measures are used such as mutual information [ 10 ], log likelihood [ 10 ], t-test [ 7, 8 ], modifiability and its variants [ 11, 8 ]. The measures for termhood are either term frequency-based (unsupervised approaches) or reference corpora-based (semi-supervised approaches). The most used frequency-based metrics are TF/IDF [ 12, 13 ], weirdness [ 14 ], and domain pertinence [ 15 ]. More recently, hybrid approaches were proposed, that combine unithood and termhood measurements in a single value. A representative measure is c/nc-value [ 16 ]. C/Nc-value-based approaches to ATE have received their further evolution in many works: [ 7, 15, 17 ] to mention a few.

Linguistic processing is organized and implemented in a very similar fashion in all ATE methods, except some of them that also include filtering out stop words. Stop Domain- Superindepen- vizion dence (U/SS) (+/-)

+ U Method [Source] TTF [ 25 ] ATF [ 23 ] TTF-IDF [ 26 ] RIDF [ 27 ] C-value [ 16 ] Weirdness [ 14 ] GlossEx [ 21 ] TermEx [ 15 ] PU-ATR [ 18 ] + + + + + + +/words could be filtered out also at a cut-off step after statistical processing. Statistical processing is sometimes further split in two consecutive sub-phases of term candidate scoring, and ranking. For term candidates scoring, reflecting its likelihood of being a term, known methods could be distinguished by being based on (c.f. [ 12 ]) measuring occurrences frequencies (including word association), assessing occurrences contexts, using reference corpora, e.g. Wikipedia [ 18 ], topic modelling [ 19, 20 ].

The cut-off procedure, takes the top candidates, based on scores, and thus distinguishes significant terms from insignificant (or non-) terms. Many cut-off methods rely upon the scores, coming from one scoring algorithm, and establish a threshold in one or another way. Some others that collect the scores from several scoring algorithms use (weighted) linear combinations [ 21 ], voting [ 9, 3 ], or (semi-)supervised learning [ 22 ]. In our set-up [ 3 ], we do cut-offs after term extraction based on retaining a simple majority vote. Therefore, the ATE solutions, which perform cut-offs together with scoring, are not relevant for our approach.

Based on the evaluations in [ 9, 12, 23 ], the most widely used ATE algorithms, for which their performance assessments are published, are listed in Table 1. The table also provides the assessments based on the aspects we use for selection.

Comments to Table 1:

Domain Independence: “+” stands for a domain-independent method; “-“ marks that the method is either claimed to be domain-specific by its authors, or is evaluated only on one particular domain. We look for a domain-independent method.

Supervision: “U” – unsupervised; “SS” – semi-supervised. We look for an unsupervised method. Term Significance: “+” – the method returns a value for each retained term, which could further be used as a measure of its significance compared to the other terms; “-“ marks that such a measure is not returned or the method does the cut-off itself. We look for doing cut-offs later. Cut-off: “+” – the method does cut-offs itself and returns only significant terms; “-” – the method does not do cut-offs. We look for “-”.

U U U U SS SS SS SS

Measure(s) Term (Total) Frequency Average Term Frequency TTF+Inverse Document Frequency Residual IDF C-value, NC-value Weirdness Lexical (Term) Cohesion, Domain Specificity Domain Pertinence, Domain Consensus, Lexical Cohesion, Structural Relevance Nc-value, Domain Specificity

Term Cut- Precision Run Time Tool Signi- off (GENIA; (related to ficance (+/-) average) c-value

method) + + + + + + 0.70; 0.35 0.71; 0.33 0.75; 0.32 0.82; 0.51 0.71; 0.32 0.80; 0.49 0.73; 0.53 0.77; 0.56 0.77; 0.47 0.82; 0.48 0.70; 0.41 0.87; 0.46 0.78; 0.57 0.34 0.37 0.35 0.35 0.53 0.37 1.00 1.00 0.41 1.67 0.42 0.52

ATR4S JATE ATR4S JATE ATR4S JATE ATR4S JATE ATR4S JATE ATR4S JATE ATR4S JATE ATR4S

JATE 809.21 ATR4S

JATE Precision and Run Time: The values are based on the comparison of the two cross-evaluation experiments reported in [ 12 ] and [ 23 ]. Empty cells in the table mean that there was no data for this method in this experiment using this tool. Survey [ 12 ] used ATR4S [ 12 ] – an open-source software tool for automated term recognition (ATR) written in Scala (4S). It evaluated 13 different methods, implemented in ATR4S, on five different datasets, including the GENIA dataset [ 28 ]. Survey [ 23 ] used JATE 2.0 [ 23 ], free software for automated term extraction (ATE) written in Java (J). It evaluated nine different methods, implemented in JATE, on two different datasets, including GENIA. Hence, the results on GENIA are the baseline for comparing the precision. Two values are given for each reference experiment: precision on GENIA; average precision. Both [ 12 ] and [ 23 ] experimented with c-value method, which was the slowest on average for [ 23 ]. So, the execution times for c-value were used as a baseline to normalize the rest in the Run Time column.

Tool: The last column in the table names the tools used in the corresponding experiments.

The information in Table 1 supports the conclusion of [ 23 ] stating that c-value is the most reliable method. The c-value method obtains consistently good results, in terms of precision, on the two different mixes of datasets [ 23, 12 ]. It could also be noted that c-value is one of the slowest in the group of unsupervised and domain-independent methods, though its performance is comparable with the fastest ones. Still, c-value outperforms the domain-specific methods, sometimes significantly – as it is in the case of PU-ATR. Therefore, we have chosen c-value as the method for our experimental framework. 4

Motivation and Formal Framework ATE is known to be computationally expensive in the terms of run time versus the volume of input text. The c-value method that we have chosen for our terminological saturation measurement pipeline (Table 1) is more expensive than the other unsupervised and domain neutral methods. Furthermore, ATE implementations are often constrained2 in the volume of input text. Hence, reducing the volume of text to be processed by the method, and partitioning it in relatively small chunks, may substantially lower this expense and contribute to the better scalability of the solution. 4.1

Motivation

The c-value method [ 16 ], as mentioned in Sect. 3, is hybrid and combines linguistic and statistical steps applied to the entire document collection (text corpus). The method starts with the linguistic pipeline, which outputs the list of term candidate strings. It then continues with the statistical part, which computes significance scores for these term candidates as c-values. The diagram of the measured run time versus the volume 2 For example, the UPM Term Extractor software [ 29 ], which is based on the c-value method and used in our experiments, does not take in texts of more than 15 Mb in volume. of input text, provided in Sect. 5 (Fig. 3), in the case of the conventional pipeline illustrates, by run time values, the computational complexity of the c-value method.

Let us now consider a document collection as a composition of its disjoint parts.

Definition 1 (A partial collection and a partition of a document collection). tition of if the following conditions hold: , = 1, … , are the partial document collections of and { } = { } =1 is the parCondition 1:

= ⋃ =1 , Condition 2: ⋂ =1 = ∅. (1) more than if

> 1 • In the case of partial collections:

The linguistic part processes separate sentences. Therefore: (i) its computational complexity is the function of the number of sentences in a document collection; and (ii) could at least be parallelized, which results in the runtime gain of times. the partial collections , = 1, … , (Definition 1) could be processed independently and the outputs further merged. Hence, applying the linguistic step to the partition of

In the case of OntoElect pipeline, the linguistic step is iteratively applied to the incrementally enlarged datasets (see Sect. 2). Therefore, the same chunks of text are processed many times. Let us suppose that contains documents and increment to enlarge datasets. Then, the number of documents to be processed is: = / is the In the case

of + 2 ∙ + ⋯ + ∙ = ∙ the 1+2+⋯+ incrementally enlarged

datasets: ≈ ( + 1)/2 ∙ , which is substantially • •

Hence, processing partial collections instead of incrementally enlarged datasets gives a substantial additional gain in runtime, which is ( +1 − 1) ∙ times.

Similarly, it might be reasonable to apply the statistical step of the pipeline not to the incrementally enlarged datasets, but to partial collections. However, it is not 2 straightforward that: • Computing c-values for the terms extracted from the partial collections; and

Merging further these bags of terms with their significance scores will give the same result as applying the statistical step to incrementally enlarged datasets. In the remainder of this section, we prove that partitioning c-value computation with later results merging gives correct results.

C-value [ 16 ], further labelled as

in formulae and equations, is built using several statistical characteristics of the corresponding term candidate string. These characteristics are: • The total frequency (number) of occurrence(s) of the candidate string in the docu• The frequency (number) of occurrence(s) of the candidate string as a part of other ment corpus longer candidate terms • The number of these longer candidate terms • The length of the candidate string (in the number of words) these . Then a (complete)

of is denoted [ 16 ] as follows:

Let: be a term candidate string; | | – the length of in words; – a longer term candidate string in which is nested as a sub-string; (. ) – the frequency (number) of occurrence(s) of a term candidate string in a collection of textual documents ; – the set of extracted term candidate strings that nest ; and ( ) – the number of ( ) = 2(| |) ∙ ( ) 2(| |) ∙ ( ) − 1 ( ) ∑ ∈ ( )

ℎ .

(2) 4.2

Merged Partial C-values

extracted from the partial document collection is computed as:

Definition 2 (Partial c-value). The partial c-value of the term candidate string ℎ 2(| |) ∙ ( ) ( ) = ∑ partial collections of : the number of occurrences of or in . where: is the set of term candidate strings , that nest , extracted from ; (. ) is

Lemma 1 (The total frequency of nested occurrences). The total value of the frequency of nested occurrences, in , of a term candidate string in longer term candiis the sum of the total frequency values of nested occurrences in all ( ) = ∑ ∈ ( ) = ∑ =1 ∑ ∈ ( ) = ∑ =1( ( )).

(4) Proof. It implies from Definition 2 (of partial c-value), that ( ) is the total number of occurrences of the term candidate string in all longer term candidate strings extracted from the partial collection . The number of these longer term candidate strings equals to ( ). Due to the disjointness of the partial collections 2 of Definition 1), ( ) = ∑ =1 ( ). Therefore, and due to the Condition 1 of Defi(Condition nition 1, the total number of occurrences of in all extracted from is: , (3)

( ) = ∑ ∈ ( ) = = ∑ ∈ ∑ =1 ( ) = ∑ ∈⋃ =1

(∑ =1 ( )) =

= ∑ =1 ∑ ∈ ( ) = ∑ =1( ( )).

□ (5)

Definition 3 (Merged partial c-value). The merged partial c-value of the term candidate string is computed as:

( ) = ∑ =1 ( ). tions , = 1. … , of .

The following Theorem 1 allows computing ( ) for the whole collection based on the merging of the known partial c-values ( ), = 1, … , for the partial collec as { } = { } =1, which means that Conditions 1 and 2 (1) hold, then Theorem 1 (Equality of and

). If a document collection is partitioned ( ) = ( ) (6) (7) (8) 2( ).

□

( ) = ∑ =1 ( ) = = ∑ =1 an :

Proof. The proof is structured in three cases: (1) is never nested in ; (2) ∀ , is nested at least once and at least in one ; and (3) is nested in for some .

Case 1: not nested. If, ∀ = 1, … , , extracted from is not nested in any extracted from , then is not nested in any extracted from . Therefore, for such = ( ) = ∑ =1 ( ) = ∑ =1

2(| |) ∙ ∑ =1 ( ) =

2(| |) ∙ ( ) = 2(| |) ∙ ( ) = ( ), due to Conditions 1 and 2 (2) and the definition of (. ). (ii)

∈ ⊂ – because ⊂ due to condition 1. Therefore:

Case 2: all nested. If, ∀ = 1, … , , extracted from is nested in an extracted from , then: (i) this (extracted from ) is nested in this (extracted from ); and 1

1 2(| |) ∙ ( ) − ( ) ∑ ∈ ( ) = ( )

Here “≈|ℎ1” stands for hypothetically approximately equal. The hypothesis ℎ1 about the approximate equality in ∑ ever, asymptotically, ( ) = ( ( )) and ( ) =

( ) 1 =1 ( ) ≈ ( ) 1 . Formally, ∑

1 =1 ( ) > ( ) 1 . Howdue to: (i) the overlaps in ; and (ii) possible nestings in several instances of . Therefore, the influence of those denominators in (8) becomes lower with the growth of the volume of and its partial collections . This is a promise that ℎ1 might be true.

Case 3: is sometimes nested in . There exist several partial collections , for which Case 2 is applied. For the rest of partial collections Case 1 is applied. In this situation two partial sums – subsets of the partition of . Similarly to Case 2, 1( ) and 2( ) – are computed for these disjoint ( ) ≈|ℎ1 1( ) + Hence, if the hypotheses ℎ1 holds true, Cases 1-3 prove Theorem 1.

For checking ℎ1, complete ( ) and merged partial ( tally computed and compared, as presented in Sect. 5. on the partitioning of a document collection.

A straightforward corollary from

Theorem 1 is that c-values do not depend Corollary 1 (Size of a partial collection). Let { } =1; =1 ; ≠ be two dif ) c-values are experimenferent partitions of a document collection . Then: ∀ , ( )| = ( )| ≈|ℎ1 ( ), (9) the partition { } of ; string computed for the partition { } of .

( )| where: is a term candidate string extracted from the document collection ; ( )|{ } is the merged partial c-value of the term candidate string computed for is the merged partial c-value of the term candidate

Based on Corollary 1, the size of a partial collection ∈ { } may be reasonably chosen based on the specifics of the problem and available hardware resources – RAM in particular. One possible scenario and problem might be extracting terms from a stream of textual documents, like blog posts or tweets. In this setting, the size of a partial collection has to be smaller than the size of the stream window. Algorithm MPCV. Merge partial c-values from two Bags of Terms Input: Ti, Ti+1 – the bags of retained significant terms.

Each term Ti.term is accompanied with its Ti.pcv.

Ti, Ti+1 are sorted in the descending order of Ti.pcv, Ti+1.pcv. Output: the bag of terms Ti+1 with merged Ti.pcv into Ti+1.pcv for every term 1. 2. 3. 4. 5. 6. 7. 8. 9.

resort := .FALSE. for k := 1 to |Ti| match := .FALSE. for m := 1 to |Ti+1| if (.NOT. match) if (Ti.term[k] = Ti+1.term[m])

then begin Ti+1.pcv[m] += Ti.pcv[k]; match := .TRUE.; end then begin append(Ti.term[k]+Ti.pcv[k], Ti+1); resort := .TRUE.; end end for 10. end for 11. if (resort) then sort(Ti+1, Ti+1.pcv, desc)

The MPCV algorithm (Fig. 1) is used for merging partial c-values in the bags of significant terms retained from the textual datasets representing the partial document collections of the complete document collection. 5

Experimental Evaluation

The idea of experimental evaluation is to compare the conventional and optimized processing pipelines based on checking: • The correctness. Are the merged partial c-values computed using the optimized pipeline practically the same as the c-values computed by the conventional pipeline? • Execution time. What is the difference in the duration of the extraction of the same bags of retained significant terms between the conventional and optimized pipelines? Checking correctness validates the hypothesis ℎ1 (Sect. 4) to fully prove Theorem 1. If ℎ1 holds true, the optimized processing pipeline could be used for extracting terms in the process of measuring terminological saturation in document collections. Comparing the execution times of the conventional and optimized processing pipelines allows assessing the efficiency of the optimized pipeline. collection for the optimized pipeline The document collection used in our experiments is the DMKD-300 collection, which contains the subset of (300) full text articles from the Springer journal on Data Mining and Knowledge Discovery3 published between 1997 and 2010. These papers have been automatically pre-processed to plain texts [ 24 ] and have not been cleaned. Therefore, the resulting datasets, representing partial collections, were moderately noisy. We have chosen the increment ( ) for generating the datasets to be 20 papers. Hence, based on the available texts, we have generated, using our Dataset Generator software (Sect. 5.3):

The descending citation frequency (DCF) order [ 6 ] of adding documents to partial collections has been used in both cases. Our experimental workflow is appropriately supported by the developed and used instrumental software. The toolset is concisely presented in Table 2. 3 https://link.springer.com/journal/10618 4 DMKD-300 collection in plain texts: http://dx.doi.org/10.17632/knb8fgyr8n.1#folder637dc34c-fa29-4587-9f63-df0e602d6e86; incrementally enlarged datasets generated of these texts: http://dx.doi.org/10.17632/knb8fgyr8n.1#folder-b307088c-9479-43fb-8197a12a66ff685b 5 The partition of the DMKD-300 collection: https://github.com/OntoElect/Data/blob/ master/DMKD-300-DCF-Part.zip

Phase /

Task

Pre-Processing Phase

Generate Datasets Dataset Generator

Post-processing Phase

Compute Terminological Differen-ces

Terms Extraction Phase

Extract Terms

UPM Term Extractor [ 29 ] the folder with plain text datasets Merge par- MPCV tial c-values the folder with the bags of terms; the list the folder with the bags of terms with of files to be processed merged c-values; the the folder with the

Java, bags of terms; the table https://github.co only, c-value with run-time per bag m/ontologylearn method [ 16 ],

English texts of terms times. The set-up of our experiments includes the configuration of the execution flow in two parallel processing pipelines – conventional and optimized, as pictured in Fig. 2.

The conventional pipeline implements the processing of incrementally extended document sub-collections, as explained in Sect. 2. It takes in the files of the document collection in the specified (DCF) order and generates the incrementally extended datasets (Sect. 5.1) using the dataset generator (Table 2). At the next step, the datasets are fed into the term extractor software (Table 2) which outputs the bags of extracted terms and measures run times .

The optimized pipeline implements the processing of the partitioned document subcollections as explained in Sect. 4. It takes the files of the document collection in the same order (DCF) and generates partition datasets (Sect. 5.1) using the dataset generator (Table 2). At the next step, the datasets are fed into the term extractor software (Table 2) which outputs the bags of extracted terms and measures run times. At the subsequent step, the extracted sets o terms are fed into the merger module (Table 2) which applies the MPCV algorithm (Sect. 4.2) consequently to the pairs { , +1} as pictured in Fig. 2. As a result, the merged bags of terms = ⋃ =1 are generated.

The run times of the merge operation are also measured. The required total run times ) are computed as the sums of the respective term extraction and merge run

Document Collection (Plain Texts) Conventional Pipeline

Generate Datasets inc

D1 D2 … Di-1 … Dn-1 D1 D2 D3 + inc + inc Extract & Retain

Significant Terms T1 T2 T3 …

… Di + inc Ti

Dn + inc … inc

Optimized Pipeline

Generate Datasets D1 D2 D3

Di

Dn …

… Extract & Retain

Significant Terms measure Tn

T1 T2 T3

Merge pcv in the Bags

of Terms 1 2 3 … …

Ti … …

Tn measure

Pairwise Compare

Bags of Terms ℎ (…), i =1, …, n After executing these two parallel branches, if ℎ1 (Sect.4) holds true, coming from the conventional pipeline and contain very similar sets of terms with approximately the same c-values. This is checked by applying the THD algorithm [ 4 ] implemented in the Baseline THD module (Table 2). THD is applied: (i) to the pairs { , +1} and { , +1} for comparing satu ration curves for conventional and optimized cases; and (ii) to the pairs { , } for computing terminological difference between hypothetically the same sets of terms.

All the computations, except term extraction, have been run on a Windows 7 64-bit HP ZBook 17 G3 PC with: Intel® Core™ i7-6700HQ CPU, E7400 @ 2.60 GHz; 8.0 Gb on-board memory; NVIDIA Qadro M3000M GPU. Term extraction has been run on an Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz, 64 cores, 256GB server.

coming from the optimized pipeline have to terms resulting from the optimized pipeline ( − , column in Table 3 and Merged Partial curve in Fig. 3(a)) and conventional pipeline ( − , column in Table 3 and Incremental curve in Fig. 3(a)) are practically the same, except the last two measurements. The deviation at the tail could be explained that regular noise is accumulated differently in these two cases. A nice side result in this context is that the optimized pipeline using merged partial c-values accumulates less regular noise. Fig. 3(b) clearly pictures

and old the fact that the difference between the bags of terms does not exceed approximately 1/3 of the individual term significance threshthat is used to cut-off insignificant terms. In the combination, these two observations reliably prove6 our hypothesis ℎ1 (Sect. 4).

The comparison of the run times presented in Table 3 and Fig. 4 clearly demonstrates that the optimized pipeline, with near to constant values, outperforms the conventional pipeline significantly. 6

One may argue that the reported experiment is just an experiment with one document collection. Hence for a different document collection the results might be different regarding the validity of ℎ1. Our counter-argument is that the computation of c-values is collection and domain-independent. Furthermore, the terms with the same c-value are randomly distributed in the documents of the collection.

(a) Saturation measurements (b) Terminological differences The contribution of this paper is the proposal of computing significance scores (c-values) for term candidates extracted from a document collection using not the incrementally extended datasets, representing sub-collections, but the partitions of the collection. It has been proven formally, up to the validity of the ℎ1 hypothesis, that this optimized approach is correct – i.e. gives practically the same results. The hypothesis has been validated experimentally, by comparing the outputs coming from the conventional and optimized processing pipelines.

The experiment also clearly showed that the proposed way of text processing very substantially outperforms the conventional approach. The run times measured while processing partitions of the document collection remained almost constant in consecutive steps. A tiny increase was observed due to the very small overhead for merging the bags of terms extracted from the partition datasets. Yet one more advantage of the proposed approach is that partition datasets could be processed independently as these do not overlap in data. Hence, the optimized pipeline is straightforwardly parallelizable. This fact opens the way to process real world document collections at industrial scales for finding terminological cores within these collections. Choosing a proper partition size also removes the limitation of many software term extractors on the volume of input data.

Our plan for the future work is to apply the optimized processing pipeline to detect terminological saturation in the industrial size paper collection in the domain of Knowledge Management.

References

1. Wong , W. , Liu, W. , Bennamoun , M. : Ontology learning from text: a look back and into the future . ACM Comput. Surv. , 44 ( 4 ), Article 20 , 36 pages ( 2012 ). http://doi.acm. org/10 .1145/2333112.2333115

2. Ermolayev , V.: OntoElecting requirements for domain ontologies. The case of time domain . EMISA Int J of Conceptual Modeling 13 ( Sp . Issue), 86 -- 109 ( 2018 )

3. Tatarintseva , O. , Ermolayev , V. , Keller, B. , Matzke , W.-E.: Quantifying ontology fitness in OntoElect using saturation- and vote-based metrics . In: Ermolayev, V. , et al. (eds.) Revised Selected Papers of ICTERI 2013 , CCIS , vol. 412 , pp. 136 -- 162 ( 2013 )

4. Chugunenko , A. , Kosa , V. , Popov , R. , Chaves-Fraga , D. , Ermolayev , V. : Refining terminological saturation using string similarity measures . In: Ermolayev, V , et al. (eds.): Proc. ICTERI 2018 . Volume I: Main Conference, Kyiv, Ukraine, May 14 -17, 2018 , CEUR-WS vol. 2105 , pp. 3 -- 18 ( 2018 , online) http://ceur-ws. org/ Vol- 2105 /10000003.pdf

5. Ermolayev , V. , Batsakis , S. , Keberle , N. , Tatarintseva , O. , Antoniou , G.: Ontologies of time: review and trends . International Journal of Computer Science and Applications 11 ( 3 ), 57 -- 115 ( 2014 )

6. Kosa , V. , Chaves-Fraga , D. , Naumenko , D. , Yuschenko , E. , Moiseenko , S. , Dobrovolskyi , H. , Vasileyko , A. , Badenes-Olmedo , C. , Ermolayev , V. , Corcho , O. , and Birukou , A. : The influence of the order of adding documents to datasets on terminological saturation . Technical Report TS-RTDC-TR-2018-2-v2 , 21 . 11 . 2018 , Dept. of Computer Science, Zaporizhzhia National University, Ukraine, 72 p. ( 2018 )

7. Fahmi , I. , Bouma , G., van der Plas, L.: Improving statistical method using known terms for automatic term extraction . In: Computational Linguistics in the Netherlands, CLIN 17 ( 2007 )

8. Wermter , J. , Hahn , U. : Finding new terminology in very large corpora . In: Clark, P. , Schreiber , G . (eds.) Proc.3rd Int Conf on Knowledge Capture, K-CAP 2005 , pp. 137 -- 144 , Banff, Alberta, Canada, ACM ( 2005 ) http://doi.org/10.1145/1088622.1088648

9. Zhang , Z. , Iria , J. , Brewster , C. , Ciravegna , F. : A comparative evaluation of term recognition algorithms . In: Proc. 6th Int Conf on Language Resources and Evaluation , LREC 2008 , Marrakech, Morocco ( 2008 )

10. Daille , B. : Study and implementation of combined techniques for automatic extraction of terminology . In: Klavans, J. , Resnik , P. (eds.) The balancing act: combining symbolic and statistical approaches to language , pp. 49 -- 66 . The MIT Press. Cambridge, Massachusetts ( 1996 )

11. Caraballo , S. A. , Charniak , E.: Determining the specificity of nouns from text . In: Proc. 1999 Joint SIGDAT Conf on Empirical Methods in Natural Language Processing and Very Large Corpora , pp. 63 -- 70 ( 1999 )

12. Astrakhantsev , N.: ATR4S: toolkit with state-of-the-art automatic terms recognition methods in scala . arXiv preprint arXiv:1611.07804 ( 2016 )

13. Medelyan , O. , Witten , I. H. : Thesaurus based automatic keyphrase indexing . In: Marchionini, G. , Nelson , M. L. , Marshall , C. C. (eds.) Proc. ACM/IEEE Joint Conf on Digital Libraries, JCDL 2006 , pp. 296 -- 297 , Chapel Hill , NC , USA, ACM ( 2006 ). http://doi.org/10.1145/1141753.1141819

14. Ahmad , K. , Gillam , L. , Tostevin , L.: University of surrey participation in trec8: Weirdness indexing for logical document extrapolation and retrieval (wilder) . In: Proc. 8th Text REtrieval Conf, TREC-8 ( 1999 )

15. Sclano , F. , Velardi , P.: TermExtractor: A Web application to learn the common terminology of interest groups and research communities . In: Proc. 9th Conf on Terminology and Artificial Intelligence , TIA 2007 , Sophia Antipolis , France ( 2007 )

16. Frantzi , K. T. , Ananiadou , S.: The c/nc value domain independent method for multi-word term extraction . J. Nat. Lang. Proc. 6 ( 3 ), 145 -- 180 ( 1999 ). http://doi.org/10.5715/jnlp.6.3_ 145

17. Kozakov , L. , Park , Y. , Fin , T. , Drissi , Y. , Doganata , Y. , Cofino , T. : Glossary extraction and utilization in the information search and delivery system for IBM Technical Support . IBM System Journal 43 ( 3 ), 546 -- 563 ( 2004 ). http://doi.org/10.1147/sj.433.0546

18. Astrakhantsev , N.: Methods and software for terminology extraction from domain-specific text collection . PhD thesis , Institute for System Programming of Russian Academy of Sciences ( 2015 )

19. Bordea , G. , Buitelaar , P. , Polajnar , T. : Domain-independent term extraction through domain modelling . In: Proc. 10th Int Conf on Terminology and Artificial Intelligence , TIA 2013 , Paris, France ( 2013 )

20. Badenes-Olmedo , C. , Redondo-García , J. L. , Corcho , O. : Efficient clustering from distributions over topics . In: Proc. K-CAP 2017 , ACM, New York, NY, USA, Article 17 , 8 p. ( 2017 )

21. Park , Y. , Byrd , R. J. , Boguraev , B. : Automatic glossary extraction: beyond terminology identification . In: Proc. 19th Int Conf on Computational linguistics , pp. 1 -- 7 . Taipei, Taiwan ( 2002 ). http://doi.org/10.3115/1072228.1072370

22. Nokel , M. , Loukachevitch , N.: An experimental study of term extraction for real information-retrieval thesauri . In: Proc 10th Int Conf on Terminology and Artificial Intelligence , pp. 69 -- 76 ( 2013 )

23. Zhang , Z. , Gao , J. , Ciravegna , F. : Jate 2.0: Java automatic term extraction with Apache Solr . In: Proc.LREC 2016 , pp. 2262 -- 2269 , Slovenia ( 2016 )

24. Kosa , V. , Chaves-Fraga , D. , Naumenko , D. , Yuschenko , E. , Badenes-Olmedo , C. , Ermolayev , V. , Birukou , A. : Cross-evaluation of automated term extraction tools by measuring terminological saturation . In: Bassiliades, N. , et al. (eds.) ICTERI 2017 . Revised Selected Papers . CCIS , vol. 826 , pp. 135 -- 163 ( 2018 )

25. Justeson , J. , Katz , S. M.: Technical terminology: some linguistic properties and an algorithm for identification in text . Natural Language Engineering 1 ( 1 ), 9 -- 27 ( 1995 ). http://doi.org/10.1017/S1351324900000048

26. Evans , D. A. , Lefferts , R. G.: Clarit-trec experiments . Information processing & management 31(3) , 385 -- 395 ( 1995 ). http://doi.org/10.1016/ 0306 - 4573 ( 94 ) 00054 - 7

27. Church , K. W. , Gale , W. A. : Inverse document frequency (idf): a measure of deviations from Poisson . In: Proc. ACL 3rd Workshop on Very Large Corpora , pp. 121 -- 130 , Association for Computational Linguistics, Stroudsburg, PA, USA ( 1995 ). http://doi.org/10.1007/ 978 - 94-017-2390-9_ 18

28. Kim , J.-D. , Ohta , T. , Teteisi , Yu, Tsujii, J.: GENIA corpus - a semantically annotated corpus for bio-textmining . Bioinformatics . 19 ( suppl . 1), i180 -- i182 ( 2003 )

29. Corcho , O. , Gonzalez , R. , Badenes , C. , Dong , F. : Repository of indexed ROs . Deliverable No. 5 .4. Dr Inventor project ( 2015 )