We present work on the task of reducing noise in nominal terminology extraction. Based on a comparative evaluation of statistical measures aimed at capturing domain specificity, we propose strategies to increase the typically quite low accuracy of classical hybrid nominal multi-word term extraction. Our experiments on a set of German do-it-yourself instruction texts show that using linguistic filters that determine the right span of the MWE before applying a suitable combination of statistical measures improves results.
The automatic extraction of terminology from domain-specific text is a task that has gained interest in the research community over the last twenty years. It is an important prerequisite for applications such as ontology creation or knowledge extraction from texts.
The work presented here is part of a project that deals with knowledge extraction and ontology creation from German texts from the do-ityourself domain. As a first step, we aim at high quality terminology extraction of nominal candidates, as these describe the objects of the domain, followed by the extraction of verbal items and verb+complement patterns before we bring them together to build up partial ontologies of the domain. This paper describes strategies to reduce noise in nominal term extraction.
We consider single-word terms and multi-word terms, but focus on the latter because the extraction of multi-word terms (MWTs) is more difficult. As they are of variable length it is in many cases nontrivial to ensure the correct span of the term. We aim at extracting noun phrases (NPs) of different levels of complexity, such as adjective and noun combinations as well as NPs containing a genitive or prepositional modifier.
Nominal terms may contain embedded prepositional phrases (PPs), such as in example (1). (1) Bohrer mit Diamantspitze1
(drill with diamond bit)
However, we do not want to extract PPs that are not syntactically attached to a term, e.g. because they are verb-dependent, such as in example (2). (2) die *Oberfl a¨che mit Lein o¨lfirnis bedecken (cover the *surface with linseed oil varnish) Thus, one of the noise reduction steps is to ensure that the extracted nominal candidates are syntactically valid, and do not cover too long spans.
There are also cases where the term extraction
may return too short candidates. Sometimes, the
extracted terms only occur as part of bigger terms,
and are not valid on their own, such as in
example (3).
(3) elektromagnetisch *angetriebene Spritzpistole
(electromagnetically *operated spray gun)
There are both statistical and hybrid approaches
to term extraction
1Extracted term candidates are underlined. The * here denotes wrongly extracted MWT candidates. (2004) investigated the use of association measures for this task and came to the conclusion that these standard measures are outperformed by more sophisticated approaches. They do not focus on domain-specificity (“termhood”) and thus do not perform better at terminology extraction than mere frequency based approaches (Pazienza et al. (2005), confirmed by our own experiments, where the maximal F1-score obtained with association measures is 0.45).
Termhood is addressed through statistical measures that only use the candidate’s frequency in a domain-specific corpus (e.g. Frantzi et al. (2000)), as well as by measures based on a comparison of a candidate’s frequency in a domain and in generallanguage corpora (cf. Ahmad et al. (1992) and section 2.3). However, among others due to data sparseness in small size specialized corpora, both approaches perform much better on single-word terms (SWTs) than on MWTs.
Most hybrid systems (linguistic pattern-based search for candidates plus statistical ranking) often do not address variable length and syntactic validity satisfactorily (with a few noteworthy exceptions, cf. for example Chen et al. (2008)): part-of-speech (POS) sequence patterns are typically flat and cannot identify phrase boundaries and grammatical functions (cf. example (2)). As the POS patterns do not model phrase structure, they may cut off essential parts from a multi-word, returning unattested candidates (cf. example (3)).
We address the above issues by means of a three-step approach which modifies and extends the classical hybrid scenario: (i) nominal candidates are selected via part-of-speech patterns; (ii) they are filtered wrt syntactic validity and embedding and finally (iii) ranked according to statistical measures that involve a comparison between specialized and general-language corpus. The system with which we experiment extracts lemma combinations, morphosyntactic properties and textspecific metadata. Our method is evaluated on a 2.7 million word corpus of German do-it-yourself (DIY) instruction texts against a gold standard.
The main contributions of this paper are a study on the suitability of standard statistical measures for the extraction of nominal single- and multiword terms, as a basis for further adaptations and methods to improve the noise-silence ratio, based on experiments with linguistic filters (we use parsing information to ensure the MWE is a valid nominal phrase (NP)), and experiments on the combination of statistical measures. We believe that the methods we propose are generalizable to other domains of specialization and to other languages. More experiments will however be needed to confirm this.
In the present work, we only deal with nominal candidates. To maximize recall on (comparatively) small specialized corpora, we use POS patterns that account for basic terms2 (N, Adj N, N P N, N D Ngenitive) and for their potential variants (step (i) in the summary above). The set of patterns is described by the regular expressions given below. – (Adv? Adj? Adj)? N – (N D)? (Adv? Adj)? N P D? (Adv? Adj)? N – (Adv? Adj)? N D (Adv? Adj)? Ngenitive
These patterns are flat and thus do not
adequately represent syntactic structure. In particular,
they cannot distinguish between cases (a) where
NP and PP are sister nodes vs. (b) where the PP
is embedded in the NP (cf. examples (1) and (2)
above). Thus, step (ii) is added to remove noise:
we exclude items from our candidate set which are
syntactically invalid (too long ones) by checking
phrase boundaries and we use the C-value score
Candidates covering too long spans typically occur when part of the extracted MWT is actually attached to the verbal phrase, e.g. in example (4) and example (5). (4) die *Schablone mit Farbe bespru¨hen (spray the *template with paint) (5) ein *Loch in die Wand bohren (drill a *hole into the wall) 2POS tags: N-noun, Adj-adjective, P-preposition, Adv-adverb, D-determiner.
We filter these by using the dependency parser
mate
The boundary violation filter works as follows: If an instance of a MWT candidate comprises two sister phrases, i.e. if the POS sequence identified goes beyond the end point of an NP, it is not counted as a valid occurrence of the respective lemma sequence. As an example for a violation, consider the following MWT candidate where ‘ein Loch’ and ‘in die Wand’ are sister phrases (‘ein Loch in die Wand bohren’ (drill a hole into the wall), (5)), whereas in the example (6) (‘Bohrer mit Diamandspitze benutzen’ (use a drill with diamond bit)) there is no violation.
(5) (6)
NP ein Loch
VP PP
VP in die Wand 2.2
An example to show the necessity for an accurate treatment of nested terms was found in our extraction result: ‘zuga¨ngliche Stelle’ (accessible place) and ‘schlecht zuga¨ngliche Stelle’ (poorly accessible place). It should be obvious that from occurrences of the latter term we do not want to extract the former.
Thus, with nested MWEs, not all fragments of
a longer expression might be suitable candidate
terms. C-value
C(a) = (log2 |a| ⇤ f (a) log2 |a| ⇤ f (a) if a not nested PbP2T(Taaf)(b) otherwise (1) |a| = term length of a (number of words) f(.) = frequency in the domain corpus Ta = set of longer candidate terms that contain a
P (Ta) = number of these longer candidate terms
Furthermore, C-value reflects the idea that longer sequences have a tendency to be more (domain-) specific than shorter ones.
Based on this, we consider German noun
compounds as pseudo-MWEs and compute the term
length |a| from formula 1 by using the result of
compound splitting as produced by the compound
splitting tool COMPOST
termlength(a) = X(1 + log(cslen(w))) (2) w2 a w = a word cslen(w) = number of compound elements in w
Frantzi and Ananiadou (1996) propose C-value as a termhood measure using only the frequencies in the domain corpus; thus, most general-language noise cannot be filtered out. We therefore suggest to use C-value as a corrected frequency and to combine it with further statistical measures. 2.3
In section 4, we will compare the following statistical measures designed to rank candidate terms by domain-specificity. A detailed description of these measures is given in Scha¨fer (2015). As these measures place general-language candidates V bohren
V benutzen
N Bohrer
NP
PP mit Diamantspitze
The candidate sequence is not removed from the list of possible candidate terms, as other occurrences might not have been analyzed as violating syntactic boundaries. The filter is thus a “soft” one as it only affects the frequency of the lexeme combination candidate. We also experiment using a “hard” filter, where the lexeme combination candidate is removed altogether.
3In the current experiments only for subject and object
phrases.
at the bottom of the list, a selection of top
candidates shows a reduced amount of noise. The
measures are defined as follows using the domain
frequencies f , the general-language frequencies F as
well as the sizes of the corpora: s for the domain
corpus and S for the general-language corpus.
• Weirdness ratio for domain specificity (DS)
f /s
Weirdness = F /S
(3)
• Corpora-comparing log-likelihood (LL)
✓ LL = 2 f ⇤ log ✓ f ◆
Ef + F ⇤ log ✓ F ◆◆
EF (4) With Ef = s⇤ (f+F ) and EF = S⇤ (F +f) .
s+S S+s
• Contrastive Selection via Heads (CSvH)
cwct = log(fh(ct)) ⇤ log( ) ⇤ fct (5)
S
Fh(ct)
• Term Frequency Inverse Term Frequency
(TFITF)
wt = log(f (t)) ⇤ log (6)
S
F (t)
4As the LL formula is obviously symmetric in the two
corpora, we multiplied the result for candidates by 1 if their
relative frequency in the domain is smaller than the one in
general language, in order to place candidates with a
significantly high general-language frequency at the bottom of the
list.
• Contrastive Selection of multi-word terms
(CSmw)
Tool. We used the TTC5 (Terminology extraction,
translation tools and comparable corpora
(20102012)) term extraction research prototype
The pipeline involves the following components: corpus pre− processing pattern search ranking
term candidate list • Pattern-based term candidate extraction: use of simple as well as extended POS-based patterns to identify term candidates;for the patterns used see section 2. 5TTC-project: http://www.ttc-project.eu/ 6http://www.sdl.com/de/cxc/language/ terminology-management/multiterm/ # tokens 62,131 6,868 5,150 15,104 35,302 2,160,008 444,381 text do-it-yourself handbook encyclopedia entries list of FAQs with answers tips and tricks for do-it-yourselfers marketing texts user generated project descriptions user generated wiki content
• Ranking: sorting of the candidate lists produced by the preceding step, according to different measures (cf. section 2.3).
Domain corpus. We use a corpus of expert and user-generated German texts from the DIYdomain, consisting a.o. of manuals, practical tips, marketing texts and project descriptions (cf. table 1). This corpus is highly heterogeneous since the domain texts were acquired by various methods resulting in fundamentally different types of texts. The texts also differ with regard to the level of expertise of the author and the intended reader. Some texts are written by a domain expert as instructions for users and some are user-generated context.
As the texts differ wrt authorship and text style, several statistical measures are implemented in order to identify different properties of terms providing multiple lists of term candidates. A domain expert can then select the most relevant lists for the construction of a terminological representation of the domain language. For the experiments presented in this work we treated the corpus as a single unit. A source identifier was included as meta data annotation. In future work on a larger version of the corpus, subsets by text type and author/intended reader may be analyzed separately.
SdeWaC corpus
N 4,238 Kreissa¨ge
(circular saw) Adj N 604 thermische Zersetzung (thermal decomposition) N P N 148 Bohren von Du¨bello¨chern (drilling of dowel holes) N D Ngen 107 Viskosita¨t der Farbe (viscosity of the paint)
Gold standard. A gold standard (GS) has been
developed for the basic POS patterns (cf. section
2) which we take to capture the core terminology
of the domain. Lemma sequences with a minimum
frequency of four were extracted from the domain
corpus matching these patterns. The gold standard
contains those terms which were marked as terms
by at least two out of three independent annotators
carefully following defined guidelines
We compare the suitability of the measures mentioned in section 2.3, as a basis for further adjustments. The measures DS and CSmw seem to be the most suited overall (cf. figure 2), while TFITF 7An alternative source would be Wikipedia, as it covers a broad variety of specialized topics.
Interpretation according to
33 2,344 1,094 44 510 133
0.5 1 1.5 2 2.5 top-n candidates (normalized by the length of the GS) produces superior results only for very short candidate term lists. CSmw achieves a maximum F1-score of 0.59 (48% precision and 77% recall) which is an improvement of 20 % over the simple frequency baseline. For very short lists TFITF achieves a precision of above 80%, for example 84% in the top 150 extracted candidates where other measures barely reach 70%. In the following we analyze these observations in detail and illustrate the reasons with examples from the extraction result as presented in table 3. This table shows in columns from left to right: extracted terms, their frequency in the domain corpus and their rank in the result lists according to the measures frequency, DS, TFITF and CSmw.
Sorting the candidates by their DS-value shows a high density of highly domain-specific terms at the top. For example, ‘Drehmomentvorwahl’ (torque pre-selection) on rank 65 (out of 226,715 candidates). DS seems to strongly focus on very specialized terms of the domain texts which do not occur in general language, or only with very low frequency. However, as a consequence it misses important domain terms which also occur with a moderate frequency in general language. For example the term ‘Bohrer’ (drill) which is essential for the domain (domain corpus frequency: 1,094) is only on DS-rank 158,094. This shows that the DS-value approach is not suitable to provide a list of important terms in the domain, but rather to identify its very specialized terms.
The TFITF-measure determines termhood by including the domain corpus frequency of a candidate term logarithmically (cf. formula in section 2.3) - unlike the DS-measure which uses it relatively. As a result several top candidate terms of the TFITF list are of a different kind than the best DS terms. For example, the abovementioned term ‘Bohrer’ (drill) is now at rank 40. 3 TFITF even puts the candidate ‘Mutter’ (screw nut/mother) which, due to its homography, is hard for a terminology extractor to identify as a term, at rank 2,276; this is acceptable, considering that there are 5,097 terms listed in the gold standard. The measure puts a stronger emphasis on the domain corpus frequency producing a considerable amount of noise in form of general-language candidates, which explains its rather mediocre overall performance. We thus suggest to use TFITF to extract a relatively small set of terms with a very high precision, for example for bootstrapping approaches or for an ontology learning which not only focuses on special technical terms of the domain but rather on its key topics.
The results acquired by the measure CSmw are in between TFITF and DS. It also gives more emphasis to the domain corpus frequency of a candidate term than DS, however not as much as TFITF. In the statistical analysis this approach reached the best overall F-scores. The CSmw-measure identifies the same top terms as the DS-measure, namely those which are highly domain specific and rare in general language, for example: ‘Drehmomentvorwahl’ (torque pre-selection) on rank 67. Furthermore, it ranks comparatively high the essential objects of the domain which are also used in general language, for example: ‘Bohrer’ (drill) is on rank 2,554. Consequently, the measure also produces some noise (as TFITF does) which is why it does not outperform the DS-measure. The ranking by CSmw turns out to be the most recommendable for a general terminology extraction which focuses on 0.6 Frequency DS CSvH CSmw
C-value LL TFITF CSmw+C 0.5 1 1.5 2 2.5 top-n candidates (normalized by the length of the GS) technical terms as well as on essential objects of the domain.
While LL outperforms the frequency baseline, for our domain it has proven to be one of the weaker measures and we could not identify any further useful characteristics.
The unsatisfactory result of the ranking by CSvH is based on its strong emphasis on MWTs which have a head with a very high domain corpus frequency. Thus, the result lists show many general-language candidates at their top which is why the measure underperforms the baseline.
The above results were obtained from an evaluation of a mixed set of SWTs and MWTs. A separate analysis of the MWT extraction (cf. figure 3) shows that the two best-performing measures (DS and CSmw) achieve an F1-score of only about 0.49 (recall 64%, precision 40%). In comparison, the maximum F1-score for the extraction of single-word terms was about 0.65. The low performance of the MWT extraction is due to the fact that this task also includes the determination of the right length of the MWT and therefore leads to more noise (in total approximately 80% noise in the MWT candidates selected by the basic POS patterns). Thus, a further filtering of the multiword candidates is necessary. 4.2
Out of the 226,715 items that follow our extended patterns (frequency 1), C-value successfully removes 58,491 cases of noise that only occur embedded (25.8%). In our GS-based evaluation, Cvalue outperforms mere frequency, as shown in figure 2 in the extraction of the basic term patterns. candidate term Band Klebeband doppelseitiges Klebeband
The positive effect of C-value is illustrated with a few examples in table 4. The frequency of occurrence in the domain corpus of the first can3 didate term ‘Band’ (tape) is relatively similar to the one of the second candidate ‘Klebeband’ (adhesive tape). They are both single-word nouns and thus would be considered almost equally as terms for the domain. After applying the C-value approach however their termhood values differ clearly with ‘Klebeband’ having a value twice as high as the value of ‘Band’. This mainly follows from the term length computation based on the number of components of compounds (‘Klebeband’ is a compound with two components: ‘kleben’ (to glue) and ‘Band’). Furthermore, the frequency of ‘doppelseitiges Klebeband’ (doublesided adhesive tape) is only approximately a third of the frequency of the single noun ‘Klebeband’. The C-value method here also rewards the length of the multi-word and computes a value that is half of the C-value of the single noun despite the much lower absolute frequency of ‘doppelseitiges Klebeband’. Note that the termhood value of ‘doppelseitiges Klebeband’ is also greater than the value for ‘Band’ even though it has a lower frequency. This shows that the fine-grained measurement of the length characteristic of candidate terms including a special treatment for compounds is beneficial for terminology extraction.
However, it has to be noted that the ranking of candidate terms by C-value alone is not sufficient for term extraction, as extracted top lists with a recall of greater than 50% still contain a considerable amount of noise (at least 78%), mostly in the form of general-language candidates.
We found that CSmw, one of the bestperforming measures in the comparative evaluation, improves when domain-specificity is computed on C-value instead of frequency (CSmw+Cplot in Figure 3, maximum F1-score 0.51). This is due to C-value’s sensitivity for nested terms which is combined with the domain-specificity filter from CSmw. 0.2 0 ignore PBs ‘soft’ PB filter ‘hard’ PB filter 0.5 1 1.5 2 top-n candidates (normalized by length of GS) 2.5 As the syntactic filter only affects POS sequences with prepositions whose proportion in the GS is rather small, the effect of phrase boundaries (PBs) cannot be shown in Figure 3 and was thus tested in two other settings. First, we evaluated against the 107 N P N terms in the gold standard. Figure 4 shows the effects of applying the “soft” filter (frequency adjustments, section 2.1) and the “hard” filter (candidate removed altogether) on the F1score. Both filters clearly improve the standard extraction based on CSmw. However, the filter affects more than just the terms in the GS (17,4% of all NP+PP candidate occurrences affected) and we would like to observe the effects on all variants of prepositional patterns. Thus, in a precisionbased evaluation, we ranked the MWT candidates by the number of times they violated the syntactic filter and manually checked, for the top 500 removal candidates, whether the removal was justified. The result, as shown in Table 5, indicates that the quality of the parser output is sufficient to predict syntactic validity: the overall precision for these top 500 candidates was 83%.
Top n Precision Top n Precision
We presented three steps to remove noise and to increase performance in nominal terminology extraction. We also suggested a combination of statistical measures that is particulary suitable for this task: Our best setting has proven to be a combination of C-value and CSmw, together with a syntactic validity check. A qualitative analysis of the extraction results showed that different termhood measures emphasize different characteristics of terms, as their top lists differ. Therefore, a combination of statistical measures can also be considered for further improvements, instead of only focusing on one single best performing measure. One could for example think of ways to combine the top lists of a set of best-performing measures, or try an approach that combines or ranks different scores of certain measures in one formula. Future work will also be based on English data where we will evaluate further steps to improve term extraction results, e.g. by combining the termhood measures also with association measures and by further improving the syntactic analysis through the use of an additional constituency parser. A further objective of this work will be to assess the generality of the approach on different domains.