-Verbalizing numeric text tokens is a required task for various speech-related applications, including automatic speech recognition and text-to-speech synthesis. In morphologically rich languages, such conversion involves predicting implicit morphological properties of a corresponding numeral. In this paper, we propose first-order skip-chain Conditional Random Field (CRF) models and various prepossessing techniques to leverage different contextual information. We show that our best skip-chain CRF models achieve over 80% accuracy on the set of 2000 Lithuanian sentences.
Number normalization is the task of replacing numeric
tokens in a sentence by numerals (word tokens) using an
appropriate inflected form of a numeral. Number normalization
usually involves disambiguation as the same numeric token
needs to be mapped into different word forms depending on
the context (e.g. ’5 vaikai eina’ 7! ’Penki vaikai eina’ (five
children are going) vs. ’5 vaiku˛ ne˙ra’ 7! ’Penkiu˛ vaiku˛ ne˙ra’
(five children are missing)). Although number normalization
can be considered as a part of a broader task of text
normalization, formulating it as the separate task might be beneficial,
since the process of number normalization can be quite
complex depending on morphological features of a language. In
this paper, we describe the process of building and evaluating a
number normalization system for Lithuanian. However, some
techniques and models are language-independent and might be
applied to other languages. In Lithuanian, for example, number
’5’ depending on sentence context may represent any of 63
different words. Predicting this relationship directly is rather
difficult and it would require huge data-set to properly learn
Numeral grammar. A simpler approach is predicting Part Of
Speech (POS) tag and then generating numerals accordingly.
POS tag contains all necessary morphological information
to use language-specific grammar based numbers-to-words
system [
As far as we know, there are no published works or
publicly available applications performing number
normalization based on the sentence context for Lithuanian. Although,
many languages deal with similar morphological
disambiguation problems. Russian and Lithuanian numbers share many
morphological properties, including types (cardinal, ordinal),
genders and cases. There are existing research and systems
for the Russian language on general text normalization,
handwritten language-general grammar [
A small text corpus for training and evaluating text normalization models was collected and manually annotated. It consists of 1955 sentences containing 3143 numeric tokens. Sentences were inspected by linguists who suggested a numeral word form as an appropriate replacement for every numeric token. In some ambiguous cases, a few reasonable alternatives were proposed. Some ambiguities were related to the use of the pronominal numeral forms (e.g. ’15 savaite˙’ (15th week) 7! ’penkiolikta savaite˙’ (non-pronominal form) or ’penkioliktoji savaite˙’ (pronominal form)). Other ambiguities were related to numeral case (e.g. ’2019 vasar˛ı’ (2019 February) 7! ’du tu¯kstancˇiai devynioliktu˛ju˛ vasar˛ı’ or ’du tu¯kstancˇiai devynioliktaisiais vasar˛ı’). All suffixes that represented a ’normalization hint’ were eliminated from the data set (e.g. ’2019aisiais’ was replaced by ’2019’). This had an effect of making training subset of the corpus more interesting for the training algorithm, increased the complexity of the normalization task, and reduced the normalization accuracy estimates on the test subset of the corpus.
Sentences of the corpus were pre-processed by the Hidden
Markov Models (HMM) based POS tagger [
Since important prediction decisions are based on tagger provided POS tags and lemmas, to ensure optimal performance, morphological annotations were hand-corrected in training-testing data-set. When using morphological analysis data, it is beneficial to divide POS tags into sub-labels to build more abstract grammar rules and filter out redundant information.
Similar more saturated Natural Language Processing (NLP)
sequence labeling tasks, POS tagging and Named Entity
Recognition (NER), does not require hand-written
languagespecific grammar rules to achieve state-of-the-art [
All rules are constructed as conditional functions without any prior weighting. Here are different techniques used to approximate and generalize number relationship with sentence context:
Lemma classification. (Replacing certain word lemmas with a dedicated class name, for example, month names with ’%Month’) Number classification. (See Table I) Verb classification. (Based on syntactic features of controlling case of other POS) Syntactic linking (See Section V. Long-Distance Dependencies)
In this paper, mainly variations of Conditional Random Fields (CRF) are explored, since CRF got better baseline performance than its neural version (LSTM-CRF) and appears to be more suitable for particular data-set and grammar ruleset.
To create a single sequence tagging model for this task, we
would need 79 (different combinations of sub-labels shown in
Table II) detailed morphological labels corresponding to the
output classes. With the currently available corpus, this would
cause significant data scarcity problems. There are no training
examples for a considerable number of classes and many
others are barely represented. A good way to address this data
scarcity problem is creating three independent CRF models
for Case, Type, Gender prediction and to combine these
predictions at a later stage. This is preferable since there is no
direct dependency relationship between these morphological
categories and operating with sub-labels allows creating a
more abstract rule-set. Although, it is worth noting that
sublabel dependencies have been proven useful for NLP sequence
labeling using CRF [
The linear-chain structure is usually used for sequence
labeling CRF since additionally modeling non-linear
relationships requires complicated inference algorithms and prior
specification of such dependencies [
Although skip-chain structure quite reliably models some important long-distance relationships, it is not able to capture distant dependencies between number and non-number tokens (e.g. in ’3 didžiu˛ju˛ mobiliojo ryšio operatoriu˛’ 7! ’triju˛ mobiliojo ryšio operatoriu˛’ (three major mobile network operators) numbers ’3’ case is determined by words ’operatoriu˛’ (operators) case). CRF is generally unable to leverage such features and requires either hybridization such as LSTMCRF, or additional pre-processing. We propose identifying position-distance independent relationships using an ad-hoc set of linkage rules and formulating perceived syntactic links as conditional functions of CRF. For Lithuanian language, we discern three directly related parts of speech (Noun, Verb, Preposition) in the number normalization task. For each, we use a different set of linkage rules, to identify related tokens to every number in the sentence, effectively performing partial syntactic analysis. To link prepositions and verbs to numbers, our rules solely rely on morphological labels provided by the POS tagger. For nouns, the task of linking could be more precisely formulated as an identification of a noun which represents an object or quantity being counted by some number in the sentence. This is extremely important since countable 7 0 f m 7 f m red 0 f m red 0 apples 0 f m apples 0
We determine the most likely countable noun in a twostep process. First, for a given numeric token d we select all potentially countable noun tokens fnig according to the ad-hoc set of linkage rules for nouns. We can not make an educated choice among selected nouns on the basis of available morphosyntactic annotation since noun morphology does not have the property of ’countability’. To discriminate among potential countable nouns, semantic analysis is needed. We need to rate the set of selected nouns fnig according to some ’countability’ measure that is dependent on the numeric token d being normalized and select the noun nbest with the highest (d; ni) rating nbest = arg max (d; ni)
Suppose that we have vector embeddings v(n); v(n) 2 RD
for every noun n, that were obtained by an algorithm such as
’word2vec’ [
(d; ni) = cosine-similarity( (d); v(ni)) (1)
We have tested a few different approaches to design the above-mentioned mapping (d). We sought large unannotated text corpus for number and noun adjacent co-occurrences and made noun frequency lists per every numeric token that was found (around 350 thousand co-occurrences). Information present in a frequency list can be aggregated into a single vector by estimating the weighted average of noun embeddings making up that list. Thus a representative (or central) embedding vector can be obtained per every numeric token. Although this tabular mapping from numeric tokens into representative vectors can be used in (1), it has serious limitations. The table contains many unreliable vectors for rare numbers, because of the lack of co-occurrences in the unannotated corpus. To circumvent the limitations of this tabular mapping we used the Neural Network (NN) approach to build a continuous cooccurrence model. We built two different neural networks: one with a single input (corresponding to the mathematical value of the numeric token) and one with 7 inputs, corresponding to the decomposition of the numeric token into sub-parts (thousands, hundreds,...) and including number features similar to Table I. NN had 200 output units.
The evaluation of these models is shown in Table III. The baseline performance is obtained by the simple rule "take the first potentially countable noun to the right of a numeric token". Accuracy is measured using whole CRF training data, extracting situations where a choice between two or more nouns (2.41 avg.) is needed.
We evaluate models with 5-fold cross-validation (except for countable noun identification in Section V, since training and testing data-sets were obtained from different sources). The accuracy of different models is shown in Table IV. Combined accuracy estimates the accuracy of all three models. The combined answer is considered to be correct if all three sublabels are correct.
It is worth noting, that our model is focused on grammatically correct, as ’spoken’ number normalization. This might not be desirable for systems like text-to-speech synthesis, hence a more standardized approach can be chosen. For Lithuanian language, numeral definiteness property could be removed from the prediction model, since it is not strictly constrained by grammar. This would increase language correctness and improve Type prediction models and combined accuracy as shown last line of Table IV (best performing model without definiteness property).
Accuracies above represent the lower bound accuracies of the real-world number normalization performance. Firstly, in certain situations, some sub-label prediction mistakes might be irrelevant for numeral generation. For example, both ’5, Cardinal, Genitive, Feminine’ and ’5, Cardinal, Genitive, Masculine’ will generate the same word representation ’penkiu˛’. Secondly, real-world sentences often contain suffixes (e.g. ’Kovo 11-a˛ja˛’ 7! ’Kovo vienuolikta˛ja˛’ (March 11th)) that either offer an unambiguous hint that solves the number normalization problem or at least provides most of the needed morphological information, which can be used to correct prediction mistakes.
In this paper, we describe the number normalization
disambiguation model, which is needed to develop a context
dependant number-to-words system. Sequence-labeling
approach allows us to normalize countable abbreviations and
symbols (next to number) effortlessly, since countable noun
morphological form can be extracted from predicted label
(e.g. ’nuo 5%’ (from 5%) 7! ’nuo penkiu˛ procentu˛’). Our
implementation based on this model is publicly available [
Number normalization errors are often directly dependant on morphological analysis mistakes and we are currently working on improving both vocabulary-grammar and disambiguation sides of Lithuanian POS tagging to consequentially increase number normalization accuracy.
Currently, we use ’word2vec’ [
Our achieved number normalization accuracy could be further improved by expanding annotated training data since a considerable amount of errors are a direct result of data scarcity. Although, our approach generally lacks semantic and syntactic language understanding, so performing full syntactic sentence analysis in the preprocessing stage would be highly beneficial.
This research was supported by the project “Semantika 2“ (No. 02.3.1-CPVA-V-527-01-0002). Special gratitude goes to our colleagues Lina Majauskaite˙ and Dovile˙ Stukaite˙ who helped us in collecting and annotating text corpus.