English. Starting from a wide set of linguistic features, we present the first in depth feature analysis in two different Native Language Identification (NLI) scenarios. We compare the results obtained in a traditional NLI document classification task and in a newly introduced sentence classification task, investigating the different role played by the considered features. Finally, we study the impact of a set of selected features extracted from the sentence classifier in document classification.
Native Language Identification (NLI) is the
research topic aimed at identifying the native
language (L1) of a speaker or a writer based on
his/her language production in a non-native
language (L2). The leading assumption of NLI
research is that speakers with the same L1 exhibit
similar linguistic patterns in their L2 productions
which can be viewed as traces of the L1
interference phenomena. Thanks to the availability
of large-scale benchmark corpora, such as the
TOEFL11 corpus
In this paper we want to further contribute to
NLI research by focusing the attention on the role
played by different types of linguistic features in
predicting the native language of L2 writers.
Starting from the approach devised by
In this work, we built a classifier based on SVM
using LIBLINEAR
We carried out two experiments devoted to
classify L2 documents and sentences. The training
and development set distributed in the 2017 NLI
shared task, i.e. the TOEFL11 corpus
The obtained macro average F1-scores were: 0.8747 in the document classification task and 0.4035 in the sentence one. As it was expected, the identification of the L1 of the sentences turned out as a more complex task than L1 document classification. Both document and sentence classification are influenced by the number of words but with a different impact. Figure 1 shows that the average performance on sentences is reached for sentences 21–token long, which corresponds to the average sentence length for this dataset. As the sentence length increases, the accuracy increases as well. Due to the smaller amount of linguistic evidence, the classification of short sentences is a more complex task. The performance of document classification is more stable: the best f–score is already reached for documents of 200–tokens, which corresponds to a very short document compared to the average size of TOEFL11 documents (330 tokens).
Figures 2(a) and 2(b) report the confusion matrices of the two experiments1. As it can be seen, both for sentences and documents the best classification performance is obtained for German, Japanese and Chinese, even though with some differences in the relative ranking positions, e.g. German is the top ranked one in the sentence classification scenario and the 2nd ranked one in the document classification one, while Japanese is the best classified L1 in the document experiment and the 4th ranked one in the sentence classification scenario. Conversely, we observe differences with respect to the worst recognized L1s, which are Turkish, Hindi and Korean in the document classification task and Arabic, Spanish and Turkish in the sentence classification one. The two confusion matrices also reveal a peculiar error distribution trend: the confusion matrix of the sentence classification model is much more sparse than the 1Since the number of documents and sentences in the two experiments is different, in order to make comparable the values of the two confusion matrices, the sentence classification values were normalized to 100. document classification one. This means that for each considered L1, the errors made by the sentence classifier are quite similarly distributed over all possible L1s; instead, errors in the document classification scenario are much more prototypical, i.e. the wrong predicted label is assigned to only one or two L1 candidates, which change according to the specific L1. This is shown e.g. by languages belonging to same language family such as Japanese and Korean which belong to the same Altaic family. Specifically, in the document classification scenario Korean is mainly confused with Japanese (10% of errors). This trend holds also in the sentence classification experiment where 17.8% of errors were due to the confusion of Korean with Japanese and vice versa (18.2% of errors). Interestingly enough, the most prototypical errors were also made when contact languages were concerned. This is for example the case of Hindi and Telugu: Hindi documents were mainly confused with Telugu ones (16% of errors) and Telugu documents with Hindi ones (13% of errors). Similarly, in the sentence classification scenario, Hindi sentences were wrongly classified as Telugu sentences in about 20% of cases and vice versa. As previously shown by Cimino et al. (2013), even if these two languages do not belong to the same family, such classification errors might originate from a similar linguistic profile due to language contact phenomena: for instance, both Hindi and Telugu L1 essays are characterized by sentences and words of similar length, or they share similar syntactic structures such e.g. parse trees of similar depth and embedded complement chains governed by a nominal head of similar length.
The behavior of the two classifiers may suggest that i) some features could play a different role in the classification of sentences with respect to documents and ii) the document classifier can be improved using features extracted from the output of a sentence classifier in a stacked configuration. To investigate these hypotheses, we carried out an extensive feature selection analysis to study the role of the features in the two classification scenarios. 3.1
Feature Selection
In the first step of the feature selection process, we
extracted all the features from the training set and
pruned those occurring less than 4 times, obtaining
4,000,000 distinct features both for document
and sentence classification. In the second step, we
ranked the extracted features through the
Recursive Feature Elimination (RFE) algorithm
implemented in the Scikit-learn library
Figure 3(a) compares the percentage of different types of features used in the classification of documents and sentences. As it can be noted, the document classifier uses more words n-grams, especially n-grams characters. Instead, morpho– syntactic and syntactic features are more effective for sentence classification, and the n-grams of lemmas even more than 4 times. Figures 3(b), 3(c) and 3(d) show the variation of relevance of the 40k raw text, morpho-syntactic and syntactic features grouped in bins of 100 features. The lines in the charts correspond to the differences between document and sentence in terms of percentage of a single type of feature in the bin with respect to its total distribution in the whole 40k selected features2. Negative values mean that this distribution in the bin is higher for sentence classification.
Among the raw text features (Figure 3(b)), ngrams of words occur more in the 1st bins of document classification, while n-grams of characters and lemma are more relevant in the 1st bins of sentence classification. The n-grams of coarse parts– of-speech are equally distributed in the two rankings, instead both the n-grams of coarse parts–ofspeech followed by a lemma and the n-grams of functional words occur more in the 1st bins of sentence classification (Figure 3(c)). This confirms the key role played by lemma in sentence classification.
For what concerns syntactic information
(Figure 3(d)), the features that properly capture
sentence structure (dependency subtree and the
hierarchical syntactic dependencies) are all contained
in the first bins of document classification even if
their total distribution is lower than in the
sentence. This shows that syntactic information is
very relevant also when longer texts are classified
and that this kind of information is not captured by
2Spline interpolation applied for readability purpose.
(a)
(b)
n-grams of words. Feature types with low
number of instances are not reported in these charts.
Among these, etymological n-grams appears in the
first bins both for sentence and document,
confirming the relevance of the etymological
information already proven for NLI document
classification
The different role of the features in the two L1
classification tasks suggests that we may improve
the traditional NLI document classification by
combining sentence and document classifiers. We
thus evaluated and extended the stacked
sentencedocument architecture proposed by
We introduced a new NLI scenario focused on sentence classification. Compared to document classification we obtained different results in terms of accuracy and distribution of errors across the L1s. We showed the different role played by a
Model Cimino and Dell’Orletta (2017) Base classifier Sentence features Base class. + avg Base class. + stddev Base class. + prod Base class. + top–3 max-min Base class. + all sentence feat. wide set of linguistic features in the two NLI scenarios. These differences may justify the performance boost we achieved with a stacked sentencedocument system. We also assessed which features extracted from the sentence classifier maximizes NLI document classification. 6
The work presented in this paper was partially supported by the 2–year project (20182020) SchoolChain – Soluzioni innovative per la creazione, la certificazione, il riuso e la condivisione di unita` didattiche digitali all’interno del sistema Scuola, funded by Regione Toscana (BANDO POR FESR 2014-2020).