English. The paper investigates impact and role of different feature types for the specific task of Automatic Genre Classification with the final aim of identifying the most predictive ones. The goal was pursued by carrying out incremental feature selection through Grafting using different sets of linguistic features. Achieved results for discriminating among four traditional textual genres show the key role played by syntactic features, whose impact turned out to vary across genres.
Italiano. L’articolo intende indagare il ruolo svolto da diversi tipi di caratteristiche linguistiche nella classificazione automatica del genere testuale al fine di identificare le piu` efficaci e rilevanti. A questo scopo e` stata messa a punto una metodologia basata su un processo incrementale di selezione realizzato mediante un algoritmo di Grafting usando diversi tipi di caratteristiche. I risultati raggiunti mostrano il ruolo chiave delle caratteristiche sintattiche, il cui impatto varia in modo significativo tra generi diversi.
Automatic classification of textual genres has
always received significant attention from both
theoretical and application perspectives. On the one
hand, it has been considered relevant by linguists
and educators to teach students the correct way
of writing in specific communicative scenarios
A variety of different approaches to Automatic
Genre Classification (AGC) has been proposed so
far differing at the level of the genre and the
typology of features considered. According to the
widely acknowledged fact that no established
classification of genres exists (see e.g. Sharoff (2010)
or Biber (2009)), previous studies focused on
‘traditional genres’ such as journalism, handbooks,
academic prose, see among the others
Despite the great interest in the investigation
of which linguistic features qualify a text genre
In this paper, we tackle the AGC task for traditional genres (namely literary, scientific, educational and journalistic texts) by using different types of linguistic features, i.e. lexical, morphosyntactic and syntactic. In particular, the following research questions are addressed: i) which are the most effective features to classify a textual genre, and ii) whether and to what extent features identified as most effective remain the same across different genres. These questions have been addressed by carrying out incremental feature selection with the final aim of identifying the most predictive ones. So far, studies focused on the best set of features to classify textual genres have been carried out mainly on English. In this paper, this issue is investigated for a typologically different language, Italian. 2
In order to identify and rank the most
important features playing a role in genre classification,
we used GRAFTING
For selecting the features and estimating their
weights, we used TINYEST1, a grafting-capable
maximum entropy parameter estimator for
ranking tasks
Various types of features have been proposed in
the literature for the automatic classification of
text genres. Following Stamatatos (2001) and Lim
(2004), we combine token–based and structural
features. Token–based features were extracted
from the top list of the most frequent lemmata in
the training corpus and represented in terms of the
relative frequency of each lemma in each
document. Structural features were extracted from the
considered corpora morpho–syntactically tagged
by the POS tagger described in
We used an Italian corpus including documents
representative of four different genres:
educational material
To assess the influence of including structural features over simply using the most frequent words (lemmata), we used two sets of features (each consisting of about 200 features). The first set of features (taken as the baseline) corresponds to the relative frequency of the 200 top-most frequent words (henceforth referred to as the tw200 set). 2 The second set combines token-based and structural features: i.e. in addition to the relative frequency of the 100 top-most frequent words, it contains the (90) structural features illustrated above and detailed in Table 1 (this set is henceforth referred to as the lingtw set). To guarantee comparability of values, for each feature the values were scaled between 0 and 1 on the basis of the data from the training set. If a (non-scaled) feature value in the held-out test set exceeded the maximum non-scaled value of that feature in the training set, it was set to the maximum value (1).
The feature ranking for each genre was obtained using grafting on the full training data set. The performance (i.e. the percentage of correctly classified documents) of the algorithm was evaluated for an increasing number of features (starting from including only the first (best) feature for each genre to including all features for each genre) against both a 10-fold cross-validation test set and a held-out test set.
The 10-fold cross-validation procedure was performed on the basis of the training set (i.e. the feature weights were determined on the basis of 90% of the training data, whereas the performance was evaluated on the remaining 10% of the training data; this procedure was repeated 10 times). As stated before, the genre of the document in the test set was assigned to the genre whose binary classification model (in this case with the same number of features) resulted in the highest score.
The classification accuracy was assessed with respect to the held-out test set for different numbers of features: i) the number of features associ2In our preliminary analyses, we also assessed the effect of including the most frequent bigrams as features. However, as the performance was similar to only using unigrams, we did not include bigrams as features. Morpho-syntactic Syntactic
Feature Sentence and token length Rate of words in the Basic Italian Vocabulary, Type/Token ratio Part-Of-Speech unigrams, Lexical density, Verbal mood Dependency type unigrams, Parse tree depth features, Arity of verbal predicates, Distribution of subordinate vs main clauses, Length of dependency links ated with the best performance on the cross validation set, and ii) the lowest number of features such that the performance dropped when a new feature was added (i.e. performance kept increasing for each additional feature up to the selected number of features). Results reported below can be replicated by downloading the docker image italianlp-wieling/dockergenreclas sification which contains all data and scripts necessary for the feature extraction and the grafting procedure, and also contains all results. The Docker file including all commands to setup the virtual machine can be found at https://github.com/italianlpwieling/dockergenreclassification.
S 38.75 33.75 42.5 42.5 formance (72.08%) is obtained using 133 features. 5.2
In order to investigate the typology of linguistic features most significantly contributing to AGC we focused on the lingtw set. In particular, we carried out an in-depth analysis of the graftingbased feature ranking resulting from the classification of the held-out test set. Ranked features were categorized into five classes: syntactic, morphosyntactic, lexical, raw and token-based features. Figure 2 provides a genre-independent view reporting the percentage average distribution (across genres) of different feature types within the first 10, 50 and 80 ranked feature sets. As shown, syntactic features play the most relevant role. They cover the 45% and 42% of the first 10 and 50 features respectively, and remain the most predictive ones also when 80 features are considered (representing 39.38% of the set). On the other hand, the distribution of token-based features increases as far as a wider amount of ranked features is considered (they cover 2.5%, 14.5% and 24.06% in the 10, 50 and 80 feature sets respectively).
Consider now the distribution of different types of features across genres reported in Table 2: notable differences can be observed. In particular, Literature and Scientific prose represent two opposite poles. Token-based features (W) are more predictive for literary texts with respect to other genres (i.e. they represent 10%, 28% and 32.50% in the top 10, 50 and 80 features respectively). On the contrary, syntactic features (S) play for Scientific prose a more important role than for the other genres (covering respectively 60%, 50% and 42.50% of the top 10, 50 and 80 features).
Let’s focus now on the role played by individual features across genres. Table 3 reports the different rank positions associated with a selection of features in the classification of the four genres. Raw text features (i.e. sentence and word length) resulted to play a key role in the classification of educational materials (Education) with respect to the other genres (e.g. Literature). A feature capturing the lexical richness of texts such as Type/Token Ratio (TTR), which refers to the ratio between the number of lexical types and the number of tokens (considered as single forms) within a text, is similarly ranked for Journalism and Science while it plays a less relevant role in the classification of educational material and literary texts. Moving to syntax, it should be noted that two features characterizing the overall sentence structure, i.e. the depth of the whole parse tree (calculated in terms of the longest path from the root of the dependency tree to some leaf) and the maximum length of dependency links (calculated in terms of the words occurring between the syntactic head and the dependent), play a key role in the classification of the Literature and Journalism genres. For the latter, it is interesting to contrast the high rank associated with the parse tree depth feature and the irrelevant role played by sentence length (typically taken as a proxy of the underlying grammatical structure): this clearly shows that syntactic features are more effective in discriminating genres. Other features which turned out to play a relevant role in ACG are concerned with the relative ordering of subject and object with respect to the verbal head: their non-canonical orders, i.e. postverbal subject and pre-verbal object, play a key role in the classification of Literature and Journalism genres. On the contrary, the use of passive voice (inferred from the presence of passive subjects) is less relevant for the classification of Literature, whereas it is highly ranked in the characterization of scientific writing and newspaper articles. 6
In this paper we investigated impact and role of different feature types for Automatic Genre Classification. The goal was pursued by carrying out incremental feature selection through Grafting augmented with TinyEst. Two sets of features were taken into account, token-based and structure-based. Achieved results show the key role played by syntactic features, a result which is new with respect to the AGC literature. Another original contribution is concerned with the role of different feature types which turned out to vary across textual genres, suggesting the specialization of features in binary genre classification tasks (e.g. Literature vs. other genres). The features contributing to AGC for Italian are possibly influenced by the language dealt with. Although it is widely acknowledged that linguistic variation across genres is a language universal, the question is whether similar linguistic features are expected to play a similar role across languages. If this might be the case of features such as e.g. TTR, use of passive voice, tenses or pronouns, on the other hand features concerned with the ordering of sentence constituents or the overall sentence structure (e.g. parse tree depth or dependency length) may be distinctive to a specific language or language family. Further directions of research thus include comparison of results in a multilingual perspective as well as across a wider variety of genres.
Reported research started in the framework of a Short Term Mobility program of international exchanges funded by CNR, and continued within the project “Smart News, Social sensing for breakingnews”, funded by the Tuscany Region under the FAR-FAS 2014 program.