Funnelling (Fun) is a method for cross-lingual text classification (CLTC) based on a two-tier learning ensemble for heterogeneous transfer learning (HTL). In this ensemble method, 1st-tier classifiers, each working on a diferent and language-dependent feature space, return a vector of calibrated posterior probabilities (with one dimension for each class) for each document, and the final classification decision is taken by a metaclassifier that uses this vector as its input. In this paper we describe (gFun), a generalization of Fun consisting of a HTL architecture in which 1st-tier components can be arbitrary view-generating functions, i.e., language-dependent functions that each produce a languageindependent representation (“view”) of the document. We describe an instance of gFun in which the metaclassifier receives as input a vector of calibrated posterior probabilities (as in other embedded representations that embody other types of correlations. We describe preliminary results that we have obtained on a large standard dataset for multilingual multilabel text classification.
According to [
Bearing in mind that most of the languages in the world are low-resource, it is appealing to develop methods and techniques capable of exploiting the high-quality resources available for the few resource-rich languages, in order to improve the performance on tasks carried out on the resource-poor languages. Cross-Lingual Transfer Learning (CLTL) is a class of machine learning tasks in which, given a training set of textual labelled data sampled from one or more nEvelop-O source languages, we must issue predictions for unlabelled documents written in one or more target languages. In other words, the goal of CLTL is to transfer (i.e., reuse) the knowledge that has been obtained from the training data in the source languages, to the target languages of interest, for which few labelled data (or no labelled data at all) exist.
Cross-Lingual Text Classification (CLTC) is a specific instance of CLTL, in which classification is the task to be carried out. In CLTC, documents are written in one of a finite set ℒ = { 1, ..., |ℒ |} of languages, and labelled according to a shared codeframe (a.k.a. classification scheme ) = { 1, ..., | | }. In such a scenario, it is common to have diferent numbers of training documents for the diferent languages, with the languages with fewer training documents usually being also the ones with fewer (if at all) available external resources (such as bilingual dictionaries, thesauri, pre-trained sets of word-embeddings, language models) that could otherwise be leveraged for this task.
Funnelling (Fun – [
Fun is a two-level architecture [
The main intuition behind Fun is to leverage the fact that all the documents are classified according to the same set of labels. Documents, regardless of the language they are written in, can be represented as vectors of posterior probabilities, i.e., vectors encoding, at each dimension , the probability for a given document to be labeled as belonging to the respective class . Once all the documents are homogenized (i.e., they are all represented as vectors of posterior probabilities), they can be stacked vertically and fed to the second-tier (the metaclassifier) regardless of the language they were originaly written in.
We generalize this architecture, and call it Generalized Funnelling (gFun). The first tier of Fun is redesigned in order to accommodate for a a set Ψ of view-generating functions (VGFs) that can expand the shared vector space on which the meta-classifier operates. VGFs are languagedependent functions that map documents into language-independent vectorial representations (views) aligned across languages. Since each view is aligned across languages, it is easy to aggregate (e.g., by concatenation) the diferent views into a single representation aligned across languages, that is then given as input to the metaclassifier. Notice that, according to this definition, the original implementation of Fun can be seen as a specific setting of gFun equipped with one single VGF.
The key idea is to leverage the VGFs in order to inject into the model information about
diferent correlations between the main elements of a Text Classification task.In this research,
we consider four kinds of correlations: Class-Class correlation, Document-Class correlation,
WordClass correlation, Word-Word correlation, Document-Word correlation. We bring to bear these
stochastic correlations by means of the following VGFs:
• the Posteriors VGF (encoding document-class correlations): it maps documents into the
space defined by calibrated posterior probabilities (as in the original Fun).
• the MUSEs VGF (encoding word-word correlations): it uses the Multilingual Unsupervised
/ Supervised Embeddings (MUSEs) made available by the authors of [
The diferent views produced by the VGFs need to be aggregated before being issued to the metaclassifier. In this work, we propose to
average the diferent views. 1 Before averaging the representations, we must ensure all views to have same dimensionality, and to be aligned.2 In order to do so, we learn additional mappings of the views to the space of class-conditional posterior probabilities, i.e., for each VGF (other than the Posteriors VGF, which already returns vectors of | |
calibrated posterior probabilities) we train a classifier that maps the view of a document into a vector of | |
calibrated posterior probabilities.
Finally, we have found that applying some routine normalization techniques consistently
increases the performance of gFun. This normalization consists of imposing unit L2-norm to
the vectors computed by the view generators, removing the first principal component of the
document embeddings obtained via WCEs or MUSEs, and standardizing the columns of the
shared space before passing the vectors [
In order to maximize comparability with the previous results, we adopt an experimental setup
identical to the one used in [
We carry out experiments on JRC-Acquis, a parallel corpus of legislative texts published by the European Union, consisting of 11 diferent languages. We retain the 300 most frequent simply concatenating the diferent views. both views.
1In preliminary work, we have observed experimentally that avaraging tends to produce better results than 2Two views are said to be aligned when the semantics of the dimensions (whatever it may be) is common to 3Standardizing (a.k.a. “z-scoring”, or “z-transforming”) consists of having a random variable , with mean and standard deviation , translated and scaled as =
− , so that the new random variable has zero mean and unit
variance. The statistics and are unknown, and are thus estimated on the training set.
target classes and use the same splits as in [
In Table 1, we directly compare our results with the naïve solution (i.e., one monolingual classifier for each language), Fun and multilingual BERT (mBERT). We group gFun results in three diferent batches: the first one groups the results obtained by deploying one single VGF at the time; in the second one we report the results combining multiple generators; in the latter we deployed all the proposed VGFs jointly. We use the notation -X to refer to the Posteriors VGF, -M denotes the MUSEs VGF, -W the WCEs VGF, and -B the BERT VGF.
The superior results of gFun-Xwith respect to Fun indicate that the normalization steps are beneficial. It is noteworthy how by simply leveraging the class-class correlations (brought to bear by the metaclassifier) gFun-B outperforms its counterpart mBERT. The best results are obtained by the combination of Posterior, MUSEs, and BERT VGFs.
In this paper we propose Generalized Funnelling (gFun), a revised variant of Fun [
4We have validated our method also using RCV1/2, but we leave the discussion of this dataset out of this short paper for the sake of brevity.