This paper illustrates work done on "Gender Identi cation in Russian texts (RusPro ling)" shared task, hosted by PAN in conjunction with FIRE 2017. The task is to predict the author's gender, based on the Twitter data corpus which is in Russian. We will give a brief introduction to the task at hand, elaborate on the data-set provided by the competition organizers, discuss various feature selection methods, provided experimental analysis that we followed for feature representation and show comparative outcomes of di erent classi ers that we used for validation. We submitted a total of 3 models and their respective prediction for each test data-set with slightly di erent pre-processing technique based upon the test corpus content. As each of the test corpus were sourced from various platforms, this made it challenging to stick to one representation alone. As per the global ranking published for the shared task[6] our team secured 2nd position overall (Concatenating all Data-set) and our 3rd submission model performed the best among the 3 submission models from the overall test data corpus. Further under extended work we discuss in brief how hyper parameter tuning of certain attributes extend our validation accuracy by 6% from baseline.
The Internet, it is a vast platform where anyone can have access to myriads of information, from online news media articles to various social media platforms, from personal blogs to personalized websites, all this literally at the end of our ngertips, and in this present age, life is becoming unimaginable without it. With the availability to all this resources, people are writing and share information more avidly over the internet than ever before, and it also provides a certain degree of anonymity while doing so. Access to such multitudinous information brings in certain set of problems like theft of identity/content and plagiarism to name a few and this we are trying to address with tasks such as "Author Pro ling".
In "Author pro ling" [
With this paper on RusPro ling sharedtask, we focus on cross-genre
gender identi cation in Russian texts[
In this task we have Twitter as training data corpus and as test
data corpus we have dataset from multiple social media domain
platforms like Twitter, Facebook, online reviews (where texts are
describing images, or letters to a friend), product and services. The
focus with this task is on gender pro ling in social media and the
main interest is in the everyday language and on how the basic
social and personality skills re ects on their writing [
The main challenge in this task is the language itself, as it is not a native, thus we used certain pre-processing methods and built our baseline representation on which we implemented classical machine learning algorithms for this text classi cation task. 2
The Corpus for the training data was mainly sourced from social media website Twitter and the labels were annotated for each of the document with the author gender "male" or "female". The training corpus is a collection of 600 data le in XML format which consists of exactly half female and half male genre documents, the le name are annotated by their associated gender label in a separate le called "truth" which is in text format.
Training Dataset
Total number of documents Total number of male documents Total number of female documents
A cursory analysis of the training corpus reviled that each training data le has a combination of di erent tags and hyperlinks, further the documents varied in count of content words i.e one document went from no text to others over 3000 plus words in a single document. Few of the les had mixed data of Russian and English, where as few other where completely in English language.
The test corpus is presented in 5 folders varied by the category of di erent sources. Each set contains di erent amount of les, the count of documents for each category varies from 96 to 776 les each. On further inspection the text format provided in each folders apart from the 3rd folder is di erent when comparing with the training corpus, namely o ine texts, Facebook, Twitter, product and online reviews and gender imitation corpus in order of their folder number respectively as shown in Table 1-2.
We have also taken the statistical data of the complete vocabulary size that we gained from grid search of attributes namely n-gram_range and min_df count. In each combination their respective corpus size is found, and the statistics are tabulated in Table 3. 3
The Figure 1 gives a rudimentary picture of the architecture that
we have implemented for our 3 submissions, in all of these models
we mainly focused on data pre-processing methods to incorporate
various features and build upon each one of them to improve the
feature representation. We started from a simple count based model,
the same methods are discussed next.
The feature selection was a process in which we started by building
a baseline model [
The 1st approach from the dataset was to form a simple count based Term Document(TD) and Term Frequency Inverse Document Frequency (TFIDF) matrix which became the baseline for our accuracy and further went with adding general features to previous representation.
With the knowledge of the social media network "Twitter", we essentially narrowed down our focus on search for features to tags, like ’@’ which is mainly used to address people/gathering and hash tag ’#’ which is based on the context or the image of the adjoining content. Moving on, we found that URLs were being used widely across most of the dataset which linked to various internet sources, so we then incorporated these as a feature to the earlier feature representation which proved to show slight improvement on all of the classi cation algorithms, captured below in Figure 3-4.
On further analysis we found that, individual URL’s in itself as a feature seemed fruitless, thus only considering the hyperlink itself, we focused on normalizing these across the dataset and went with the count of the URL and those of the tag’s as feature to represent a document. It proved to increase the accuracy little more, This further led to normalizing of various emoticons represented by a keyword and various other punctuation like the exclamation mark ’!’, period ’.’ and hyphen ’-’ which occurred multiple time or in continuous repetition were converted to a single instance of each.
As we were not familiar with the language, we considered the average word length as the total number of character per document to the total number of feature instance in that document and appended that list as an average word length '
Training Data: was not at this game and did not see the game, now Processing Data: & % per document making it an independent feature. This is to accommodate for the fact that the average vocabulary word length that gender used can also be taken as a discriminative feature between the 2 classes.
As a part of experimental analysis we manually ran over few random training documents based on the individual sizes of the le to gather a glimpse of the overall change in data, then ran snippets on these training set data to see the scale of, improvement of, accuracy with the above considered parameters. Thus to distinguish for better feature representation for the classi cation.
After going through various transitions, the selected certain features were extracted and then used as a part of pre-processing of the entire training corpus these features were individually added one set at a time to show the increase or decrease in their accuracy corresponding to various classi ers by cross validation with di erent classical ML classi ers, namely Logistic regression (LR), Support Vector Machine (SVM) using linear kernel, Decision tree $ (1) A simple count based matrix is taken to achieve baseline accuracy, for this we have considered both TD and TFIDF matrix representation from which we set a base-line accuracy of 80.5% ( We randomly initialized few attributes like ngram_range = 2 , min_df = 3 and used a linear SVM classi er to get the baseline). (2) Count of ’http’ and ’https’ are taken and converted to a single key word ’https’ as this will help in adding feature to see the usage of URLs between the 2 class distinguishing which gender base might have used more number of hyperlinks within their tweets. (3) Count of ’#’ tags was further attributed to the previous representation. (4) Replaced emoticons with keyword. (5) Took the average word length in a document i.e count of character to number of feature instances as the language, this we chose as a preferred method. 5
We submitted a total of 3 models/run’s and for each individual run the following pre-processing method have been followed: Submission 1 : We have considered feature representation 2, 3, 4, 5 and also the normalization of ’@’ followed by content tags to simple key word( Splitting tags from their context otherwise to preserve the word content in particular did not show much di erence in validation accuracy), and used SVM classi er for classi cation. Based on learned model from training corpus the prediction for the test corpus’s were taken.
Submission 2 : The same feature representation as the 1st run was considered, but we used a di erent classi er, we took Adaboost based training model and the prediction for the test corpus’s were taken Submission 3 : In this run, we considered mostly with regard to the other test datasets 1,2,4 and 5 where the content are in longer and in paragraph form rather than the shorter version and there was less to no use of tags and or hyperlinks. Thus to normalize this we disregarded the above used tags and reduced any extended repeat of punctuation’s to a single count(e.g:’...’ is shortened to a single ’.’)
A sample of this is shown in Figure 2. 6
As per the global ranking published for the shared task by the
organizers[
Individually, submission 3 gaining 2nd best accuracy in "o line texts (picture descriptions, letter to a friend etc.)from RusPro ling corpus" whereas the submission 1 gained our team 2nd place for "gender imitation corpus" and 3rd in "product and service online reviews corpus". 7
In our earlier experiments we randomly initialized our attributes like max feature length, n_gram and min_df with 10000, 2 and 3 respectively. As a motive to increase validation accuracy we performed a grid search for the hyper-parameter namely word count, n_gram and min_df based values with the TFIDF model, where we considered the following range of data values for each: Word count : 10000 - 50000 ngram-range : 2 - 6 Min-df : 1 - 4
After applying grid search we pushed the baseline accuracy to 82.5% when initializing max_feature with 10,000, n_gram with 2 and min_df with 1 and applying a linear SVM classi er. We further pushed our validation to 86.49% by applying Adaboost classi er. Over all we found that the trend of accuracy of TD feature representation model decreased with increase in all the attributes, and the accuracy of TFIDF feature representation model increased but saturated after n_gram value exceeds 6 and the min_df value exceeds 4, the same is show in Figure 5 where the best of the attributes, feature combination were taken for each TD and TFIDF representation. The challenge in this shared task we faced was the fact that we were working on a language corpus that is non native to us, thus we mainly focused on pre-processing and normalizing the data corpus to get improved feature representation. We built from a basic count representation and incorporated simple modi cation on iterating feature representation and observed the various accuracy changes involved with those features. Based on the experimental analysis and further discussion on optimizing of the various attributes in the extended work, we could make an inference that the baseline can further be increased which could better improve the prediction, fetching us better gender identi cation model.
As a future study we considered making various embedded
representation for the Russian corpus and use deep learning techniques
for categorizing author gender [