One of the major challenges nowadays is Information retrieval from social media platforms. Most of the information on these platforms is informal and noisy in nature. It makes the Information retrieval task more challenging. The task is even more difficult for twitter because of its character limitation per tweet. This limitation bounds the user to express himself in condensed set of words. In the context of India, scenario is little more complicated as users prefer to type in their mother tongue but lack of input tools force them to use Roman script with English embeddings. This combination of multiple languages written in the Roman script makes the Information retrieval task even harder. Query processing for such CodeMixed content is a difficult task because query can be in either of the language and it need to be matched with the documents written in any of the language. In this work, we dealt with this problem using Vector Space Models which gave significantly better results than the other participants. The Mean Average Precision (MAP) for our system was 0.0315 which was second best performance for the subtask.
• Information Systems ➝ Information Retrieval ➝ Retrieval models and ranking ➝
➝ • Computing methodologies
Social media has a plentitude of user generated data in numerous
languages which are predominantly informal in nature. Most of
these languages have their own native scripts. Some of these
scripts include Arabic, Chinese, Hebrew, Greek, and Indic etc. For
most of these languages, major user-generated content is
transliterated into the Roman script with English embeddings. The
trend in Indian social media is to use such informal text
containing a mixture of multiple South-Asian languages with
English embeddings. This mixture makes the Information
Retrieval (IR) task very challenging. In Forum for Information
retrieval (FIRE)1 (2016), a similar task was proposed, which
required Mixed Script IR on Code-Mixed Hindi-English tweets.
The difference of Code-Mixed IR from MixedScript IR is subtle.
In MixedScript content, query is written in Roman or native
script [
The subtask II of shared task of Mixed Script IR on Code-Mixed Hindi-English tweets was to retrieve 20 most relevant tweets from a document given a query. The query as well as the document was in Roman script but with CodeMixed Hindi and English languages. The corpus had set of documents with each document containing several hundred (or thousand) tweets. The corpus was further classified based on topics and queries. Each topic had at least one query related to the topic description. Table 1 explains about the structure of training/testing corpus provided for the subtask. The total number of topics for training and testing corpus was 10 and 3 respectively. There were several queries based on each topic (See Table 1) and there was at least one query per topic. The total number of queries for training was 23 and for testing, it was 12. A narrative on each topic was also given in the corpus describing the details about the tweets under that topic. The topic 001 (Aam Aadmi Party) has four queries under the same description (Table 1). All these four queries had separate documents with a corresponding number of tweets. Let be the given query, IR task was to rank the tweets in the corresponding document from most relevant with the query to the least.
Vector-Space-Models (VSMs) are used to represent documents as
a vector (of terms) that occurs within a collection [
To imagine the representation of term-document matrix, think of a multiset from set theory. A multiset is a set but it allows multiple instances of the same element. For example, = { , , , , , } is a multiset containing elements , and . Just like sets, order of elements in multiset could be anything. That means, multiset 1 = { , , , , , } is same as multiset 2 = { , , , , , }. Multisets are also called as bags and we can represent these bags as a vectors with vector component denoting the frequency of the elements of multiset i.e. = < 3, 2, 1 > is vector representation of the bag M in which 3 is the frequency of and 2 is the frequency of etc. Using the same analogy, we can imagine a document as a bag and set of documents as set of bags aligned as columns in a matrix, say . This matrix, , is term document matrix with columns representing a bag and rows representing a unique member. A particular element in the matrix corresponds to the frequency of ℎ term in the ℎ document (or bag). To capture the whole intuition, let’s assume 3 documents as: Doc1: We stayed very closely connected.
Doc2: Charger stayed connected with phone.
Doc3: His phone charger closely resembled mine.
The term document matrix of frequency for above three documents could be: In the above matrix, terms are the rows and columns are documents. It has 3 documents (Doc1, Doc2 & Doc3) and 11 unique terms (tokens in this case) with dimension 11x3. In a similar way a given query could be represented as bag of words and estimating the relevance of query with the documents in such a manner is called bag of words hypothesis in Information retrieval. This hypothesis states that a column vector in a termdocument matrix captures the meaning of the corresponding document (to some extent). It should be observed that the column vector which correspond to a document in a collection tell us about the frequency of the words in the document with loss of actual order of the words. The vector may not capture the structure of a document as it is but it works surprisingly well with the search engines. We can compare the column (document) vectors to compute the similarity among them. This similarity can be computed using euclidean distance if we are assuming columns (documents) as points in the document space. If we are assuming columns (documents) as vectors in documents space, we can use cosine similarity to measure the similarity by the angle between the vectors. Larger the cosine, more semantically related the documents are. If 1 and 2 are two document vectors, then cosine of angle between them is computed as: cos( 1, 2) = || (
1, 1||. || 2) 2|| Where || 1|| and || 2|| are the length (or norm) of the vectors. The basic intuition behind using cosine similarity is that it captures the idea that the angle between the vectors is important, length of the vector is not (See Figure 1). The cosine is 1 when vectors are same or they point in the same direction ( ). The cosine value varies from 0-1, zero being not similar and one being exactly similar.
Generally, most frequent terms will have lower information than the less frequent or surprising terms. To capture this idea, most efficient way is to use − (term frequency-inverse document frequency). An element in a term-document matrix gets a higher weight when a term in corresponding document is very frequent ( ) that means the term is rare in collection of documents( ). Hence the weight of a particular terms appearance is computed as: = ×
demonstrated in [
The subtask II in FIRE was Mixed Script Information Retrieval on Code-Mixed Hindi-English tweets. There were total 23 files containing tweets for training and 12 files for testing. Each file had a corresponding query. Given a query , the information retrieval task was to compute the similarity between the query and tweets in each file and return the top 20 most relevant tweets. As explained in the last section, this query processing task can be successfully executed using VSMs.
Each file was treated as a collection of documents and each tweet within the collection is referred as document. The dataset comprised of Hindi-English code-mixed tweets. As twitter data is generally noisy and requires some preprocessing, it was subjected to some preprocessing modules. The preprocessing in our implementation included tokenizing, removing stop words, stripping punctuations, stripping repetitions (hiiiiiii→ hi) etc. The major issue in tokenizing twitter data is to capture the key attributes of tweets such as: hashtags (#aap), @ mentions (@timesnow), URLs, symbol, emoticons etc. These attribute were captured using regular expressions. A sample tweet after capturing these nuances, stripping punctuation and tokenizing appeared as: Original @respectshraddie shhhhh :( salman ko jail hojaegi :( #badday ‘@respectshraddie’, ‘shh’, ‘:(’, ‘salman’, ‘jail’, ‘hojaegi’, ‘:(’ , ‘#badday’ 2 https://www.math10.com/en/geometry/geogebra/geogebra.html Tokenization was done for all the queries too. The document vector (column) size, as well as the query vectors, were in same vector space. The preprocessing was performed for each file (collection) over each document (tweet). After preprocessing over each file (collection), it was fed to Information Retrieval system. The similarity scores for each tweet in a collection given a query were computed and results were saved in a list. The top 20 tweets related to the given query were retrieved from the index values of top 20 similarity scores in the list.
There was a provision of submitting three systems per team. We submitted two systems. One system was same as explained above. In second system, we manually removed some Hindi stop words like , , , , , etc. It didn’t reflected any better results. All the implementations were done in Python 2.7. Related code will be made available at author’s Github page.
The result were declared roughly after two weeks of the
submission. There were total 7 teams and our system performed
well as compared to others [
The shared task on CodeMixed Information retrieval was indeed a unique task. It captured the latest trend in social media. We used Vector Space Models (VSMs) of semantics to compute the similarity between the tweets and given query. The performance of our system was ranked 2 among all the participants. But the Mean Average Precision (MAP) value was very low in terms of performance. That suggests, CodeMixed IR task is a difficult task and existing algorithms do not perform as expected and require sufficient attention to perform well for such data.
The authors would like to thank the organizers of Forum for Information Retrieval Evaluation (FIRE) for organizing this event. The authors would also like to thank the organizers of shared task on Mixed Script Information Retrieval (MSIR) for organizing the much coveted task for Indian social media.