A sizable percentage of online user generated content is susceptible to code switching and code mixing owing to a variety of reasons. Thus, an expected consequence is that adhoc user queries on such data are also inherently code mixed. This paper thus presents our solution for a similar scenario : information retrieval on code mixed Hindi-English tweets. We explore techniques in information extraction, clustering and query expansion as part of this work and present our results on the test dataset. Our system achieved a MAP of 0.0217 on the test set and placed third on the rankings.
A large number of languages like Russian, Hindi, Arabic, etc._ that although have indigenous writing scripts, are often expressed in the Roman script in online communication due a variety of socio-cultural reasons. Furthermore, it is not uncommon to see multilingual speakers switching between di erent languages when expressing themselves in an informal setting like social media. Hence, a large percentage of user generated content, typically on social media is code switched or code mixed, i.e., contains content in more than one language that may or may not involve multiple writing systems.
Given the nature of the content, it is thus not unexpected
to see adhoc user queries on such data to be also code mixed.
For example, consider Table 1, that lists some sample queries
and sample target tweets that illustrate the given scenario.
The given problem was more formally introduced recently
by [
Given a topic T , speci ed using a name, description and narrative; a set of associated English-Hindi code mixed tweets tT ; and a set of similarly code mixed queries QT against the topic, the task involves returning the top k results for every such query. A summary of the training dataset is presented in Table 2. Some of the key challenges thus, can be enumerated as: 1. Short and comparable document lengths : Judging relevance on curated tweets can be especially hard given that not only the documents are short in length, thus, giving little context to work with but also that all documents are comparable in length and are de nitely somewhat relevant to the query / topic to begin with. 2.
The training dataset can be summarized as follows:
2. Queries: Total of 23 queries, ranging from 1 to 4 per topic 3. Tweets: Total of 6142 tweets split between topics/queries and ranging from 34 (Topic 9 / Q 21) to 3531 (Topic 1 / Q 4)
Based on the question types and available data, we make the following assumptions / observations: 1. Each query is an informational query about some Named
Entity (NE) 2. Every query can also be completely described by a triple as A[-rel -B] where [xx] indicates an optional clause
S.no Sample Query Example Tweets 1 delhi me election - rohit sharma ko dilli ka chunav ladwao !! newsmaker of the day 2 india ki haar #aapkasting thanks india ki haar ka dukh bhula diya tum logo ne 3 nirbhayaa ka rapist asharam bapu ka kehna hai mujhe nabalig samajhkar hi chod do . #nirbhayarapistout 4. When not a NE, A or B, are most likely a noun 5. NEs would usually be represented by a xed number
of variants 6. Nouns on the other hand, could be represented by any number of synonyms that could be drawn from both languages (chunaav versus election for example) 7. The ordering of the clause (Noun-rel -NE versus NErel -Noun) is dependent on the underlying language (chunaav in delhi versus dilli mein election)
Apart from the training dataset, no relevance judgments were provided, binary or nuanced. The narrative with each topic provided some insight into what was considered relevant and non-relevant, but there were no clues to di erentiate between two tweets given they both referenced the same number of query tokens. Thus, we need to derive some notion of aboutness or information content of each tweet to determine relevance. Thus, a possible search strategy can be formulated as: 1. Determine NE or NEs within the query and the tweets - this could be a xed list or processed from Wikipedia or other Knowledge Bases (KBs) 2. Find tweets that match as many tokens or as much of
the remaining query as possible 3. Find some proxy for tweet aboutness and use it to
determine relevance
Using these three guiding principles, we now present the di erent techniques we tried out and how they contributed to our nal system run.
In this section, we describe the di erent experiments we conducted and the various system components that comprise our nal runs. 3.1
We used Apache Solr - an open source, scalable text search engine written in Java and built over Apache Lucene as the back-end for our system. It is fairly straightforward to index any sort of data in Solr after de ning a schema. We present our full schema de nition in Table 3. Thus, apart from the data already provided, we augment it using two elds : NE (see Section 3.2) and cluster id (see Section 3.3). 3.2
We implemented the NE tagger in two parts : (a) a regex based extractor that generates a list of known NEs and (b) a simple string matching based tagger that are explained as follows. For the NE extractor, we used the topic description les and automatically extracted longest possible sequence of tokens wherein the sequence is delineated by a token that begins with a capital letter. We then sorted the list extracted as thus and eliminated duplicates and subsequences. However, for every sub-sequence that we eliminated, we tagged the parent entity as being capable of being present as a sub-string. For example, the entity Narendra Modi could be present as a whole or as individual tokens namely Narendra or Modi. At the end of this step, we had thus generated a list of NEs (of interest at least) and a boolean ag indicating sub-sequence occurrence. A snapshot of the extracted lists from the training dataset is shown in Figure 1.
The actual tagger followed a similar philosophy where we
started with the above list and tagged each tweet as follows.
We simply created a mapping from token to entity, wherein
the token was either de ned as the full entity or any of its
sub-tokens if it could be present partially. For example, in
the above case, for Narendra Modi, we created two mappings
as Narendra ! Narendra Modi and Modi ! Narendra Modi.
For each tweet, we then performed a look-up and added the
matching NE to the index if a match was found. Further,
we also added a Solr plugin for the modi ed Levenshtein
distance [
Finally, we attempted to mine cross-lingual equivalents
and spelling variants for di erent words. Much akin to the
work of query expansion by [
Description
Unique identi er for each tweet, UUID generated from raw text
Raw text for the tweet, stored after tokenization and minimal processing
Query Number
Topic Number
List of Named Entities referenced within this tweet
Union of cluster ids for all constituent tokens within the tweet
of Brown Clustering [
Thus, we automatically generated a synonym list using the above results using a two step process. First, we removed all words that were found in a standard English dictionary. This was done to remove any false positives. Secondly, we deemed a set of words to be equivalents if their edit distance was less than a pre-determined threshold of three. The said threshold was manually determined by trial and error over a small test set. All such matching pairs thus generated were then combined into a single synonym le as partially shown in Figure 2.
Having described the individual components that comprise our system, we now turn our attention to presenting the overall system in the following section.
The basic system architecture is shown in Figure 3. It includes the following components as introduced earlier: Solr (and its associated index), Query Processor (that includes NE tagger) and the extracted synonyms. Two additional inputs are (a) the search con guration that de nes the different weights, choice of scoring mechanism etc._ and (b) the topic narrative les that are additionally ingested as described below. Note that the diagram shows three di erent con gurations, additive in nature for each of the three runs submitted.
Before we describe the three runs, we present some additional details on the query processor and Solr con guration. In continuation with the schema as presented in Table 3, given: - Query Q of n tokens as q1, q2, ... , qn - Predetermined query weights Wf = w1, w2, ... , wk for elds f1, f2, ... , fk as stored within the con guration
The processor then partitions or generates eld level parses of the query as Qf = q1f , q2f , ... , qkf and passes the nal query to Solr as Qf Wf .
Secondly, as mentioned in Section 1, we also needed to
gure out some notion of relevance. Given the lack of any
additional information, we chose a simple voting based scheme.
Solr supports a variety of di erent ranking models and thus,
we con gured four distinct Solr instances each with a di
erent relevance scheme as enumerated below:
1. Lucene Similarity: This is an implementation of the
classic tf-idf similarity that uses ptf and 1 + log( Nd+f1 )
as normalization factors
2. Okapi similarity: A probabilistic relevance model as
introduced by [
For the rst three models, we used the default settings and for DFR, we manually experimented with a few queries and returned results to settle on the selected model. For a given query, the system executes it in parallel on the four
Topic subsystems, normalizes the returned scores and combines the results into a single ranked list.
We enumerate the three runs as follows: 1. Run 1 - Named Entity boosts : In the rst run, we performed two levels of query matching - one was boosting the documents based on their NE matches from the query, i.e., the query was parsed to extract NEs and each document (tweet) that matched the given NE was provided a small numeric boost. The second level of boosting utilized phrase matching, i.e., documents that more closely matched the input query phrase were ranked higher than those that did not. 2. Run 2 - Synonym expansion: We merely expanded the given query based on these synonyms over the ranking mechanism presented for Run 1. 3. Run 3 - Narrative based weighting: For the nal run, we extracted NEs from the provided topic narratives and assigned positive or negative boosts based on the associated word usage \relevant" and \not relevant". These additional weights were applied over the scheme presented in Run 2.
We present a summary of the test dataset and the results from each of the runs in Table 4. We make the following observations:
Overall, the best system performance was our baseline system. Synonym expansion performs second best and narrative based weighting performs the worst.
In hindsight, the techniques used by us perhaps improved recall at the cost of precision The only queries where synonym expansion works better than not having any synonyms is perhaps where there is a spelling mismatch between query and text (queries 10 and 12).
Thus, we could have used \pessimistic" expansion only use it if adequate results not available.
When multiple NEs are present (queries 1-3), our system gets confused and returns non relevant results.
As part of future improvements, we thus need de ne a better notion of relevance and aboutness at a tweet level, speci cally ascertain the information content of each tweet. In summary, our work showed three important results : (a) Named Entities have a very important role in IR on tweets and system performance could be improved by using more sophisticated NE taggers (b) simple clustering on topic wise tweets can give considerable insight into the constituent words - enough to derive spelling and in ectional variants and (c) in the given setting, precision is very sensitive to small changes and hence, typical recall improving techniques should be used as a \last resort". 6.