This paper describes the participation of the Syllabs Team in the content analysis task of the CLEF MC2 Evaluation lab. In the current state of our work, we offer preliminary solutions to first detect the language of the microblogs used within the task, then extract the named entities that will be later used to recognize Wikipedia entities and finally, detect microblogs that deal with festivals.
The presence of festivals on the social media is constantly increasing. An analysis1 of the 30 biggest festivals in France made in 2015 was already showing that 97% of them were holding a Twitter account, while the number of followers was larger77% than in 2014. Tweets are a wonderful mean to be visible for events such as festivals. However, there are a few limitations about using Twitter: first of all, a single tweet may provide partial information due to the limitation of 140 characters; secondly, there is a lot of noise on Twitter (advertising, close events, etc.), or thirdly, important information may be drawn among the huge volume of tweets produced.
The first task – Content Analysis – of the MC2 evaluation [
Our goal2 when participating in the MC2 evaluation task is threefold: firstly, by evaluating the Syllabs in-house technologies we hope to improve their performance, especially regarding linking data and Wikipedia recognition. Secondly, by experimenting festival detection we try to settle a method for event detection, based on data gathering, and that could lead to event analysis or specific information retrieval. Lastly, by working on microblogs we continue our exploration on limited content data that can be rich in relevant information and usable data.
In Section 2, we describe our current approaches regarding the different subtasks of CLEF MC2 Lab we participated in. 1 http://www.socialband.fr/docs/les_festivals_et_les_reseaux_sociaux_en_2015.pdf 2 http://www.agence-nationale-recherche.fr/?Project=ANR-14-CE24-0022
We participated in three tasks of the MC2 Lab: - Task 1.1: Filtering microblogs dealing with festivals; - Task 1.2: Language Detection; - Task 1.5: Wikipedia Entity Recognition.
Our main stream is starting with Task 1.2, since Language Detection is the precondition to the two other tasks as it helps us focus on the analysis of a specific language. 3 3.1
Specific data used for the training of Language Detection is detailed in Section 4.1.
Regarding the MC2 data, and due to time constraints, we only worked on text and kept aside metadata. Thus, we reduced the tuning work, but also the performance gains that metadata could have provided. Test data is of course processed to submit the results, and we also used the full stream of June 2016 for Task 1.1 to help the filtering of tweets dealing with festivals. While the test set is composed of 1,100 tweets, the June 2016 set is much bigger with more than 4.3 million tweets. 3.2
Analyzing microblogs needs specific context and treatment due to the peculiar nature of the data. Information is reduced to its simplest concept and the shortness of each microblog makes this information hard to retrieve.
Noise is one of the most relevant characteristics to filter. In microblogs, noise usually prevails over relevance but the few pieces of useful information can have a strong impact.
Thus, data preparation, noise filtering and preprocessing are essential parts of the full process, and vital for the following subtasks.
Basically, we prepared the input data using the following steps: - Removing hashtags and nicknames, specifically for language detection: although they convey meaning, we suppose that most of the hashtags and nicknames do not disambiguate languages (in French, for instance, there are plenty of them written in English or pseudo-English); - Removing all symbols and punctuation marks: although some of those characters could help for language detection, such as “¿” used in Spanish, our tests showed that they were not relevant; - Keeping one character when it is duplicated more than twice (“okkkkkk” for instance) so as to better fit the “traditional” language models; - Not considering case.
Many methods and studies exist to detect the language of textual contents, including microblog-oriented data. The specific nature of tweets, through their size and specific vocabulary, makes the task particularly complex. Microblogs are usually coming with metadata but language is generally not reliable, firstly because the language of a given writer is not always the same, secondly because the geolocalization cannot determine the language of a tweet. Therefore, we need to apply other methods to detect the language of a given microblog.
We used, adapted and tested four different methods (cf. Sections 4.1.1 to 4.1.4) that are not specific to microblogging. The main difficulty is to find a training corpus that would simulate microblogging. We describe the corpus in the following section and the results of our experimentations are detailed below. In our first experiment we tested a very simple method using microblog n-grams, implemented from scratch. Microblogs being, by definition, short, we took away the possibility to use word n-grams, and thus we focused on letter n-grams.
Then, we first built a small microblogs training corpus, composed of 3 sources:
- The corpus from the SEPLN Workshop 2014 [
Thus, the whole corpus represents more than 200k annotated tweets. 95% of this corpus was arbitrarily used for the model training, the other 5% being used for testing purposes. Table 1 shows the results obtained using from 2-grams to 7-grams on the test corpus.
After those tests, we determined that using 4-grams gives the best results.
4.1.2
LangID [
CLD2 [
Guess Language [
The test corpus was filtered to keep 5 languages: French, Catalan, English, Portuguese and Spanish. Table 2 shows the associated figures. The low number of French tweets can be explained by both the lack of data and the need to keep part of the French tweets for training.
We computed precision so as to estimate the performance of the method. We also tracked the computation time for all the methods. Results can be found in Table 3.
Our basic method obtains good results despite of its simplicity, and we guess that with more training data, results could be comparable to other methods. On the other hand, our method is more adapted to the test since it was trained with the same limited languages while other methods were trained with more languages.
Higher results are obtained by LangID with more than 92% correctly identified. Considering the low quality of some tweets, this result is quite good.
The cleaning we apply to the tweets shows a slightly improvement of the performance, except for the Guess Language method, while the timing remains similar.
One last interesting result is the timing of our method, although CLD2 obtains way better results, with a comparable precision and more languages available.
In the context of the CLEF MC2 Lab, we use the LangID method with cleaning, since it obtains a higher performance. 4.2
(Wikipedia) Entity Recognition (Task 1.5) We have used the named entity recognition system that was developed by Syllabs in this task. It is a rule-based system which has been used for years but not adapted to tweets. Since tweets are short, uppercase is not a relevant clue to find named entities. So, we had to improve our lexicon. We added a large number of organisms, places and persons. In the last few months, more than 1,5M entries have been added to our system, plus a few rules regarding new extractions for specific contexts, adaptation to new entry features.
Regarding the specific task of the CLEF MC2 Lab, we have simply used the Wikipedia API3 to link our entities to the encyclopedia and translate them into other languages, for English, French, Portuguese and Spanish.
For each tweet in a given language, we provide the original entity extracted, and the translation for the other languages. 4.3
Filtering contents to a certain type of event is particularly complex especially when dealing with microblogs. Most of the time, in microblogs, the information given is not relevant enough to do that filtering. Hashtags may help, as well as specific keywords, but we cannot count on them for most of the microblogs.
Moreover, the concept of festival and how people write about a given festival are hard to define. A person who is happy to be in Cannes during the festival may be in Cannes for another reason. Depending on the context, we may, or may not, keep that kind of microblog.
Our method to filter microblogs works on a full microblog corpus (vs a single microblog) and is based on clustering the microblogs. It applies the following steps: 3 https://www.mediawiki.org/wiki/API:Main_page
Removing duplicates and retweets, while keeping track of them: We then avoid
biasing the clustering by using similar tweets, as well as giving very large
weights to the information given;
Detecting the languages (from Task 1.2) on the remaining microblogs: The
language detection is twofold, first by reducing the clustering process on a large
data set, second by doing a first gathering of similar linguistic information;
Clustering for a first time the microblogs on a time frame (i.e. a month or a
week), per language: A DBSCAN clustering is applied that, according to our
experiments, gathers similar (i.e. very close to duplicates) tweets and thus
reduces our data set again [
By regrouping topics on the fourth step, we increase the possibility of finding tweets related to festivals that could not have been found by simply extracting festival-related ones. 5
We submitted results on the three tasks presented above. Although evaluation results are not available yet, we provided a few statistics below. 5.1
The 1,100 tweets from the test corpus have been preprocessed, then analyzed using LangID. Results are shown in Table 4:
Tweets are mainly written in English (more than half of the test corpus), then Spanish, Portuguese, French, Italian, Japanese, etc. More unusual languages are also present in the corpus, such as Aragonese, Bengalî or Bokmål. The Wikipedia Entity Recognition has been processed on English, French, Portuguese and Spanish. Table 5 shows the entities found per language. 339 tweets contain at least one Wikipedia entity and a total of 550 entities (for 414 unique entities) have been found. Translation is possible only when the link in Wikipedia is available.
Unfortunately, our Entity Recognition system has found many more Entities that do not exist in Wikipedia. This is certainly due to the less well-known events or specific information included in the corpus.
By using our method, we have found that 734 tweets are dealing with festivals. Most of them are tweets in English (439), then Spanish (124), Portuguese (36), French (31), etc. 6
This article describes the first experimental results on three tasks of the MC2 evaluation Lab. We mainly enhanced our stream system by adding preprocessing that allowed us to improve the results of the Language Detection slightly.
Wikipedia Entity Recognition and Microblogs filtering are both basic methodologies and, although first results seem promising, they can be improved in many ways. Our rule-based system for Entity Recognition requires more lexicon and further work should be done on the rules, while the linking with Wikipedia could be done using disambiguation or word variation techniques. The clustering process shows limitations to filter microblogs, especially because we need a large amount of tweets, which leads to performance decrease.
We will continue this preliminary work within the “gallery of festival” project (GaFes) so as to extract proper content and to be able to represent festivals in a social context.