A group of bachelor students in information science at the University of Groningen applied off-the-shelf tools to the detection of events on Twitter, focusing on Dutch. Systems were built in four socially relevant areas: sports, emergencies, local life, and news. We show that (i) real time event detection is a feasible and suitable way for students to learn and employ data mining and analysis techniques, while building end-to-end potentially useful applications; and (ii) even just using off-the-shelf resources for such applications can yield very promising results.
The availability of a constant flow of information in the
form of short texts makes it in theory possible to collect real
time information about virtually all sorts of events that are
being written about. The attractiveness of this is evident,
and so is its potential social utility. ReDites, an event
detection and visualisation system
However, how the twitter stream can be successfully exploited to this end isn’t straightforward, neither conceptually nor practically. First, specific events must be detected, and related tweets clustered. This step must rely on a definition of what an event is, which is often application dependent. Second, the retrieved tweets must be filtered and processed to minimise noise, both in terms of pertinence as well as in terms of the noise typical to the nature of tweets. Third, in order to produce a meaningful tool, the output must be evaluated at a development stage, and made as usable as possible for the end user in its final form, for example providing customisation and visualisation features.
In this short paper, we report a series of efforts within a bachelor programme in information science, where a group of nine students developed different systems with different specific aims, all exploiting real time tweet-derived information, and all socially relevant. The systems work with Dutch tweets, but their architecture is virtually languageindependent, as long as tweets and basic language processing tools are available. Suggesting novel methodologies or applications for event detection wasn’t our primary concern when developing the systems and when writing this contribution. Instead, by describing this collection of different student projects in the area of real time event detection towards social utility, we have a twofold aim. First, we show that even leveraging off-the-shelf tools and basic preprocessing can yield interesting, promising, and even unexpected results, with a variety of applications. Second, as we have observed that event detection on Twitter has been a useful and suitable task for students, helping them gain familiarity with data mining and data analysis while putting together end-to-end systems, we hope to inspire others to embark on similar exercises. 2.
The University of Groningen extracts Dutch tweets
from the Twitter Firehose and provides access to the stream
for students and employees
The definition of an event was very application dependent, but always informed by some secondary information, be that peaks in Twitter usage, shared time and location of tweets or overlap with news headlines. Practically and generally speaking, tweets about a single event overlap in some way, and finding sophisticated ways to detect this overlap was a core challenge for all projects.
For none of the developed applications, typical noise in tweets was of any particular concern, although hashtags and URLs were often removed or replaced by placeholders in preprocessing.
Visualization for end users was provided by a few of the
projects
The students’ systems that we describe showcase four different application areas: sports, emergencies, local events, and news.
Sports fans like to stay up to date with real time scores
by accessing websites such as livescore.com, which
provides live overviews of football matches for major
Leagues.1 The overview consists of tables including major
events in the game, like goals and yellow/red cards. It takes
a lot of time to manually input in-game events, which is
why obtaining reliable real-time updates can be expensive
and automatising this process leveraging real time Twitter
data becomes attractive. Three of the projects were
concerned with automatically reproducing such tables by using
the stream of Twitter data to automatically detect and
classify in-game events. We describe here one of the developed
systems, where matches of the Dutch national team in the
1Companies such as Livescore.com buy real-time information
for prices that vary according to the prestige of the League.
2014 World Cup of soccer are used as a case to demonstrate
the approach
The selected five matches featured a total of 19 goals, 15 yellow cards and no red cards, so only the first two types of events were predicted. For each match, two hours of Dutch Twitter data from the first minute of the match was collected, resulting in a total of 4.376 relevant tweets.
The data was then modified to allow for a more general application of the system. This involved replacing specific scores and players with placeholders. Occurrence of specific keywords that denote an event were used as features, along with the length of the tweet and the tf-idf term vector.
Beyond detecting events in single tweets, detecting
events has to do with grouping relevant tweets in the right
way. Detection of peaks in Twitter activity has been used
to detect events
Using the Multinomial Naive Bayes implementation in
Scikit learn
Beyond achieving good results for a case of very
specific matches in a very specific sport,
Twitter allows for detection of real-time sub-events in sports because relevant tweets follow these events almost instantly. The delay between a real-time occurrence and its social resonance are thus minimal. This adds to the social relevance of detection of events that are particularly timesensitive, such as emergency situations. This section will look at two different emergency scenarios: earthquakes in the Dutch province of Groningen, and detection of context for events reported by Dutch emergency services.
The detection of earthquakes on Twitter has been
extensively documented
Detection of earthquakes in Groningen has only
recently become relevant since gas extraction in the province
led to a 200% increase in earthquakes over the past ten
years
To develop his system for detecting earthquake events
via Twitter,
Using Weka
P20003 is a live repository of all emergency services active in a given area. All reports are publicly available and real-time updated communications of and between Dutch police, ambulance and fire department services are available. You can think of the first reports of a fire, the cars inbound to the location of the reports and the way the distress is handled.
The work described in
One month of emergency reports and tweets were downloaded from the P2000 website in April 2015 for the larger Groningen area. This resulted in 700 ‘matches’ of reports to one or more tweets with a similar location and time. A Naive Bayes model was trained on 80% of the annotated data, using simply word occurrences as features, to classify tweets as ‘relevant’ or ‘irrelevant’. Testing on the remainder 20% resulted in a significant improvement over the baseline (75% of tweets were annotated as irrelevant) with a final accuracy of 91%. The limited amount of data, and the skewedness between relevant and irrelevant tweets does not make this result generalizable to a real-time application for detecting emergency situations, but successful matches do add some context to otherwise formal reports.
The approach in which words are used to predict relevancy allows for an overview of the most indicative words (Table 2).
A critical reflection on these results can be that
The meta-data attached to tweets can be useful for
certain instances of event detection.
Detecting events using geo-locations from Twitter has
previously been done by
All geo-tagged tweets from a month of Dutch Twitter data were used for training. This resulted in a total of 566.549 geo-tagged tweets. The geo-information was translated into a geoHash that denotes a specific area, and tweets with a similar geoHash and comparable timestamp were grouped and added to a list of event candidates. To handle the hard borders of the geohash area, candidates with matching timestamps in adjacent areas were then merged (Figure 1).
Two judges annotated the event candidates in the
training data and the test data with the following labels: No
event, Meeting, Entertainment, Incident, Sport and Other,
with the most frequent category being No event (triggering
a 46% baseline). Inter-annotator agreement was measured
via Cohen’s Kappa
Several models were built and tested on development data, with the final system being a Naive Bayes model which yielded an accuracy of 84% on test data. Especially considering the inter annotator agreement was measured at K = 0:87, this is a very good result. This research also shows that meta-information from tweets can successfully be used to detect events. The end-user output of the system is a map with classified events (Figure 2). 2.4.
Twitter data has also been used to detect real time commentary on news events. News media websites often feature their own social plugins which allow readers to discuss news items. These discussions are relatively structured and easy to relate to the news article. However, when people post their commentary to Twitter, it becomes problematic to link the tweet back to the article and to group all relevant discussion together.
In
To deal with the computational effort involved in making this many comparisons (an hour of Dutch twitter data often contains more than 30,000 tweets), a first module of the system makes a pre-selection of candidates to be considered. The candidates are made up of the 25 tweets with the highest cosine similarity compared to the title of the news items. Generally, these 25 tweets contain tweets that are too similar to the title (a retweet for example), too dissimilar (are not about the article) and actual relevant tweets that add meta-commentary. It is the latter type that one would want to detect and use, while discarding the former two as non-relevant.
To distinguish between relevant and non-relevant tweets, four non linear machine learning algorithms were trained and tested. Four news articles from the 21st of May 2015 were selected to be compared to 24 hours of Dutch Twitter data from the same day. For training and testing, 250 candidate tweets were selected using the approach mentioned above. This cut off point was chosen because after annotation it became clear that for each of the articles, the number of relevant tweets that could be found after rank 250 was negligible.
In the 1,000 candidates, 593 were annotated as relevant to the articles which results in a baseline of 59.3%. The system was trained on six features: (1) the difference in timestamp between the publication of the article and the publication of the tweet, (2) the cosine similarity between the title of the article and the tweet, (3) the difference in length between the title of the article and the tweet, (4) the cosine similarity between the abstract of the article and the tweet, (5) the amount over overlapping Named Entities, (6) the cosine similarity between bigrams in the abstract of the article and the tweet.
Named entities were extracted by means of a Named
Entity Recognizer trained on the CoNLL2002 Dutch
corpus using NLTK
The selection of viable candidates before automatic classification proves successful in reducing computational effort, while still keeping the detection of relevant commentary possible. This approach is therefore suitable for a realtime application of the system. demonstrates this by applying the system to a specifically dedicated website that updates its news articles and tweets hourly (http://www. nieuwstwiets.nl, Figure 3). Rather than representing the results of the research with an F-score, it provides insight by showing users the resulting tweets.
3.
In this overview we have reported efforts of bachelor students in the field of automatic event detection exploiting the (Dutch) Twitter stream.
Besides the differences in fields of application, this overview gives insight in the considerations that were made in dealing with the inherent challenges in event detection. For the emergency detection and the detection of local events, geo-information of tweets was used. Since this information is not always available, this sometimes resulted in very little data to work with. For the detection of subevents in soccer, peaks of tweets with certain key patterns were used. This worked well for important subevents (goals) and worse for minor subevents (yellow cards).
The students used similar ways to remove noise from
tweets, by removing or replacing URLs and hashtags. Even
when hashtags were crucially used to detect events, such as
in
Finally, evaluation showed good results in all the theses.
Some students chose to apply their findings in a real time
setting by visualizing them. This lead to demonstration of
the systems by
With this exercise we observed that real time event detection on social media is a field that students can successfully experiment with. Although the aim was not to build the next generation event detection applications, the choices that the students made in the course of such a research reflect some of the core challenges and considerations central to this task and we believe are useful lessons for future endeavors, both from a research and a teaching perspective. It also reflects that there are no ready-made best practices when it comes to defining events and selecting data, and that each socially useful task will have its own needs and therefore strategies.