In this paper we present a system for personalized o ers based on two main components: a) a hybrid method, combining rules and machine learning, to nd users that post life events on social media networks; and b) an entity matching algorithm to nd out possible relation between the detected social media users and current clients. The main assumption is that, if one can detect the life events of these users, a personalized o er can be made to them even before they look for a product or service. This proposed solution was implemented on the IBM InfoSphere BigInsights platform to take advantage of the MapReduce programming framework for large scale capability, and was tested on a dataset containing 9 million posts from Twitter. In this set, 42K life event posts sent by 19K di erent users were detected, with an overall accuracy of 89% e precision of about 65% to detect life events. The entity matching of these 19K social media users against an internal database of 1.6M users returned 983 users, with accuracy of about 90%.
Social Media Networks (SMN), such as Twitter and
Facebook, engage thousands of people that post, on a daily
basis, a huge amount of content represented by texts, images,
videos, etc [
One way to nd potential customers for services or
products is by detecting life events from public user activities
on SMNs, in special microbloggings. Generally, a life event
can be de ned as something important that happened, is
happening, or will be happening, in a particular individual's
life, such as getting married, get graduated, having a baby,
buying a house, and thus forth. That is, if a life event is
properly detected, a product or service can be o ered to
someone even before she looks for it, anticipating her needs.
For instance, if a person posts on the SMN that her marriage
will be happening in a few days (or weeks or months), a loan
or an insurance (for the honey moon trip for example) can
be o ered to her in advance. Furthermore, as state in [
For this reason, this work focuses on presenting a system
that can detect life events from textual posts on SMNs,
and can match the corresponding users with an existing
database, i.e. entity matching with current clients, using
basic information such as the name and the location available
on the SMN. Entity matching is important to understand
whether a given user of a SMN is already a customer or not,
and adapt the way the person can be approached.
Both life event detection and entity matching are complex
tasks which are subject of various research in elds such
as arti cial intelligence, machine learning [
In the light of this, in this work we describe and evaluate our
proposed solution to tackle the life event detection problem
and the entity matching. For the rst task, we propose a
hybrid system combining rules and machine learning (ML).
In contrast to the system speci cally focused on life event
detection presented in [
The entire system has been implemented on the IBM
InfoSphere BigInsights platform [
Since the work proposed in this paper is a hybrid solution
on which we integrate a ML-based classi er with an Entity
Matching solution, the background and related work is
presented separated for both as follows:
Life Event Detection: as already mentioned, a life event
can be de ned as something important regarding the users'
lives in SMNs. It is important to di erentiate it from some
related work which uses the event detection expression to
refer to the problem of detecting unexpected event exposed
by several users in SMNs like a rumor, a trend, or emergent
topic. In the case of the work proposed in this paper,
detection means to classify a short post, like Twitter's or
Facebook's status messages in one of the life event categories,
which could be considered, for instance, topics. Therefore,
as related work, any approach of topic classi cation of short
messages could be considered like [
The former problem can be considered a subset of the latter
if we ignore the fact that the second set contains real people
information rather than SMN's pro les. And generally, as
summarized by [
Regarding, syntactic-based similarity approach, we
summarize here the ones most used for URI, numeric-based
attributes and, in the context of SNMs, two users' full names.
Levenshtein or Edit Distance [
Another approach is the N-Gram name similarity, on which
N-grams are sub-strings of length n and an n-gram
similarity between two strings is calculated by counting the
number of n-grams in common (i.e., n-grams contained in both
strings) and dividing by either the number of n-grams in the
shorter string (called Overlap coe cient), or the number of
n-grams in the longer string (called Jaccard similarity), or
the average number of n-grams in both strings. 2-grams and
3-grams have been used to calculate the similarity between
the two users' full names. Finally, the VMN name similarity
approach proposed by [
In this paper, we use two versions of ED preceded by Jaro's similarity as described in the next section.
In this section we describe in detail both systems for life event detection system and entity matching.
Given a social media network, the life event detection system has as main goal to return a list of users that posted life events within a given time window. This task involves a crawler to gather data, and a system to search for life events on the data. Note that not only accuracy is important in this case, to nd the largest list of users with a high precision, but also performance is important since the system is likely 1http://www.foaf-o-matic.org/ 2http://www.okkam.org/ to face a large amount of data. In addition, on a production environment, the system must allow for easy ne-tuning, addition and removal of life events classes. To cope with the aforementioned issues, we propose a hybrid life event detection approach, combining both rules and machine learning (ML). Such a system, depicted in Figure 1, is basically composed of three subsequent phases or modules, namely Ingest, Filter, and Detect. The rst phase, i.e. Ingest, captures a database of posts to be used for the search for life events. This is done by considering a set of words that can possibly relate to all life events of the system. We assume that the larger this dataset, the larger the set of users that will be returned. Once the set of posts has been totally crawled, the Filter module selects the set of posts that is more likely to contain life events. That is, by considering a set of simple rules such as words and combinations of words (but more elaborated rules than those of Ingest), but in this case a set of rules for each type of life event, the posts that match these rules are marked with the corresponding possible life events.
Despite these rules can indicate a possible life event, a large portion of these message can be false candidates. For this reason, the Detected phase is then carried out to validate the possible life events with their corresponding probability. For each post found in the Filter phase, we apply the ML classi er of the corresponding possible life events and compute the probability of that the post contains the given life events. With this information, all posts with life event probability above the threshold are selected and users of the corresponding posts are generate as the output of the system.
It is worth noting that currently ML is well-known to produce the best solutions to deal with ambiguous and noisy texts such as microbloggings' posts. However, the proposed hybrid solution takes advantage of the rule-based ltering to reduces the search space for the ML classi er, which can reduce both the number of errors and processing time. Moreover, by treating types of life events independently it makes it easy for ne-tuning, addition and removal of life event classes. For instance, to add a new type of life events, one need to append the corresponding keywords for the Ingestion phase, the rules for Filter, and a binary classi er in the Detect phase. This can be done with no impact on the accuracy of existing life events.
Given the output of the life event detection system, i.e. users (aka entities) that posted life events on social media, the main goal of the entity matching system is to nd corresponding people in a database of real names. For achieving this task accurately, the system must use as much information as possible to decrease the level of uncertainty. Dealing with users found on SMNs, though, is very challenging. First of all, on most SMNs the basic information about the user (e.g. name, location, age) is very limited (on Twitter only the name and location of the user are available). In addition, such personal information may be lacking or not relevant since lling them may be not mandatory, and the content led is not veri ed. Besides that, when the information is seriously provided by the user, other di culty factors can appear, such as the use of simpli ed names (Claudio Pinhanez instead of Claudio Santos Pinhanez), the use of social media pen-names (@cinhanez instead of Claudio Santos Pinhanez), or the use of nickname (Darth Vader instead of Claudio Santos Pinhanez.
To deal with some of the aforementioned di culty factors, for this work we have developed a system to match names and locations of users using three di erent string distance functions: 1. Exact matching (EM): a match is found if all the names of an SMN user are identical to those of a client 2. Entity Distance 1 (ED1): designed to consider misspellings and transpositions between adjacent characters as a match. For instance, the user \Jooa Paulo" matches the client \Joao Paulo", and the user \Carolina" matches \Carolina". In this case, the threshold 1 is used to de ne a match only if the similarity value is above this threshold. 3. Entity Distance 2 (ED2): designed to match abbreviations and some nicknames. For example, the user \Joseph S." matches the client \Joseph Salem"; the user \Fabinho" matches the client \Fabio", and \Mari" matches \Mariana". Similarly to ED1, the threshold 2 is used to de ne a match.
The execution of three aforementioned matching algorithms results in three distinct sets of users, denoted EM , ED1 and ED2. The resulting set of users All corresponds to the union of those individual sets. That is, All = EM [ ED1 [ ED2, where EM \ ED1 \ ED2 6= ; or EM \ ED1 \ ED2 = ;, depending on the data.
It is worth mentioning that the Jaro Winkler similarity
ltering [
In this section we present the results of applying the
proposed system on a dataset containing 9 millions of posts
from Twitter, which have been produced by about 1.4
million users. This data has been gathered by means of the
GNIP social media data provider [
Mainly, these experiments have two di erent purposes. First
we aim at evaluating the numbers related to applying the
system on this 9 million dataset, i.e. how many posts and
users are returned by using the system. And second, we
focus on a quality analysis to validate those numbers by
means of a manual inspection of samplings of this dataset.
The life event detection system has been implemented for
six types of life events: Marriage, Graduation, Travel,
Birthday, Birth,and Death. For each one, a training dataset of
about 2 thousand samples has been manually labeled as
either life event or non life event, and a distinct classi er has
been trained. The training data has been obtained with the
Twitter Search API [
As we mentioned, the rst experiment has as main purpose to evaluate how many posts and users are returned after carrying out each phase of the proposed system. The results of applying the implemented life event detection system on 9-million-tweet dataset is summarized in Figure 2. In this case, the Filter phase has returned 347 thousand posts from about 220 thousand users. Then, after going through the Detect module, 42 thousand posts, from about 19 thousand users, have been detected as life events. It is worth noting the large di erence in terms of proportion from one phase to another. The Ingest phase captures a very large dataset, i.e. 9 million posts. Then, Filter nds out that only 3.7% of these posts can be of interest. However, the Detect phase shows that from these 347K% of posts, only 42 thousand (0.45% of 9M or 12% of 347K) are really those that the application is looking for. Considering that many of the current search system are rule-based, these results indicate that our proposed system can avoid a useless search on about 88% of the posts returned, 307 thousand posts in this case. In Table 1 we present the results of the experiment above for each type of life event. We can observe that about 12% of the posts ltered have been generally con rmed as life events, but this proportion can vary according to the type of life event. For instance, for the Marriage class, from the 182,096 posts that the lter considered as possible life event, the machine learning algorithm detected 19,475 (10.6%) as being actually life events, which is close to the average. The Graduation type, on the other hand, presented a much larger proportion (43.21%), while Death and Travel smaller ones (5.47% and 8.26% respectively). We believe that this difference can happen either due to the period of the year in which the data is gathered (Graduation supposedly has more posts in certain periods of the year), or even due to the type of life event that may contain more non life events (Travel for example, which may present many posts from marketing agencies) or even less life event posts (for instance Death, whereas people might to be more introspective). containing 1.6 million users using publicly-available data. The users on this dataset have been matched against the 19 thousand users that have been detected as the ones that posted life events in the 9M dataset. The results and this process are illustrated in Figure 3. Note that we have conducted two di erent experiments. The rst one matches these users by taking into account only their names, since we consider this as the minimum information we will be able to obtain from the SMN. In this case, 983 users have been found as probable matches. In the second experiment, where both names and locations are considered, only 5 users have been found. This shows that the precision of entity matching can be increased considering more for this process. On the other hand, this will also reduce the size of the resulting matching set.
In order to validate the above results, we performed a
random sampling of 23 thousand posts (from the 9 million set)
focusing on quality analysis. The number of posts ltered
and detected are shown in Figure 4. The total of posts
ltered is 1,008, from which 105 have been detected as life
events. Similar to the results on the 9 million set, only about
10% of the ltered posts have been detected as life events.
Detailed numbers, for each type of life event, are presented
in the columns Filtered and Detected in Table 2.
To evaluate the entity matching, we have a built a dataset
Those 1,008 posts resulting from the Filter module have
been then manually inspected in order to verify whether the
Detect phase has assigned the correct probability or not.
The total of posts for each type of life event are listed in the
Ground-Truth column in Table 2. It can be observed that
our system presents numbers that are close to what was
found by the manual inspection. By comparing the manual
inspection with the results of the system, we have been able
to compute the confusion matrix presented in Table 3, which
contains the total number of true positives, true negative,
false positives and false negatives. This has allowed us to
compute the values for accuracy, precision and recall [
In this work we presented a system for personalized o er based on life event detection. Once the system detects users posting life events on a social media network, these users are matched against an internal database of clients to decide what is the best approach to o er them a service or product. We described a way to implement the entire system, and presented the results of applying the system on a dataset of 9 million posts. From this set, a total of 42 thousands life events have been found, with a projected accuracy of 88.90% and precision of 65%. This indicates that, in a normal day of 20 million posts published by Brazilian users, for instance, the system presents the ability to detect around 91 thousand posts a day, being about 60 thousand of them correct. Besides that, it is worth mentioning that the system is scalable since it has been implement with the MapReduce programming paradigm.
Future work can follow many di erent and complementary paths. Accuracy is important and could be improved by evaluating other types of classi ers and features, as well as increasing training data. The addition and evaluation of other types of life events could be important to better understand the way people behave on the SMNs. Furthermore, the adaptation to a real-time streaming platform such as the IBM InfoSphere Streams would allow the system react very quickly (near to real-time) once the users post life events.