- With the growing popularity of Twitter, numerous issues surround the usefulness of the technology for intelligence, defense, and security. For security, Twitter provides a real-time opportunity to determine unrest and discontent. For defense, twitter can be a source of opensource intelligence (INT) information related to areas of contested environments. However, the semantic content, location of tweets, and richness of the information requires big data analysis for understanding the use of the information for intelligence. In this paper, we describe some results in using twitter data to determine events, the semantic implications of the results from the data, as well as discuss pragmatic uses of twitter data for multi-INT data fusion. The results collected during the period of Egypt Arab spring conclude that (1) many tweets are clutter or noise in analysis, (2) location information does not always convey the accuracy of the information, and (3) the aggregate processing of the twitter data results in real-time trends of possible events that warrant more conventional information gathering.
INTRODUCTION
Over the last decade, there has been a surge of the use of constrained messages sizes in 140 characters or less, known as “tweets.” The popularity of tweets has three emerging issues: (1) big data as the number of users grows, (2) semantic extraction of meaningful content from cryptic phrases and nonstandard terminology, and (3) the large amount of semantic clutter that reduces the signal-to-noise ratio of identifying salient content (e.g., key words of phrases).
While the use of open source information becomes popular such as Facebook, imagery, and text; it is well established that tweets are being used by anyone anywhere from distributed mobile platforms. The presentation of different semantic formats and the number of users require pragmatic approaches to searching and deriving meaningful content from tweets. Meaningful content is further exacerbated as the source of tweets does not always correspond to the location of the user; however, timing and general trends over many users can determine the status of an emerging event.
Twitter data, while popular, suffers from various content issues that have to be solved with advanced and tailored methods. Examples of problems include users with hidden meanings, masked source of origin, possible deceit and deception, as well as non-descriptive and non-important discussions of social issues (e.g., where to go for dinner). However, tweets do provide a forum where users can express their social and political views, news reports of immediate actions that are not available to the regular media, and links of semantic content such as to video collected and posted to the web from cell phones.
It is the interplay between the availability and enormity of tweets to that of extraction of meaningful content that is derived from users close to the action. Tweets provide reports that are not available from other intelligence sources of information.
An approach taken by most search engines over twitter data
is to organize documents and their terms in a Vector Space
Model (VSM) [
With successful implementation of VSM by the search
engines, researchers have attempted to apply VSMs to other
areas of natural language processing (NLP). For a long time it
had been considered that to understand the meaning of words
it is enough to consider statistical word usage, the so called
statistical sematic hypothesis [
Twitter has a lot of informal and abbreviated text. For
example, current tools, while practical on news articles and
similar types of well written documents, perform quite poorly
for Part-of-Speech (POS) tagging and Named Entity
Recognition (NER) when applied to tweets. The accuracy of
tools falls from 97% accuracy for news articles to about 80%
for tweets [
To perform further analysis such as sentiment analysis, it
would make sense to perform all of the steps in the papers
cited i.e. set of tweets needs to be found, the features from
tweets are extracted, preprocessed, tagged, and statistical
analysis is performed. Sentiment analysis can be used in order
to identify anger, tension, and other emotions that may be tied
to significant offline events [
The rest of the paper is as follows. In Section II, we describe the use case of Twitter data from the Egyptian uprising. In Section III, we discuss how the data is structured. Section IV and VI describe analytics and visualization, respectively. Finally, Section VII describes conclusions.
II.
TWEETS FROM THE ARAB SPRING SCENARIO
The Egyptian uprising of 2011 is an example of an
important historical event which has been captured via social
media sites such as Facebook and Twitter. Some argue that
without social media, like Twitter, the uprising would have not
achieved the same level of success [
In regards to Egypt, January 25th 2011 had become known as the “day of rage” with protests in Cairo. Social media and Internet played such a key role that the Egyptian government had begun limiting Internet access on January 27th [23]. Egypt related topics continued circulation until February 11 when President Mubarak resigned. During the course of events, it was noted that information was coming from tweets, but the intelligence sources were not mobilized to use the technology and even if available, to what extent that content could be gleamed from the experience of multiple users presenting tweets.
In this paper we attempt to analyze 738,717 tweets from
that time period in 2011. The Egypt Twitter data has been
grouped into 10 classes by Army Research Lab (ARL) [
In relation to the above methods, our novel approach is to apply a VSM model to the groupings made by the Tri-HITS model in order to extract top one hundred features associated with each grouping. The top features that are extracted can be used for evaluating the quality of grouping and can be used by the analyst when searching for similar events. The algorithm is fast and straight forward to implement and does not require human in the loop involvement. To the best of our knowledge we are not aware of papers applying a VSM model on rankings generated with Tri-HTTS model for the Egypt data.
• • • •
with some class label. Figure 1 shows the basic tweet to class label structure. There are a total of 10 classes c0, c1, …, c9. Tokenize Approach A. Feature = anything that is no more than 50 characters in length and that contains only digits and ascii_letters i.e. any other characters are removed.
http://www.google.com is converted to
httpwwwgooglecom which becomes our feature.
Tokenize Approach B. Feature = Twitter hashtags (Twitter topic that begins with “#”) and Twitter at -mentions (atmentions begin with “@”). There were a total of 106,322 features using this approach.
Figure 2 shows a vision for all of the main steps listed in
the referenced papers. The first step is to find information of
interest whether it is related to some event, an organization, a
product, etc. Open IE systems can help retrieve the data we are
interested in, if we have a broad enough set of terms that cover
the topic of interest. The event of interest for us is the Egypt
uprising data supplied by the ARL [
The second step is to tokenize, i.e. extract features. Most often terms of interest are separated by white space, but researchers need to consider how they want to treat URLs, punctuation, and multiword features such as “ daylight savings”. We had two approaches. Our first approach had been to simply use white space as delimiter, join on punctuation, and disregard features over 50 characters in length (this gets rid of most websites). Our second approach was to focus on specific topics and people on Twitter:
The third step is to normalize. Normalizing reduces similar
features. For instance it is common to use a stemmer in order
to turn words like fixing, fixed, fixer fix. The Metaphone
algorithm mentioned in [
can be tagged as a verb roll/VB (to rotate around an axis) or a noun roll/NN (a small loaf of bread). Annotation is the inverse of normalization so it tends to improve precision and decrease recall. Annotation can be performed using well established lexicons that contain basic rules of grammar for such operations; examples include WSJ and Brown corpora as well as WordNet and Moby lexicons. In this paper we have not attempted any annotation.
The fifth step is to use the features in a frequency matrix. It begins by recording how many times each feature appears in each class. Table 1 is the result of an SQL query which orders features by total times used over all classes using tokenizing approach A.
Instead of using classical frequency, most papers use the
term frequency multiplied by inverse document frequency
(TFIDF) [
The sixth step is to rank features. The counts can then be used to calculate probabilities and to rank features. Classical frequency only considers how probable a term is to occur within a class. For example consider that the feature "to" was seen 1000 times for class 1 and 2000 times for class 2. Classic accuracy is going to be: "to" appears 1000/3000 = 33.33% of time in class 1, and 2000/3000 = 66.66% of time for class 2.
Classic accuracy could be taken to mean that the feature "to" is associated with class 2 with 66.66% accuracy. But consider that class 1 had a total of 5000 records and class 2 had a total of 10000 records. This additional information tells us that "to" had appeared in every record of class 1 and every record of class 2. Hence "to" is not a relevant feature. Besides using the TF-IDF measure we rank features by calculating how each feature contributes to each class using the following percent contribution formula:
Feature i contribution to class j
Total times feature i appeared for class j ( Total number of tweets in class j ) =
Total times feature appeared for class N ( Total number of tweets in class N ) (where n is the number of classes involved) Both measures consider the number of records in all classes which is better than using classical frequency. Using the percent contribution formula on the example from above, we see that class 1 and class 2 evenly contribute to this ratio, i.e.: denominator = 1000/5000 + 2000/10000 = 2000/5000 %contribution by "to" feature from class 1 =
(1000/5000)/(denominator) = 50% %contribution by "to" feature from class 2 =
(2000/10000)/(denominator) = 50% TF-IDF will rank these two classes as equals as well: (1000/5000)*(log(2/2)) = 0 (2000/10000)*(log(2/2)) = 0
Table 1 is used to generate measures for each feature using classical accuracy, percent contribution, and TF-IDF. Table 2 shows the calculation for the three measures for the feature “cairo”. The examples had been shown using data from tokenization approach A, but the same approach and tables are produced when performing tokenization approach B.
IV.
RANKING FEATURES
Given a score for each feature, we are able to go through all of the original tweets and classify the tweet using the feature within the tweet that has the highest score. We keep track of how many times a feature is used. Ordering on times that the feature had been used to predict a class gives us a ranking of all the features. Table 3 shows top features used by the three measures.
Classical Accuracy
Percent Contribution
From the table, we see that TF-IDF has identified many stopwords as important because those features are a big percentage of the tweets. We see that percentage contribution used the feature “i" less than classical accuracy (score for feature “i” is slightly less by perce nt contribution) and the feature “square” more than classical accuracy (score for feature “square” is ranked slightly higher by percent contribution. Percent contribution should be more accurate because it takes into account number of records within each class when calculating its scores.
When looking at top 100 features associated with each class there is a clear difference between classes as we go from class c0 to class c9. Features seem to be going from clear topics during the Egypt revolution to features corresponding to personal tweets. Tables 5 and 6 show percent contribution measure top features for classes c0, c1, c8 and c9 for tokenization approach A and B (respectively). c9_Feature
#fb @nevinezaki @etharkamal @mamoudinijad @khalawa69 @amrwaked @saraaayman @theonly warman #grammys @terradaki #feb11 #weather … @travellerw #alarabiya @huffington … @mosaa post berizing #ghonim @shmpongo … @litfreak #iheard @guardian @washington … @marionnette90 #mbmemories post @addtoany @adel_salib … @samiyusuf #prayforegypt Table 6: Top Features Percent Contribution
(tokenization Approach B)
Having found ranked features for each class, the analyst can verify if the features captured make sense and use those features in order to filter and collect more data from Twitter.
V.
Twitter is a large noisy data source. There are 400 million tweets a day with most of the messages not relevant to the analyst. Simply grabbing a lot of data and trying to fit a model to the messages is not relevant. An analyst needs to first understand how to query Twitter just as an ordinary human being knows how to query the World Wide Web. Querying Twitter is equivalent to understanding the types of features (query terms) to use. We have illustrated a means of ranking features and then using those features for classifying a 10 class problem. Calculating accuracy is simple in the sense that we can just count how many times we have accurately identified a record vs. number of records attempted. The features are ranked by the overall accuracy for the 10 classes achieving 0.695% for percent contribution error, but the final classifier had hundreds of thousands of features that appeared only once. For this reason we choose to look at only the top 1000 features, with accuracies shown in Table 7 and 8. … … … … …
The actual accuracy is whether the features that were extracted make sense and can these features be used for finding relevant tweets that are of interest to the analyst. We have seen that the features identified distinguish classes. We saw for instance, that class c9 carries features that are associated with more personal messages and class c0 carries features that are closely associated with the Egypt uprising news topics (all other classes are somewhere in between). Going through the messages by hand in the classes we see that c0 may contain tweets that should not be associated with class c0, such as: a) “OMFGCould you believe it? My wife just purchased an Iphone for 42US$!!! http://moourl.com/5td4g Cairo #famouslies White Stripes DiPietro” b) “WOW OMG JUST WOW -- search Twitter annnd Google side by side - http://bit.ly/hBxUBC #### #ifyouonlyknew Cairo Charles Barkley”
Likewise other classes probably have tweets that have been misclassified. We use the top 100 features to reclassify the ten classes, but it is up to the analyst to determine if those features are enough (tweets that do not have the features are thrown away).
Among other things that might help in extracting useful tweets and increasing accuracy include analyzing how many times a tweet has been reposted (retweeted), how many people replied to it, and considering the geospatial component so that we focus only on tweets from a certain area. A way to filter irrelevant tweets would be to get rid of tweets that consist of features that are mentioned by less than N number of people (bottom up approach). This can be established through regression to determine a threshold. Another way is to try and understand most important features extracted and include the features from all of the tweets that mention those features (a top down approach). Filtering would get rid of spam and selfcentered messages that do not give any insight in understanding the event of interest. In the next section we consider allowing the user to focus on features coming from a specific geo-location.
VI.
Tweets have a geospatial component to them so that they may be shown on a map. We have used JavaScript and Google Map API in order to visually present Twitter data for the Egypt dataset. The intention was to allow a user to click and drill down into tweets corresponding to some geographic location. In this way a user could perform analysis on how the features in one geographic location are different from tweets associated with a different geographic locations. The thought is that there will be more conversations in the local area where the event is actually taking place then in the rest of the world.
Based on the latitudes and longitudes in the Egypt dataset, we divide the world into a ten by ten grid. This grid serves effectively as a histogram and displays rings that correspond to number of tweets coming from a particular area (the center of the ring is the center for the particular cell on grid and so
Classic Accuracy Percent Contribution It should be kept in mind that this is a 10 class problem so random guessing would produce around 10% accuracy. TFIDF actually exhibits worse errors rates because there are few classes so that many features appear in all classes and thus get ranked 0. some circles may appear on water, Figure 3). The Egyptian government limited Internet access so we actually see that most tweets that do have a geospatial location are from around the world (in particular from South Africa).
For the Egypt dataset, unfortunately only about 1% of data
had a geo location associated with it, but this is typical as less
than three percent of all tweets have geo-location information
[
The benefit to using geo-location is that the user can focus on main features corresponding to the area of interest vs. discussions about the topic in neighboring regions. For example a victory in a sporting event will be discussed differently in the hometown vs. the rest of the country. The features coming from hometown will probably be positive about the hometown team. These features can then be used for finding towns that have similar feelings about the sport’s team. This is an iterative process whereby more and more features can be discovered but at the root of those features will be the features associated with the hometown. The features can be listed in a hierarchical fashion and can be a means of organizing based on features and locations. Unfortunately only 1% of features have geo-locations, again results are still to follow for ranking using this approach. VII. DISCUSSION AND CONCLUSIONS
In this paper, we explored the use of Twitter as a source of intelligence for determining the status of a pending, emerging, or on-going event. We believe Twitter can be queried for relevant data similar to queries performed on the World Wide Web. In order to make queries, an analyst should have a list of key features (query terms) related to some event of interest. In an attempt to extract those key features, we investigated a 10 pre-labeled class dataset by the ARL covering the Egypt uprising. The features from the labeled classes are used in a frequency matrix so that ranked features can be used to identify other relevant tweets (like Vector Space Model [VSM]). Once top features are identified, an analyst can use an open IE system to make queries for relevant tweets just as a person is searching for web documents on Google. These extra tweets are used to get at an even more robust feature set. Each loop generates a list of features that an analyst has to go through and approve. In this way, we foresee an iterative process between the analyst (feature approver), VSM (feature rankings), and an Open IE system (Twitter queries) in order to create catalogs of useful features for semantic analysis of activities. Catalogs of useful features can then be used for filtering and identifying events and activities of interest.
VIII. ACKNOLEDGEMENTS
We thank Sue E. Kase and Liz Bowman at the Army Research Lab and Mike Hinman at the Air Force Research Lab for their help in getting access to the Egypt data.