We present an initial examination of the (alt)metric ageing factor to study posts in Twitter. Ageing factor was used to characterize a sample of tweets, which contained a variety of astronomical terms. It was found that ageing factor can detect topics that both cause people to retweet faster than baseline values, and topics that hold people's attention for longer than baseline values.
Our long-term goal is to study public communication about science: understanding
what the public thinks about scientific research is important for many reasons, from
developing science policy, through making the case for technological developments
and determining the impact of research, to simply being able to characterize the
vibrant public discourse which marks a healthy society, in which science plays as
important a part as politics, popular culture or the arts. The new media of the social web,
which are open to all, offer a fresh insight into public opinion to supplement the
surveys, and so forth, used in Public Understanding of Science (PUS) research [
Because of the scale of data available (Twitter claims there are 340M tweets per
day1), quantitative metrics are needed to aggregate the contributions of many
individuals. Informetrics research has developed and used quantitative measures to study
scholarly communication in traditional media for decades. Informetric methods have
been shown to transfer to communication on the web [
1.1
While reservations about the use of social media in formal work still exist [
However, scientists play only one part in the bigger picture of science
communication - science organizations, journalists (both science and public-interest media),
lobbyists and the general public also have important contributions to make. The current
ethos of science communication, as discussed by Nisbet & Scheufele [
Compared to the numerous works on politics and marketing, relatively few studies
exist about the public’s (as opposed to researchers’) scientific communication in
microblogs. Hubmann-Haidvogel et al. [
Weller et al. [
Our earlier work [
The metric we test in the experiments presented here, called aging factor, is based
on a well-established informetric measure. We follow the convention in which, by
analogy, retweets are treated as citations (tweetations) by researchers working on
altmetrics. For example, Eysenbach calculated a range of metrics for retweets of
announcements by the Journal of Medical Internet Research about the publication of
new papers [
Half-life is similar to the ageing factor metric used in this paper with a number of
important differences, which we consider make ageing factor more suitable for the
kind of data we are studying. Half-life takes what is called the diachronous view, i.e.
the metric observes a fixed set of documents, such as one year’s articles in a given
journal or one year’s tweets from a particular organization). It is therefore useful for
organizations which want to judge the impact of their own tweets and are monitoring
the occurrence of their Twitter name in retweets on an on-going basis. This is
necessary in order to harvest every tweetation of a particular tweet in the first n days. By
contrast, we want to take a snapshot of general discussion on scientific topics in a
given time period. This is what is termed the synchronous view and requires a metric
which does not rely on the originating tweet being present in the sample. For this we
adapted Avremescu’s ageing factor measure as presented in [
AF = i
k k + l
We examined two values of i, i=1 giving the one hour ageing factor (1hAF), and i=24 giving the 24 hour ageing factor (24hAF). A convenient feature of 1hAF is that it is simply the ratio of retweets in a sample that originated more than one hour after the original createdAt time over the total number of retweets in the sample; this makes it easy to understand. The 24hAF is the 24th root of the similar ratio for a 24 hour cutoff. In either case, AF values are produced in the range 0-1 with higher values indicating more retweets originating after the cut-off. 2
In the context of communication on Twitter, low values of AF would suggest a
flurry of activity typical of a trending topic, such as might happen following the
posting of tweets about an exciting topic. This might be a special event - in our
experiments we looked at retweeting about meteor showers. This fits with the findings in
[
Assumption 1: ageing factors for topics which concern special events will be lower than suitable baselines.
Assumption 2: ageing factors which are higher than suitable baselines are associated with topics in which interest is sustained over time.
The question of what constitutes a “suitable” baseline therefore arises. Unfortunately no benchmark corpus of Twitter presently exists (plans for a corpus to be held at the United States Library of Congress are believed to be underway at the time of writing). In this experiment, we have taken a pragmatic approach. We know there is a high level of noise in the samples (see table 1) - the majority of posts for selected terms are not about science. Therefore, we take the superset of tweets as a sample of general usage of that term on Twitter at that time, and use ageing factors for these sets as our baselines.
We chose the topic of astronomy for the experiment, because it has an enthusiastic following of amateur stargazers who own their own telescopes and are interested in observing events in the night sky for themselves. This led us to believe that it would be possible to harvest posts from Twitter, which discussed astronomical events and might provide evidence about the validity of assumption 1. We collected data on two nights when meteor showers were expected. Our data collection harvested tweets containing a broad range of astronomical terms in order to compare ageing factors for meteor showers with those for other astronomical topics. 2.1
As in our previous experiment [
For the AF calculations we needed to pick terms with reasonable levels of retweets.
Our previous experiment [
We considered terms with 99 or more retweets in the test data to be worth considering in the experiment. These were classified by reading the tweets and making a judgement about whether or not they had scientific content. For example, “when I was little I thought the sun and moon followed me around everywhere!” was judged not scientific, whereas “If the Sun exploded we wouldn’t know for 8m 20s. Light & gravity take that long to reach us. Then we’d vaporize” was judged scientific. Some retweets needed more research, for example, “RT @VirtualAstro: Make sure You watch a Night with the stars with your illustrious leader on Sunday night :)” was judged scientific after establishing the @VirtualAstro describes himself as “The Basil Fawlty of Astronomy, Science, Nature and more.”; he fits the profile of an amateur stargazer. As can be seen from Table 1, this classification exercise made clear the low level of scientific retweets.
Having identified UNESCO terms for which we could harvest reasonable levels of scientific retweets (albeit along with significant amounts of noise), we selected two baselines: the UNESCO thesaurus term Space, and a compound term we labelled Astro, which bundled together the UNESCO terms Earth, Moon, Sun, Stars, Universe and Space. In their raw condition, these both contain high levels of non-scientific usage of terms. Therefore, each can be considered as a sample of general use of those terms on Twitter at the sampled point in time. Subsets of the baseline, selected to filter out noise and represent scientific usage of the terms then had to be extracted.
Ideally, we would use a natural language processing method to identify scientific
use of the terms. However, these experiments have the objective of testing whether
AF is an appropriate metric for studying scientific communication. Therefore, we
took a simple approach to identifying scientific retweets using SQL queries to reduce
the noise in samples by adding narrower terms. We accept that this approach, based
on human interpretation of the language of the domain, has limitations for practical
implementation on the large scale and will need to be replaced in future work with
advanced NLP methods as advocated by [
The training data was a small enough sample to be analysed by hand. The retweets that had previously been identified as scientific were reviewed and topically related terms, which co-occurred with more than one UNESCO term, were identified. The terms were then sorted into topical queries. For example, terms related to space exploration equipment (e.g. Hubble) were in one set and terms which would be ambiguous (e.g. program) were all grouped together in another. One search (e.g. Space and Bodies+) added the names of the planets (plus Pluto) based on background knowledge.
Table 2 presents short form versions of the queries used, in which | represents OR, and standard parts of the query have been omitted for clarity. An actual search statement for the search Space AND sci for the training data set would read: SELECT statusid, createdat, retweetid, retweetcreatedat FROM 'twitter' WHERE (retweetid != "" AND batch = "1323902158593" AND (text like '%space%') AND (text like '%nasa%' OR text like '%science%' OR text like '%station%' OR text like '%soyuz%' OR text like '%satellite%' OR text like '%hubble%' OR text like '%interstellar% 'OR text like '%program% 'OR text like '%physics% 'OR text like '%plane% 'OR text like '%voyager%')) ORDER BY retweetid ASC;
Batch Space Space AND sci
Astro AND events Astro AND @
Table 3 shows both 1hAF and 24hAF for the searches. The 24hAF values for this dataset were all in the range 0.8-0.95 (zero values were assigned when all retweets collected were within the 24 hour window), whereas 1hAF ranged from 0.25-0.65. In general, 24hAF tracks 1hAF. The culture of Twitter places high value on currency, and 24h is a long time for many Twitter users. 24hAF appears to be an insensitive metric and we used 1hAf only for the remainder of the experiments.
1hAF for the training data searches are often based on small samples of retweets, such that just two or three retweets can make a big difference to the 1hAF. For example, the 1hAf value for Space and gear of 0.73 in the training data is based on 11 retweets, and consequently the 0.4 difference compared to 1hAF in the test data (0.33) is unlikely to be significant. Therefore, the following observations look only at test data, and use Batch, Space and Astro test values as baselines. In this first experiment, we made a naïve interpretation of the results, simply looking for values of 1hAF that appeared high or low, then trying to explain them in terms of the content of retweets.
For the Space set, two searches have 1hAF values that look different to the baselines: 1hAF for Space AND bodies is increased (0.45 compared to 0.34 in the Space baseline and 0.37 for the batch), 1hAF for Space AND bodies+ is decreased (0.26 compared to the same baselines). For the Astro set of queries, all queries except Astro NOT Meteor show differences when compared to the baseline Astro. Searches with increased 1hAF are: Astro AND @ (0.65 compared to 0.37 for the Astro baseline, but with only 23 retweets in the sample we should be cautious about its significance), and Astro AND tech (0.58). Astro AND events shows decreased 1hAF (0.26). The search Meteor was run to isolate tweets concerning the Geminid meteor shower. As can be clearly seen, it has a low value of 1hAF (0.22 compared to 0.37 for the batch).
Assumption 1 would associate low 1hAF with an event of some kind. The text of the retweets was examined and we found a high level of retweets of “NASA launch new rover to Mars” tweet in both Space AND bodies+ (which contains the term Mars) and Astro AND events (which contains the term launch). It seems the high level of retweeting of this post brings the 1hAF down for these two subsets. These initial results were sufficiently encouraging to make us want to study 1hAF in more detail with a larger dataset. 2.2
A larger sample of the public Twitter stream was then collected. This was filtered using the same 32 UNESCO astronomy terms and covers the full 24 hours of the 3rd of January 2012. This was the night on which the annual Quadrantid meteor shower was expected and our aim was to see if 1hAF values were low for this event, as per assumption 1, and whether the time of day matters (it must be dark to see meteors).
Initially we filtered out subsets using the searches we had developed using the training data for the first experiment (see Table 2). The day was divided into four periods 0:00-5:59 GMT (labelled 6), 6:00-11:59 GMT (12), 12:00-17:59 GMT (18) and 18:00-23:59 (24). Figure 1 shows the 1hAF values for these searches.
The first observation is that although the batch baseline 1hAF is steady in the range 0.32-0.37 through the day, the other two baselines each have one quarter of the day when they are high or low (24 for Space and 12 for Astro).
As in the previous experiment, the 1hAF values for Astro are more variable than those for Space. For example, both the Astro AND Events 1hAF and the Meteor 1hAF (both of which contain the term shooting star) are relatively low compared to the Astro baselines, especially for period 12 (2nd quarter of the day). It would be easy, but incorrect, to infer that 1hAF had identified a flurry of retweets about the meteor shower. Examination of the text of posts in the second quarter show that of 275 total retweets 18 contain the term quadrantid while 213 contain the term wish. There are various original tweets, but “@iQuoteFresh: #IfYouWereMine I’d stop wishing on 11:11’s, birthday candles, dandelions and shooting stars...Because I’d have my wish ...” is a typical example. Noise from non-scientific posts clearly remains an important issue, a fact underlined by examination of the high 1hAF values observed for four of the Space searches. These turned out to be due largely to retweets of variants of humorous posts on the lines of “Oh really? You need space? You might as well join NASA.”, for which the original tweets were more than one hour old.
The searches were based on the sample of training data collected about two weeks earlier around the Geminid meteor shower. It seems that even in this short time, the ways terms were being used had changed. We therefore took further steps to remove noise from our samples. Three astronomical events that took place around the 3rd of Jan. 2012 were used as background knowledge to add narrower terms to three of the original searches. The events were the Quadrantid meteor shower on the night of 3-4 Jan., the second of the twin Grail spacecraft moving into orbit around the Moon on the 2nd of Jan., and the proximity of the Moon and the planet Jupiter in the night sky on the 2nd of Jan. Searches excluding the event related terms were also conducted (see Table 4) as non-overlapping sets in order to assess the significance of results.
Space AND gear AND (grail|lunar|moon) Space AND gear AND NOT (grail|lunar|moon) Space AND bodies+ AND (jupiter AND moon) Space AND bodies+ AND NOT (jupiter AND moon) Astro AND (quadrantid|meteor shower)
Astro AND NOT (quadrantid|meteor shower)
Naïve interpretation of the results in Figure 2 (left) now seems promising. There is a low 1hAF for the Space and Grail search in the third quarter of the day (@18, 0.23), which contains retweets of posts about Grail tweeted by NASA and SETI in the first half of the (USA) working day. 1hAF for Space AND jupiter is generally high, and particularly in the third quarter (@18, 0.83), with retweets typically of links to pictures taken the previous night. Finally, the 1hAF values for Astro and quad are in the range 0.11-0.22, some of the lowest we saw, compared to between 0.32 and 0.39 for Astro AND NOT quad. These retweets are of messages from several sources reminding people to get up before dawn in order to see the meteors.
However, the differences in sample size between the searches about events and the
exclusion searches we are using for comparison could be extreme: the largest (Astro
NOT quad @24) contains 26327 retweets, the smallest (Space and jupiter @18)
which generates 1hAF of 0.83 contains just 6 retweets. We therefore require a method
of determining whether the 1hAF values we are seeing are significant or are merely
the effect of small samples. To gain insight into the significance of results we used a
funnel plot (see Figure 2 right). Funnel plots are employed in meta-analyses to detect
publication bias and other biases [
Based on the funnel plot, two of the Astro AND quad points still look interesting (@time 1hAF RT: @18 0.15 182, @24 0.22 330). Therefore, we can infer that retweeting activity around the Quadrantid meteor shower was significant in the hours of darkness for the UK and USA, where the largest concentrations of English speaking Twitter users in the northern hemisphere would be expected to be. Other data points which previously looked interesting, such as (@18 0.23 57) for Space and grail, on the funnel plot look like normal fluctuation within the expected variance of the population. This suggests that, at least on the 3rd of Jan 2012, the progress of Grail was not exciting the public to a significant degree.
A third data point also deserves investigation, that for Space NOT grail (@6 0.71 274). Examination showed that 216 of the retweets in this set contained the phrase “join NASA”, from retweets of the humorous posts we identified earlier. We infer that people had a long attention span for that particular joke. 3
Our objective in carrying out the two exploratory experiments described here was not to rate jokes, but to test ageing factor as an altmetric for analysing scientific communication in social media, and specifically to test whether it can give any insights for the smaller datasets typical of scientific content. For ageing factor calculated with a window of one hour (1hAF), several of our naïve observations fitted both assumption 1 (ageing factors for topics which concern special or exciting events will be lower than suitable baselines) and assumption 2 (ageing factors which are higher than suitable baselines are associated with topics in which interest is sustained over time). However, when a funnel plot was used to identify data points which lay outside the area of expected variance, only three data points appear significant: two with low 1hAF for retweets about the Quadrantid meteor shower, and one with high 1hAF for a (non-scientific) humorous post. We conclude that further investigation of 1hAF would be worthwhile, but that interpretation of the metric without reference to sample size must be avoided.
In these exploratory experiments, topics were identified using SQL searches. It
would be intriguing to compare topics with low and high AF to the statistical
linguistics approach used by Hu et al. [
24hAF gave similar values for all the searches in experiment 1. Therefore, we did
not use 24hAF in experiment 2. As in [
The overall aim of this work is to contribute to the nascent development of methods and metrics that will support analysis of public online scientific communications. It is clear that the big issue in achieving this is the level of noise in samples coupled with low actual levels of scientific communication in social media. These combine to make it difficult to get big enough samples to get statistically significant results. As an additional problem, the usage of terms on Twitter clearly varies considerably even over a few weeks: our experiments used data collected only a few weeks apart, but the searches developed in experiment 1 proved useless in experiment 2. This may make it difficult to devise standard filters for on-going monitoring of scientific communication. Noise was addressed in this study by writing SQL queries to produce disambiguated subsets. However, in the future we will need to identify, and possibly develop, more subtle, NLP-based techniques for classifying tweets on science related topics. These techniques will need to adjust dynamically to pick up new topics as they arise.
As for future work, although our interest in ageing factor was stimulated by the small sample sizes we found for typical scientific topics, we are investigating the application of the technique to larger datasets and longer sequences of events. Furthermore, we have not explored the differences between types of participants. For example, is there a difference between ageing factors observed for private individuals’ tweets vs professional scientists’ vs organizations’? Techniques for distinguishing these groups will be particularly important in achieving our overall goal of analysing public opinion about science. 4
A-S Dadzie is currently funded by the UK MRC project Time to Change (129941). Thanks to Mike Thelwall and the Statistical Cybermetrics Research Group for encouragement with pursuing the informetrics approach, and to Gary Simpson at Aston University for advice on the statistical issues. 5