Does a journal's actively curated Twitter account cause a rise in Journal Impact Factor (JIF)? The present paper analyses this causal assumption by looking at the JIF trajectories of various journals that set up a Twitter account in 2015. Having identified one suitable journal (namely, International Sociology), a counterfactual unit was generated with the the synthetic control method, a control unit that resembles the same journal but one that would not have registered a Twitter account in 2015. That control unit is synthetised via dozens of other journals that exhibit similar characteristics like the treated unit. The paper finds that the actual International Sociology enjoyed higher JIFs than its counterfactual control unit after 2015. The scale of this efect, however, is negligible and the finding is statistically insignificant. The main result of this paper, therefore, is that the null hypothesis claiming no-efect-of-Twitter-upon-JIF cannot be rejected.
In disseminating research findings, scientific journals compete for attention and innovativeness
that is (imperfectly) measured with citation counts. Greater visibility can be achieved through
community engagement, such as when journals curate their own blogs, newsletters, podcasts, or
social media channels [
One dificulty with validating this assumption lies in the fast-growing landscape of scholarly
publications, the increasing number of references per paper, and therefore an accompanying
‘inflation’ of the JIF [
An alternative to a randomized controlled experiment would be to use quasi-experimental
designs. For instance, one could use a statistical matching approach by comparing journals that
bear similar values of selected covariates – such as by matching journals that have set up a
Twitter account in a certain year with journals that have no Twitter accounts based on similar
citation counts. However, scientific journals are so heterogeneous in terms of citation counts,
discipline, topical niche, number of articles, number of authors, editorial prestige, use of special
issues etc. that the approach of matching risks too much of an extrapolation [
Given that the untreated units are often unsuitable control groups for a journal, there is another quasi-experimental approach that would not be apt for the present question, namely the diference-in-diferences approach [ 8, ch. 5.1-5.2]. It would observe the trajectory of tweeting journals’ JIFs with that of similar, but non-tweeting journals’ JIF across years, assuming that the two sets of units may not necessarily start at the same level of JIF, but that all other factors afecting each journal’s JIF develop in similar ways. This ‘parallel trend’-assumption, however, seems unrealistic as citation counts fluctuate too greatly – for instance, due to editorial board changes, due to the setting up of new journals in a given field, due to the size of the subdiscipline it is dedicated to, due to the salience of the issues the journal publishes about in society at large, due to open access and open peer review policies, and so on. Journals exhibit completely diferent trajectories with regards to their citation life cycle. The diference-in-diferences design would nevertheless give all the untreated units an equal weight.
The approach of a synthetic control avoids some of the problems [9, ch. 10]. Given a unit of observation (such as a journal that established a Twitter account in 2015), the synthetic control method does not require a comparison unit (such as a journal without a Twitter account) that exhibits otherwise highly similar covariates, and it does not give the same weight across every single comparison unit. Instead, it combines multiple units of comparison and thereby generates an artificial, synthetic control unit which “is selected as the weighted average of all potential comparison units that best resembles the characteristics of the case of interest” [10, p. 496]. Given a journal with a Twitter account, the absence of an otherwise highly similar (but non-tweeting) journal does not pose a hindrance for analysing the causal hypothesis anymore. The synthetic control method takes X1, or specific parameters of the treated unit (the journal with the Twitter account since 2015) before the intervention (before the setting up of a Twitter account), and compares them with X0, or with the same parameters of synthesised comparison units given specific weights of the variables, W. By taking the formula of X1 - X0W, the approach selects the synthetic control that carries the minimum diference, or W*. The pre-intervention synthetic control should thus closely reproduce the values of the parameters of the treated unit. As a result, the diference in the post-intervention outcomes between the treated unit and the synthetic control, or Y1 - Y0W*, allows one to quantify the causal efect of a given intervention (the setting up of a Twitter account) on the outcome of interest (the JIF). Another problem that is tackled by the synthetic control method is that given an extended period in a longitudinal dataset, the approach controls for unmeasured factors that may influence the outcome variable [10, p. 498]; the heterogeneity of the individual units does not pose a problem anymore, as it assumes that the treated unit and the synthetic control behave similarly and are therefore alike for a stretched period of pre-intervention time.
Meta-scientific studies have already used synthetic controls so as to model efects of Nobel
prize bestowals on citations [
The present paper adds to this strand of causality-assessing meta-scientific research by tackling the question whether the creation of an (actively curated) Twitter account by a scholarly journal leads to a higher JIF in the years thereafter. The following section outlines the synthetic control method. The results section finds an almost negligible positive efect, but one that lacks statistical significance. The final section discusses possible reasons for the null result (such as the limited sample comprising only three disciplines and just one treatment journal) and points to possible future research venues.
The sample from which both the treated and the control units are to be taken consist of all journals that are indexed in Web of Science’s 2021 Social Science Citation Index (SSCI) in the three categories Sociology, Political Science and International Relations. Their Twitter accounts were collected manually. The Twitter API was then used to find out the respective year of the Twitter accounts’ establishment. Their respective two-year JIFs ranging from 2005 to 2020 were likewise collected for all these journals.
Possible journals to be chosen as the treated units were those that fulfilled three criteria. First, they had to be located in the middle quartiles (Q2 and Q3) of the respective ranking (ordered by JIF) in the 2021 version of the SSCI (the most recent one at the time of data collection). The journals in the top quartile (Q1) and in the bottom one (Q4) were not considered due to the risk that they could exhibit extreme values regarding the predictors (citation counts, number of Tweets about their articles and the JIF) so that no synthetic equivalent could be generated.
Second, the journals had to have established their Twitter accounts in the year 2015. That year was chosen because it ensures a long enough pre-intervention period spanning 15 years, thereby controlling for possible unobserved confounders, and a long enough post-intervention period so as to estimate an efect of the Twitter account registration.
The third criterion was that the Twitter account pertaining to the journal should be an actively curated one, indicated by an average of at least one Tweet a week since the inception of the Twitter account. As an approximation, therefore, only journals that had issued at least 364 Tweets so far (52 weeks multiplied by 7 years) would be used as treated units.
The donor pool, or the “reservoir of potential comparison units” [10, p. 497] out of which the control group would be synthetised, consisted of all the other journals indexed in those SSCI categories, given that they had not created any Twitter accounts yet, or only did so no earlier than 2021.
Using the International Standard Serial Number (ISSN) of the journals, the scholarly metadata service CrossRef’s API was accessed to obtain the Digital Object Identifiers (DOIs) of all papers published in the sample journals between 2005 and 2021. For each DOI, the citation count was fetched using the OpenCitations API, and the Tweeter count via the Altmetric API.
A total of 20 journals matched the profile for the first two treatment criteria – that is, they were located in the middle quartiles of the JIF-based rankings and they created their Twitter accounts in 2015. However, 15 of these journals had missing values across 2005 to 2021 (such as due to their later indexing in Web of Science). To regain balanced data, or a longitudinal dataset with all values available for each unit in every year [10, p. 497], the 15 journals were removed. Only the following five journals were left: Acta Sociologica, American Politics Research, British Journal of Sociology, International Sociology, and Political Studies. However, the Twitter accounts of both Acta Sociologica and American Politics Research did not reach the threshold of 364 Tweets; the former one exhibited 318 Tweets, the latter just 21 Tweets in seven years. Thus, only three journals – British Journal of Sociology, International Sociology, and Political Studies – fulfilled all three criteria to count as potential treatment units.
The donor pool – journals in the same three SSCI categories that did not have Twitter accounts as of 2020 – originally harboured 93 journals, but only 43 of them had complete (i.e., balanced) data. The analysis thus remains with three journals in the possible treatment group and with 43 journals in the donor pool.
To generate the synthetic control group, three predictors were used that were known to afect the JIF: first, the average count of Tweeters per article (as of December 2021); second, the average count of citations for each article (as of December 2021); and finally, the JIF as calculated by Web of Science (note that the predictors “may include pre-intervention values of the outcome variable”[14, p. 497]). With the R library RSynth, the use of the predictors was optimized from 2005 until the year of treatment in 2015, and a time plot was generated from 2005 to 2020.
Only one of the three potential treatments journals received an apt synthetic control, namely International Sociology, a journal located in Q3 that established its Twitter account in June 2015, having issued 3033 Tweets ever since. The two other journals’ synthetic controls had pre-intervention values that deviated too much from the actual journal’s predictors.
Table 1 shows the mean values of the predictors prior to 2015 for the journal and its synthetic equivalent, with the final column suggesting that the synthetic control provides a much better comparative unit than the mean values of the whole sample.
In line with the literature that found article-level citation counts to be better predictors of a
journal’s JIF than Twitter activities [
A low pre-intervention mean squared prediction errors (MSPE) of 0.026 (in the unit of the target variable – that is, in terms of the JIF) likewise indicates that the synthetic control matches the actual journal closely in the pre-intervention period.
The control of International Sociology was synthesised with the help of the Korean Journal of Defense Analysis (31%), Space Policy (9.1%), the American Journal of Sociology (8.9%), Politická Ekonomie (8.2%), Society (6.7%), Political Science (5.2%), Parliamentary Afairs (2%) and 35 other journals whose respective weight remained below 2%.
The following section will present the synthetic control efect of the treatment (the setting up of the journal’s Twitter account in 2015) upon the JIF, and infer the significance of the finding.
The adoption of a Twitter account exhibits a minuscule positive efect on the increase of the JIF in the five years thereafter, albeit there are two caveats to this finding: first, the scale of this efect is negligible, and second, the finding is statistically insignificant. In other words, to obtain an answer to the question whether an active Twitter account generally causes an increase in a journal’s JIF in the five years thereafter, a diferent dataset would be needed to find a generalisable response. While the synthetic control method may be suitable in answering the question at hand, it does not seem to be so when one uses a sample of just the middle-ranking journals in Sociology, Political Science and International Relations whose Twitter accounts were established in 2015.
One may nevertheless have a look at the results based on the data at hand. The path plot in Figure 1 shows that International Sociology enjoyed higher JIFs than the control group after 2015 (in the post-intervention period) than the counterfactual journal (the synthetic control) that did not establish a Twitter account, with the exception of the first post-treatment year (in 2016) where the journal underperformed relatively to its synthetic control.
Figure 1 also shows that there is a certain pre-intervention divergence between the two units. While the mean values of the predictors exhibited a close fit when seen across the whole pre-intervention period (see Table 1 above), the year-by-year JIFs deviate from each other with a mean annual gap of 0.07 (median: 0.08) prior to the intervention in 2015. A merely graphical analysis would thus indicate that the fit between the treated unit and the synthetic control was not as close as previously suggested by Table 1, unless one resorts to mean values summarised across the whole pre-intervention period.
Assuming nevertheless that the pre-intervention synthetic control reconstructs the actual International Sociology closely, one may hold that the efect of the establishment of an actively curated Twitter account in 2015 upon the JIF in the five years thereafter is rather minuscule. After the treatment in 2015, the average gap in JIF in a year was reduced to a mere 0.01 (median: 0.01); see Figure 2 for a gap plot showing how International Sociology’s JIF difered from its synthetic control year by year. A clear causal efect would have shown how “the two lines diverge substantially” [10, p. 503], but this is not the case here.
What is the statistical significance of this minuscule efect? The usual inferential techniques
(such as the t-test etc.) are suitable for large samples, but not necessarily for comparative cases
with a low number of units as in the present case where only one unit received the treatment.
Inference in the synthetic control method is thus exercised via so-called placebo tests [
The placebo tests in Figure 3, however, does not show any unusual treatment efect. On the contrary, the treated unit’s line is positioned at the very center of the distribution of placebos. A large number of other lines exhibit more pronounced trajectories – even though “if another [journal] is chosen at random, identified as treated, and the synthetic control is applied, no efect should be visible for these” [ 19, p. 296]. Figure 3 seems to suggest the opposite, as if not-having-established-a-Twitter-account leads to more volatile JIF changes than the treatment.
Note, however, that “[t]he placebo efects may be quite large if those units were not matched well in the pretreatment period” [18, p. 837]. It is also possible that the post-treatment period was chosen to be too brief for a useful analysis. As a further means of inference, therefore, one may go beyond a graphical analysis and look at the distribution of the ratios of the MSPE in the post- and the pre-intervention period. If the intervention of the Twitter account registration was significant with regards to its efect upon the journal’s JIF, then International Sociology’s ratio would be much greater than the ratios of the placebos.
Figure 4 lists the post/pre-intervention MSPE ratios for the treated journal and the placebos. Again, International Sociology does not stand out at all.
The p-value is 0.91, indicating that even a random assignment of the intervention would render it highly probable that one obtained the same post/pre-intervention MSPE ratio as International Sociology’s [10, p. 500]. In other words, the null hypothesis, or the assumption that the establishment of an actively curated Twitter account had absolutely no efect upon the JIF at all, still holds.
The efect of a scholarly journal setting up an active Twitter account amounts to a minuscule increase in its JIF (of an average of just 0.01 a year). However, this efect is statistically insignificant; in other words, despite having found a negligible efect in a one-unit sample pertaining to the journal International Sociology, the general causal relation between a Twitter account and the JIF cannot be determined with suficient clarity for a broader population of scholarly journals with the method used in this paper – that is, not when one uses the synthetic control method with regards to middle-ranked Sociology (and related) journals, not when one places the intervention in 2015, and not when one generates the synthetic control group via journals of the same categories with the three predictors of an average citation count per publication, an average Tweeter count per publication, and the JIF. The main result of this paper, therefore, is that the null hypothesis claiming no-efect-of-Twitter-upon-JIF cannot be rejected.
Perhaps one reason for the failure to identify a statistically significant causal efect is that the (2-year) JIF itself is immensely volatile. Reasons behind citations are complex and manifold, and chance encounters of serendipity push the scientific system to such unpredictable bibliometric links that a statistical analysis with a limited number of clear-cut quantitative parameters cannot capture the dynamics suficiently. In addition, as Twitter is a relatively new tool for scholarly journals, perhaps the pre- and post-intervention periods for generating synthetic controls and for analyzing their efects were too brief for a meaningful estimation. Finally, drawing from Q2 and Q3 journals from just three social scientific disciplines may have been too narrow of a sample, indicated by the fact that only one single suitable unit with an apt control group was identified; a larger sample should be considered, taking into account disciplinespecific diferences with regards to a community’s use of Twitter. Even if the question whether a journal’s community outreach causally afects its scientific impact may be worthwhile to investigate, a limited data sample hinders the generalizability of the result found.
Further research could thus use a diferent sample of journals, one that could cover all kinds of
scientific disciplines. The dataset of Twitter accounts from almost 3500 Web of Science-indexed
journals may ofer a good starting point [