Uncovering impacts: a case study in using altmetrics tools

Abstract

Growing scholarly use of Web tools present an opportunity to track alternative impacts along heretofore invisible paths like reading, bookmarking, and discussing. We present two tools, CitedIn and total-impact, that gather and report these and other \altmetrics" After discussing the tools features, we use a set of 214 articles from a national research center as a demonstration case study. We nd that both tools present a meaningful number and variety of altmetrics in a form that could be used for immediate evaluation, and call for more research into the properties and validity of altmetrics.

Keywords

• altmetrics
• scholarly communication
• impact
• tools

-

The future of scholarly communication is one in which a large part of scholarly communication is conducted online [ 3 ]. A key part of the scholarly communication lifecycle is trying to understand the impact of work. The process of understanding impact helps scientists, science administrators and others both nd, evaluate, and access scholarly products. Traditionally, this impact assessment has been done primarily through the tracking of formal citations. This is possible because citations counts, for all their occasional ambiguity [ 2 ], do re ect use of scholarly products. However, this re ection is of a restricted spectrum; scholarly products are often used by scholars, and others, in ways that do not perturb the citation record [ 5 ]. Furthermore, traditional citation does not re ect the rapid nature of communications a orded by the Web. Thus, we need new approaches for measuring impact in this changed world.

Indeed, because of the Web scholarly communication, formerly \underground" uses like reading, bookmarking, sharing, discussing, and rating are beginning to leave online traces. The are becoming visible on Web pages [ 8, 13 ], on blogs [ 6 ], in downloads [ 1, 4 ], on social media like Twitter [ 9 ], and in social reference managers like CiteULike, Mendeley, and Zotero [ 7 ]. These alternatives to traditional citation analysis have been labeled altmetrics [ 11 ]. Altmetrics o er potential for gathering information on more diverse types of impact, from more diverse scholarly products, including blog posts, slides, datasets, or even tweets. They also have the important bene t of speed; altmetrics typically accumulate in days or weeks rather than the years citations require. This is particular useful in as the research process increases pace where users of scienti c content need to understand the impact of it rapidly. To begin to make practical use of altmetrics for measuring impact requires both a greater understanding of the properties and validity of these new metrics, and practical tools for obtaining them [ 10 ]. Others have begun the former [ 12 ]; here we will pursue the latter, presenting two new tools for gathering and presenting altmetrics. 2

Tools for Altmetrics: CitedIn and total-impact

Case study: altmetrics for a national research center Data repositories, including locating datasets associated with a given publication Social bookmarking and reference management tools Blogs and social media Traditional citation

Google Books mentions Other PLoS ALM

PubMed subsets, Citations from Wikipedia (pmid) N/A ABS, Ares, Alzgene, Biogrid, BredeWiki, Ctdatabase, cancerCell, ChdWiki, Cosmic, Ctd, Cutdb, Dejavu, HIFTFBS, HNF4, HaemB, Jaspar, Kegg, Mgi, Mint, Mpidb, N Regulome Resource, Oreganno, MIDNCI, BIDReactome, BDB, PleiadesGenes, Gregransbase, Balmer Retinoic, Uniprot, Wikipathways, Wormbase, YTPdb, Z n CiteULike, Mendeley

Connotea, Google Blogs, Nature Blogs total-impact Dryad (downloads of most popular le, package views, total downloads and le views) CiteULike, Delicious, Mendeley (groups, readers) Facebook (clicks, ments, likes, shares) Citation in PubMed Central Citations from Wikipedia

comConnotea, citations (CrossRef, PubMed Central, Scopus), blog mentions(Nature Blogs, ResearchBlogging, Bloglines, Postgenomic), downloads, PubMed activity is smaller. Both tools showed that altmetric activity as measured by number of \altmetric events" (bookmarks, downloads, etc.) is relatively widespread across articles: CitedIn found at least one event on 95% of its articles, and total-impact on 85%. There were a mean of 28 and median of 16 events per CitedIn article, with a maximum of 678. Total-impact had a per-article mean of 92 events and a median of 19; the higher mean is due to Dryad dataset downloads, which accumulate more easily than other metrics, reaching a maximum of 2769 on one article. We visualized the activity across articles using heatmaps, shown in Figures 3 and 4 to create a sort of \impact genome." Only altmetrics with nonzero counts are shown, and counts of each altmetric are normalized by that metric's maximum. Articles are arranged so that those with higher mean event counts across all metrics are further left. Altmetrics have potential to improve the speed and breadth of scienti c evaluation. CitedIn and total-impact are two tools in early development that aim to gather altmetrics. A test of these tools using a real-life dataset shows that they work, and that there is a meaningful amount of altmetrics data available for use. These tools continue to improve: check out the current versions for up to date capabilities.

The properties and validity of these data, however, are still unclear, and call for additional research. What is the scholarly value of, for instance, a Mendeley bookmark or a Wikipedia citation? Future work should also investigate how altmetrics for di erent sets of articles can be compared; this is a particularly tricky problem given the high dimensionality of altmetrics data, and may bene t from better visualization techniques, or statistical approaches like principle component analysis and factor analysis. - Source code for CitedIn: http://code.google.com/p/citedin - Source code for total-impact: https://github.com/mhahnel/total-impact - Source code and data for analysis in this paper:

https://github.com/jasonpriem/altmetrics-tools-iConference-poster - The authors of the paper are key developers on CitedIn and Total-Impact

References

1. Bollen , J., Van de Sompel, H., Hagberg , A. , Chute , R.: A principal component analysis of 39 scienti c impact measures . PLoS ONE 4 ( 6 ), e6022 (06 2009 )

2. Bornmann , L. , Daniel , H.D.: What do citation counts measure? A review of studies on citing behavior . Journal of Documentation 64 ( 1 ), 45 { 80 ( 2008 )

3. Bourne , P. , Clark , T. , Dale , R., de Waard, A. , Herman , I. , Hovy , E. , Shotton , D. , on behalf of the Force11 community: Force11 White Paper: Improving the Future of Research Communication and e- Scholarship ( 2011 ), http://force11.org/white_ paper

4. Brody , T. , Harnad , S. , Carr , L. : Earlier web usage statistics as predictors of later citation impact: Research articles . J. Am. Soc. Inf. Sci. Technol . 57 ( 8 ), 1060 {1072 (Jun 2006 ), http://dx.doi.org/10.1002/asi.v57: 8

5. Cronin , B. : The Hand of Science: Academic Writing and Its Rewards . Scarecrow Press ( 2005 )

6. Groth , P. , Gurney , T. : Studying Scienti c Discourse on the Web using Bibliometrics: A Chemistry Blogging Case Study. In: WebSci10 Extending the Frontiers of Society OnLine ( 2010 )

7. Hull , D. , Pettifer , S.R. , Kell , D.B. : Defrosting the digital library: bibliographic tools for the next generation web . PLoS computational biology 4 ( 10 ), e1000204 ( 2008 ), http://www.ncbi.nlm.nih.gov/pubmed/18974831

8. L. , V. , K. , H.: Relationship between links to journal web sites and impact factors . Aslib Proceedings: new information perspectives 54 ( 6 ), 356 { 361 ( 2002 )

9. Priem , J. , Costello , K.L. : How and why scholars cite on Twitter . Proceedings of the 73rd ASIS&T Annual Meeting 73 , http://jasonpriem.org/self-archived/ Priem\_Costello\_Twitter.pdf

10. Priem , J. , Hemminger , B.H. : Scientometrics 2.0: New metrics of scholarly impact on the social Web . First Monday 15 ( 7 ) ( Jul 2010 ), http://firstmonday.org/htbin/ cgiwrap/bin/ojs/index.php/fm/article/view/2874

11. Priem , J. , Taraborelli , D. , Groth , P. , Neylon , C. : Alt-metrics: a manifesto ( 2010 ), http://altmetrics.org/manifesto/

12. Shema , H. , Bar-Ilan , J. : Characteristics of Researchblogging.org science Blogs and Bloggers. altmetrics 11 , http://altmetrics.org/workshop2011/shema-v0/

13. Thelwall , M. , Vaughan , L. , Bjorneborn, L.: Webometrics . Annual Review of Information Science and Technology 39 ( 1 ), 81 {135 (Oct 2006 ), http://dx.doi.org/ 10.1002/aris.1440390110