In this paper we explore post retraction citations to retracted papers. The reasons for retractions in our sample were data manipulation, small sample size, scientific misconduct, and duplicate publication by the authors. We found, that the huge majority of the citations are positive, and the citing papers usually fail to mention that the cited article was retracted. Retracted articles are freely available at the publishers' site, which is probably a catalyst for receiving more citations.
Introduction
Studies on retracted articles show that the amount of retracted articles has increased in
relative measure to the overall increase in scientific publications [
In this study we sought out to find the context around post-retraction citations with the main purpose of finding out whether they are negatively, positively or neutrally mentioned. In this case study we present a sample of five retracted articles that have post-retraction citations tracked in 2015 and 2016. 2
ScienceDirect, Elsevier’s full text database was accessed in October 2014. The database was queried for the term “RETRACTED” in the article title and its retraction notice. In ScienceDirect, each retracted article is preceded with the word “RETRACTED”. In addition, each Elsevier journal incorporates a retraction notice which explains who retracted article and the reason for retraction. This allowed us to manually code each article in our dataset with an additional field “retracted by” that represented the person/s requesting the retraction.
A total of 1,203 results retrieved from which 988 were retracted articles. The results excluded were retraction notices, duplicates and papers whose original titles included the word "retracted".
For this study we selected the five top articles that were cited most (more than 20 times) since 2015. This way we made sure that the papers all cite retracted articles (since they were all retracted before October 2014). The reason for this decision is that the retraction date of many of the retracted articles is unknown. For each article we extracted the citing documents and analyzed the ones appearing in 2015 and 2016. Overall, we analyzed located 125 citing documents and analyzed 109 of them; 16 documents were unavailable to us mostly because they appear in books to which we did not have access. Each citing document was inspected to identify the precise mention of the retracted article within the text. Each mention was categorized as follows: Positive: A positive citation indicates that the retracted article was cited as legitimate prior work and its findings used to corroborate the author/s current study. Negative: A negative citations indicates that the authors mentioned the retracted article as such and its findings as inappropriate.
Neutral: A neutral citation indicates that the retracted article was mentioned as a publication that appears in the literature and does not include judgement on its validity. 3 3.1
Case study 1: Donmez, G., Wang, D., Cohen, D. E., & Guarente, L. (2010). RETRACTED: SIRT1 Suppresses β-Amyloid Production by Activating the α-Secretase Gene ADAM10. Cell, 142(2), 320-332.
This article was published in 2010 in Cell and retracted in 2014 due to irregularities in graphs and data misrepresentation in the images. Although the graphs and images did not have any bearing on the validity of the results, according to the retraction notice, the editors stated that “…the level of care in figure preparation in Donmez et al. falls well below the standard that we expect, and we are therefore retracting the paper”.
We conducted an individual content analysis of the most recent 36 citations which
were tracked in 2015 and 2016. We were able to analyze 32 citing articles in context.
Our results show that the citations are mostly positive (see Fig. 1). One negative
mention was found in a letter to the editor of Journal of Korean Medical Science written
“giving the above article as an example of how altered graphics are causing bias in the
biomedical field and result in numerous articles being retracted” [
In this case, the editor indicated that the actual results of the study were valid, and this could be the reason for the continuous citations of the article. In one other case, although the article was cited positively in the paper, in the reference list it was noted that the article was retracted.
40 s fero iltrec 6
1 b a -2020
5 uNm iitgn 102 0 c 4 NA
1
This article, published in 2012 was the subject of a debate surrounding the validity of the findings, use of animals and even accusations of fraud. Its publication and retraction process have resulted in the “Séralini affair” which became a big media news item. The article described a 2-year study of rats which were fed genetically modified (GM) crops and showed increased tumors. The study, which was also scrutinized by government agencies, received major media attention that resulted in the creation of a social movement against GM food. The demand to label of all GM foods is still underway. Despite the accusation of fraud and fabrication of results, the editors found no such evidence to that effect. However, the article was retracted because of the “low number of animals” used in this study which lead to the conclusion that “no definitive conclusions can be reached with this small sample size”.
This article was cited 109 times since its publication in 2012 with 23 citations tracked after its retraction (2015-2016) out of which 18 citing articles were accessible to us. As can be seen in Fig. 2 post-retraction citations are divided. Although more citations are seen to be negative, the positive and neutral ones are also present. The negative citations mostly point to the media frenzy around the results. Positive mentions appear in similar studies which claim that concerns raised by the GM study are valid and the dangers of GM foods to humans should be studied further. 5 7 5 6 8 s fero iltrec 66
1 b a -24 0 5 uNm iitgn 1022 c 0
15 s fero iltrec 1610 0 2 b a -5 5 uNm iitgn 102 0 c
NA
The study was republished in 2014 by Environmental Sciences Europe [
The republished article received 17 citations in 2015 and 2016. The vast majority
of them being positive mentions (see Fig. 3). In addition, some criticism towards the
peer-review practices of the retracting editors were also detected [
13 2 NA 1
1
Case 3: Mukherjee, S., Lekli, I., Gurusamy, N., Bertelli, A. A., & Das, D. K. (2009). RETRACTED: Expression of the longevity proteins by both red and white wines and their cardioprotective components, resveratrol, tyrosol, and hydroxytyrosol. Free Radical Biology and Medicine, 46(5), 573-578.
The leading author of the paper, Dipak Das and his lab at the University of Connecticut Health Sciences Center were the subject of an ethical investigation by the university. The results of the university’s investigation led to the retraction of all of Dr. Das' papers due to scientific misconduct and data manipulation. This particular paper was investigated by the journal’s ethics committee along with an additional paper that appeared in the same journal. The retraction notice states that the journal’s ethics committee “analyzed the data presented, and then further concluded that …. on re-examination of these two FRBM (Free Radical Biology and Medicine) papers that they contain clear evidence of obvious cutting, pasting and manipulation of data in experimental blots.” The article, which was retracted in 2012, received 85 citations since its publication in 2009, 21 of which occurred in 2015 through March 2016. All 17 citing accessible citing articles referred to the article’s findings as legitimate. For example, “Plants containing resveratrol, a potent antioxidant, has been used widely in the treatment of various ailments” [12 p.1286] or “Recent studies have also shown that red wine upregulates the protein expression of sirtuin “[13, p.1213]. 3.4
Case 4: Walumbwa, F. O., Wang, P., Wang, H., Schaubroeck, J., & Avolio, B. J. (2010). RETRACTED Psychological processes linking authentic leadership to follower behaviors. The Leadership Quarterly, 21(5), 901-914. This article was retracted in 2014 due to serious data manipulation and falsification. In the retraction notice of this article, the editors of the journal went to great lengths to examine and re-examine the statistical claims made by the authors using the services of three separate methodologists. Following the methodologists’ findings of irregularities in the reported data and falsification of results, and the authors’ lack of proper response to their findings, the article was retracted from the journal. However, the article continued to be cited despite the lengthy and detailed retraction notice.
A close examination of the post retraction citations (2015- March 2016 – 24 citations of which 23 were analyzed) shows that all citations were positive citations, meaning that the citing authors used findings from this article to support their findings. The subject of “authentic leadership” is popular in management studies and has seen a surge in publications since 2012. This could explain the overall positive citations of the article. 3.5
Case 5: Li, C., Tao, X. M., & Choy, C. L. (1999). RETRACTED: On the microstructure of three-dimensional braided preforms. Composites Science and Technology, 59(3), 391-404.
This article, published in 1999 was retracted due to an identical version which was published 2 years earlier. In the retraction notice the editors state that “The article duplicates significant parts of a paper that had already appeared in [J China Textil Univ, 1997, 14(3), 8-13]”. The authors in this case re-used data they already published on and re-published it in a different journal. However, this article has been cited even in recent years despite being retracted for many years. A content analysis of the 18 out of the 21 recently citing articles from 2015 and 2016 shows that this article is being referred to mostly in positive context or mentioned as a legitimate piece in the literature. Here too, there is one paper that cites the article positively in the text, but in the references it appears as retracted. 4
Discussion and Conclusions As can be seen from the examples above, retracted articles continue to be cited years after retraction and despite retraction notices being posted on publishers’ platforms.
In some cases, the continuous citations rates could be the result of general interest by the public or media. For example, the Séralini article evoked an ongoing public debate regarding the safety of GM foods which resulted in a call to label all such food. This could explain the continuing interest in the study and its citations. The article was also republished and thus continues to be cited despite of the fact that the authors did not modify it. In the case of the Mukherjee article, again, public interest could explain its continuing citations. Resveratrol was hailed by the media as an important supplement that could ensure longevity and good health and is an off the counter supplement available in vitamin shops. Finally, the Walumbwa article which describes ‘authentic leadership’ and followers’ dynamic is also a topic of media and business management interest. With numerous management books published on this topic it has been accepted as a management style encouraged by corporations.
In other cases, the reason for retraction does not deter others from citing the article.
For example, the Donmez article (case study 1 above) was retracted because of poor
graphing and data representation. However, the editors do state in the retraction notice
that these faults do not apply to the results of the study, even though on PubPeer [
Regardless of the reasons speculated for the post-retractions citations, the fact that invalid and falsified research is continuing to appear as valid research is concerning. We recommend that publishers use reference checks to all submitted articles to detect citations of retracted articles and remove them or at least request an explanation from the authors for citing a retracted paper in a positive or neutral manner. This explanation should clearly appear in the paper. In addition, we would recommend the deletion of retracted articles from publishers’ websites. Currently, at least for the major publishers: Elsevier, Springer Nature and Wiley, but possibly a general practice, retracted articles are not only available on the publishers' site, but they are freely available, without the need for a subscription or for a one-time payment. While leaving a retraction notice, the article itself should not appear on platforms such as ScienceDirect or others. Although versions of these articles may appear elsewhere, the journal websites should not carry these versions and make it difficult for authors to download, read and consequently cite retracted articles. 5
Acknowledgement The first author was supported by EU COST Actions PEERE (TD1306) and KnowEscape (TD1210).