Poorly Cited Articles in Peer-Reviewed Cardiovascular Journals from 1997 to 2007CLINICAL PERSPECTIVE
Analysis of 5-Year Citation Rates
Background—The extent to which articles are cited is a surrogate of the impact and importance of the research conducted; poorly cited articles may identify research of limited use and potential wasted investments. We assessed trends in the rates of poorly cited articles and journals in the cardiovascular literature from 1997 to 2007.
Methods and Results—We identified original articles published in cardiovascular journals and indexed in the Scopus citation database from 1997 to 2007. We defined poorly cited articles as those with ≤5 citations in the 5 years following publication and poorly cited journals as those with >75% of journal content poorly cited. We identified 164 377 articles in 222 cardiovascular journals from 1997 to 2007. From 1997 to 2007, the number of cardiovascular articles and journals increased by 56.9% and 75.2%, respectively. Of all the articles, 75 550 (46.0%) were poorly cited, of which 25 650 (15.6% overall) had no citations. From 1997 to 2007, the proportion of poorly cited articles declined slightly (52.1%–46.2%, trend P<0.001), although the absolute number of poorly cited articles increased by 2595 (trend P<0.001). At a journal level, 44% of cardiovascular journals had more than three-fourths of the journal’s content poorly cited at 5 years.
Conclusion—Nearly half of all peer-reviewed articles published in cardiovascular journals are poorly cited 5 years after publication, and many are not cited at all. The cardiovascular literature and the number of poorly cited articles both increased substantially from 1997 to 2007. The high proportion of poorly cited articles and journals suggests inefficiencies in the cardiovascular research enterprise.
The cardiovascular literature has expanded rapidly, with the number of cardiovascular journals more than doubling between 1998 and 2010.1 Scientific advances, greater access to data, increasing resources for research, and the demand on researchers to produce publications are likely contributors to this rapid growth. Concurrently, the availability of many articles online and the emergence of open-access journals provide increasing access to this body of literature. On the surface, these changes are encouraging because they suggest growth and more effective dissemination of knowledge in the cardiovascular sciences.
Editorial see p 1739
Clinical Perspective on p 1762
However, many of these articles and journals may have limited impact. For example, not all research is novel or exhibits high quality. Limitations such as unnecessary duplication of research,2 prolonged delays to publication,3 and methodological flaws4–7 may reduce the impact and value of an article to readers. Furthermore, the academic phenomenon of publish or perish may lead to persistent pressure to publish less useful or poorly performed science. Many medical journals are for-profit enterprises or generate substantial revenues for nonprofit entities, and the growth in journals may potentially reflect publishers’ motivation to increase profits. A potential downside of increased publications may be that useful and high-quality articles may be overlooked. In this context, an article may make minimal impact because its scientific contribution and value are unrecognized. The generation of research output of limited impact, irrespective of the reason, may ultimately reflect inefficiency and waste in the research enterprise.
One measure of the impact or importance of a published article is the number of citations it accrues. Citation, a widely used metric in publishing, reflects recognition that the knowledge generated by the article has influenced others’ research or practice.8 Although the number of citations has limitations as a measure of impact, highly cited articles generally identify research that is useful to others. In contrast, poorly cited articles may indicate resources spent on research output that is either unrecognized or of limited use. Furthermore, if the expansion of publications is attributable to growth in useful research output, an increasing number of published articles are more likely to be cited often, especially given the increased number of articles available to cite them. In contrast, if the expansion is attributable to articles that are either unrecognized or of limited use, then more of these cardiovascular articles may remain poorly cited.
To provide perspective on the value of increased publications, we determined the number of poorly cited articles published in cardiovascular journals 5 years after publication, whether these articles clustered within certain journals, and examined the trends from 1997 through 2007. Furthermore, because article access and visibility may affect the number of citations, we evaluated the 5-year citation of articles in the subset of open-access cardiovascular journals.
We identified articles in cardiovascular journals from Scopus, a citation database produced by Elsevier. Scopus is the largest database available for citation analysis and contains >50 million reported records from >21 000 titles including 100% coverage of articles indexed in Medline and PubMed.9 Scopus captures the citation of any article by other records within the Scopus database and provides a yearly citation count for each article. Scopus has more coverage and retrieves a greater number of citations in comparison with other citation databases.10 As stated by Scopus, all indexed journals in Scopus publish peer-reviewed content and have, at minimum, an English language abstract for each article.
Selection of Cardiovascular Articles and Journals
From this data set, we identified all journals from 1997 to 2007 containing primarily cardiovascular content by using SCImago, a third-party provider that categorizes journals into fields based on data obtained from Scopus.11 We then reviewed and refined the SCImago list of cardiovascular journals to ensure that we only included journals primarily publishing content relevant to the cardiovascular field. For each journal, we determined the open-access status of the journal based on journal data published by Elsevier.12 From the identified cardiovascular journals, we then selected documents indexed in Scopus as articles for analysis because these are typically original articles and contain original content. We excluded documents indexed as reviews and other noncitable documents such as editorials, letters, and conference abstracts because these documents do not typically contain original content and are normally excluded from citation analysis.13
Rates of Poorly Cited Cardiovascular Articles
We assessed the number and proportion of poorly cited articles recorded in Scopus 5 years following the year of publication. We defined poorly cited articles by using a conservative cutoff of ≤5 citations at 5 years after publication. We chose the 5-year outcome period because the peak citation of articles typically occurs within the first 2 to 4 years after publication.14 Therefore, this time horizon should provide sufficient time for an article to be recognized and potentially cited. We also included author and journal self-citations within the 5-year period because many authors (and journals) may use prior published work to advance their subsequent work.
Rates of Poorly Cited Cardiovascular Journals
To assess whether poorly cited articles cluster within certain journals, we assessed the proportion of poorly cited articles within each journal. We further defined a journal as a poorly cited journal if >75% of its article content was poorly cited 5 years after publication. We defined moderately cited journals as those with 26% to 75% of their article content poorly cited and well-cited journals as those with ≤25% of their article content poorly cited by using the above definition. For this outcome, we limited our analysis to journals publishing at least 20 articles per year to obtain an accurate estimate of the journal citation rates. To further characterize citation patterns within journals, we reviewed each journal’s stated aim, scope, and author instructions to categorize all journals into those that solely published basic science articles and those that accepted clinical or population health articles. Journals that accepted both types of publications were categorized as mixed. We then evaluated the proportion of poorly cited article content within each journal type.
We summarize data as frequencies and percentages for categorical variables. We present citation counts as median and interquartile range. For the analysis of yearly trend at the article level, we compared the absolute number and the proportion of poorly cited articles within each year from 1997 to 2007. For yearly trend analysis at a journal level, we compared the yearly trend in the proportion of poorly cited journals. To assess the 5-year citation rate among articles where access to the publication is not a barrier, we repeated our analysis (at the article and journal level) among the subset of open-access cardiovascular journals. We used the Cochran-Armitage test for all trend analyses. For all statistical analysis, significance levels were 2-sided with a P value <0.05. We used SAS 9.2 (SAS Institute Inc., Cary, NC).
We identified 222 unique cardiovascular journals indexed at any time from 1997 to 2007 that contained 164 377 original articles over the same period. The number of unique cardiovascular journals indexed in Scopus increased from 113 in 1997 to 198 in 2007, representing an increase of 75.2% (trend P<0.001, Figure 1A). Of the 113 journals from 1997, 103 (91%) were still indexed in Scopus in 2007. The remaining 10 journals (9%) from 1997 were no longer indexed in 2007. Of the 198 journals indexed in Scopus in 2007, 52% were, therefore, present in 1997 and the remaining 95 journals (42%) were new journals. The number of articles in these journals indexed by Scopus increased by 56.9%, from 12 761 in 1997 to 20 024 in 2007 (trend P<0.001, Figure 1B).
Rates of Poorly Cited Articles
Of the 164 377 articles, 75 550 (46.0%) articles were poorly cited, of which 25 650 (15.6% of all articles) were not cited at all 5 years after publication. Poorly cited articles were cited a median of 2 times (interquartile range, 1–4 citations) during the 5 years after publication. The remaining 88 827 (54.0%) articles were well cited. Well-cited articles were cited a median of 17 times (interquartile range, 10–32 citations) during the 5 years after publication.
From 1997 to 2007, the proportion of poorly cited articles decreased from 52.1% to 46.2% (trend P<0.001, Figure 2A), largely because of a decrease in the proportion of articles with no citations during the same period (19.1% overall in 1997 to 14.7% overall in 2007, trend P<0.001). However, because of the growth in volume of all articles, the absolute number of poorly cited articles increased from 6648 in 1997 to 9243 in 2007, an increase of 2595 articles, of which 410 had no citations (Figure 2B).
To rule out the probable effect of citations beyond 5 years, we conducted a post hoc analysis of 2002 to 2007 citations for journal articles published in 1997, which showed that 91% of articles that are poorly cited during the first 5 years (1997–2001) continued to be poorly cited within the next 5 years (2002–2007; Figure I in the online-only Data Supplement and Table I in the online-only Data Supplement).
Rates of Poorly Cited Cardiovascular Journals
When all cardiovascular journals published from 1997 to 2007 were ranked by the proportion of poorly cited article content, 44% were poorly cited (where more than three-fourths of the journal’s content was poorly cited 5 years after publication, Figure 3A). Of the remaining journals, 47.2% were moderately cited (ie, contained 26%–74% poorly cited content) and 7.9% were highly cited (ie, contained ≤25% of poorly cited content). Table identifies the journals with the least amount of poorly cited content (Table II in the online-only Data Supplement includes the full list of journals). Although our focus was to assess poorly cited content within journals, we performed a post hoc analysis of very highly cited articles (>25 citations). We noted that such articles were rare among poorly cited journals (Figure II in the online-only Data Supplement).
From 1997 to 2007, the proportion of poorly cited cardiovascular journals declined by 10.8% (from 49.5% to 38.7%, trend P<0.001, Figure 3B). Correspondingly, the proportion of moderate and well-cited journals increased by 6.1% and 4.7%, respectively (all trend P<0.05). However, because of the growth in the number of journals, the absolute number of poorly cited journals increased from 45 to 63 journals (trend P<0.01). Similarly, the absolute number of moderately cited journals increased from 38 to 78, and the number of well-cited journals increased from 8 to 22 (both trend P<0.01).
When journals were categorized by the type of articles published, 6 were categorized as publishing purely basic science research, 72 as publishing clinical or population health content, and 104 as publishing mixed article content. The proportion of poorly cited content was 14.4% among the basic science journals, 63.3% among clinical or population health journals, and 40.3% among journals that published mixed article content.
Poorly Cited Articles in the Subset of Open-Access Cardiovascular Journals
Of the cardiovascular journals included in the study, 25 (11.3%) were open-access journals that contributed 12 829 (7.8%) of all included articles. Of the articles in these open-access journals, 71.8% were poorly cited with 28.7% overall having no citations at 5 years. From 1997 to 2007, the proportion of poorly cited articles in open-access journals declined by 30.2% (from 91.5% to 61.3%, trend P<0.001, Figure 4A). Despite the declining trend, the proportion of uncited and poorly cited articles published in open-access journals remained substantially higher than non–open-access journals throughout the study period. At a journal level, 14 of the 25 open-access journals (56%) were poorly cited journals with more than three-fourths of the journal’s content poorly cited at 5 years (Figure 4B). Comparatively, 42% (66/157) of non–open-access journals were poorly cited journals.
In our study of all articles published in peer-reviewed cardiovascular journals from 1997 through 2007, we found that nearly half of all articles were poorly cited 5 years after publication. Although this proportion has changed modestly in over a decade, the proliferation of journals over the same period has resulted in a substantial increase in the absolute number of poorly cited articles. Furthermore, we found that 44% of journals have more than three-fourths of their content poorly cited 5 years after publication. Using a conservative measure of poor citation that included self-citations, our findings suggest that many of the articles that are published in the cardiovascular literature are not cited in subsequent articles. Such high rates of poorly cited articles may indicate significant waste in the research enterprise.
Several reasons may explain poor citation of articles. The literature frequently notes studies that lack novelty, address topics of little relevance, and methodological limitations; such studies, lacking in quality or usefulness, may be poorly cited because they make minimal contributions to scientific advancement. Even novel and high-quality articles may remain poorly cited because of lack of recognition of their scientific merit.15,16 For example, it is well known that researchers often fail to recognize and incorporate appropriate references, a problem that may be compounded by the ever increasing volume of articles that are produced.16 Furthermore, factors relating to article recognition, such as the number of keywords and authors, are known to influence citation independent of the article’s scientific content. However, our observations occurred during a period of marked expansion of Internet access and the emergence of open-access journals, public repositories such as PubMed Central, and social media; articles’ accessibility, visibility, and potential for recognition are perhaps more favorable now than ever before. For this reason, lack of recognition alone is unlikely to justify the near 50% poor citation rates observed in our study.
Our findings extend the literature by showing the temporal growth in the number of poorly cited articles. In the early 1990s, several cross-sectional studies reported that 16% to 55% articles in the field of science were not cited and rates as high as 98% reported in the arts and humanities literature.17–19 These observations raised the question of whether such large-scale uncitedness reflected a waste of research resources. However, concern existed that citations were not comprehensively captured because citation databases at the time were relatively small and indexed only the main journals.15,16,20 We use a citation database that is severalfold larger and more likely to comprehensively capture citations than previous investigations of this topic. We show the presence of a high proportion of poorly cited articles in the cardiovascular literature and a concerning growth in the number of poorly cited articles. We focus on the cardiovascular literature because citation patterns can be variable between subspecialties with autonomous journal networks15,16,21; focusing on the cardiovascular literature allowed us to closely examine citation patterns within a specialty that has experienced rapid publication growth. However, the growth in publications is a widespread phenomenon in medicine,1 and our findings are likely to be of relevance to the broader medical literature.
We also show for the first time that a large proportion of cardiovascular journals are poorly cited and the absolute number of poorly cited journals is increasing, an observation that questions the value of many of the new cardiovascular journals. Journal publishing is a profitable enterprise. The continued expansion of cardiovascular journals raises the possibility that monetary gain in the context of demand to publish, rather than the presence of high-quality articles, is the primary driver of the continued journal expansion. Indeed, an increasing presence of open-access pay-for-publishing journals with dubious peer-review processes, or so-called predatory journals, is well documented.22 Although journals indexed in Scopus are required to meet ethical publishing guidelines, and unlikely to include such journals, it nevertheless highlights the presence of perverse incentives for continued journal expansion. Of note, we observe a modest decline in the proportion of poorly cited journals with time that may suggest a gradual improvement in the distribution of cited articles among cardiovascular journals.
Irrespective of the cause, poorly cited articles are a cause for concern. Research typically consumes researchers’ time, is expensive, creates demand on regulatory processes such as the institutional board review, and may involve sizable contributions from study subjects. This is equally applicable to both clinical and basic science research. For example, 53% of the US $35.5 billion in federal research funding for Health in 2009 was for basic science research.23 Furthermore, basic science research also involves and requires many (and often extensive) regulatory processes such as institutional board review approvals for animal studies, tissue banking or genetic testing. If the end research output is of limited use, or its value is not recognized, then these resources and contributions may be wasteful. Many of the poorly cited articles that we identify may well be low-cost studies produced with minimal resources. However, poor-quality studies, and misleading findings, are frequent even among trials registered in clinical trials registries,6 leading some to suggest that waste in the research enterprise is a widespread and costly problem.24 Several recently proposed strategies may reduce such waste and increase the use of research, including (1) prioritizing research addressing the concerns of patients and clinicians to ensure research output is relevant to end users25; (2) improving research design, conduct, analysis, and reporting to improve the quality of research output26; (3) ensuring that research protocols, full study reports, and data are accessible to the end users27; and (4) streamlining regulatory and study management processes so that they are appropriate to the risks associated with the research, thereby minimizing undue burden on researchers and patients.26 Rigorous application of these steps in the research process may generate output of greater use to others.
Our findings have implications for how we value articles, journals, and research productivity more broadly.28 The number of publications by the author is the most commonly used measure of the research performance29 and often a deciding factor in awarding qualifications, academic promotions, and granting of competitive research funding. However, our observations suggest that the number of publications alone may be a poor indicator of the use of the research output. Alternatives such as the H-index,30 which considers both the number of publications and the number of citations, or adoption of a panel of indices31 or article level metrics,32 may provide a more robust assessment of the author’s research productivity and the value of the research output.
Our analysis has several limitations. First, article citation is the most common quantitative indicator of article impact. However, research may have different types of impact and achieve these impacts by a multitude of diverse routes, which may not be readily measurable or transparent. For example, citation does not consider how well an article is viewed, read, discussed, and integrated into grant applications for future research. Alternative measures of research impact include the willingness of others to invest in the research or pay for the knowledge generated,28 financial returns of the research output, dissemination of the findings into guidelines and clinical practice, and the broader health policy, social, economic, and environmental impact. Second, increasing numbers of publications are likely to lead to increased citation of prior publications. This endogeneity bias, over time, may lead to an increase in citation of articles and a progressive reduction in the proportion of articles that are poorly cited. However, despite the endogeneity bias, we only observed a relatively modest reduction in the proportion of poorly cited articles with time with an increase in the absolute number of poorly cited articles. Third, we selected documents indexed in Scopus as articles to identify original research rather than individually reviewing articles. Although the latter method is perhaps more precise, it was impractical because of the volume of articles included in this analysis. Fourth, research that is likely to lead to the highest impact often tends to be high risk; in high-risk, high-reward situations, the research may still warrant investment despite a high degree of anticipated failure. However, even negative or unexpected findings from a well-conducted study are likely to generate important knowledge and one would expect such research output to be well cited. Last, the knowledge within an article may be slow to be recognized. However, 5 years are a reasonable duration of time from publication for an article to be recognized.
Nearly half of all peer-reviewed articles published in cardiovascular journals are poorly cited 5 years after publication with continued growth in the absolute numbers of poorly cited articles from 1997 to 2007. Furthermore, 44% of cardiovascular journals have more than three-fourths of its article content poorly cited at 5 years. Although citation is one of many measures of article use and value, such a high proportion of poorly cited articles and journals suggests the presence of waste in the cardiovascular research enterprise spent on generating of research output that is either unrecognized or of limited use to end users.
Sources of Funding
This study was supported by grant 1U01HL105270-03 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. Dr Ranasinghe is supported by an Early Career Fellowship cofunded by the National Health and Medical Research Council of Australia and the National Heart Foundation of Australia. Dr Ross is supported by the National Institute on Aging (K08 AG032886) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program.
Drs Krumholz and Ross receive support from Medtronic, Inc. and Johnson & Johnson to develop methods of clinical trial data sharing, from the Centers of Medicare and Medicaid Services to develop and maintain performance measures that are used for public reporting, and from the Food and Drug Administration and Medtronic to develop methods for postmarket surveillance of medical devices. Dr Krumholz is Editor of the American Heart Association’s Circulation: Cardiovascular Quality and Outcomes and of the Massachusetts Medical Society’s Journal Watch Cardiology and CardioExchange. Dr Masoudi is Associate Editor of the American Heart Associations’ Circulation: Cardiovascular Quality and Outcomes and of the Massachusetts Medical Society’s Journal Watch Cardiology. Dr Spertus is the Deputy Editor of the American Heart Associations’ Circulation: Cardiovascular Quality and Outcomes. Dr Ross is Associate Editor of the American Medical Association’s JAMA Internal Medicine. The other authors report no conflicts.
Guest Editor for this article was Donna K. Arnett, PhD.
The online-only Data Supplement is available with this article at http://circ.ahajournals.org/lookup/suppl/doi:10.1161/CIRCULATIONAHA.114.015080/-/DC1.
- Received September 18, 2014.
- Accepted March 3, 2015.
- © 2015 American Heart Association, Inc.
- Zhang Y,
- Kou J,
- Zhang XG,
- Zhang L,
- Liu SW,
- Cao XY,
- Wang YD,
- Wei RB,
- Cai GY,
- Chen XM.
- Cooper NJ,
- Jones DR,
- Sutton AJ.
- Hewitt C,
- Hahn S,
- Torgerson DJ,
- Watson J,
- Bland MJ.
- Rutjes AWS,
- Reitsma JB,
- Di Nisio M,
- Smidt N,
- van Rijn JC,
- Bossuyt PMM.
- Glasziou P,
- Meats E,
- Heneghan C,
- Shepperd S.
- Elkana Y,
- Lederberg J,
- Merton RK.,
- Thackray A,
- Zuckerman H
- Garfield E MM,
- Small H.
- 9.↵Scopus. New York, NY: Elsevier; 2013:2013.
- 11.↵SCImago. SJR — SCImago Journal & Country Rank. 2007;2013.
- 12.↵Scopus Title List. New York, NY: Elsevier. http://www.elsevier.com/__data/assets/excel_doc/0005/226742/title_list.xlsx. Accessed March 31, 2014.
- Pendlebury DA.
- Hamilton DP.
- Hamilton DP.
- Garfield E.
- 23.↵Federal Funds for Research and Development: Fiscal Years 2009–11. Arlington, VA: National Science Foundation; 2012.
- Krumholz HM.
- Patel VM,
- Ashrafian H,
- Ahmed K,
- Arora S,
- Jiwan S,
- Nicholson JK,
- Darzi A,
- Athanasiou T.
- Hirsch JE.
The cardiovascular literature is expanding at a tremendous pace, yet whether this expansion reflects growth in useful research output is uncertain. One measure of the impact of an article is the number of citations it accrues after publication, because citation is recognition that the knowledge generated by the article has influenced others’ research or practice. Using this metric, we assessed the 5-year citation of articles published in cardiovascular journals from 1997 to 2007 in the Scopus citation database. Our analysis of 164 377 articles from 222 journals show that nearly half (46.0%) of all cardiovascular articles were poorly cited (<5 citations), and 15.6% articles had no citations 5 years after publication. Whereas the proportion of poorly cited articles declined slightly (52.1% to 46.2%) from 1997 to 2007, the absolute number of poorly cited articles increased by 2595 articles over the same period. At a journal level, 44% of all cardiovascular journals had more than three-fourths of the journal’s content poorly cited at 5 years. These findings suggest that many journals and articles that are increasingly produced have limited value to end users, or the value of the research output is unrecognized. Our findings are cause for concern because research is expensive and resource intensive, and it may involve sizable contributions from study subjects. The generation of research output of limited impact, irrespective of the reason, ultimately reflects inefficiency and waste in the research enterprise. Prioritizing research of high quality and relevance to end users, and increasing accessibility to research findings, may reduce waste and increase the value of research output.