Published online before print September 8, 2008, doi: 10.1161/CIRCULATIONAHA.108.794016
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Circulation. 2008;118:1321-1327.)
© 2008 American Heart Association, Inc.
Health Services and Outcomes Research |
Differential Citation Rates of Major Cardiovascular Clinical Trials According to Source of Funding
A Survey From 2000 to 2005
From the Center for Cardiovascular Disease Prevention, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Mass.
Reprint requests to Paul M Ridker, MD, Center for Cardiovascular Disease Prevention, Brigham and Women’s Hospital, 900 Commonwealth Ave E, Boston MA 02215. E-mail pridker{at}partners.org
Received May 23, 2008; accepted July 10, 2008.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Background— Prior work indicates that therapeutic trials funded by for-profit organizations are more likely to report positive findings than trials funded by not-for-profit organizations. What impact, if any, funding source has on subsequent dissemination of trial data is uncertain. To address this issue, we used the number of citations per publication per year to assess differences in trial dissemination according to funding source.
Methods and Results— We assessed 303 consecutive superiority trials of cardiovascular medicine published between January 1, 2000, and July 30, 2005, in the Journal of the American Medical Association, The Lancet, and the New England Journal of Medicine. The primary outcome measure was the number of citations per publication per year up to December 31, 2006. Overall, the median number of citations per publication per year was 46 for trials funded exclusively by for-profit organizations, 37 for trials jointly funded, and 29 for trials funded by not-for-profit organizations (P=0.0007). Higher citation rates for trials funded by for-profit organizations were consistently observed in analyses stratified by journal and various trial design features and were most striking when the new intervention was favored over the standard of care; in this subgroup, the median number of citations per publication per year was 52 for trials funded by for-profit organizations compared with 25 for trials funded by not-for-profit organizations (P=0.0006). In marked contrast, in analyses limited to trials in which the new intervention was significantly worse than the standard of care, an inverse pattern was observed with fewer citations per publication per year for trials funded by for-profit organizations compared with not-for-profit organizations (33 versus 41; P=0.048). Higher citation rates were observed for industry-funded trials than for federally funded trials even when the trials dealt with similar issues and were published back-to-back in the same journal.
Conclusions— Dissemination of clinical trial results is important for clinical practice but appears to be biased in favor of for-profit entities. Consideration should be given to more extensive promotion of clinical trial results that are funded by not-for-profit organizations.
Key Words: bibliometrics • cardiovascular diseases • financial support • organizations, nonprofit • randomized controlled trials as topic
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Major therapeutic trials funded by for-profit organizations are more likely to report positive findings than trials funded by not-for-profit organizations,1 suggesting that incentives within for-profit organizations may influence outcomes in clinical trials. However, the impact of any individual trial result on physician behavior is dependent not only on initial publication but also on the subsequent dissemination of results to the medical community. For example, a recent study demonstrated that active propagation of current guidelines for the management of ST-elevation myocardial infarction significantly increased the use of evidence-based therapy, subsequently improving the outcome of patients.2
Clinical Perspective p 1327
The citation rate of an article in the medical literature is viewed as a measure of its dissemination.3 Previous studies have shown that several factors influence the number of citations for a given publication, the strongest being the impact factor of the publishing journal.4 Citation rates also may depend on study design, sample size, or medical subspecialty to which the article applies.5–7 However, few studies have addressed whether the funding source of a clinical trial has an impact on the citation rate of its publication.5 This is an important issue, given that for-profit organizations have more resources than governmental funding agencies to effectively and selectively disseminate the results of their studies.
To address this issue, we surveyed all cardiovascular clinical trials published in the Journal of the American Medical Association, The Lancet, and the New England Journal of Medicine between 2000 and 2005 and sought data on funding source, trial outcome, and subsequent citations in the medical literature.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Study Design
We evaluated a consecutive series of 349 randomized trials in cardiovascular medicine published between January 1, 2000, and July 30, 2005, in Journal of the American Medical Association, The Lancet, or the New England Journal of Medicine.1 Trials were included if they represented the primary report of the trial itself or were a major secondary report based on additional end points that could be critically evaluated on the basis of the randomization process.
One of the authors (D.C.) searched the Institute for Scientific Information Web of Science database to extract the number of citations for each individual trial (www.isinet.com) unaware of either trial outcome or funding source. For each publication, the publication date, publication journal, total number of citations, and number of citations per calendar year were ascertained through December 31, 2006. This date was chosen to ensure an identical closure date for all trials and all potential citation sources. To evaluate whether the citation system used had any impact on outcomes, analyses were repeated with Google Scholar.
A second author (J.T.) reviewed each publication unaware of trial outcome and citation rate and extracted information about funding sources, categorizing trials into 1 of 4 groups: those financed exclusively by federal, state, or other not-for-profit foundations (not-for-profit trials); those financed exclusively by for-profit pharmaceutical or device manufacturers (for-profit trials); those financed jointly (not-for-profit and for-profit); and those for which no source of funding was provided. Trials financed in their entirety by federal, state, or foundation support for which drugs or devices were donated without charge were considered not-for-profit trials.
The third author (P.M.R.) reviewed each clinical trial unaware of the funding source and citation rate and classified all trials into 1 of 3 categories: those for which there was clear evidence of a statistically significant benefit in favor of the newer treatment over the standard of care (SOC); those for which no statistically significant difference between SOC and newer treatment was observed; and those for which there was a statistically significant benefit in favor of the SOC over the newer treatment. Trials were further evaluated according to study design (superiority trials versus others), sample size, number of sites (single center versus multicenter), end-point type (clinical versus surrogate), and whether the treatment of interest was a drug, a device, or any other intervention (eg, behavioral intervention or novel procedure).1
Of the 349 trials reviewed, 25 (7%) were equivalence trials and were excluded from analysis. Of the remaining 324 superiority trials, 303 (94%) provided evidence of funding source (not-for-profit, not-for profit and for-profit, or for-profit funding) and thus were included in this analysis.
Statistical Analysis
Characteristics of the trials according to funding source were compared by use of Wilcoxon rank-sum tests for continuous variables and 
2 tests for categorical variables. Numbers of citations are presented as median (interquartile range [IQR]) unless indicated otherwise. To account for differences in time since publication, we calculated the number of citations per publication per year (total number of citations up to December 31, 2006, divided by the difference in years between the publication date and December 31, 2006) as the principal outcome variable. To evaluate the consistency of the results, we repeated all analyses using the total number of citations per article up to December 31, 2006, assuming that the funding source should not influence the publication date. We used the Kruskal-Wallis test to compare the citation rates across categories of funding source. Stratified analyses were performed according to the outcome of the primary end point (new therapy worse than the SOC, no difference between new therapy and the SOC, new therapy superior to the SOC), the journal in which the trial was published, sample size (<400, 400 to 2000, >2000 participants), type of end point used (clinical versus surrogate), and type of intervention (drug trial, device trial, other).
Finally, we compared the proportion of publications across funding source categories for the 50 most frequently cited publications and the 50 least often cited publications with that of the corresponding 253 remaining trials using 2 tests. All analyzes were carried out with SAS version 9 (SAS Institute Inc, Cary, NC). A 2-tailed value of P<0.05 was considered to indicate statistical significance.
The authors had full access to and take responsibility for the integrity of the data. All authors have read and agree to the manuscript as written.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Characteristics of the 303 superiority trials stratified by funding source are shown in Table 1. As published previously, trials funded by for-profit organizations more often reported positive results than trials funded by not-for-profit organizations (67.2% versus 49.0%).1
|
For the study population as a whole, the median number of citations per publication was 135 (IQR, 64 to 269) during a median follow-up time per publication of 4.1 years (IQR, 2.8 to 5.4 years). Thus, the median number of citations per publication per year was 36 (IQR, 18 to 66). As anticipated, there was significant variation in the median number of citations per publication per year according to journal (New England Journal of Medicine, 46; The Lancet, 32; Journal of the American Medical Association, 24; P<0.0001), sample size (<400, n=24; 400 to 2000, n=35; >2000, n=54; P<0.0001), type of end point (clinical, n=43; surrogate, n=24; P<0.0001), type of study (multicenter, n=40; single center, n=18; P<0.0001), and type of intervention used (drug, n=40; device, n=49; other, n=28; P=0.002) (Table 2).
|
Table 2 presents the number of citations per publication per year according to trial characteristic and funding source. The median number of citations increased from 29 (IQR, 14 to 51) per publication per year for trials funded by not-for-profit organizations to 37 (IQR, 21 to 71) for trials jointly funded to 46 (IQR, 22 to 81) for trials funded exclusively by for-profit organizations (P=0.0007). This trend was consistent across all of the subgroups evaluated, with 1 notable exception; specifically, trials funded by for-profit organizations were cited more than twice as often than trials funded by not-for-profit organizations among studies when the trial outcome significantly favored the new treatment over the SOC (52 versus 25; P=0.0006). In marked contrast, no such differential excess of citations according to funding source was observed when the new treatment was significantly worse than the SOC; in this setting, studies funded by for-profit organizations were, if anything, cited significantly less often than trials funded by not-for-profit organizations (33 versus 41; P=0.048).
We repeated these analyses using the total number of citations per publication rather than the citation rate per year and observed almost identical results (Table 3). In this analysis, the median number of citations increased from 103 (IQR, 50 to 195) per publication for trials funded by not-for-profit organizations to 133 (IQR, 52 to 280) per publication for trials jointly funded to 164 (IQR, 88 to 314) per publication for trials funded exclusively by for-profit organizations (P=0.0006). Similar to the findings reported in Table 2, we again found evidence of a differential citation rate by funding source according to whether the outcome was in favor of the new treatment or not. We also observed that the increased citation rate for trials funded by for-profit organizations is independent of the year of publication (Table 3). On a posthoc basis, we found no evidence that country of origin altered these results.
|
Of the 50 most frequently cited trial publications (range, 89 to 739 citations per year), 31 (62%) were exclusively funded by a for-profit organization, 10 (20%) were jointly funded, and 9 (18%) were funded exclusively by a not-for-profit organization (P=0.015). Thus, 23% of the for-profit trials, 16% of the jointly funded trials, and 9% of the not-for-profit trials were among the 50 most frequently cited studies. Among the 31 trials funded by for-profit organizations in this subgroup, 28 (90%) favored the new treatment, 3 (10%) found no difference between treatments, and none favored the SOC. In contrast, of the 9 trials in this highly cited subgroup funded by not-for-profit organizations, 3 (33%) favored the SOC, 4 (44%) favored the new treatment, and 2 (22%) found no difference between the 2 treatment groups.
Of the 50 least frequently cited trial publications (range, 0 to 14 citations per year), 16 (32%) were funded exclusively by a for-profit organization, 6 (12%) were jointly funded, and 28 (56%) were funded exclusively by a not-for-profit organization (P=0.002). Thus, 12% of the for-profit trials, 10% of the jointly funded trials, and 27% of the not-for-profit trials were among the 50 least often cited studies.
Our primary findings were not altered when we used Google Scholar rather than the Institute for Scientific Information Web of Science. The number of citations per paper per year was highly correlated in these 2 citation tracking systems (Spearman r=0.91).
On a posthoc basis, we found 3 examples of trials dealing with related topics that were published side by side in the same journal, and in each case, the for-profit trial was subsequently cited more often than the not-for-profit or jointly funded trial. For example, in 2001, 2 trials of radiation to prevent restenosis after coronary stenting were presented side by side. One (for profit) was cited 56 times per year for a total of 333 citations, whereas the other (not-for-profit) was cited 24 times per year for a total of 142 citations.8,9 Similarly, in 2002, 2 analyses of novel indications for cardiac pacing were presented simultaneously. One (for profit, surrogate end point) was cited 152 times per year for a total of 696 citations, and the other (jointly funded, hard clinical end point) was cited 40 times per year for a total of 181 citations.10,11 Finally, in 2001, 2 trials of β-blockers for heart failure were presented side by side. One (for profit, positive trial) was cited 134 times per year for a total of 746 citations; the other (jointly funded, negative trial) was cited 48 times per year for a total of 269 citations.12,13
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
These data indicate that cardiovascular clinical trials published in the Journal of the American Medical Association, the New England Journal of Medicine, and The Lancet between 2000 and 2005 were significantly more likely to be cited in subsequent medical publications if funded by for-profit rather than not-for-profit organizations. This discrepancy in citation rate between for-profit and not-for-profit trials was most prominent when the trial outcome favored the new intervention over the SOC. In contrast, among cardiovascular clinical trials in which the outcome significantly favored the SOC over the new intervention, studies funded by for-profit organizations were cited less frequently than those funded by not-for-profit organizations.
The present study expands previous observations that cardiovascular trials funded by for-profit organizations are more likely to report a positive finding.1 In line with previous studies, these data demonstrate that commercial incentives have the ability to influence not only the initial trial outcomes but also subsequent dissemination of clinical trial results. In this regard, among the 50 most cited studies in our analysis, 62% were exclusively funded by a for-profit organization, and 90% of these significantly favored the new treatment.
The present study results emphasize that beyond the requirement of clinical trial registration,14,15 further steps may be needed to ensure equal dissemination of trial results independently of funding organization. The recently introduced public-access policy of government-funded research in the United States16 may be a step toward increasing the availability and dissemination of its research.17 However, our data indicate that more active steps are needed to further increase the dissemination of research funded by not-for-profit organizations and that these steps need to be considered on an international basis.
Previous work evaluating funding source and subsequent citation rates is scarce but consistent with our observations.5,18 Some prior studies also may have been somewhat limited in their ability to avoid confounding by factors known to affect citation rates, including area of investigation, outcome of trial, and journal of publication. To avoid these confounding effects as much as possible, we elected in the present analysis to evaluate only superiority trials in 1 medical specialty that were published in high-impact journals with rigorous review, and we further stratified citation rates according to several important trial characteristics. We also found differential effects on citations according to funding source in 3 sets of side-by-side publications that dealt with similar topics.
We suspect that there are several mechanisms through which for-profit organizations are able to increase dissemination of study results that are not available to not-for-profit organizations. First, study results may be selectively propagated to thought leaders through the dense network of sales representatives who are in regular contact with academic physicians.19 Similarly, dissemination of favorable study results during sponsored sessions at scientific meetings may selectively increase knowledge among thought leaders of certain publications at the expense of others. Second, for-profit organizations have a greater ability to ensure media coverage of their research, which has been shown to increase the citation rate by up to 73%.20 Third, for-profit organizations are likely to invest greater resources to increase the number of secondary publications for a given trial, further increasing exposure for the primary article.21 For-profit organizations also are more likely to support favorable review articles, journal supplements, and promotional materials on an "author-for-hire" basis.22,23 Finally, it has been shown that financial relationships between industry and scientific investigators are associated with an increased likelihood of a pro-industry conclusion.24–26 These same financial ties may lead to selective citations of studies favoring for-profit organizations.
Our study may be limited by the fact that we were not able to exclude self-citations or citations that derived from secondary articles. However, when we repeated our analyses using an alternative citation system, similar results were obtained.
Although the present study provides evidence that trials funded by for-profit organizations are cited more often than trials funded by not-for-profit organizations, it is noteworthy that the single most frequently cited trial in our analysis was the National Institutes of Health–funded Women’s Health Initiative (WHI), which demonstrated that among healthy postmenopausal American women, hormone replacement therapy has more health risks than benefits.27 This report has been cited >700 times annually since its publication in 2002, a rate almost 60% higher than any industry-funded trial in our analysis. Thus, the WHI provides reassuring evidence that at least in some settings, not-for-profit findings can be widely disseminated. Unfortunately, the WHI was a major outlier in our data and may be an exception that in part proves the rule; in our survey, this not-for-profit trial was virtually unique in that its publication was accompanied by a substantial public-health educational campaign, akin to that more commonly observed for trials funded by for-profit entities.
|
Acknowledgments |
|---|
Sources of Funding
The authors received no direct funding to support the data collection, analysis, or presentation of this manuscript. Dr Conen is supported by grants from the Swiss National Science Foundation (PASMA 118586/1), the Janggen-Poehn Foundation, St Gallen, Switzerland, and the Stiftung für Gesundheit und kardio-/neurovaskuläre Forschung, Basel, Switzerland.
Disclosures
During the 5-year period covered in this survey, Dr Ridker received research funding support from multiple not-for-profit entities, including the American Heart Association; the Doris Duke Charitable Foundation; the Leducq Foundation; the National Cancer Institute; the National Heart, Lung, and Blood Institute; the Donald W. Reynolds Foundation; and the James and Polly Annenberg La Vea Charitable Trusts, as well as investigator-initiated research support from for-profit entities, including Abbott, AstraZeneca, Novartis, Pharmacia, Roche, and Sanofi-Aventis. Dr Ridker is listed as a coinventor on patents held by the Brigham and Women’s Hospital that relate to the use of inflammatory biomarkers in cardiovascular disease and has served as a consultant to AstraZeneca, Dade-Behring, Isis Pharmaceutical, Sanofi-Aventis, and Schering-Plough. None of the above entities played any role in the design, conduct, or interpretation of this study. Dr Conen and J. Torres report no conflicts.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
1. Ridker PM, Torres J. Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000–2005. JAMA. 2006; 295: 2270–2274.
2. Kalla K, Christ G, Karnik R, Malzer R, Norman G, Prachar H, Schreiber W, Unger G, Glogar HD, Kaff A, Laggner AN, Maurer G, Mlczoch J, Slany J, Weber HS, Huber K. Implementation of guidelines improves the standard of care: the Viennese registry on reperfusion strategies in ST-elevation myocardial infarction (Vienna STEMI registry). Circulation. 2006; 113: 2398–2405.
3. Garfield E. How can impact factors be improved? BMJ. 1996; 313: 411–413.
4. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002; 287: 2847–2850.
5. Kulkarni AV, Busse JW, Shams I. Characteristics associated with citation rate of the medical literature. PLoS ONE. 2007; 2: e403.[CrossRef]
6. Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA. 2005; 293: 2362–2366.
7. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006; 295: 90–93.
8. Leon MB, Teirstein PS, Moses JW, Tripuraneni P, Lansky AJ, Jani S, Wong SC, Fish D, Ellis S, Holmes DR, Kerieakes D, Kuntz RE. Localized intracoronary gamma-radiation therapy to inhibit the recurrence of restenosis after stenting. N Engl J Med. 2001; 344: 250–256.
9. Verin V, Popowski Y, de Bruyne B, Baumgart D, Sauerwein W, Lins M, Kovacs G, Thomas M, Calman F, Disco C, Serruys PW, Wijns W. Endoluminal beta-radiation therapy for the prevention of coronary restenosis after balloon angioplasty: the Dose-Finding Study Group. N Engl J Med. 2001; 344: 243–249.
10. Abraham WT, Fisher WG, Smith AL, Delurgio DB, Leon AR, Loh E, Kocovic DZ, Packer M, Clavell AL, Hayes DL, Ellestad M, Trupp RJ, Underwood J, Pickering F, Truex C, McAtee P, Messenger J. Cardiac resynchronization in chronic heart failure. N Engl J Med. 2002; 346: 1845–1853.
11. Lamas GA, Lee KL, Sweeney MO, Silverman R, Leon A, Yee R, Marinchak RA, Flaker G, Schron E, Orav EJ, Hellkamp AS, Greer S, McAnulty J, Ellenbogen K, Ehlert F, Freedman RA, Estes NA 3rd, Greenspon A, Goldman L. Ventricular pacing or dual-chamber pacing for sinus-node dysfunction. N Engl J Med. 2002; 346: 1854–1862.
12. Packer M, Coats AJ, Fowler MB, Katus HA, Krum H, Mohacsi P, Rouleau JL, Tendera M, Castaigne A, Roecker EB, Schultz MK, DeMets DL. Effect of carvedilol on survival in severe chronic heart failure. N Engl J Med. 2001; 344: 1651–1658.
13. Beta-Blocker Evaluation of Survival Trial Investigators. A trial of the beta-blocker bucindolol in patients with advanced chronic heart failure. N Engl J Med. 2001; 344: 1659–1667.
14. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet. 2004; 364: 911–912.[CrossRef][Medline] [Order article via Infotrieve]
15. Zarin DA, Tse T, Ide NC. Trial registration at ClinicalTrials.gov between May and October 2005. N Engl J Med. 2005; 353: 2779–2787.
16. Steinbrook R. Public access to NIH-funded research. N Engl J Med. 2005; 352: 1739–1741.
17. Eysenbach G. Citation advantage of open access articles. PLoS Biol. 2006; 4: e157.[CrossRef][Medline] [Order article via Infotrieve]
18. Patsopoulos NA, Ioannidis JP, Analatos AA. Origin and funding of the most frequently cited papers in medicine: database analysis. BMJ. 2006; 332: 1061–1064.
19. Campbell EG, Gruen RL, Mountford J, Miller LG, Cleary PD, Blumenthal D. A national survey of physician-industry relationships. N Engl J Med. 2007; 356: 1742–1750.
20. Phillips DP, Kanter EJ, Bednarczyk B, Tastad PL. Importance of the lay press in the transmission of medical knowledge to the scientific community. N Engl J Med. 1991; 325: 1180–1183.[Abstract]
21. Ross JS, Hill KP, Egilman DS, Krumholz HM. Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA. 2008; 299: 1800–1812.
22. Bero LA, Galbraith A, Rennie D. The publication of sponsored symposiums in medical journals. N Engl J Med. 1992; 327: 1135–1140.[Abstract]
23. Rochon PA, Gurwitz JH, Cheung CM, Hayes JA, Chalmers TC. Evaluating the quality of articles published in journal supplements compared with the quality of those published in the parent journal. JAMA. 1994; 272: 108–113.
24. Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003; 289: 454–465.
25. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Med. 2007; 4: e5.[CrossRef][Medline] [Order article via Infotrieve]
26. Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007; 335: 1202–1205.
27. Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, Jackson RD, Beresford SA, Howard BV, Johnson KC, Kotchen JM, Ockene J. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial. JAMA. 2002; 288: 321–333.
|
Footnotes |
|---|
Guest Editor for this article was Steven E. Nissen, MD.
Related Article:
Circulation 2008 118: 1307-1308.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||