Results-Based Accountability for Cardiovascular Epidemiology Research by the National Heart, Lung, and Blood Institute
The US National Institutes of Health (NIH) has been at the forefront of advances in biomedical research for over a century. It comprises 27 institutes and centers, including the National Heart, Lung, and Blood Institute (NHLBI), each with its own specific research agenda and priorities. Collectively, NIH funding has contributed to many scientific and technological breakthroughs that have resulted in substantial gains in population health and longevity, both in the United States and globally. Moreover, there are numerous examples of investments in NIH research that yielded returns extending well beyond health benefits to encompass social and economic ramifications. For instance, it has been estimated that the $3.8 billion NIH contribution to the Human Genome Project generated $1 trillion in economic growth, corresponding to a 178-fold return on initial investment.1
Article see p 2028
Nevertheless, the present challenging economic landscape has compromised the NIH’s capacity to maintain its long-established role as the largest funding agency for medical research in the world. The NIH budget grew steadily in the 1990s, peaked at $35.6 billion in 2004, and then declined by ≈2% per year in real terms thereafter (Figure). In contrast, the proportion of medical research funded by industry in the United States increased from 46% in 1994 to 58% in 2012.2 From an international perspective, the annual compound growth rate in medical research from 2004 to 2011 was slowest in the United States (1%) in comparison with, for example, 16.9% in China, 9.3% in Australia, 6.8% in Japan, 4.5% in Canada, 4.1% in Europe, and 3.5% globally.2 Other metrics, such as patent statistics and new drugs and devices, likewise suggest that the United States is at risk of relinquishing its global leadership in health research if current trends are not reversed.3
During these times of tight control over public spending, countries have espoused various strategies to optimize the allocation of research funds. The United Kingdom’s Department for Business Innovation and Skills protects research funding in cash terms and has adopted 9 criteria to guide the distribution of funds, favoring centers of proven excellence, research that attracts leveraged funding, and strategies that support economic growth.4 The Singapore government invests 1% of the national gross domestic product on research and development, with emphasis on basic science, attraction of talent, competitive funding, synergy across public and private sectors, commercialization, and economic benefits.5 The Japanese government’s long-term strategy favors funding innovation and research toward achieving a healthy and active aging society, while prioritizing the commercialization of outcomes.6 The European Union is allocating an unprecedented €80 billion over a 7-year period as part of its Horizon 2020 program, with the objective of maintaining an active and independent lifestyle in the elderly, supporting safer and more effective interventions, and sustaining healthcare systems.7
All large-scale funding programs are ultimately subject to accountability, surveillance, and reassessment. They may monitor, for example, to what extent investments appropriately reflect societal needs and specific priorities. Moreover, they may attempt to assess the quality and quantify the impact of the research produced. Within this context, the article by Fabsitz et al8 in this issue of Circulation provides an excellent example of a careful self-examination by a prominent funding organization committed to results-based accountability. Conceptually, this approach is intended to provide evidence-based data to improve program performance and guide strategic decision making.9
The National Heart Institute was established in 1948. It expanded its purview in 1972 as the NHLBI to incorporate lung and blood vessel diseases and to include a cardiovascular epidemiology branch.10 Epidemiological research is observational in nature and, hence, inherently subject to selection and information biases and numerous potential confounders. In contrast to randomized clinical trials, which are rightfully considered the methodological gold standard to determine treatment efficacy, epidemiological studies are most often hypothesis generating. They seldom directly and immediately alter clinical practice. By their nature, epidemiological investigations typically include large numbers of subjects, incur substantial costs, require long follow-up periods to capture outcomes, and necessitate elaborate resources, with sizeable interdisciplinary teams. Moreover, they might (like all research) produce results that are redundant, spurious, or, at worst, downright misleading. Some critics have therefore argued that in this era of unparalleled budget constraints, cardiovascular epidemiological research should be assigned a low priority.11
Fabsitz et al have undertaken an ambitious task: establishing the impact of NIH-funded cardiovascular epidemiology research. The authors approached this objective by quantifying the performance of projects and scientists supported by the cardiovascular epidemiology branch of the NHLBI during a 15-year period (1998–2012), focusing on key performance indices based on annual citation metrics.8 More specifically, they analyzed the top 100 cardiovascular epidemiology publications worldwide across 10 research domains, and identified those supported by the NHLBI. They assessed changing trends over successive 5-year intervals and determined the proportion of cardiovascular epidemiology studies supported by the NHLBI that were published in the most represented high-impact journals. The findings are staggering. Overall, the NHLBI supported ≈25% of the most highly cited cardiovascular epidemiology studies worldwide. Significant increases over time were observed in the proportion of NHLBI-supported high-impact publications related to genetics and quality-of-care/clinical epidemiology domains. The proportion of publications with NHLBI support was even higher among the 5 most represented journals, surpassing 40% in several domains.
Quantifying performance and returns on investments for research-funding organizations is undeniably challenging. A handful of metrics cannot realistically tell the whole story. Prominent scientometricians have cautioned against assessing performance on the basis of bibliometrics alone, such as number of publications overall and in high-impact journals, citations, and impact factors. They contend that doing so compromises innovation, represents an extrapolation of metrics to applications for which they were not intended, and may distort reality.12,13 Citation counts, for example, do not necessarily reflect the quality of a publication. Rather, they are indicative of the attention the publication received by scientists who may cite articles for various reasons, including to dispute or critique findings. For example, the controversial study linking the measles, mumps, and rubella vaccine to autism,14 which was later retracted on the basis of false claims, was cited an average of 53.5 times per year from 1998 to 2015, according to Web of Science (Thomson Reuters). Furthermore, citations do not reveal the influence of a publication beyond academic circles, such as on policy makers, clinical practice, or the economy. Self-citations may inflate statistics and annual citation counts favor end-year studies, because citation rates generally decrease over time.
Notwithstanding such reservations, and when considered within the context of a more global assessment strategy, performance metrics can provide critical information to supplement qualitative analyses in guiding decision making. In assessing the impact of NHLBI-supported cardiovascular epidemiology research, Fabsitz et al8 have produced a thoughtful and reasoned analysis that respects the key tenets of results-based accountability, including transparency and simplicity. Objectives were clearly stated and detailed methodology was provided to enable reproducibility. Importantly, citation rates, which vary substantially across disciplines, were subjected to the most robust normalization method, ie, percentiles. Each publication was weighted according to its rank within 1 of 10 research domains in cardiovascular epidemiology. In addition, self-citations were taken into consideration.
A research project that assesses other research projects is itself observational in nature and, hence, subject to associated limitations. The impact of potential selection bias is difficult to ascertain. For example, it is theoretically possible, but far from certain, that NHLBI-supported studies are more likely to use the particular domain-specific search terms used to identify cardiovascular epidemiology research by virtue of the grant application process. Moreover, selection criteria to define NHLBI support were broadly defined to include, for example, coauthorship by an NHLBI employee or acknowledged assistance of NHLBI staff. Although such an approach maximizes sensitivity, the trade-off may be lower specificity.
From our qualitative perspective, overwhelming evidence suggests that the cardiovascular epidemiology branch of the NHLBI has supported revolutionary discoveries, which have contributed substantially to reducing morbidity and mortality related to cardiovascular disease worldwide. The Table summarizes 10 selected landmark cardiovascular epidemiology studies supported by the NHLBI. Together, they enrolled >100 000 subjects and have generated >7300 publications. These studies answered key questions that informed and are complementary to, but cannot be addressed by, randomized clinical trials. They assessed common and rare exposures, directly measured risks, quantified population disease patterns and trends, identified physiological, environmental, lifestyle, and genetic determinants of disease development and progression, and evaluated multiple cardiovascular outcomes. In so doing, they addressed large care-limiting knowledge gaps, paved the way for risk stratification schemes that guide clinical decision making, spawned an era of personalized medicine, and set the stage for definitive randomized clinical trials.
We fully support the NHLBI’s efforts to critically appraise its programs, continuously strive to improve performance, and direct resources toward innovative research that yields the greatest scientific, societal, and economic returns. In competing for increasingly constrained budgets, the field of cardiovascular epidemiology is challenged to modernize and streamline its approaches to maximize the generation of novel data, prioritize the most meritorious research questions, ensure affordability, and improve efficiency in responding to changing realities and new opportunities.10 The study by Fabsitz et al8 provides empirical evidence to complement qualitative assessments of performance in defending the NHLBI’s track record for identifying and supporting impactful cardiovascular epidemiology research. It is our hope that these types of thoughtful performance analyses will help inform the future direction of the NHLBI and ensure continued prioritization of investments in trailblazing cardiovascular epidemiology research.
Sources of Funding
Dr Nattel's research is funded by the Canadian Institutes of Health Research and the Heart and Stroke Foundation of Canada.
Dr Khairy participated on an NHLBI working group to establish research priorities in adult congenital heart disease and is a coapplicant on several NIH grants. Dr Nattel has no conflicts to declare.
The opinions in this article are not necessarily those of the editors or of the American Heart Association.
- © 2015 American Heart Association, Inc.
- 4.↵The Allocation of Science and Research Funding 2015/16: Investing in World-Class Science and Research. London, UK: Department for Business Innovation and Skills, Crown copyright; 2014.
- 5.↵Singapore Ministry of Trade and Industry. Research, Innovation and Enterprise 2015. Singapore: RIE Secretariat; 2015.
- 6.↵Japanese Council for Science, Technology and Innovation. Comprehensive Strategy on Science, Technology and Innovation 2014 – Bridge of Innovation toward Creating the Future. Cabinet Office: Japan; 2014.
- Galsworthy M,
- McKee M
- Fabsitz RR,
- Papanicolaou GJ,
- Sholinsky P,
- Coady SA,
- Jaquish CE,
- Nelson CR,
- Olson JL,
- Puggal MA,
- Purkiser KL,
- Srinivas PR,
- Wei GS,
- Wolz M,
- Sorlie PD
- Lauer MS
- Sorlie PD,
- Bild DE,
- Lauer MS
- Rosbash M
- Wakefield AJ,
- Murch SH,
- Anthony A,
- Linnell J,
- Casson DM,
- Malik M,
- Berelowitz M,
- Dhillon AP,
- Thomson MA,
- Harvey P,
- Valentine A,
- Davies SE,
- Walker-Smith JA