Policy Research

Descriptive Health Policy Research

The most frequently conducted type of health policy research seeks to describe or characterize health and healthcare delivery across an entire population or system of care or within subpopulations (the uninsured or Californians) or subsectors of care (safety-net hospitals or outpatient clinics). This type of policy research is critical in the evaluation of the healthcare delivery system, examining how the system is functioning for patients, clinicians, or hospitals and identifying problems or opportunities to improve healthcare delivery.

The Cooperative Cardiovascular Project (CCP) provides an excellent example of descriptive health policy research because it was undertaken to identify and characterize problems in healthcare delivery. In the early 1990s, the Health Care Financing Administration (known today as the Centers for Medicare and Medicaid Services) transitioned from being a large payer of healthcare for older Americans to an insurer also focused on quality of care. Accordingly, it began a series of quality initiatives to characterize the state of quality of care for all Medicare beneficiaries. The CCP focused on the treatment of patients with acute myocardial infarction (AMI), first convening a steering committee to draft quality indicators (measures of processes and outcomes of care) heavily based on clinical practice guidelines, then collecting information on all Medicare beneficiaries admitted to all US hospitals with a principal diagnosis of AMI over an 8-month period between 1994 and 1996.^3,4 The CCP examined 10 quality indicators, including 3 outcome measures (inpatient, 30-day, and 1-year mortality) and 7 process measures with varying eligibility criteria (reperfusion within 12 hours of admission, aspirin on admission and discharge, β-blocker and angiotensin-converting enzyme inhibitors on discharge, avoidance of calcium channel blockers, and tobacco cessation counseling) and demonstrated substantial underutilization of high-quality care; only 67% of eligible patients underwent reperfusion during hospitalization, only 50% received β-blocker therapy at discharge, only 59% received angiotensin-converting enzyme inhibitors at discharge, and only 42% received tobacco cessation counseling.^5,6 Moreover, there was wide geographic variation in healthcare quality, such that, as an example, in some regions, 100% of patients received an aspirin on admission, whereas only 68% did in others.^5,6

Descriptive health policy research frequently makes use of publicly available data to characterize health and healthcare delivery, often collected through US government–sponsored programs, such as the National Center for Health Statistics.⁷ For instance, the Behavioral Risk Factor Surveillance System (collected by the Centers for Disease Control and Prevention) was used to describe healthcare utilization among adults with and without health insurance, finding the uninsured and those with lower incomes to be less likely to receive recommended preventive and chronic disease care, including cholesterol and hypertension screening, cholesterol monitoring among patients with hypertension or cardiovascular disease, and weight loss and smoking cessation advice from a physician.^8,9 In addition, the Medical Expenditure Panel Survey (collected by the Agency for Healthcare Research and Quality) was used to describe how the financial burden from healthcare has changed in the past decade, finding that people with chronic diseases, such as cardiovascular disease and hypertension, are at greater risk of incurring large healthcare costs.^10,11

It is important to distinguish descriptive policy research from clinical effectiveness studies. Clinical effectiveness research seeks to determine the relative risks and benefits of different treatment strategies in routine clinical practice. For instance, for the treatment of AMI, substantial amounts of research have been devoted to determining whether a medication, such as an aspirin, improves patient outcomes. In other words, an outcome, such as mortality, is compared between 2 groups: 1 receiving aspirin after AMI, another receiving placebo (a comparison that when studied found 36 fewer deaths, AMIs, and strokes per 1000 aspirin users over 2 years¹²). However, this and most other effectiveness studies are clinical research investigations. In contrast, “descriptive” policy research examining the treatment of AMI would focus on whether aspirin is being appropriately and widely used, that is, aspirin use after AMI might be compared among men and women (a comparison that when studied has consistently found lower rates of aspirin use among women^13,14) or among patients receiving care at nonprofit and for-profit hospitals (a comparison that when studied found nonprofit hospitals discharging post-AMI patients with aspirin at higher rates¹⁵). Importantly, there are a number of policy-level solutions that could improve the frequency and appropriateness of use of aspirin for AMI or use of another medication for another disease, including changing reimbursement schemes or providing performance feedback.

Analytical Health Policy Research

Analytical health policy research focuses on the relationship between existing health policies and healthcare delivery. These studies, which also generally involve using large samples of healthcare utilization data, allow “big-picture” hypotheses to be examined and are often cautiously interpreted, providing a starting point for future study through experimental research (or impetus for policy change). In this way, analytical research offers important initial opportunities to explore healthcare delivery and outcomes in the context of existing health policy (or within existing health systems), functioning as a stepping stone of sorts between the first and third broad categories of policy research.

Studies examining the effectiveness of Medicare insurance provide a good example of analytical health policy research because these studies attempted to characterize healthcare utilization among all Medicare beneficiaries receiving care governed by the same health policies. For instance, several studies demonstrated that important preventive and chronic care services that were poorly reimbursed, including cholesterol and breast and colorectal cancer screening and management of hypertension and cardiovascular disease,^16–18 were underused among Medicare beneficiaries, potentially resulting in poorer health outcomes. These findings stimulated policy changes, including the Medicare Cholesterol Screening Coverage Act of 2003 to guarantee Medicare coverage of preventive cardiovascular screenings as well as a provision in the Balanced Budget Act of 1997 to ensure Medicare reimbursement for colon cancer screening procedures (the colon cancer screening provision was subsequently found to be associated with an increased use of colonoscopy among Medicare beneficiaries and an increased probability of early-stage diagnosis¹⁹).

Research examining geographic variation in healthcare utilization and medical spending offers another example of analytical health policy research. Medicare has long provided fixed fee-for-service payments for treatment of its beneficiaries for both hospital and physician services, with modest regional fee variation based on regional price level. Despite this policy, 2 decades of research have documented persistent and substantial geographic variations in use of services, generally focused on rates of admission to the hospital and of procedures, such as hysterectomy or coronary artery bypass surgery, that are not accounted for by variations in price or patient illness.^20–24 Health policies have been proposed in an attempt to reduce variation in healthcare delivery, particularly because higher spending does not result in more effective care or better health outcomes,^20,25 although none have been enacted.

Experimental Health Policy Research

Experimental health policy research focuses on examining how policy changes might affect healthcare delivery. There are 2 broad approaches to such research, experimental and quasi-experimental, the latter of which make use of observational methods. However, this type of policy research is challenging because it is costly to conduct experimental studies and quite difficult to use observational methods to isolate the specific effect of changing a policy on healthcare delivery.

Study Population

When policy research is evaluated, it is important to consider context, such as the size and diversity of the study population, as well as the health system in which the patients are receiving care. Findings from 1 healthcare system (ie, insurance plan or hospital) or location (ie, country, state, city, or smaller) may not be generalizable to another healthcare system or location. In other words, narrowly focused study populations are unlikely to have strong external validity with regard to extrapolating findings to a broader population. Ideally, data used for policy research should reflect as large and diverse a segment of the population as possible. Hence, population-based data are ideal.

Nevertheless, studies focused on specific population subgroups, such as members of a large health plan or even participants in a particular clinical trial, might still yield valuable insights. Investigators are increasingly making use of existing clinical trial data to address topics that are distinct from the primary study outcome, such as patterns of care and costs. For instance, data from the Global Utilization of Streptokinase and t-PA for Occluded Coronary Arteries (GUSTO) trial have generated numerous studies relevant to the care and outcomes of patients with AMI. Reports based on GUSTO data suggested important sex-based disparities in angiography use and regional variation in β-blocker use.^55,56 A key question is whether these reports were idiosyncratic findings particular to the GUSTO trial participants or an accurate representation of the larger national healthcare system. To address this concern, investigators compared findings from these GUSTO reports to study results based on data collected through the CCP (discussed above). Because the CCP involved a comprehensive and systematic record review of Medicare patients with AMI at the national level, CCP data are thought to be generalizable to all older adults. When comparing study results derived from each of these 2 databases, the authors found that the degree of variability in β-blocker use (across geographic regions) and angiography use (between sexes) was remarkably similar when studied with the use of either CCP or GUSTO data.^5,55–58 As expected, the overall rates of death and treatment-related complication were higher in the GUSTO patients. These findings suggest that large data repositories, such as data collected within clinical trials, may provide valuable and accurate assessments of health and healthcare delivery for policy research. Nevertheless, care must be taken to assess the generalizability of the data source,⁵⁹ considering issues such as patient selection and the degree to which trial participants differ from other community-based patients with regard to age, sex, health status, comorbidities, or healthcare access.^60–65

Study Design

Policy research study designs should attempt to maximize internal validity by minimizing bias and confounding. Randomized controlled trial evidence is rare in policy research, due to the challenge of funding such studies, as well as the complexity involved in randomizing people to blindly receive care within differing healthcare delivery systems or under differing policies.

Commonly, observational studies using routinely or previously collected data are used for policy research and therefore to guide policy. However, the observational nature of most health policy research should instill caution when interpreting findings, recognizing that cross-sectional studies do not (and cannot) imply causation. Nevertheless, well-designed observational research can provide evidence as informative and accurate as that provided by randomized controlled trials.⁶⁶

In addition, some policies are difficult to evaluate by either experimental or observational methods because they are implemented at an organizational level or within a geographic area, necessitating the use of other methodologies. For instance, door-to-balloon time, in reference to providing percutaneous coronary intervention within 120 minutes to patients with ST-elevation AMI, is a quality indicator reported by the Health Quality Alliance, and hospital performance on this measure has historically lagged behind performance on other measures. Qualitative research methods identified common organizational approaches among higher-performing hospitals,^67,68 which were subsequently tested in a cross-sectional, national study.⁶⁹

Outcomes

Good policy research examines outcomes that are important to patients, providers, and payers. Preferred outcomes include patient quality of life, satisfaction, mortality, re/hospitalization, and disability, in other words, outcomes that patients and policy makers can identify and make reasonable determinations of value and importance. Outcomes that provide scientific value but are less important to patients (and clinicians and policy makers) include laboratory measurements, diagnostic test findings, or other surrogate markers of disease. This is particularly important because surrogate outcomes do not always reflect central outcomes of interest. For instance, the recent ILLUMINATE trial of the pharmacological agent torcetrapib found that it increased levels of high-density lipoprotein cholesterol, a surrogate marker for better health outcomes, but increased deaths.⁷⁰ This same counterintuitive relationship has been observed in quality of care research as well because publicly reported AMI process measures were found to capture only a small proportion of the variation in hospitals’ risk-standardized short-term mortality rates.⁷¹

Policy research that examines costs must be evaluated by which costs were considered, how they were calculated, and from whose perspective. All outcomes should be considered in the context of their value, utility, and tradeoffs. For instance, studies of pay-for-performance programs tend to focus on whether incentivized processes of care are performed at higher rates. However, these studies would provide more information if they also evaluated the costs required to increase performance rates, in fiscal terms and in clinical terms, such as whether high-quality processes of care that are not being measured are “crowded out” by measured care.⁷²

Statistical Methods

Statistical approaches used within policy research will vary in order to deal with missing variables, account for nonindependence of outcomes, and minimize bias. Missing data are a common problem, especially when large, administrative data sources are used. Ignoring missing data may lead to inefficiency or, worse, bias, although a variety of methods exist for addressing this problem.⁷³

In addition, large data sources often have a hierarchical structure, in which there may be multiple levels of nested observations that are therefore not independent. For example, some hospitals or physicians may be more likely to care for patients with specific characteristics. This structure can be accounted for with the use of hierarchical regression modeling to account for patients nested within clinicians nested within hospitals nested within cities. If studies do not account for the nonindependence of these observations, results may be biased.⁷⁴

Standard statistical methods are likely to produce unbiased findings when the distribution of unmeasured factors is similar between the groups being compared, such as after randomization. Observational studies are more prone to bias if groups differ in unmeasurable ways from one another and should utilize multivariable adjustment for measured variables that are different between the groups being compared. Both instrumental variable analysis and propensity score analysis are methods for inferring the effect of policy on a range of potential outcomes in situations in which randomization is unethical or impractical and strong selection bias exists. Propensity score analysis involves matching individuals from each group to address overt bias, whereas an instrumental variable is an observable variable that can be used in lieu of a coin flip in assigning patients to the policy of research interest. This type of analysis minimizes the confounding that complicates observational studies without random assignment because unobserved characteristics of patients and clinicians may influence outcomes and whether or not patients receive care under an examined policy.⁷⁵ As an example, recent systematic reviews of randomized controlled trials of routine invasive versus conservative management of AMI have found that between 8% and 21% improved relative survival in the more invasively treated groups,^76,77 but a population-based observational study found little benefit to invasive therapy in US regions in which medical management was of higher quality.⁷⁸ Attempting to reconcile these contrasting findings, subsequent research determined that the estimated effect was sensitive to the analytical method used, in that the estimated benefit from invasive therapy depended on the statistical method used to adjust for overt (measured) and hidden (unmeasured) bias. In this study, invasive therapy predicted a 50% relative decrease in mortality with the use of standard risk adjustment methods, including a rigorous propensity-based matching analysis, but only a 16% relative decrease in mortality with the use of instrumental variable methods.⁷⁹ Hence, estimates of relative treatment effectiveness varied by a factor of 3, depending on which analytical technique is used, underscoring the importance of the statistical method.

Data Access

Data Quality

At the core, the quality of health policy research is determined by the quality of the data. Many databases are not collected specifically for the purpose of health policy research, particularly nationally representative data used to derive population estimates. Policy research typically examines complex organizational structures with the use of the best available data. Often, key information is not collected, limiting the potential insight from the work. In addition, increasing amounts of data will be available that are generated from health information technology. Managing these data to ensure that meaningful, well-designed studies are conducted, rather than studies that create ever more “noise” in which to lose informational signals, is paramount.

Funding

Perhaps the greatest challenge facing health policy research is its low prioritization within state and federal budgets. The total federal funding for health services research in fiscal year 2006 was ≈$1.5 billion,⁸³ principally funded by the National Institutes of Health ($938 million), the Agency for Healthcare Research and Quality ($319 million), and the Centers for Disease Control ($140 million). Yet this is <5% of a total federal research budget that exceeds $35 billion,⁸⁴ the vast majority of which was used to support the conduct of biomedical (75%) and clinical research (20%). The Agency for Healthcare Research and Quality, a division within the Department of Health and Human Services (not within the National Institutes of Health), is the natural funding home for health policy research. But the budget of the Agency for Healthcare Research and Quality, which has not been increased in 4 years,⁸³ allows for little discretionary funding because it is nearly completely preappropriated to specific research projects related to health information technology, patient safety, and comparative effectiveness.

Moreover, because the purpose of health policy research is to inform future health policy and improve (ideally) future health and healthcare delivery, payers should also be expected to invest in and prioritize research. However, the Centers for Medicare and Medicaid Services, the largest insurance plan in the United States, spent only ≈$58 million for research in fiscal year 2006,⁸³ a substantial decrease from previous years and a fraction of the total Centers for Medicare and Medicaid Services budget that exceeds $545 billion.⁸⁵ The Department of Veterans Affairs, which provides medical care for veterans, invested more heavily in research, ≈$412 million in fiscal year 2006,⁸³ but this remains a small proportion of a total Veterans Affairs budget that currently exceeds $30 billion.⁸⁶ Amounts invested in policy research by private payers are unknown. Unless health policy research is sufficiently prioritized and sufficiently funded, its greater impact on health and healthcare will remain below its potential.

Conflicts of Interest

All types of research can be threatened by conflicts of interest, and health policy research is no exception. A conflict of interest occurs when a secondary interest of any type can affect judgment about a primary interest, such as a research study.⁸⁷ Financial conflicts of interest have been associated with lapses in research integrity within clinical and cost-effectiveness research.^88–91 Financial conflicts of interest have not been as thoroughly explored within health policy research.

Although the broad ethical issues raised by conflicts of interest apply to all settings, health policy research is distinct in some respects. In clinical research, investigators may hold patents, stock, or other investments that may increase substantially in value if a trial yields favorable results. Because of the nature of health services research, this type of financial arrangement is much less likely. Rather, financial ties may be more aligned either with the researcher’s ability to solicit future funding from the sponsor or with the sponsoring organization’s financial security.⁹² It is also important to consider nonfinancial conflicts because the close relationship between health policy researchers and policy makers may lead to policy agendas affecting researchers’ objectivity. Above all, health policy research should be planned and conducted within a framework that supports investigator independence and ensures objectivity and transparency in the planning and conduct of research.⁹²