doi: 10.1161/CIRCULATIONAHA.107.742775
(Circulation. 2009;119:1442-1452.)
© 2009 American Heart Association, Inc.
Key Issues in Outcomes Research |
Qualitative and Mixed Methods Provide Unique Contributions to Outcomes Research
From the Robert Wood Johnson Clinical Scholars Program, Department of Internal Medicine, (L.A.C., E.H.B.), and Section of Health Policy and Administration (L.A.C., I.M.N., E.H.B.), Department of Epidemiology and Public Health at Yale University School of Medicine, New Haven, Conn.
Correspondence to Leslie Curry, Yale School of Public Health, 60 College St, PO Box 208034, New Haven, CT 06520-8034. E-mail leslie.curry{at}yale.edu
Key Words: outcomes research • qualitative research • policy
 |
Introduction |
|---|
 Top Introduction Defining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Outcomes research examines the effects of medical care interventions and policies on the health outcomes of individuals and society.1 Investigators conducting outcomes research seek to inform the development of clinical practice guidelines, to evaluate the quality of medical care, and to foster effective interventions to improve the quality of care.2 Outcomes research has traditionally used quantitative sciences to examine the utilization, cost, and clinical effectiveness of medical care through randomized and nonrandomized experimental designs. Quantitative methods are not as well suited to measure other complex aspects of the healthcare delivery system, such as organizational change, clinical leadership in implementing evidence-based guidelines, and patient perceptions of quality of care, which are also critical issues in outcomes research.3–7 These more nuanced aspects of healthcare delivery may be most appropriately examined with qualitative research methods.8–10
Qualitative approaches are becoming more common in clinical medicine and health services research.5,11–15 Federal encouragement of qualitative research is regularly reflected in funding program announcements issued by the National Institutes of Health.16 For more than a decade, federal agencies and foundations such as the National Science Foundation have demonstrated a commitment to supporting qualitative research through funding scientific conferences, workshops, and monographs on this field of inquiry.17–20 Despite this steady growth in qualitative research, outcomes investigators in cardiology have relatively little guidance on when and how best to implement these methods in their investigations.
The purpose of the present report is to introduce qualitative methods as providing unique and critical contributions to outcomes research. This report will describe the situations in which qualitative approaches are most helpful; summarize the primary principles and practices in study design, sampling, data collection, and data analysis for qualitative studies; present representative examples of cardiovascular outcomes research that uses qualitative methods; and synthesize current standards for ensuring rigor and enhancing credibility of qualitative research.
|
Defining Qualitative Research |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Qualitative research is a form of scientific inquiry that spans different disciplines, fields, and subject matter and comprises many varied approaches.21 Qualitative methods can be used to understand complex social processes, to capture essential aspects of a phenomenon from the perspective of study participants,5 and to uncover beliefs, values, and motivations that underlie individual health behaviors.22–24 Such research can also illuminate aspects of organizational context and healthcare delivery that influence organizational performance and quality of care.10,25 Qualitative studies are often exploratory in nature and seek to generate novel insights9,23,26,27 using inductive (starting with observations and developing hypotheses) rather than deductive (starting with extant hypotheses and testing them with observations) approaches.
Qualitative research can be distinguished from quantitative research in several ways. First, whereas quantitative research counts occurrences (eg, estimates prevalence, frequency, magnitude, incidence), qualitative research describes the complexity, breadth, or range of occurrences or phenomena. Second, whereas quantitative research seeks to statistically test hypotheses, qualitative research seeks to generate hypotheses about a phenomenon, its precursors, and its consequences. Third, quantitative research is performed in randomized or nonrandomized experimental and natural settings and generates numeric data through standardized processes and instruments with predetermined response categories. Qualitative research occurs in natural (rather than experimental) settings and produces text-based data through open-ended discussions and observations.
Mixed methods, in which quantitative and qualitative methods are combined, are increasingly recognized as valuable, because they can capitalize on the respective strengths of each approach.28 Pairing quantitative and qualitative components of a larger study can achieve various aims, including corroborating findings, generating more complete data, and using results from 1 method to enhance insights attained with the complementary method.29,30 Approaches to mixed-methods studies differ on the basis of the sequence in which the components occur and the emphasis given to each.29,30 The qualitative and quantitative components may be performed concurrently or sequentially, and emphasis may be placed on either component or equal weight given to both. For instance, a preliminary qualitative component may serve to generate hypotheses or to develop the content for a questionnaire to be used in a follow-up quantitative study. Conversely, a preliminary quantitative component may generate surprising or inconsistent findings that may be examined in greater depth with a follow-up qualitative component. Strategies to enhance the validity of mixed-methods studies include recognizing the role of the complementary strategy and adhering to the methodological assumptions of each method.31 Principles and practices specific to mixed-methods research have been described extensively28,29; current applications are reported in the recently established Journal of Mixed Methods.32 Although the present report focuses primarily on qualitative research, illustrative examples are also provided from mixed-methods studies to highlight the full range of potential applications.
|
When to Consider Using Qualitative Methods |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Qualitative methods should be considered when the research aim is 1 or more of the following: to investigate complex phenomena that are difficult to measure quantitatively, to generate data necessary for a comprehensive understanding of a problem, to gain insights into potential causal mechanisms, to develop sound quantitative measurement processes or instruments, or to study special populations (Table 1).
|
Investigating Complex Phenomena That Are Difficult to Measure Quantitatively
Complex phenomena such as organizational processes, change processes over time, and social interactions underlying specific outcomes may be difficult to measure quantitatively.9 Qualitative methods can be helpful in identifying and characterizing multifaceted organizational dynamics that can influence outcomes, including organizational culture, clinical leadership, and team-based interactions.10 For instance, 1 large qualitative study examined the process of implementing an innovative technology for cardiac surgery in hospitals with top-tier cardiac surgery departments. The researchers conducted interviews across 16 hospitals with an average of 10 key staff per site to characterize the changes in organizational and group routines triggered by technology adoption. The findings generated a process theory of how collective learning within teams influences the success of technology adoption in hospital settings.33
Generating Data Necessary for a Comprehensive Understanding of the Problem
In cases in which quantitative data alone will not fully address the research question, qualitative approaches can be useful. A qualitative component can provide detailed perspectives of individuals or descriptions of processes, thereby ensuring a more comprehensive understanding of the phenomenon of interest. For instance, quantitative evidence regarding self-care practices of patients with congestive heart failure indicated interventions designed to enhance self-care have had modest success, which suggests a need to improve understanding of potentially modifiable patient-level factors. Horowitz and colleagues34 illuminated the dynamics of self-regulation processes in patients with congestive heart failure using qualitative methods. The researchers interviewed 19 congestive heart failure patients to characterize these processes and found that patients had limited knowledge of congestive heart failure, failed to recognize and act on symptom exacerbation, and encountered a variety of barriers to care. Findings were used to modify an existing model of self-care to fit the unique experiences of patients with congestive heart failure.
Gaining Insights Into Potential Causal Mechanisms
Qualitative research can also illuminate the potential causal mechanisms that are associated with a given outcome and generate hypotheses about such mechanisms. A qualitative approach can be useful when researchers are interested in looking beyond identified variables that are statistically linked with a desired effect to understand why a given intervention has a specific impact, how the impact occurs, and in what organizational context. Exploration of the causal mechanisms underlying a statistical association may be performed either before or after the quantitative association has been found. For instance, Bradley and colleagues used qualitative methods to characterize potential causal factors that influence increased β-blocker use after acute myocardial infarction,35 as well as reduced time to treatment for patients with ST-segment elevation myocardial infarction.36,37 These studies set a foundation for subsequent quantitative research to determine the statistical associations between qualitatively identified factors and outcomes in broader samples.
Developing Sound Quantitative Measurement Processes or Instruments
Qualitative methods can be useful in the development of patient-centered quantitative instruments for outcomes research. For example, qualitative methods have been used in the early stages of developing patient-centered quantitative measures of health-related constructs.38 Kressin and colleagues39 used this approach in developing a psychometrically sound, patient-centered survey instrument measuring health-related beliefs, attitudes, and experiences of cardiac patients in the Veterans Administration system. In the first phase, qualitative data were gathered from 13 patients who had undergone cardiac stress testing with positive results to elicit patients’ perceptions of their interaction with providers, including communication of patient concerns and preferences.40 These findings subsequently were used in the construction and validation of a standardized questionnaire to assess patient beliefs, preferences, and experiences surrounding cardiac decision making.
Qualitative approaches are also important when there are concerns about cross-cultural equivalency of existing tools, because psychological principles and measures that have not been cross-validated with different populations may be based on researchers’ preconceived, and perhaps erroneous, assumptions.41 Qualitative studies can help researchers evaluate the saliency and acceptability of defined constructs, identify missing constructs, employ appropriate language in the tool, and create suitable response formats.
Studying Special Populations
Qualitative research can offer important benefits for studies that involve special populations, including those that are traditionally underrepresented in research and those with low literacy.42 Qualitative data collection methods, such as open-ended interviews, may be more effective and less intimidating than surveys for those who historically have been marginalized in research. Qualitative approaches can also be used to improve methods of recruitment and retention of underrepresented groups in health research. For instance, Koops and colleagues43 sought to address ethical challenges in conducting research into thrombolysis for acute ischemic stroke patients. The large clinical trials required to determine the potential benefits of this treatment have been constrained by inherent ethical barriers, including the known risk of early fatal intracranial hemorrhage and the patient’s potentially compromised ability to provide timely informed consent. The researchers explored views on informed consent for participation in clinical trials on stroke by gathering feedback on trial recruitment materials from 54 potential study participants. Their findings guided the development of the recruitment and informed consent procedures, which resulted in an ethically appropriate trial design.
|
Conducting Rigorous Qualitative Research |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
There is a substantial body of literature defining the principles and practices of scientifically sound qualitative and mixed methods in health services research.9,23,27,29,44–48 Qualitative methods involve the systematic collection, organization, and interpretation of data in accordance with rigorous and widely accepted techniques for research strategy, sampling, data collection, and analysis.
Research Strategy
Primary qualitative research strategies include grounded theory, ethnography, case study, and phenomenology.9,49 Each approach is uniquely suited for specific types of investigations, and the choice of design is determined by the overarching aim of the study.23,50,51
Grounded theory uses systematic procedures to generate theory or insights describing a phenomenon and is grounded in the views expressed by study participants.9,27,49 Based in sociology and inductive in nature, this approach involves an iterative process of data collection and analysis. Grounded theory relies on "theoretical sampling," in which the sample is developed and refined throughout the course of the study to explore insights that emerge from the data.27 Analysis in grounded theory studies is typically guided by the constant comparative method, in which verbatim quotations or observations are catalogued into their essential concepts by use of codes or labels developed iteratively to reflect the data.
Ethnography is a form of field research that seeks to learn the culture of a particular setting or environment.52 It often relies on participant observation through prolonged field work and may include other qualitative and quantitative methods.53 The researcher becomes embedded in ongoing relationships with research participants for the purpose of observing and recording talk and behavior. In such cases, the researcher (as opposed to, for instance, surveys or questionnaires) is the primary instrument for data collection and analysis.52,54 The researcher seeks to place specific events into a broader, more meaningful context, with a focus on the culture and social interaction of the observed people or groups.55 Ethnography is particularly valuable in understanding the influence of social and cultural norms on the effectiveness of health interventions.52,53
Case studies involve investigation of a specific, unique system with patterned behavior, dynamic properties, and defined features.56–58 Such designs may involve a single, in-depth case study or a set of several case studies, depending on the scope and goals of the research project. A case may be defined as a clinical case, critical incident, activity, process, family, organization, or system. A case study design is most useful when an individual case appears to produce new insights that might be transferred to a larger group of cases or activities. The aim of a case study may be exploratory (to define questions or hypotheses), descriptive (to depict a phenomenon within its context), or explanatory (to identify cause-and-effect relationships) in nature.59 The quality of a case study is enhanced by the use of diverse data sources, including documentation, archival records, interviews, direct observation, participant-observation notes, and physical artifacts.56
Phenomenology seeks to describe how individuals experience a specific phenomenon. Rooted in philosophy, this approach characterizes individuals’ lived experiences of a phenomenon (such as grief) through gathering extensive narrative data from a small number of participants. The goal is to generate a deeper understanding of the "essence" or meaning of a particular phenomenon from the individual’s perspective.9
An additional research approach that may use qualitative methods (independently or in combination with quantitative methods) is community-based participatory action research.60 This topic will be addressed in depth in a forthcoming article in this outcomes research series.
Sampling Strategy and Size
Systematic, scientifically sound methods for developing samples for qualitative and mixed-methods studies are well established.9,44,61 In contrast to quantitative sampling techniques that rely on statistical probability theory, qualitative sampling is based on purposeful or theoretical sampling principles. The aim is to identify "information-rich" participants who have certain characteristics, detailed knowledge, or direct experience relevant to the phenomenon of interest.26 A common approach known as purposeful sampling seeks to include the full spectrum of cases and reflect the diversity within a given population by including extreme or negative cases.9 The sample size varies depending on the breadth and complexity of the inquiry, although samples are generally smaller than those used in quantitative studies and are studied intensively. Adequacy of the sample size is determined by the principle of theoretical saturation. Theoretical saturation refers to the point at which no new concepts emerge from the review of successive data from a sample that is diverse in pertinent characteristics and experiences.27,45,62 Although it is not possible to define the number of participants in advance, a range of 20 to 30 interviews9 or 4 to 6 focus groups63 may achieve saturation.
Data Collection Methods
Primary methods of qualitative data collection include in-depth interviews, focus groups, observation, and document review (Table 2).
|
In-Depth Interviews
An in-depth interview is typically a 1-to-1 interaction between a researcher and a study participant. Interviews allow for the exploration of individual experiences and perceptions in great detail.9,54,64 Interviews are particularly useful when rapport between the researchers and respondents is required to ensure candor, or in instances when privacy may alleviate fear of reprisal for negative statements, such as program evaluation or patient satisfaction studies. Although interviews are typically conducted either in person or via telephone, electronic (online) formats are emerging as a method that is particularly useful for isolated, difficult-to-reach study participants.65
In-depth interviews (or guided conversations) allow the respondent to direct the course of discussion as much as possible.66 The format allows the respondent to identify and describe concerns or concepts that may not have been anticipated or considered by the researchers. Interviews are highly interactive. The interviewer aims to be responsive to the language and concepts used by the interviewee. Data collection instruments are often referred to as discussion guides and may be semistructured (using open-ended questions within a predetermined set of topics) or in-depth (1 or 2 very broad questions that explore an issue or experience in great detail) in design. In addition to the established questions, the interviewer uses prompts and probes to clarify concepts, elicit detail, and extend the narrative.67
These interview methods require highly experienced researchers with specific skills, including the ability to establish rapport with respondents, to use discussion guides flexibly, and to use probes and follow-up questions to draw out responses. Interviewers must be skilled in passive listening and in the use of neutral, nonjudgmental language in encouraging respondents to speak in detail. At the same time, interviewers must maintain control of the data gathering through vigilant attention to the purpose of the interview, asking the right questions to obtain relevant information, and giving appropriate verbal and nonverbal feedback.9 Staff of diverse backgrounds can conduct high-quality interviews if provided adequate training in these methods.
Focus Groups
Focus groups are guided discussions among a small group of people who share a common characteristic central to the topic of interest.46,63,68 The group interaction can serve as a catalyst to generate unique insights into understanding shared experiences and social norms. Focus groups are appropriate when the goal is to understand differences in perspectives between groups or categories of people or to uncover factors that influence opinions or behavior.68 They may be especially effective with socially marginalized populations and can be useful in facilitating comfort among members in discussing potentially sensitive or intimate topics.69 Focus group discussions can be conducted in person or via telephone, and electronic online formats are also emerging as an effective tool, particularly with adolescents.70 The decision of whether to use focus groups or individual interviews is dependent on which format is most likely to encourage participants to speak with the greatest candor (the privacy of individual interviews versus the reassurance of group shared experience), the nature of data desired (eg, individual experiences versus social norms), and feasibility concerns such as logistics and resources of the research team.
Focus groups rely on interactions among group members to widen the range of responses, activate forgotten details of individual experiences, or release inhibitions that otherwise discourage participants from disclosing information.46 Depending on the depth and complexity of the topic, saturation may occur in 4 to 6 focus groups.63 Discussion guides generally consist of 5 to 10 open-ended questions designed to stimulate discussion on central topics of interest.68 Focus group moderators must be adept at guiding the dialogue while permitting free exchange and ensuring that participants feel comfortable in expressing discordant views. Encouraging members to describe and compare their experiences and opinions with other group members can uncover the degree of consensus or diversity on the topic.46 There are a number of important considerations in the construction of groups, such as whether the group will comprise novices or experts, strangers or acquaintances, and other factors that may negatively influence the group dynamic, such as race, gender, income, or power differentials.68 The guiding criterion is to ensure that the group composition does not inhibit members from speaking, so that the discussion generates comprehensive information that reflects the full spectrum of opinions and experiences.
Observation
Observational data collection involves the systematic, detailed observation of people and events to learn about behaviors and interactions in natural settings.26 Such study designs are useful when the study goal is to understand cultural aspects of a setting or phenomenon,71 when the situation of interest is hidden from the public, or when those in the setting appear to have notably different views than do outsiders. Observational data collection is accomplished through an extended period of interaction between the researcher and study participants, in the participants’ milieu.72 The researcher may be a participant or nonparticipant observer. Participant observers are not known to the study subjects as researchers, although the role of nonparticipant observers is made clear to all in the setting. A notable challenge of the latter method is the Hawthorn effect, or the degree to which participants may alter their talk or behavior because they are being watched. Although covert observation (in which participants are unaware of the researcher’s role) may minimize this effect, it raises ethical questions and is therefore rare in health research. The data consist of extensive, detailed observational field notes that the researcher collects systematically and unobtrusively. Notes may include verbatim or paraphrased commentary, observations of the environment, and researcher reflections.
Document Review
A wide variety of written materials may serve as a valuable source of data. Documents include but are not limited to institutional documents (clinical, programmatic, or organizational records), personal documents (diaries, letters, artistic expressions), and public historical documents (legislative testimony, legal documents).9 One method of systematic document review is content analysis, a strategy that generates inferences through objective and systematic identification of core elements of written communication.73 Content analysis involves the categorization and classification of data to make inferences about the antecedents of a communication, describe and make inferences about characteristics of a communication, and make inferences about the effects of a communication.73
Data Analysis
Unlike quantitative studies, in which data collection is generally completed before data analysis begins, qualitative data collection and analysis occur in an iterative fashion. The research team moves back and forth between the data collection and data analysis processes to allow new avenues of inquiry to develop as additional data are collected.23,74
A commonly used analytic approach is the constant comparative method.27,44,45 In this form of analysis, data are reviewed line by line in detail. As a concept becomes apparent, a code is assigned to that segment of the document (or an entire document). Codes are tags44 or labels that help catalogue key concepts while preserving the context in which these concepts occur. To ascertain whether a code is assigned appropriately, the analyst compares text segments with segments that have been assigned the same code previously and decides whether they reflect the same concept. Using this constant-comparison method, the researchers refine dimensions of existing codes and identify new codes. Through this process, the code structure evolves inductively, in accordance with analytic principles of grounded theory.27,45
Qualitative data collection generates a substantial amount of data; for instance, 20 one-hour, in-depth interviews may generate up to 400 single-spaced pages of transcripts. Researchers often use computer software to facilitate qualitative data analysis through computerized coding, organization, searching, and retrieval of the data. Given the volume of data typically generated in a single study, software offers important efficiencies in organizing and retrieving data. Software provides a number of other important benefits, including more consistent and reproducible analysis, and access to analytic methods not available by hand.75,76 There are a number of software packages available; the most commonly used include AtlasTi, MAXqda, NVivo7, and ETHNOGRAPH version 5.0.77–80
Products of Qualitative Research
Although there are many products generated through qualitative inquiry (such as detailed descriptions of culture-sharing groups, case reports, narratives, and oral histories), certain forms of analytic output are particularly useful in outcomes research (Table 3). These include taxonomies, themes, and theories.74 Taxonomy is a formal system for classifying multifaceted, complex phenomena according to a set of common conceptual domains and dimensions. Taxonomies promote increased clarity in defining and hence comparing of diverse, complex healthcare interventions common to outcomes research.10 Themes are recurrent unifying concepts or statements81 about the subject of inquiry. They can be used to describe a phenomenon in depth, such as patient preferences regarding a specific treatment outcome, and to generate hypotheses for subsequent quantitative investigation. Themes can also be interrelated to form a chronology of events or to portray a theory. Theory is a set of general, modifiable propositions that help explain, predict, and interpret events or phenomena of interest, such as uptake of evidence-based guidelines in clinical practice. Theory is important for understanding potential causal links and confounding variables, understanding the context within which phenomenon occur, and providing a potential framework for guiding subsequent empirical research.
|
|
Illustrations From Qualitative Research in Cardiovascular Disease |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Qualitative and mixed methods offer unique opportunities to contribute to the empirical literature on key aspects of cardiovascular outcomes.82 The following examples illustrate the use of these approaches in selected areas of cardiovascular outcomes research, including effectiveness, equity, efficiency, timeliness, and patient-centeredness.
Effectiveness
Translation of evidence-based practices in primary care is a major focus of outcomes and effectiveness research. For example, Fuat and colleagues83 examined barriers to uptake of guidelines for diagnosis and management of heart failure in primary care. Focus groups with 30 general practitioners generated data to characterize the beliefs, current practices, and decision making of general practitioners in the diagnosis and management of suspected heart failure in primary care. Three central challenges were identified by general practitioners: uncertainty about clinical practice, lack of awareness of current evidence regarding therapeutic approaches, and influences of individual preference and local organizational factors. Findings can be used in the development of targeted interventions to increase uptake of evidence-based treatment strategies for heart failure.
Timeliness
Qualitative methods have also been used to examine aspects of organizational change in implementing guidelines for improving timeliness of cardiac care for patients with ST-segment elevation myocardial infarction.36,37 One hundred twenty-two key informants at 11 top-performing hospitals described innovations that were critical to their success in reducing door-to-balloon times in the management of myocardial infarction with ST-segment elevation. Findings informed the development of a follow-up quantitative survey used in a larger study to identify strategies that were significantly associated with a significant reduction in door-to-balloon times.84 Another study explored the relationship between cardiac self-care behaviors and treatment timing among survivors of myocardial infarction.85 Using mixed methods, the researchers analyzed a quantitative data set of 2972 myocardial infarction survivors and examined the relationship between time to treatment and 4 categories of patient characteristics known to influence timing (sociodemographics, clinical/risk factors, symptom experiences, and self-care activities). The researchers then analyzed qualitative illness narrative data from interviews with 35 patients who survived myocardial infarctions to further explore cardiac self-care responses, which generated novel and important insights on illness decision making and subsequent treatment actions.
Patient Centeredness
Development of patient-centered outcomes measures has been identified as a primary goal in outcomes research.82 In an effort to improve understanding of patient reservations regarding antihypertensive medication, Benson and Britten86 conducted in-depth interviews with 38 patients who were taking antihypertensive medication despite having concerns about the drug. They found diverse and highly individualized appraisal of risks and benefits of the medication therapy. The study highlighted the importance of having explicit discussions with patients regarding their balancing of reservations and reasons for taking medications when initiating antihypertensive therapy.
Patient-centered outcomes research requires development of functional assessment benchmarks that reflect patient perceptions of important aspects of function in everyday life during stroke recovery. Using in-depth interviews with 40 male stroke survivors from 3 ethnic groups (Hispanic, black, and non-Hispanic white), Gubrium and colleagues87 sought to understand the daily experiences of stroke survivors and to define components of function that are relevant from the patient’s perspective. Findings revealed that methods of formal functional assessment do not address aspects of daily life defined as important by patients and indicated new domains to be incorporated into functional measures to more fully capture essential elements of daily function for stroke survivors.
Equity
Disentangling the potential sources of documented disparities in access to and outcomes of cardiac care has been identified as a priority. Little is known about factors that influence patient decision-making in ischemic heart disease, as well as whether racial groups differ in regard to these factors. To advance knowledge in these areas, Ferguson and colleagues88 conducted a study using 15 focus groups with black and white patients who had received treatment for ischemic heart disease in 1 of 2 university-affiliated hospital emergency departments. Findings suggested several important factors related to perceptions of racial and financial discrimination in the healthcare system that emerged as important for the black participants, thereby illuminating important new areas for intervention.
|
Ensuring Rigor of Qualitative and Mixed-Methods Research |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Despite the integration of qualitative and mixed methods in biomedical and health services research, as well as a vast body of literature defining principles and practices of these methods, questions about scientific rigor persist. Concerns include the potential for researcher bias, a lack of reproducibility, and limited generalizability of findings.89 A number of useful guidelines exist for assessing the soundness of qualitative and mixed-methods research.89–96
The core concepts of validity, reliability, and generalizability that are essential markers of sound quantitative research apply to some degree in qualitative studies as well. Analogous principles in qualitative research are known as credibility, dependability, and transferability, respectively.91,97 The credibility of a study is determined by assessing the findings in several ways, including the degree to which the findings plausibly explain the phenomenon of interest, the extent to which findings cohere with what is already known, the attention paid to alternative or rival explanations or interpretations, and the correspondence between the researcher’s and respondent’s portrayal of respondent experience. Dependability refers to the degree to which the researchers account for and/or describe the changing contexts and circumstances during the study. Dependability can be enhanced by altering the research design or data collection as new findings emerge during data collection, by using multiple coders who engage in multicoder agreement analysis, and by the establishment of a multidisciplinary research team. Transferability is the degree to which themes or research protocols can be transferred or generalized to other settings, contexts, or populations.98 The researchers must provide sufficiently detailed information to allow the reader to determine whether the findings are applicable (or can transfer to) to another specific environment.
There are a number of specific techniques recognized by qualitative and mixed methods experts23,29,44,46,51,97,99,100 to enhance credibility, dependability, and transferability of qualitative research. These include strategies for study design, data collection, analysis, and reporting. Experts caution that the rote use of these techniques does not necessarily confer rigor and that principles and assumptions of qualitative research design and analysis must be applied consistently.8,74,101 Guidelines should be used with judgment and integrity and with the primary aim of conducing scientifically sound research.
Study Design
The rationale for the study design must be explicit and compelling, including justification for the use of the qualitative approach in general, as well as the choice of a specific method (eg, focus group, in-depth interview, or observation).89,90,98 For instance, in-depth interviews would be appropriate for examining individual perceptions and beliefs, whereas observational methods are necessary for assessing behaviors in a specific environmental context.
Sampling Strategy and Size
The sample should be constructed in accordance with established principles of qualitative sampling.9,44 The process should be systematic, conceptually sound, and thoroughly documented. The sample must include enough participants to provide sufficient representation of relevant characteristics, and negative or disconfirming cases or viewpoints should be reflected in the sample. The most commonly used criterion for determining adequate sample size is theoretical saturation.
Data Collection
There are several specific strategies to enhance the integrity and dependability of the data. First, given the critical role of the interviewer in data gathering, it is essential to employ interviewers with substantial training and experience in qualitative data collection. Second, although handwritten notes or summaries of interviews are a useful reference, audiotape or videotape of the interviews, focus group discussions, or observations ensures accuracy of the data and ready accessibility to other researchers. Third, some experts recommend the use of a professional transcription service with independent verification of transcription accuracy. This ensures that all data are intact and included in a standard master data set. Finally, although not essential, there is increasing importance placed on the use of computer software to organize and maintain data.
Data Analyses
Qualitative researchers can and should use methods of analysis that are explicit, systematic, and reproducible.96 Some experts argue that a single researcher conducting all the coding is both sufficient and preferred, particularly in ethnographic studies in which the individual researcher is inextricably enmeshed in the data collection and analysis. In these cases, awareness of and attention to the researcher’s potential biases is important.98,102 Other experts suggest that the quality and breadth of analysis are enhanced by ongoing and close involvement of multiple analysts from differing disciplines.8,21,91,103 A multidisciplinary analytical team can generate unique insights from differing perspectives, engage in critical discussion of unclear or subjective data, and ensure consideration of multiple interpretations of the data. In either approach, researchers must look for and examine disconfirming, negative, and deviant cases (those that do not fit neatly within the explanatory scheme) and provide possible explanations for why the data vary. Deviant cases can also be useful in refining the existing theory to increase its reliability and validity.8,103
Because of the integrative and fluid nature of qualitative analysis, it is critical to maintain systematic, detailed documentation of analytic decisions, sometimes referred to as an audit trail.44 Documentation might include memos about insights at different points in the process, the regularly updated coding schemes with definitions and properties described, and analytic decisions made through the process. The goal is to provide sufficient detail so that another researcher could analyze the same data in the same way and come to essentially similar conclusions.8
Presentation of Findings
Authors should provide an explicit and well-cited description of methods and a thorough, detailed report of findings. Because a central benefit of qualitative research is to discover novel insights, particular caution must be exercised to avoid telling a story that is relatively obvious or, as experts have described, "rhetorically convincing but scientifically incomplete."8,89 It is also incumbent on the researchers to place the findings in the context of what is already known and to describe the specific and novel contributions made by their findings.
Other Techniques for Ensuring Rigor
Other strategies for enhancing rigor include triangulation, participant validation, and coder reliability checks, although opinions vary30,44,47,62,71,101,104 as to their utility. Because they are time- and resource-intensive techniques, they should be considered carefully at the onset of the study. They should be incorporated in the study design only if they are determined to be appropriate for the research and if there are sufficient resources to implement them fully and in accordance with established principles and practices.
Triangulation is a process by which a single phenomenon is examined with multiple observers, theories, methods, or data sources to determine the degree of convergence across components.9,21,91 This process offers the benefits of cross-validation of findings and a more holistic understanding by capturing the context in which phenomena occur. Triangulation can also minimize common method bias,28 which can occur if the same source provides information on all variables of interest.21,105 Challenges in the use of triangulation include securing funding to address essentially the same research question twice and addressing discordant findings across components.30
Participant validation strategies involve presenting findings to the study participants for confirmation as to whether the findings represent a reasonable account of their experience.97 Although participant feedback can be useful for assessing the face validity of findings, fidelity to the participants’ perception is only 1 marker of validity. Some experts suggest it is helpful to view participant validation as 1 step in a process of error reduction.8,89 Practical considerations for participant confirmation include the point at which feedback is solicited, the scope and depth of findings to be shared, the systematic process to be used for gathering feedback, whether to request input from some or all participants, and plans for addressing disconfirming or discordant views.
Coder reliability refers to the degree to which 2 coders agree in their independent assignment of codes to text. Coder reliability checks have been suggested as 1 tool for establishing rigor, and percentages of agreement44 are occasionally reported in the empirical literature. However, some experts question the utility of computing percentages of agreement among codes. They suggest that the greater value is in the insights generated by team discussions of disagreements and the subsequent refinement of codes as indicated.106,107 Others observe that these approaches need not be mutually exclusive and that it is possible for a research team to use both strategies.29
|
Future Directions for Outcomes Research on Cardiovascular Disease |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Although cardiovascular outcomes research has contributed to improvements in both practice and policy over the past decade, important challenges remain.82 Qualitative and mixed methods represent a body of scientifically sound, well-established research approaches that are uniquely suited to help address these challenges. For instance, qualitative methods can shed light on facilitators of and barriers to the uptake of promising interventions or evidence-based guidelines through characterization of key aspects of organizational context and clinical processes. Optimal outcomes require supporting principles of patient centeredness and promoting understanding of patient experience. By emphasizing the patient’s perspective in the conceptualization and implementation of research, qualitative methods can illuminate aspects of the patient experience that have been largely ignored. The importance of patient-centered perceptions of outcomes has been noted specifically in the areas of acute and long-term care for people with stroke108 and persons with disabilities.109 Critical areas of research to address racial/ethnic disparities in cardiovascular disease outcomes have been identified. Qualitative methods can identify barriers to implementing and sustaining community-based prevention programs to improve cardiovascular health.110 Qualitative approaches can also inform the development of tailored intervention strategies for areas of high disease risk and burden for ethnic/racial minority groups.111 Finally, ensuring patient safety, including reduction of misuse of medical therapies and oversights in the course of clinical care, has been identified as a priority. Patient and provider views on sensitive healthcare quality issues such as adverse events and medical errors are topics well suited for qualitative investigation.112,113
|
Conclusion |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
Not everything that can be counted counts, and not everything that counts can be counted.—Albert Einstein, attributed (German-born US physicist, 1879 to 1955)114
Research that draws on the strengths of both quantitative and qualitative approaches has become increasingly recognized as essential in a number of fields intrinsic to outcomes research. Qualitative and mixed methods have been used in a number of areas of clinical medicine, including critical care,13,115,116 oncology,117 surgery,118 primary care,119,120 and orthopedics.121 The present report introduces qualitative and mixed methods as providing unique and critical opportunities for cardiovascular outcomes research. A clear understanding of such methodologies and systematic incorporation of established techniques for ensuring rigor can help outcomes researchers successfully adopt and integrate qualitative approaches when they are appropriate.
|
Acknowledgments |
|---|
We thank John Creswell, PhD, and Benjamin Crabtree, PhD, for their helpful review and comments on earlier versions of this report.
Sources of Funding
Dr Bradley is supported by the Patrick and Catherine Weldon Donaghue Medical Research Foundation Investigator Award.
Disclosures
None.
|
References |
|---|
|
TopIntroductionDefining Qualitative ResearchWhen to Consider Using...Conducting Rigorous Qualitative...Illustrations From Qualitative...Ensuring Rigor of Qualitative...Future Directions for Outcomes...ConclusionReferences |
|---|
1. Clancy CM, Eisenberg JM. Outcomes research: measuring the end results of health care. Science. 1998; 282: 245–246.
2. Krumholz HM. The year in epidemiology, health services, and outcomes research. J Am Coll Cardiol. 2005; 46: 1362–1370.
3. Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care. 2003; 12: 47–52.
4. Green J, Britten N. Qualitative research and evidence based medicine. BMJ. 1998; 316: 1230–1232.
5. Malterud K. The art and science of clinical knowledge: evidence beyond measures and numbers. Lancet. 2001; 358: 397–400.[CrossRef][Medline] [Order article via Infotrieve]
6. Pope C, van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care. 2002; 11: 148–152.
7. Greenhalgh T. Integrating qualitative research into evidence based practice. Endocrinol Metab Clin North Am. 2002; 31: 583–601, ix.[CrossRef][Medline] [Order article via Infotrieve]
8. Mays N, Pope C. Rigour and qualitative research. BMJ. 1995; 311: 109–112.
9. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Thousand Oaks, Calif: Sage; 2002.
10. Sofaer S. Qualitative methods: what are they and why use them? Health Serv Res. 1999; 34: 1101–1118.[Medline] [Order article via Infotrieve]
11. Shortell SM. The emergence of qualitative methods in health services research. Health Serv Res. 1999; 34: 1083–1090.[Medline] [Order article via Infotrieve]
12. McKinlay JB, Marceau LD. To boldly go. Am J Public Health. 2000; 90: 25–33.
13. Sinuff T, Cook DJ, Giacomini M. How qualitative research can contribute to research in the intensive care unit. J Crit Care. 2007; 22: 104–111.[CrossRef][Medline] [Order article via Infotrieve]
14. McKibbon KA, Gadd CS. A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year. BMC Med Inform Decis Making. 2004; 4: 11.[CrossRef]
15. Hoff TJ, Witt LC. Exploring the use of qualitative methods in published health services and management research. Med Care Res Rev. 2000; 57: 139–160.
16. National Institutes of Health. Methodology and Measurement in the Behavioral and Social Sciences (RO1) Program Announcement. Available at: http://grants.nih.gov/grants/guide/pa-files/PA-07-060.html. Accessed September 11, 2007.
17. National Institutes of Health. NIH Summer Institute: The Design and Conduct of Qualitative and Mixed-Method Research in Social Work and Other Health Professions. Available at: http://obssr.od.nih.gov/conf_wkshp/sw/. Accessed September 11, 2007.
18. Office of Behavioral and Social Sciences Research, National Institutes of Health. Qualitative Methods in Health Research: Opportunities and Considerations in Applications and Reviews. Available at: http://obssr.od.nih.gov/Documents/Publications/Qualitative.pdf. Accessed August 21, 2007.
19. Gerontological Society of America/American Public Health Association. Qualitative and Mixed Methods Research: Improving the Quality of Science and Addressing Health Issues of Diverse Populations. Available at: http://www.apha.org/membergroups/newsletters/sectionnewsletters/geron/annual+meeting05/2205.htm. Accessed September 11, 2007.
20. Ragin CC, Nagel J, White P. National Science Foundation Workshop on Scientific Foundations of Qualitative Research. National Science Foundation. Available at: http://www.nsf.gov/pubs/2004/nsf04219/nsf04219_1.pdf. Accessed September 11, 2007.
21. Denzin N, Lincoln Y. Handbook of Qualitative Research. 2nd ed. Thousand Oaks, Calif: Sage; 2000.
22. Berkwits M, Inui TS. Making use of qualitative research techniques. J Gen Intern Med. 1998; 13: 195–199.[CrossRef][Medline] [Order article via Infotrieve]
23. Crabtree B, Miller W, eds. Doing Qualitative Research. 2nd ed. Newbury Park, Calif: Sage; 1999.
24. Inui TS. The virtue of qualitative and quantitative research. Ann Intern Med. 1996; 125: 770–771.
25. Sofaer S, Firminger K. Patient perceptions of the quality of health services. Annu Rev Public Health. 2005; 26: 513–559.[CrossRef][Medline] [Order article via Infotrieve]
26. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ. 1995; 311: 42–45.
27. Glaser B, Strauss A, eds. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, Ill: Aldine; 1967.
28. Jick T. Mixing qualitative and quantitative methods: triangulation in action. Adm Sci Q. 1979: 602–610.
29. Creswell JW, Piano Clark V. Designing and Conducting Mixed Methods Research. Thousand Oaks, Calif: Sage; 2007.
30. Morgan DL. Connected contributions as a motivation for combining qualitative and quantitative methods. In: Curry L, Shield R, Wetle T, eds. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association and Gerontological Society of America; 2006: 53–63.
31. Morse J, Wolfe R, Niehaus L. Principles and procedures of maintaining validity for mixed-method design. In: Curry L, Shield R, Wetle T, eds. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association and Gerontological Society of America; 2006: 65–78.
32. Journal of Mixed Methods Research. Available at: http://mmr.sagepub.com/. Accessed September 11, 2007.
33. Edmondson AC, Bohmer RM, Pisano G. Disrupted routines: team learning and new technology implementation in hospitals. Adm Sci Q. 2001; 46: 685–716.[CrossRef]
34. Horowitz CR, Rein SB, Leventhal H. A story of maladies, misconceptions and mishaps: effective management of heart failure. Soc Sci Med. 2004; 58: 631–643.[CrossRef][Medline] [Order article via Infotrieve]
35. Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. A qualitative study of increasing beta-blocker use after myocardial infarction: why do some hospitals succeed? JAMA. 2001; 285: 2604–2611.
36. Bradley EH, Curry LA, Webster TR, Mattera JA, Roumanis SA, Radford MJ, McNamara RL, Barton BA, Berg DN, Krumholz HM. Achieving rapid door-to-balloon times: how top hospitals improve complex clinical systems. Circulation. 2006; 113: 1079–1085.
37. Bradley EH, Roumanis SA, Radford MJ, Webster TR, McNamara RL, Mattera JA, Barton BA, Berg DN, Portnay EL, Moscovitz H, Parkosewich J, Holmboe ES, Blaney M, Krumholz HM. Achieving door-to-balloon times that meet quality guidelines: how do successful hospitals do it? J Am Coll Cardiol. 2005; 46: 1236–1241.
38. Krause N. The use of qualitative methods to improve quantitative measures of health-related constructs. Med Care. 2006; 44: S34–S38.[CrossRef][Medline] [Order article via Infotrieve]
39. Kressin NR, Clark JA, Whittle J, East M, Peterson ED, Chang BH, Rosen AK, Ren XS, Alley LG, Kroupa L, Collins TC, Petersen LA. Racial differences in health-related beliefs, attitudes, and experiences of VA cardiac patients: scale development and application. Med Care. 2002; 40: I72–I85.[Medline] [Order article via Infotrieve]
40. Collins TC, Clark JA, Petersen LA, Kressin NR. Racial differences in how patients perceive physician communication regarding cardiac testing. Med Care. 2002; 40: I27–I34.[Medline] [Order article via Infotrieve]
41. Stewart AL, Napoles-Springer A. Health-related quality-of-life assessments in diverse population groups in the United States. Med Care. 2000; 38: II102–II124.[Medline] [Order article via Infotrieve]
42. Napoles-Springer AM, Stewart AL. Overview of qualitative methods in research with diverse populations: making research reflect the population. Med Care. 2006; 44: S5–S9.[CrossRef][Medline] [Order article via Infotrieve]
43. Koops L, Lindley RI. Thrombolysis for acute ischaemic stroke: consumer involvement in design of new randomised controlled trial. BMJ. 2002; 325: 415.
44. Miles M, Huberman AM, eds. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, Calif: Sage; 1994.
45. Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Newbury Park, Calif: Sage; 1998.
46. Morgan DL, Krueger RA, eds. The Focus Group Kit. London, United Kingdom: Sage; 1998.
47. Morse J, Richards L. README First for a User’s Guide to Qualitative Methods. Thousand Oaks, Calif: Sage; 2002.
48. Curry L, Shield R, Wetle T, eds. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association and Gerontological Society of America; 2006.
49. Creswell J. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. 2nd ed. Thousand Oaks, Calif: Sage; 2003.
50. Edmondson A, McManus S. Methodological fit in management field research. Acad Manage Rev. 2007; 32: 1155–1179.
51. Creswell J. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. 2nd ed. Thousand Oaks, Calif: Sage; 2007.
52. Schensul J. The Ethnographer’s Toolkit: Walnut Creek, Calif: AltaMira Press; 1999.
53. Savage J. Ethnography and health care. BMJ. 2000; 321: 1400–1402.
54. Britten N. Qualitative interviews in medical research. BMJ. 1995; 311: 251–253.
55. Tedlock B. Ethnography and ethnographic representation. In: Denzin N, Lincoln Y, eds. Strategies of Qualitative Inquiry. 2nd ed. Thousand Oaks, Calif: Sage; 2003: 165–213.
56. Yin RK. Case Study Research: Design and Methods. 1st ed. Beverly Hills, Calif: Sage; 1984.
57. Stake R. The Art of Case Study Research. Thousand Oaks, Calif: Sage; 1995.
58. Anderson RA, Crabtree BF, Steele DJ, McDaniel RR Jr. Case study research: the view from complexity science. Qual Health Res. 2005; 15: 669–685.
59. Yin RK. Applications of Case Study Research. Beverly Hills, Calif: Sage; 1993.
60. Meyer J. Qualitative research in health care: using qualitative methods in health related action research. BMJ. 2000; 320: 178–181.
61. Kuzel A. Sampling in qualitative research. In: Crabtree B, Miller W, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, Calif: Sage; 1999: 33–46.
62. Morse J. The significance of saturation. Qual Health Res. 1995; 5: 147–149.
63. Morgan DL. Focus groups. Annu Rev Sociol. 1996; 22: 129–152.[CrossRef]
64. Dicicco-Bloom B, Crabtree BF. The qualitative research interview. Med Educ. 2006; 40: 314–321.[CrossRef][Medline] [Order article via Infotrieve]
65. Hamilton RJ, Bowers BJ. Internet recruitment and e-mail interviews in qualitative studies. Qual Health Res. 2006; 16: 821–835.
66. Lofland J. Analyzing Social Settings. Belmont, Calif: Wadsworth; 1971.
67. Miller W, Crabtree B. Depth interviewing. In: Crabtree B, Miller W, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage; 1999: 89–107.
68. Krueger R, Casey M. Focus Groups. 3rd ed. Thousand Oaks, Calif: Sage; 2000.
69. Halcomb EJ, Gholizadeh L, DiGiacomo M, Phillips J, Davidson PM. Literature review: considerations in undertaking focus group research with culturally and linguistically diverse groups. J Clin Nurs. 2007; 16: 1000–1011.[CrossRef][Medline] [Order article via Infotrieve]
70. Fox FE, Morris M, Rumsey N. Doing synchronous online focus groups with young people: methodological reflections. Qual Health Res. 2007; 17: 539–547.
71. Lambert H, McKevitt C. Anthropology in health research: from qualitative methods to multidisciplinarity. BMJ. 2002; 325: 210–213.
72. Bogdewic S. Participant observation. In: Crabtree B, Miller W, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, Calif: Sage; 1999: 47–69.
73. Holsti O. Content Analysis for the Social Sciences and Humanities. Reading, Mass: Addison-Wesley; 1969.
74. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007; 42: 1758–1772.[CrossRef][Medline] [Order article via Infotrieve]
75. Maietta R. State of the art: integrating software with qualitative analysis. In: Curry L, Shield R, Wetle T, eds. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association and Gerontological Society of America; 2006: 117–139.
76. MacMillan K, Koenig T. The wow factor: preconceptions and expectations for data analysis software in qualitative research. Soc Sci Comput Rev. 2004; 22: 179–186.[Abstract]
77. AtlasTi. Available at: http://www.atlasti.com/. Accessed September 17, 2007.
78. MAXqda. Available at: http://www.maxqda.com/. Accessed September 17, 2007.
79. NVivo 7. Available at: http://www.qsrinternational.com/products_nvivo.aspx. Accessed September 17, 2007.
80. ETHNOGRAPH v5.0. Available at: http://www.qualisresearch.com/. Accessed September 17, 2007.
81. Boyatzis R. Transforming Qualitative Information: Thematic and Code Development. Thousand Oaks, Calif: Sage; 1998.
82. Krumholz HM. Outcomes research: generating best evidence for practice and policies. Circulation. 2008; 118: 309–318.
83. Fuat A, Hungin AP, Murphy JJ. Barriers to accurate diagnosis and effective management of heart failure in primary care: qualitative study. BMJ. 2003; 326: 196.
84. Bradley EH, Herrin J, Wang Y, Barton BA, Webster TR, Mattera JA, Roumanis SA, Curtis JP, Nallamothu BK, Magid DJ, McNamara RL, Parkosewich J, Loeb JM, Krumholz HM. Strategies for reducing the door-to-balloon time in acute myocardial infarction. N Engl J Med. 2006; 355: 2308–2320.
85. Schoenberg NE, Amey CH, Stoller EP, Drew EM. The pivotal role of cardiac self-care in treatment timing. Soc Sci Med. 2005; 60: 1047–1060.[CrossRef][Medline] [Order article via Infotrieve]
86. Benson J, Britten N. Patients’ decisions about whether or not to take antihypertensive drugs: qualitative study. BMJ. 2002; 325: 873.
87. Gubrium JF, Rittman MR, Williams C, Young ME, Boylstein CA. Benchmarking as everyday functional assessment in stroke recovery. J Gerontol B Psychol Sci Soc Sci. 2003; 58: S203–S211.
88. Ferguson JA, Weinberger M, Westmoreland GR, Mamlin LA, Segar DS, Greene JY, Martin DK, Tierney WM. Racial disparity in cardiac decision making: results from patient focus groups. Arch Intern Med. 1998; 158: 1450–1453.
89. Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ. 2000; 320: 50–52.
90. Frankel R. Standards of qualitative research. In: Crabtree B, Miller W, eds. Doing Qualitative Research. Newbury Park, Calif: Sage; 1999: 333–347.
91. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999; 34: 1189–1208.[Medline] [Order article via Infotrieve]
92. Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res. 1998; 8: 341–351.
93. Giacomini MK, Cook DJ; Evidence-Based Medicine Working Group. Users’ guides to the medical literature, XXIII: qualitative research in health care B: what are the results and how do they help me care for my patients? JAMA. 2000; 284: 478–482.
94. Giacomini MK, Cook DJ; Evidence-Based Medicine Working Group. Users’ guides to the medical literature, XXIII: qualitative research in health care A: are the results of the study valid? JAMA. 2000; 284: 357–362.
95. Daly J, Willis K, Small R, Green J, Welch N, Kealy M, Hughes E. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007; 60: 43–49.[CrossRef][Medline] [Order article via Infotrieve]
96. Greenhalgh T, Taylor R. How to read a paper: papers that go beyond numbers (qualitative research). BMJ. 1997; 315: 740–743.
97. Lincoln Y, Guba E. Naturalistic Inquiry. Newbury Park, Calif: Sage; 1985.
98. Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet. 2001; 358: 483–488.[CrossRef][Medline] [Order article via Infotrieve]
99. Yin RK. Enhancing the quality of case studies in health services research. Health Serv Res. 1999; 34: 1209–1224.[Medline] [Order article via Infotrieve]
100. Devers KJ. How will we know "good" qualitative research when we see it? Beginning the dialogue in health services research. Health Serv Res. 1999; 34: 1153–1188.[Medline] [Order article via Infotrieve]
101. Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001; 322: 1115–1117.
102. Gubrium JF. Qualitative methods today. In: Curry L, Shield R, Wetle T, eds. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association and Gerontological Society of America; 2007: 15–25.
103. Pope C, Ziebland S, Mays N. Qualitative research in health care: analysing qualitative data. BMJ. 2000; 320: 114–116.
104. Cohen D, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008; 6: 331–339.
105. Podaskoff P, MacKenzie S, Lee L, Podsakoff N. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003; 88: 879–903.[CrossRef][Medline] [Order article via Infotrieve]
106. Morse J. Perfectly healthy but dead: the myth of inter-rater reliability. Qual Health Res. 1997; 7: 445–447.
107. Armstrong D, Gosling A, Weinman J, Martaeu T. The place of inter-rater reliability in qualitative research: an empirical study. Sociology. 1997; 31: 1–6.[Abstract]
108. McKevitt C, Redfern J, Mold F, Wolfe C. Qualitative studies of stroke: a systematic review. Stroke. 2004; 35: 1499–1505.
109. Jette AM, Keysor JJ. Uses of evidence in disability outcomes and effectiveness research. Milbank Q. 2002; 80: 325–345.[CrossRef][Medline] [Order article via Infotrieve]
110. Veazie MA, Galloway JM, Matson-Koffman D, LaBarthe DR, Brownstein JN, Emr M, Bolton E, Freund E Jr, Fulwood R, Guyton-Krishnan J, Hong Y, Lebowitz M, Ochiai E, Schoeberl M, Robertson RM. Taking the initiative: implementing the American Heart Association Guide for Improving Cardiovascular Health at the Community Level. Circulation. 2005; 112: 2538–2554.
111. Yancey AK, Robinson RG, Ross RK, Washington R, Goodell HR, Goodwin NJ, Benjamin ER, Langie RG, Galloway JM, Carroll LN, Kong BW, Leggett CJ, Williams RA, Wong MJ. Discovering the full spectrum of cardiovascular disease: Minority Health Summit 2003: report of the Advocacy Writing Group. Circulation. 2005; 111: e140–e149.
112. Gallagher TH, Waterman AD, Ebers AG, Fraser VJ, Levinson W. Patients’ and physicians’ attitudes regarding the disclosure of medical errors. JAMA. 2003; 289: 1001–1007.
113. Kaldjian LC, Jones EW, Rosenthal GE, Tripp-Reimer T, Hillis SL. An empirically derived taxonomy of factors affecting physicians’ willingness to disclose medical errors. J Gen Intern Med. 2006; 21: 942–948.[CrossRef][Medline] [Order article via Infotrieve]
114. Einstein, attributed. Available at: http://www.quotationspage.com/quote/26950.html. Accessed September 17, 2007.
115. Doering LV, McGuire AW, Rourke D. Recovering from cardiac surgery: what patients want you to know. Am J Crit Care. 2002; 11: 333–343.
116. Rubenfeld G, Angus D, Pinsky M, Curtis R, Connors A, Bernard G. Outcomes research in critical care: results of the American Thoracic Society Critical Care Assembly Workshop on Outcomes Research. Am J Respir Crit Care Med. 1999; 160: 358–367.
117. Woodgate R. Part II: a critical review of qualitative research related to children’s experiences with cancer. J Pediatr Oncol Nurs. 2000; 17: 207–228.
118. O'Neill T, Jinks C, Ong BN. Decision-making regarding total knee replacement surgery: a qualitative meta-synthesis. BMC Health Serv Res. 2007; 7: 52.[CrossRef][Medline] [Order article via Infotrieve]
119. Creswell JW, Fetters MD, Ivankova NV. Designing a mixed methods study in primary care. Ann Fam Med. 2004; 2: 7–12.
120. Twohig PL, Putnam W. Group interviews in primary care research: advancing the state of the art or ritualized research? Fam Pract. 2002; 19: 278–284.
121. Verbeek J, Sengers MJ, Riemens L, Haafkens J. Patient expectations of treatment for back pain: a systematic review of qualitative and quantitative studies. Spine. 2004; 29: 2309–2318.[CrossRef][Medline] [Order article via Infotrieve]
This article has been cited by other articles:
 |
|
 |
|
  M. Aira, P. Mantyselka, A. Vehvilainen, and E. Kumpusalo Occupational isolation among general practitioners in Finland Occup. Med., September 1, 2010; 60(6): 430 - 435. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. Santana, M. Nunez-Smith, A. Camp, E. Ruppe, D. Berg, and L. Curry Quality improvement in community health centres: the role of microsystem characteristics in the implementation of a diabetes prevention initiative Qual. Saf. Health Care, August 1, 2010; 19(4): 290 - 294. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. D. Peterson, D. Dai, E. R. DeLong, J. M. Brennan, M. Singh, S. V. Rao, R. E. Shaw, M. T. Roe, K. K. L. Ho, L. W. Klein, et al. Contemporary Mortality Risk Prediction for Percutaneous Coronary Intervention: Results From 588,398 Procedures in the National Cardiovascular Data Registry J. Am. Coll. Cardiol., May 4, 2010; 55(18): 1923 - 1932. [Abstract] [Full Text] [PDF]
|
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||