Manual for ACC/AHA Guideline Writing Committees

Methodologies and Policies from the ACC/AHA Task Force on Practice Guidelines

Section II: Tools and Methods for Creating Guidelines

• Step Three: Sort and Evaluate the Evidence

Stages of Sorting Evidence
After the literature search results have been imported into the computerized database managed at the ACC, the Research Analyst reviews the abstracts and removes non-relevant citations. At this step, only the article's title and abstract are assessed, so any article likely to be relevant to the guideline is maintained. Additionally, the Research Analyst sorts the abstracts to correspond with the specific clinical objectives identified in Step One. This initial sort creates a comprehensive set of potentially relevant studies.

It is important to err on the side of over-inclusion because once a trial has been excluded from the selection process it is unlikely to be reconsidered. Questionable articles which are included at one stage can be excluded at a latter stage when more information on the study is available. - Mulrow, 1996

Although the Research Analyst does a preliminary level of sorting, the clinical expertise of writing committee members is necessary to make the final decision as to whether the article is a relevant piece of evidence that should be included in the development of a recommendation. This often requires review of the article's full text and critique of the research methodology employed. As necessary, the research analyst will provide the full text of all peer-reviewed, published:

• Randomized controlled trials
• Meta-analysis
• Systematic reviews of evidence
• Diagnostic studies using comparison with a gold standard

Along with each full text article, the writer will receive Checklist 2, which asks the writer to make the final determination of the article's role within the guideline. Documentation of completed checklists will be maintained by the ACC in the methodology files for the guideline. The Appendix includes three articles from the British Medical Journal and articles by Pogue and Lau that provide further information on evaluating the quality of published literature.

Unpublished Data
Guideline writers are frequently familiar with data from abstracts and late breaking trials that may impact the guideline's content. The results from unpublished data should not be considered except in few instances, should be no older than 2 years, and should be clearly stated to be unpublished data in the guideline text. Only trials presented at a major national or international scientific meeting are allowed, and may not be used to support any recommendation. When trial data are discussed, the text should clearly state that the data are preliminary. Additionally, guideline writers should obtain slides from the trial presentation, perform a detailed review, and ask the presenter of the trial for guidance, keeping in mind that the trial group has the prerogative to request that the information not be published in a guideline.

Balancing Scientific Rigor with Feasibility
The Cochrane Collaboration publishes perhaps the most rigorous and comprehensive guide to conducting systematic reviews of evidence, and their methodology has provided the basis for much of this manual. However, due to time and economic constraints, some components of their methodology (such as creating and validating criteria for which articles to include, and removing the journal and author names from articles being reviewed) are beyond the scope of ACC/AHA guideline development.

A less resource-intensive, more feasible approach is to establish a few basic criteria (such as randomized controlled trials only or studies with at least six month follow-up) and be as inclusive and unbiased as possible. The Task Force recommends rigorous review of the articles used in evidence tables and meta-analyses—those articles that are most fundamental to the guideline recommendations. Documentation of why studies are included and excluded from consideration will provide additional scientific rigor to the document and will be published on the ACC web site as a component of the guideline methodology. The Research Analyst assigned to the guideline can and should be used in scoring the articles and synthesizing the evidence (Step Four) to ease the burden on the writing committee.

 

Checklist 2. Determining the Evidence-Based for Guideline Recommendations* Guideline writers are asked to consider the merits, quality, and generalizability of each article relevant to the clinical objective. This checklist should be completed only for articles from the peer-reviewed, published literature that are: randomized controlled trials, meta-analysis/systematic reviews, or diagnostic studies using comparison with a gold standard. Article/Author: Please indicate one of the following conclusions about the article: _____ Yes, this is a relatively high quality study that provides credible results and should be included in the evidence table and references that support the recommendation(s) for this clinical objective. _____ No, this study is not of sufficient quality to be included in the evidence table, but Yes, this study contains some useful information about the clinical objective and should be maintained as a reference for the text accompanying the recommendation. _____ No, this is a relatively poor study that should not be used in the evidence table or in the references for this clinical objective. _____ No, this study is not directly relevant to the clinical objective Comments:   * Our methodology for grading individual studies is a work in progress. We are currently pilot testing this checklist with a number of writing groups as well as investigating more detailed checklists for evaluating study quality. Please re-visit this site in the coming months for newer checklists.