Conversion of Cardiovascular Conference Abstracts to PublicationsClinical Perspective
Background—The transition of scientific knowledge from discovery into practice is less than ideal. A key step in this translation occurs when presentations from major meetings are published in peer-reviewed literature, yet the completeness and speed of this process are not known. We performed a systematic and automated evaluation of rates, timing, and correlates of publication from scientific abstracts presented at 3 major cardiovascular conferences.
Methods and Results—Using an automated computer algorithm, we searched the ISI Web of Science to identify peer-reviewed publications of abstracts presented at the American Heart Association (AHA), American College of Cardiology (ACC), and European Society of Cardiology (ESC) scientific sessions from 2006 to 2008. We compared abstract publication rates and journal impact factor between the 3 meetings using multivariable logistic regression modeling. From 2006 to 2008, 11 365, 5005, and 10 838 abstracts were presented at the AHA, ACC, and ESC meetings, respectively. Overall, 30.6% of presented abstracts were published within 2 years of the conference; ranging from 34.5% for AHA to 29.5% for ACC to 27.0% for ESC (P<0.0001). Five years after conference presentation in 2005, these rates had risen slightly to 49.7% for AHA, 42.6% for ACC, and 37.6% for ESC (P<0.0001). After adjustment for abstract characteristics and contributing countries, abstracts presented at the AHA meeting remained more likely for publication relative to the ESC (adjusted odds ratio, 1.24; 95% confidence interval, 1.16–1.34) and the ACC (adjusted odds ratio, 1.20; 95% confidence interval, 1.11–1.29). Median impact factors for subsequent publications varied from 4.8 (interquartile range, 3.8–10.1) for AHA to 4.0 (interquartile range, 3.1–7.5) for ACC and 3.9 (quartile 1–3, 2.5–5.8) for ESC (P for difference between groups <0.01). Clinical science and population science were less likely to be published compared with basic science.
Conclusions—One third of abstracts were translated into publications by 2 years after presentation and less than one half by 5 years after presentation. Our findings suggest that efforts to understand the barriers to publication and to facilitate the rapid dissemination of new knowledge are needed to speed up the transition of scientific discovery into clinical practice.
Major scientific conferences remain important venues for sharing novel, rapidly developing medical findings among peers; however, results presented at conferences are preliminary and are generally not considered final until published in full detail in a peer-reviewed journal. Optimally, a very high percentage of abstracts selected for presentation at these meetings would be published in a timely fashion so that these findings could influence subsequent scientific discovery and/or knowledge translation into practice. Small manual studies for selected meetings have shown that publication rates vary substantially,1,2 and meta-analyses gathering these data showed a long delay in publication.3–5 No prior study has systematically assessed the scientific yield of medical conferences, and no comparative data exist for the major scientific sessions in cardiology. Some even question the core purpose of medical conferences and have asked specifically for data on the utility of these meetings, being that they are expensive and seemingly inefficient.6,7
Clinical Perspective on p 2825
To address these issues, we developed a novel algorithm for searching the published literature. After validating this tool, we conducted a comprehensive assessment to determine how many abstracts presented at the 3 major international cardiovascular conferences (the American Heart Association [AHA], American College of Cardiology [ACC], and European Society of Cardiology [ESC] annual scientific sessions) were published as peer-reviewed papers within 2 years of their initial meeting presentation.
Using publicly available data from the Web sites of the AHA, ACC, and ESC for their main journals (Circulation, Journal of the American College of Cardiology, and European Heart Journal)8–10 we identified all abstracts presented at their corresponding annual scientific sessions meeting in 2006, 2007, and 2008. Obtainable information from all 3 societies included author names, institution names, abstract title, scientific categories, number of authors on the abstract, number of countries involved with the abstract, and year of the meeting. For abstracts with >1 country involved, these were attributed to all mentioned countries. The AHA and ESC used a major-category categorization scheme for the scientific content of the abstract but the ACC did not, so these were categorized manually according to the subcategorization provided. Most programs were indicated with a title and often a category designation (eg, clinical, basic, or epidemiology). If the larger program did not include such a keyword in the title, then each session included in the program was examined by title and category. Duplicate abstracts within the same conference were removed from the data set.
Search Algorithm and the Linked Database
Using an automatic computer algorithm, we fed abstract information into the ISI Web of Science search engine11 (Thomson Reuters). The search was done for each abstract separately between the month before the conference (March for ACC, October for AHA, and August for ESC) and 2 years after the month of the meeting, allowing equal and substantial follow-up for all years and conferences. A 2-year follow-up window was chosen to assess publication rates during a time frame when publications preferably would be published representing new knowledge. We also conducted a sensitivity analysis to explore rates of delayed publication up to 5 years after the conference using data from the 2006 conferences.
For the first iteration, the search used the following criteria: the last name of the first author of the abstract, the last name of the last author of the abstract, and 3 keywords from the abstract title in the topic search field. All search terms (author names and keywords) were concatenated by an “and” term, and keywords were selected by ranking. This ranking was accomplished by counting the number of times words appeared in all of the abstract titles. Commonly used title words such as “and” and “the” received low rankings, whereas rarely used terms received higher ranks. Words longer than 4 characters were truncated by last character and extended with an asterisk for the purpose of generalizing the search criteria. We further created a brief list of priority phrases for the search (Table I in the online-only Data Supplement). If one of these words/phrases were in the abstract title, the search would be forced to up-rank them. The program allowed ISI to refine the search result and included only the article document type, which excluded results from conferences, letters, reviews, patents, etc.
A second iteration of the search was performed automatically if the search returned >1 corresponding publication. Then the search was rerun automatically including an additional ranked keyword. A third iteration was carried out if there was still >1 publication found or if the publication was identified for a different abstract. This iteration included adding the second and third authors on the abstract to the search.
For abstracts with a corresponding publication, we retrieved the information on date of publication, journal name, title, authors, and number of times cited. For the minority of abstracts (5.7%) with >1 identified publication, we classified the publication on the basis of that first published. Journal names were then automatically paired with information from ISI Journal Citation Reports to assess the impact factor (IF) for the journal. We used the 5-year averaged journal IF for our analyses as a surrogate for the quality of the published article. If the 5-year IF average was not available, the 2010 IF was used instead.
To test the applied methodology, we conducted a manual search of 200 abstracts in a blinded fashion to assess the precision of the finalized search algorithm. Using an automated assignment tool in the SAS software package (SAS 9.2, SAS Institute, Cary, NC), we randomly assigned abstracts from each of the major clinical categories for review (ie, basic science, clinical science, and population science). A medical doctor/clinical investigator (E.L.F.) conducted a manual search for a corresponding article on the basis of the title of the abstract and the author list. This procedure was also used in the development and iterative process of fine tuning of the algorithm. Sensitivity, specificity, positive predictive value, and negative predictive value were calculated with the use of the manual search as the “gold standard.”
We examined the overall number, IF, and timing of abstract publication. We also further classified these items by countries, by calendar year, and by scientific category (basic science, clinical science, and population science). We used the Kruskal-Wallis test for continuous variables and χ2 tests for categorical variables to test for differences between groups. For time-to-publication analyses, we performed unadjusted Kaplan–Meier curves comparing conferences and tested for differences between them with the log-rank test. We calculated the average monthly IF of published articles and illustrated the relationship with time after presentation. A linear regression line and 95% confidence limits were calculated to illustrate time trends for quality of published articles after conferences. Multivariable logistic regression was performed to identify factors associated with publication (versus no publication), and all available variables from the abstracts were included in the model. For country effects, we included only countries with >5 publications in the model to have meaningful estimates. We also modeled associations with high-impact publication (defined as IF ≥10 versus IF <10) among those abstracts that were published. A value of P<0.05 was considered statistically significant.
From 2006 to 2008, a total of 27 564 scientific abstracts were presented at the AHA, ACC, and ESC annual scientific meetings. Of these, 356 abstracts were excluded from the analysis because of double entries, making the final number of presented abstracts for our study 27 208. These included 11 365 from AHA, 5005 from ACC, and 10 838 from ESC.Table 1shows the characteristics of the abstracts in total and by conference. Overall number of abstracts tended to decline from 2006 to 2008. ACC abstracts included more classified as population science (23%) than ESC (15%) or AHA (10%). In contrast, AHA had the most basic science of the 3 conferences (38%) compared with ACC (15%) and ESC (11%). The distribution of abstracts by country to each meeting is supplied in Figure I in the online-only Data Supplement.
Validation of Search Algorithm
We assumed that a manual search was the gold standard, so we conducted a blinded manual validation of the search algorithm. A total of 200 randomly selected abstracts were reviewed; 80 abstracts were true-positive records, 11 abstracts were false-positive records, 102 abstracts were true-negative records, and 7 abstracts were false-negative records. Hence, our search performed well with a sensitivity of 92% (95% confidence interval [CI], 0.83–0.96) and a specificity of 90% (95% CI, 0.82–0.95). The negative predictive value was 94% (95% CI, 0.86–0.97), meaning that if the search did not find a corresponding publication, one can be reasonably certain of this result. The positive predictive value was 88% (95% CI, 0.79–0.94).
Publication of Abstracts
Table 2shows the results derived from the search algorithm and the characteristics of the identified published articles. Overall, 30.6% of presented abstracts were found to be available in peer-reviewed journals within 2 years of the conference: 34.5% for AHA, 29.6% for ACC, and 27.0% for ESC (P for unadjusted difference between groups <0.0001). This stayed relatively stable from the 2006 to 2008 meetings (Figure 1), although there was a statistically significant trend for an overall increase in publication rates from 2006 to 2008 (P=0.042 for trend over time). This was due to a trend seen for ESC (P for trend over time=0.004), whereas neither AHA (P for trend=0.66) nor ACC (P for trend=0.2) was associated with differences in publication rates.Figure 2 illustrates the time to publication for the respective conferences (P for difference between conferences <0.0001). The curves separate at 6 months and were significantly different at 2 years.
The number of abstracts per scientific category differed substantially between the groups, yet publication rates by scientific categories suggested a similar relationship between conferences; a similar proportion of clinical science and population science abstracts were published at 2 years, whereas basic science abstracts were published at a higher rate than the other categories (P for difference between groups <0.0001;Table 2).
The identified articles were published in 683 different journals. Table II in the online-only Data Supplement shows the number of publications per journal for the 30 journals with the most published articles. Overall, 23.4% of publications were in a journal having an IF >10, with both ACC and AHA abstracts generating a greater amount of high-impact publications than ESC (P<0.0001;Table 2). The median IF of all publications was 4.2 (interquartile range [Q1–Q3], 3.2–7.5), and basic science had the highest median IF compared with the other categories: basic science, 5.6 (Q1–Q3, 3.9–10.2); clinical science, 3.9 (Q1–Q3, 2.7–6.4); and population science, 4.2 (Q1–Q3, 3.0–7.3; overall P for difference <0.0001, Kruskal-Wallis test).
High-impact publications were generally published closer to the conference compared with lower-impact publications, and the average IF declined with time after the conference (Figure 3). Median time to publication was 11 months for high-impact publications (Q1–Q3, 6–17 months) compared with 13 months (Q1–Q3, 7–19 months) for lower-impact publications (P<0.0001). There was no statistically significant difference in time to publication for the major scientific categories; the median time to publication was 12 months (Q1–Q3, 6–19 months) for basic science, 13 months (Q1–Q3, 7–18 months) for clinical science, and 12 months (Q1–Q3, 6–18 months) for population science (P for difference=0.16).
Factors Associated With Publication
Figure 4shows results from the logistic regression analysis of factors associated with publication. Countries were also entered into the model, and these results are shown separately in Figure 5. After adjustment for scientific category, calendar year, country, and number of authors, AHA was associated with a 24% higher odds of publication compared with ESC (adjusted odds ratio [OR], 1.24; 95% CI, 1.16–1.34), whereas ACC was associated with a publication odds similar to that of ESC (adjusted OR, 1.04; 95% CI, 0.95–1.13). For AHA versus ACC, the model yielded an OR of 1.20 (95% CI, 1.11 to 1.29). Clinical science and population science were significantly associated with similar and lower rates of publication than basic science. This did not change after exclusion of ACC data because ACC did not have a standardized scientific categorization. Increments in number of authors and countries were associated with an increased chance of publication. In an examination of a high-impact publication as the outcome relative to a low-impact publication, the AHA continued to have a higher adjusted publication record than ESC (adjusted OR for AHA versus ESC, 1.56; 95% CI, 1.35–1.81); however, more ACC abstracts were published in high-impact publications compared with ESC (OR, 1.40; 95% CI, 1.17–1.67). This was opposite of the trend seen with publication in which ACC and ESC showed a similar relationship. A high IF was also associated with more authors on the abstract (adjusted OR, 1.10; 95% CI, 1.08–1.12) per increment of 1 coauthor. Again, relative to basic science, both clinical science and population science were associated with a lower likelihood of having a high-impact publication (OR, 0.67; 95% CI, 0.59–0.75) for clinical versus basic science and for population versus basic science (OR, 0.62; 95% CI, 0.51–0.74).
Figure 5 provides 5-year publication rates for those abstracts from 2006 conferences. Although there was some cumulative increase in the number of publications during years 3 to 5 after the conference, the publication rates slowed dramatically. Specifically, abstract publication rates decreased 3-fold after 2 years compared with years 0 to 2. For the ACC, the 5-year publication rate was 42.6%; at AHA, it was 49.7%; and at ESC, it was 37.6%.
This study used a novel informatics approach for evaluating the scientific yield of major medical meetings. We found that only approximately 1 in 3 presented abstracts was published within 2 years of initial presentation at the ACC, AHA, or ESC international cardiovascular meeting. These publication rates varied significantly across major meetings and remained <50% by up to 5 years after the conferences.
The annual scientific sessions organized by the AHA, ACC, and ESC are the largest and most influential medical meetings in cardiology with science from the 3 core branches of medical research: basic science, clinical science, and population science. Overall, approximately one third of the presented abstracts were published within the following 2-year period. These findings are similar to those in other fields of medical research noted in small prior studies from the 1980s and1990s.1 One prior study has examined cardiology meetings and found publication rates varying from 49% to 59%; however, this study was conducted in 1980 and assessed the outcome of only 276 randomly selected abstracts.2 A meta-analysis also has shown a relatively low publication rate in the 2 years after the conference, but the final publication rate was close to half after 9 years.4 It is unknown whether differences in publication rates between prior studies and ours reflect real changes over time or differences in the methodological approach. Importantly, we believe that a delay of >2 years is far from optimal although common in the current system for publication of research findings. Nevertheless, our approach had good validity compared with a manual search, and the estimated rates are based on a much larger sample of abstracts.
Our results showed overall differences in publication rates among the AHA, ACC, and ESC meetings, and these differences were still evident after accounting for imbalances in the scientific profiles of the conferences. The AHA meeting was associated with the highest publication rate and the highest IF for subsequent published articles. This association held true even after adjustment for the contribution from basic science compared with other scientific categories. Prior reports/studies from the AHA and ESC have shown that ≈26% to 30% and 35% to 38% of submitted abstracts, respectively, are accepted for presentation.12,13 We were unable to find published data for the ACC. Differences in acceptance rates for presentation could potentially explain some of the variance between the conferences that was seen in our study. Perhaps a lower acceptance rate is indicative of higher quality, which, in turn, could translate into better publication rates. In synthesis, ≈30% of submitted abstracts are being accepted for presentation at the ACC, AHA, or ESC, and ≈30% of these abstracts turn into published manuscripts within 2 years of their initial meeting presentation. We were not able to assess publication rates for those abstracts not accepted for presentation, but one would assume that this rate is even lower, as previously suggested by von Elm et al.14
Time to publication was similar between conferences and between scientific categories. Although basic science generally published with a higher rate compared with population science and clinical science, rates of publication according to scientific category were similar. Lack of publication does not seem to be associated with a specific type of research within cardiology, although basic science was associated with higher-IF publications. This finding is similar to prior findings that surgery and medicine conferences have similar publication rates.1 An effort to increase the publication rate could seemingly be targeted broadly and implemented widely. It appears that the dissemination of knowledge from meetings is a long and immature process, as documented by the declining average IF with time from conference.
Many factors are involved in the pathway for traditional peer-reviewed publication, yet it appears that in this era of social media and real-time sharing of knowledge, sharing of new scientific findings is flawed by old habits and time-consuming steps not designed to benefit either patients or researchers. Although the peer-review system is frequently helpful and occasionally even extremely beneficial for researchers, we share the perspective with others that this system needs a fresh look toward innovative ways of improvement.15⇓–17 Ross et al12 have previously shown that various core factors in peer-review systems are important for acceptance and independent of the work quality (ie, favoring authors from the United States, English-speaking countries outside the United States, and prestigious academic institutions). We also found that these factors were associated with a higher likelihood of publication and a high-impact publication. Our results could, in turn, also be explained by an abstract review and acceptance process that is too rough and does not hold up in terms of quality work ready for publication. One important feature of a researcher is his or her ability to see projects through to the end, yet our study suggests that a minority of cardiovascular researchers see abstracts as such projects. On a provocative note, abstracts may serve as vessels for obtaining funding for clinicians to go to conferences; hence, the science may not be the main purpose of the conference for the participant. As a result, the science may be de-emphasized or down-prioritized after the conference is over. Scientific communities and journals need to think of more contemporary solutions for patients to benefit from new knowledge.6,7
Although conferences allow abstracts public airing and media attention, we find it perplexing that two thirds of these abstracts will not be published within a 2-year period after the meeting. Furthermore, those abstracts that successfully become published manuscripts are published with a substantial lag period. To avoid these issues, some professional societies (ie, Southern Thoracic Surgical Association) require the investigators to send a full manuscript describing the work in detail if their abstract is chosen for presentation. This is 1 approach that would eliminate abstracts that are incomplete and would not stand the test of thorough peer review. We believe that the translation of one third of abstracts into publications by 2 years after presentation and fewer than one half by 5 years after presentation is far from ideal. Conversely, we recognize that one third may be appropriate if all abstracts do not represent thoroughly conducted science that will benefit patients and the scientific community. Studies fundamentally flawed would, we hope, be corrected or stopped in the peer-review system, but we (as a scientific community) have to be more aware of the knowledge-dissemination problem. Regardless, because at least one half of accepted conference abstracts were not published, we believe there is room for improvement in the number and timeliness of publications. Future studies should examine whether abstracts with higher reviewer ratings are published more frequently than those with lower ratings. One potential solution is the movement toward open-source publishing.17 The need for more modern approaches to the publication of scientific findings is further emphasized by cross-institutional initiatives and collaborations for knowledge sharing.18–20 There is an increasing need among researchers for fruitful collaboration to build “meta-knowledge,”18 it has been suggested that the current peer-reviewed publication system is unhelpful in this regard.17
Our study had several limitations. To the best of our knowledge, our study is the first to evaluate data from major scientific meetings using an automatic computerized algorithm. Data were collected retrospectively from publicly available sources and were found to be in general accordance with what has been reported from the professional societies. Because we used an automatic informatics approach for our study, we also validated this method. We used a manual search as the gold standard to assess the performance of the algorithm, but this implies that the manual search is a true gold standard, which could be uncertain; as a result, calculated sensitivity and specificity may not be completely accurate. Overall, the algorithm performed well. We were not able to assess the type of presentation (oral presentation versus poster presentation) and university power ranking because of unavailable data. We did not assess any investigator-specific covariates (eg, H index) other than country in which the researcher resided. In addition, the ISI system provides the date of publication as the date for when the article was in print; therefore, for journals with early online publication, time from presentation to publication may be overestimated. Our primary results examined a 2-year follow-up period that was arbitrary and driven by the availability of follow-up information on our cohort. As noted, we considered publications up to 5 years after presentation but only saw marginal increases in the overall publication rates. Our study examined the relationship between abstract presentation and publication, but we did not explore the converse question, How many high-quality articles were once presented as abstracts at scientific congresses? Furthermore, our results are based on a single field in medicine, and whether these results are generalizable to medicine and other branches of research is unknown.
This study has shown that the scientific yield, as measured by number of abstracts from the 3 major cardiovascular conferences resulting in a peer-reviewed article, was about 1 in 3. Abstracts presented at the AHA conferences were associated with a higher likelihood of publication and a higher IF compared with those of the ACC and ESC. Efforts are needed to ensure more complete sharing of knowledge presented at large medical meetings, but quality should never be euthanized. Such efforts should focus on increasing the publication rate broadly but also reducing the time from abstract presentation to publication. Patient care is not improved through incomplete and preliminary data from abstracts but is informed and changed on the basis of information found in fully published reports after state-of-the-art peer review.
Sources of Funding
This work was supported by an award from the American Heart Association–Pharmaceutical Roundtable and from David and Stevie Spina.
Drs E.L. Fosbøl and Eapen report being a member of the AHA, ACC, and ESC. Dr Harrington reports serving as vice chair of the scientific sessions of the AHA. He is a Board of Trustees member for ACC and is on multiple ACC and AHA scientific committees. He is a fellow of AHA, ACC, and ESC. Dr Peterson reports being a fellow of the AHA and the ACC. He is the principal investigator of the data coordinating center for multiple clinical registries funded and run by ACC (the National Cardiovascular Data Registries) and AHA (AHA-Get With The Guidelines registries). Dr P.L. Fosbøl reports no conflicts.
We would like to thank Erin LoFrese, MS, for her editorial contributions to this manuscript. She did not receive compensation for her contributions apart from her employment at the institution where this study was conducted.
The online-only Data Supplement is available with this article at http://circ.ahajournals.org/lookup/suppl/doi:10.1161/CIRCULATIONAHA.112.120535/-/DC1.
- Received May 25, 2012.
- Accepted October 24, 2012.
- © 2012 American Heart Association, Inc.
- Byerly WG,
- Rheney CC,
- Connelly JF,
- Verzino KC
- Scherer RW,
- Langenberg P,
- von Elm E
- Green M
Circulation archives. Circulation Web site. http://circ.ahajournals.org/content/by/year. Accessed March 29, 2012.
JACC archives. Journal of the American College of Cardiology Web site. http://content.onlinejacc.org/archive/. Accessed March 29, 2012.
European Heart Journal archives. European Heart Journal Web site. http://eurheartj.oxfordjournals.org/content/by/year. Accessed March 29, 2012.
ISI Web of Knowledge. Web of Knowledge Web site. http://apps.webofknowledge.com/WOS_GeneralSearch_input.do?highlighted_tab=WOS&product=WOS&last_prod=WOS&SID=4EG7gB3GNMLo5KHh1c@&search_mode=GeneralSearchFirst. Accessed September 20, 2011.
Abstracts selected for ESC 2008 30 August-3 September 2008, Munich-Germany. Eur Heart J. 2008;29:iii–vi.
- Gitlin JM
- Lin T
- Evans JA,
- Foster JG
- Jones BF,
- Wuchty S,
- Uzzi B
- Wuchty S,
- Jones BF,
- Uzzi B
Major scientific conferences remain important venues for sharing novel, rapidly developing medical finds among colleagues, yet results presented are preliminary and not considered final until published in full detail in a peer-reviewed journal. Unfortunately, the completeness and speed of this process are not known, although prior meta-analyses gathering these data indicated a long delay in publication. To the best of our knowledge, no prior study has systematically assessed the scientific yield of medical conferences, and no comparative data exist for the major scientific sessions in cardiology. In hopes of addressing these gaps, we developed an algorithm for searching the published literature and then conducted a comprehensive assessment to determine how many abstracts presented at the 3 major international cardiovascular conferences (the American Heart Association, American College of Cardiology, and European Society of Cardiology) were published as peer-reviewed papers within 2 years of their initial meeting presentation. We found that the number of abstracts resulting in a peer-reviewed manuscript was about 1 in 3. We believe that efforts need to be made to increase the publication rate and to decrease the time from presentation to publication. We believe that our study is significant to the field of cardiology in that it highlights the importance of better understanding publication barriers and the need to facilitate the rapid dissemination of scientific discovery to the patient bedside.