Is Anybody Listening?
When I was in my teens, I built a small radio transmitter from individual components. It was my first homebrew project, and I was anxious to see whether I could connect with someone far away from my home in St. Louis. After hours on the air desperately sending “CQ CQ CQ this is WAØAJX” in Morse code, I gave up. I had raised no one. Despite my broadcasts, nobody, it seemed, was aware of my work. I set the project aside. A few months later, I received a postcard from Mexico. Someone had picked up my signal, but I could not hear his response. Although I had succeeded in transmitting, the time frame and the mechanism of response were not what I had expected.
Article see p 1755
What does this have to do with cardiovascular research? In this issue of Circulation, an article by Ranasinghe and coworkers looks at “poorly cited articles” in peer-reviewed cardiovascular journals.1 The authors’ premise is that articles that are infrequently cited are of limited impact and therefore “may reflect inefficiency and waste in the research enterprise.” Their key assumption is that impact can be measured by citations within the first 5 years from publication. If your work is important to others, they will cite it when they report their own work; this will happen within a few years of publication. This assumption may often be true, but is it also true that if no one rapidly cites your work, it is unimportant? By their statement about “inefficiency and waste in the research enterprise,” Ranasinghe et al seem to think so. But just as I had to wait to learn that my meager radio signal had reached a faraway place, recognition of the importance of research may take longer than expected.
In this article, the authors report on the outcome of an organized search for “poorly cited research.” To do so, they used the Scopus database from Elsevier to identify articles published from 1997 through 2007 in 222 peer-reviewed cardiovascular journals. They examined the citations to all published articles, stratified by journal, and counted the number of times each article was cited in the 5 years after its publication. They then applied their case definition of a poorly cited article, <5 citations in the 5 years after publication, to these data. With this information, and counts of all the articles published by a given journal, they calculated the fraction of articles in that journal that were poorly cited.
Overall they found that about half of all published cardiovascular articles were poorly cited, with ≈1 in 6 not cited at all within 5 years. The other half, which they term “well cited,” was cited a median of 17 times over the 5 years after publication. Among the top cardiovascular journals, the fraction of poorly cited articles ranged from 1% (Circulation Research) to 17% (Journal of Hypertension). In other journals, however, more than half the published material met the case definition of poorly cited.
They conclude that “poorly cited articles are a cause for concern” and propose methods for reducing waste in research. If you agree with their underlying premise—if work is not cited within 5 years of publication it has been a waste of effort and resources—the data they bring to the table support you. But is their premise reasonable? I think not.
One well-known proof is the work of Gregor Mendel, whose research on allele sorting in pea plants did not garner much interest when it was published in 1866.2 Although citation databases from the 1860s and 1870s are not wholly reliable, Mendel’s work probably fit Ranasinghe’s case definition of poorly cited research; most agree that Mendel’s work was largely ignored for 35 years. Then, as modern genetics began to emerge, his work was discovered and we still today use the term Mendelian inheritance to describe the inheritance of clearly dominant or recessive traits.
The cynic will say that in the 1860s and 1870s the scientific world was not as closely connected as it is today. Mendel’s work may have been published in an established European journal, but very few people knew about it. Consider, then, another example, one that for me is closer to home. As an applied physics student at Tufts University, I studied under Alan Cormack, who, although his main interest was nuclear physics, had done some work on the problem of how to infer the shape and characteristics of an object embedded inside another object from the absorption characteristics of an energy beam that penetrated both. His theoretical and experimental work on this problem was published in 1963 and 1964.3,4 In his own words, “There was virtually no response.”5 Each of these publications met the case definition of a poorly cited article; his work could have been labeled waste research. But the calculus changed dramatically when Hounsfield and coworkers, who were developing the first generation of computerized tomographic scanners, needed an answer to the very problem Cormack had solved. Cormack and Hounsfield shared the Nobel Prize in Physiology or Medicine in 1979. Were Cormack’s published articles from 1963 and 1964 then waste research?
The case definition of Ranasinghe and coworkers allowed them to designate an article as poorly cited with absolute accuracy and without having to read the work they were labeling. I have not examined their entire corpus of poorly cited articles, but my curiosity got the better of me and I looked at a few published articles that met their definition to see whether in my eyes they were waste research. Sure enough, I too would have assigned this label to some, but others were well-done studies. For example, in a study cited only 4 times since it was published in 2006, the investigators used extant cardiac surgery databases to show that patients with diabetes mellitus had poorer improvement in physical functioning after cardiac surgery than did patients without diabetes mellitus.6 Despite the lack of citations, this is clearly an issue that health care professionals should keep in mind when counseling patients about potential cardiac surgery. In another article, cited only twice since it was published in 2007, the investigators asked whether a Food and Drug Administration public health notification about drug-eluting cardiac stents had an impact on the implantation of such stents.7 They found that it did, but only temporarily. Although these studies were labeled waste research, in each case, the investigators framed a question of importance to them, outlined an approach to test it, and reported the outcome of their experiment and whether it answered or failed to answer their question. My conclusion is that citations may be a surrogate for community interest but not a surrogate for science well done.
Because all published articles need not be research, consider case reports that are a staple of medical publishing and often of value to other clinicians; case reports are counted as articles in the Scopus database. As a practical example, consider the “Case Records of the Massachusetts General Hospital”: these are clinical case reports that we publish 40 weeks a year in the New England Journal of Medicine, of which I am the editor-in-chief. In 2010, we published 40 Case Records that by March 2015 had been cited in the aggregate 124 times or 3.1 citations per case; only 9 of the 40 had been cited >5 times. Are these waste publications? Our readers do not think so, because they are one of the most valued features on our Website; they are used for teaching medicine to students around the world.
Science well done is its own reward. After a well-done experiment, you understand something in a way that you had not understood it before. With answer in hand, you go on to the next question. My work with the transmitter was not a true scientific experiment. It was a technical one with the implicit question, “Do I have the engineering prowess to build a functional radio transmitter on my own?” Looking at the immediate response to my test, the answer was no. But someone was listening and eventually responded. My time and resources were not wasted.
Before we decide that an endeavor is unworthy of the time and resources it consumed only because others have not quickly taken note of it in a publication, we should take a careful look at its intrinsic worth.
I thank Generva Pittman for doing background research for this article.
Dr Drazen is a full-time employee of the Massachusetts Medical Society, owner and publisher of the New England Journal of Medicine.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
- © 2015 American Heart Association, Inc.
- Ranasinghe I,
- Shojaee A,
- Bikdeli B,
- Gupta A,
- Chen R,
- Ross JS,
- Masoudi FA,
- Spertus JA,
- Nallamothu BK,
- Krumholz HM.
- Mendel JG.
- Lindsten J
- Cormack AM.