Editorial

The Promise and Problems of Meta-Analysis

John C. Bailar, III, M.D., Ph.D.

N Engl J Med 1997; 337:559-561August 21, 1997DOI: 10.1056/NEJM199708213370810

Article

Meta-analysis has acquired a substantial following among both statisticians and clinicians. The technique was developed as a way to summarize the results of different research studies of related problems. Meta-analysis may be applied even when the studies are small and there is substantial variation in the specific issues studied, the research methods applied, the source and nature of the study subjects, and other factors that may have an important bearing on the findings. In this issue of the Journal, LeLorier et al.1 compare the findings of 12 large randomized, controlled trials with the results of meta-analyses of the same problems. They find important discrepancies. When a large randomized, controlled trial — commonly considered the gold standard for determining the effects of medical interventions — disagrees with a meta-analysis, what should the reader conclude? Perhaps more important, when only one of the two tools is used, how much uncertainty should the reader add to the confidence limits and other statistical measures of uncertainty reported by the author?

The core of meta-analysis is its systematic approach to the identification and abstracting of critical information from research reports. Doing a meta-analysis correctly demands expertise in both the method and the substance and hence almost always requires collaboration between clinicians and an experienced statistician. The questions must be defined carefully to maximize the relevance of the reports to be included and to reduce uncertainties about procedures. The investigators must then try to find every relevant report by searching data bases, reviewing bibliographies, and asking widely about unpublished work. The collected reports are then winnowed to the few (often less than 10 percent) that meet the requirements for the meta-analysis. The reports must be searched carefully to identify problems and validate the quantitative findings of interest. These findings must be expressed on a common scale (often as odds ratios), and some reports may have to be dropped for lack of information. Those doing a meta-analysis may also abstract information from each report to produce a quantitative measure of research quality. Each of the individual quantitative estimates must be scrutinized for problems, and this may require the efforts of a range of specialists. When the analysis is completed and submitted for publication, the editor and the reviewers must assure themselves of its quality. A rigorous technical review of a meta-analysis requires the reviewer to identify, reabstract, and interpret a fair sample of the original papers. Very few editors and reviewers will do this, which may be one reason why there are so many poor meta-analyses in the literature.

Although some meta-analyses stop with the presentation and discussion of the results of the individual studies, many others proceed further and combine the results into a single, comprehensive “best” estimate, generally with statistical confidence bounds, that is meant to summarize what is known about the clinical problem. This last step — preparing and presenting a single estimate as the distillation of all that is known — is the one that has drawn the most criticism. This is because there are often biologic reasons, statistical evidence, or both, showing that the studies included in the meta-analysis have in fact measured somewhat different things, so that a combined estimate cannot be meaningful unless additional, doubtful assumptions are made. One such assumption is that the effects reported in the studies actually performed can be seen as a random sample of the effects observed in all possible studies that might have met the author's criteria. Unfortunately, there is little evidence to support an assumption such as this.

LeLorier et al.1 searched four leading general medical journals to identify all the large randomized, controlled trials (those with 1000 subjects or more) whose results were published from 1991 through 1994, then searched for meta-analyses of similar topics that were published before each trial. Twelve large randomized, controlled trials and 19 meta-analyses met their criteria. Because some of the trials and meta-analyses reported on more than 1 outcome, they studied a total of 40 outcomes. Overall, there was somewhat better than chance agreement between these meta-analyses and the subsequent large randomized, controlled trials, with kappa values in a range commonly considered to represent “fair-to-slight agreement.” In terms of an ordinary diagnostic test, the results could be described as average. The results obtained with the two approaches usually pointed in the same direction, and there were no cases in which they gave statistically significant results in opposite directions. However, the discrepancies with regard to the estimated size of an effect were sometimes quite substantial, and occasionally they differed significantly despite their agreement in direction. Stewart and Parmar2 have shown how some such differences can arise.

It is impossible to say, on the basis of present evidence alone, whether the results of a large randomized, controlled trial or those of a meta-analysis of many smaller studies are more likely to be close to the truth. Much depends on the details of both the research studies and the analyses. When both the trial and the meta-analysis seem to be of good quality, however, I tend to believe the results of the trial. A history of 40 years of generally successful use of randomized, controlled trials that have made important contributions to progress in many branches of medicine must not be overlooked. In addition, major problems with the implementation of meta-analyses have been common.3-6 There have been a wide variety of these, including failure of the investigator performing the meta-analysis to understand the basic issues, carelessness in abstracting and summarizing appropriate papers, failure to consider important covariates, bias on the part of the meta-analyst, and, perhaps most often, overstatements of the strength and precision of the results. It is not uncommon to find that two or more meta-analyses done at about the same time by investigators with the same access to the literature reach incompatible or even contradictory conclusions.7-10 Such disagreement argues powerfully against any notion that meta-analysis offers an assured way to distill the “truth” from a collection of research reports.

Other observers, including policy makers, also have reservations about meta-analyses, and there is some general concern about the credibility of the findings of meta-analysis. I know of no instance in medicine in which a meta-analysis led to a major change in policy before the time when a careful, conventional review of the literature led to the same change. Showing that a sequence of meta-analyses performed over time converged to have some value as published findings accumulated does not mean that it was the meta-analyses that gave a convincing answer to the particular clinical question.

In my own review of selected meta-analyses,3 problems were so frequent and so serious, including bias on the part of the meta-analyst, that it was difficult to trust the overall “best estimates” that the method often produces. On present evidence, we can generally accept the results of a well-done meta-analysis as a way to present the results of disparate studies on a common scale (as is shown by the two figures in the article by LeLorier et al.), but any attempt to reduce the results to a single value, with confidence bounds, is likely to lead to conclusions that are wrong, perhaps seriously so. I still prefer conventional narrative reviews of the literature, a type of summary familiar to readers of the countless review articles on important medical issues.

Meta-analysis may still be improved, by a combination of experience and theory, to the point at which its findings can be taken as sufficiently reliable when there is no other analysis or confirmation available, but that day seems to be well ahead of us. LeLorier et al. also imply, however, that large randomized, controlled trials should be regarded more circumspectly than published reports commonly suggest. We never know as much as we think we know.

John C. Bailar, III, M.D., Ph.D.
University of Chicago, Chicago, IL 60637

References

References

1. 1

   LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-542
   Free Full Text | Web of Science | Medline

2. 2

   Stewart LA, Parmar MK. Meta-analysis of the literature or of individual patient data: is there a difference? Lancet 1993;341:418-422
   CrossRef | Web of Science | Medline

3. 3

   Bailar JC III. The practice of meta-analysis. J Clin Epidemiol 1995;48:149-157
   CrossRef | Web of Science | Medline

4. 4

   Macarthur C, Foran PJ, Bailar JC III. Qualitative assessment of studies included in a meta-analysis: DES and the risk of pregnancy loss. J Clin Epidemiol 1995;48:739-747
   CrossRef | Web of Science | Medline

5. 5

   Shapiro S. Meta-analysis shmeta-analysis. Am J Epidemiol 1994;140:771-778
   Web of Science | Medline

6. 6

   Shapiro S. Is meta-analysis a valid approach to the evaluation of small effects in observational studies? J Clin Epidemiol 1997;50:223-229
   CrossRef | Web of Science | Medline

7. 7

   Kerlikowske K, Grady D, Rubin SM, Sandrock C, Ernster VL. Efficacy of screening mammography: a meta-analysis. JAMA 1995;273:149-154
   CrossRef | Web of Science | Medline

8. 8

   Smart CR, Hendrick RE, Rutledge JH III, Smith RA. Benefit of mammography screening in women ages 40 to 49 years: current evidence from randomized controlled trials. Cancer 1995;75:1619-1626
   CrossRef | Web of Science | Medline

9. 9

   Rosenfeld RM, Post JC. Meta-analysis of antibiotics for the treatment of otitis media with effusion. Otolaryngol Head Neck Surg 1992;106:378-386
   Web of Science | Medline

10. 10

    Williams RL, Chalmers TC, Stange KC, Chalmers FT, Bowlin SJ. Use of antibiotics in preventing recurrent acute otitis media and in treating otitis media with effusion: a meta-analytic attempt to resolve the brouhaha. JAMA 1993;270:1344-1351[Erratum, JAMA 1994;271:430.]
    CrossRef | Web of Science | Medline

Citing Articles (124)

Citing Articles

1. 1

   Statistical Foundations of Clinical Decisions. In: Jekel's Epidemiology, Biostatistics, Preventive Medicine, and Public Health. Elsevier, 2014:93-104.
   

2. 2

   Nandu Goswami, Jerry J. Batzel, Gilles Clément, T. Peter Stein, Alan R. Hargens, M. Keith Sharp, Andrew P. Blaber, Peter G. Roma, Helmut G. Hinghofer-Szalkay. (2013) Maximizing information from space data resources: a case for expanding integration across research disciplines. European Journal of Applied Physiology 113:7, 1645-1654
   

3. 3

   Massimo Di Maio. (2013) The role of meta-analysis in defining clinical practice in advanced non-small-cell lung cancer. Clinical Investigation 3:6, 523-530
   

4. 4

   Salah Fallatah, Eugene Wai, Christopher S. Baily. (2013) The value of adding posterior interbody fusion in the surgical treatment of degenerative lumbar spine disorders: A systematic review. The International Journal of Spine Surgery
   

5. 5

   Quoc-Dien Trinh, Anders Bjartell, Stephen J. Freedland, Brent K. Hollenbeck, Jim C. Hu, Shahrokh F. Shariat, Maxine Sun, Andrew J. Vickers. (2013) A Systematic Review of the Volume–Outcome Relationship for Radical Prostatectomy. European Urology
   

6. 6

   Georgina Guilera, Maite Barrios, Juana Gómez-Benito. (2013) Meta-analysis in psychology: a bibliometric study. Scientometrics 94:3, 943-954
   

7. 7

   Salah Fallatah. (2013) Posterior inter-body fusion in lumbar spine surgery: A systematic review. Journal of Taibah University Medical Sciences
   

8. 8

   P. C. McGovern, A. Quintana. (2012) Patient-Level Data Provide Additional Insight for Tigecycline All-Cause Mortality Meta-analysis. Clinical Infectious Diseases 55:12, 1739-1740
   

9. 9

   Mahmoud A. Abdel-Aleem, Ahmed Nasr. (2012) Synthesizing a proper meta-analysis: Are we almost there?. Middle East Fertility Society Journal 17:4, 293-295
   

10. 10

    Bibliography. In: A Statistical Guide for the Ethically Perplexed. Chapman and Hall/CRC, 2012:493-532.
    

11. 11

    D R Mould. (2012) Model-Based Meta-Analysis: An Important Tool for Making Quantitative Decisions During Drug Development. Clinical Pharmacology & Therapeutics 92:3, 283-286
    

12. 12

    P. Prasad, J. Sun, R. L. Danner, C. Natanson. (2012) Excess Deaths Associated With Tigecycline After Approval Based on Noninferiority Trials. Clinical Infectious Diseases 54:12, 1699-1709
    

13. 13

    S. A. Daneshvar, S. H. Rahimtoola. (2012) Still searching for the elusive Scarlet Pimpernel of today. European Heart Journal
    

14. 14

    Zhiping Zhou, Xingxiang Zhang, Shoujin Tian, Yaogang Wu. (2012) Minimally Invasive Versus Conventional Dynamic Hip Screw for the Treatment of Intertrochanteric Fractures in Older Patients. Orthopedics
    

15. 15

    Brian M. Ilfeld, Christopher L. Wu. (2012) Meta-Analysis. Regional Anesthesia and Pain Medicine 37:6, 571-573
    

16. 16

    Mei Yang, Anne M Robinson, Paul F Pollack. (2011) When Meta-Analysis Misleads. The American Journal of Gastroenterology 106:11, 2043-2043
    

17. 17

    John A. Bittl. (2011) Interventional Strategies for ST-Segment Elevation Myocardial Infarction and Multivessel Coronary Artery Disease. Journal of the American College of Cardiology 58:7, 712-714
    

18. 18

    Lars Andersson, Jens O. Andreasen. (2011) Important considerations for designing and reporting epidemiologic and clinical studies in dental traumatology. Dental Traumatology 27:4, 269-274
    

19. 19

    Sune Bo, Ahmad Abu-Akel, Mickey Kongerslev, Ulrik Helt Haahr, Erik Simonsen. (2011) Risk factors for violence among patients with schizophrenia. Clinical Psychology Review 31:5, 711-726
    

20. 20

    R. F. Soll. (2011) Meta-analysis in Neonatal Perinatal Medicine. NeoReviews 12:1, e8-e12
    

21. 21

    W.E. Herfel, Y. Gao, D.J. Rodrigues. Chinese Medicine and Complex Systems Dynamics. In: Philosophy of Complex Systems. Elsevier, 2011:675-719.
    

22. 22

    Sumit Mohan, Jai Radhakrishnan. (2010) Do meta-analyses in nephrology change the way we treat patients?. Kidney International 78:11, 1080-1087
    

23. 23

    Adam Felton, Emma Knight, Jeff Wood, Charlie Zammit, David Lindenmayer. (2010) A meta-analysis of fauna and flora species richness and abundance in plantations and pasture lands. Biological Conservation 143:3, 545-554
    

24. 24

    Ming Liu, Zhiming Yang, Fuxing Pei, Fuguo Huang, Shiqiang Chen, Zhou Xiang. (2010) A meta-analysis of the Gamma nail and dynamic hip screw in treating peritrochanteric fractures. International Orthopaedics 34:3, 323-328
    

25. 25

    Stacey J. Bell, Gregory T. Grochoski. (2008) How Safe is Vitamin E Supplementation?. Critical Reviews in Food Science and Nutrition 48:8, 760-774
    

26. 26

    Marcel Zwahlen, Andrew Renehan, Matthias Egger. (2008) Meta-analysis in medical research: Potentials and limitations. Urologic Oncology: Seminars and Original Investigations 26:3, 320-329
    

27. 27

    John H. Powers. (2008) Noninferiority and equivalence trials: deciphering ‘similarity’ of medical interventions. Statistics in Medicine 27:3, 343-352
    

28. 28

    Jonathan C Ipser, Catherine M Kariuki, Dan J Stein. (2008) Pharmacotherapy for social anxiety disorder: a systematic review. Expert Review of Neurotherapeutics 8:2, 235-257
    

29. 29

    Martin J. Tobin, Amal Jubran. (2008) Meta-analysis under the spotlight: Focused on a meta-analysis of ventilator weaning*. Critical Care Medicine 36:1, 1-7
    

30. 30

    P. G. Sand. (2007) Comments on the paper by D. Li and L. He: Meta-analysis shows association between the tryptophan hydroxylase (TPH) gene and schizophrenia. Human Genetics 122:3-4, 409-411
    

31. 31

    J. H. Powers. (2007) Increasing the Efficiency of Clinical Trials of Antimicrobials: The Scientific Basis of Substantial Evidence of Effectiveness of Drugs. Clinical Infectious Diseases 45:Supplement 2, S153-S162
    

32. 32

    Craig S. Radnay, Kevin J. Setter, Locky Chambers, William N. Levine, Louis U. Bigliani, Christopher S. Ahmad. (2007) Total shoulder replacement compared with humeral head replacement for the treatment of primary glenohumeral osteoarthritis: A systematic review. Journal of Shoulder and Elbow Surgery 16:4, 396-402
    

33. 33

    John H. Powers. (2007) Diagnosis and Treatment of Acute Otitis Media: Evaluating the Evidence. Infectious Disease Clinics of North America 21:2, 409-426
    

34. 34

    S. S. Mahid, C. A. Hornung, K. S. Minor, M. Turina, S. Galandiuk. (2006) Systematic reviews and meta-analysis for the surgeon scientist. British Journal of Surgery 93:11, 1315-1324
    

35. 35

    D. A. GRANTZ, S. GUNN, H.-B. VU. (2006) O3 impacts on plant development: a meta-analysis of root/shoot allocation and growth. Plant, Cell and Environment 29:7, 1193-1209
    

36. 36

    Jaime Guzman, Rosmin Esmail, Kaija A Karjalainen, Antti Malmivaara, Emma Irvin, Claire Bombardier, Jaime Guzman. Multidisciplinary bio-psycho-social rehabilitation for chronic low-back pain. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 2006.
    

37. 37

    Dan J Stein, Jonathan C Ipser, Soraya Seedat, Dan J Stein. Pharmacotherapy for post traumatic stress disorder (PTSD). In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 2006.
    

38. 38

    Verena Henkel, Roland Mergl, Antje-Kathrin Allgaier, Ralph Kohnen, Hans-Jürgen Möller, Ulrich Hegerl. (2006) Treatment of depression with atypical features: A meta-analytic approach. Psychiatry Research 141:1, 89-101
    

39. 39

    Yulan Liang, Arpad Kelemen. (2006) Associating phenotypes with molecular events: recent statistical advances and challenges underpinning microarray experiments. Functional & Integrative Genomics 6:1, 1-13
    

40. 40

    Spencer S. Liu, Wyndam M. Strodtbeck, Jeffrey M. Richman, Christopher L. Wu. (2005) A Comparison of Regional Versus General Anesthesia for Ambulatory Anesthesia: A Meta-Analysis of Randomized Controlled Trials. Anesthesia & Analgesia1634-1642
    

41. 41

    Shayne Gad. Meta- Analysis. In: Statistics and Experimental Design for Toxicologists and Pharmacologists, Fourth Edition. CRC Press, 2005:167-193.
    

42. 42

    Donna F. Stroup, Stephen B. Thacker. Meta-Analysis in Epidemiology. In: Encyclopedia of Biostatistics. John Wiley & Sons, Ltd, 2005.
    

43. 43

    Jonathan M. Samet, Leon Gordis. Epidemiology, Overview. In: Encyclopedia of Biostatistics. John Wiley & Sons, Ltd, 2005.
    

44. 44

    David M. Stieb, Marc Smith Doiron, Philip Blagden, Richard T. Burnett. (2005) Estimating the Public Health Burden Attributable to Air Pollution: An Illustration Using the Development of an Alternative Air Quality Index. Journal of Toxicology and Environmental Health, Part A 68:13-14, 1275-1288
    

45. 45

    Siu-Sun Yao, Farooq A. Chaudhry. (2005) Assessment of Myocardial Viability with Dobutamine Stress Echocardiography in Patients with Ischemic Left Ventricular Dysfunction. Echocardiography 22:1, 71-83
    

46. 46

    J. Melzer, W. Rosch, J. Reichling, R. Brignoli, R. Saller. (2004) Meta-analysis: phytotherapy of functional dyspepsia with the herbal drug preparation STW 5 (Iberogast). Alimentary Pharmacology and Therapeutics 20:11-12, 1279-1287
    

47. 47

    Jose M. Ordovas, Dolores Corella. (2004) NUTRITIONAL GENOMICS. Annual Review of Genomics and Human Genetics 5:1, 71-118
    

48. 48

    Tho Bella Dinh-Zarr, Cynthia W Goss, Elizabeth Heitman, Ian G Roberts, Carolyn DiGuiseppi, Tho Bella Dinh-Zarr. Interventions for preventing injuries in problem drinkers. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 2004.
    

49. 49

    Stephen Bent, Kaveh G Shojania, Sanjay Saint. (2004) The use of systematic reviews and meta-analyses in infection control and hospital epidemiology. American Journal of Infection Control 32:4, 246-254
    

50. 50

    Thomas Rutledge, Cathy Loh. (2004) Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research. Annals of Behavioral Medicine 27:2, 138-145
    

51. 51

    Paris P Tekkis. (2004) Reply. Journal of the American College of Surgeons 198:3, 498-500
    

52. 52

    David S Fedson, Charles Liss. (2004) Precise answers to the wrong question: prospective clinical trials and the meta-analyses of pneumococcal vaccine in elderly and high-risk adults. Vaccine 22:8, 927-946
    

53. 53

    Rodrigo Arriagada, Cécile Le Péchoux, Jean-Pierre Pignon. (2003) Resected non-small cell lung cancer: need for adjuvant lymph node treatment? From hope to reality. Lung Cancer 42:2, 57-64
    

54. 54

    I. Lopez-Azpiazu, A. Sanchez-Villegas, L. Johansson, J. Petkeviciene, R. Prattala, M. A. Martinez-Gonzalez, . (2003) Disparities in food habits in Europe: systematic review of educational and occupational differences in the intake of fat. Journal of Human Nutrition and Dietetics 16:5, 349-364
    

55. 55

    Martin J. Tobin. (2003) Writing a Review Article for AJRCCM. American Journal of Respiratory and Critical Care Medicine 168:7, 732-734
    

56. 56

    MA Papadopoulos. (2003) Meta-analysis in evidence-based orthodontics. Orthodontics & Craniofacial Research 6:2, 112-126
    

57. 57

    MA Papadopoulos. (2003) Meta-analysis in evidence-based orthodontics. Orthodontics and Craniofacial Research 6:2, 112-126
    

58. 58

    Seymour H Levitt, Dorothee Aeppli, Mary Beth Nierengarten. (2003) Evidence-based medicine: its effect on treatment recommendations as illustrated by the changing role of postmastectomy irradiation to treat breast cancer. International Journal of Radiation Oncology*Biology*Physics 55:3, 645-650
    

59. 59

    (2003) ATS/ACCP Statement on Cardiopulmonary Exercise Testing. American Journal of Respiratory and Critical Care Medicine 167:2, 211-277
    

60. 60

    Neville W. Goodman. (2002) Evidence-Based Medicine Needs Proper Critical Review. Anesthesia & Analgesia 95:6, 1817
    

61. 61

    L Rajdev. (2002) Neoadjuvant and adjuvant therapy of non-small cell lung cancer. Surgical Oncology 11:4, 243-253
    

62. 62

    (2002) Metaanalysis, Pallidotomy, and Microelectrodes. Journal of Neurosurgery 97:5,
    

63. 63

    Pierluigi Granone, Steven Keller, Lucio Trodella, Alfredo Cesario, Stefano Margaritora, Venanzio Porziella, Salvatore Valente, Giuseppe Maria Corbo, Rolando Maria d'Angelillo. (2002) Postoperative radiotherapy in completely resected, early-stage, non-small-cell lung cancer: reflections and future directions.. The Lancet Oncology 3:10, 589-590
    

64. 64

    G. Marie Swanson, John C. Bailar. (2002) Selection and description of cancer clinical trials participants?Science or happenstance?. Cancer 95:5, 950-959
    

65. 65

    Fergus Macbeth, Jens Overgaard. (2002) Expert reviews, systematic reviews and meta-analyses. Radiotherapy and Oncology 64:3, 233-234
    

66. 66

    Ruth M Belin, Paul W Ladenson, Karen A Robinson, Neil R Powe. (2002) Development and use of evidence-based clinical practice guidelines for thyroid disease. Endocrinology and Metabolism Clinics of North America 31:3, 795-817
    

67. 67

    Michel E. Safar, Martin G. Myers, Frans Leenen, Roland Asmar. (2002) Gender influence on the dose-ranging of a low-dose perindopril–indapamide combination in hypertension: effect on systolic and pulse pressure. Journal of Hypertension 20:8, 1653-1661
    

68. 68

    Sanjay Sharma. (2002) Meta-Analysis of Clinical Trials Comparing Scleral Bucking Surgery to Pneumatic Retinopexy. Evidence-Based Eye Care 3:3, 125-128
    

69. 69

    Robert O Bonow. (2002) Myocardial viability and prognosis in patients with ischemic left ventricular dysfunction. Journal of the American College of Cardiology 39:7, 1159-1162
    

70. 70

    P. Rodger W. Gregson, Andrew G. Meal, Mark Avis. (2002) Meta-analysis: the glass eye of evidence-based practice?. Nursing Inquiry 9:1, 24-30
    

71. 71

    P Sjögren, A Halling. (2002) Medline search validity for randomised controlled trials in different areas of dental research. British Dental Journal 192:2, 97-99
    

72. 72

    J Guzmán, R Esmail, K Karjalainen, A Malmivaara, E Irvin, C Bombardier, Jaime Guzman. Multidisciplinary bio-psycho-social rehabilitation for chronic low-back pain. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 2002.
    

73. 73

    John G.F Cleland. (2002) Is aspirin ″the weakest link” in cardiovascular prophylaxis? The surprising lack of evidence supporting the use of aspirin for cardiovascular disease. Progress in Cardiovascular Diseases 44:4, 275-292
    

74. 74

    Mark J Eisenberg. (2001) Risk stratification for arrhythmic events: are the bangs worth the bucks?. Journal of the American College of Cardiology 38:7, 1912-1915
    

75. 75

    John G.F. Cleland, Joseph John, Tim Houghton. (2001) Does aspirin attenuate the effect of angiotensin-converting enzyme inhibitors in hypertension or heart failure?. Current Opinion in Nephrology and Hypertension 10:5, 625-631
    

76. 76

    John H. Rex, Thomas J. Walsh, Mary Nettleman, Elias J. Anaissie, John E. Bennett, Eric J. Bow, A. J. Carillo‐Munoz, Pascal Chavanet, Gretchen A. Cloud, David W. Denning, Ben E. de Pauw, John E. Edwards Jr., John W. Hiemenz, Carol A. Kauffman, Gabriel Lopez‐Berestein, Pietro Martino, Jack D. Sobel, David A. Stevens, Richard Sylvester, Jan Tollemar, Claudio Viscoli, Maria A. Viviani, Teresa Wu. (2001) Need for Alternative Trial Designs and Evaluation Strategies for Therapeutic Studies of Invasive Mycoses. Clinical Infectious Diseases 33:1, 95-106
    

77. 77

    Blake Cady. (2001) Fundamentals of contemporary surgical oncology: biologic principles and the threshold concept govern treatment and outcomes1 1No competing interests declared.. Journal of the American College of Surgeons 192:6, 777-792
    

78. 78

    David W. Ramey, Martin L. Lee, N.T. Messer. (2001) A review of the Western language equine acupuncture literature. Journal of Equine Veterinary Science 21:2, 56-60
    

79. 79

    J. P. A. Ioannidis, J. Lau. (2001) Evolution of treatment effects over time: Empirical insight from recursive cumulative metaanalyses. Proceedings of the National Academy of Sciences 98:3, 831-836
    

80. 80

    DJ Stein, N Zungu-Dirwayi, GJH van der Linden, S Seedat, Dan Stein. Pharmacotherapy for post traumatic stress disorder (PTSD). In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 2000.
    

81. 81

    Andrew Saxon. (2000) Good “Negative Results”. Clinical Immunology 96:2, 77-78
    

82. 82

    Steven H. Woolf, James N. George. (2000) EVIDENCE-BASED MEDICINE. Hematology/Oncology Clinics of North America 14:4, 761-784
    

83. 83

    Anne Sabers, Lennart Gram. (2000) Newer Anticonvulsants. Drugs 60:1, 23-33
    

84. 84

    James A. Schoenberger. (2000) Evidence-based treatment of hypertension. Current Hypertension Reports 2:4, 398-398
    

85. 85

    GLENN S. GERBER. (2000) SAW PALMETTO FOR THE TREATMENT OF MEN WITH LOWER URINARY TRACT SYMPTOMS. The Journal of Urology 163:5, 1408-1412
    

86. 86

    James L. Sorensen, Amy L. Copeland. (2000) Drug abuse treatment as an HIV prevention strategy: a review. Drug and Alcohol Dependence 59:1, 17-31
    

87. 87

    Laura D Kubzansky, Ichiro Kawachi. (2000) Going to the heart of the matter. Journal of Psychosomatic Research 48:4-5, 323-337
    

88. 88

    Steven M. Keller. (2000) Adjuvant therapy of resected non–small-cell lung cancer. Current Opinion in Oncology 12:2, 149-155
    

89. 89

    David C Bellinger. (2000) Effect modification in epidemiologic studies of low-level neurotoxicant exposures and health outcomes. Neurotoxicology and Teratology 22:1, 133-140
    

90. 90

    Aditya K. Gupta, Neil H. Shear. (2000) A Risk-Benefit Assessment of the Newer Oral Antifungal Agents Used to Treat Onychomycosis. Drug Safety 22:1, 33-52
    

91. 91

    Eleftherios C. Vamvakas. (2000) Evaluation of clinical studies of the efficacy of therapeutic apheresis. Journal of Clinical Apheresis 15:1-2, 6-17
    

92. 92

    Petteri Sjögren, Arne Halling. (2000) Trends in dental and medical research and relevance of randomized controlled trials to common activities in general dentistry. Acta Odontologica Scandinavica 58:6, 260-264
    

93. 93

    Charles W Hogue, Mary L Hyder. (2000) Atrial fibrillation after cardiac operation: risks, mechanisms, and treatment. The Annals of Thoracic Surgery 69:1, 300-306
    

94. 94

    J. H. Kilwein. (1999) Biases in medical literature. Journal of Clinical Pharmacy and Therapeutics 24:6, 393-396
    

95. 95

    Richard J. Beale, David J. Bryg, David J. Bihari. (1999) Immunonutrition in the critically ill: A systematic review of clinical outcome. Critical Care Medicine 27:12, 2799-2805
    

96. 96

    T Dinh-Zarr, C DiGuiseppi, E Heitman, I Roberts. Interventions for preventing injuries in problem drinkers. In: The Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd, 1999.
    

97. 97

    Brenda E. Sirovich, Harold C. Sox. (1999) BREAST CANCER SCREENING. Surgical Clinics of North America 79:5, 961-990
    

98. 98

    (1999) Passive Smoking and Coronary Heart Disease. New England Journal of Medicine 341:9, 697-700
    Free Full Text

99. 99

    Gordon B. Drummond, Christopher A. Ludlam. (1999) IS ALBUMIN HARMFUL?. British Journal of Haematology 106:2, 266-269
    

100. 100

     Suojanen , James N. , . (1999) False False Positive Rates. New England Journal of Medicine 341:2, 131-131
     Free Full Text

101. 101

     Aditya K. Gupta, Neil H. Shear. (1999) The new oral antifungal agents for onychomycosis of the toenails. Journal of the European Academy of Dermatology and Venereology 13:1, 1-13
     

102. 102

     Joyce A. Cramer, Robert Fisher, Elinor Ben-Menachem, Jacqueline French, Richard H. Mattson. (1999) New Antiepileptic Drugs: Comparison of Key Clinical Trials. Epilepsia 40:5, 590-600
     

103. 103

     Michael D. Privitera. (1999) Evidence-based Medicine and Antiepileptic Drugs. Epilepsia 40:s5, s47-s56
     

104. 104

     Harold L. Kennedy. (1999) The importance of randomized clinical trials and evidence-based medicine: A clinician's perspective. Clinical Cardiology 22:1, 6-12
     

105. 105

     Richard Horton. (1999) The information wars. The Lancet 353:9148, 164-165
     

106. 106

     Ralph B. D'Agostino, Michael Weintraub, Heidy K. Russell, Miganush Stepanians, Ralph B. D'Agostino, Louis R. Cantilena, James F. Graumlich, Sam Maldonado, Peter Honig, Charles Anello. (1998) The effectiveness of antihistamines in reducing the severity of runny nose and sneezing: A meta-analysis*. Clinical Pharmacology & Therapeutics 64:6, 579-596
     

107. 107

     Eleftherios C. Vamvakas. (1998) Meta-analysis of randomized controlled trials of the efficacy of white cell reduction in preventing HLA-alloimmunization and refractoriness to random-donor platelet transfusions. Transfusion Medicine Reviews 12:4, 258-270
     

108. 108

     Peter Boyle, Richard Horton, George Radda, David Sharp, Umberto Veronesi, Alexander M Walker. (1998) Open debate on hormone-replacement therapy. The Lancet 352:9131, 836
     

109. 109

     Sven B. Vercauteren, Jean-Louis Bosmans, Monique M. Elseviers, Gert A. Verpooten, Marc E. De Broe. (1998) A meta-analysis and morphological review of cyclosporine-induced nephrotoxicity in auto-immune diseases. Kidney International 54:2, 536-545
     

110. 110

     Lehel Somogyi, Phillip P. Toskes. (1998) Can a Meta-Analysis That Mixes Apples With Oranges Be Used to Demonstrate That Pancreatic Enzymes Do Not Decrease Abdominal Pain in Patients With Chronic Pancreatitis?. The American Journal of Gastroenterology 93:8, 1396-1397
     

111. 111

     STEPHEN B. THACKER, DONNA F. STROUP, HERBERT B. PETERSON. (1998) Meta-Analysis for the Practicing Obstetrician-Gynecologist. Clinical Obstetrics and Gynecology 41:2, 275-281
     

112. 112

     Brendon Smith. (1998) Evidence based medicine and practice guidelines for management of fever without source in young children. Emergency Medicine 10:2, 129-140
     

113. 113

     Karl Goodkin, Daniel J. Feaster, Teri Baldewicz. (1998) Meta-analysis: A time for closer scrutiny in applications to pain research?. Pain Forum 7:2, 100-103
     

114. 114

     N. W. Goodman. (1998) Anaesthesia and evidence-based medicine. Anaesthesia 53:4, 353-368
     

115. 115

     Bertram L. Kasiske. (1998) Meta-analysis as a clinical tool in nephrology. Kidney International 53:4, 819-825
     

116. 116

     Judith M. Stephenson. (1998) Systematic review of hormonal contraception and risk of HIV transmission. AIDS 12:6, 545-553
     

117. 117

     R.J. Koerner. (1998) This letter was shown to Dr Koerner whose reply follows. Journal of Hospital Infection 38:3, 234-235
     

118. 118

     Joyce A. Cramer, Richard H. Mattson, Richard D. Scheyer, Jacqueline French. (1998) REPLY. Epilepsia 39:2, 233-233
     

119. 119

     David A. Clark, Salim Daya. (1998) Is There Hope for IVIG ???. American Journal of Reproductive Immunology 39:2, 65-68
     

120. 120

     (1998) Meta-Analyses and Large Randomized, Controlled Trials. New England Journal of Medicine 338:1, 59-62
     Free Full Text

121. 121

     Michael Borenstein. The Shift from Significance Testing to Effect Size Estimation. In: Comprehensive Clinical Psychology. Elsevier, 1998:313-349.
     

122. 122

     Harvey. (1998) Guidelines, standards and evidence in cervical screening: a personal view. Cytopathology 9:1, 2-5
     

123. 123

     Stephen A. Feig. (1997) Increased benefit from shorter screening mammography intervals for women ages 40-49 years. Cancer 80:11, 2035-2039
     

124. 124

     Jan P Vandenbroucke. (1997) Homoeopathy trials: going nowhere. The Lancet 350:9081, 824
     

Letters