Special Article
Rating the Appropriateness of Coronary Angiography — Do Practicing Physicians Agree with an Expert Panel and with Each Other?
N Engl J Med 1998; 338:1896-1904June 25, 1998
- Abstract
Background
Evaluations of the appropriateness of medical care are important to monitor the quality of care and to contain costs and enhance safety by reducing inappropriate care. Experts' views are usually incorporated into evaluations of appropriateness. However, practicing physicians may not concur with these views, and physicians' clinical backgrounds may influence their beliefs.
Methods
We asked 1058 internists, family practitioners, and cardiologists in California, Florida, New York, Pennsylvania, and Texas to rate the appropriateness of coronary angiography after acute myocardial infarction for 20 common indications. Nine clinical experts also rated these indications using an established consensus method.
Results
For 17 of the 20 indications, median ratings of surveyed physicians and the expert panel agreed within 1 unit on a 9-unit scale. Patients' older age had a negative effect on ratings by the expert panel but not on ratings by surveyed physicians. In multivariable analyses of surveyed physicians, cardiologists rated angiography as significantly more appropriate than did primary care physicians for complicated indications, and for uncomplicated indications cardiologists who performed invasive procedures gave higher appropriateness ratings for angiography than did cardiologists who did not perform such procedures and primary care physicians. For uncomplicated indications, physicians from hospitals providing coronary angioplasty and bypass surgery rated angiography as more appropriate than physicians from other hospitals. Physicians from New York and those employed by health maintenance organizations rated angiography as less appropriate than other physicians.
Conclusions
Surveyed physicians agreed with clinical experts about the appropriateness of coronary angiography after myocardial infarction for most indications, indicating that well-designed expert panels can closely reflect the views of practicing physicians. Variations in beliefs among practicing physicians suggest that evaluations of medical practice should incorporate the views of a range of relevant types of physicians.
Media in This Article
Figure 2
Distributions of Estimated Regression Coefficients Describing the Relation between Physicians' Fields of Practice and Their Ratings of the Appropriateness of Coronary Angiography for Complicated (Panel A) and Uncomplicated (Panel B) Indications.Figure 3
Distributions of Estimated Regression Coefficients Describing the Relation between Physicians' Access to Coronary Procedures and Their Ratings of the Appropriateness of Coronary Angiography for Complicated (Panel A) and Uncomplicated (Panel B) Indications.Article Activity
- Article
Evaluations of the appropriateness of medical care have become more common in recent years. Investigators at RAND have developed a formal method for clinical experts to rate the appropriateness of medical care,1-3 and they have applied this method in numerous studies of cardiac procedures in the United States, Canada, and the United Kingdom.4-7 Major clinical organizations have also convened panels of clinical experts to create guidelines for the use of cardiac procedures after myocardial infarction.8-10
Although substantial effort has been devoted to developing and applying these methods, little attention has been directed to the views of practicing physicians whose care is evaluated by ratings of appropriateness. Clinical experts' ratings may be limited by the quality of scientific data and the challenges of applying these data to clinical practice.11-13 Members of expert panels are often drawn from academic medical centers, so their ratings may not reflect the experiences of physicians in other settings.14 Moreover, understanding differences among practicing physicians in their views about appropriate medical care can help ensure that clinical guidelines and evaluations encompass the views of a range of relevant types of physicians.15
We therefore surveyed internists, family practitioners, and cardiologists in five states to assess their ratings of the appropriateness of coronary angiography after acute myocardial infarction for 20 common clinical indications. These indications were also rated by a panel of clinical experts using an established consensus method. Our study was designed to address two primary questions: Do practicing physicians agree with an expert panel about the appropriateness of coronary angiography? And do practicing physicians vary according to specialty, type of hospital, state, or other characteristics in their beliefs about the appropriateness of coronary angiography?
Methods
Study Population
The study population consisted of cardiologists, internists, and family practitioners in five states: California, Florida, New York, Pennsylvania, and Texas. We focused on these types of physicians because they care for almost all patients with acute myocardial infarction16,17 and they are primarily responsible for recommending coronary angiography. These five states were selected because they have the largest numbers of patients with acute myocardial infarction in the population covered by Medicare, and their rates of coronary angiography after myocardial infarction vary widely.18
Physicians were identified from a list of names, addresses, and self-reported specialties obtained from the American Medical Association's master file (which includes members and nonmembers). We selected a random sample of 4345 physicians stratified according to state and specialty, including 329 cardiologists, 265 internists, and 275 family practitioners from each of the five study states. Physicians were ineligible if they reported at the beginning of the survey that they were in training or that they had not served as attending or consulting physicians for at least one patient hospitalized with myocardial infarction during the prior 12 months.
Ratings of the Expert Panel
During October 1995 we convened a national panel of nine physicians with expertise in the treatment of acute myocardial infarction, including five cardiologists, two cardiac surgeons, one general internist, and one family practitioner. These physicians were nominated by their professional organizations, including the American Academy of Family Physicians, the American Association for Thoracic Surgery, the American College of Cardiology, the American College of Physicians, the American College of Surgeons, the American Heart Association, and the Society of Thoracic Surgeons. Eight of these nine physicians had participated in a prior panel organized by RAND, and we used a similar method to obtain their ratings of the appropriateness of coronary angiography after acute myocardial infarction.3
Panelists were provided with an updated review of the literature that synthesized evidence on the appropriateness of coronary angiography.19 Eight hundred ninety clinical indications were specified for the use of this procedure within 12 weeks after acute myocardial infarction. For the subgroup of 92 indications relating to angiography during the initial hospitalization, key variables included the patient's age (≥75 or <75 years), time since the onset of symptoms (<6, 6 to 12, or >12 hours), eligibility for and receipt of thrombolytic therapy, and presence of key complications, such as persistent or recurrent chest pain, stress-induced ischemia, and pulmonary edema. The panelists initially rated the appropriateness of each clinical indication on a 9-unit numerical scale ranging from 1, extremely inappropriate, to 5, uncertain, and 9, extremely appropriate. Using a modified Delphi method to foster consensus, the group discussed their initial ratings and each panelist then assigned final ratings to the individual indications.
Survey Instrument
From the 92 indications for coronary angiography during the initial hospitalization, we selected 20 indications to be rated by practicing physicians (see the Appendix). Our objective was to include a varied set of indications (differing in patient's age, presence or absence of complications, and expert panelists' ratings) that would apply to a substantial proportion of patients with myocardial infarction. In preliminary analyses of data from the Cooperative Cardiovascular Project of the Health Care Financing Administration at the time we designed the survey, approximately four fifths of Medicare patients with myocardial infarction in the five study states were represented by these 20 indications. The survey also collected professional information about the physicians and asked them to rate the importance of consultations, patient or family requests, guidelines, insurance-plan approvals, and the availability of angiography in their decisions about whether to recommend this procedure for patients who had had acute myocardial infarction during the previous 12 months, using questions adapted from a prior survey.20
Data Collection
During November 1996 the survey was mailed to physicians with a letter of support from the American College of Cardiology (for cardiologists), the American College of Physicians (for internists), or the American Academy of Family Physicians (for family practitioners). Physicians who did not respond to the initial mailing received at least two additional mailings. Eight or more attempts were also made to call nonresponding physicians to request their participation.
Statistical Analysis
Characteristics of survey respondents and the importance they attributed to factors influencing the use of coronary angiography were compared according to specialty with use of the Pearson chi-square test. In preliminary analyses, ratings of appropriateness by internists and family practitioners were similar, but the groups' ratings differed according to the number of patients physicians had treated for myocardial infarction in the prior year (≤12 vs. >12), so we grouped these specialties together and stratified them in subsequent analyses according to this measure of volume. We also divided cardiologists according to whether they had performed coronary angiography during the previous year, because one earlier study had shown that the ratings of appropriateness differed according to this variable,15 and our preliminary analyses suggested that this variable influenced cardiologists' ratings more than the number of patients they treated for myocardial infarction.
Observed median ratings of appropriateness (and interquartile ranges) were calculated for each indication for the expert panel and the surveyed physicians. Using the Pearson correlation coefficient, we correlated the surveyed physicians' average ratings of appropriateness for patients under the age of 75 and 75 or older with their reports of the proportion of their patients with acute myocardial infarction in each of these age groups who had received coronary angiography during the prior year.
To compare more reliable estimates of the appropriateness of coronary angiography as rated by the surveyed physicians and the expert panelists and to determine the precision associated with these estimates, hierarchical regression models21 were fitted separately to the observed ratings from each group. This method accounted for three important sources of variability: the underlying appropriateness of each indication, each physician's propensity to rate angiography as more or less appropriate in general, and measurement error. To understand differences in ratings between the surveyed physicians and the expert panelists, we then incorporated patients' characteristics, including age, time since the onset of symptoms, and the presence or absence of complications, into the hierarchical regression models. Finally, to assess differences in ratings among the surveyed physicians in multivariable analyses, we used hierarchical regression models to compare the distribution of appropriateness ratings according to numerous physician's characteristics, including state, specialty, age, sex, number of patients treated for myocardial infarction in the prior year (for internists and family practitioners), performance of coronary angiography (for cardiologists), availability of coronary procedures at each physician's principal hospital, and employment by a health maintenance organization (HMO).
Hierarchical models were estimated with “Bayesian inference using Gibbs sampling” (BUGS) software, which employs simulation techniques to obtain parameter estimates in complex models.22 Estimates of appropriateness and the effects of the patient and physician characteristics listed above were obtained, and corresponding 95 percent confidence intervals were calculated. We also plotted smoothed density estimates of the empirical distributions for the effects of physician characteristics from the multivariable analysis of survey respondents. Effects were considered significant if the 95 percent confidence intervals for the groups being compared did not overlap.
Results
Response Rates
Completed surveys were received from 1058 physicians, including 602 cardiologists, 239 internists, and 217 family practitioners. An additional 842 physicians were ineligible because they were in training or had not cared for a patient with acute myocardial infarction in the prior year, and 647 physicians were deleted from the sample because they had moved with no forwarding address. Among physicians known to be eligible, 79 refused to participate. No response was received from 1719 physicians, and we estimate on the basis of the physicians who did respond that 57 percent would have been eligible.23 Thus, assuming this estimated eligibility rate among nonrespondents, the overall response rate was 50 percent, with rates of 47 percent for internists, 53 percent for family practitioners, and 49 percent for cardiologists. Response rates by state were 45 percent for California, 46 percent for Florida, 54 percent for New York, 56 percent for Pennsylvania, and 46 percent for Texas.
Surveyed Physicians' Characteristics and Factors Influencing Decisions about Angiography
The characteristics of respondents are presented according to specialty in Table 1Table 1
Characteristics of the Survey Respondents According to Field of Practice.. Among the cardiologists, 62 percent had performed coronary angiography during the previous year. Internists and family practitioners were more likely than cardiologists to cite consultations and hospital guidelines as very or moderately important factors in their recommendations about coronary angiography after myocardial infarction (Table 2Table 2
Factors Affecting Recommendations about Coronary Angiography after Acute Myocardial Infarction.). About half the physicians in each group rated patient or family requests and the availability of angiography at their own hospital as very or moderately important factors in their recommendations. Approximately one fifth of the physicians perceived insurance-plan guidelines or requirements for approval as important factors.Appropriateness Ratings of the Surveyed Physicians and the Expert Panel
When we compared the ratings of the surveyed physicians with those of the expert panel (Table 3Table 3
Ratings of the Appropriateness of Coronary Angiography after Acute Myocardial Infarction by Surveyed Physicians and an Expert Panel.), the median ratings were identical for 6 of the 10 indications in patients under the age of 75 and within 1 unit on the 9-unit scale for the remaining 4 indications (the survey respondents' ratings were higher in 2 cases and the expert panelists' ratings higher in 2). Agreement was less consistent for patients who were 75 or older, with identical median ratings for four indications and higher median ratings by the survey respondents than by the expert panelists for the other six indications. Two of these six indications (indications K and M) differed by 3 units on the 9-unit scale, and one indication (indication T) differed by 2 units. Surveyed physicians' observed ratings of appropriateness were moderately correlated with their self-reported use of coronary angiography after myocardial infarction for patients under the age of 75 (r=0.31, P<0.001) and 75 or older (r=0.37, P<0.001).Hierarchical linear regression models largely confirmed these observed findings (Figure 1AFigure 1
Mean Ratings of the Appropriateness of Coronary Angiography after Acute Myocardial Infarction by an Expert Panel and Surveyed Physicians for 10 Common Clinical Indications in Patients under the Age of 75 (Panel A) and 75 or Older (Panel B). and Figure 1B). Estimates of appropriateness from surveyed physicians were within the 95 percent confidence intervals of estimates from the expert panel for 16 of the 20 indications. Two indications for patients under the age of 75 (indications B and I) were rated slightly less appropriate by the surveyed physicians than by the panel members. Two indications for patients 75 or older (indications M and T) were rated as more appropriate by the surveyed physicians than by the panel members. Patients' older age had a significant negative effect on the ratings of the expert panel (mean difference in ratings between ≥75 and <75 years, –1.4 units; 95 percent confidence interval, –2.1 to –0.6), but not on those of the surveyed physicians (mean difference, –0.4 unit; 95 percent confidence interval, –1.2 to +0.5).Characteristics Associated with the Surveyed Physicians' Appropriateness Ratings
In multivariable analyses, differences in beliefs were evident between cardiologists and primary care physicians and within each of these groups. For patients with cardiac complications (Figure 2A), cardiologists who performed invasive procedures and those who did not gave similar ratings of appropriateness, and ratings by all cardiologists combined were significantly higher than those of primary care physicians with high numbers of patients with myocardial infarctions and those with low numbers (mean differences, 0.6 and 1.0 unit, respectively). For patients with uncomplicated myocardial infarctions (Figure 2B), however, cardiologists who did not perform invasive procedures and primary care physicians reported similar ratings, whereas the ratings of cardiologists who performed invasive procedures were significantly higher (mean differences, 0.5 to 0.6 unit).
Access to coronary procedures at the physicians' hospitals was not associated with their ratings of the appropriateness of angiography for patients with complications (Figure 3A), but for patients with uncomplicated infarctions (Figure 3B), physicians with access to coronary angioplasty and bypass surgery gave significantly higher ratings than physicians from hospitals offering coronary angiography alone or no coronary procedures (mean difference, 0.6 unit). Smaller significant differences (0.3 unit) were noted between ratings by physicians in New York and those by physicians in Florida and Pennsylvania (Figure 4AFigure 4
Distributions of Estimated Regression Coefficients Describing the Relation between Physicians' Ratings of the Appropriateness of Coronary Angiography and Their States (Panel A) and Whether They Were Employed by a Health Maintenance Organization (HMO) (Panel B).) and between physicians employed by HMOs and other physicians (Figure 4B); these patterns were similar for both complicated and uncomplicated indications (data not shown). Physicians' age and sex were not associated with their beliefs about the appropriateness of angiography.Discussion
In this survey of more than 1000 physicians in five states, ratings of the appropriateness of coronary angiography after acute myocardial infarction were very similar to those of a panel of clinical experts. This convergence suggests that evaluations of medical practice based on the judgments of expert panels can closely reflect the beliefs and experiences of practicing physicians for well-studied procedures. However, variations in ratings among practicing physicians suggest that the judgments of physicians participating in panels will be influenced by their professional characteristics, so the composition of panels should be carefully considered when one is interpreting their results.
Agreement between the expert panel and the surveyed physicians was greatest for patients under the age of 75. For patients who were 75 or older, the surveyed physicians rated angiography as more appropriate than did the expert panel, particularly for elderly patients with uncomplicated myocardial infarctions who presented within six hours after the onset of symptoms. Differences of opinion between clinical experts and surveyed physicians highlight the need for further clinical research to assess the risks and benefits of coronary angiography in older patients who may be candidates for angioplasty or thrombolytic therapy.24-27
Beliefs about the appropriateness of coronary angiography differed substantially among primary care physicians, cardiologists who did not perform invasive procedures, and cardiologists who did, as seen in expert panels.15 Other studies have demonstrated that cardiologists' patients are much more likely than the patients of primary care physicians to undergo coronary angiography.16,17,28 Among primary care physicians in our survey, those who treated larger numbers of patients with myocardial infarctions rated angiography as more appropriate for complicated indications than did physicians with fewer such patients, possibly because of their greater experience in referring such patients for invasive coronary procedures. Previous surveys have found that about half of primary care physicians underestimate the benefits of coronary-artery bypass surgery,29 and some primary care physicians may also overestimate the risks of coronary angiography.30 The difference we found in ratings of angiography for uncomplicated indications between cardiologists who performed invasive procedures and those who did not suggests persistent uncertainty or disagreement about the role of coronary angioplasty in patients with acute myocardial infarction.26,27,31 Future studies should explore the consequences of differing beliefs and practices within and between specialties.
Physicians who practiced in hospitals that offered coronary angioplasty and bypass surgery were more likely to view angiography as appropriate for uncomplicated infarctions than physicians in hospitals without these procedures — a finding consistent with the results of other studies in which the availability of coronary procedures strongly predicted their use.18,32-34 Physicians employed by HMOs were somewhat less likely than other physicians to view angiography as appropriate, but differences in practice within HMOs may be more consequential.35 Coronary angiography was also perceived to be somewhat less appropriate by physicians in New York than by physicians in other states, which may contribute to the lower rate of coronary angiography after myocardial infarction in New York.18,36,37 Larger differences in appropriateness ratings have been noted between expert panels from the United States and the United Kingdom.5
Our study has four potential limitations. First, our response rate was similar to that of other published surveys of physicians,38 but the physicians who responded may have been more familiar than those who did not with studies of the appropriate use of coronary procedures, thereby fostering greater agreement with the expert panel. However, median ratings of appropriateness would have had to differ substantially between respondents and nonrespondents to alter the high level of agreement between the surveyed physicians and the expert panel for most indications. Second, although the elements of the clinical indications were determined by expert panelists from clinical experience and an extensive review of the literature,19 practicing physicians may incorporate other clinical factors, such as the anatomical site or size of a myocardial infarction, in decisions about angiography for individual patients. Third, coronary angiography is a relatively well studied procedure for which guidelines have been disseminated by respected professional organizations,39 so the generalizability of our findings to other important procedures needs to be assessed. Fourth, we did not assess physicians' actual use of coronary angiography.40 Nonetheless, physicians' ratings of appropriateness in this survey were correlated with self-reported rates of angiography, and in another recent study physicians' responses to clinical scenarios regarding coronary angiography were significantly associated with actual angiography rates in the areas they served.41
In conclusion, surveyed physicians in five large states agreed closely with an expert panel about the appropriateness of coronary angiography after acute myocardial infarction, particularly for patients under the age of 75. Thus, studies and guidelines that use the judgments of expert panels to evaluate and improve medical care can be highly concordant with the views of practicing physicians. Areas of disagreement between clinical experts and practicing physicians or within either group — over the appropriateness of angiography for older patients or those with uncomplicated infarctions, for example — point to important topics for further clinical research to guide practice. Local and national organizations striving to promote more appropriate medical care should identify and address the systematic ways in which clinical beliefs may differ among practicing physicians.
Supported by a grant (HS07081) from the Agency for Health Care Policy and Research. Dr. Ayanian is a Generalist Physician Faculty Scholar of the Robert Wood Johnson Foundation.
We are indebted to Christine Kreider and the staff of Northeast Research for assistance with data collection; to Christine K. Cassel, M.D., of the American College of Physicians, Richard P. Lewis, M.D., of the American College of Cardiology, and Daniel J. Ostergaard, M.D., of the American Academy of Family Physicians, for providing letters of support to participating physicians; and to Arnold M. Epstein, M.D., and Matthew H. Liang, M.D., M.P.H., for reviewing an earlier draft of the manuscript.
Source Information
From the Department of Medicine, Division of General Medicine and Primary Care, Section on Health Services and Policy Research (J.Z.A.), and the Department of Radiology (B.J.M.), Brigham and Women's Hospital and Harvard Medical School; the Department of Health Care Policy, Harvard Medical School (J.Z.A., M.B.L., S.-L.T.N., E.G., B.J.M.); and the Department of Biostatistics, Harvard School of Public Health (S.-L.T.N.) — all in Boston.
Address reprint requests to Dr. Ayanian at the Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave., Boston, MA 02115.
Appendix
Physicians were asked to rate the appropriateness of coronary angiography according to the following definition and clinical indications on a 9-point scale ranging from extremely inappropriate (1) to uncertain (5) and extremely appropriate (9):
Appropriateness means that the expected health benefits (i.e., increased life expectancy or functional capacity and relief of pain or anxiety) exceed the expected negative consequences (i.e., mortality, morbidity, pain, or anxiety associated with the procedure) by a sufficiently wide margin, exclusive of cost, that angiography is worth doing. Since angiography does not reduce morbidity or mortality, its benefit or harm is generally related to its effect on the subsequent use of medical therapy, percutaneous transluminal coronary angioplasty, or coronary-artery bypass graft surgery. Use your own best clinical judgment for patients presenting to you in each of the following situations during an initial hospitalization for acute myocardial infarction.
In a patient under the age of 75 with acute myocardial infarction, how would you rate the appropriateness of coronary angiography?
A. Within six hours of the onset of symptoms, the patient has not received thrombolytic therapy because of strong contraindications, and the myocardial infarction is uncomplicated.
B. Within six hours of the onset of symptoms, the patient has not received thrombolytic therapy because of strong contraindications and has persistent chest pain.
C. Within six hours of the onset of symptoms, the patient has not received thrombolytic therapy but has no strong contraindications, and the myocardial infarction is uncomplicated.
D. Within six hours of the onset of symptoms, the patient has not received thrombolytic therapy but has no strong contraindications and has persistent chest pain.
E. Within six hours of the onset of symptoms, the patient has received thrombolytic therapy, and the myocardial infarction is uncomplicated.
F. Within six hours of the onset of symptoms, the patient has received thrombolytic therapy and has persistent chest pain.
G. Between 12 hours after the onset of symptoms and discharge, the patient has not received thrombolytic therapy, and the myocardial infarction is uncomplicated.
H. Between 12 hours after the onset of symptoms and discharge, the patient has not received thrombolytic therapy and has persistent chest pain.
I. Between 12 hours after the onset of symptoms and discharge, the patient has not received thrombolytic therapy and has persistent pulmonary edema.
J. Between 12 hours after the onset of symptoms and discharge, the patient has not received thrombolytic therapy and has stress-induced ischemia.
Indications K through T were phrased identically for patients who were 75 or older.
- References
References
1
Park RE ,Fink A ,Brook RH , et al. Physician ratings of appropriate indications for six medical and surgical procedures. Am J Public Health 1986;76:766-772
CrossRef | Web of Science | Medline2
Kosecoff J ,Chassin MR ,Fink AL , et al. Obtaining clinical data on the appropriateness of medical care in community practice. JAMA 1987;258:2538-2542
CrossRef | Web of Science | Medline3
Bernstein SJ, Laouri M, Hilborne LH, et al. Coronary angiography: a literature review and ratings of appropriateness and necessity. Santa Monica, Calif.: RAND, 1992.
4
Chassin MR ,Kosecoff J ,Solomon DH ,Brook RH . How coronary angiography is used: clinical determinants of appropriateness. JAMA 1987;258:2543-2547
CrossRef | Web of Science | Medline5
Brook RH ,Kosecoff JB ,Park RE ,Chassin MR ,Winslow CM ,Hampton JR . Diagnosis and treatment of coronary disease: comparison of doctors' attitudes in the USA and the UK. Lancet 1988;1:750-753
CrossRef | Web of Science | Medline6
Bernstein SJ ,Hilborne LH ,Leape LL , et al. The appropriateness of use of coronary angiography in New York State. JAMA 1993;269:766-769
CrossRef | Web of Science | Medline7
McGlynn EA ,Naylor CD ,Anderson GM , et al. Comparison of theappropriateness of coronary angiography and coronary artery bypass graft surgery between Canada and New York State. JAMA 1994;272:934-940
CrossRef | Web of Science | Medline8
Ryan TJ ,Anderson JL ,Antman EM , et al. ACC/AHA guidelines for the management of patients with acute myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Acute Myocardial Infarction). J Am Coll Cardiol 1996;28:1328-1428
CrossRef | Web of Science | Medline9
American College of Physicians
Web of Science | Medline10
Peterson ED ,Shaw LJ ,Califf RM . Risk stratification after myocardial infarction. Ann Intern Med 1997;126:561-582
Web of Science | Medline11
Phelps CE . The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med 1993;329:1241-1245
Full Text | Web of Science | Medline12
Naylor DC . Grey zones of clinical practice: some limits to evidence-based medicine. Lancet 1995;345:840-842
CrossRef | Web of Science | Medline13
Battista RN ,Hodge MJ ,Vineis P . Medicine, practice and guidelines: the uneasy juncture of science and art. J Clin Epidemiol 1995;48:875-880
CrossRef | Web of Science | Medline14
Layde PM ,Broste SK ,Desbiens N , et al. Generalizability of clinical studies conducted at tertiary care medical centers: a population-based analysis. J Clin Epidemiol 1996;49:835-841
CrossRef | Web of Science | Medline15
Kahan JP ,Park RE ,Leape LL , et al. Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures. Med Care 1996;34:512-523
CrossRef | Web of Science | Medline16
Jollis JG ,DeLong ER ,Peterson ED , et al. Outcome of acute myocardial infarction according to the specialty of the admitting physician. N Engl J Med 1996;335:1880-1887
Full Text | Web of Science | Medline17
Ayanian JZ ,Guadagnoli E ,McNeil BJ ,Cleary PD . Treatment and outcomes of acute myocardial infarction among patients of cardiologists and generalist physicians. Arch Intern Med 1997;157:2570-2576
CrossRef | Web of Science | Medline18
Gatsonis CA ,Epstein AM ,Newhouse JP ,Normand SL ,McNeil BJ . Variations in the utilization of coronary angiography for elderly patients with an acute myocardial infarction: an analysis using hierarchical logistic regression. Med Care 1995;33:625-642
CrossRef | Web of Science | Medline19
Bates DW ,Miller E ,Bernstein SJ ,Hauptman PJ ,Leape LL . Coronary angiography and angioplasty after acute myocardial infarction. Ann Intern Med 1997;126:539-550
Web of Science | Medline20
Pilote L ,Granger C ,Armstrong PW ,Mark DB ,Hlatky MA . Differences in the treatment of myocardial infarction between the United States and Canada: a survey of physicians in the GUSTO trial. Med Care 1995;33:598-610
CrossRef | Web of Science | Medline21
Landrum MB, Normand S-LT. Applying Bayesian ideas to the development of medical guidelines. Stat Med (in press).
22
Gilks WR ,Thomas A ,Spiegelhalter DJ . A language and program for complex Bayesian modelling. Statistician 1994;43:169-178
CrossRef23
Aday LA. Designing and conducting health surveys: a comprehensive guide. San Francisco: Jossey-Bass, 1989.
24
Gurwitz JH ,Col NF ,Avorn J . The exclusion of the elderly and women from clinical trials in acute myocardial infarction. JAMA 1992;268:1417-1422
CrossRef | Web of Science | Medline25
Stone GW ,Grines CL ,Browne KF , et al. Predictors of in-hospital and 6-month outcome after acute myocardial infarction in the reperfusion era: the Primary Angioplasty in Myocardial Infarction (PAMI) trial. J Am Coll Cardiol 1995;25:370-377
CrossRef | Web of Science | Medline26
Lange RA ,Hillis LD . Should thrombolysis or primary angioplasty be the treatment of choice for acute myocardial infarction? Thrombolysis -- the preferred treatment. N Engl J Med 1996;335:1311-1312
Full Text | Web of Science | Medline27
Grines CL . Should thrombolysis or primary angioplasty be the treatment of choice for acute myocardial infarction? Primary angioplasty -- the strategy of choice. N Engl J Med 1996;335:1313-1316
Web of Science | Medline28
Borowsky SJ ,Kravitz RL ,Laouri M , et al. Effect of physician specialty on use of necessary coronary angiography. J Am Coll Cardiol 1995;26:1484-1491
CrossRef | Web of Science | Medline29
Friedmann PD ,Brett AS ,Mayo-Smith MF . Differences in generalists' and cardiologists' perception of cardiovascular risk and the outcomes of preventive therapy in cardiovascular disease. Ann Intern Med 1996;124:414-421
Web of Science | Medline30
Poses RM ,McClish DK ,Smith WR , et al. Physicians' judgments of the risks of cardiac procedures: differences between cardiologists and other internists. Med Care 1997;35:603-617
CrossRef | Web of Science | Medline31
Di Salvo TT ,Paul SD ,Lloyd-Jones D , et al. Care of acute myocardial infarction by noninvasive and invasive cardiologists: procedure use, cost and outcome. J Am Coll Cardiol 1996;27:262-269
CrossRef | Web of Science | Medline32
Blustein J . High-technology cardiac procedures: the impact of service availability on service use in New York State. JAMA 1993;270:344-349
CrossRef | Web of Science | Medline33
Every NR ,Larson EB ,Litwin PE , et al. The association between on-site cardiac catheterization facilities and the use of coronary angiography after acute myocardial infarction. N Engl J Med 1993;329:546-551
Full Text | Web of Science | Medline34
Pilote L ,Miller DP ,Califf RM ,Rao JS ,Weaver WD ,Topol EJ . Determinants of the use of coronary angiography and revascularization after thrombolysis for acute myocardial infarction. N Engl J Med 1996;335:1198-1205
Full Text | Web of Science | Medline35
Selby JV ,Fireman BH ,Lundstrom RJ , et al. Variation among hospitals in coronary-angiography practices and outcomes after myocardial infarction in a large health maintenance organization. N Engl J Med 1996;335:1888-1896
Full Text | Web of Science | Medline36
Guadagnoli E ,Hauptman PJ ,Ayanian JZ ,Pashos CL ,McNeil BJ ,Cleary PD . Variation in the use of cardiac procedures after acute myocardial infarction. N Engl J Med 1995;333:573-578
Full Text | Web of Science | Medline37
Pilote L ,Califf RM ,Sapp S , et al. Regional variation across the United States in the management of acute myocardial infarction. N Engl J Med 1995;333:565-572
Full Text | Web of Science | Medline38
Asch DA ,Jedrziewski MK ,Christakis NA . Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129-1136
CrossRef | Web of Science | Medline39
Tunis SR ,Hayward RS ,Wilson MC , et al. Internists' attitudes about clinical practice guidelines. Ann Intern Med 1994;120:956-963
Web of Science | Medline40
Grilli R ,Lomas J . Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care 1994;32:202-213
CrossRef | Web of Science | Medline41
Wennberg DE ,Dickens JD Jr ,Biener L ,Fowler JF Jr ,Soule DN ,Keller RB . Do physicians do what they say? The inclination to test and its association with coronary angiography rates. J Gen Intern Med 1997;12:172-176
CrossRef | Web of Science | Medline
- Citing Articles (57)
Citing Articles
1
Martha S. Linet, Thomas L. Slovis, Donald L. Miller, Ruth Kleinerman, Choonsik Lee, Preetha Rajaraman, Amy Berrington de Gonzalez. (2012) Cancer risks associated with external radiation from diagnostic imaging procedures. CA: A Cancer Journal for Cliniciansn/a-n/a
CrossRef2
G. Haller, D. S. Courvoisier, H. Anderson, P. S. Myles. (2011) Clinical factors associated with the non-utilization of an anaesthesia incident reporting system. British Journal of Anaesthesia 107:2, 171-179
CrossRef3
Thomas C. Ricketts. (2011) The Health Care Workforce: Will It Be Ready as the Boomers Age? A Review of How We Can Know (or Not Know) the Answer. Annual Review of Public Health 32:1, 417-30
CrossRef4
Robert A. Warriner, Marissa J. Carter. (2011) The Current State of Evidence-Based Protocols in Wound Care. Plastic and Reconstructive Surgery 127, 144S-153S
CrossRef5
T. Higashi, R. Machii, A. Aoki, C. Hamashima, H. Saito. (2010) Evaluation and Revision of Checklists for Screening Facilities and Municipal Governmental Programs for Gastric Cancer and Colorectal Cancer Screening in Japan. Japanese Journal of Clinical Oncology 40:11, 1021-1030
CrossRef6
Cynthia LeRouge, Bengisu Tulu, Pamela Forducey. (2010) The Business of Telemedicine: Strategy Primer. Telemedicine and e-Health 16:8, 898-909
CrossRef7
Philipp Wagdi, Hatem Alkadhi. (2010) The impact of cardiac CT on the appropriate utilization of catheter coronary angiography. The International Journal of Cardiovascular Imaging 26:3, 333-344
CrossRef8
Bruce I. Reiner, Eliot L. Siegel. (2009) Decommoditizing Radiology. Journal of the American College of Radiology 6:3, 167-170
CrossRef9
Katsunori Shimada, Hiroshi Kasanuki, Nobuhisa Hagiwara, Hiroshi Ogawa, Naohito Yamaguchi. (2008) Routine coronary angiographic follow-up and subsequent revascularization in patients with acute myocardial infarction. Heart and Vessels 23:6, 383-389
CrossRef10
Claudia Sanmartin, Kellie Murphy, Nicole Choptain, Barbara Conner-Spady, Lindsay McLaren, Eric Bohm, Michael J. Dunbar, Suren Sanmugasunderam, Carolyn De Coster, John McGurran, Diane L. Lorenzetti, Tom Noseworthy. (2008) Appropriateness of healthcare interventions: Concepts and scoping of the published literature. International Journal of Technology Assessment in Health Care 24:03,
CrossRef11
Christopher Lee Sistrom. (2008) In Support of the ACR Appropriateness Criteria®. Journal of the American College of Radiology 5:5, 630-635
CrossRef12
Anna R. Gagliardi, Novlette Fraser, Frances C. Wright, Louise Lemieux-Charles, Dave Davis. (2008) Fostering knowledge exchange between researchers and decision-makers: Exploring the effectiveness of a mixed-methods approach. Health Policy 86:1, 53-63
CrossRef13
ROD STABLES. 2008. Patient selection for percutaneous coronary intervention. , 19-27.
CrossRef14
Anna Gagliardi, Louise Lemieux-Charles, Adalsteinn Brown, Terry Sullivan, Vivek Goel. (2008) Stakeholder preferences for cancer care performance indicators. International Journal of Health Care Quality Assurance 21:2, 175-189
CrossRef15
Cynthia LeRouge, Alan R. Hevner, Rosann Webb Collins. (2007) It's more than just use: An exploration of telemedicine use quality. Decision Support Systems 43:4, 1287-1304
CrossRef16
Andrew D Oxman, John N Lavis, Atle Fretheim. (2007) Use of evidence in WHO recommendations. The Lancet 369:9576, 1883-1889
CrossRef17
D M Charytan, S Setoguchi, D H Solomon, J Avorn, W C Winkelmayer. (2007) Clinical presentation of myocardial infarction contributes to lower use of coronary angiography in patients with chronic kidney disease. Kidney International 71:9, 938-945
CrossRef18
Umesh T. Kadam, Kelvin Jordan, Peter R. Croft. (2006) A comparison of two consensus methods for classifying morbidities in a single professional group showed the same outcomes. Journal of Clinical Epidemiology 59:11, 1169-1173
CrossRef19
Anna R. Gagliardi, Michael Fung Kee Fung, Bernard Langer, Hartley Stern, Adalsteinn D. Brown. (2005) Development of ovarian cancer surgery quality indicators using a modified Delphi approach. Gynecologic Oncology 97:2, 446-456
CrossRef20
Olivier Terraz, Vincent Wietlisbach, Jean-Gabriel Jeannot, Bernard Burnand, Florian Froehlich, Jean-Jacques Gonvers, Jennifer K. Harris, John-Paul Vader. (2005) The EPAGE Internet Guideline as a Decision Support Tool for Determining the Appropriateness of Colonoscopy. Digestion 71:2, 72-77
CrossRef21
Rosalind Raine, Colin Sanderson, Andrew Hutchings, Simon Carter, Kirsten Larkin, Nick Black. (2004) An experimental study of determinants of group judgments in clinical guideline development. The Lancet 364:9432, 429-437
CrossRef22
Donna L. Washington, Steven J. Bernstein, James P. Kahan, Lucian L. Leape, Caren J. Kamberg, Paul G. Shekelle. (2003) Reliability of Clinical Guideline Development Using Mail-Only versus In-Person Expert Panels. Medical Care 41:12, 1374-1381
CrossRef23
Arnold M. Epstein, Joel S. Weissman, Eric C. Schneider, Constantine Gatsonis, Lucian L. Leape, Robert N. Piana. (2003) Race and Gender Disparities in Rates of Cardiac Revascularization. Medical Care 41:11, 1240-1255
CrossRef24
Petersen, Laura A., Normand, Sharon-Lise T., Leape, Lucian L., McNeil, Barbara J., . (2003) Regionalization and the Underuse of Angiography in the Veterans Affairs Health Care System as Compared with a Fee-for-Service System. New England Journal of Medicine 348:22, 2209-2217
Full Text25
Nandini Dendukuri, Sharon-Lise T. Normand, Barbara J. McNeil. (2003) Impact of Cardiac Service Availability on Case-Selection for Angiography and Survival Associated with Angiography. Health Services Research 38:1p1, 21-40
CrossRef26
S. Joffe. (2002) Views of American Oncologists About the Purposes of Clinical Trials. CancerSpectrum Knowledge Environment 94:24, 1847-1853
CrossRef27
A.W. Bradbury, J. Bell, A.J. Lee, R.J. Prescott, I. Gillespie, G. Stansby, F.G.R. Fowkes. (2002) Bypass or Angioplasty for Severe Limb Ischaemia? A Delphi Consensus Study. European Journal of Vascular and Endovascular Surgery 24:5, 411-416
CrossRef28
Shemy Carasso, Walter Markiewicz. (2002) Medical treatment of patients with stable angina pectoris referred for coronary angiography: Failure of treatment or failure to treat. Clinical Cardiology 25:9, 436-441
CrossRef29
Nancy Wolff, Mark Schlesinger. (2002) Clinicians as Advocates. The Journal of Behavioral Health Services & Research 29:3, 274???287
CrossRef30
Nancy Wolff, Mark Schlesinger. (2002) Clinicians as advocates: An exploratory study of responses to managed care by mental health professionals. The Journal of Behavioral Health Services & Research 29:3, 274-287
CrossRef31
Debra Saliba, John F. Schnelle. (2002) Indicators of the Quality of Nursing Home Residential Care. Journal of the American Geriatrics Society 50:8, 1421-1430
CrossRef32
MB Nienhuis, M Berge, NJGM Veeger, JW Viersma, JF May. (2002) Afwachten bij atypische angina pectoris en een negatief inspanningselektrocardiogram gerechtvaardigdangina pectoris elektrocardiografie hart- en vaatziekten. Huisarts en Wetenschap 45:7, 408-411
CrossRef33
Pushkal P. Garg, Mary Beth Landrum, Sharon-Lise T. Normand, John Z. Ayanian, Paul J. Hauptman, Thomas J. Ryan, Barbara J. McNeil, Edward Guadagnoli. (2002) Understanding Individual and Small Area Variation in the Underuse of Coronary Angiography Following Acute Myocardial Infarction. Medical Care 40:7, 614-626
CrossRef34
V. L. Roger, S. J. Jacobsen, S. A. Weston, P. A. Pellikka, T. D. Miller, K. R. Bailey, B. J. Gersh. (2002) Sex Differences in Evaluation and Outcome After Stress Testing. Mayo Clinic Proceedings 77:7, 638-645
CrossRef35
D.A Alter, C.D Naylor, P.C Austin, J.V Tu. (2002) Biology or bias: practice patterns and long-term outcomes for men and women with acute myocardial infarction. Journal of the American College of Cardiology 39:12, 1909-1916
CrossRef36
William G. Carnett. (2002) Clinical Practice Guidelines: A Tool To Improve Care. Journal of Nursing Care Quality 16:3, 60-70
CrossRef37
Curtis E Margo. (2002) Peer and expert opinion and the reliability of implicit case review. Ophthalmology 109:3, 614-618
CrossRef38
Michael Hermann, Gunter Alk, Rudolf Roka, Karl Glaser, Michael Freissmuth. (2002) Laryngeal Recurrent Nerve Injury in Surgery for Benign Thyroid Diseases. Annals of Surgery 235:2, 261-268
CrossRef39
Alison Murray, Jana E. Montgomery, Hong Chang, William H. Rogers, Thomas Inui, Dana Gelb Safran. (2001) Doctor discontent. Journal of General Internal Medicine 16:7, 451-459
CrossRef40
Alison Murray, Jana E. Montgomery, Hong Chang, William H. Rogers, Thomas Inui, Dana Gelb Safran. (2001) Doctor Discontent. A Comparison of Physician Satisfaction in Different Delivery System Settings, 1986 and 1997. Journal of General Internal Medicine 16:7, 452-459
CrossRef41
E. F. Philbin, P. A. McCullough, T. G. DiSalvo, G. W. Dec, P. L. Jenkins, W. D. Weaver. (2001) Underuse of invasive procedures among Medicaid patients with acute myocardial infarction. American Journal of Public Health 91:7, 1082-1088
CrossRef42
Batami Sadan. (2001) Patient data confidentiality and patient rights. International Journal of Medical Informatics 62:1, 41-49
CrossRef43
Edward Guadagnoli, Mary Beth Landrum, Sharon-Lise T. Normand, John Z. Ayanian, Pushkal Garg, Paul J. Hauptman, Thomas J. Ryan, Barbara J. McNeil. (2001) Impact of Underuse, Overuse, and Discretionary Use on Geographic Variation in the Use of Coronary Angiography After Acute Myocardial Infarction. Medical Care 39:5, 446-458
CrossRef44
Steven J. Bernstein, Pablo Lázaro, Kathryn Fitch, María Dolores Aguilar, James P. Kahan. (2001) Effect of Specialty and Nationality on Panel Judgments of the Appropriateness of Coronary Revascularization. Medical Care 39:5, 513-520
CrossRef45
Hemingway, Harry, Crook, Angela M., Feder, Gene, Banerjee, Shrilla, Dawson, J. Rex, Magee, Patrick, Philpott, Sue, Sanders, Julie, Wood, Alan, Timmis, Adam D., . (2001) Underuse of Coronary Revascularization Procedures in Patients Considered Appropriate Candidates for Revascularization. New England Journal of Medicine 344:9, 645-654
Full Text46
Brenda R. Hemmelgarn, William A. Ghali, Hude Quan, Rollin Brant, Colleen M. Norris, Kenneth J. Taub, Merril L. Knudtson, Approach Investigators. (2001) Poor Long-Term Survival After Coronary Angiography in Patients With Renal Insufficiency. American Journal of Kidney Diseases 37:1, 64-72
CrossRef47
Petersen, Laura A., Normand, Sharon-Lise T., Daley, Jennifer, McNeil, Barbara J., . (2000) Outcome of Myocardial Infarction in Veterans Health Administration Patients as Compared with Medicare Patients. New England Journal of Medicine 343:26, 1934-1941
Full Text48
Epstein, Arnold M., Ayanian, John Z., Keogh, Joseph H., Noonan, Susan J., Armistead, Nancy, Cleary, Paul D., Weissman, Joel S., David-Kasdan, Jo Ann, Carlson, DianeFuller, Jerry, Marsh, DouglasConti, Rena M.. (2000) Racial Disparities in Access to Renal Transplantation — Clinically Appropriate or Due to Underuse or Overuse?. New England Journal of Medicine 343:21, 1537-1544
Full Text49
Steven H. Woolf, James N. George. (2000) EVIDENCE-BASED MEDICINE. Hematology/Oncology Clinics of North America 14:4, 761-784
CrossRef50
Susannah C Daly, Veronique L Roger, Cynthia Leibson, Todd D Miller, Patricia A Pellikka, Kent Bailey, Steven J Jacobsen. (2000) Cardiology services after stress testing. Journal of Clinical Epidemiology 53:7, 661-668
CrossRef51
Timothy M. Powell, Jeffrey P. Thompsen, Katherine S. Virgo, Eric T. Johnson, Danny Chan, John W. Colberg, David K. Ornstein, Frank E. Johnson. (2000) Geographic Variation in Patient Surveillance After Radical Prostatectomy. Annals of Surgical Oncology 7:5, 339-346
CrossRef52
Joanne K. Tobacman, Ingrid U. Scott, Stacey Cyphert, Bridget Zimmerman. (1999) Reproducibility of Measures of Overuse of Cataract Surgery by Three Physician Panels. Medical Care 37:9, 937-945
CrossRef53
Vincent Wietlisbach, John-Paul Vader, François Porchet, Michael C. Costanza, Bernard Burnand. (1999) Statistical Approaches in the Development of Clinical Practice Guidelines From Expert Panels. Medical Care 37:8, 785-797
CrossRef54
Patrick J Scanlon, David P Faxon, Anne-Marie Audet, Blase Carabello, Gregory J Dehmer, Kim A Eagle, Ronald D Legako, Donald F Leon, John A Murray, Steven E Nissen, Carl J Pepine, Rita M Watson, James L Ritchie, Raymond J Gibbons, Melvin D Cheitlin, Kim A Eagle, Timothy J Gardner, Arthur Garson, Richard O Russell, Thomas J Ryan, Sidney C Smith. (1999) ACC/AHA guidelines for coronary angiography. Journal of the American College of Cardiology 33:6, 1756-1824
CrossRef55
Joel S. Weissman, John Z. Ayanian, Scott Chasan-Taber, Marjorie J. Sherwood, Carol Roth, Arnold M. Epstein. (1999) Hospital Readmissions and Quality of Care. Medical Care 37:5, 490-501
CrossRef56
(1998) Assessing the Appropriateness of Medical Care. New England Journal of Medicine 339:20, 1478-1481
Full Text57
Naylor, C. David, . (1998) What is Appropriate Care?. New England Journal of Medicine 338:26, 1918-1920
Full Text
- Letters
