Special Article
The Reproducibility of a Method to Identify the Overuse and Underuse of Medical Procedures
N Engl J Med 1998; 338:1888-1895June 25, 1998
- Abstract
Background
To assess the overuse and underuse of medical procedures, various methods have been developed, but their reproducibility has not been evaluated. This study estimates the reproducibility of one commonly used method.
Methods
We performed a parallel, three-way replication of the RAND–University of California at Los Angeles appropriateness method as applied to two medical procedures, coronary revascularization and hysterectomy. Three nine-member multidisciplinary panels of experts were composed for each procedure by stratified random sampling from a list of experts nominated by the relevant specialty societies. Each panel independently rated the same set of clinical scenarios in terms of the appropriateness of the relevant procedure on a risk–benefit scale ranging from 1 to 9. Final ratings were used to classify the procedure in each scenario as necessary or not necessary (to evaluate underuse) and inappropriate or not inappropriate (to evaluate overuse). Reproducibility was measured by overall agreement and by the kappa statistic. The criteria for underuse and overuse derived from these ratings were then applied to real populations of patients who had undergone coronary revascularization or hysterectomy.
Results
The rates of agreement among the three coronary-revascularization panels were 95, 94, and 96 percent for inappropriate-use scenarios and 93, 92, and 92 percent for necessary-use scenarios. Agreement among the three hysterectomy panels was 88, 70, and 74 percent for inappropriate-use scenarios. Scenarios involving necessary use of hysterectomy were not assessed. The three-way kappa statistic to detect overuse was 0.52 for coronary revascularization and 0.51 for hysterectomy. The three-way kappa statistic to detect underuse of coronary revascularization was 0.83. Application of individual panels' criteria to real populations of patients resulted in a 100 percent variation in the proportion of cases classified as inappropriate and a 20 percent variation in the proportion of cases classified as necessary.
Conclusions
The appropriateness method is far from perfect. Appropriateness criteria may be useful in comparing levels of appropriate procedures among populations but should not by themselves be used to direct care for individual patients.
Media in This Article
Figure 3
Effect of the Three Panels' Appropriateness Ratings on the Determination of Overuse of Hysterectomy.Figure 2
Effect of the Three Panels' Appropriateness Ratings on the Determination of Underuse of Coronary Revascularization.Article Activity
- Article
The appropriateness of health procedures has commanded considerable attention recently.1-3 Escalating health care costs and identification of inappropriate care have led to the critical examination of possible overuse and underuse of many medical and surgical procedures and questions as to when or whether they are needed. Central to this examination is the determination of what constitutes appropriate indications for any given procedure. Ideally, this determination would be derived solely from rigorously conducted research that established conclusively the clinical circumstances under which patients benefit from the procedure. Unfortunately, satisfactory data on efficacy and effectiveness are unusual.4 In fact, several studies estimate that only 15 to 20 percent of medical practices can be justified on the basis of rigorous scientific data establishing their effectiveness.5,6 For most conditions, something other than rigorous data on efficacy or effectiveness must be used to determine criteria of appropriateness.
One frequently used method that combines expert opinion, the type of information most commonly employed, with available scientific evidence is the RAND–University of California at Los Angeles appropriateness method, which was developed in 1984 by the Health Services Utilization Study.7 This method has been used to evaluate the appropriateness of a variety of medical and surgical interventions.1-3,8,9 It combines a systematic review of the scientific literature with expert opinion and yields specific criteria of appropriateness that can be used as the basis for review criteria, practice guidelines, or both. In general, it quantitatively assesses the expert judgment of a multidisciplinary group of clinicians concerning a comprehensive series of clinical indications on a risk–benefit scale ranging from 1 to 9. It is iterative, with two rounds of anonymous ratings and a face-to-face group discussion between rounds. Each panelist has equal weight in determining the final result: an explicit appropriateness rating for clinically detailed patient scenarios.
A central criticism of the appropriateness method is the potential sensitivity of the results to the selection of particular experts, leading to concern about the results' validity.10,11 To address this concern, we conducted a rigorous test of the reproducibility of the appropriateness method as used to identify the overuse and underuse of medical procedures.
Methods
We performed a parallel, three-way replication of the appropriateness panel process for two medical procedures, coronary revascularization and hysterectomy. We chose these procedures because they are commonly performed and they differ in the amount of available scientific evidence concerning efficacy. We examined all indications for coronary revascularization (948 clinical scenarios) and nonemergency, nononcologic indications for hysterectomy (1718 clinical scenarios). Table 1Table 1
Examples of the Indications for Coronary Revascularization and Hysterectomy Rated by Expert Panels. presents examples of indications that were rated.Selection of Panelists
We solicited nominations for the coronary-revascularization and hysterectomy panels from a variety of relevant, respected medical and surgical societies and organizations. From all sources, 69 cardiologists, 30 primary care physicians, and 81 cardiovascular surgeons were nominated for the coronary-revascularization panel, and 57 obstetrician–gynecologists and 30 primary care physicians were nominated for the hysterectomy panel.
We requested a current curriculum vitae from each nominee. Physicians who had previously served as expert panelists for assessments of the appropriateness of coronary revascularization or hysterectomy were excluded. Each panelist was classified according to specialty, location of practice, type of practice (academic or private), and sex. Drawing from the pool of qualified nominees by stratified random sampling, we made assignments to four panels for each procedure. We sent the panelists who were selected a letter inviting them to participate. Those who declined were replaced with new physicians from the appropriate strata until four panels for each procedure had been composed. Our interaction with one of these panels was only by mail. We report here the results from the three panels that followed the conventional appropriateness method, which includes a face-to-face panel discussion.
Synthesis of the Literature and Selection of Moderators
For each procedure, a synthesis of the scientific literature was prepared and peer-reviewed by external experts for completeness and accuracy. Three experienced moderators were selected, one for each panel. Moderators were aware only of the names of their own panelists and their own results; they were unaware of the names of other panelists and of the actions and results of the other panels.
Operation of the Panels
Each panel was conducted in identical fashion, with panelists receiving the same literature synthesis, set of clinical scenarios, and instructions. The panelists first independently rated the appropriateness of using the relevant procedure in each scenario and returned their rating forms by mail. The ratings were then tabulated before the face-to-face panel meeting. Each coronary-revascularization panel had a 2-day face-to-face meeting (all three of which took place over a 10-day span in October 1994). Likewise, the three hysterectomy panels met independently for two days each in November 1994. All panel meetings occurred in the same room at the RAND office in Washington, D.C. In the only departure from usual practice, we did not allow panelists to alter clinical scenarios, because we wanted an identical set of scenarios in order to compare results among panels. To minimize the potential effect of this change, we extensively tested our scenarios with nonpanelists for clinical sensibility before we used them.
After obtaining the final-round appropriateness ratings, we had the coronary-revascularization panelists rate again each scenario that they had judged appropriate for use of the relevant procedure, this time according to necessity criteria. The concept of necessity goes beyond that of appropriateness, in that withholding a procedure that was deemed necessary for a person's clinical situation would constitute wrongful underuse of the procedure.12 Because our study was restricted to the use of hysterectomy for nonemergency, nononcologic indications, we did not ask the hysterectomy panel for necessity ratings.
Statistical Analysis
With final ratings from each panel, we assigned an appropriateness category to each clinical indication. Disagreement was considered to have occurred when at least three panelists rated an indication in the top third of the risk–benefit scale (7, 8, or 9) and at least three panelists rated the same indication in the bottom third (1, 2, or 3). A median panel rating of 7, 8, or 9 without disagreement defined an indication as appropriate. A median panel rating of 1, 2, or 3 without disagreement defined an indication as inappropriate. Indications with a median rating of 4, 5, or 6, and all indications with disagreement, were classified as uncertain. Indications judged appropriate with a median panel rating of 7, 8, or 9 on the necessity scale without disagreement were considered evidence of a procedure's necessity.
We analyzed the final-round appropriateness ratings using the pairwise percentage of agreement between panels, the kappa statistic (a measure of agreement that takes into account the agreement due to chance), and the three-way kappa statistic among panels. We used terminology suggested by Landis and Koch13 to assign descriptive terms to numerical values of kappa. To identify overuse, we used the ratings to classify each procedure as “inappropriate” or “not inappropriate.” To identify underuse of coronary revascularization, we used the classification of “necessary” or “not necessary.” These classifications are the same as those used in previous studies of overuse and underuse. For each calculation, the indication was weighted by the frequency with which it occurs in practice. For the weights for overuse of coronary revascularization, we used data from 2532 persons (randomly selected from 15 hospitals in New York State) who had undergone coronary revascularization. For the weights for underuse of coronary revascularization, we used data from 1294 persons (randomly selected from 15 New York hospitals) who had undergone coronary angiography. For hysterectomy, we used data from 636 women (randomly selected from seven managed-care organizations) who had undergone hysterectomy for nonemergency, nononcologic indications. The methods used for collecting data and assigning appropriateness criteria based on medical records have been previously reported.1-3 In brief, clinical data were collected from the medical records in sufficient detail to allow each case to be matched with one of the clinical scenarios rated by the panels for appropriateness.
Stata software (version 5.0, Stata, College Station, Tex.) was used for calculations. Confidence intervals were calculated by the bias-corrected bootstrap method.
Results
Participation rates were extremely high among those invited to serve as panel members. Of the cardiovascular panelists invited, 98 percent agreed to participate, and of the hysterectomy panelists, 91 percent agreed to participate. The three panels for each procedure were well matched with regard to all measured characteristics (Table 2Table 2
Composition of the Expert Panels.).The respective final-round ratings of Panels A, B, and C showed disagreement on 1, 4, and 4 percent of the coronary-revascularization scenarios and 9, 6, and 2 percent of the hysterectomy scenarios.
The degree of agreement on appropriateness among the panels was mixed. Table 3Table 3
Comparisons of Panel Ratings of Overuse and Underuse. shows the pairwise agreement, pairwise kappa statistic, and three-way kappa statistic for overuse and underuse. For coronary revascularization, there were high levels of agreement among panels, with moderate agreement beyond chance with regard to overuse and almost perfect agreement beyond chance with regard to underuse. For hysterectomy, Panels A and B had a very high level of agreement, and substantial agreement beyond chance, with regard to overuse. Panel C had a lower level of overall agreement with the other two panels. For both procedures, the three-way agreement beyond chance with regard to overuse was moderate, and for coronary revascularization, the three-way agreement beyond chance with regard to underuse was almost perfect.Figure 1Figure 1
Effect of the Three Panels' Appropriateness Ratings on the Determination of Overuse of Coronary Revascularization. shows the effect of using the appropriateness ratings of the three coronary-revascularization panels to classify the 2532 cases of coronary revascularization in New York. Had Panel A's ratings alone been used to classify care, 160 procedures would have been labeled as inappropriate. Of these, none would have been rated as necessary by either of the other two panels, and 18 would have been rated as appropriate by one of the other panels. Similarly, if Panel B's ratings alone had been used to classify care, 186 procedures would have been labeled as inappropriate, and none of these would have been rated as necessary or appropriate by either of the other two panels. Finally, if Panel C's ratings alone had been used to classify care, 97 procedures would have been labeled as inappropriate; none of these would have been rated as necessary, but 2 would have been rated as appropriate by one of the other panels. In no instance was a case rated as necessary by one panel and inappropriate by another.Figure 2Figure 2
Effect of the Three Panels' Appropriateness Ratings on the Determination of Underuse of Coronary Revascularization. provides similar data about the underuse of coronary revascularization. Of 1294 uses of angiography, 498, 464, and 402 would have been rated as necessary by Panels A, B, and C, respectively. No use of angiography judged necessary by one panel was rated as inappropriate by either of the other two panels; some were rated as uncertain by at least one other panel (24, 31, and 4 by Panels A, B, and C, respectively).Finally, Figure 3Figure 3
Effect of the Three Panels' Appropriateness Ratings on the Determination of Overuse of Hysterectomy. shows the effect of using the appropriateness ratings of the three hysterectomy panels to classify 636 cases of hysterectomy. Using Panel A's ratings or Panel B's ratings alone would have labeled 200 or 153 hysterectomies, respectively, as inappropriate, with 7 of them for each panel rated as appropriate by one of the other two panels. Using Panel C's ratings alone would have labeled 331 hysterectomies as inappropriate, with 92 of them rated as appropriate by one of the other two panels.We examined the indications for which results were discordant among panels and found none in which conclusive evidence from randomized, clinical trials supported a given action. For overuse of revascularization, three indications involved discordant ratings (in a total of 20 cases). Sixteen cases were accounted for by one indication (patients with chronic stable angina, mild or moderate angina, and single-vessel disease who had less than strongly positive results on an exercise stress test or in whom the stress test was not done). For underuse of revascularization, 13 indications involved discordant ratings (in a total of 46 cases). Four indications accounted for 35 cases, including three that involved patients presenting within 21 days after an acute myocardial infarction and one that involved asymptomatic patients with three-vessel disease. The 92 cases of hysterectomy with discordant results were spread over 28 indications, of which 26 (93 percent, involving 90 [98 percent] of the cases) involved uterine bleeding (or pelvic discomfort) with “major impairment” of the patient, which was defined as follows: “during the last 3 months the patient had had a significant worsening in level of activity (e.g., 2 or more days per month) due to her bleeding or pain, or the bleeding or pain is continuing to have a significant negative effect on her functional ability.”
Discussion
Our results show that the appropriateness method of identifying overuse is far from perfect. The degree of agreement among panels about care identified as inappropriate was only moderate. Furthermore, the number of cases categorized as inappropriate varied by a factor of about two for both procedures. However, our results for identifying underuse are more reassuring. Agreement among panels was nearly perfect, and the number of cases classified as necessary varied by only 20 percent among panels.
The literature is sparse on studies evaluating the reproducibility and reliability of alternative methods for determining appropriateness. We do know, however, that alternative methods are certain to be less than perfect. The reliability of individual surgeons' decisions to recommend hysterectomy has been estimated to have a kappa of 0.23.14 Although imperfect, the reproducibility of the appropriateness method is markedly better. Three-to-fivefold variations in the rate of use of hysterectomy have been reported15-17 and have been attributed to variability among physicians.18 Although imperfect, the appropriateness method is less variable. A recent report on coronary angiography after myocardial infarction reported a 2.5-fold variation in the rate of use among 16 Kaiser Permanente hospitals.19 For cases in which coronary angiography was judged necessary (by a process identical to that described here), there was a 1.6-fold variation. Again, although imperfect, the results of the appropriateness method for coronary revascularization are less variable.
Although systematic data are lacking, the results of other methods, such as meta-analysis, decision analysis, and cost-effectiveness analysis, have also been variable. For example, meta-analyses on the same topics have reached different conclusions,20 and meta-analyses do not always agree with subsequent clinical trials.21,22 A recent systematic evaluation of the agreement between meta-analyses and subsequent large clinical trials reported a kappa of 0.3.23 Likewise, three independent decision analyses on the use of isoniazid prophylaxis for patients with positive results on tuberculin skin tests came to three different conclusions.24 The estimates of the cost effectiveness of autologous blood donation have also varied greatly, even for the same surgical procedure.25-30 Whether any of these methods is more or less reliable than the appropriateness method remains to be studied systematically.
The area of medicine with the largest amount of rigorous data on reliability is diagnostic testing. Although not a diagnostic test, the appropriateness method shares many characteristics with diagnostic tests, in that both involve classifying patients into two or more categories and both therefore have a reproducibility, false positive, and false negative rate. In ischemic cardiac disease and in women's health, the reliability of thallium scintigraphy for the diagnosis of ischemic cardiac disease has been estimated to have a kappa of 0.4531 and a kappa of 0.66,32 the reliability of coronary angiography in determining the presence or absence of stenosis has been estimated to have a kappa of 0.53,33 the reliability of screening mammography has been estimated to have a kappa of 0.47,34 and the reliability of the classification of cervical smears with grade III histologic features has been estimated to have a kappa of 0.5035 and a kappa of 0.58.36 Given these values, the reproducibility of the appropriateness method is about the same as that of several well-accepted diagnostic tests.
However, the variability we observed in the appropriateness method does have important implications for clinical use. When the method is used to measure rates in a single population, the fact that the classification of inappropriate use varies by a factor of two means that precise estimates are not possible. At best, in a single population, the appropriateness method can estimate whether the proportion of cases with overuse is small or large. The appropriateness method will perform more acceptably as a way to assess the relative proportions of overuse and underuse among populations. Bias due to misclassification will be present in all comparison groups. Although the absolute measure of overuse and underuse may be biased because of misclassification, the relative difference among groups is less likely to be biased.
In making decisions for individual patients, however, the situation is different. Like diagnostic tests, the appropriateness method does not have sufficient reproducibility to justify its use as a gold standard of appropriateness. Clinicians and patients may wish to use results of the appropriateness method as a starting point for discussions about the expected net outcome of a medical procedure. Purchasers, however, should consider the appropriateness method as no more than a screening test to identify care that may be inappropriately under- or overdelivered. Care that is so identified should then be examined at the next level, which must involve direct contact with the provider, and possibly the patient as well, to ascertain additional details about the care delivered. Under no circumstances should the care of individual patients be guided solely by the results of the appropriateness method without additional clinical information.
Our data certainly make it clear that the reproducibility of the appropriateness method could be improved. Although our results for coronary revascularization may be acceptable, we need to know whether the difference between groups of experts considering other procedures is likely to be of a magnitude similar to that seen for hysterectomy between Panel C and the other two panels. The variability in the effect of “major impairment” of function on the appropriateness ratings reflects the different way that Panel C interpreted the trade-off between risk and benefit for these patients; the symptom of major impairment was not judged sufficient to outweigh the risk of the procedure. This finding underscores the variability of physicians' interpretations of the importance of patients' symptoms (as opposed, for example, to mortality or the probability of a myocardial infarction). It also highlights the need for clinical trials of hysterectomy that directly measure symptoms as a primary outcome and the need to involve patients in quality-of-life decisions.
Further research is needed to identify which procedures are likely to be associated with reliable appropriateness-method results. We can conjecture that the more firm evidentiary basis underlying the indications for revascularization resulted in a more reliable extrapolation beyond the evidence on the part of the experts. For hysterectomy, where the evidence was scant and the judgments were dependent on individual values, reliability was reduced. This hypothesis can be further explored by examining in detail the panel discussions or analyzing the results of different panels for different procedures. Multiple determinations have also been suggested as a way to improve the reliability of some diagnostic tests, such as mammography and coronary angiography.33,34
The use of stratified random sampling, the high participation rate achieved, the similarity of the panelists in many features, and the identical nature of the process in each panel all strengthen this study as a fair and rigorous test of the reproducibility of the expert-panel component of the appropriateness method. However, our study has several limitations. Additional components not tested include the development of the systematic review and the construction of the clinical scenarios, each of which may contribute to variability. Also, we studied only two procedures. Although this was a deliberate choice designed to identify likely upper and lower boundaries of reproducibility (with coronary revascularization and hysterectomy, respectively), values for other procedures may be below the values reported here for hysterectomy.
Future studies of the reproducibility of methods identifying overuse and underuse of health procedures should be conducted as rigorously as the study reported here. Only then can we inform with empirical evidence what has thus far been a debate based largely on theory and opinion about how best to determine what care is appropriate.
Supported by a grant (HSO7185-02) from the Agency for Health Care Policy and Research. Dr. Shekelle is the recipient of a Senior Research Associate Career Development Award from the Department of Veterans Affairs.
We are indebted to Mark Chassin, M.D., for helpful comments and to the physicians who served as panelists.
Source Information
From the West Los Angeles Veterans Affairs Medical Center, Los Angeles (P.G.S.); RAND, Santa Monica, Calif. (P.G.S., J.P.K., C.J.K., R.E.P.); the Ann Arbor Veterans Affairs Medical Center and the Departments of Internal Medicine and Health Management and Policy, University of Michigan, Ann Arbor (S.J.B.); and the Harvard School of Public Health, Boston (L.L.L.).
Address reprint requests to Dr. Shekelle at RAND, 1700 Main St., P.O. Box 2138, Santa Monica, CA 90407-2138.
- References
References
1
Leape LL ,Hilborne LH ,Park RE , et al. The appropriateness of use of coronary artery bypass graft surgery in New York State. JAMA 1993;269:753-760
CrossRef | Web of Science | Medline2
Bernstein SJ ,Hilborne LH ,Leape LL , et al. The appropriateness of use of coronary angiography in New York State. JAMA 1993;269:766-769
CrossRef | Web of Science | Medline3
Bernstein SJ ,McGlynn EA ,Siu AL , et al. The appropriateness of hysterectomy: a comparison of care in seven health plans. JAMA 1993;269:2398-2402
CrossRef | Web of Science | Medline4
Fink A ,Brook RH ,Kosecoff J ,Chassin MR ,Solomon DH . Sufficiency of clinical literature on the appropriate uses of six medical and surgical procedures. West J Med 1987;147:609-614
Medline5
Institute of Medicine. Assessing medical technologies. Washington, D.C.: National Academy Press, 1985.
6
Dubinsky M ,Ferguson JH . Analysis of the National Institutes of Health Medicare coverage assessment. Int J Technol Assess Health Care 1990;6:480-488
CrossRef | Medline7
Brook RH ,Chassin MR ,Fink A ,Solomon DH ,Kosecoff J ,Park RE . A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care 1986;2:53-63
CrossRef | Medline8
Gray D ,Hampton JR ,Bernstein SJ ,Kosecoff J ,Brook RH . Audit of coronary angiography and bypass surgery. Lancet 1990;335:1317-1320
CrossRef | Web of Science | Medline9
Bengtson A ,Herlitz J ,Karlsson T ,Brandrup-Wognsen G ,Hjalmarson A . The appropriateness of performing coronary angiography and coronary artery revascularization in a Swedish population. JAMA 1994;271:1260-1265
CrossRef | Web of Science | Medline10
Phelps CE . The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med 1993;329:1241-1245
Full Text | Web of Science | Medline11
Hicks NR . Some observations on attempts to measure appropriateness of care. BMJ 1994;309:730-733
CrossRef | Web of Science | Medline12
Kahan JP ,Bernstein SJ ,Leape LL , et al. Measuring the necessity of medical procedures. Med Care 1994;32:357-365
CrossRef | Web of Science | Medline13
Landis JR ,Koch GG . The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174
CrossRef | Web of Science | Medline14
Rutkow IM ,Gittelsohn AM ,Zuidema GD . Surgical decision making: the reliability of clinical judgment. Ann Surg 1979;190:409-419
CrossRef | Web of Science | Medline15
Roos NP . Hysterectomy: variations in rates across small areas and across physicians' practices. Am J Public Health 1984;74:327-335
CrossRef | Web of Science | Medline16
Hysterectomies in New York State: a statistical profile. Albany: New York State Department of Health, 1988:1-13.
17
Haas S ,Acker D ,Donahue C ,Katz ME . Variation in hysterectomy rates across small geographic areas of Massachusetts. Am J Obstet Gynecol 1993;169:150-154
Web of Science | Medline18
Carlson KJ ,Nichols DH ,Schiff I . Indications for hysterectomy. N Engl J Med 1993;328:856-860
Full Text | Web of Science | Medline19
Selby JV ,Fireman BH ,Lundstrom RJ , et al. Variation among hospitals in coronary-angiography practices and outcomes after myocardial infarction in a large health maintenance organization. N Engl J Med 1996;335:1888-1896
Full Text | Web of Science | Medline20
Chalmers TC ,Berrier J ,Sacks HS ,Levin H ,Reitman D ,Nagalingam R . Meta-analysis of clinical trials as a scientific discipline. II. Replicate variability and comparison of studies that agree and disagree. Stat Med 1987;6:733-744
CrossRef | Web of Science | Medline21
Borzak S ,Ridker PM . Discordance between meta-analyses and large-scale randomized, controlled trials: examples from the management of acute myocardial infarction. Ann Intern Med 1995;123:873-877
Web of Science | Medline22
Cappelleri JC ,Ioannidis JP ,Schmid CH , et al. Large trials vs. meta-analysis of smaller trials: how do their results compare? JAMA 1996;276:1332-1338
CrossRef | Web of Science | Medline23
LeLorier J ,Gregoire G ,Benhaddad A ,Lapierre J ,Derderian F . Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-542
Full Text | Web of Science | Medline24
Colice GL . Decision analysis, public health policy, and isoniazid chemoprophylaxis for young adult tuberculin skin reactors. Arch Intern Med 1990;150:2517-2522
CrossRef | Web of Science | Medline25
Birkmeyer JD ,AuBuchon JP ,Littenberg B , et al. Cost-effectiveness of preoperative autologous donation in coronary artery bypass grafting. Ann Thorac Surg 1994;57:161-168
CrossRef | Web of Science | Medline26
Birkmeyer JD ,Goodnough LT ,AuBuchon JP ,Noordsij PG ,Littenberg B . The cost-effectiveness of preoperative autologous blood donation for total hip and knee replacement. Transfusion 1993;33:544-551
CrossRef | Web of Science | Medline27
Goodnough LT ,Grishaber JE ,Birkmeyer JD ,Monk TG ,Catalona WJ . Efficacy and cost-effectiveness of autologous blood predeposit in patients undergoing radical prostatectomy procedures. Urology 1994;44:226-231
CrossRef | Web of Science | Medline28
Kattan MW ,Eastham JA ,Yawn DH ,Scardino PT . A decision analysis of the cost effectiveness of preoperative autologous blood donation prior to radical prostatectomy for clinically localized prostate cancer. Med Decis Making 1995;15:429-429 abstract.29
Etchason J ,Petz L ,Keeler E , et al. The cost effectiveness of preoperative autologous blood donations. N Engl J Med 1995;332:719-724
Full Text | Web of Science | Medline30
Sonnenberg FA ,Nizam RA ,Yomtovian RA , et al. Cost-effectiveness of autologous blood donation revisited: the impact of increased risk of bacterial infection following allogeneic transfusion. Med Decis Making 1995;15:428-428 abstract.31
Wackers FJ ,Bodenheimer M ,Fleiss JL ,Brown M . Factors affecting uniformity in interpretation of planar thallium-201 imaging in a multicenter trial. J Am Coll Cardiol 1993;21:1064-1074
CrossRef | Web of Science | Medline32
Atwood JE ,Jensen D ,Froelicher V , et al. Agreement in human interpretation of analog thallium myocardial perfusion images. Circulation 1981;64:601-609
CrossRef | Web of Science | Medline33
DeRouen TA ,Murray JA ,Owen W . Variability in the analysis of coronary arteriograms. Circulation 1977;55:324-328
Web of Science | Medline34
Elmore JG ,Wells CK ,Lee CH ,Howard DH ,Feinstein AR . Variability in radiologists' interpretations of mammograms. N Engl J Med 1994;331:1493-1499
Full Text | Web of Science | Medline35
Kato K ,Santamaria M ,De Ruiz PA , et al. Inter-observer variation in cytological and histological diagnoses of cervical neoplasia and its epidemiologic implication. J Clin Epidemiol 1995;48:1167-1174
CrossRef | Web of Science | Medline36
Ismail SM ,Colclough AB ,Dinnen JS , et al. Observer variation in histopathological diagnosis and grading of cervical intraepithelial neoplasia. BMJ 1989;298:707-710
CrossRef | Web of Science | Medline
- Citing Articles (107)
Citing Articles
1
Yvonne M. Drewes, Jacobijn Gussekloo, Victor Meer, Henk Rigter, Janny H. Dekker, Marleen J. B. M. Goumans, Job F. M. Metsemakers, Riki Overbeek, Sophia E. Rooij, Henk J. Schers, Marieke J. Schuurmans, Ferd Sturmans, Kerst Vries, Rudi G. J. Westendorp, Annet W. Wind, Willem J. J. Assendelft. (2011) Assessment of Appropriateness of Screening Community-Dwelling Older People to Prevent Functional Decline. Journal of the American Geriatrics Societyn/a-n/a
CrossRef2
Karen E Hansen, H Alexander Wilson, Carol Zapalowski, Howard A Fink, Salvatore Minisola, Robert A Adler. (2011) Uncertainties in the prevention and treatment of glucocorticoid-induced osteoporosis. Journal of Bone and Mineral Research 26:9, 1989-1996
CrossRef3
, Teryl K. Nuckols, Melinda Maggard Gibbons, Neil G. Harness, Walter T. Chang, Kevin C. Chung, Steven M. Asch. (2011) Clinical quality measures for intraoperative and perioperative management in carpal tunnel surgery. HAND 6:2, 119-131
CrossRef4
Jozette J. C. Stienen, Merit M. Tabbers, Marc A. Benninga, Mirjam Harmsen, Mariëlle M. T. J. Ouwens. (2011) Development of quality indicators based on a multidisciplinary, evidence-based guideline on pediatric constipation. European Journal of Pediatrics
CrossRef5
, Teryl Nuckols, Philip Harber, Karl Sandin, Douglas Benner, Haoling Weng, Rebecca Shaw, Anne Griffin, Steven Asch. (2011) Quality Measures for the Diagnosis and Non-Operative Management of Carpal Tunnel Syndrome in Occupational Settings. Journal of Occupational Rehabilitation 21:1, 100-119
CrossRef6
Amy H. Warriner, Nivedita M. Patkar, Jeffrey R. Curtis, Elizabeth Delzell, Lisa Gary, Meredith Kilgore, Ken Saag. (2011) Which fractures are most attributable to osteoporosis?. Journal of Clinical Epidemiology 64:1, 46-53
CrossRef7
Susanne Kleudgen, Franziska Diel, Friederike Burgdorf, Ingrid Quasdorf, Werner de Cruppé, Max Geraedts. (2011) KBV entwickelt Starter-Set ambulanter Qualitätsindikatoren – AQUIK®-Set. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 105:1, 54-63
CrossRef8
T. To, A. Guttmann, M. D. Lougheed, A. S. Gershon, S. D. Dell, M. B. Stanbrook, C. Wang, S. McLimont, J. Vasilevska-Ristovska, E. J. Crighton, D. N. Fisman. (2010) Evidence-based performance indicators of primary care for asthma: a modified RAND Appropriateness Method. International Journal for Quality in Health Care 22:6, 476-485
CrossRef9
Jennifer M. Grossman, Rebecca Gordon, Veena K. Ranganath, Chad Deal, Liron Caplan, Weiling Chen, Jeffrey R. Curtis, Daniel E. Furst, Maureen McMahon, Nivedita M. Patkar, Elizabeth Volkmann, Kenneth G. Saag. (2010) American College of Rheumatology 2010 recommendations for the prevention and treatment of glucocorticoid-induced osteoporosis. Arthritis Care & Research 62:11, 1515-1526
CrossRef10
Partho P. Sengupta, Bijoy K. Khandheria. (2010) A Rising Paradigm of Appropriateness. Journal of the American Society of Echocardiography 23:11, 1205-1206
CrossRef11
T. Higashi, R. Machii, A. Aoki, C. Hamashima, H. Saito. (2010) Evaluation and Revision of Checklists for Screening Facilities and Municipal Governmental Programs for Gastric Cancer and Colorectal Cancer Screening in Japan. Japanese Journal of Clinical Oncology 40:11, 1021-1030
CrossRef12
Melinda A. Maggard, Neil G. Harness, Walter T. Chang, Janak A. Parikh, Steven M. Asch, Teryl K. Nuckols. (2010) Indications for Performing Carpal Tunnel Surgery: Clinical Quality Measures. Plastic and Reconstructive Surgery 126:1, 169-179
CrossRef13
RM Mendelson, PJ Bairstow. (2010) Inappropriate imaging: Why it matters, why it happens, what can be done. Journal of Medical Imaging and Radiation Oncology 54:3, 173-177
CrossRef14
Theodorus G. Mettes, Wil J.M. van der Sanden, Leontien van Eeten-Kruiskamp, Jan Mulder, Michel Wensing, Richard P.T.M. Grol, Alphons J.M. Plasschaert. (2010) Routine oral examination: Clinical vignettes, a promising tool for continuing professional development?. Journal of Dentistry 38:5, 377-386
CrossRef15
Karl J. Sandin, Steven M. Asch, Charles K. Jablecki, David D. Kilmer, Teryl K. Nuckols, . (2010) Clinical quality measures for electrodiagnosis in suspected carpal tunnel syndrome. Muscle & Nerve 41:4, 444-452
CrossRef16
Nadia Danon-Hersch, Dino Samartzis, Vincent Wietlisbach, François Porchet, John-Paul Vader. (2010) Appropriateness Criteria for Surgery Improve Clinical Outcomes in Patients With Low Back Pain and/or Sciatica. Spine 35:6, 672-683
CrossRef17
N. F. de Larrea, J. A. Blasco, U. Aguirre, S. Garcia, B. Elizalde, G. Navarro, S. Perez, . (2010) Appropriateness of phacoemulsification in Spain. International Journal for Quality in Health Care 22:1, 31-38
CrossRef18
S Evans, C Metcalfe, B Patel, F Ibrahim, K Anson, F Chinegwundoh, C Corbishley, D Gillatt, R Kirby, G Muir, V Nargund, R Popert, P Wilson, R Persad, Y Ben-Shlomo. (2010) Clinical presentation and initial management of Black men and White men with prostate cancer in the United Kingdom: the PROCESS cohort study. British Journal of Cancer 102:2, 249-254
CrossRef19
Robert W. Dubois. (2009) On the second European Panel on the Appropriateness of Crohn's disease Therapy (EPACT-II). Journal of Crohn's and Colitis 3:4, 223-224
CrossRef20
J. M. Molina-Linde, Juan Ramon Lacalle-Remigio, R. Villegas-Portero, D. Díaz-Gómez, L. Gómez-Bujedo, P. Parra-Membrives. (2009) Assessing the appropriate use of laparoscopic surgery for hepatobiliary diseases. Journal of Hepato-Biliary-Pancreatic Surgery 16:6, 844-849
CrossRef21
(2009) For Which Glaucoma Suspects Is It Appropriate to Initiate Treatment?. Ophthalmology 116:4, 710-716.e82
CrossRef22
Jinoos Yazdany, Pantelis Panopalis, Joann Zell Gillis, Gabriela Schmajuk, Catherine H. MacLean, David Wofsy, Edward Yelin, . (2009) A quality indicator set for systemic lupus erythematosus. Arthritis & Rheumatism 61:3, 370-377
CrossRef23
Richard M. Mendelson, Phillip J. Bairstow. (2009) Imaging Pathways: Will They Be Well Trodden or Less Traveled?. Journal of the American College of Radiology 6:3, 160-166
CrossRef24
Laurent Gerbaud, Géraud Manhes, Juliette Debourse, Gérald Gouby, Phyllis-Yvonne Glanddier, John-Paul Vader, Louis Boyer, Patrice Deteix. (2008) The Appropriateness of Renal Angioplasty. The ANPARIA Software: A Multidisciplinary Expert Panel Approach. CardioVascular and Interventional Radiology 31:6, 1059-1068
CrossRef25
J. M. CAVAZOS, A. D. NAIK, A. WOOFTER, N. S. ABRAHAM. (2008) Barriers to physician adherence to nonsteroidal anti-inflammatory drug guidelines: a qualitative study. Alimentary Pharmacology & Therapeutics 28:6, 789-798
CrossRef26
Christopher L. Sistrom, Niccie L. McKay. (2008) Evidence-Based Imaging Guidelines and Medicare Payment Policy. Health Services Research 43:3, 1006-1024
CrossRef27
D.A. O'Reilly, M. Chaudhari, M. Ballal, P. Ghaneh, A. Wu, G.J. Poston. (2008) The Oncosurge strategy for the management of colorectal liver metastases – An external validation study. European Journal of Surgical Oncology (EJSO) 34:5, 538-540
CrossRef28
C. Craig Blackmore, L. Santiago Medina. (2008) Commentary on “In Support of the ACR Appropriateness Criteria®”. Journal of the American College of Radiology 5:5, 636-637
CrossRef29
Christopher Lee Sistrom. (2008) In Support of the ACR Appropriateness Criteria®. Journal of the American College of Radiology 5:5, 630-635
CrossRef30
H. S. Hermanides, M. E. J. L. Hulscher, J. A. Schouten, J. M. Prins, S. E. Geerlings. (2008) Development of Quality Indicators for the Antibiotic Treatment of Complicated Urinary Tract Infections: A First Step to Measure and Improve Care. Clinical Infectious Diseases 46:5, 703-711
CrossRef31
ROD STABLES. 2008. Patient selection for percutaneous coronary intervention. , 19-27.
CrossRef32
S. M. Campbell, J. A. Cantrill. (2008) Consensus methods in prescribing research. Journal of Clinical Pharmacy and Therapeutics 26:1, 5
CrossRef33
Teryl K. Nuckols, Yee-Wei Lim, Barbara O. Wynn, Soeren Mattke, Catherine H. MacLean, Philip Harber, Robert H. Brook, Peggy Wallace, Rena H. Garland, Steven Asch. (2008) Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers. Journal of General Internal Medicine 23:1, 37-44
CrossRef34
K.-P. CHUNG, M.-S. LAI, S.H. CHENG, S.-T. TANG, C.C. HUANG, A.-L. CHENG, P.-C. HSIEH. (2007) Organization-based performance measures of cancer care quality: core measure development for breast cancer in Taiwan. European Journal of Cancer Care 0:0, 070726095645001-???
CrossRef35
Cesare Hassan, Gianluca Bersani, Luigi Buri, Angelo Zullo, Marcello Anti, Maria Antonia Bianco, Emilio Di Giulio, Leonardo Ficano, Sergio Morini, Giovanni Di Matteo, Piero Loriga, Vincenzo Pietropaolo, Livio Cipolletta, Guido Costamagna. (2007) Appropriateness of upper-GI endoscopy: an Italian survey on behalf of the Italian Society of Digestive Endoscopy. Gastrointestinal Endoscopy 65:6, 767-774
CrossRef36
Zsolt Mogyorósy, Gábor Mogyorósy. (2007) Klinikai auditról szóló közlemények kritikus elemzése. Orvosi Hetilap 148:21, 985-991
CrossRef37
Ethan A. Halm. (2007) Sydenham Society: assessing the appropriateness of carotid endarterectomy. Journal of Clinical Epidemiology 60:2, 203-207
CrossRef38
Eugene Oddone. (2007) Sydenham Society: racial variations in carotid endarterectomy. Journal of Clinical Epidemiology 60:2, 208-211
CrossRef39
Pascal Juillerat, Florian Froehlich, Christian Felley, Valérie Pittet, Christian Mottet, Jean-Jacques Gonvers, Pierre Michetti, John-Paul Vader. (2007) EPACT II: Project and Methods. Digestion 76:2, 84-91
CrossRef40
Ronald J. Halbert, Robert A. Figlin, Michael B. Atkins, Myriam Bernal, Thomas E. Hutson, Robert G. Uzzo, Ronald M. Bukowski, Khuda Dad Khan, Christopher G. Wood, Robert W. Dubois. (2006) Treatment of patients with metastatic renal cell cancer. Cancer 107:10, 2375-2383
CrossRef41
Umesh T. Kadam, Kelvin Jordan, Peter R. Croft. (2006) A comparison of two consensus methods for classifying morbidities in a single professional group showed the same outcomes. Journal of Clinical Epidemiology 59:11, 1169-1173
CrossRef42
C.O. Prys-Picard, S.M. Campbell, J.G. Ayres, J.F. Miles, R.M. Niven. (2006) Defining and investigating difficult asthma: Developing quality indicators. Respiratory Medicine 100:7, 1254-1261
CrossRef43
Nananda F. Col, Christine Duffy, Michele G. Cyr. (2006) The pitfalls of non-evidence-based guidelines. Menopause 13:3, 334-337
CrossRef44
Bruce Ettinger, Elizabeth Barrett-Connor, Lalima A. Hoq, John-Paul Vader, Robert W. Dubois. (2006) When is it appropriate to prescribe postmenopausal hormone therapy?. Menopause 13:3, 404-410
CrossRef45
Robert W. Dubois, L. Tim Goodnough, William B. Ershler, Lloyd Van Winkle, Allen R. Nissenson. (2006) Identification, diagnosis, and management of anemia in adult ambulatory patients treated by primary care physicians: evidence-based and consensus recommendations. Current Medical Research and Opinion 22:2, 385-395
CrossRef46
S BARTELS. (2005) Evidence-Based Geriatric Psychiatry: An Overview. Psychiatric Clinics of North America 28:4, 763-784
CrossRef47
MICHAEL H. ALLEN, GLENN W. CURRIER, DANIEL CARPENTER, RUTH W. ROSS, JOHN P. DOCHERTY. (2005) Introduction: Methods, Commentary, and Summary. Journal of Psychiatric Practice 11:Supplement 1, 5-25
CrossRef48
Elizabeth W. Loder, Fred Sheftell. (2005) The Quality of Headache Treatment in The United States: Review and Analysis of Recent Data. Headache: The Journal of Head and Face Pain 45:7, 939-946
CrossRef49
Debra Saliba, David Solomon, Laurence Rubenstein, Roy Young, John Schnelle, Carol Roth, Neil Wenger. (2005) Quality Indicators for the Management of Medical Conditions in Nursing Home Residents. Journal of the American Medical Directors Association 6:3, S36-S48
CrossRef50
Kathleen W. Wyrwich, William M. Tierney, Ajit N. Babu, Kurt Kroenke, Fredric D. Wolinsky. (2005) A Comparison of Clinically Important Differences in Health-Related Quality of Life for Patients with Chronic Lung Disease, Asthma, or Heart Disease. Health Services Research 40:2, 577-592
CrossRef51
Bonnie T. Zima, Michael S. Hurlburt, Penny Knapp, Heather Ladd, Lingqi Tang, Naihua Duan, Peggy Wallace, Abram Rosenblatt, John Landsverk, Kenneth B. Wells. (2005) Quality of Publicly-Funded Outpatient Specialty Mental Health Care for Common Childhood Psychiatric Disorders in California. Journal of the American Academy of Child & Adolescent Psychiatry 44:2, 130-144
CrossRef52
Debra Saliba, David Solomon, Laurence Rubenstein, Roy Young, John Schnelle, Carol Roth, Neil Wenger. (2005) Quality Indicators for the Management of Medical Conditions in Nursing Home Residents. Journal of the American Medical Directors Association 6:SUPPLEMENT, S35???S48
CrossRef53
Debra Saliba, David Solomon, Laurence Rubenstein, Roy Young, John Schnelle, Carol Roth, Neil Wenger. (2004) Quality Indicators for the Management of Medical Conditions in Nursing Home Residents. Journal of the American Medical Directors Association 5:5, 297-309
CrossRef54
R. B. Hakim, M. B. Benedict, N. J. Merrick. (2004) Quality of Care for Women Undergoing a Hysterectomy: Effects of Insurance and Race/Ethnicity. American Journal of Public Health 94:8, 1399-1405
CrossRef55
Curtis E Margo. (2004) Quality care and practice variation: the roles of practice guidelines and public profiles. Survey of Ophthalmology 49:3, 359-371
CrossRef56
M Gabel. (2004) A social choice approach to expert consensus panels. Journal of Health Economics 23:3, 543-564
CrossRef57
Ted R. Mikuls, Catherine H. MacLean, Jason Olivieri, Fausto Patino, Jeroan J. Allison, John T. Farrar, Warren B. Bilker, Kenneth G. Saag. (2004) Quality of care indicators for gout management. Arthritis & Rheumatism 50:3, 937-943
CrossRef58
Michael H Allen, Glenn W Currier. (2004) Use of restraints and pharmacotherapy in academic psychiatric emergency services. General Hospital Psychiatry 26:1, 42-49
CrossRef59
R. W. Dubois, G. Y. Melmed, J. M. Henning, L. Laine. (2004) Guidelines for the appropriate use of non-steroidal anti-inflammatory drugs, cyclo-oxygenase-2-specific inhibitors and proton pump inhibitors in patients requiring chronic anti-inflammatory therapy. Alimentary Pharmacology and Therapeutics 19:2, 197-208
CrossRef60
Donna L. Washington, Steven J. Bernstein, James P. Kahan, Lucian L. Leape, Caren J. Kamberg, Paul G. Shekelle. (2003) Reliability of Clinical Guideline Development Using Mail-Only versus In-Person Expert Panels. Medical Care 41:12, 1374-1381
CrossRef61
Arnold M. Epstein, Joel S. Weissman, Eric C. Schneider, Constantine Gatsonis, Lucian L. Leape, Robert N. Piana. (2003) Race and Gender Disparities in Rates of Cardiac Revascularization. Medical Care 41:11, 1240-1255
CrossRef62
C.P Wilkinson, Frederick L Ferris, Ronald E Klein, Paul P Lee, Carl David Agardh, Matthew Davis, Diana Dills, Anselm Kampik, R Pararajasegaram, Juan T Verdaguer. (2003) Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110:9, 1677-1682
CrossRef63
McGlynn, Elizabeth A., Asch, Steven M., Adams, John, Keesey, Joan, Hicks, Jennifer, DeCristofaro, Alison, Kerr, Eve A., . (2003) The Quality of Health Care Delivered to Adults in the United States. New England Journal of Medicine 348:26, 2635-2645
Full Text64
Elizabeth A. McGlynn, Eve A. Kerr, John Adams, Joan Keesey, Steven M. Asch. (2003) Quality of Health Care for Women. Medical Care 41:5, 616-625
CrossRef65
Jeroan J. Allison. (2003) Quality Assessment Tools. Medical Care 41:5, 575-578
CrossRef66
Kathleen W. Wyrwich, Stephan D. Fihn, William M. Tierney, Kurt Kroenke, Ajit N. Babu, Fredric D. Wolinsky. (2003) Clinically Important Changes in Health-related Quality of Life for Patients with Chronic Obstructive Pulmonary Disease. An Expert Consensus Panel Report. Journal of General Internal Medicine 18:3, 196-202
CrossRef67
MICHAEL H. ALLEN, GLENN W. CURRIER, DOUGLAS H. HUGHES, JOHN P. DOCHERTY, DANIEL CARPENTER, RUTH ROSS. (2003) Treatment of Behavioral Emergencies: A Summary of the Expert Consensus Guidelines. Journal of Psychiatric Practice 9:1, 16-38
CrossRef68
MICHAEL H. ALLEN, DANIEL CARPENTER, JOHN L. SHEETS, STEVEN MICCIO, RUTH ROSS. (2003) What Do Consumers Say They Want and Need During a Psychiatric Emergency?. Journal of Psychiatric Practice 9:1, 39-58
CrossRef69
Nancy Wolff, Mark Schlesinger. (2002) Clinicians as Advocates. The Journal of Behavioral Health Services & Research 29:3, 274???287
CrossRef70
Nancy Wolff, Mark Schlesinger. (2002) Clinicians as advocates: An exploratory study of responses to managed care by mental health professionals. The Journal of Behavioral Health Services & Research 29:3, 274-287
CrossRef71
Debra Saliba, John F. Schnelle. (2002) Indicators of the Quality of Nursing Home Residential Care. Journal of the American Geriatrics Society 50:8, 1421-1430
CrossRef72
V. L. Roger, S. J. Jacobsen, S. A. Weston, P. A. Pellikka, T. D. Miller, K. R. Bailey, B. J. Gersh. (2002) Sex Differences in Evaluation and Outcome After Stress Testing. Mayo Clinic Proceedings 77:7, 638-645
CrossRef73
Joseph D. Restuccia, Michael Shwartz, Bernard E. Kreger, Susan M.C. Payne, Arlene S. Ash, Lisa I. Iezzoni, Janelle Heineke, Harry P. Selker, Theresa Gomes, Alan Labonte, John R. Butterly. (2002) Does More “Appropriateness” Explain Higher Rates of Cardiac Procedures Among Patients Hospitalized With Coronary Heart Disease?. Medical Care 40:6, 500-509
CrossRef74
Thomas M. Wickizer, Daniel Lessler. (2002) U
TILIZATION MANAGEMENT : Issues, Effects, and Future Prospects. Annual Review of Public Health 23:1, 233-254
CrossRef75
Curtis E Margo. (2002) Peer and expert opinion and the reliability of implicit case review. Ophthalmology 109:3, 614-618
CrossRef76
Joseph P. Mathew, Manuel L. Fontes, Susan Garwood, Elizabeth Davis, William D. White, Gerard McCloskey, Jane C. K. Fitch, Sherif Afifi, David L. Lee, Phillip Kraker, Terence D. Rafferty, Paul G. Barash, Linda Gillam, Edward Prokop. (2002) Transesophageal Echocardiography Interpretation: A Comparative Analysis Between Cardiac Anesthesiologists and Primary Echocardiographers. Anesthesia & Analgesia 94:2, 302-309
CrossRef77
McNeil, Barbara J., . (2001) Hidden Barriers to Improvement in the Quality of Care. New England Journal of Medicine 345:22, 1612-1620
Full Text78
GEORGE S. ALEXOPOULOS, IRA R. KATZ, CHARLES F. REYNOLDS, DANIEL CARPENTER, JOHN P. DOCHERTY, RUTH W. ROSS. (2001) Pharmacotherapy of Depression in Older Patients: A Summary of the Expert Consensus Guidelines. Journal of Psychiatric Practice 7:6, 361-376
CrossRef79
Paul G Shekelle, R.E Park, James P Kahan, Lucian L Leape, Caren J Kamberg, Steven J Bernstein. (2001) Sensitivity and specificity of the RAND/UCLA Appropriateness Method to identify the overuse and underuse of coronary revascularization and hysterectomy. Journal of Clinical Epidemiology 54:10, 1004-1010
CrossRef80
Steven J. Bernstein, Pablo Lázaro, Kathryn Fitch, María Dolores Aguilar, James P. Kahan. (2001) Effect of Specialty and Nationality on Panel Judgments of the Appropriateness of Coronary Revascularization. Medical Care 39:5, 513-520
CrossRef81
LORI L. ALTSHULER, LEE S. COHEN, MARGARET L. MOLINE, DAVID A. KAHN, DANIEL CARPENTER, JOHN P. DOCHERTY, RUTH W. ROSS. (2001) Treatment of Depression in Women: A Summary of the Expert Consensus Guidelines. Journal of Psychiatric Practice 7:3, 185-208
CrossRef82
Hemingway, Harry, Crook, Angela M., Feder, Gene, Banerjee, Shrilla, Dawson, J. Rex, Magee, Patrick, Philpott, Sue, Sanders, Julie, Wood, Alan, Timmis, Adam D., . (2001) Underuse of Coronary Revascularization Procedures in Patients Considered Appropriate Candidates for Revascularization. New England Journal of Medicine 344:9, 645-654
Full Text83
Shekelle, Paul G., . (2001) Are Appropriateness Criteria Ready for Use in Clinical Practice?. New England Journal of Medicine 344:9, 677-678
Full Text84
S. M. Campbell, J. A. Cantrill. (2001) Consensus methods in prescribing research. Journal of Clinical Pharmacy and Therapeutics 26:1, 5-14
CrossRef85
José M. Quintana, Inmaculada Aróstegui, Jesús Azkarate, J.Ignacio Goenaga, Xabier Elexpe, Jon Letona, Andoni Arcelay. (2000) Evaluation of explicit criteria for total hip joint replacement. Journal of Clinical Epidemiology 53:12, 1200-1208
CrossRef86
Epstein, Arnold M., Ayanian, John Z., Keogh, Joseph H., Noonan, Susan J., Armistead, Nancy, Cleary, Paul D., Weissman, Joel S., David-Kasdan, Jo Ann, Carlson, DianeFuller, Jerry, Marsh, DouglasConti, Rena M.. (2000) Racial Disparities in Access to Renal Transplantation — Clinically Appropriate or Due to Underuse or Overuse?. New England Journal of Medicine 343:21, 1537-1544
Full Text87
Marjorie L. Pearson, Jan L. Lee, Betty L. Chang, Marc Elliott, Katherine L. Kahn, Lisa V. Rubenstein. (2000) Structured Implicit Review. Medical Care 38:11, 1074-1091
CrossRef88
K Fitch. (2000) European criteria for the appropriateness and necessity of coronary revascularization procedures. European Journal of Cardio-Thoracic Surgery 18:4, 380-387
CrossRef89
Alexander S. Young, Sandra L. Forquer, Anh Tran, Midge Starzynski, Jess Shatkin. (2000) Identifying clinical competencies that support rehabilitation and empowerment in individuals with severe mental illness. The Journal of Behavioral Health Services & Research 27:3, 321-333
CrossRef90
John-Paul Vader, François Porchet, Tania Larequi-Lauber, Robert W. Dubois, Bernard Burnand. (2000) Appropriateness of Surgery for Sciatica. Spine 25:14, 1831-1836
CrossRef91
Susannah C Daly, Veronique L Roger, Cynthia Leibson, Todd D Miller, Patricia A Pellikka, Kent Bailey, Steven J Jacobsen. (2000) Cardiology services after stress testing. Journal of Clinical Epidemiology 53:7, 661-668
CrossRef92
Lisa V. Rubenstein, Brian S. Mittman, Elizabeth M. Yano, Cynthia D. Mulrow. (2000) From Understanding Health Care Provider Behavior to Improving Health Care. Medical Care 38, I-129-I-141
CrossRef93
Eric J. Thomas, E. John Orav, Troyen A. Brennan. (2000) Hospital Ownership and Preventable Adverse Events. Journal of General Internal Medicine 15:4, 211-219
CrossRef94
Timothy M. Powell, Jeffrey P. Thompsen, Katherine S. Virgo, Eric T. Johnson, Danny Chan, John W. Colberg, David K. Ornstein, Frank E. Johnson. (2000) Geographic Variation in Patient Surveillance After Radical Prostatectomy. Annals of Surgical Oncology 7:5, 339-346
CrossRef95
Eric J. Thomas, David M. Studdert, Helen R. Burstin, E. John Orav, Timothy Zeena, Elliott J. Williams, K. Mason Howard, Paul C. Weiler, Troyen A. Brennan. (2000) Incidence and Types of Adverse Events and Negligent Care in Utah and Colorado. Medical Care 38:3, 261-271
CrossRef96
Timothy P. Hofer, Steven J. Bernstein, Sonya DeMonner, Rodney A. Hayward. (2000) Discussion Between Reviewers Does Not Improve Reliability of Peer Review of Hospital Quality. Medical Care 38:2, 152-161
CrossRef97
Robert Peter Gale, Rolla Edward Park, Robert Dubois, Jacob D Bitran, Aman Buzdar, Gabriel Hortobagyi, Stephen E Jones, Gary S Lazar, Gary Spitzer, Sandra M Swain, Clarence B Vaughn, Charles E Vogel, Silvana Martino. (2000) Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in women with breast cancer. Clinical Transplantation 14:1, 32-41
CrossRef98
Harald Herholz. (2000) Diagnostischer Overkill in der invasiven Kardiologie?. Herz 25:1, 62-64
CrossRef99
Robert Peter Gale, Rolla Edward Park, Robert W. Dubois, Geoffrey P. Herzig, William G. Hocking, Mary M. Horowitz, Armand Keating, Sanford Kempin, Charles A. Linker, Charles A. Schiffer, Peter H. Wiernik, Daniel J. Weisdorf, Kanti R. Rai. (1999) Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in chronic myelogenous leukemia in chronic phase. Leukemia Research 23:9, 817-826
CrossRef100
Stephen M. Campbell, Mark Hann, Martin O. Roland, Julie Ann Quayle, Paul G. Shekelle. (1999) The Effect of Panel Membership and Feedback on Ratings in a Two-Round Delphi Survey. Medical Care 37:9, 964-968
CrossRef101
Joanne K. Tobacman, Ingrid U. Scott, Stacey Cyphert, Bridget Zimmerman. (1999) Reproducibility of Measures of Overuse of Cataract Surgery by Three Physician Panels. Medical Care 37:9, 937-945
CrossRef102
Robert Peter Gale, Rolla Edward Park, Robert W. Dubois, Geoffrey P. Herzig, William G. Hocking, Mary M. Horowitz, Armand Keating, Sanford Kempin, Charles A. Linker, Charles A. Schiffer, Peter H. Wiernik, Daniel J. Weisdorf, Kanti R. Rai. (1999) Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute myelogenous leukemia in 1st remission. Leukemia Research 23:8, 709-718
CrossRef103
Vincent Wietlisbach, John-Paul Vader, François Porchet, Michael C. Costanza, Bernard Burnand. (1999) Statistical Approaches in the Development of Clinical Practice Guidelines From Expert Panels. Medical Care 37:8, 785-797
CrossRef104
Joel S. Weissman, John Z. Ayanian, Scott Chasan-Taber, Marjorie J. Sherwood, Carol Roth, Arnold M. Epstein. (1999) Hospital Readmissions and Quality of Care. Medical Care 37:5, 490-501
CrossRef105
Benjamin W. Johnson. (1998) Cost-effectiveness and pain medicine. Current Review of Pain 2:4, 254-266
CrossRef106
(1998) Assessing the Appropriateness of Medical Care. New England Journal of Medicine 339:20, 1478-1481
Full Text107
Naylor, C. David, . (1998) What is Appropriate Care?. New England Journal of Medicine 338:26, 1918-1920
Full Text
