Correspondence
Report Cards on Cardiac Surgeons
N Engl J Med 1995; 333:938-939October 5, 1995
- Article
To the Editor:
Green and Wintfeld (May 4 issue)1 raise valid questions about the severity coding in the New York State Department of Health's Cardiac Surgery Reporting System (CSRS). Some errors in their analysis, however, may cause policy makers to draw the wrong conclusions about the usefulness of hospital rankings in general.
The main statistical concern of the authors is that of “predictive accuracy.” They incorrectly suggest that a low R2 value for the correlation between observed and expected death rates for hospitals (or surgeons), as estimated by the CSRS model,2 indicates that the CSRS model has poor predictive accuracy. A poor correlation between observed and expected rates might be due to differences in the quality of care provided by hospitals (or surgeons) or to similarities in the initial severity of disease and the case mix, even if the underlying CSRS model perfectly reflected the true probability of death per patient. Other methods to evaluate the CSRS model should have included the standard techniques used for evaluating logistic-regression models.3
Furthermore, changes in rank “so extensive that in one year 46 percent of the surgeons had moved from one half of the ranked list to the other” provide little evidence of deficiencies in the predictive accuracy of identifying outliers, which is how the model was intended to be used. The authors would do better to determine the stability of the outlier hospital (or surgeon) rankings over time. In earlier work criticizing the Health Care Financing Administration's mortality model, the authors correctly performed a similar type of analysis.4
Green and Wintfeld criticize the use of in-hospital death as an outcome measure, yet the evidence against its use is weak. Stating that 38 percent of surgeons “changed rank by at least 10 positions” (or that 62 percent changed rank by fewer than 10 positions) when the data were reanalyzed, the authors again fail to provide evidence that the model lacks accuracy in identifying outliers. In constructing a test between in-hospital death within 30 days and any in-hospital death, the authors have also failed to determine whether any bias would have occurred had all deaths within 30 days been analyzed.
4 ReferencesJeffrey H. Silber, M.D., Ph.D.
University of Pennsylvania, Philadelphia, PA 191041
Green J ,Wintfeld N . Report cards on cardiac surgeons -- assessing New York State's approach. N Engl J Med 1995;332:1229-1232
Full Text | Web of Science | Medline2
Hannan EL ,Kilburn H Jr ,Lindsey ML ,Lewis R . Clinical versus administrative data bases for CABG surgery: does it matter? Med Care 1992;30:892-907
CrossRef | Web of Science | Medline3
Silber JH ,Rosenbaum PR ,Ross RN . Comparing the contributions of groups of predictors: which outcomes vary with hospital rather than patient characteristics? J Am Stat Assoc 1995;90:7-18
CrossRef | Web of Science4
Green J ,Wintfeld N ,Sharkey P ,Passman LJ . The importance of severity of illness in assessing hospital mortality. JAMA 1990;263:241-246
CrossRef | Web of Science | Medline
To the Editor:
Drs. Green and Wintfeld imply that the CSRS model is inadequate. Although we agree with many of the points made by the authors, we do not agree with the implication that the CSRS model is primitive and could be greatly improved. Few models have been as thoroughly researched as those predicting mortality after coronary-artery bypass grafting. Since the CSRS model contains the most important elements of the other models and performs about as well, it is unlikely that the use of additional, generally available, objective risk factors would more than incrementally improve the ability of the CSRS model to predict mortality.
Instead of using results from other studies to evaluate the performance of the CSRS model, the authors used absolute standards. They imply that the model did not perform well because it accounted for only 7.3 percent of the variance in surgeon-specific mortality rates, and the risk-adjusted mortality rate for one year accounted for only 4.9 percent of the variance in the risk-adjusted mortality rate for the following year. Because of the low number of deaths per surgeon, however, there is a large amount of random variation in the mortality rates.
We evaluated the performance of a perfect model with 100 simulated data sets that had the same average mortality rate, the same variation in mortality rates, and the same average number of patients per surgeon as the CSRS data. In these data sets the true physician-specific mortality rates accounted for 31 percent of the variation in the observed mortality rates, and the risk-adjusted mortality rate for one year accounted for only 18 percent of the variance in the risk-adjusted mortality rate for the following year. The CSRS model appears somewhat better when compared to these standards of perfection.
Our view is that the CSRS model needs fine-tuning rather than an overhaul. After it has been limited to well-defined, objectively determined data elements, possibly with a few additional data elements, the CSRS model should set the standard for risk-adjusted outcome models in the future.
Arthur J. Hartz, M.D., Ph.D.
Evelyn M. Kuhn, Ph.D.
Kenneth L. Kayser, M.S.
Medical College of Wisconsin, Milwaukee, WI 53226-0509Author/Editor Response
The authors reply:
To the Editor: We reported that the CSRS risk model explains only 7.3 percent of the variation in surgeon-specific mortality rates and a nonsignificant 0.4 percent of the variation in hospital mortality. We were optimistic that the CSRS model could be improved but not without substantial modification of the data-collection protocol.
Hartz et al. are incorrect in attributing the limited correlations obtained from the CSRS model to sampling variability. The correlation between surgeons' predicted and observed mortality rates is a function of both the sizes of surgeons' caseloads and the accuracy of the underlying model that generates patient-level risk estimates. A patient-level model that discriminated survivors from decedents with perfect calibration and resolution would also predict surgeon-specific death rates with perfect accuracy, regardless of the sample size.
Silber offers another possible explanation of the low R2 values we reported. He suggests that if surgeons had similar patients but differed substantially in the quality of the care they provided, patients' risk factors would account for very little of the variance. However, CSRS data indicate that the surgeons differed greatly with respect to case mix. Furthermore, if the quality of the care provided by individual surgeons is the chief determinant of outcomes, then adding variables to the CSRS model that represent individual surgeons should substantially improve its predictive power; on re-estimation, however, such improvement does not occur.
We emphasized rankings rather than outliers in our assessment because the annual CSRS report1 and the media coverage of it2 emphasized rankings. Our results did not depend on this choice, however, because the outlier designation, like rank, was a poor predictor of a surgeon's future performance.
Hartz et al. suggest that CSRS collect only “well-defined, objectively determined data elements.” This proposal overlooks our finding that even left ventricular ejection fraction, an objective variable, had extremely poor reliability in CSRS.3 Our study identified serious obstacles in obtaining accurate data for “report cards.” In particular, the objectivity of CSRS was compromised when the surgeons who collected the data realized that they were the subjects of the study. It is difficult to conceive of a stronger inherent bias.
3 ReferencesJesse Green, Ph.D.
Neil Wintfeld, Ph.D.
New York University Medical Center, New York, NY 100161
Coronary artery bypass graft surgery in New York State 1989-1991. Albany: New York State Department of Health, December 1992.
2
Zinman D. Cardiac centers ranked. New York Newsday. December 8, 1992:4.
3
Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State's approach: final report to the United Hospital Fund, New York, N.Y. New York: New York University Medical Center, September 30, 1994.
- Citing Articles (1)
Citing Articles
1
David M. Shahian, Sharon-Lise Normand, David F. Torchiana, Stanley M. Lewis, John O. Pastore, Richard E. Kuntz, Paul I. Dreyer. (2001) Cardiac surgery report cards: comprehensive review and statistical critique11This review is an abridged version of a report submitted by the Massachusetts Cardiac Care Quality Commission to the Massachusetts Legislature, May 2001.. The Annals of Thoracic Surgery 72:6, 2155-2168
CrossRef
