Correspondence
Practice Profiles
N Engl J Med 1994; 331:201-203July 21, 1994
- Article
To the Editor:
The article by Welch et al. (March 3 issue)1 on physician profiling describes that method well but also illustrates its shortcomings. Adjustment for case mix is the Achilles' heel of physician profiling. All physicians know how flimsy the relation is between actual clinical severity and statistical diagnosis-related groups, not just in individual cases, but for a population as well. Welch et al. omit the age of the Medicare population studied in Oregon as compared with the age of the one studied in Florida. Age alone could explain the difference in utilization found in their study.
1 ReferencesJoseph C. Masdeu, M.D.
New York Medical College, Valhalla, NY 105951
Welch HG ,Miller ME ,Welch WP . Physician profiling -- an analysis of inpatient practice patterns in Florida and Oregon. N Engl J Med 1994;330:607-612
Full Text | Web of Science | Medline
To the Editor:
On January 1, 1993, Blue Shield for the National Capital Area “deselected” about 3000 doctors from the Federal Blue Shield Program, the insurance program used by about one third of the members of the Congress and, according to CNN, the First Family. The computer system used, the ProFile system, has been mentioned previously in the Journal1. It operates in total secrecy, and all efforts to review its methods, accuracy, or validity publicly have been rebuffed by Blue Shield. The only way physicians in Washington will be able to find out why they were excluded by Blue Shield is to file a lawsuit at great expense.
I submit that you did not go far enough in your editorial on physician profiling (March 3 issue)2. You called for a debate on profiling. It is too late for such a debate. It is time to call for congressional hearings to ascertain the validity, accuracy, uses, and impact of physician-profiling schemes on medical practice in this country. It is also time to demand that physicians and the scientific practice of medicine receive federal protection from the profiling industry, by requiring that the methods and use of profiling services be regulated and controlled by the Department of Medical Devices of the Food and Drug Administration.
It is common knowledge that an entire generation of physicians has learned to practice expensive defensive medicine, fearing lawsuits. Welch et al.3 compared medical care costs in Florida and Oregon and found those in Florida higher than statistically “expected” on the basis of the comparison. If these authors knew anything about the malpractice situation in Florida, they would have cogently expected a differential of 200 percent, instead of only 53 percent. I guess statisticians never get sued.
I cite this example of costs being driven up by the fear of lawsuit to show how dangerous the profiling industry is. The only injury to a doctor from a lawsuit is one involving time and irritation. Doctors are insured from economic loss. The injury they are suffering every day from these adverse profiles is absolute economic destruction. By separating patients from their own physicians by strong or absolute economic penalties or disincentives, these schemes will produce a reign of terror over physicians, whose only defense from arbitrary and summary termination by insurance companies will be to offer cheapened, primitive, minimalist medical practice. This is contrary to the public welfare and must be condemned and stopped as an emergency.
3 ReferencesJohn H. Lossing, M.D.
730 24th St., NW, Washington, DC 200371
Iglehart JK . The American health care system -- managed care. N Engl J Med 1992;327:742-747
Full Text | Web of Science | Medline2
Kassirer JP . The use and abuse of practice profiles. N Engl J Med 1994;330:634-636
Full Text | Web of Science | Medline3
Welch HG ,Miller ME ,Welch WP . Physician profiling -- an analysis of inpatient practice patterns in Florida and Oregon. N Engl J Med 1994;330:607-612
Full Text | Web of Science | Medline
To the Editor:
A deeper problem underlies practice profiles than those you discuss. Stein's paradox is the counterintuitive finding that a comparison of three or more entities based on the use of averages will be systematically biased. Averages overestimate the true value when they are high and underestimate it when they are low. It is always possible to find another estimator (e.g., the James-Stein estimator1) that is more accurate than the naive average.
Why doesn't the naive average work for groups, when it works for individuals? The naive average misleads when no prior hypotheses are specified for the physicians. With no presupposition that one outlier would be lowest and another highest, their appearance at those extremes is due as much to chance as to innate performance. Whatever the unit of observation, if three or more estimates are needed, the James-Stein estimator will outperform the naive average.
This leads to several recommendations for better use of profile results. First, ignore extreme observations. The good ones are not as good as they appear, nor the bad as bad. Second, beware of absolute standards. Since the extreme values of the naive averages have the greatest error, those falling below some cutoff value are the ones most likely to be wrong. Finally, large variations in profiles need not imply problems with care. The true range may not be nearly as wide as it appears from the naive averages.
Using the James-Stein estimator should help focus attention where it belongs -- on the distribution of results as a whole, and particularly on moving the entire distribution in the desired direction. Ignoring Stein's paradox may lead to unnecessary and potentially harmful changes in practice, with no corresponding benefit.
1 ReferencesRobert L. Wears, M.D., M.S.
University of Florida, Jacksonville, FL 32209-65111
Efron B ,Morris C . Stein's paradox in statistics. Sci Am 1977;236:119-127
CrossRef | Web of Science
Author/Editor Response
The authors reply:
To the Editor: Dr. Masdeu is concerned about the adequacy of our case-mix adjustment and wonders whether the age of the Medicare population in Florida differs from that in Oregon. Florida does have a higher proportion of elderly residents (18 percent) than Oregon (14 percent). Within the Medicare population (people over 65 years of age), however, the age structure in the two states is remarkably similar (Table 1Table 1
Medicare Population of Florida and Oregon, According to Age Group.). Because our profiling data are derived from the Medicare population, age cannot explain the differences observed. To confirm this, we calculated relative-value units per beneficiary (i.e., physicians' services to inpatients and outpatients) adjusted for age, sex, and race, using 1990 Medicare data and found the rate in Florida to be 77 percent higher than that in Oregon.On the other hand, we do not dispute that there are shortcomings in our case-mix adjustment. We doubt that physicians' concern in this area can ever be fully allayed at the level of the individual case. But the problem of adjustment for case mix highlights the advantages of focusing physician review on patterns of care involving many patients instead of using a case-by-case approach. Although all physicians care for some patients who are sicker (and require more resources than average), only a few will do so consistently.
Dr. Lossing laments the “deselection” of physicians in Washington, D.C., because the methods behind it were unclear. Ironically, one of our motivations in publishing this work was to make one method of profiling explicit and expose it to criticism. Unfortunately, Dr. Lossing goes on to confuse the measurement itself (the profile) with the actions based on it. There are many other possible outlets for physician profiles beyond economic credentialing. Simply telling physicians where they stand seems like a reasonable place to start.
Dr. Lossing further asserts that the malpractice environment explains the differences between Florida and Oregon. We made no attempt to identify which non-patient-related factors might best explain the observed differences (e.g., defensive medicine, entrepreneurial practice styles, or availability of technology). Nevertheless, we have provided a measure with which the question of the effect of the local malpractice environment may be explored.
Dr. Lossing's closing plea to condemn all efforts to refine such measurements seems to imply that physicians should never examine their own patterns of practice. We understand the desire to make individual clinical decisions unfettered by micromanagement. However, such day-to-day autonomy comes with the responsibility for the use of resources over the long run. Dr. Lossing seems to deny the prerogative of society to care about the cost of services over time. Were the entire profession to share this view, physicians would be inviting the kind of oversight they most despise.
H. Gilbert Welch, M.D., M.P.H.
Department of Veterans Affairs, White River Junction, VT 05009Mark E. Miller, Ph.D.
W. Pete Welch, Ph.D.
Urban Institute, Washington, DC 20037Author/Editor Response
Dr. Kassirer replies:
In my editorial I expressed great concern about how commercially generated profiles are being used. In addition, I said, “Companies in the business of producing profiles are monitoring physicians, but at present nobody is monitoring them.” In my opinion, organizations that represent physicians have not been sufficiently vocal in opposing the use of industry-generated profiles based on proprietary, unverified criteria. Nonetheless, I do believe that better measures of physicians' performance must be developed, that practice profiles are a promising method, and that we must be engaged in developing profiles that accurately reflect the quality of a physician's practice.
The point Dr. Wears makes is relevant to any management decisions based on short-term performance. In fact, the long-range performance of outliers, whether at the high or the low end of the scale of excellence, is likely not to be as atypical as their short-term performance would imply. It follows that decisions designed to sanction poor performers or reward top performance must be tempered by the realization that the observed short-term performance may overstate fundamental attributes of quality. In constructing profiles, we should assess performance over periods long enough to reduce the possibility that variations are the consequence of chance alone, and we should bear in mind that an approach that focuses only on the performance of outliers may be misdirected.
Jerome P. Kassirer, M.D.
- Citing Articles (1)
Citing Articles
1
David M. Shahian, Sharon-Lise Normand, David F. Torchiana, Stanley M. Lewis, John O. Pastore, Richard E. Kuntz, Paul I. Dreyer. (2001) Cardiac surgery report cards: comprehensive review and statistical critique11This review is an abridged version of a report submitted by the Massachusetts Cardiac Care Quality Commission to the Massachusetts Legislature, May 2001.. The Annals of Thoracic Surgery 72:6, 2155-2168
CrossRef
