Perspective

A Map to Bad Policy — Hospital Efficiency Measures in the Dartmouth Atlas

Peter B. Bach, M.D., M.A.P.P.

N Engl J Med 2010; 362:569-574February 18, 2010

Article

The Debate over Regional Variation in Health Care Spending

The regional variations in health care spending that are documented by the Dartmouth Atlas of Health Care have been cited by many as a justification, and possible basis, for changes in provider payment rates. The article below — and the response that follows it — address various concerns about the Dartmouth data.

In showing that regional spending variations do not correlate with differences in disease burden or outcome, the Dartmouth Atlas of Health Care has felled the notion that higher health care spending necessarily leads to improved health outcomes. Policymakers have seized on two possible ways to wring savings out of this information. The health care reform bill passed by the House of Representatives contains provisions for identifying regions where Medicare spending appears disproportionately high and adjusting payment rates accordingly. And some policymakers, including President Barack Obama, have proposed that the features of high-performing, “efficient” health care systems should be identified and their lower-cost practices emulated.

Dartmouth Atlas researchers have also begun attempting to convert their observations into cost-saving policies. By analyzing Medicare claims for people who have been treated in hospitals and have died, they aim to identify and rank high- and low-efficiency hospitals.1 Because the Atlas is so influential, their rankings could have broad effects on policy. They could affect hospitals' payments and prestige, and Consumer Reports is already publicizing them to consumers (www.consumerhealthreports.org). Given their potentially far-reaching implications, it is concerning that the rankings are unsound, both conceptually and methodologically.

The conceptual problem lies in the fact that in Atlas analyses all health care costs that are incurred by patients over the 2 years before their death are attributed to the hospital where they were admitted most frequently during that period. This method assumes that the hospital controls all, or at least most, patient care, even if it occurs outside the hospital or in another hospital. It thus seems to presuppose a system in which hospitals are accountable for all care — perhaps a noble long-term objective, but not a current reality.

Hospitals can sometimes influence the immediate follow-up care of discharged patients, but much care is beyond their control. According to the Center for Studying Health System Change, less than 20% of U.S. physicians have a financial connection with any hospital. Home health agencies and inpatient facilities for subacute care — which tend to have high costs — are designed to serve multiple hospitals, not a single hospital that can exert control over them. Frequently, even the hospital to which a patient is assigned has very limited contact with that patient. In one Atlas analysis, one third of the patients who were included in an assessment of hospital efficiency had been admitted to the hospital in question only once.2

Atlas-based analyses are also hampered by methodologic problems, starting with their implicit definition of efficiency. A true analysis of efficiency would ask “whether healthcare resources are being used to get . . . improved health,”3 weighing both resources consumed and outcomes. Yet Atlas efficiency rankings consider only costs (i.e., resources consumed).

Conceptually, this approach would be appropriate only if outcomes were the same in all hospitals, so that costs equaled efficiency. But since outcomes vary among hospitals and providers, both costs and outcomes must be assessed in evaluating efficiency. Atlas researchers might correctly argue that costs correlate poorly with outcomes. But poor correlation does not imply that outcomes are homogeneous, but rather that there are high-spending hospitals that use resources in a manner that improves outcomes and others that squander resources, failing to improve health. The same goes for low-spending hospitals. Figuring out which is which is the purpose of efficiency assessment, which therefore requires consideration of both costs and outcomes.

Say Hospital A and Hospital B each has a group of patients with a fatal disease. Hospital A gives each patient a $1 pill and cures half of them; Hospital B provides no treatment. An Atlas analysis would conclude that Hospital B was more efficient, since it spent less per decedent. But all the patients die at Hospital B, whereas only half of the patients do at Hospital A, where the cost per life saved is a bargain at $2. Although $1 cures are rare, changing the price or efficacy of the pill does not alter the fundamental problem with examining costs alone when cost differences are sometimes associated with outcome differences.

Another methodologic problem is that Atlas analyses assess hospital efficiency overall on the basis of costs incurred for nonrepresentative patients — decedents who were enrolled in fee-for-service Medicare. This group varies among hospitals in terms of severity of illness and is not representative of a given hospital's overall spending pattern.

Regarding illness severity, Atlas researchers note on their Web site (www.dartmouthatlas.org/faq/hospital.shtm) that they focus on “patients who died so that [they can be sure] that patients were similarly ill across hospitals,” further explaining that “by definition, the prognosis of all patients [who died was] identical — all were dead . . . therefore, variations [in resource use] cannot be explained by differences in the severity of illness.” But since some hospitals take care of sicker patients than others, the average severity of illness of patients who die also varies among hospitals. This fact is being ignored when all spending differences are attributed to differences in efficiency.4

My analysis of data from the 2006 Nationwide Inpatient Sample on decedents 65 years of age or older illustrates how far off the Atlas assumption is (see graphDistribution among Hospitals of the Average Predicted Risk of Death at Admission among Patients 65 Years of Age or Older Who Died.). Using “All-Patient–Refined Diagnosis-Related Groups,” I found that among such decedents, the predicted risk of death at the time of admission varied widely among hospitals. At the average hospital, the average risk was 15%. But the severity of illness was far lower in hospitals at the 10th percentile (6% risk of death) and far higher in those at the 90th percentile (22% risk of death). Differences in illness severity result in differences in resource consumption. For example, the average length of stay for this population was 4.4 days in hospitals at the 10th percentile but 9.3 days in those at the 90th percentile. In other words, the average decedent in a hospital with a low average severity of illness started out much less sick than the average decedent in a hospital with high average severity, and differences in resource consumption should be expected. The Atlas, however, assumes that all decedents in all hospitals were equally sick before death, an error that tends to make low-severity hospitals look more efficient than high-severity hospitals even if the hospitals are equally efficient.

Resources that are consumed in the care of fee-for-service Medicare patients who died are also a poor surrogate for costs of care for other patients at the same hospital. If resource consumption is measured by the number of hospital days, the correlations are only 0.40 with patients in Medicare Advantage plans and 0.34 with patients covered by commercial insurance.5 If most care and resources went to dying fee-for-service Medicare patients, then maybe these weak correlations wouldn't matter. But I estimate that these patients account for less than $3 of every $100 spent on health care.

Only scientifically valid measures of efficiency should guide policy. Several organizations are developing such measures — for example, Medicare's Physician Group Practice Demonstration recently released data incorporating measures of cost and quality with respect to the management of specific conditions; the Centers for Medicare and Medicaid Services is developing “value scores” for physicians on the basis of quality performance and resource use; the National Committee for Quality Assurance publishes health-plan-level “Relative Resource Utilization” measures for various conditions; and Prometheus Payment has developed approaches to prospective payment that reward efficiency by enabling providers to share in savings for avoiding complications.

Of course, more work is required. But there are dangers in using scientifically shaky measures of efficiency to identify high-performing providers. Patients may be misled in selecting providers. Providers may base delivery changes on the wrong models. And the whole system may waste precious time and resources pursuing a path that fails to lead toward quality improvement and cost containment.

See also “Looking Back, Moving Forward,” by Jonathan Skinner, Douglas Staiger, and Elliott S. Fisher, and the following response by Dr. Bach.

Dr. Bach replies:

Until now, the way costs have been measured for research purposes has not mattered that much. Prospectively gathered costs for all patients and retrospectively gathered costs for decedents yield the same general conclusion: spending is poorly correlated with outcome.

Going forward, however, the method will matter. Assessing costs of care prospectively for all patients, not only for those who die, will provide hospital and practice CEOs with the right action item, not a perverse incentive. They will seek to streamline care for all patients rather than withhold care from the seriously ill.

“Bundling” gets this effort right. It provides prospective payment to an organization on the basis of the average costs of care for patients, not decedents.6 Accountable care organizations do this, too — bonuses are for savings per patient, not per decedent.7

Regionally adjusting payment rates to counteract differences in regional spending is less appealing. Nowhere is care regionally organized. When regions have overall spending that is too high and, as a result, rates are cut, greater finger pointing, not greater care coordination, will follow.

In any case, spending differences among regions are a lot smaller than we originally thought. The spending ranks of regions change a lot from year to year, so regions are not simply “low-spending” or “high-spending.”8 Indeed, McAllen, Texas, is not even the highest-spending region in Texas, according to the most recent analysis.9 Moreover, differences in spending among regions, though still present, are not nearly as dramatic after adjustment for regional differences in severity of illness and input prices.9

Financial and other disclosures provided by the author are available with the full text of this article at NEJM.org.

Source Information

From Memorial Sloan-Kettering Cancer Center, New York.

References

References

1. 1

   Wennberg JE, Fisher ES, Baker L, Sharp SM, Bronner KK. Evaluating the efficiency of California providers in caring for patients with chronic illnesses. Health Aff (Millwood)2005;W5-526
   Medline

2. 2

   Wennberg JE, Fisher ES, Stukel TA, Skinner JS, Sharp SM, Bronner KK. Use of hospitals, physician visits, and hospice care during last six months of life among cohorts loyal to highly respected hospitals in the United States. BMJ 2004;328:607-607
   CrossRef | Web of Science | Medline

3. 3

   Palmer S, Torgerson TJ. Economic notes: definitions of efficiency. BMJ 1999;318:1136-1136
   CrossRef | Web of Science | Medline

4. 4

   Bach PB, Schrag D, Begg CB. Resurrecting treatment histories of dead patients: a study design that should be laid to rest. JAMA 2004;292:2765-2770
   CrossRef | Web of Science | Medline

5. 5

   Baker LC, Fisher ES, Wennberg JE. Variations in hospital resource use for Medicare and privately insured populations in California. Health Aff (Millwood) 2008.27(2):w123-w134.
   

6. 6

   Miller HD. From volume to value: better ways to pay for health care. Health Aff (Millwood) 2009;28:1418-1428
   CrossRef | Web of Science | Medline

7. 7

   Fisher ES, McClellan MB, Bertko J, et al. Fostering accountable health care: moving forward in Medicare. Health Aff (Millwood) 2009;28:w219-w231
   CrossRef | Web of Science | Medline

8. 8

   Chernew ME, Sabik L, Chandra A, Newhouse JP. Ensuring the fiscal sustainability of health care reform. N Engl J Med 2010;362:1-3
   Full Text | Web of Science | Medline

9. 9

   Medicare Payment Advisory Committee. Report to Congress: measuring regional variation in service use. Washington, DC: MEDPAC, 2009.
   

Citing Articles (9)

Citing Articles

1. 1

   Tom Rosenthal. (2012) Geographic Variation in Health Care. Annual Review of Medicine 63:1, 493-509
   CrossRef

2. 2

   Todd H. Driver, Robert M. Wachter. (2012) Can healthcare go from good to great?. Journal of Hospital Medicine 7:1, 60-65
   CrossRef

3. 3

   Jack Hadley, Timothy Waidmann, Stephen Zuckerman, Robert A. Berenson. (2011) Medical Spending and the Health of the Elderly. Health Services Research 46:5, 1333-1361
   CrossRef

4. 4

   Alvin C Kwok, Marcus E Semel, Stuart R Lipsitz, Angela M Bader, Amber E Barnato, Atul A Gawande, Ashish K Jha. (2011) The intensity and variation of surgical care at the end of life: a retrospective cohort study. The Lancet 378:9800, 1408-1413
   CrossRef

5. 5

   Amy S Kelley. (2011) Treatment intensity at end of life—time to act on the evidence. The Lancet 378:9800, 1364-1365
   CrossRef

6. 6

   Richard A. Cooper. (2011) Geographic Variation in Health Care and the Affluence-Poverty Nexus. Advances in Surgery 45:1, 63-82
   CrossRef

7. 7

   Dan D. Matlock, Jean S. Kutner, Caroline B. Emsermann, Sana M. Al-Khatib, Gillian D. Sanders, L. Miriam Dickinson, John S. Rumsfeld, Arthur J. Davidson, Lori A. Crane, Frederick A. Masoudi. (2011) Regional Variations in Physicians’ Attitudes and Recommendations Surrounding Implantable Cardioverter-Defibrillators. Journal of Cardiac Failure 17:4, 318-324
   CrossRef

8. 8

   Jonathan Skinner. 2011. Causes and Consequences of Regional Variations in Health Care. , 45-93.
   CrossRef

9. 9

   Richard A. Cooper. (2010) Health Care Reform. Transactions of the ... Meeting of the American Surgical Association 128, 176-181
   CrossRef