Early Performance of Accountable Care Organizations in Medicare

Study Data and Population

We analyzed Medicare claims and enrollment data from 2008 through 2013 for a random 20% sample of fee-for-service beneficiaries. For each study year from 2009 through 2013, we included beneficiaries who were continuously enrolled in Parts A and B of fee-for-service Medicare in that year (during the time they were alive, in the case of beneficiaries who subsequently died) and in the previous year (to assess preexisting conditions).

Using the CMS ACO Provider-level Research Identifiable File, which defines each ACO as a collection of provider taxpayer identification numbers and CMS Certification Numbers (for safety-net providers),⁹ we attributed each beneficiary in each study year to the ACO or non-ACO taxpayer identification number accounting for the most allowed charges for qualifying outpatient evaluation and management services delivered to the beneficiary by a primary care physician during the year (see the Supplementary Appendix, available with the full text of this article at NEJM.org). Beneficiaries attributed to non-ACO providers constituted the control group. We excluded beneficiaries with no qualifying services.

We followed the MSSP rules for attributing beneficiaries¹⁰ but limited qualifying services to those provided by primary care physicians, because many ACOs include no or few specialty practices. This restriction improved balance in precontract spending levels between the ACO group and the control group. In a sensitivity analysis, we modified ACO definitions to account for major changes in the taxpayer identification numbers used by participating providers. (For more details on attribution of beneficiaries and modification of ACO definitions, see the Supplementary Appendix.)

Study Variables

Spending and Quality Measures

The primary outcome of our study was total annual Parts A and B Medicare spending per beneficiary, as defined by ACO program specifications.¹¹ In secondary analyses, we analyzed spending according to the type of service and care setting.

As in a previous study,⁵ we assessed several claims-based measures of quality of care included in ACO contracts¹²: hospitalizations for two ambulatory care–sensitive conditions (ACSCs; conditions for which appropriate ambulatory care could potentially reduce the need for inpatient care) — congestive heart failure and chronic obstructive pulmonary disease or asthma^13,14; all-cause 30-day readmissions among hospitalized beneficiaries; and screening mammography. We also assessed hospitalizations for ACSCs related to cardiovascular disease and diabetes¹⁴ and three preventive services for diabetes, because many contract measures that cannot be assessed with claims focus on these conditions. We specified hospitalizations and readmissions as annual counts and assessed the receipt of preventive services on an annual basis. Finally, we used previously developed methods to assess the use of 31 low-value services (Table S1 in the Supplementary Appendix), which were defined as services that provide minimal average benefit in clinical settings that can be identified from claims with reasonable specificity.^15,16 We analyzed the annual count of low-value services received by each beneficiary.

ACO Characteristics

We prespecified several subgroups of ACOs for comparison. First, we categorized ACOs as financially integrated with hospitals (vertically integrated organizations), independent multispecialty physician groups, or independent primary care physician groups, using CMS descriptions, information on the websites of participating organizations, and physician specialty information from claims. Second, we categorized each ACO according to whether risk-adjusted Medicare spending at baseline for the control group in its service area was above or below the median among ACO service areas. Third, we categorized each ACO according to whether risk-adjusted baseline spending for its attributed beneficiaries was above or below risk-adjusted spending for the control group in its service area. To mitigate bias from regression to the mean, we used claims from 2008 (before the study period) to assess baseline spending for these measures. (For more details on the categorization of ACOs, see the Supplementary Appendix.)

Fourth, we used the Decision Resources Group proprietary ACO database¹⁷ to identify MSSP ACOs with at least one ACO risk contract with a commercial insurer starting in 2012 or 2013. We considered payment arrangements involving global budgets with one- or two-sided risk sharing, global payments, or capitation to be ACO contracts. When comparing ACOs that entered commercial ACO contracts in 2012 or 2013 with other ACOs, we excluded 12 ACOs with commercial contracts that started before 2012 because these may have had spillover effects that caused lower Medicare spending before the start of Medicare ACO contracts.¹⁸

Covariates

From Medicare Master Beneficiary Summary Files, we assessed the age, sex, race or ethnic group,^19,20 and Medicaid coverage of the beneficiaries, as well as whether disability was the original reason for Medicare eligibility and whether beneficiaries had end-stage renal disease. From the Chronic Conditions Data Warehouse (CCW), which draws from diagnoses since 1999 to describe beneficiaries’ accumulated disease burden,²¹ we assessed whether beneficiaries had any of 27 conditions in the CCW by the start of each study year. From diagnoses in the preceding year of claims, we also calculated a Hierarchical Condition Category risk score for each beneficiary in each study year.²² We determined whether beneficiaries were long-term nursing home residents using a validated claims-based algorithm.²³ Finally, from U.S. Census data,²⁴ we assessed area-level sociodemographic characteristics.

Statistical Analysis

Table 1. Table 1. Characteristics of Beneficiaries in the 2012 Cohort of Medicare Shared Savings Program (MSSP) Accountable Care Organizations (ACOs) and the Control Group, before and after the Start of ACO Contracts in 2012.

We used linear regression and a difference-in-differences approach to estimate changes in spending or quality in the ACO group from the precontract to postcontract period that differed from concurrent changes in local control groups (see the Supplementary Appendix). Our models included the characteristics listed in Table 1 (except with age specified categorically and CCW conditions as indicators and counts) and fixed effects for each ACO to adjust for precontract differences between ACOs and the control group and for any changes in the distribution of ACO-attributed beneficiaries across ACOs. Models also included fixed effects for each hospital referral region (HRR) in each year to compare each beneficiary attributed to an ACO with beneficiaries in the control group living in the same area and to adjust for HRR-specific changes in spending or quality occurring in the control group. Thus, unlike comparisons with financial benchmarks set by CMS, our analysis estimated savings or losses by comparing spending in the postcontract period for ACO-attributed patients with spending that would be expected in the absence of ACO contracts, with the use of local changes in a similar population to establish that counterfactual scenario. We compared our estimates of savings with CMS estimates, using publicly available data on ACO spending and benchmarks to describe the latter (see the Supplementary Appendix).³

We used robust variance estimators to account for clustering within ACOs (for the ACO group) or HRRs (for the control group).²⁵ In sensitivity analyses, we analyzed count variables and spending using generalized linear models with a log link and proportional-to-mean variance function, and we analyzed binary quality measures using logistic regression.

As prespecified in our study protocol, we treated 2013 as the postcontract period for both the 2012 and 2013 cohorts of MSSP ACOs because the 2012 cohort did not enter the MSSP until April 2012 (24% of ACOs) or July 2012 (76%). Our models treated 2009 through 2011 as the precontract period for 2012 MSSP entrants (excluding the 2012 transition year) and 2009 through 2012 as the precontract period for 2013 MSSP entrants. Per the study protocol, we estimated differential changes in total spending and quality separately for each cohort of MSSP ACOs and analyzed savings according to type of service (for descriptive purposes) only if total savings in a cohort were significant (two-sided P<0.05). We analyzed the 2012 and 2013 cohorts separately because early participants may have differed from later participants in their responsiveness to new incentives and because CMS-reported savings differ between the cohorts.^2,3 To compare savings between ACO subgroups of interest, we added interaction terms to models to determine whether savings differed according to ACO characteristics. To provide context for interpreting MSSP results, we also updated analyses of the 32 ACOs that entered the Pioneer model in 2012. (For more details on multiple testing in subgroup comparisons and on analyses of the Pioneer ACO model, see the Supplementary Appendix.)⁵

We conducted several analyses to gauge potential sources of bias. First, we compared trends in total spending between the ACO group and the control group during the precontract period. A key assumption of our difference-in-differences comparisons is that changes from the precontract to postcontract period would have been similar in the ACO group and the control group in the absence of the ACO programs. Similar trends in the precontract period would support this assumption. Precontract trends were similar for all comparisons except for one subgroup comparison, which we note and consider to be uninterpretable.

Second, we tested whether patients’ sociodemographic and clinical characteristics differentially changed from the precontract to postcontract period in the ACO group relative to the control group. Third, we used propensity-score methods to balance observed patient characteristics between the ACO group and the control group within each HRR and study year (see the Supplementary Appendix).²⁶ Fourth, we conducted falsification tests treating 2010 or 2011 as a postcontract year.

Patient Characteristics

After adjustment for HRR, differences in beneficiaries’ clinical and sociodemographic characteristics between the ACO group and the control group were mostly small, and differential changes from the precontract period to the postcontract period were consistently minimal (Table 1 for the 2012 MSSP cohort, and Table S2 in the Supplementary Appendix for the 2013 cohort).

Changes in Spending and Quality

In the precontract period, adjusted total annual Medicare spending per beneficiary was similar in the ACO group and the control group ($133 per beneficiary higher in the 2012 ACO cohort [2012 MSSP entrants] than in the control group, P=0.22; $2 per beneficiary higher in the 2013 ACO cohort [2013 MSSP entrants] than in the control group; P=0.97). Spending trends in the precontract period were also similar ($19 per beneficiary slower per year in the 2012 ACO cohort than in the control group, P=0.50; $11 per beneficiary slower per year in the 2013 ACO cohort than in the control group, P=0.54).

Table 2. Table 2. Differential Changes in Spending and Quality for the 2012 Cohort of MSSP ACOs versus the Control Group.

The adjusted differential change (i.e., the adjusted between-group difference in the change from the precontract period) in mean total annual Medicare spending in 2013 was −$144 per beneficiary in the 2012 cohort of MSSP ACOs as compared with the control group (P=0.02), representing a 1.4% estimated savings relative to an expected mean of $9,942 in 2013 for the ACO group. In contrast, the differential change in total spending in 2013 was only −$3 per beneficiary among ACOs entering the MSSP in 2013 as compared with the control group (P=0.96). Contributing to the estimated savings in the 2012 cohort were a 1.4% reduction in spending on inpatient care, a 2.1% reduction in spending on hospital outpatient care that was mostly offset by a 1.5% increase in spending on office-based outpatient care, a 6.1% reduction in spending on postacute care at skilled nursing facilities, and a 2.7% reduction in spending on home health care (Table 2).

Performance on quality measures differentially improved in the ACO cohorts as compared with the control group for a few measures and did not differentially change for other measures, and there was no differential change in the use of low-value services in either ACO cohort (Table 2, and Table S3 in the Supplementary Appendix). Use of generalized linear models did not alter our conclusions, falsification tests revealed no significant differential changes in spending in 2010 or 2011, and results were robust with respect to other sensitivity analyses (Table S4 in the Supplementary Appendix).

ACO Subgroup Comparisons

Organizational Type

Figure 1. Figure 1. Differential Changes in Total Medicare Spending in 2013 According to Accountable Care Organization (ACO) Subgroup.

Differential changes (i.e., between-group differences in the change from the pre-ACO contract period) in total Medicare spending per beneficiary in 2013 are displayed according to ACO subgroup, with negative estimates indicating estimated savings. Subgroup estimates are provided separately for the 2012 (blue) and 2013 (red) cohorts of Medicare Shared Savings Program (MSSP) ACOs. Estimated savings in the 2012 cohort were greater for ACOs in high-spending areas than for those in low-spending areas, but the subgroup difference was not significant (P=0.09) and not interpretable because precontract spending growth in high-spending areas was already significantly slower in the ACO group than in the control group. All the 2008 ACO spending values were either above or below the local average. Of 220 MSSP ACOs, 33 (15%) entered a total of 45 commercial ACO contracts in 2012 or 2013 without having entered a commercial ACO contract before 2012.

Estimated savings among independent primary care groups were consistently seen in the 2012 and 2013 cohorts of ACOs (Figure 1) and were significantly greater than savings among vertically integrated organizations (difference between ACO subgroups pooled across cohorts, −$371 per beneficiary; P=0.005). Post hoc analyses suggested that this difference in total savings was driven largely by greater differential reductions in spending on inpatient care and hospital outpatient care among independent primary care groups. In contrast, CMS estimates of savings did not differ substantially according to ACO type, apparently understating subgroup differences as a result of the actuarial methods used. (For more details, see the Supplementary Appendix, including Tables S5 and S6.)

Baseline Spending

Estimated savings were significantly greater for ACOs in the 2012 MSSP cohort that had baseline spending above local averages than for those that had baseline spending below local averages (difference, −$285 per beneficiary; P=0.02). Savings in the 2012 cohort followed a similar pattern for ACOs in high-spending versus low-spending areas (Figure 1), but the difference was not significant (P=0.09) and was uninterpretable owing to differences in precontract trends. In the 2013 cohort, savings were not evident in any subgroup defined by baseline spending.

Commercial ACO Contracting

In the 2012 MSSP cohort, estimated savings in 2013 did not differ significantly between ACOs concurrently entering commercial ACO contracts and ACOs without such contracts (Figure 1). In the 2013 cohort, differential changes in spending for the 11 ACOs that entered commercial ACO contracts in 2012 or 2013 indicated significant savings in 2013, differing significantly from differential changes in spending among other ACOs in the cohort (subgroup difference, −$611; P=0.001).

The first full year of participation in the MSSP was associated with early savings among ACOs that entered the program in 2012 but not among those that entered in 2013. Savings in the 2012 cohort were on a par with savings estimated for Pioneer ACOs (see the Supplementary Appendix), suggesting that one-sided contracts without downside risk (shared savings only) also may elicit effective efforts to reduce health care utilization. Owing to the one-sided nature of almost all MSSP contracts, however, the aggregate $238 million spending reduction suggested by our estimates for the 2012 MSSP cohort did not result in net savings to Medicare, because Medicare paid $244 million in bonuses without recouping losses from ACOs that had spending above benchmarks.³

Moreover, our results suggest that early gains observed among early MSSP participants may not generalize to later participants. We examined only 12 to 18 months of exposure to MSSP incentives for most ACOs in our study, and savings may grow over time.²⁷ Nevertheless, if meaningful savings from the expanding MSSP materialize, our findings suggest that they may be slow to develop, because participants with less advanced systems for managing care may enter later and require more time to improve care efficiency. Regulatory changes that strengthen incentives for ACOs to lower spending — incentives that have been very weak thus far^28-30 — may accelerate savings and will be important to implement and evaluate.³¹

Although our estimate of aggregate savings in the MSSP is similar to the $234 million in savings determined by CMS from comparisons with benchmarks, we found a different pattern of savings according to ACO structure (see the Supplementary Appendix). Whereas CMS data suggest that savings were generally similar for different types of ACOs, we estimated substantially greater savings for independent primary care groups than for groups integrated with hospitals when comparing spending changes in ACOs with local concurrent changes. There are both theoretical considerations and previous observational studies that would support the pattern of savings that we observed.^32-35 In particular, independent physician groups have stronger incentives to lower inpatient and hospital outpatient spending than groups integrated with hospitals because their shared-savings bonuses are not offset by forgone profits from reductions in hospital care.³⁶ Our findings suggest that financial integration between physicians and hospitals, which may increase commercial health care prices,^37,38 is not necessary for ACO success. Early signals from the MSSP, however, may not predict the long-term efficiencies from various organizational structures under new payment models.³⁹

Savings in the 2012 MSSP cohort were greater for ACOs that had baseline spending above local averages than for those with below-average spending. This finding is consistent with those of previous studies of Pioneer ACOs^5,16 and suggests that providers with more opportunities to cut spending can do so more easily. This finding also confirms the importance of encouraging participation by high-spending organizations as CMS considers transitioning benchmarks for ACOs to average fee-for-service spending in their region.^30,31,40 Also consistent with findings from the Pioneer model⁵ is that the decomposition of savings according to type of service in the 2012 cohort suggested a substitution of lower-priced outpatient care in independent office settings for higher-priced hospital outpatient care and a prominent reduction in spending for postacute care, which is known to vary widely across markets and is thought to be a major source of wasteful care.⁴¹ Unlike the Pioneer model, however, the MSSP was not associated with significant reductions in the use of low-value services.

Our study had several limitations. First, because the ACO programs are voluntary, participating providers may differ from other providers in ways related to subsequent spending changes. The similar precontract spending levels and trends between ACOs and non-ACO providers, however, show that ACOs were not already constraining spending. Organizations already planning to limit spending may have selectively participated, but implementing such plans under fee-for-service payment would have run counter to their financial self-interest. ACOs probably were more capable of managing care effectively in response to new incentives than nonparticipating providers, but latent differences in the capacity for change would not invalidate inferences that changes in practice were due to changes in incentives. Second, our estimates of savings did not consider the costs to ACOs of efforts to lower spending or improve quality. Third, because statistical tests of subgroup differences were not adjusted for multiple comparisons in our main analysis (see the Supplementary Appendix), they overstate somewhat the confidence with which we could reject the null hypothesis of no difference. Nevertheless, several findings were observed consistently across the Pioneer model and MSSP or across cohorts of MSSP ACOs.

In conclusion, our findings extend evidence of small but meaningful reductions in spending, with unchanged or improved quality of care, early in the Medicare ACO programs and suggest that progress toward net savings to Medicare or society may be slow. Continued quasi-experimental evaluations will be important for identifying successful subgroups of ACOs that have developed strategies that could be disseminated and for assessing progress from regulatory efforts to strengthen incentives and encourage participation.