Correspondence

A Computer-Assisted Management Program for Antiinfective Agents

N Engl J Med 1998; 338:1775-1776June 11, 1998

Article

To the Editor:

Evans et al. (Jan. 22 issue)1 have developed an innovative computer-assisted management program for antiinfective agents, but they have compared three groups that are not comparable: a group in which the recommendations of the computer-assisted management program were followed, a group in which recommendations were overridden, and all patients seen before the program was implemented. This is scientifically unsound. Selection bias could well account for any differences between groups. Equally likely explanations are differences between physicians who adhered to the program and those who did not or differences in the severity of disease. The failure to adjust for the prescribing physician and to provide an analysis of reasons for physician nonadherence (available to the investigators) seriously weakens the study. In addition, the authors have only partially adjusted for the severity of disease by using the Computer Severity Index at the time of admission to the intensive care unit and for mortality. It is revealing, and a better way to minimize selection bias, to compare the preintervention and entire postintervention groups. This comparison shows that there were no changes in mortality rates, lengths of stay, or total costs, negating the main conclusions of the report. Some measures one might expect to decrease with the implementation of a computer-assisted management program, such as the number of doses, the number of days of excessive antiinfective doses, and drug costs, did decrease, but can this be attributed to the program?

Sumit R. Majumdar, M.D.
Dennis Ross-Degnan, Sc.D.
Harvard Medical School, Boston, MA 02115

Stephen B. Soumerai, Sc.D.
Harvard Pilgrim Health Care, Boston, MA 02215

1 References

1
Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338:232-238
Full Text | Web of Science | Medline

To the Editor:

Evans et al. contend that the use of a computerized antiinfectives-management program improved the quality of care in their intensive care unit. Although the data they collected may support such a claim, the data presented could also lead one to the opposite conclusion. If the authors' argument is correct, the patients for whom the computer's advice was ignored should have fared the same as the patients in the preintervention control group. However, they fared much worse. For example, although the mortality rate decreased in the group that followed the computer regimen, it increased in the group in which the computer regimen was overridden.

Instead of concluding that the computer aided or impaired physicians' judgment, it is more parsimonious to conclude that it had no effect other than to select patients with different outcomes. Again, if the mortality data are used as an example, the mortality rate in the combined intervention group was the same as in the control group (22 percent). . . .

I urge the authors to reanalyze their data to explain why some patients did worse after the intervention began. Those of us who favor computer systems need this information to prove that the systems do not cause harm.

Curtis L. Cole, M.D.
Cornell University Medical College, New York, NY 10021

To the Editor:

. . . As Evans et al. suggest, a possible source of selection bias in their study could have been the severity of illness, which may have been greater in the subgroup in which the computer regimen was overridden. The authors discount this possibility, but data on the severity of illness that could empirically address this hypothesis (the Computer Severity Index score) were collected and should be presented explicitly, rather than used only as a hidden adjustment factor in the analyses of Table 4 of the article. No amount of post hoc adjustment can render the self-selected subgroups of the intervention period a suitable basis for valid controlled assessment of the effect of the intervention on outcomes.

James R. Johnson, M.D.
Minneapolis Veterans Affairs Medical Center, Minneapolis, MN 55417

To the Editor:

The conclusions of Evans et al. rely on the premise that any noncompliance with computer-assisted suggestions, however trivial, constitutes a substantial compromise in quality. This all-or-none criterion of compliance introduces considerable bias regarding the length of stay in the intensive care unit. Patients who require lengthy treatment in the intensive care unit are overrepresented in the noncompliant group, since they have more daily opportunities to override the computer's advice.

If the reductions in the length of stay and total costs were as substantial as portrayed, one would expect overall improvement in the intervention group, with the greatest effect in patients whose treatment most closely followed the computer-assisted regimen, but with little change in others. However, Table 3 shows that the group in which the regimen was overridden had decreases in all but 2 of 11 measures as compared with values in the preintervention period. These data suggest that the regimen provided only limited benefits and warrant reexamination. . . .

Robert A. Duncan, M.D., M.P.H.
Lahey Hitchcock Clinic, Burlington, MA 01805

Author/Editor Response

The authors reply:

To the Editor: All the letter writers believe that selection bias could account for the differences among the three groups. In addition, Dr. Majumdar and colleagues question whether a scientifically sound analysis, which necessarily requires a linear, reductionist model, can ever be conducted in a natural and complex environment. In our report, the threshold for detecting differences in clinical outcomes was unusually high because we depended on real, unsolicited clinical data instead of restricted, protocol-directed data.1 We did present a comparison of the overall preintervention and intervention groups, adjusted for age, sex, severity of illness, medical service, and mortality by linear regression. Because patient care involved a large number of rotating house officers and decisions were usually made as a team, adjustment for physicians was simply not feasible. However, the same group of intensivists supervised care for all three study years. The reasons for nonadherence were diverse and most often related to data that were not available to the computer program (such as findings on physical examination).

Dr. Duncan suggests that longer stays could increase opportunities for overrides and account for the poorer outcomes in the group in which the computer regimen was overridden. However, this seems an implausible explanation since most of the overrides, 153 of 195 (78 percent), occurred within 72 hours after admission to the intensive care unit and 86 (44 percent) occurred on the day of admission to the intensive care unit (when the severity-of-illness score was calculated). However, an unadjusted comparison of the preintervention and intervention periods is also scientifically unsound. No method of adjustment for severity of illness is perfect, and even a randomized, controlled trial cannot answer questions about optimal treatment for individual patients. At best, it can yield an average value for efficacy in groups of patients.2 Certainly, there were patients who benefited from overrides as well as others who benefited from adherence to the computer recommendations, and the opposite circumstances also occurred. Our pragmatic study analyzed clinical and cost outcomes, but it was not a fastidious cost–benefit analysis. It would be a misuse of our data to imply that cost savings alone must be required to justify expanded use of computerized decision-support tools.

John P. Burke, M.D.
David C. Classen, M.D.
R. Scott Evans, Ph.D.
LDS Hospital, Salt Lake City, UT 84143

2 References