Correspondence

Computer-Based Diagnostic Systems

N Engl J Med 1994; 331:1023-1024October 13, 1994

Article

To the Editor:

In their report on the performance of four commercially available computer-based diagnostic systems in internal medicine (June 23 issue),1 Berner and colleagues conclude that “the programs should be used by physicians who can identify and use the relevant information and ignore the irrelevant information that can be produced.”

The decision-making process of such systems should always be challengeable, yet surprisingly, no mention was made of whether the programs were in any way able to explain their reasoning. Current computational techniques permit the decisions of computers to be queried and the reasoning process formally verified. The use of such techniques should allow more rigorous validation of the programs' workings and should assist physicians in interpreting their suggestions. Surely this might be one way of helping clinicians identify relevant advice and ignore irrelevant suggestions?

Eldon D. Lehmann, M.B., B.S.
Hammersmith Hospital, London W12 0NN, United Kingdom

1 References

1. 1

   Berner ES, Webster GD, Shugerman AA, et al. Performance of four computer-based diagnostic systems. N Engl J Med 1994;330:1792-1796
   Full Text | Web of Science | Medline

To the Editor:

The paper by Berner et al. and your editorial1 raise interesting questions about the diagnostic applications of expert computer systems in medicine. The term “expert” is, of course, relative. To a layperson, the physician in general practice is an expert, and to the physician in general practice, the specialist is an expert. In the evaluation, the four diagnostic programs were considered to be “super-experts” that could diagnose problems involving “atypical presentations, rare diseases, multiple disorders presenting simultaneously, or elements sufficiently complex that the physician would be likely to request a diagnostic consultation.” In other words, they were asked to function at a level considerably above the level of a family physician. When evaluated at the super-expert level, they were only moderately successful.1

Regardless of their design assumptions, what if the programs were viewed not as super-experts but as experts with a family physician's level of expertise? They would probably do quite well in diagnosing most of the conditions that a family physician encounters. If an expert system is as good at diagnosis as a family physician (and as good at referring patients for whom no diagnosis can be made), the redundancy is interesting to consider.

Who would use such a program? Certainly not the human super-expert, and probably not the family physician. Would the program be used by nonphysicians who are expert data gatherers but not expert diagnosticians? Could a substantial percentage of typical patients be managed by a human data gatherer and a computer program?

Robert L. Yolton, Ph.D.
Pacific University College of Optometry, Forest Grove, OR 97116

1 References

1. 1

   Kassirer JP. A report card on computer-assisted diagnosis -- the grade: C. N Engl J Med 1994;330:1824-1825
   Full Text | Web of Science | Medline

Author/Editor Response

The authors reply:

To the Editor: Drs. Lehmann and Yolton raise some provocative questions about the decision-support systems we studied. Their questions address one of the key issues that we highlighted -- namely, that it is essential that the physician using a system evaluate the diagnostic suggestions it produces.

As Dr. Lehmann suggests, a program's explanation of its reasoning might help in the interpretation of the suggested diagnoses. Most of the programs permit queries, and they “explain” their diagnoses, in that they describe, with various means of quantification, how the individual case findings relate to a given diagnosis in the program's knowledge base, the level of confidence that can be placed in a particular diagnosis, and what additional data would support the diagnosis. In addition to an explanation of the program's reasoning, information on the sources and methods used to build its knowledge base might also assist users in interpreting the material it produces. The design of our study did not permit us to evaluate the explanatory features or construction of the knowledge base, but Dr. Lehmann's point is well taken; systematic evaluation of additional features of the programs may be warranted. However, even if the reasoning behind a program is readily accessible, the user of the system must still judge the appropriateness of the explanation, as well as the suggested diagnoses.

We agree with Dr. Yolton that these systems would be unlikely to be used by subspecialists for most patients within their disciplines. However, they might be used by any physician when confronted with a patient with a puzzling illness. We do not know what kind of nonphysician data gatherer Dr. Yolton had in mind as a potential user, since a nonphysician could be a layperson, a medical student, or another health professional. The data gatherer must have enough medical knowledge to select the appropriate information on the patient to enter into the systems. Furthermore, as these systems currently function, medical knowledge is needed to understand their vocabulary and to sift through the diagnostic suggestions. What is not known is how much medical knowledge is necessary. The key question to be addressed is not whether these systems can replace the physician, either generalist or specialist, but whether their use leads to improved diagnostic decision making and ultimately to better care of patients.

Eta S. Berner, Ed.D.
University of Alabama at Birmingham School of Medicine, Birmingham, AL 35294-2041

George D. Webster, M.D.
InforMed, Bryn Mawr, PA 19010

Alwyn A. Shugerman, M.D.
University of Alabama at Birmingham School of Medicine, Birmingham, AL 35294-2041

Citing Articles (3)

Citing Articles

1. 1

   Eldon D. Lehmann. (2004) Computerised Decision-Support Tools in Diabetes Care: Hurdles to Implementation. Diabetes Technology & Therapeutics 6:3, 422-429
   CrossRef

2. 2

   Eldon D. Lehmann. 2004. Computer-assisted Diabetes Education and Information Technology in Diabetes Care. .
   CrossRef

3. 3

   Eldon D. Lehmann. (2003) Research Use of the AIDA www.2aida.org Diabetes Software Simulation Program: A Review—Part 2. Generating Simulated Blood Glucose Data for Prototype Validation. Diabetes Technology & Therapeutics 5:4, 641-651
   CrossRef