April 26, 2007

How Doctors Think

Posted by Ben Simo

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.

I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.

Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.

About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.
Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.



April 28, 2007  
Ben Simo wrote:

I just got the book today. I've only read the introduction so far. The introduction explains why Dr. Groopman did the research that led to the book.

He describes watching medical students and new doctors follow written process and decision trees instead of looking closely at patients' problems and really thinking about those problems. Instead of applying critical thinking, he says that the new generation of doctors are trained to process pre-written diagnosis scripts like computers.

Sounds a lot like factory school software testing to me. I am looking forward to the rest of the book.

Ben Simo

April 29, 2007  
Erkan Yilmaz wrote:

Hello Ben,

I will put this book on my buy list. I have listened to the interview only and what I liked from it was, that Dr. Jerome Groopman correctly said: "We are fallible" (I mean he talks at the beginning of having caused a death: position 3:40) - there should be no arrogance. Arrogance will probably happen at least once to everybody of us (no matter if one is tester or doctor). Let`s hope this does not cost too much then.

About the quote of Linda Lewis (position 21:00):
"You know, medicine is not that complicated: a doctor should be able to explain in clear language to every patient what is wrong and it should make sense. It's not quantum physics."

If you set this in relation to testing - actually being able to communicate is also very important for testers. Because, after finding bugs what next? Are they communicated good - if not, they would then be not known or fixed fast enough.
Hopefully we testers do this in the best way. Here are links, how to write good bug reports and to convince the reader of the bug report to fix them:
Cem Kaner, Effective bug reporting (slides, video)

Until I get my copy, waiting for some stories from you.

Good night,

April 29, 2007  
Ben Simo wrote:

Hi Erkan,

Dr. Jerome Groopman correctly said: "We are fallible" ... there should be no arrogance.

You have hit on an important point. If we testers aren't careful, we can easily come across as arrogant to others on our project teams. We software testers need to team with the rest of the project, not lord over them as quality cops. We are all in it together. We need to let developers see our mistakes too. Otherwise, we come across as arrogant perfectionists. (I will admit to being a perfectionist at heart.) I've made as many mistakes as a tester as the worst requirements and code writers I've encountered.

actually being able to communicate is also very important for testers. Because, after finding bugs what next?

Communication is very important. Not only do we need to report bugs, but we need to communicate their potential impact. There was a time that I just pointed to the requirements and demanded that developers fix the bugs. Then I encountered a product manager that would say something like "Revenue is king. Liability is queen. Tell me how they are impacted by this bug." He taught me that it is necessary to communication the impact of bugs in terms that those impacted understand.


April 30, 2007  
Anonymous wrote:

Hi Ben,

I am glad that there is another perfectionist at heart. :-)

"There was a time that I just pointed to the requirements and demanded that developers fix the bugs."
Yes, I made the same mistakes. Unfortunately - when I started testing (officially) I had then nobody telling me this. But with time I learned my lessons.
Well, the good thing with mistakes it: they are invaluable and thank God we also make them, so we get the chance to learn from it - hopefully.