Are You Smarter Than a Doctor?

Percentages confuse even the experts. The accuracy of a typical medical test, for instance, is usually expressed as a percentage: “The test is 90 percent reliable.” But it has been found that doctors, no less than patients, are often hopelessly confused when it comes to interpreting what this means in human terms.

Gerd Gigerenzer, a German psychologist, asked two-dozen physicians to tell him the chance of a patient truly having breast cancer when a mammogram that was 90 percent accurate at spotting those who had it, and 93 percent accurate at spotting those who did not, came back positive. Gigerenzer then added one other important piece of information: that the condition affected about 0.8 percent of the population for the group of 40- to 50-year-old women being tested.

Of the 24 doctors, just two worked out correctly the chance of the patient really having the condition. Most were hopelessly wrong. Quite a few of the physicians assumed that, since the test was 90 percent accurate, a positive result meant a 90 percent chance of having the condition.

Ok, here is your chance to see if you smarter than your average doctor. Can you determine what the chance of having breast cancer if you get a positive result from the test?

In fact, more than nine out of 10 positive tests under these assumptions are false positives, and those nine patients are in the clear.

To see why, look at the question again, this time expressed in terms that make more human sense: natural frequencies.

Imagine 1,000 women. Typically, eight have cancer, for whom the test, a fairly but not perfectly accurate test, comes back positive in seven cases. The remaining 992 do not have cancer, but remember that the test can be inaccurate for them, too. Nearly 70 of them will also have a positive result. These are the false positives, people with positive results that are wrong.

Now we can see that there will be about 77 positive results in total (the true positives and the false positives combined), but that only about seven of them will be accurate. This means that for any one woman with a positive test, the chance that it is accurate is low (one in about 11) and not, as most physicians thought, high.

A couple of points on this.

First, the rate of cancer in the population is crucial to determining the chance that you actually have cancer from a positive result. Which raises an interesting question. If you were to get the test and then move to a population that had a 20% chance of having cancer (or if you were subdefined into a category that had a higher risk of cancer, such as those with a history of cancer in the family), would the accuracy of the result go up? While the calculation would say yes, your chance of having cancer from a positive result would be 74%, it doesn't make much sense given that nothing has changed with you personally or the result of your test. I don't know my statistics well enough to be able to explain this.

Second, given the low incidence of cancer, the rate of false positives is much more important than that of false negatives. If this test was 100% percent accurate at spotting those who had it (and remains 93% accurate at spotting those who did not), a positive result would still mean that only around 10% of individuals had cancer. On the other hand, if the test was 99% accurate at spotting those who did not have it (and remained 90% accurate at spotting those who did have it) then a positive result would mean a 42% chance of the individual having cancer.

Third, this reinforces my belief that probability is much more useful than calculus and that high schools should change their curriculum accordingly.

Fourth, along with the previous point, it makes me question whether medical school gives doctors the skills they need to practice medicine effectively today. As the Obama administration looks at trying to lower health costs, one thing that I think should also be put into the equation is medical education. Part of the reason for the expense of medicine is that getting a medical degree is expensive and takes a long time. Does all of this education really make doctors better? Many other nations do not require medical students to first get an undergraduate degree. If we could cut down on the years it takes to get a medical degree, the price of seeing a doctor would go down.

And finally, while the concept of preventative medicine is good, this example shows how with tests that aren't 100% accurate it could be bad in practice. Imagine that everyone who tested positive for cancer was then put on a anti-cancer regiment. Only 10% of them would actually gain any benefit from it, and yet all would go through the rigor of it. And then 90% of them would be "cured" regardless of what the treatment was! The financial expense would be great and the health of the 90% without cancer would be impacted negatively.

via The Week