Seeing patterns where none exists

Researchers at the Institute for Clinical Evaluative Sciences in Toronto did a fun study using data from 10 million Ontario residents looking for associations between various health problems and....their astrological signs. And they actually found associations! In fact, each of the twelve astrological signs had at least two medical disorders associated with them.

Speaking at the American Association for the Advancement of Science conference this weekend, lead researcher Peter Austin, Ph.D., senior scientist with the Institute for Clinical Sciences, a nonprofit organization in Ontario which evaluates the evidence for health care services, explained their study.

Their point was not to validate star signs but to illustrate that when researchers dredge through a database looking for patterns, they are sure to find them. That doesn’t mean the associations mean anything, however. These researchers went on to test their initial results in a second group of people and were not able to reproduce them, adding a second caution for us about not believing the conclusions of any single study.

The Statesman of India reports:

Of medical disorder and astrological destiny

...“Replace astrological signs with another characteristic such as gender or age [or obesity, a specific diet, or exposure], and immediately your mind starts to form explanations for the observed associations,” Austin said. “Then we leap to conclusions, constructing reasons for why we saw the results we did....”

“Scientists take pains to make sure their clinical studies are conducted accurately,” Austin said. “But sometimes erroneous conclusions will be obtained solely due to chance.” Statistical chance means that five per cent of the time, scientists will incorrectly conclude that an association exists, when in reality no such association exists in the population that the scientists are studying....

I call data dredge studies the “Rorschach tests” of epidemiology, because researchers can pull out characteristics about people in almost unlimited combinations to find all sorts of correlations and conclude just about anything they set out to find. Just like the Rorschach test, seeing patterns where none exists, finding connections that are there but not as strongly as believed, and seeing what one expects to see, are common. The biggest and longest-running Rorschach test is the Nurses Health Study — a huge quarry of questionnaires gathered since 1976 from over 120,000 nurses, headed by JoAnne Manson, M.D., DrPH of Brigham and Women’s Hospital in Boston. Over 500 “studies” have been released by Harvard School of Public Health and Brigham and Women’s Hospital, all using the Nurses Health Study data. The correlations they’ve found frequently contradict each other.

No matter how strong an association between two things might be and no matter how many times it has been reproduced, that will still never show one to be the cause for the other, as was explained in “Loopy Loops.” Yet, when an association seems to validate what we believe to be true, we are naturally quick to jump and put the links together to show cause, when in actuality, we oftentimes have them backwards or they have nothing to do with the cause at all, just like the stars.