Reputable science isn’t apparent by looks, popularity or age

One of the most common fallacies about scientific research is that the validity of a study can be determined by the date it was done. “Outdated” studies are not to be trusted, we’re often led to believe. It’s sort of the junk scientists’ version of numerology.

Sadly, when we become convinced to dismiss studies out-of-hand because of some measure other than the strength of the science itself, we make ourselves vulnerable to being taken advantage of by unsound ideas and to miss out on the best available science...

As was discussed earlier here and here, the soundness of a study is not determined by who conducted, published or funded it... anymore than the date is was published.

The date on a paper can never change its basic science or findings. If it was a carefully-done study with strong findings that haven’t been nullified by better studies, it doesn’t matter if it was done last year or 50 years ago. With the increasing shenanigans appearing in research today, most notably the misuse of epidemiological studies and their interpretations, oftentimes older studies are more credible.

For example, the most definitive and comprehensive clinical trial ever created to examine the effects of dieting was conducted by Ancel Keys, Ph.D., in 1950. The harmful emotional and physical effects were so extraordinary and catastrophic that even he admitted decades later that nothing like it will ever be done again because laws have since been put in place against cruelty and endangerment of human research subjects. Instead, they’ve been confirmed in countless large observational and smaller clinical studies ever since. To dismiss its findings because of the date in the legendary two-volume, Biology of Human Starvation where the results were published, would be to lose invaluable insights.

To better understand how we can be hurt by believing a publication date or some other arbitrary characteristic about a study matters, let’s go back to what we discussed here last week. The value of the scientific process comes in how it tests plausible hypotheses and helps us discover and learn what things we can actually place our trust in. As experiments are able to support a preliminary hypothesis — by testing an idea from various angles and using various methodologies; and testing a hypothesis in increasingly rigorous, detailed and sophisticated ways — a body of evidence is built to support it. Science is continually progressing and advancing, and ideas are fine-tuned, in light of new and better information.

While an idea can never be entirely proven, in time the body of evidence supporting it will be so strong that it will become an accepted hypothesis or a scientific theory: a rational explanation or model based on observation, careful experimentation and reasoning, especially one that has been tested and confirmed as a principle helping to explain and predict natural phenomena — or a scientific fact: things which can be observed and/or measured; something that is supported by unmistakable evidence.

In science, 'fact' can only mean confirmed to such a degree that it would be perverse to withhold provisional assent. — Stephen Jay Gould (1941-2002), former president of the American Association for the Advancement of Science

But during the process of science, careful or new studies can show an idea to be without merit and disprove a theory. When that happens, sound scientists move on and do not continue to test and fund studies to repeat the same things ad nauseam. An “old” study disproving something doesn’t need to be continually repeated. Progress is not made by continuing to dredge through ideas that were disproven ages ago or implausible to begin with.

Not so with junk scientists. They will assert their unsound ideas and continue to reassert their ideas and insist more studies are endlessly needed, hoping that one will come up with something different.

When they continue to rely on poorly conducted studies and ideas that have been disproven, that is when our caution is warranted. A flawed study — whether old or new — is still a bad study. They are often counting on the fact, however, that the media and many of us haven’t done our homework to realize they are citing uncredible sources.

We have to be careful here, too, not to believe that we can tell who the junk scientists are by some arbitrary characteristic. Bad “science” can appear anywhere and be popularly believed by even mainstream, seemingly credible sources. Prestige and credentials, and the volume of funding and political momentum behind it, are not necessarily reliable clues to the soundness of the science.

And most importantly, the popularity of an idea is never a measure of its soundness.

Do not believe in anything simply because you have heard it.

Do not believe in anything simply because it is spoken and rumored by many.

Do not believe in anything simply because it is found written in your religious books.

Do not believe in anything merely on the authority of your teachers and elders.

Do not believe in traditions because they have been handed down for many generations.

But after observation and analysis, when you find that anything agrees with reason and is conducive to the good and benefit of one and all, then accept it and live up to it.— Gautama Buddha (563 - 483 BC)

When dates can make a difference

When interpreting findings, putting scares into perspective and balancing public health risks in today’s world, that is when using more recent studies and data can help us. Remembering, for instance, advances in medical care; the dramatic improvements in our air quality and pollution levels over recent decades; and changes among our population, such as its aging and increasing diversity, and bettering prosperity and availability of food and immunizations which have led us to be healthier and living longer than ever; can all assist us in putting claims into perspective for our lives today. Examples abound, such as using the most recent data on influenza pandemics rather than deaths of a century ago, or the most recent figures on dropping cancer deaths or astounding reductions in heart disease deaths in light of our diets and weights. :))

© 2007 Sandy Szwarc