Err on the Side of Bologna – According to Scientists

by melonakos on August 9, 2010

in Education,Engineering,Grad School,Media,Politics,Science

eric

Brother Eric in 1986 - Sleeping on Stomach

david

Son David in 2004 - Sleeping on Back

In 1986, science said putting a baby on its stomach was the “right” thing to do, so that’s how my siblings and I slept. By 2004, science had changed its mind, so my firstborn slept on his back. He cried, and we were miserable. In 2006 and 2008, we were told to put our daughters on their backs. We ignored the experts and had a much better experience!

How many times have you read an article or heard a news broadcast where the phrase “according to scientists…” is used? The phrase is used to lend credibility. Occasionally, journalists actually cite the particular scientists; most times they do not.

There is a lot of trashy science out there. It is increasingly difficult to discern truth from error.

The root of the problem is that scientists have skewed incentives. Publishing is king, along with building a network of scientific allies who make it easier for you to publish. Yet, the peer-review system often does nothing to verify results, and scientists have no accountability for reproducibility.

There are a few initiatives trying to fix this problem. For example, in my field of computer vision, there exists the Insight Journal. This journal promotes the concept of “open science” wherein you submit your paper along with code and data so that other people can reproduce your results. The reviews are totally open. Everything is made public. Unfortunately, this initiative seems to have lost steam over time. We’re all still wedded to the system of closed reviews and irreproducible results.

Intensifying this problem are too many students flooding the world with too many papers. Unfortunately, scores of those papers are erroneous and sloppy, another chunk of them are simply fluff that no one cares about, and some of them are blatant lies designed to further an agenda. The truly good ones often get lost in the weeds.

Another problem, less openly discussed, is that some scientists can be flat-out cocky. A variety of things scientists study and claim to “know” are just preposterous. Man-made measurement systems are often too inadequate to study complex natural systems. So scientists build “models” to sweep the true complexity under the rug. Wherever models are used, we should have a Surgeon-General-like warning on the results: “Warning: Man-made models were used to convert God’s complex creation into a feeble human-designed computer simulation.”

Examples of scientific areas that I believe to be extraordinarily complex include neuroscience, climate science, and archeaology. Because so little is truly understood in these areas, true discoveries can be extremely fruitful. But bad things happen when we start placing faith in results that are overly contrived or extrapolated from inaccurate and sparse measurements.

Problems with science lead to problems in policy, as the media attempts to create headlines out of research. This invariably leads to the phrase “according to scientists…” and the dangerous practice of believing that scientific consensus is worth beans.

Michael Crichton gave an awesome lecture at Caltech on the issue of consensus science. Go read that now!

Over time, I’ve become very skeptical of articles claiming this or that. I find myself deciding to err on the side of calling most stuff bologna.

Wish it were otherwise, and wish I had a solution to this problem. I don’t have a solution; at least I don’t have a practical solution. Do you?

Tagged as: education, engineering, grad school, media, politics, publications, science

  • Pingback: Wallowing in Education — Table Conversations

  • Anonymous
  • Tadizi

    This is an interesting topic and something I have given thought to in the past. I think there is fault at many levels of the system. For instance, I have had long discussions with a knowledgeable and respected Earth scientist regarding global warming. I admitted some skepticism based on the years of hearing about impending doom and find that more than a decade later, nothing seems all that different despite being told the problem is getting “worse.” He made a convincing case for global warming (much based on computer models) and laid the blame at the feet of the media. Basically overzealous journalist sensationalizing actual scientific work and disseminating it to a wide audience. Claims are misrepresented and taken out of context and 99.9% of people never go back and read the original scientific paper that the article was based on.

    I think there is truth to that. I also think its very easy to get information out (thanks to the internet) so you need to be very careful of the source. There are still strongly peer reviewed publications but they are not the only voices anymore.

    Another “problem” is the desire to protect IP. I expect that if you make major breakthroughs with Jacket, you will not publish that code. That doesn’t mean your results are not true, just that you want to protect your IP. Of course, claims such as these must be taken with a significant grain of salt, though they could be just as accurate.

    I do agree with your apprehension of “consensus science,” though. Moreover, I think there are popular areas of scientific research that are beneficial to ones career to pursue. Researchers, even those in academic institutions, are capitalists. They compete for research funding. I think this is a good thing but one of the byproducts seems to be the certain topics become “hot” and thus people flock to that area and often support a specific thesis in order to help their careers and chances of receiving funding. I think you see examples of that in the subjects you innumerated in your post.

    Interesting topic of discussion, though. Notice I present no solutions. Just be skeptical of everything, even if some claims: According to scientists…

  • Anonymous

    Thanks for the comment. I agree with all of your points. There are indeed faults at many levels of the system, including all the things you mentioned: media, protecting IP, competition for research funding, etc.

    As for there being “strongly peer reviewed publications”, some are certainly more strongly reviewed than others. In my experience, there is a high variance in the level of scrutiny within a given publication. Typically, there are 3-5 reviewers ranging from those who fly through the review in 10 minutes and others who spend a day or more reviewing the paper in depth. So it often seems very hit-or-miss on the integrity of the process.

Previous post:

Next post: