Autism News Beat

An evidence-based resource for journalists

Autism News Beat header image 2

Why is peer review important?

November 12th, 2007 · 10 Comments · Research

A belated response to ANB reader Joanna who asks about peer review. Hat tip to ANB reader Ms. Clark, who points us to senseaboutscience.org .

SUMMARY

  • Science has a system for assessing the quality of research before it is published. This system is called peer review.
  • Peer review means that other scientific experts in the field check research papers for validity, significance and originality – and for clarity.
  • Editors of scientific journals draw on a large pool of suitable experts to scrutinise papers before deciding whether to publish them.
  • Many of the research claims you read in newspapers and magazines, find on the internet, or hear on television and the radio are not published in a peer-reviewed journal.
  • Some of this research may turn out to be good but much of it is flawed or incomplete. Many reported findings, such as claims about “wonder cures” and “new dangers”, never come to anything.
  • Unpublished research is no help to anyone. Scientists can’t repeat or use it and as a society we can’t base decisions about our public safety – or our family’s health for example – on work that has a high chance of being flawed.
  • So, no matter how exciting or compelling new scientific or medical research is, you must always ask:

Is it peer reviewed? If not, why not?

If it is peer reviewed, you can look for more information on what other scientists say about it, the size and approach of the study and whether it is part of a body of evidence pointing towards the same conclusions.

Share

Tags:

10 responses so far ↓

  • 1 notmercury // Nov 12, 2007 at 1:02 pm

    Many of the publications from the Geiers claim to be peer-reviewed. How do we know if the reviewers are honest, qualified, or even real?

  • 2 Timelord // Nov 12, 2007 at 4:04 pm

    They’re not. The constant queries aimed at the work by the Geiers prove it. If they had been, where are the experts who reviewed their work? They should be coming forward and backing up the Geiers. But the silence is deafening on that score isn’t it?

  • 3 Iterverbal // Nov 12, 2007 at 6:28 pm

    Thank you for this concise and valuable summery. It is good to remember why peer review is so important.

  • 4 Interverbal // Nov 12, 2007 at 6:33 pm

    Anon, well…..

    I can think of several counter examples to your claim. And while negative results are harder to get published in any science field (not just medicine) you are bounding right into a logical fallacy if you will play the pharma shill gambit.

  • 5 Maya M // Nov 14, 2007 at 12:38 am

    The company producing secretin that funded a trial to probe its effectiveness in treating autism didn’t prevent the negative results of the trial from being published.
    As for the Geiers’ publications, they are a mystery to me. They would be relatively easy to explain if they were all in one and the same journal, but they are in different journals.
    I wish to know how these guys get their works published. Such knowledge could be very helpful when time comes to submit a new manuscript for publication.

  • 6 Prometheus // Nov 14, 2007 at 4:40 pm

    Maya,

    As a person involved – albeit peripherally – in the Repligen secretin trial you mention, I feel obligated to point out that although they didn’t prevent the news from getting out (they, in fact, announced that secretin didn’t work), they haven’t yet approved a publication of the data.

    Every center that participated in the Repligen secretin study was asked (required) to sign a form giving Repligen a “first look” at any publications and – if I remember correctly – their approval was necessary before publication.

    To be honest, I don’t know how many people in the multi-center study have asked to get the data so that they could publish, but you’ll notice that there hasn’t been a single publication in a scientific or medical journal.

    As for the journals that publish the Geier and Geier “studies” – there are a couple of possible explanations:

    [1] Some – Medical Hypotheses, for instance, don’t do peer review. If you doubt me, look in their “Instructions to Authors”.

    [2] Other journals are looking for something to raise their impact factor. Controversial – or even incorrect – studies gets a lot of citations (primarily from studies proving that they’re wrong) and thereby raise the impact factor of the journal they’re published in.

    [3] Journal editors are human, too. They are just as susceptible to unscientific and emotion-based arguments as the next person. An editor who feels – for example – that mercury causes autism might be inclined to give an article supporting that point of view an “easy ride” by sending it to reviewers that are known to be sympathetic.

    Prometheus

  • 7 Timelord // Nov 15, 2007 at 4:20 am

    “If the journals that peer reviewed articles appear in are funded by drug company advertising, that means that science that shows the drug companies screwed up will never appear. This fact makes your position on peer review null and void.”

    Very funny, Best (we know it’s you). There are plenty of publications that aren’t supported by Big Pharma. But still the so called peer reviewers are silent! Care to explain that? Or are we going to hear more red herrings based on wild conspiracy theories?

  • 8 Schwartz // Nov 18, 2007 at 2:26 am

    A couple more points:

    * Peer review does not guarantee good science
    * Many peer reviewed studies contradict each other (this means that many of them draw incorrect conclusions)
    * Peer reviewed studies that have been deemed non-credible after publication are rarely retracted, and often continue to be referenced.
    * Peer reviewers are anonymous (for some good reasons) but this means that they do not risk their reputation for poor review quality
    * Peer review is often frustrating for researchers because it can be an impediment for publication, and sometimes due to very minor points or opinioned ommissions — other times, it is fully justified
    * Negative results are less likely to be published, but my understanding is that is not due to peer-review. That is more due to the fact that most people are not interested in reading about negative results and thus people tend not to go through the effort of publishing them.
    * Most main-stream medical journals do receive significant funding from Pharamceuticals
    * Industry funded studies have been shown to be biased toward results that favour the funding party — this is well documented in peer-reviewed studies
    * The practise of using Ghost Study authors (especially statisticians) is quite prevalent in medical studies — even more concerning is that these ghost authors are often provided by the funding party. This is also documented in peer-reviewed studies.
    * Conflict of Interest guidelines are not consistent between publications, and they are not consistently enforced. This has also been documented in peer-reviewed studies.

    Some things I don’t necessarily agree with in the summary above:

    Point #1
    * Peer review does not rate the quality of the article published, it only determines if the study has passed the minimum constraints for publish-ability for the particular journal. I might be wrong here, but I’ve never seen a ranking or scale of quality scores from peer-reviewers. That statement is very misleading.

    Point #5
    Absolutely true for both peer-reviewed and non-peer reviewed studies.

    Point #6
    I agree with this. Unfortuanately, peer-reviewed studies are not generally accessible to the public (unless you are wealthy or plan to go through volumes of paper at the library). Even more concerning, is that safety trials used to justify the use of drugs/vaccines are not available to the general public, and the raw data is almost never made available to independent third parties for analysis.

    Additionally, a study does not have to be published in a peer-reviewed journal to be useful as inferred by this post.

    My advice: Read the study and the conclusions yourself. Check the references, and read those studies as well. Even a lay person can get a good idea as to the quality of the study by reading it. Often the stated conclusions clearly overreach the data provided in the study body.

    Also, you can’t disregard non-peer reviewed studies just because it isn’t peer-reviewed. Peer review will guarantee a minimum level of scientific merit, but that doesn’t mean non-peer reviewed studies aren’t scientific. It does mean you should ask why it isn’t peer-reviewed and read the details carefully as you would with any study.

  • 9 Schwartz // Nov 18, 2007 at 2:29 am

    Timelord,

    To my knowledge, peer-reviewers never speak up regarding any studies and they always carefully guard their anonamynity. That is pretty universal.

    That is not a good argument to discredit Geiers’ publications. There are much better ways.

  • 10 sara // Feb 6, 2010 at 12:32 pm

    not always we can trust articles if they are not peer view this is becuse they can have false information, amd it can sometimes harm people which depends on the product they have published.
    peer view includes different scientist to collect different data snd they can compare them with diiffernt results and come with a conclusion.
    this why it is very important to get the report to be peer viewed coz who noe if we shud trust the report or it can be fake.

Leave a Comment