On Tue, Jun 16, 2009 at 3:33 AM, Pippa Smart <pippa.smart --
googlemail.com> wrote:
>
> Peer review has been widely studied in the biomedical sciences with different conclusions about its efficacy, and whether more open or more closed models make any difference to the quality.
>
> A review of many different studies (135 studies considered, and 19 included in the review), are presented in "Effects of Editorial Peer Review: A Systematic Review" by Tom Jefferson, MD; Philip Alderson, MBChB; Elizabeth Wager, MA; Frank Davidoff, MD, published in JAMA. 2002;287:2784-2786.
>
> Their conclusions are "Editorial peer review, although widely used, is largely untested and its effects are uncertain." (so not terribly helpful!)
Let's not get carried away. There's room for improvement in every
system of quality control, but that's not the same as saying there's
no need for quality control!
Anyone who thinks it is just as safe to treat seriously ill members of
their family on the basis of Wikipedia or Google as on the basis of
the top journals of the refereed biomedical literature -- until and
unless (refereed? unrefereed?) studies have first demonstrated
otherwise -- is taking a serious risk that we all instinctively
reject, for obvious reasons: Before we tamper with the health of our
family, we want to make sure qualified experts have reviewed and
approved what we are about to do.
In other words, "peer review" just means systematic quality control by
qualified experts. The rest is just about the most reliable and
effective way of implementing the quality control (in which, as noted,
there is always room for improvement).
It is absurd (and exceedingly foolhardy) to cast this issue in terms
of the premise that "peer review, although widely used, is largely
untested and its effects are uncertain." That puts the shoe on
entirely the wrong foot. It is the *absence* of peer review that has
not been tested to see whether no quality control at all -- or
Wikipedia- or Google-style "anything-goes and the rest is
up-to-popular-intervention" -- would keep our family's health at least
as safe as biomedical peer review does today.
In other words, the null hypothesis is not that our family's health is
as safe without peer review until (refereed? unrefereed?) evidence
demonstrated the contrary. The null hypothesis is the contrary --
family's health is unsafe without peer review until (refereed?
unrefereed?) evidence demonstrated the contrary -- and that is where
the burden of proof lies.
I have deliberately cast this issue in terms of life and death matters
for those we hold dear, because it is the only way to get people to
think seriously about what is really being claimed when the results of
a meta-analysis of disparate hit-and-run bit-studies on various
aspects of how peer review is implemented (e.g., blind vs. non-blind,
etc.) are announced as "peer review, although widely used, is largely
untested and its effects are uncertain."
(I have not, by the way, read the meta-analysis in question. With a
sweeping conclusion like that -- and knowing well the methods and
limits of meta-analyses, as well as the scope of what is here implied
to be the null hypothesis under investigation -- I have no need to. As
we all know, peer review is not 100% reliable in every individual
instance, even at the JAMA level...)
Such a conclusion, thus construed, is, as I said, utter, irresponsible
nonsense, if it is taken to be an empirical conclusion about the
reliability and validity of biomedical research that has been
published with versus without some form of systematic quality control
by qualified experts. (No study of such scope, scale and public-health
risk has ever been done.) The rest is just about the rigor and level
of the implementation of the quality-control standards (in which, as
noted, there is always room for improvement).
As to the rest of science and scholarship, where the health of kith
and kin is not at immediate risk -- it all depends on how seriously
you take the rest of science and scholarship...
Stevan Harnad
Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature
[online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B.
(2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield.
Pp. 235-242.
http://cogprints.org/1646/
Received on Tue Jun 16 2009 - 13:11:10 BST