Re: Peer Review Reform Hypothesis-Testing

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Wed, 21 May 2003 21:03:45 +0100

On Wed, 21 May 2003 [identity deleted] wrote:

> [I have been asked] to find some information on International
> Peer Review standards. Having read your article I
> thought you would have some information. Would you point me
> in the right direction to any standards information please?

I'm afraid there is no answer to your question because the question
is based on a misconception of (research journal) peer review. Peer
review is merely qualified experts evaluating the quality of the work
of fellow experts. The evaluation is meant to give expert feedback on
things that need to be fixed and then, if the fixing is successful, to
certify the work as having met the quality standards of the journal in
question. Journals differ in their quality standards; there is a hierarchy
of them, from those with the highest standards (usually, but not always,
reflected by their rejection rates and impact factors) in their field at
the top, grading all the way down to almost a vanity press at the bottom.

No one can explicitly formalize the quality standards of each journal at
each level, any more than one can say in advance for a jury exactly what
features make someone guilty or not guilty of a crime. In both jury duty
and peer review there are criteria: base your judgment on evidence,
not hearsay or opinion, do not be swayed by prejudice, etc. In the case
of peer review there is rather more that one could say by way of general
scientific and scholarly criteria, but it still always comes down to a
matter of human judgment: qualified expert judgment, one hopes, and
answerable to the meta-judgment of a qualified expert editor.

Some journals consult one referee, some consult many; some give long
criterial check-lists, some just ask for judgments as to soundness,
originality, importance and comprehensibility. But what standards there
are are those of the domain of expertise in question, be it cancer
research or postmodernism -- and how reliably and responsibly the editor
sees to it that they are adhered to, in choosing qualified, unbiassed
referees, and adjudicating which of their recommendations the author
must meet in revisision, and hence when the revision has successfully
met that journal's standards. That is what a journal's track-record
attests to. Otherwise, there are no "peer review standards,"
international or otherwise, for research journal peer review.
(About peer review in professional practise I have no knowledge
at all.)

Harnad, S. (1998/2000) The invisible hand of peer review. Nature
[online] (5 Nov. 1998)
http://helix.nature.com/webmatters/invisible/invisible.html

Stevan Harnad
Received on Wed May 21 2003 - 21:03:45 BST

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:46:57 GMT