On Sat, 23 Feb 2002, Sergio Della Sala and Jordan Grafman wrote:
> We are convinced that peer-review is central to scientific credibility.
> However, as it stands the process is far from watertight. Is there any
> way we can improve it by suggesting any modification, either radical or
> minimal? Time is ripe for such a discussion to be launched (see the
> JAMA and BMJ four congresses on peer review in biomedical publication:
> www.jama-peer.org).
Discussion, yes; testing, yes; but implementation: not yet!
Peer review is imperfect (as is human judgment and performance, in all
areas, as far as I know). But it has yielded the current quality-level
of our refereed research literature. Peer review reform proposals need
to be empirically tested and demonstrated to be able to yield at least
a comparable level of quality. And the sample needs to be large and
representative enough to give a reasonable likelihood that positive
findings are not just a hawthorn effect, and will reliably scale up to
the literature as a whole.
Until such data are available, it does not seem wise to put research
quality at risk by recommending or implementing untested variants of or
alternatives to classical peer review on the "live" literature. (The
trials should at least be conducted in parallel with classical peer
review as a backup and a control baseline for comparison.)
> One possible alternative is to substitute referees with sponsors,
> chosen by the authors, who overtly review and promote the papers (with
> their names as sponsors on it) they regard worthy of publication prior
> to submission. The manuscript would therefore be submitted to the
> journal ready to be published, although the editorial board could still
> reject it on scientific or non-scientific grounds (e.g.,
> inappropriateness for the journal audience). On the one hand this will
> make the refereeing process overt and rewarding for the reviewers. On
> the other hand, it will require more responsibility, because the
> sponsors names would be on it. Some problems are however immediately
> apparent with this kind of review process: nepotism, favouritism
> towards better known authors, over-commitment of the most popular
> sponsors, awkwardness in requesting the review from sponsors, and, more
> dangerously, scientific or personal covert blackmailing. All true,
> though not dissimilar from the imperfections of the current system.
These variants have all been proposed before, and some have even
been given some experimental trials. But the jury is still out, and
until there is reliable positive evidence, these are still merely
speculative hypotheses, rather than empirical findings ready to be
applied in practise.
> Would this proposal, all things considered, improve the present
> cumbersome and not indisputable process? If not, does anybody have a
> better idea? Is it even worth thinking about possible modifications of
> the present system?
It is not clear whether it would work, and even less clear what quality
papers it would yield, in the short and long run. It needs to be tested
before it can be implemented.
See:
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (c. 5 Nov. 1998)
http://helix.nature.com/webmatters/invisible/invisible.html
Longer version:
http://www.exploit-lib.org/issue5/peer-review/
http://www.ecs.soton.ac.uk/~harnad/nature2.html
"Peer Review Reform Hypothesis-Testing"
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/0479.html
"A Note of Caution About 'Reforming the System'"
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/1169.html
Stevan Harnad
Received on Sun Feb 24 2002 - 17:12:14 GMT