> In the editorial announcing the launch of NetPrints, the editors and
> publishers stated (see BMJ 1999;319:1515-1516, 11 December; full text
> at: http://www.bmj.com/cgi/content/full/319/7224/1515):
>
>> "Opponents of free availability place a high value on peer review,
>> regardless of the contrary evidence. They argue that it efficiently
>> winnows out the wheat from the chaff and that articles not yet through
>> the mill of peer review may harm public health or even science itself.
>> Yet many unreviewed findings already find their way into the public
>> arena with the connivance of researchers and conference organisers,
>> usually in an uninterpretable form. Far better, we believe, for a
>> study to appear in full, with the opportunity for readers to assess
>> its claims and append their criticisms. We are alert to the danger
>> that members of the public may be misled by what they read on the
>> website, but an explicit warning appears on its opening screen and is
>> repeated above every article ...".
This is controversial, and based on conjecture and opinion rather than evidence.
There is no evidence that (peer?) commentary can replace peer review and
still maintain the current quality of the literature. In fact, if we are
conjecturing, let me add a conjecture (based on a trifle more evidence,
having implemented both peer review and peer commentary for over 20 years
now) to the contrary:
"There is a way to test our intuitions about the merits of this sort
of proposal a priori, using a specialist domain that is somewhat
more urgent and immediate than abstract "learned inquiry"; if we
are not prepared to generalise this intuitive test's verdict to
scholarly/scientific research in general, we really need to ask
ourselves how seriously we take the acquisition of knowledge: If
someone near and dear to you were ill with a serious but
potentially treatable disease, would you prefer to have them
treated on the basis of the refereed medical literature or on the
basis of an unfiltered free-for-all where the distinction between
reliable expertise and ignorance, incompetence or charlatanism is
left entirely to the reader, on a paper by paper basis?
"A variant on this scenario is currently being tested by the British
Medical Journal
<
http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>, but instead
of entrusting entirely to the reader the quality control function
performed by the referee in classical peer review, this variant,
taking a cue from some of the developments and goings-on on both
the Internet and Network TV chat-shows, plans to publicly post
submitted papers unrefereed on the Web and to invite any reader to
submit a commentary; these commentaries will then be used in lieu
of referee reports as a basis for deciding on formal publication.
Expert Opinion or Opinion Poll?
"Is this peer review? Well, it is not clear whether the
self-appointed commentators will be qualified specialists (or how
that is to be ascertained). The expert population in any given
speciality is a scarce resource, already overharvested by classical
peer review, so one wonders who would have the time or inclination
to add journeyman commentary services to this load on their own
initiative, particularly once it is no longer a rare novelty, and
the entire raw, unpoliced literature is routinely appearing in this
form first. Are those who have nothing more pressing to do with
their time than this really the ones we want to trust to perform
such a critical QC/C function for us all?
"And is the remedy for the possibility of bias or incompetence in
referee-selection on the part of editors really to throw
selectivity to the winds, and let referees pick themselves?
Considering all that hangs on being published in refereed journals,
it does not take much imagination to think of ways authors could
manipulate such a public-polling system to their own advantage,
human nature being what it is.
Peer Commentary vs. Peer Review
"And is peer commentary (even if we can settle the vexed "peer"
question) really peer review? Will I say publicly about someone who
might be refereeing my next grant application or tenure review what
I really think are the flaws of his latest raw manuscript? (Should
we then be publishing our names alongside our votes in civic
elections too, without fear or favour?) Will I put into a public
commentary -- alongside who knows how many other such commentaries,
to be put to who knows what use by who knows whom -- the time and
effort that I would put into a referee report for an editor I know
to be turning specifically to me and a few other specialists for
our expertise on a specific paper?
"If there is anyone on this planet who is in a position to attest to
the functional difference between peer review and peer commentary
(Harnad 1982, 1984), it is surely the author of the present
article, who has been umpiring a peer-reviewed paper journal of
Open Peer Commentary (Behavioral and Brain Sciences [BBS]
<
http://www.princeton.edu/~harnad/bbs.html>, published by Cambridge
University Press) for over 2 decades (Harnad 1979), as well as a
brave new online-only journal of Open Peer Commentary, likewise
peer-reviewed (Psycoloquy, sponsored by the American Psychological
Association, <
http://www.princeton.edu/~harnad/psyc.html>), which
entered its second decade with the millennium.
"Both journals are rigorously refereed; only those papers that have
successfully passed through the peer review filter go on to run the
gauntlet of open peer commentary, an extremely powerful and
important supplement to peer review, but certainly no substitute
for it. Indeed, no one but the editor sees [or should have to see]
the population of raw, unrefereed submissions, consisting of some
manuscripts that are eventually destined to be revised and accepted
after peer review, but also (with a journal like BBS, having a 75%
rejection rate) many manuscripts not destined to appear in that
particular journal at all. Referee reports, some written for my
eyes only, all written for at most the author and fellow referees,
are nothing like public commentaries for the eyes of the entire
learned community, and vice versa. Nor do 75% of the submissions
justify soliciting public commentary, or at least not commentary at
the BBS level of the hierarchy." (Harnad 1998/2000)
This is not to say that the peer review system could not stand with
some empirical tetsing and improvement. But the test has to be done
first, and its outcome shown to be an improvement (or at least not the
opposite) before any alternative is implemented. In the meanwhile, let
us not blur the difference between Open Archives with Journals, and
between commentary and peer review. (In particular, archives will
contain both prefereeing preprints and peer-reviewed reprints.)
Needless to say, this is in direct disagreement with the blurring that
Ralph Youngen recommends in Jill's quote from Ralph Youngen below:
On Tue, 4 Apr 2000, Jim Till wrote:
> http://chemweb.com/alchem/2000/news/nw_000331_publish.html
>
> This url provides access to an article by Chris Leonard ("Leave Us
> Alone!"). The article includes a comment attributed to Ralph Youngen
> (ACS Publications):
>
> "Journal/database blurring - as journals are increasingly available in
> electronic format and it becomes easier to search across journal titles,
> the notion of what a journal is starts to become vague. Scientists may
> well end up publishing their paper not with a journal, but with a
> publisher who maintains a database of different manuscripts".
>
> It remains to be seen whether or not the ClinMed NetPrints website will
> gain acceptance as a database of this kind. In the meantime, perhaps
> there needs to be as little blurring as possible with regard to the
> criteria applied for inclusion of articles into such databases?
Harnad, S. (ed.) (1982) Peer commentary on peer review: A case
study in scientific quality control, New York: Cambridge
University Press
Harnad, S. (1985) Rational disagreement in peer review. Science,
Technology and Human Values 10: 55 - 62.
Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A
difficult balance: Peer review in biomedical publication.) Nature
322: 24 - 5.
Harnad, S. (1996) Implementing Peer Review on the Net: Scientific
Quality Control in Scholarly Electronic Journals. In: Peek, R. &
Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier.
Cambridge MA: MIT Press. Pp. 103-118.
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer
Review, Peer Commentary and Copyright. Learned Publishing 11(4)
283-292. Short version appeared in 1997 in Antiquity 71:
1042-1048. Excerpts also appeared in the University of Toronto
Bulletin: 51(6) P. 12.
http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (5 Nov. 1998)
http://helix.nature.com/webmatters/invisible/invisible.html Longer
version in Exploit Interactive 5 (2000):
http://cabaret.ukoln.ac.uk/exp-int-development/issue5/
http://www.ecs.soton.ac.uk/~harnad/nature2.html
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 23-80 592-582
Computer Science fax: +44 23-80 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
Received on Mon Jan 24 2000 - 19:17:43 GMT