Re: ClinMed NetPrints

From: Stevan Harnad <>
Date: Wed, 5 Apr 2000 15:14:21 +0100

> In the editorial announcing the launch of NetPrints, the editors and
> publishers stated (see BMJ 1999;319:1515-1516, 11 December; full text
> at:
>> "Opponents of free availability place a high value on peer review,
>> regardless of the contrary evidence. They argue that it efficiently
>> winnows out the wheat from the chaff and that articles not yet through
>> the mill of peer review may harm public health or even science itself.
>> Yet many unreviewed findings already find their way into the public
>> arena with the connivance of researchers and conference organisers,
>> usually in an uninterpretable form. Far better, we believe, for a
>> study to appear in full, with the opportunity for readers to assess
>> its claims and append their criticisms. We are alert to the danger
>> that members of the public may be misled by what they read on the
>> website, but an explicit warning appears on its opening screen and is
>> repeated above every article ...".

This is controversial, and based on conjecture and opinion rather than evidence.
There is no evidence that (peer?) commentary can replace peer review and
still maintain the current quality of the literature. In fact, if we are
conjecturing, let me add a conjecture (based on a trifle more evidence,
having implemented both peer review and peer commentary for over 20 years
now) to the contrary:

   "There is a way to test our intuitions about the merits of this sort
    of proposal a priori, using a specialist domain that is somewhat
    more urgent and immediate than abstract "learned inquiry"; if we
    are not prepared to generalise this intuitive test's verdict to
    scholarly/scientific research in general, we really need to ask
    ourselves how seriously we take the acquisition of knowledge: If
    someone near and dear to you were ill with a serious but
    potentially treatable disease, would you prefer to have them
    treated on the basis of the refereed medical literature or on the
    basis of an unfiltered free-for-all where the distinction between
    reliable expertise and ignorance, incompetence or charlatanism is
    left entirely to the reader, on a paper by paper basis?

   "A variant on this scenario is currently being tested by the British
    Medical Journal
    <>, but instead
    of entrusting entirely to the reader the quality control function
    performed by the referee in classical peer review, this variant,
    taking a cue from some of the developments and goings-on on both
    the Internet and Network TV chat-shows, plans to publicly post
    submitted papers unrefereed on the Web and to invite any reader to
    submit a commentary; these commentaries will then be used in lieu
    of referee reports as a basis for deciding on formal publication.

    Expert Opinion or Opinion Poll?

   "Is this peer review? Well, it is not clear whether the
    self-appointed commentators will be qualified specialists (or how
    that is to be ascertained). The expert population in any given
    speciality is a scarce resource, already overharvested by classical
    peer review, so one wonders who would have the time or inclination
    to add journeyman commentary services to this load on their own
    initiative, particularly once it is no longer a rare novelty, and
    the entire raw, unpoliced literature is routinely appearing in this
    form first. Are those who have nothing more pressing to do with
    their time than this really the ones we want to trust to perform
    such a critical QC/C function for us all?

   "And is the remedy for the possibility of bias or incompetence in
    referee-selection on the part of editors really to throw
    selectivity to the winds, and let referees pick themselves?
    Considering all that hangs on being published in refereed journals,
    it does not take much imagination to think of ways authors could
    manipulate such a public-polling system to their own advantage,
    human nature being what it is.

    Peer Commentary vs. Peer Review

   "And is peer commentary (even if we can settle the vexed "peer"
    question) really peer review? Will I say publicly about someone who
    might be refereeing my next grant application or tenure review what
    I really think are the flaws of his latest raw manuscript? (Should
    we then be publishing our names alongside our votes in civic
    elections too, without fear or favour?) Will I put into a public
    commentary -- alongside who knows how many other such commentaries,
    to be put to who knows what use by who knows whom -- the time and
    effort that I would put into a referee report for an editor I know
    to be turning specifically to me and a few other specialists for
    our expertise on a specific paper?

   "If there is anyone on this planet who is in a position to attest to
    the functional difference between peer review and peer commentary
    (Harnad 1982, 1984), it is surely the author of the present
    article, who has been umpiring a peer-reviewed paper journal of
    Open Peer Commentary (Behavioral and Brain Sciences [BBS]
    <>, published by Cambridge
    University Press) for over 2 decades (Harnad 1979), as well as a
    brave new online-only journal of Open Peer Commentary, likewise
    peer-reviewed (Psycoloquy, sponsored by the American Psychological
    Association, <>), which
    entered its second decade with the millennium.

   "Both journals are rigorously refereed; only those papers that have
    successfully passed through the peer review filter go on to run the
    gauntlet of open peer commentary, an extremely powerful and
    important supplement to peer review, but certainly no substitute
    for it. Indeed, no one but the editor sees [or should have to see]
    the population of raw, unrefereed submissions, consisting of some
    manuscripts that are eventually destined to be revised and accepted
    after peer review, but also (with a journal like BBS, having a 75%
    rejection rate) many manuscripts not destined to appear in that
    particular journal at all. Referee reports, some written for my
    eyes only, all written for at most the author and fellow referees,
    are nothing like public commentaries for the eyes of the entire
    learned community, and vice versa. Nor do 75% of the submissions
    justify soliciting public commentary, or at least not commentary at
    the BBS level of the hierarchy." (Harnad 1998/2000)

This is not to say that the peer review system could not stand with
some empirical tetsing and improvement. But the test has to be done
first, and its outcome shown to be an improvement (or at least not the
opposite) before any alternative is implemented. In the meanwhile, let
us not blur the difference between Open Archives with Journals, and
between commentary and peer review. (In particular, archives will
contain both prefereeing preprints and peer-reviewed reprints.)

Needless to say, this is in direct disagreement with the blurring that
Ralph Youngen recommends in Jill's quote from Ralph Youngen below:

On Tue, 4 Apr 2000, Jim Till wrote:

> This url provides access to an article by Chris Leonard ("Leave Us
> Alone!"). The article includes a comment attributed to Ralph Youngen
> (ACS Publications):
> "Journal/database blurring - as journals are increasingly available in
> electronic format and it becomes easier to search across journal titles,
> the notion of what a journal is starts to become vague. Scientists may
> well end up publishing their paper not with a journal, but with a
> publisher who maintains a database of different manuscripts".
> It remains to be seen whether or not the ClinMed NetPrints website will
> gain acceptance as a database of this kind. In the meantime, perhaps
> there needs to be as little blurring as possible with regard to the
> criteria applied for inclusion of articles into such databases?

    Harnad, S. (ed.) (1982) Peer commentary on peer review: A case
    study in scientific quality control, New York: Cambridge
    University Press

    Harnad, S. (1985) Rational disagreement in peer review. Science,
    Technology and Human Values 10: 55 - 62.

    Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A
    difficult balance: Peer review in biomedical publication.) Nature
    322: 24 - 5.

    Harnad, S. (1996) Implementing Peer Review on the Net: Scientific
    Quality Control in Scholarly Electronic Journals. In: Peek, R. &
    Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier.
    Cambridge MA: MIT Press. Pp. 103-118.

    Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer
    Review, Peer Commentary and Copyright. Learned Publishing 11(4)
    283-292. Short version appeared in 1997 in Antiquity 71:
    1042-1048. Excerpts also appeared in the University of Toronto
    Bulletin: 51(6) P. 12.

    Harnad, S. (1998) The invisible hand of peer review. Nature
    [online] (5 Nov. 1998) Longer
    version in Exploit Interactive 5 (2000):

Stevan Harnad
Professor of Cognitive Science
Department of Electronics and phone: +44 23-80 592-582
             Computer Science fax: +44 23-80 592-865
University of Southampton
Highfield, Southampton
Received on Mon Jan 24 2000 - 19:17:43 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:45:44 GMT