On Thu, 1 Feb 2007, eric.buffetaut wrote:
> The idea that peer-reviewed papers are necessarily good and self-published
> papers are just "vanity press" seems rather naive.
Not necessarily, but the odds are certainly a lot better...
> Has the overall quality of published science been significantly increased
> by peer-review ?
Do the comparison: Pick a discipline that has a high-quality peer-reviewed
journal, and advertise the availability of a free-for-all journal in the same
field. Then compare the quality of the result for the expert-vetted versus the
ad-lib content.
Until then, based on many, many other areas of expertise where output
tested and certified by qualified experts has proved more reliable
and valuable than ad-lib output, the default assumption is that quality
control is greatly preferable to free-for-alls.
In particular, if we are serious about this question, we should ask ourselves
whether, if we have a loved one who is suffering from a serious illness, we would
rather have them treated on the basis of peer-reviewed medical findings or Web
wisdom. (I am not casting aspersions either on non-traditional medicine nor on web
wisdom as a potential *supplement* to peer-reviewed research-based medicine: we
are discussing whether it is a suitable *substitute* for it.)
> Taking obvious advances in general knowledge into consideration, is 1907 (mostly
> non-peer-reviewed) scientific publication intrinsically inferior to 2007
> (mostly peer-reviewed) scientific publication ? I very much doubt it.
In 1907, research was occurring on a scale where an exchange of letters
between the few qualified experts in the world was sufficient. With 2.5
million articles being published per year today, and an equal burgeoning
of would-be users, those leisurely days are long over (*if* it is
true that there was little peer review in 1907, rather than just much
less publication: the Societe des Savans and the Royal Society, both
peer associations, considerably predated all that, it seems to me, and
are generally regarded as the origins of peer reviewed learned journals).
> Peer-review is at best a necessary evil (to cope with the increase in the
> sheer bulk of papers submitted for publication), not a panacea - and at
> worst a serious hindrance to innovative thinking.
To repeat, with few authors and few users, everyone's a peer. The peer-reviewed
journal corpus is actually a hierarchy, with journals practising peer-review at a
variety of levels of rigour, and corresponding quality. This is all known from
their track records. Yes, these structures are a consequence of scale. But we are
where we are, in terms of quantity and quality, and demography, and not elsewhere.
That peer review quashes innovation is, I think, considerably exaggerated, if it
is true at all. Besides, today, of all times, peer review can no longer suppress
making writings accessible: Unrefereed preprints can be put on the web too. It is
just that they do not have the credibility of peer-reviewed content, and probably
a good thing they don't. (As a general rule: One cannot make rules based on the
possibility of rare exceptions. Jury trial has its failings too, but it sure beats
the alternative or a free for all...)
Peer review does not really decide whether something gets published today: It just
decides at what *level* in the journal hierarchy it gets published (and it often
improved its quality through mandated revision beforehand).
> Moreover, departmental journals don't have to dispense with peer-review, if
> they see any merit in it. I don't see anything to be gained from the
> disappearance of such journals, except for commercial publishers.
The trouble with in-house publications is that, unlike neutral third-party
publications, they have a conflict of interest: They want their work to be good,
but they want it published too. A neutral third-party publisher (i.e., other than
the author or his institution) can be a lot more objective. Moreover, overloaded
referees are more likely to lend their (free) services to a neutral journal than
to a house journal. (Law Reviews are house journals, and they tend to be
peer-reviewed in-house, often by students, or adjudicated by students. The result
is the result: A few good articles by peerless luminaries of the institution, and
a lot of less reliable content that one would hardly call peer-reviewed.)
"The Special Case of Law Reviews"
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/4090.html
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (c. 5 Nov. 1998) Exploit Interactive version
http://helix.nature.com/webmatters/invisible/invisible.html
http://www.exploit-lib.org/issue5/peer-review/
Peer Review Reform Hypothesis-Testing (started 1999)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#480
A Note of Caution About "Reforming the System" (2001)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#1170
Self-Selected Vetting vs. Peer Review: Supplement or
Substitute? (2002)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2341
Stevan Harnad
Received on Fri Feb 02 2007 - 15:53:41 GMT