Re: Hefce backs off citations in favour of peer review in REF

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Wed, 24 Jun 2009 10:34:39 -0400

    [ The following text is in the "windows-1252" character set. ]
    [ Your display is set for the "iso-8859-1" character set. ]
    [ Some characters may be displayed incorrectly. ]

On Wed, Jun 24, 2009 at 9:25 AM, Stephen J
Bensman <notsjb_at_lsu.edu> wrote: 

      that is pretty much how Garfield recommended citations
      should be used and how they are used in US evaluations. 
      You don?t use citations by themselves but to balance your
      subjective judgments.  

Gene is of course right that citations alone are not and never were
enough for research evaluation; they not only need to be "balanced"
against subjective (peer expert) evaluations, but they need to be
formally validated against them, discipline by discipline.  

Moreover, it's not just about citations any more. A growing battery
of research performance metrics need to be jointly validated and
initialized against peer ranking. That's what the UK RAE/REF makes
possible, uniquely, at a national, pandisciplinary
scale. http://bit.ly/etLvL

Until the full-scale joint validation exercise is conducted and
analyzed, discipline by discipline, no one can say what percentage of
the variance in the peer rankings the metric battery can predict. If
it's 20-40%, then metrics can only be advisory, merely supplementary
adjuncts to the more expensive and time-consuming peer rankings; if
it's 70-90%, then it's the peer rankings that are the supplementary
adjuncts to the metrics.

Stevan Harnad

On Wed, Jun 24, 2009 at 9:25 AM, Stephen J Bensman <notsjb_at_lsu.edu>
wrote:

      I hate to say it, but that is pretty much how Garfield
      recommended citations should be used and how they are
      used in US evaluations.  You don?t use citations by
      themselves but to balance your subjective judgments.  For
      Garfield?s recommendations, see the two URL below:

      http://www.garfield.library.upenn.edu/essays/v6p354y1983.pdf

       http://www.garfield.library.upenn.edu/essays/v6p363y1983.pdf

      For the most the most recent US National Research Council
      Data and methodology, see the following URLs:

      http://www.nap.edu/readingroom/books/researchdoc/

      http://sites.nationalacademies.org/pga/Resdoc/index.htm

      Given the politics of the thing, nobody in his right mind
      would use a purely metric approach, if he/she had any
      instinct for survival.    

      Stephen J. Bensman

      LSU Libraries

      Louisiana State University

      Baton Rouge, LA   70803

      USA

      notsjb_at_lsu.edu

      From: ASIS&T Special Interest Group on Metrics
      [mailto:SIGMETRICS_at_LISTSERV.UTK.EDU] On Behalf Of Stevan
      Harnad
      Sent: Wednesday, June 24, 2009 3:26 AM
      To: SIGMETRICS_at_LISTSERV.UTK.EDU
      Subject: [SIGMETRICS] Hefce backs off citations in favour
      of peer review in REF 

      Hefce backs off citations in favour of peer review in REF

      18 June 2009

      By Zoë Corbyn

Research assessments in hard sciences will now be 'informed' by
bibliometrics. Zoe Corbyn writes

The use of citations to determine the quality of academic work
in the hard sciences is to be abandoned in favour of peer
review in the new system being designed to replace the research
assessment exercise.

However, information about the number of citations a scholar's
work accrues could be provided to assessment panels to help
"inform" their judgments in a range of subjects....

http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=407
041&c=1

·         Richard Hull 20 June, 2009

So finally common sense prevails. But I would now like to know
exactly which stupid, thoughtless person, blinded by the New
Labour mantra of "evidence-based this that and the other",
first proposed the hair-brained idea to use citations?? Time
for some journalistic digging, I think. This person must be
exposed, as they have effectively wasted a huge amount of the
time and energy of HEFCE and indeed the academics who actively
opposed the idea.

·         Stevan Harnad 22 June, 2009

It's probably alright that instead of scrapping panel rankings
altogether and hard-wiring the outcome to metrics, the new REF
will continuing doing rankings and metrics in parallel, using
the metrics as advisory rather than binding.  

      That's fine; it will give the metrics a better chance to
      be cross-validated against peer judgment (though the
      hybrid metric-influenced rankings of the new REF will not
      be as independent a criterion against which to validate
      metrics as the RAE rankings were, when they were not
      influenced by metrics). 

The important thing is to make the battery of candidate metrics
as broad and rich as possible. It is true that metrics today
are still relatively sparse, but with the growth of open access
and a rich variety of web-based metrics emerging therefrom, the
power and scope of metrics will now grow and grow. 

About the possibility of abuse: Yes, one can abuse individual
metrics. Downloads are the easiest to abuse. But genuine
downloads generate genuine citations, and the correlation is
there and can be measured. There are other intercorrelations in
multiple metric profiles too. There are endogamy/exogamy
metrics: Self-citations, co-author citations, author-circle
citations, same-institution citations, same-journal citations.
With these, anomalies and abuses can be detected, named and
shamed. 

Multiple metrics create a pattern, a profile. If you
artificially manipulate one of them (say, downloads, or citing
others in your institution) it will be detectable as a
deviation from the normal profile. Once a few of these abuses
are prominently exposed and shamed, that will create a strong
deterrent against trying such tricks, since the objective is
the exact opposite: to increase one's prestige, not to tarnish
it. 

And unlike (some) individual metrics, multiple metric profiles
are almost impossible to manipulate jointly: Try writing
software to generate bogus downloads of your work looking as if
they all come from different IPs the world over, and then try
to generate the non-institutional citations that would normally
be the correlate of such high downloads. Even that 2-metric
trick is not easy to accomplish! 

Stevan Harnad University of Southampton

REPLY TO RICHARD HULL: ON EXPOSING THE CULPRIT -- Harnad, S.
(2001) Research access, impact and assessment. Times Higher
Education Supplement 1487: p. 16. http://cogprints.org/1683/
Received on Wed Jun 24 2009 - 15:44:08 BST

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:49:49 GMT