about the Australian RQF, particularly in relation to IRs and the way in
which panels will access submitted publications, are accurate. However,
his "definition" of quality and impact in the RQF context is seriously
misleading. Yes, the terms are used in an unusual way, but his attempt
to paraphrase the meaning is way off. The definitions contained in the
official document are:
^Õ the quality of original research including its intrinsic merit and
academic impact. Academic
impact relates to the recognition of the originality of research by peers
and its impact on the
development of the same or related discipline areas within the community
of peers; and
^Õ the impact or use of original research outside the peer community that
will typically not be
reported in traditional peer reviewed literature (that is, the extent to
which research is
successfully applied during the assessment period for the RQF). Broader
impact relates to the
recognition by qualified end users that methodologically sound and
rigorous research has
been successfully applied to achieve social, economic, environmental
and/or cultural
outcomes.
Quality is NOT a solely metrics-based exercise. It is the peer
assessment of 4 outputs per active researcher (as in the RAE), informed
by quantitative indicators supplied to the panel (citations, competitive
grants, ranked outputs - details of proposed measures are on the DEST
website in the background papers).
Impact, the most difficult to assess, is judged from an "evidence-based
statement of claims". Obviously, there is a lot of detail behind that
statement - again, background papers are available on the DEST website.
It will definitely not be judged in the way outlined below.
Linda Butler
Research Evaluation and Policy Project
The Australian National University
At 03:34 PM 17/11/2006, you wrote:
Adminstrative info for SIGMETRICS (for example unsubscribe):
http://web.utk.edu/~gwhitney/sigmetrics.html
---------- Forwarded message ----------
Date: Fri, 17 Nov 2006 14:44:39 +1100
From: Arthur Sale <ahjs_at_ozemail.com.au>
To:
AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
The Australian Government has released a definitive, if
incomplete,
description of Australia's Research Quality Framework (RQF)
which is our
equivalent of the UK's RAE. If familiar with the RAE, you
will recognize the
family resemblance. I extract the essentials of the RQF for
an international
readership, and analyze some of the consequences likely to
flow from it. To
see the documentation, see
http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_i
ssues/research_quality_framework/rqf_development_2006.htm.
ESSENTIAL POINTS
1. The first RQF assessment will be based on submissions
by the 38
Australian universities by 30 April 2008. Funding based on
the assessment
will flow in calendar year 2009. Six years will elapse before
the next
assessment (ie 2014), but there is provision to shorten this.
2. The Unit of Assessment is the Research Group.
Research Groups will
be defined by up to three RFCD four-digit codes (to allow for
multi-disciplinary groups). The RFCD classification is
uniquely Australian,
and for example there are six four-digit codes in the field
of ICT.
Engineering has more but for example Civil Engineering is
one. If you are
interested in the codes see
http://www.research.utas.edu.au/publications/docs/14_rfcd.doc,
the four
digit codes are the sub-headings.
3. Each Research Group will be allocated to and assessed
by one of 13
Panels. The Panel is determined by the primary RFCD code.
Thus Mathematics,
Computing and Information Technology is Panel 4.
4. Each University will submit an Evidence Portfolio
(EP) for each
identified Research Group. There is provision for
cross-university Research
Groups.
5. The ratings will be based on Quality and Impact
separately. These
words have peculiar (ie not common-usage) meanings.
Approximately, Quality
is a bag of quantifiable metrics, and Impact is all the soft
things like
Fellowships of Academies, Honors, journal associate
editorships, etc. The
relative importance of Quality and Impact will vary by Panel
and is
similarly not yet resolved. Quality is based on the best four
publications
(Research Output) of each researcher in the group over the
six years
2002-2007, on a full list of all Research Output from the
group including
honorary and emeritus professors, and on competitive grants
received over
the period. Impact is covered in the Context Statement of the
EP
6. Impact for each Research Group will be assessed on a
scale of 1 (not
important) to 5 (prestigious)..
7. Impact is rated A (outstanding) to E (poor).
8. Research Groups which rate below 2 for Quality, or
below D for
Impact, will attract no funding to their university, though
the two factors
are separately aggregated for the University. The weighting
of funding is
stated to be linear with rating, but the gradient will be
determined during
2007.
9. The Panels require access to the electronic versions
of any of the
Research Output within four working days. The Panels will (a)
rank the
outputs by things like journal impact factors, journal
standing, etc, (b)
assess citation counts, both in aggregate and by the
percentage that fall in
the top decile for the discipline, and (c) competitive grant
income.
10. The RQF is based on a semi-centralized IT model (or
semi-decentralized). In other words, the full-texts of the
research outputs
(publications) will be held in IRs in each university, while
the RQF
secretariat will run a repository with all the EPs and
develop the citation
counts independent of the universities (in conjunction with
Thomson
Scientific and possibly EndNote Web). The Australian
Government will be
approached for funds to universities to establish these IRs.
ANALYSIS FOR OPEN ACCESS
* The RQF will actually use citation metrics in the
assessment, not
just test them as a "shadow exercise" as in the next RAE.
This will mean
that the OA citation advantage will suddenly look very
attractive to
Australian universities, though it is a bit late to do
anything about it
five years into a six-year window. However, with 2014 in
mind, there will be
pressure to increase citations.
* Every university will have to have an IR to hold the
full-text of
Research Outputs. About half already do, with EPrints and
DSpace being the
most popular software with a few Fedora-based repositories
and outsourced
ProQuest hosts. There will be funding to establish
repositories.
* I expect a mad scramble in the smaller universities,
with
outsourcing and hosting solutions being very attractive.
Money fixes
everything. The ones that have been dithering will regret it.
* All Research Output generated by all Research Groups
will have to
be in the IRs for the RQF. This may amount to 50% of the
university research
production over six years, or more or less depending on how
research
intensive it is. There are two corollaries: (a) this is
Mandate by Money,
and (b) there will be frantic activity over 2007 to put in
the backlog of
2002-2006 publications.
* Since one does not know what Research Output will be
needed in
2014, and only a general clue in 2007, 100% institutional
mandates are
likely to spring up all over the place, in the form of
Mandate by
Administration. What I mean by this is that the deposition of
the paper will
be integrated with the already present administrative annual
requirement to
report the publication to the Australian Government.
* Although it is nowhere stated explicitly that I can
see, I read
between the lines that the RQF may be expecting to get access
to the
publisher's pdf. This means that it will have to be in the
repository as
"restricted access" in most cases or as a link to an OA
source. There is no
reason why the OA postprint cannot be there as "open access"
as well, of
course, and if a citation advantage is to be got, it will
need to be.
Please feel free to blog this or forward this to anyone you
think may be
interested. My apologies for cross-posting.
Arthur Sale
Professor of Computing (Research)
University of Tasmania
Linda Butler
Research Evaluation and Policy Project
Research School of Social Sciences
The Australian National University
ACT 0200 Australia
Tel: 61 2 61252154 Fax: 61 2 61259767
http://repp.anu.edu.au
Received on Fri Nov 17 2006 - 11:45:28 GMT