Re: [SIGMETRICS] Davis study still lacks self-selection control group (and the sample is still small)

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Thu, 25 Nov 2010 21:39:55 +0000

Adminstrative info for SIGMETRICS (for example unsubscribe): On Thu, Nov 25, 2010 at 6:30 PM,
Ludo Waltman <> wrote

      There is an essential difference between what Stevan Harnad says and
      what I am saying. Stevan states that the results of Phil Davis can
      be interpreted as "simply the failure to detect any citation
      advantage at all". He also states that "this failure to replicate is
      almost certainly due to the small sample size as well as the short
      time-span". We seem to agree that no OA citation advantage can be
      detected in Phil's data. What we do not agree about is whether or
      not this is likely to be caused by a small sample size. As indicated
      in my previous post, I consider this to be unlikely. Phil's results
      show not only that there is no OA citation advantage in his sample,
      but also that an OA citation advantage of reasonable size is
      unlikely to exist in the underlying population.

It depends on what you mean by "the underlying population." If you mean the 3
-year population of articles in the 36 (mostly APS) journals that Phil actually
tested, I agree. 

But if you mean that general population of articles, in multiple different
fields, tested by multiple independent investigators, reporting a significant
(and sometimes quite sizeable) OA citation advantage (with the exception of a
very small number of negative or null outcomes), then Phil's study certainly has
*not* shown "that an OA citation advantage of reasonable size is unlikely to
exist" in *that* underlying population. It would take a null meta-analysis, not
just one null outcome, to be able to show that. (Otherwise any repeatedly
observed effect could be dismissed on the basis of one non-replication!)

What Phil set out to show was that the repeatedly observed OA citation advantage
was an artifact of author self-selection, and the way he hoped to show that --
and it's a good way -- was to show that randomly imposed OA  eliminates the OA
citation advantage. So far so good. But what Phil failed to do was to replicate
the OA citation advantage in the case of author self-selection, and then show
that it is eliminated by randomization -- in the same journal population and
time interval.

Instead he just found no OA citation advantage at all, for that journal
population and interval.

That, to repeat, is one non-replication, for that journal population and time
interval. It is not a demonstration that the repeatedly observed OA citation
advantage is an artifact of author self-selection.

And -- not to focus only on negative results -- let me suggest that it is
pertinent to this question that we too did a test of the self-selection
hypothesis -- on a much larger sample across more fields and a longer time
interval -- and we not only found "an OA citation advantage of reasonable size"
for self-selected OA, but we found that the advantage was the same size for
mandated OA.  

We accordingly conclude that "an OA citation advantage of reasonable size is
likely to exist in the underlying population" if you test for it, and your
sample is big enough and long enough -- except perhaps in Phil's sample of
(mostly APS) journals...

Stevan Harnad

            On Wed, Nov 24, 2010 at 11:52 PM, Ludo Waltman
            <> wrote:


            Phil Davis has published interesting results on the
            question whether open
                  access leads to a citation advantage. In my
                  view, Stevan Harnad's criticism
                  of Phil misses the point...

            In my view, Phil has convincingly shown that, at least
            for the journals and
                  the time intervals he studied, there is no
                  meaningful OA citation advantage.

            I don't understand Ludo Waltman's point, since this is
            exactly what I said:


            Phil Davis's dissertation results are welcome and
            interesting, and include
                  some good theoretical insights, but insofar
                  as the OA Citation Advantage is
                  concerned, the empirical findings turn out
                  to be just a failure to replicate
                  the OA Citation Advantage in that particular
                  sample and time-span... it is
                  most definitely not a demonstration that the
                  OA Advantage is an artifact of
                  self-selection, since there is no control
                  group demonstrating the presence
                  of the citation advantage with self-selected
                  OA and the absence of the
                  citation advantage with randomized OA across
                  the same sample and time-span:
                  There is simply the failure to detect any
                  citation advantage at all.

            Stevan Harnad
Received on Thu Nov 25 2010 - 21:46:14 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:50:17 GMT