Davis study still lacks self-selection control group (and the sample is still small)

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Tue, 23 Nov 2010 11:20:36 -0500

On 2010-11-23, at 9:46 AM, Philip Davis wrote (in the Sigmetrics Forum):
      Critics of our open access publishing experiment (read: Stevan
      Harnad) have expressed skepticism that we were too eager to report
      our findings and should have waited between 2 and 3 years.  All of
      the articles in our study have now aged 3-years and we report [1]
      that our initial findings [2] were robust: articles receiving the
      open access treatment received more article downloads but no more

      During the first year of publication, open access articles received
      more than double the number of full-text downloads (119%, 95% C.I.
      100% - 140%) and 61% more PDF downloads (95% C.I. 48% - 74%) from a
      third more unique visitors (32%, 95% C.I. 24% - 41%). Abstract views
      were reduced by nearly a third (-29%, 95% C.I. -34% - -24%)
      signaling a reader preference for the full article when available.

      Thirty-six months after publication, open access treatment articles
      were cited no more frequently than articles in the control group
      (Figure 2). Open access articles received, on average, 10.6
      citations (95% C.I. 9.2 -12.0) compared to 10.7 (95% C.I. 9.6 -
      11.8) for the control group. No significant citation differences
      were detected at 12, 18, 24 and 30 months after publication.

      1. Davis, P. M. 2010. Does Open Access Lead to Increased Readership
      and Citations? A Randomized Controlled Trial of Articles Published
      in APS Journals. The Physiologist 53: 197-201.

      2. Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., &
      Connolly, M. J. L. 2008. Open access publishing, article downloads
      and citations: randomised trial. BMJ 337: a568.

See:  Correlation, Causation, and the Weight of
Evidence http://openaccess.eprints.org/index.php?/archives/772-guid.html

Phil Davis's dissertation results are welcome and interesting, and include some
good theoretical insights, but insofar as the OA Citation Advantage is
concerned, the empirical findings turn out to be just a failure to replicate the
OA Citation Advantage in that particular sample and time-span. Phil's original
2008 sample of 247 OA and 1372 non-OA articles in 11 journals one year after
publication has now been extended to 712 OA and 2533 non-OA articles in 36
journals 2-3 years after publication. The result is a significant download
advantage for OA articles but no significant citation advantage. (Brody et al
(2006) reported -- for physics article in Arxiv -- that a download advantage in
the first 6 months after publication is correlated with a citation advantage 1.5
years or more after that; see also Gentil-Beccot et al's (2009) data, below). 

      Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage
      Statistics as Predictors of Later Citation Impact. Journal of the
      American Association for Information Science and Technology (JASIST)
      57(8) pp. 1060-1072. 

Gentil-Beccot, A, Mele, S  & Brooks TC (2009) Citing and reading
behaviours in high-energy physics. Scientometrics

The only way to describe this outcome is as a non-replication of the OA Citation
Advantage on this particular sample; it is most definitely not a demonstration
that the OA Advantage is an artifact of self-selection, since there is no
control group demonstrating the presence of the citation advantage with
self-selected OA and the absence of the citation advantage with randomized OA
across the same sample and time-span: There is simply the failure to detect any
citation advantage at all.

This failure to replicate is almost certainly due to the small sample size as
well as the short time-span. (Phil's a-priori estimates of the sample size
required to detect a 20% difference took no account of the fact that citations
grow with time; and the a-priori criterion fails even to be met for the
self-selected subsample of 65.) 

"I could not detect the effect in a much smaller and briefer sample than
others" is hardly news! Compare the sample size of Phil's negative results with
the sample-sizes and time-spans of some of the studies that found positive


And here is how the OA citation advantage builds up with time (read the curves
on the left from bottom to top to see the effect of a longer and longer time
interval: [the topmost curve should read "1998-2008" rather than "1998-2001"]):


Gargouri, Y., Hajjem, C., Lariviere, V., Gingras, Y., Brody, T., Carr, L. and
Harnad, S. (2010) Self-Selected or Mandated, Open Access Increases Citation
Impact for Higher Quality Research. PLOS ONE 10(5) e13636

Harnad, S. (2008) Davis et al's 1-year Study of Self-Selection Bias: No
Self-Archiving Control, No OA Effect, No Conclusion. Open Access
Archivangelism July 31 2008

Stevan Harnad

    [ Part 2, Image/GIF (Name: "Slide0001a.gif") 26 KB. ]
    [ Unable to print this part. ]

    [ Part 3, Image/GIF (Name: "citegrowth.gif") 47 KB. ]
    [ Unable to print this part. ]
Received on Tue Nov 23 2010 - 16:30:22 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:50:17 GMT