Examples of Analysis of Variance and Covariance

C. P. Doncaster and A. J. H. Davey

This page presents example datasets and outputs for analysis of variance (ANOVA) and covariance (ANCOVA), and computer programs for planning data collection designs and estimating power. All of the statistical models are detailed in Doncaster and Davey (2007), with pictorial representation of the designs and options for troubleshooting common issues with analysis.

Click here for the suite of commands in R (freeware statistical package, R Development Core Team 2010) that will analyze each of the example datasets below, and calculate the power of the design.

- What is a statistical model?

- Examples of ANOVA and ANCOVA models

3 Fully replicated factorial designs

- Analyses for figures and worked examples in Doncaster and Davey (2007)

- Computer programs for planning designs and estimating design power

- Key to types of statistical models

- Glossary

Any statistical test of pattern requires a model against which to test the null hypothesis of no pattern. Models for ANOVA and ANCOVA
take the form: Response = Factor(s)
+ ε, where the response refers to the data that
require explaining, the factor or factors are the putative explanatory
variables contributing to the observed pattern of variation in the response,
and ε is the residual variation in the response left unexplained by the
factor(s). For each of the examples illustrated here, we use a standard **notation**
to describe the full model and its testable terms. For example, the two-factor
nested model in Section
**2**
below is described by:

(i) The full model, packed up into a single expression: Y = B(A) + ε;

(ii) Its testable terms to declare in a statistics package, unpacked from the full model: A + B(A).

A statistics package will require you to specify the model desired for a given dataset. You will need to declare which column contains the response variable Y, which column(s) contain the explanatory variable(s) to be tested, any nesting or cross factoring of the explanatory variables, whether any of the variables are random rather than fixed factors, and whether any are covariates of the response.

Each of the links in Sections 1 to 7 below
shows a full suite of analyses of a hypothetical dataset. Where appropriate,
these include alternative restricted and unrestricted models (Searle 1971), and Model-1 and Model-2
designs (Newman *et al*. 1997). Refer
to the protocols in Doncaster and Davey (2007) to see which mean squares are
used for the *F-ratio* denominators, and consequently how
many error degrees of freedom are
available for testing significance. The examples
have not used *post-hoc* pooling though this may be an option or an alternative
to some quasi *F*-ratios,
and the underlying assumptions have not been
evaluated though this would need doing for real datasets.

A statistics package may give different
*F* ratios
and *P* values to those shown here, for a variety of reasons:

- In unbalanced and non-orthogonal designs and ANCOVA models, default use of Type-III adjusted SS for models that require Type II.

- In balanced mixed models and ANCOVA models, default use of an unrestricted model when the design may suite a restricted model.

- In designs with randomized blocks and split plots, default use of Model-1 analysis when the test hypothesis may require Model 2.

- In ANCOVA mixed models, default use of residual error for all denominator mean squares when the test hypothesis may have a different error variance.

Click here for the suite of commands that will analyze each of these example datasets in the freeware statistical package R (R Development Core Team 2010).

1.
*List all terms and degrees of freedom in any
model for analysis of variance or covariance*. Click
here to download a computer program (Terms.exe)
that will list all of the main effects and interactions and their degrees of freedom for a model of your own specification with any
number of levels for each of any number of cross-factored or nested categorical
or continuous variables (yielding up to a maximum of 2047 terms).

Specify the model as a hierarchical nesting of sampling units in factors, representing each variable by a single letter. Thus for example, requesting: 'P(B|S(A))' will yield testable terms for any of models 3.3, 5.6 or 6.3 above, depending on the nature of the variables and the replication of sampling unit P.

A text file 'Terms.txt' will be created to store the list of terms
comprising all main effects and interactions and their degrees of freedom; it
will also contain the *N* lines of factor-level combinations against which to tag
your *N* observations of the response.

2.
*List testable terms, degrees of freedom, and
critical *F*-values for any of the numbered designs above*. Click
here to download a computer program (CritiF.exe)
that allows you to specify your own sample sizes in the numbered ANOVA and
ANCOVA designs above, assuming fixed treatment factors. For
each estimable effect, it shows the test and
error
degrees of freedom, and the critical *F* at α = 0.05. For fixed effects, it shows the
standardized effect size with 80%
detection probability (the value of *θ */*σ*
that gives the test a power of 0.8).

Use the program to evaluate alternative experimental designs for a given workload of data points, targeting a low standardized effect size for treatments. This value will vary according to the distribution of data points between levels of sampling units and treatments. For a given total data points, it will be increased by the inclusion of nesting, covariates, blocking, split plots, or repeated measures. These may be desirable or intrinsic features of the experimental design, and they will increase power to detect treatment effects if they reduce error variances sufficiently to compensate for the reduction in error degrees of freedom.

A text file 'Factor_levels.txt' will be created to store the *N*
lines of factor-level combinations against which to tag your *N*
observations of the response.

3.
*Calculate statistical power for any balanced
model*. Click
here to download a computer program (Power.exe)
that calculates
statistical power prospectively for fixed or random
factors in any balanced model with a
proposed size of samples, given a threshold ratio of treatment effect size,
*θ* (the
standard deviation of the treatment variability) to
error effect size, *
σ* (the standard deviation of the random
unmeasured variation). It can also calculate the value of *θ */*σ* required to achieve a target power.

A pilot study may be needed to obtain an initial observed
*F* from
samples of size *n*. Then [(*F* - 1)/*n*]^{1/2} will provide an unbiased estimate of the
population *θ */*σ*, with which to
evaluate the potential to gain
power from more replication (e.g., Kirk 1968). Freeware is available elsewhere on the
web to further explore the relationships between *n*, *θ, *
*σ* and power for specified designs (e.g.,
Piface by Russell V. Lenth).

4.
*Calculate relative performance for any balanced
model*. Click
here to download a computer program (Performance.exe)
that calculates the performance of a balanced analysis of variance design
relative to a reference design for the same treatment(s). The relative
performance of the design is given by the fractional size of its error variance
that will just match the power of the reference. The value of relative
performance is robustly approximated by the ratio of reference to alternative *
α* quantiles of the *F* distribution, multiplied by the ratio of
alternative to reference effective sample sizes (Doncaster, Davey & Dixon 2014).
By comparing the precision of two designs at equal sensitivity, relative
performance provides a useful way to enumerate trade-offs between error variance
and error degrees of freedom when considering whether to block random variation
or to sample from a more or less restricted domain.

5.
*Find critical *
F*-values for any number of test and error
degrees of freedom, and value of *α. Click
here to download a computer program (Ftable.exe)
that provides critical
*F*-values
for a chosen *α*, and also gives
the Type-I error
probability associated with an observed value of *F*, given test and error degrees of freedom.

*Program citations *

Doncaster, C. P. (2007) Computer software for design of analysis of variance and covariance. Retrieved [date] from http://www.southampton.ac.uk/~cpd/anovas/.

Use this key to identify the appropriate section of model structures above, then look at example datasets and analyses.

1. Can you take observations with independently varying residuals that randomly sample from the populations of interest (i.e., from the levels of each factor or factor combination)?

*Yes* → 2.

*No* → identify the dependency structure and explore options to control
it in your model, for example by factoring in nuisance variables or
subpopulations or by
sub-sampling; otherwise the data may not suit
statistical analysis of any sort.

*Yes* → 3.

*No* → the data may not suit ANOVA or ANCOVA.

*Yes* → consider treating the continuous factor as a covariate and
using ANCOVA designs in Sections **
1 to 3** above; this will be the only option
if each sampling unit takes a unique value of the factor. The response and/or
covariate may require transformation to meet the assumption of linearity.
Analyze with a General Linear Model (GLM) and for non-orthogonal designs
consider using Type II adjusted SS if cross factors are fixed, or Type III
adjusted SS if one or more cross factors are random (and an unrestricted model,
checking correct identification of the denominator MS to the covariate). Be
aware that adjusted SS can increase or decrease the power to detect main
effects.

*No* → 4.

*Yes* → 5.

*No* → 9.

5. Are all combinations of factor levels fully replicated?

*Yes* → 6.

*No* → use an unreplicated design (Section
**7** above).

*Fully randomized and fully replicated designs*

6. Do your samples represent the levels of more than one explanatory factor?

*Yes* → 7.

*No* → use a one-factor design (Section
**1** above), considering options for
orthogonal contrasts (e.g., model 1.1 above).

7. Is each level of one factor present in each level of another?

*Yes* → 8.

*No* → use a nested design with each level of one factor present in only one level of another
(Section **2** above).

8.
Use a fully replicated
factorial design (Section **3** above), taking account of any nesting
within the cross factors (model 3.3 or
model 3.4 above). For balanced and orthogonal designs
with one or more random cross factors, consider using a restricted model, and
consider *post hoc* pooling if an
effect has no exact *F*-test. If cross
factors are not orthogonal (e.g., sample sizes are not balanced), use GLM and
consider using Type II adjusted SS if cross factors are fixed, or Type III
adjusted SS if one or more cross factors are random (and an unrestricted
model).

*Stratified random designs*

*Yes* → use a design with randomized blocks (Section
**4** above). If all factor combinations
are fully replicated, analyze with Section-3 ANOVA tables; otherwise consider analysis by Model 1 (assumes
treatment-by-block interactions) or Model 2 (assumes no treatment-by-block
interactions). For a single treatment factor, consider options to use a balanced incomplete block or, with cross factored blocks, a Latin square or Youden square.

*No* → 10.

*Yes* → use a design with split plots (Section
**5** above), taking account of nesting
among sampling units.* *

*No* → use a repeated-measures
design (Section **6** above) for repeated measurement of
each sampling unit at treatment levels applied in a temporal or spatial
sequence. If all factor combinations are fully replicated, analyze with
Section-**3** ANOVA tables.

Doncaster, C. P. and Davey, A. J. H. (2007) *Analysis of Variance and Covariance: How to
Choose and Construct Models for the Life Sciences*. Cambridge University
Press, Cambridge 302 pp. ISBN-13: 9780521684477. https://doi.org/10.1017/CBO978051161137

Doncaster, C. P., Davey, A. J. H.
& Dixon, P. M. (2014) Prospective evaluation of designs for analysis of variance
without knowledge of effect sizes. *Environmental and Ecological Statistics*,
21: 239-261.
https://doi.org/10.1007/s10651-013-0253-4.

Kirk, R. E. (1968, 1982, 1994) *Experimental Design: Procedures for the Behavioral Sciences*.
Brooks/Cole, Belmont, CA.

Newman, J. A., Bergelson, J. and Grafen, A. (1997)
Blocking factors and hypothesis tests in ecology: is your statistics text
wrong? *Ecology*, 78, 1312-20.

R
Development Core Team (2010). *R: A language and environment for statistical
computing*. R Foundation for Statistical Computing, Vienna, Austria. ISBN
3-900051-07-0, URL
http://www.R-project.org.

Searle, S. R. (1971, 1997) *Linear Models*. New York: John Wiley.

analysis of covariance: *see*
ANCOVA

analysis of variance: *see*
ANOVA

a priori contrasts: *see*
orthogonal contrasts

balanced non-orthogonal designs: *see*
balanced incomplete block *and*
Youden square

balanced complete block: *see*
randomized complete block

balanced incomplete block design

categorical factor: *see*
factor

continuous factor: *see*
covariate

contrasts: *see*
orthogonal contrasts

crossover trials: *see*
Latin squares

effect size: *see *
power

effective
sample size: *see* replication

error variance: *see*
residual error

fixed factor: *see*
factor

general linear model: *see*
GLM

generalized linear model: *see*
GLM

Helmert contrasts: *see*
orthogonal contrasts

independent replicates: *see*
replication

interaction: *see*
cross factor

linear model: *see*
GLM

main effect: *see *
model

mean square: *see*
*F*
ratio

missing cell: *see*
examples of
orthogonal contrasts

multiple regression: *see*
ANCOVA

null hypothesis: *see*
hypothesis

observation: *see*
replication

one-way designs: *see*
one-factor
designs

orthogonal unbalanced designs: *see*
orthogonal contrasts

planned
comparisons: *see*
orthogonal contrasts

random factor: *see*
factor

random replicates: *see*
replication

regression: *see*
ANCOVA

repeated-measures sampling units

sample: *see*
replication

sampling unit: *see*
replication

sum of squares: *see*
adjusted
SS *and*
sequential SS

test hypothesis: *see*
hypothesis

treatment: see factor

two-way designs: *see*
factorial designs
*and*
randomized-block designs

type-II error: *see* type-I error
*and* power

variance: *see*
residual error

variance
component: *see *
power

Page maintained by C. P. Doncaster

Last updated 24 March 2023