Examples of Analysis of Variance and Covariance

C. P. Doncaster and A. J. H. Davey

This page presents example datasets and outputs for analysis of variance (ANOVA) and covariance (ANCOVA), and computer programs for planning data collection designs and estimating power. All of the statistical models are detailed in Doncaster and Davey (2007), with pictorial representation of the designs and options for troubleshooting common issues with analysis.

Click here for the suite of commands in R (freeware statistical package, R Development Core Team 2010) that will analyze each of the example datasets below, and calculate the power of the design.

## What is a statistical model?

Any statistical test of pattern requires a model against which to test the null hypothesis of no pattern. Models for ANOVA and ANCOVA take the form: Response = Factor(s) + ε, where the response refers to the data that require explaining, the factor or factors are the putative explanatory variables contributing to the observed pattern of variation in the response, and ε is the residual variation in the response left unexplained by the factor(s). For each of the examples illustrated here, we use a standard notation to describe the full model and its testable terms. For example, the two-factor nested model in Section 2 below is described by:

(i) The full model, packed up into a single expression: Y = B(A) + ε;

(ii) Its testable terms to declare in a statistics package, unpacked from the full model: A + B(A).

A statistics package will require you to specify the model desired for a given dataset. You will need to declare which column contains the response variable Y, which column(s) contain the explanatory variable(s) to be tested, any nesting or cross factoring of the explanatory variables, whether any of the variables are random rather than fixed factors, and whether any are covariates of the response.

## Examples of ANOVA and ANCOVA models

Each of the links in Sections 1 to 7 below shows a full suite of analyses of a hypothetical dataset. Where appropriate, these include alternative restricted and unrestricted models (Searle 1971), and Model-1 and Model-2 designs (Newman et al. 1997). Refer to the protocols in Doncaster and Davey (2007) to see which mean squares are used for the F-ratio denominators, and consequently how many error degrees of freedom are available for testing significance. The examples have not used post-hoc pooling though this may be an option or an alternative to some quasi F-ratios, and the underlying assumptions have not been evaluated though this would need doing for real datasets.

A statistics package may give different F ratios and P values to those shown here, for a variety of reasons:

- In unbalanced and non-orthogonal designs and ANCOVA models, default use of Type-III adjusted SS for models that require Type II.

- In balanced mixed models and ANCOVA models, default use of an unrestricted model when the design may suite a restricted model.

- In designs with randomized blocks and split plots, default use of Model-1 analysis when the test hypothesis may require Model 2.

- In ANCOVA mixed models, default use of residual error for all denominator mean squares when the test hypothesis may have a different error variance.

Click here for the suite of commands that will analyze each of these example datasets in the freeware statistical package R (R Development Core Team 2010).

 Analysis of differences between sample means of one categorical factor (including orthogonal contrasts), or trend with one covariate Analysis of two or more factors in a replicated hierarchy with levels of each nested in (belonging to) levels of the next Analysis of crossed combinations of factor levels randomly assigned to sampling units in replicated samples (including orthogonal contrasts), and/or of trends with cross-factored covariates Planned orthogonal contrasts for levels of factors B and/or A, and contrasts for two-factor analysis missing one combination of levels Analysis of one or more categorical factors with levels, or combinations of levels, randomly assigned in blocked sampling units of plots within blocks, and replicated only across blocks (including orthogonal contrasts, and balanced incomplete block, Latin squares, and Youden square variants on the one-factor complete-block design) Latin square variant Y = C|B|A with replicate in blocks and stacked squares for crossover designs. Click here to download a computer program (LatinSquare.exe) that allocates treatment levels at random. Analysis of two or more categorical cross factors with levels randomly assigned in split-plot sampling units of sub-sub-plots nested in sub-plots and/or sub-plots nested in plots and/or plots nested in blocks and replicated only across levels of the nesting (repeated-measures) factor(s) Analysis of one or more categorical factors with levels, or combinations of levels, assigned in repeated-measures sampling units of subjects repeatedly tested in a temporal or spatial sequence, and replicated only across subjects Analysis of fully randomized factorial (crossed) combinations of factor levels without replication Figures and worked examples in Doncaster and Davey (2007) Analyses of illustrations to sections introducing analysis of variance and model structures, and general linear models for unbalanced designs

## Computer programs for planning designs and estimating design power

1.    List all terms and degrees of freedom in any model for analysis of variance or covariance. Click here to download a computer program (Terms.exe) that will list all of the main effects and interactions and their degrees of freedom for a model of your own specification with any number of levels for each of any number of cross-factored or nested categorical or continuous variables (yielding up to a maximum of 2047 terms).

Specify the model as a hierarchical nesting of sampling units in factors, representing each variable by a single letter. Thus for example, requesting: 'P(B|S(A))' will yield testable terms for any of models 3.3, 5.6 or 6.3 above, depending on the nature of the variables and the replication of sampling unit P.

A text file 'Terms.txt' will be created to store the list of terms comprising all main effects and interactions and their degrees of freedom; it will also contain the N lines of factor-level combinations against which to tag your N observations of the response.

2.    List testable terms, degrees of freedom, and critical F-values for any of the numbered designs above. Click here to download a computer program (CritiF.exe) that allows you to specify your own sample sizes in the numbered ANOVA and ANCOVA designs above, assuming fixed treatment factors. For each estimable effect, it shows the test and error degrees of freedom, and the critical F at α = 0.05. For fixed effects, it shows the standardized effect size with 80% detection probability (the value of θ /σ that gives the test a power of 0.8).

Use the program to evaluate alternative experimental designs for a given workload of data points, targeting a low standardized effect size for treatments. This value will vary according to the distribution of data points between levels of sampling units and treatments. For a given total data points, it will be increased by the inclusion of nesting, covariates, blocking, split plots, or repeated measures. These may be desirable or intrinsic features of the experimental design, and they will increase power to detect treatment effects if they reduce error variances sufficiently to compensate for the reduction in error degrees of freedom.

A text file 'Factor_levels.txt' will be created to store the N lines of factor-level combinations against which to tag your N observations of the response.

3.    Calculate statistical power for any balanced model. Click here to download a computer program (Power.exe) that calculates statistical power prospectively for fixed or random factors in any balanced model with a proposed size of samples, given a threshold ratio of treatment effect size, θ (the standard deviation of the treatment variability) to error effect size, σ (the standard deviation of the random unmeasured variation). It can also calculate the value of θ /σ required to achieve a target power.

A pilot study may be needed to obtain an initial observed F from samples of size n. Then [(F - 1)/n]1/2 will provide an unbiased estimate of the population θ /σ, with which to evaluate the potential to gain power from more replication (e.g., Kirk 1968). Freeware is available elsewhere on the web to further explore the relationships between n, θ, σ and power for specified designs (e.g., Piface by Russell V. Lenth).

4.    Calculate relative performance for any balanced model. Click to download a computer program (Performance.exe) that calculates the performance of a balanced analysis of variance design relative to a reference design for the same treatment(s). The relative performance of the design is given by the fractional size of its error variance that will just match the power of the reference. The value of relative performance is robustly approximated by the ratio of reference to alternative α quantiles of the F distribution, multiplied by the ratio of alternative to reference effective sample sizes (Doncaster, Davey & Dixon 2014). By comparing the precision of two designs at equal sensitivity, relative performance provides a useful way to enumerate trade-offs between error variance and error degrees of freedom when considering whether to block random variation or to sample from a more or less restricted domain.

5.    Find critical F-values for any number of test and error degrees of freedom, and value of α. Click here to download a computer program (Ftable.exe) that provides critical F-values for a chosen α, and also gives the Type-I error probability associated with an observed value of F, given test and error degrees of freedom.

Program citations

Doncaster, C. P. (2007) Computer software for design of analysis of variance and covariance. Retrieved [date] from http://www.southampton.ac.uk/~cpd/anovas/.

## Key to types of statistical models

Use this key to identify the appropriate section of model structures above, then look at example datasets and analyses.

1.       Can you take observations with independently varying residuals that randomly sample from the populations of interest (i.e., from the levels of each factor or factor combination)?

Yes → 2.

No → identify the dependency structure and explore options to control it in your model, for example by factoring in nuisance variables or subpopulations or by sub-sampling; otherwise the data may not suit statistical analysis of any sort.

2.       Are you interested either in differences between sample averages or in relationships between covariates?

Yes → 3.

No → the data may not suit ANOVA or ANCOVA.

3.       Does one or more of your explanatory factors vary on a continuous scale (e.g., distance, temperature etc) as opposed to a categorical scale (e.g., taxon, sex etc)?

Yes → consider treating the continuous factor as a covariate and using ANCOVA designs in Sections 1 to 3 above; this will be the only option if each sampling unit takes a unique value of the factor. The response and/or covariate may require transformation to meet the assumption of linearity. Analyze with a General Linear Model (GLM) and for non-orthogonal designs consider using Type II adjusted SS if cross factors are fixed, or Type III adjusted SS if one or more cross factors are random (and an unrestricted model, checking correct identification of the denominator MS to the covariate). Be aware that adjusted SS can increase or decrease the power to detect main effects.

No → 4.

4.       Can all factor levels be randomly assigned to sampling units without stratifying any crossed factors and without taking repeated measures on plots or subjects?

Yes → 5.

No → 9.

5.       Are all combinations of factor levels fully replicated?

Yes → 6.

No → use an unreplicated design (Section 7 above).

Fully randomized and fully replicated designs

6.       Do your samples represent the levels of more than one explanatory factor?

Yes → 7.

No → use a one-factor design (Section 1 above), considering options for orthogonal contrasts (e.g., model 1.1 above).

7.       Is each level of one factor present in each level of another?

Yes → 8.

No → use a nested design with each level of one factor present in only one level of another (Section 2 above).

8.       Use a fully replicated factorial design (Section 3 above), taking account of any nesting within the cross factors (model 3.3 or model 3.4 above). For balanced and orthogonal designs with one or more random cross factors, consider using a restricted model, and consider post hoc pooling if an effect has no exact F-test. If cross factors are not orthogonal (e.g., sample sizes are not balanced), use GLM and consider using Type II adjusted SS if cross factors are fixed, or Type III adjusted SS if one or more cross factors are random (and an unrestricted model).

Stratified random designs

9.       Are sampling units grouped spatially or temporally and all treatment combinations randomly assigned to units within each group?

Yes → use a design with randomized blocks (Section 4 above). If all factor combinations are fully replicated, analyze with Section-3 ANOVA tables; otherwise consider analysis by Model 1 (assumes treatment-by-block interactions) or Model 2 (assumes no treatment-by-block interactions). For a single treatment factor, consider options to use a balanced incomplete block or, with cross factored blocks, a Latin square or Youden square.

No → 10.

10.   Are different treatments applied at different spatial scales and their levels randomly assigned to blocks or to plots within blocks, etc?

Yes → use a design with split plots (Section 5 above), taking account of nesting among sampling units.

No → use a repeated-measures design (Section 6 above) for repeated measurement of each sampling unit at treatment levels applied in a temporal or spatial sequence. If all factor combinations are fully replicated, analyze with Section-3 ANOVA tables.

## References

Doncaster, C. P. and Davey, A. J. H. (2007) Analysis of Variance and Covariance: How to Choose and Construct Models for the Life Sciences. Cambridge University Press, Cambridge 302 pp. ISBN-13: 9780521684477. https://doi.org/10.1017/CBO978051161137

Doncaster, C. P., Davey, A. J. H. & Dixon, P. M. (2014) Prospective evaluation of designs for analysis of variance without knowledge of effect sizes. Environmental and Ecological Statistics, 21: 239-261

Kirk, R. E. (1968, 1982, 1994) Experimental Design: Procedures for the Behavioral Sciences. Brooks/Cole, Belmont, CA.

Newman, J. A., Bergelson, J. and Grafen, A. (1997) Blocking factors and hypothesis tests in ecology: is your statistics text wrong? Ecology, 78, 1312-20.

R Development Core Team (2010). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.

Searle, S. R. (1971, 1997) Linear Models. New York: John Wiley.

## Glossary

analysis of covariance: see ANCOVA

analysis of variance: see ANOVA

ANCOVA

ANOVA

a priori contrasts: see orthogonal contrasts

assumptions

balance

balanced non-orthogonal designs: see balanced incomplete block and Youden square

balanced complete block: see randomized complete block

balanced incomplete block design

block

blocked designs

blocked sampling units

categorical factor: see factor

continuous factor: see covariate

contrasts: see orthogonal contrasts

critical F

cross factor

crossover trials: see Latin squares

degrees of freedom

effect: see model and terms

effect size: see power

effective sample size: see replication

error variance: see residual error

F ratio

factor

factorial designs

fixed factor: see factor

general linear model: see GLM

generalized linear model: see GLM

GLM

Helmert contrasts: see orthogonal contrasts

hypothesis

independent replicates: see replication

interaction: see cross factor

Latin square

linear model: see GLM

main effect: see model

mean square: see F ratio

missing cell: see examples of orthogonal contrasts

mixed model

Model 1

Model 2

model

multiple regression: see ANCOVA

nested designs

nesting

notation

null hypothesis: see hypothesis

observation: see replication

one-factor designs

one-way designs: see one-factor designs

orthogonal contrasts

orthogonality

orthogonal unbalanced designs: see orthogonal contrasts

planned comparisons: see orthogonal contrasts

pooling

power

quasi F

random factor: see factor

randomized-block designs

randomized complete-block

random replicates: see replication

regression: see ANCOVA

repeated-measures designs

repeated-measures sampling units

replication

residual error

response

restricted model

sample: see replication

sampling unit: see replication

sequential SS

significance

split-plot designs

split-plot sampling units

sum of squares: see adjusted SS and sequential SS

test hypothesis: see hypothesis

treatment: see factor

two-way designs: see factorial designs and randomized-block designs

type-I error

type-II error: see type-I error and power

unreplicated designs

unrestricted model

variance: see residual error

variance component: see power

Youden square

Page maintained by C. P. Doncaster

Last updated 24 March 2023