Dr Wei LiuMultiple testing refers to any situation in which a collection of statistical hypotheses is evaluated. This situation applies to virtually all data analyses; it is most unusual that only one hypothesis will be tested on a data set. A data set is often analysed from every possible angle, and myriads of hypotheses are therefore tested.
Everyday one reads an article in a newspaper or magazine indicating some food, chemical, drug or human practice is dangerous to health. How can general health be so good when so many things are unhealthy? It has been argued that much of the reporting is fear mongering, based on faulty methods and misuse of statistics. For example, hundreds of possible effects are tested, and only the few results which are 'statistically significant' are reported as dire. With a large number of hypotheses tested, false significances are bound to happen!
The problem of multiplicity is gaining increasing recognition, and research in the area is proliferating. The major challenge is to devise methods that control some kind of overall type I error rate while retaining reasonable power for tests of individual hypotheses.