Optimal Experimental Designs: Squeezing Every Ounce of Information from an Experiment Event

Accurate and efficient measurement is at the core of empirical scientific research. To ensure measurement is optimal, and thereby maximize inference, there has been a recent surge of interest among researchers in the design of experiments that lead to rapid accumulation of information about the phenomenon under study with the fewest possible measurements. Statisticians have contributed to this area by introducing methods of optimizing experimental design (OED), which is related to active learning in machine learning and to computerized adaptive testing in psychometrics. The methodology involves adapting the experimental design in real time as the experiment progresses. Specifically, in OED, an experiment is run as a sequence of stages, or mini-experiments, in which the values of design variables (e.g., stimulus properties, task parameters, testing schedule) for the next stage are chosen based on the information (e.g., responses) gathered at earlier stages, so as to be maximally informative about the question of interest (i.e., the goal of the experiment). OED has become increasing popular in recent years, largely due to the advent of fast computing, which has made it possible to solve more complex optimization problems, and as such is starting to reach everyday experimental scientists. In our lab we have developed and applied our own versions of OED. In this talk I will demonstrate the application of OED in the areas of retention memory and risky choice.