Chapter 3: Producing Data

Section 3.2: Designing Experiments

Definitions:

• observational study: a study of individuals in which researcher observes individuals and measure variables of interest, but does not attempt to influence the responses
• experiment: a study of individuals in which researcher deliberately imposes some treatment on individuals in order to observe their responses
• experimental units: the individuals in the experiment
• subjects: human experimental units
• treatment: specific experiment condition applied to individuals
• factors: explanatory variables
• levels: specific values of factors

Advantages of experiments:

• can control environment (background) of units to hold lurking variables constant
  • can provide good evidence of causation
  • this ideal is not realized in practice
• can study combined effects of several factors simultaneously

Types of experiments:

Comparative experiments

• design: units --> treatment --> response
• typically rigorously controls environment
• confounding can still occur (see placebo effect in Example 3.11 on p. 189 of your text
• to prevent confounding: split individuals into two groups – experimental group (treatment) and control group (no treatment).
  • all lurking variables operate on both groups so 'any' difference is due to treatment

Randomized comparative designs

• design: see Figure 3.4 on p. 190 of text
• Randomized comparative experiments are designed to give good evidence that difference in the treatments actually cause the differences we see in the response
• randomly assigns individuals into two groups
• groups do not depend on characteristic of individuals or on judgment of researcher
• groups must be similar

Completely randomized designs – example

• simplest statistical design
• design: see Figure 3.5 on p. 191 of text
• randomly assigns individuals into three or more groups
• groups do not depend on characteristic of individuals or on judgment of researcher
• groups must be similar
• two treatment groups: goal is to compare the treatments
• one control group

The logic of randomized comparative experiments

Purpose is to obtain good evidence that differences in the treatments actually cause the differences in the response.

• random assignment of individuals forms groups that should be similar in all respects before treatment
• comparative design ensures that influences other than treatments operate equally on all groups
• Thus, differences in average responses are due either to treatments or chance (see section 4.1).

Key ideas about sample design:

• sample design is the method used to choose the sample from the population. Poor designs produce misleading conclusions.
• A design is biased if it systematically favors certain outcomes.
• Poor designs:

Principles of Experimental Design

• Control of the effects of lurking variables on the response by comparing several treatments
• Randomization: the use of chance to assign individuals to groups
• Replication of the experiment on many individuals to reduce chance variation on results

Statisitcal Significance: an observed effect so large that it would rarely occur by chance. Statisitcally significant results provide good evidence for cause–and–effect relationship between explanatory and response variables. In other words, there is good evidence that the treatment causes differences between the groups.

Generalizing experiments – applying the results of study to a wider setting.

Cautions about experimentation

Good experiments require careful attention to details.

Double-blind experiments are ones in which both individuals and researchers do not know who is receiving the treatment(s).

Lack of realism: the conditions of the study do not realistically duplicate the conditions that we are interested in. This can limit our ability to generalize the experiment.

Matched pair designs – example

This type of design compares just two treatments. To use a matched pair design, choose two individuals that are as similar as possible, then randomly assigns each of the individuals of the pair into different treatment groups.

A pair may also be the same individual who gets both treatments one after the other. In this case, the order of the treatments must be randomly presented to the individual.

Block designs – example

Design: see Figure 3.6 on p. 198 of text.

Matched pair designs are an example of block designs. A block is a group of individuals that are known before the experiment to be similar in some manner prior to treatments. In a block design, the random assignment of individuals to treatments is completed within each block.

This type of design removes systematic differences between the blocks. Blocks are used when there are unavoidable differences between parts of a population. Randomization averages out the effects of the differences and allows for unbiased comparison of the treatments.