February 2004 — Exclusive Series: SBR
Print this article | Email this articleClick here to receive your FREE subscription to T.H.E. Journal
What the Words Mean
Objective
Objective research is independent research conducted by a credible third party using rigorous and systematic methodology. Alternatives include research conducted by a company itself that includes 1) up-front design consultation that builds in safeguards against potential bias, and 2) follow-up, third-party review or audit. For research to be objective, two equally competent researchers, given the same data, must arrive independently at the same conclusions.
Reliable
Reliability is about consistency. The more reliable the research instruments and the more reliably the research procedures are applied, the greater the probability that:
- The same researcher can repeat the study with the same or highly similar results.
- Another researcher can replicate the study's methodology and obtain the same or similar results.
Reliability and its cousin, replicability, are important because one experiment or study by itself d'es not constitute an irrefutable body of evidence.
Valid
While reliability concerns consistency, validity is about accuracy. Imagine a target. Reliable marksmen would consistently hit the same spot on the target, but this spot might not be the bull's-eye. A marksman who shoots with validity would hit the center of the target. A reliable and valid marksman would hit the center of the target consistently. In research, we must be concerned with the validity of research instruments, procedures and findings. And validity must always be interpreted in terms of purpose. For example, if a test is designed to measure the extent to which a student has mastered a particular curriculum but the results are used to determine how well the student compares to students who may have been taught other curricula, then the findings of the comparison are not valid. The strength of a research design for meeting these criteria revolves around the extent to which the study rules out "alternative" explanations of the study's outcomes (i.e., explanations other than the treatment intervention).
Experimental and Quasi-Experimental Designs
Neither NCLB nor other credible documents such as the National Research Council report "Scientific Research in Education" (2002) claim that experiments and quasi-experiments are the only approaches to scientifically based research. The point of NCLB's emphasis on experimental and quasi-experimental designs is that these are the specific designs that can demonstrate cause-and-effect relationships. Clearly, other designs are appropriate, or even better suited, to addressing certain research questions. Which design, or combination of designs, is most appropriate for addressing a particular issue will depend on a number of factors, including the goals of the research, the extent to which the study is part of a larger body of research, and the claims that the researchers wish to be able to make on the basis of the research data.
The hallmark of both experimental and quasi-experimental designs is that they employ experimental and comparison groups. An experimental group is a group in a research study that receives the "treatment" or intervention. In order to determine whether or not there is an experimental effect, experiments and quasi-experiments require a basis for comparison. Sometimes a control group is used (i.e., a group that d'es not receive an experimental treatment). In medical trials, the control group d'es not receive the treatment drug. In education, a control group usually consists of a group of students taught by a traditional method rather than by the "experimental" method.
