April 2004 — Exclusive Series: SBR
Print this article | Email this articleClick here to receive your FREE subscription to T.H.E. Journal
How to Evaluate Educational Research
Questions for Evaluating Research Claims
1. D'es the research claim that a particular program or product results in student achievement, improved student achievement, or some other outcome such as teacher skill levels?
If yes: Were participants in the study randomly selected from the population to which the results will be generalized? For example, if the study claims that the intervention has a positive effect on fifth-grade students in high-poverty urban schools, were the students participating in the study randomly selected from a population of such students? Or, were participants in the study randomly assigned to the experimental versus the control/comparison groups? The Education Department places greater emphasis on random assignment than on random selection. In addition, was there a control or comparison group (i.e., a group that d'es not receive the intervention which is being studied for effectiveness)?
If you can answer "yes" to both of the above items, the study was an experiment. And if the study was rigorously conducted (i.e., used reliable and valid measures, and so forth), causal claims can be made, especially if the findings have been replicated.
If you answered "no" to the first item and "yes" to the second item, the researcher(s) have used a quasi-experimental design. If the research is quasi-experimental, ask the following question: Did the researchers take every possible precaution to ensure that the experimental and control/comparison groups were alike except for the experimental intervention? For example, were the teachers in all groups equally qualified, did the students have similar backgrounds, and so forth?
2. Whether the research was experimental, quasi-experimental or otherwise:
- Were the instruments and procedures used to measure results reliable? And, are the procedures described clearly enough that another researcher could replicate them?
- Is there enough information about the instruments that the reader can reasonably conclude that, if nothing changed in the situation, the instrument would yield the same measurement again?
- Were the instruments and procedures valid for the purpose of the study? Valid instruments and procedures measure what they purport to measure. For example, a test of mathematical reasoning should measure mathematical reasoning and not simply the ability to accurately compute.
Test manuals should include information about the reliability and validity of the test. Reliability and validity are expressed in terms of a correlation c'efficient (r). The closer "r" is to 1.00, which is perfect reliability or validity, the better; however, c'efficients above 0.7 are generally considered to be acceptable. Of course, if high-stakes decisions will be made on the basis of a particular measure, that measure should be highly reliable and valid.
3. Finally, for any study you evaluate, ask yourself:
- Did the research address your question?
- Are there other possible explanations for the results that are reported in the research study or how those results are interpreted by the authors?
- Is there more than one high-quality study to support the claims?
- Has the research been reviewed by expert researchers other than those who conducted the study? And, did the independent expert reviewers support the methods used and the conclusions drawn by the researchers?
Cite this Site
copy text (above) for proper citation
