January 2004 — Exclusive Series: SBR
Print this article | Email this articleClick here to receive your FREE subscription to T.H.E. Journal
Determining 'What Works' - An Interview With Dr. Grover 'Russ' Whitehurst
How a What Works Clearinghouse Evidence Report Is Created
The end products of the What Works Clearinghouse are evidence reports, which are designed, according to the WWC, "to provide education consumers with high-quality reviews of scientific evidence of the effectiveness of educational interventions." The WWC would like to see educators use evidence reports to make instructional decisions and education policy. The reports will be publicly available on the WWC Web site (http://w-w-c.org). The first evidence reports were due in fall 2003; however, the WWC decided to pilot test the first reports and extended the initial release date until early 2004. The process for producing an evidence report is described below.
1. A topic area is chosen by the WWC. A topic area is defined by the intended outcome (e.g., improving literacy skills), the intended population (e.g., free- or reduced-lunch elementary students) and the types of replicable interventions (e.g., a literacy-building product or program) that may produce the intended outcome for that population. Topic areas are chosen by considering the potential of interventions in the topic area to improve student outcomes, the perceived demand within the education community for evidence of effective educational interventions in the topic area, and the likely availability of high-quality scientific studies of effective educational interventions in the topic area. Examples of initial topic areas currently being reviewed include: Interventions for Beginning Reading, Curriculum-Based Interventions for Increasing K-12 Math Achievement, Programs for Preventing High School Dropout, and Programs for Increasing Adult Literacy.
2. An evidence team — consisting of a senior content and a methodology expert, as well as a project coordinator and research reviewers — creates a protocol that sets the parameters for the kinds of interventions they will look at within a given topic area. So, for instance, for the topic area of improving literacy, the protocol might define such parameters as grade-level scope, the extent of the intervention, a definition of literacy skills, and so forth.
3. Once a protocol has been defined, the evidence team begins a literature search for studies on interventions in the topic area that fits the parameters of the protocol. Publishers, organizations, educators and others can nominate studies of replicable interventions through the WWC Web site at http://w-w-c.org/topicnom.html. An initial sort eliminates those interventions that do not fit the protocol parameters. For example, if the protocol defines literacy skills as including decoding, a study of an intervention that d'es not address decoding would likely not be considered.
4. The remaining intervention studies are then submitted to a series of questions about the design of the intervention study, such as which methodology was used (e.g., random assignment, matched control group), the number of participants involved, outcome variables tested (e.g., attendance, test scores), and so forth. Studies are dropped from consideration if the examination (called a "DIAD") determines that their design d'es not meet WWC standards. In general, those studies that do not use an experimental or quasi-experimental methodology will not pass the DIAD examination. For instance, if a study of the effectiveness of a particular literacy intervention used pretests and posttests to determine effectiveness, that study would likely not be eligible for further consideration.
