February 2001 — Features
Print this articleClick here to receive your FREE subscription to T.H.E. Journal
Assessing the Impact of Instructional Technology on Student Achievement
The evaluation team extended the Sternberg Developing Expertise model to define "expertise" as student achievement measured by teacher-created rubrics. Participating teachers developed, refined, and benchmarked rubrics for student-created products over the past three years. Teachers also selected a rubric for measuring student learning processes from Marzano et al.'s (1993) Dimensions of Learning model. The rubric addressed the depth and richness of revisions to student-created products and performances.X@XOpenTag001The Survey
X@XCloseTag001Using mixed methods that consisted of the online survey, student pretest and posttest surveys, and scores on teacher-created/selected rubrics that assessed students' learning processes and final products, the evaluation team used structural equation modeling to correlate the various elements of the extended Sternberg model. The hypothesis was that motivation would drive metacognition, and that metacognition would drive thinking and learning processes (specifically, inquiry learning and application of skills; two scales derived from WEB Project-based activities). Thus, increases in thinking and learning processes would result in increases in teacher-scored measures of student achievement. The student survey was pilot-tested in spring 1999, and three derived scales (metacognition, inquiry learning, and application of skills) had high internal consistency (alpha = .72 to .84). Two ten-item sets of questions for "in this class" and "in school in general" motivation were added to the survey in spring 2000.
In January 2000, the survey was administered to 165 students in nine cooperating schools. One-hundred and thirty-seven responses were from students who had not yet been exposed to the intervention, and could therefore be used as pretests. Internal consistency and reliability for all scales (class motivation, school motivation, metacognition, inquiry learning, and application of skills) ranged from alpha = .70 to alpha = .87. In May 2000, at the end of the spring term, the survey was re-administered as a posttest to the same group of students. As of August 2000, 131 completed surveys were returned by all nine schools. About 75% of the students who responded were from high schools, and 25% were from middle schools. Gender was about equally distributed.
Seventy-six valid data sets were matched in order to conduct a true repeated measures methodology (pretest vs. posttest). Only the "application of skills" scale increased during the spring term (2-tailed significance = .0165).
For the path analysis, the posttest survey results were correlated with teacher assessments. Participating teachers assigned a "product" score of "0" (no evidence), "1" (approaches standards), "2" (meets standards), and "3" (exceeds standards) to their students' final products. Products were re-scored by a jury of experts to increase reliability, resulting in 91 reported "product" scores. One-hundred and seven teachers assigned a "process" score of "1" (low) to "4" (high) to each of their participating students for the quality and depth of revisions of their final products, which they construed as a measure of student learning processes. These data constituted two independent measures of student achievement, which served to complete the model.
Four separate simplified path analysis models were tested. The first pair addressed process and product outcomes for class motivation, and the second pair addressed school motivation. The statistically significant (p < .05) results were as follows:
