February 2001 — Features
Print this articleClick here to receive your FREE subscription to T.H.E. Journal
Assessing the Impact of Instructional Technology on Student Achievement
This was done via a virtual learning environment called The WEB Exchange, which resided on The WEB Project's server. Through threaded design conversations, students requested feedback on their works of art, music, and multimedia. They then filtered the feedback they received and used it to improve their final artistic products, which many of the participating students then posted on The WEB Exchange.Concurrently, language arts students, aided by language arts teachers and mentors from the Vermont Center for the Book, discussed curriculum-related texts. Moderating their own discussions, students engaged in deep, rich dialogue that focused on standards-based activities such as responding to text, substantiating arguments with evidence found in the text, informed decision-making, etc. These interventions were stable over the last two to three years of the five-year grant.
X@XOpenTag000Measuring the Project's Impact
X@XCloseTag000One of the research questions posed by the RMC Research Corporation evaluation team in evaluating this project was, "What is the impact of The WEB Project on student achievement?" Our intent was to generalize our methodology to other instructional technology grants in which student achievement must be reported. Findings from an online survey of The WEB Project teachers and administrators, repeated in spring 1998, 1999, and 2000, indicated that a connection between student motivation, metacognition, and learning processes, as outlined in a conceptual model developed by Sternberg (1998), might be applicable.
According to Sternberg, motivation drives metacognition, which, in turn, stimulates the development of thinking and learning skills. Thinking and learning skill development further stimulates metacognition, resulting in the development of expertise. The evaluation team extended the Sternberg Developing Expertise model to define "expertise" as student achievement measured by teacher-created rubrics. Participating teachers developed, refined, and benchmarked rubrics for student-created products over the past three years. Teachers also selected a rubric for measuring student learning processes from Marzano et al.'s (1993) Dimensions of Learning model. The rubric addressed the depth and richness of revisions to student-created products and performances.
X@XOpenTag001The Survey
X@XCloseTag001Using mixed methods that consisted of the online survey, student pretest and posttest surveys, and scores on teacher-created/selected rubrics that assessed students' learning processes and final products, the evaluation team used structural equation modeling to correlate the various elements of the extended Sternberg model. The hypothesis was that motivation would drive metacognition, and that metacognition would drive thinking and learning processes (specifically, inquiry learning and application of skills; two scales derived from WEB Project-based activities). Thus, increases in thinking and learning processes would result in increases in teacher-scored measures of student achievement. The student survey was pilot-tested in spring 1999, and three derived scales (metacognition, inquiry learning, and application of skills) had high internal consistency (alpha = .72 to .84). Two ten-item sets of questions for "in this class" and "in school in general" motivation were added to the survey in spring 2000.
