Much ado has been made about the power of instructional online videos. Many see them as powerful supplements or even replacements to classroom learning. But how are those claims really being measured?
Two Ithaca College researchers are aiming to get beyond the anecdotes about “flipped classrooms,” where static learning through videos (or other means, including textbooks) takes place outside the classroom, which is reserved for more active and engaging activities based on the material.
Aaron Weinberg and Matt Thomas—who both teach math education in the Department of Mathematics—received a $185,985 grant from the National Science Foundation to gather real empirical evidence on how such videos can impact student learning.
“If you read the literature, it’s really kind of funny, because all of these studies say something like ‘We assigned videos, and then the students watched them, and then we did all this stuff [in the classroom].'“ Thomas said.
“It's the ‘and then the students watched them’ that made us both pause and say, ‘Wait, what actually happened?’” he added.
Specifically, the two will study how instructional videos aid students in their Calculus I classes. One of the purported benefits of videos is that viewers can go at their own pace through the material. “But we don't know if they're actually doing that,” Thomas said.
To find out, their videos will be embedded on a website that the professors are setting up. The site will document all students’ interactions with the videos: when and where they pause, how far they rewind, or how often they play straight through, for example.
The researchers will also experiment with how they structure the video-watching experience to see what affects it has on learning: They might ask a set of questions prior to each video or provide an outline of material, and compare those students’ experiences watching the video with those who were not provided such resources. All the students will provide feedback about their time with the video via an online form.
Weinberg said that a small subset of their students will instead watch the videos while being measured with an eye-tracking device. “So we're going to be able to see exactly what they're looking at—what they're watching, what they're paying attention to—as they watch these videos,” he said. These students will also undergo pre- and post-viewing interviews.
Crunching the Numbers
All that data and the student responses will—ideally—provide insight into what’s working or not working within the videos. The ultimate aim is to comb through the eye-tracking and video-click data and combine it with interview and written responses to create models of student thinking.
“We’ll also be looking at the ways students solve calculus problems that are based on the video content, so we’ll be connecting the ‘what they’re thinking’ data with the ‘what they can do’ data to try to see how the two are related, and then see how various ways of structuring the video-watching process influences both,” Weinberg said.
Thomas added: “The explanatory power of the project is really going to only come from combining all this information to give us a picture of how the students are interacting with the videos, and how they're making sense of and learning from them.”
Weinberg notes that’s really the missing piece in this concept of the flipped classroom.
“No one has done research on videos. We thought it seems like a neat idea when teaching,” he said. But as two mathematicians who are also interested in understanding how students learn, a neat idea is not enough. “We want to think carefully about teaching, but we want to think scientifically about teaching, too.”
Video Production
Weinberg and Thomas will create the videos in a “digital whiteboard” format using an iPad and stylus to draw and write over images or simple animations, and will provide their own voiceover narratives. They will develop the videos based on a learning trajectory they want their students to take throughout the calculus class so they pick up core concepts in a structured way.
Each video will run between 5 and 10 minutes, they said. In this first year of the study, they plan to create up to three videos for eight specific topics covered in the class. In the second year, they will expand on another seven or eight topics.
Weinberg and Thomas are collaborating with colleagues at Central Arkansas University and Oklahoma State, as well. All the researchers will make their own videos based on a shared script. The third year of research will expand the effort to an additional 15 institutions.
Thomas said the answers they find about what works or doesn’t with their videos will likely differ from what would work for other distinct fields of study, such as chemistry. “I think in some ways this just opens more doors to figure out how the answers here are going to be different from other subject areas,” he said.
With those doors open, educators won’t have to solely rely on anecdotal evidence about the usefulness of videos or other methods of outside-the-class learning, and can start to refine methods and styles with the weight of empirical evidence behind them, Weinberg said.
“That way when we come into class, we'll have some idea of ‘OK, you didn't take away the exact things we wanted you to take away, but here's what you did take away. Let's build on that.’”