Evidence Framework for Innovation and Excellence in Education

Challenge: Provide best learning approach for each student

Stakeholders: Researchers, Developers, Teachers, Students

Tech Affordances: Rapid Prototyping

Methods: A/B Testing; Learning Analytics

Author: Barbara Means based on an interview with Ken Koedinger and references below.

The Challenge: Researchers at LearnLab, the Pittsburgh Science of Learning Center, set out to see whether they could improve the effectiveness of an online chemistry course by applying design principles based on research. Within a chemistry course developed by Carnegie Mellon University’s Open Learning Initiative (OLI), they identified the portion that deals with equilibrium as being particularly challenging for students.

Researchers hypothesized that they could help students better understand equilibrium by applying the “multimedia principle,” which states that material is learned better when presented in more than one sensory medium (for example, both pictorial representations and auditory text). The multimedia principle has been demonstrated in dozens of carefully controlled studies performed in university laboratories (Mayer, 2003), but these studies did not specifically address equilibrium in chemistry and did not necessarily involve participants that are representative. A number of classroom studies of the effects of adding diagrams to instruction on chemical equilibrium had shown generally favorable outcomes, but these studies had failed to carefully control the content in the experimental and control conditions, making it unclear whether or not the addition of diagrams was responsible for observed differences in student performance.

The Approach: The LearnLab researchers set out to test in a more controlled way whether the multimedia principle held true for teaching chemical equilibrium. They developed molecular-level diagrams of equilibrium stages and added these visual representations to the equilibrium module in the online course. Davenport, Klahr and Koedinger designed a random-assignment experiment in which the treatment group received the equilibrium content in multiple modalities—supplementing the existing text-based content with the new diagrams—while the control group received only the text, not the diagrams. They used learning analytics to compare how the two groups of students performed on conceptual multiple-choice questions and open-ended equilibrium problems embedded in the OLI Chemistry course. There was no difference between the two groups’ performance on these embedded assessments, suggesting that the difficulty students were having with the chemical equilibrium content was not one of visualizing the process.

To better understand what students were having trouble with, the LearnLab researchers conducted a study with undergraduates and chemistry graduate students acting as content experts. They asked the students to explain their thinking out loud as they worked on the equilibrium problems in the OLI course. This qualitative work found that many students did not fully understand the temporal aspects of chemical change over time. They thought of chemistry equations as machines running from left to right, and did not appreciate the fact that the same variable would have different values before, during, and after the achievement of chemical equilibrium. This new understanding of typical student thinking helped to inform a more targeted application of the multimedia principle.

Traditional chemistry lectures explain equilibrium using text and equations. David Yaron, the chemistry professor working with the learning researchers to redesign the equilibrium instruction, developed new course content integrating molecular diagrams into the explanation of equilibrium. A new A/B experiment contrasted this revised equilibrium module with the prior version of the module.

Because OLI Chemistry is an online course with embedded assessment activities for each content module, the researchers were able to rapidly test hypothesized variations of the equilibrium module within existing course structures. The first experiment was started in February 2006 at the University of British Columbia and replicated at Carnegie-Mellon. Both experiments were completed within a few months. After the lab-based qualitative research, the module was redesigned and the experiment on the new course content was run from March through June of 2007.

The Results: The revised diagram-based version of the equilibrium module tended to produce better learning for students generally and had a significant positive effect for students who had been in the lowest 25% of the class (Davenport, Yaron, Klahr, & Koedinger, 2008). The data from all three of these experiments has been placed in LearnLab’s DataShop, and can be accessed by external researchers with permission.

Value of this Approach:
For researchers, it used the affordances of the technology (rapid prototyping, embedded assessment) to test multiple implementations of a learning science principle (the multimedia principle), allowing them to better understand the conditions under which the principle does and does not operate.

For developers the same rapid testing and embedded learning analytics forwarded their mission of producing more effective courseware and helping students who otherwise struggle master difficult course content.

For teachers and students, this rapid experimental process resulted in more effective instructional affordances to help students grasp a particularly difficult concept in chemical equilibrium.

References

Davenport, J. L., Yaron, D., Klahr, D., & Koedinger, K. (2008). When do diagrams enhance learning? A framework for designing relevant representations. 2008 International Conference of the Learning Sciences.

Mayer, R. E. (2003). The promise of multimedia learning: using the same instructional design methods across different media. Learning and Instruction, 13(2), 125-139.
Pittsburgh Science of Learning Center. Visual Representations in Science Learning. LearnLab Research Wiki.