It can be used for a variety of purposes, for example, to check the accuracy and completeness of content, or to test whether instructional strategies match learners’ needs or their levels of prior understanding. It can involve conversations, surveys, checklists, observations, and many other types of instruments and tools.
Formative evaluation
Formative evaluation or formative assessment is a process that teachers and designers use to improve the effectiveness and efficiency of instruction, primarily before that instruction is delivered.
Introduction
Publicity photo of the Marx Brothers in 1946.
In the early 20th century, a group called the Marx Brothers produced a series of film comedies. The first of these, like Duck Soup (1933) and Night at the Opera (1935) were critically acclaimed and very successful. Later movies, like A Night in Casablanca (1946) and Love Happy (1949), not so much. What was the difference? The early films incorporated the vaudeville routines that the brothers had performed hundreds of times, which allowed them to revise and perfect the timing before filming. The later films were just written and filmed.
Explanation and Application
Formative evaluation is often compared with summative evaluation (e.g., Eberly Center) with an analogy: “When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative” (Stake, 2004, p. 17). As this implies, formative evaluation typically occurs early and during the design process while summative occurs at the end. However, the more important distinction is not when it happens, but for whom it is done. Formative evaluation is done by the designer (the cook) for their own benefit, to improve a lesson or course before or while it’s being delivered. Summative is done for someone else (the guest), like an accreditor checking on the value of an existing curriculum. Some also refer to a follow-up or “confirmatory” evaluation to check if quality is being maintained (when the customer returns to the restaurant and compares the meal to their memory of it).
Stage vs time & cost
When time and budget allow, for example, when thousands of learners will be served, formative evaluation is conducted in a series of stages. First, the content is checked for accuracy and completeness by a subject matter expert. Second, the designer sits down with an individual(s) who may or may not be one of the intended learners. Here the intent is to discover major problems that need to be addressed before investing more in testing. They can simply talk through the lesson and see what is revealed. This is called the one-to-one stage. Third, a small number of people are asked to complete the instruction without the designer’s interference. This gets closer to how the instruction will be delivered and leads to more detailed refinement. It’s called the small group stage, but it just means a larger number, not that they need to test the instruction together. Finally, in the field trial stage, the instruction is tested in as close a manner as possible to its intended implementation—same context and timing, with a subset of the actual learner population, and so on. Here final revisions are made. As this implies, the evaluation is designed to itself be cost effective: larger errors are corrected before greater expenditure is made (see figure).Of course, not all formative evaluations can include all these stages. When time is short, the budget is low, and/or the instruction is for a limited number of learners, just one or two stages might be used, for example, a check with an SME followed by a small-group trial.
Examples
For the faculty Summer Institute in 2020, the CFE and partners created a series of asynchronous online learning modules. A group of volunteer faculty members worked through early versions of the modules and provided valuable feedback to the designers before the Institute began.
Other examples of formative evaluation include checking in with students during a course to see how things could be improved and involving past and future students as members of a course design team. The former can be accomplished with many different types of instruments. For example, a simple anonymous fill-in-the-blank form can work well:
Dear Professor, I think that the course is going ______ and/but I wish that _____. The part I like best is ____, while the part I like least is _______. Something else I think you should know is ____.
Tools
Tools
There are many instruments and interpretive tools that can be used in formative evaluation. One designed by Kirkpatrick points out that different instruments and questions are appropriate for four different levels of evaluation: reaction:
- Reaction – Is instruction favorable, engaging, relevant?
- Learning – Do learners acquire intended knowledge, skills, attitude, confidence, commitment?
- Behavior – How do learners apply what they learn?
- Results – To what degree are outcomes achieved?
Some add a fifth level to Kirkpatrick’s scheme: 5. Societal Impact.
Tip
In the mid-20th century, before personal computers and the beginning of computer-based instruction, an approach was developed called “programmed instruction.” One of the pioneers of programmed instruction named Susan Markle developed a principle called “lean programming.” What this meant was drafting instruction with as little text as possible. Testing, specifically the questions that participants asked, would then reveal where details needed to be added. In contrast, if the initial draft were rich in detail, it would be very difficult for the designer to identify what to cut. “Lean programming” continues to be a great principle to follow.
Resources
Eberly Center (nd). What is the difference between formative and summative assessment?
Edutopia (2019). 7 smart, fast ways to do formative assessment.
Kirkpatrick Partners (nd). The New World Kirkpatrick Model.
Virginia Tech. Lesson 10. Formative Evaluation
Stake, R. E. (2004). Standards-based and responsive evaluation. Thousand Oaks, CA: Sage.
Credit
Center for Faculty Excellence, Ithaca College