The Hawthorne Effect in Educational Research

In the 1920s at the Hawthorne Works factory outside Chicago, some engineers did a series of productivity experiments with the factory workers. Though they thought they were examining how lighting and other elements in the factory environment affected productivity, their conclusion was that change itself made workers more productive—even changing back to the original environment from before the experiment started. This is now known as the Hawthorne effect, and it came up in a conversation I had at a biology education conference. Someone there bemoaned how a new style of biology teaching was getting pushed as orthodoxy (reducing lectures, increasing active learning opportunities, etc.) without any good evidence that it was effective. When I pointed out that there were lots of studies showing improvements in student learning when implementing these changes, he invoked the Hawthorne effect and said that all of these education studies he had seen were flawed. None of them would avoid the Hawthorne effect.

The argument is that the professors implementing these new classes are putting a heavy investment of time and resources into making changes, and are excited about those changes. The improvement in student learning may come about just because the class changed and there is an energy around change that makes people do better. The logical conclusion to this line of thought is that if you suddenly came up with an "innovation" to put all the students in a big room with a blackboard in front and lectured at them, and this was new and different, you should get the same improvement.

Although this sounds to me at first like a handy excuse from professors who don't want to bother changing their teaching, it also got me thinking—it's hard to argue against and makes some intuitive sense. Yet it's also hard to see how you would do an educational study that avoided the effect. Medical studies use placebos for this reason, and parts of the federal government now want double-blind placebo-controlled educational studies like in medicine. But how do you do a blind study, or implement a placebo, in teaching? Surely the student will know what treatment they are receiving, and it won't take long for an interviewing experimenter to figure it out as well.

Our own studies don't suffer as badly from this argument as we are usually comparing activities without a person involved—a computer program versus a pencil and paper exercise, for instance. So there is no excited teacher energy to rub off on the students. But for classroom innovations, invoking the Hawthorne effect to dismiss all studies really leaves your hands tied. While there are ways around it (the person suggested waiting 5 years after implementing an innovation to measure gains; measuring gains in a second or third school that adopts rather than the originating school), they would take a huge effort to implement. Lots of studies have now shown student gains from adopting more active learning techniques. Some of these have pieces of evidence that are not just improved test results—I remember one talk where the professor showed how much more detailed the end-of-class evaluations were in the new class design than the old, with students writing good constructive feedback in complete paragraphs compared to one word answers in the past. And the Hawthorne effect itself may be overblown—a reanalysis of the original Hawthorne effect dataset showed that even in that factory where the idea originated, the effect was not very strong after all.

It seems hard to dismiss all this evidence of educational gain outright, Hawthorne effect or no. But even if you did, one conclusion is still valid—focusing on improving classes tends to improve student performance. Whether or not it's through the Hawthorne effect, students are still learning more than they were. So regardless of whether you believe a particular technique has been shown to work, a culture of focusing on education and education research will help your students learn.