Does Understanding Science Help You Learn Evolution?

We're starting a large study of a new EvoBeaker lab on the evidence used in support of evolution (more details here). So I was quite interested in a new paper on whether an active learning approach to teaching evolution increased students understanding or changed their beliefs (Cavallo and McCall, American Biology Teacher, 2008). What's interesting about this study is that the researchers looked for interactions between students' learning, their initial belief in evolution, and something the authors call Nature of Science (NOS) beliefs.
The basic thesis the authors wanted to test was whether students who had a better understanding of what science means would learn evolutionary theory better. By Nature of Science (NOS), the authors were looking for whether students understood science as a set of tentative understandings, or as absolute, unchanging knowledge. For instance, they looked for whether students thought that only correct theories move science forward, or that experiments are designed by scientists to prove their ideas as definitively right or wrong. Understanding science as a process is one of the goals of many newer state and national science standards, so its important in practice to see whether a curriculum can help do that. But the more interesting question in my mind is whether students that do understand science as a process actually learn science better as a result.
The study was among 3 high school biology classes presented a four week curriculum on evolution which included a variety of active and passive teaching tools. Unfortunately, the results from this study are inconclusive, but in an interesting way. It turns out that students improved their understanding of evolution (at least according to the metrics used in the study, which aren't shown). But the students with a better understanding of NOS did not improve any more than those with a worse understanding. Students also improved equally whether or not they believed in evolution. And neither understanding NOS nor belief in evolution changed at all during the study. The latter is not new—several studies have found that learning evolution and believing in it are pretty well disconnected from each other. But it's kind of discouraging if better understanding of scientific thinking doesn't help to learn science. The hint that there might be something more was that belief in evolution and understanding NOS were correlated—those students that had a better understanding of the nature of science also tended to accept that life had evolved. So it's possible that although a 4 week unit may not change student's ability to think scientifically very much, a longer and/or better course of study which does get students to understand science may also help them think more scientifically about their life and beliefs. That would be interesting—but of course there are many other possible explanations for that correlation.
So what does understanding how science works help with? The general argument is it gives one a way of thinking about and analyzing the world that is very powerful, and that's obviously true —science is very successful. But are there parts of learning science where that really doesn't matter? I'm skeptical in general, but it could be true for some topics. If true (and of course, we'd need a lot more evidence), that would be quite useful to know.
I have one caution in interpreting this study: it's hard to know if their tests are picking up real understanding. They used tests from various sources for looking at the 3 variables in the study, including a test by Settlage and Baldwin (1996) to measure student's understanding of evolution. The latter is one of the better tests out there for evolution, but when we looked at it we still weren't convinced that it was measuring deeper understanding as well as we wanted to. So we took a few questions from there but also wrote many of our own (in press). I'm not convinced our first test on natural selection is all that excellent, either. In fact, writing tests that get reliably at misconceptions and real understanding, especially of natural selection, is devilishly difficult. It's a real conundrum for education research—you need good tests to do evidence-based teaching, and we biology educators still don't have a good formula for making standardized, easily graded tests that measure real understanding.