Advice to Colleagues on Reliable Research

Dear Colleagues,

Often we are handed reports regarding some new finding or educational strategy that demands our attention. These sometimes include phrases such as, “research shows” or “studies have proved” along with the new claim. But how can we evaluate these sources and the validity of their assertions?

First of all, any study that claims to have a definitive, absolute answer or touts an approach as the “best” should raise red flags. As you have likely experienced, there is simply no such thing as an absolute or perfect strategy. Our classrooms are as varied as the students that enter them each year and the approaches that work best with one student or one class may be fruitless with others. If you are doubting that something is the definitive solution/approach/etc. you are right in doing so. Also, this may alert you that whoever generated the study may have had an ulterior motive.

The notion of ulterior motives brings us to two crucial questions: 1) who published this research? and 2) who funded this research? Brian L. Carpenter, Executive Director of the National Charter Schools Institute calls these a “skeptics first two questions”. The idea is to consider whether the study was objective and why the study was done (and published) in the first place. For example, research about how interactive whiteboards affect learning outcomes is great, but if it’s funded by the SMART Board company or published by Promethean (who makes ActivBoards), the objectivity comes into question.

While you may feel odd acting as a skeptic and putting material through filters, trust that you are acting as an informed consumer. As Carpenter explains, rather than taking everything at face value, you are placing the burden of proof on the researcher, where it belongs. The following are useful principles to help determine if a work is indeed scientific.

The Six Guiding Principles (Shavelson & Towne, 2002, in Carpenter 2006)
1) Does it pose significant questions that can be investigated empirically?
2) Does it link research to relevant theory?
3) Does it use methods that permit direct investigation of the question?
4) Does it provide a coherent, explicit chain of reasoning?
5) Could results be replicated and generalized across studies?
6) Disclose research to encourage professional scrutiny and critque?

Keeping these questions in mind when reviewing studies or results will not only make you more informed as a consumer, but also better prepared as a beneficiary of solid educational research.

Feel free to come to me with any additional questions or items to discuss.

Yours in Reliable Research,
W. Davis