Embracing Good Science, A Lesson from SREE
A few weeks ago, I was at the spring meeting of the Society for Research in Educational Effectiveness. One of the talks that caught my attention was the evaluation of the Tennessee Voluntary PreKindergarten program by Mark Lipsey, Dale Farran and Kerry Hofer from Vanderbilt University.
In case you missed the opening chapter of their story, the Vanderbilt team in close partnership with the Tennessee Department of Education Division of School Readiness and Early Learning launched an evaluation of the impact of Tennessee’s Voluntary Pre-K program in 2009. Established in 1996 as a small pilot program, Tennessee’s Voluntary Pre-K program was scaled up in 2005 to serve 18,000 income eligible children all across the state. For the evaluation, the researchers took advantage of the fact that many of the sites are oversubscribed, and children were randomly assigned to enrollment in pre-K or to a waitlist. The full sample includes over 3000 students in 80 schools with a more intensive study conducted on a subsample of 1076 Pre-K and control children (data reported here are from the subsample). At the end of the prekindergarten year, children in the Pre-K program were substantially outperforming children in the control group in literacy, language and math outcomes.[i] At the beginning of kindergarten, children in the Pre-K program were rated by their teachers as being better prepared.[ii] At the end of kindergarten and first grade, however, children in the control condition were doing just as well as the children in the Pre-K group. In addition, the first-grade teachers reported that the Pre-K children were less well prepared for grade-level work and had poorer learning behaviors.[iii] If the Tennessee story had ended here, the Vanderbilt talk would not have been so interesting. We might hypothesize, for example, that in kindergarten and first grade, because teachers are working with the entire class that the Pre-K children were not being exposed to new information. They were not building on what they had already learned. Perhaps the Pre-K children were bored in kindergarten and first grade. At SREE, the Vanderbilt team reported on the next chapter of the Tennessee story.
In second and third grades, children in the control group outperformed children in the pre-K group. Not surprisingly, this finding has caused a bit of a dust-up.
The Tennessee study is a prime example of research in the policy arena, and I was reminded of something written by Henry Ricciuti,[iv] a developmental psychologist at Cornell University for over half a century, who was passionate about research on infants and young children. Looking back at research conducted on malnutrition and cognitive development, Ricciuti (1991), observed that in the 1960s considerable attention focused on whether early malnutrition led to poor cognitive outcomes. Both globally and nationally, policymakers and researchers were concerned about the conditions of poverty that could lead to poor developmental outcomes for children. In developing nations, there was evidence indicating that severe malnutrition was associated with poor intellectual development. Additional animal and human studies supported this position, although the findings were somewhat mixed. Furthermore, there were methodological weaknesses in the human studies. Malnutrition was confounded with other factors associated with poverty – disease, poor living conditions, low parental education, limited access to health care – such that the effect of malnutrition could not be isolated. But the thought that nutritional supplements could improve poor children’s cognitive development was incredibly enticing. In this context, “both researchers and policymakers tended to come to the rather premature and overstated conclusion that early protein-calorie malnutrition represented a major and direct cause of impaired mental development in young children (Ricciuti, 1991, p. 61). In the United States, this conclusion was used to garner support for federal programs to improve nutrition for infants and young children.
According to Ricciuti, “[d]uring this period, when there were many international conferences on malnutrition and mental development, to question the direct causal relationship between malnutrition and cognitive development seemed to be perceived by many as tantamount to arguing against the importance of good nutrition for young children in poor populations. Thus, there tended to be subtle but nevertheless, quite real social pressures not to raise such questions too publicly, particularly in the presence of policymakers” (1991, p. 66). In his look back at what had happened in the 1960s and 1970s, Ricciuti argued that what came from good intentions – the desire to protect programs that provided food to children in poverty – ultimately delayed our understanding of the role of nutrition in children’s cognitive development and that delay hindered our ability to develop effective interventions for infants and young children.
Good science needs to be open to findings that are contrary to generally expected results and use those results to further our understanding of the phenomenon. And good science is needed to develop and identify effective interventions for those we want to serve.
So, what should we take from the Tennessee Pre-K evaluation? The Vanderbilt team has been careful to situate their results in the context of decades of work on pre-kindergarten interventions and recognize that theirs is one finding among many. However, as they and others[v] have argued, their finding should not be ignored. My point is not to delve into the merits of this debate, but rather to say that in our desire to improve outcomes for young children we need to make sure that advocacy does not thwart good science. Let’s not delay our understanding of factors that improve children’s outcomes and our ability to deliver reliable and effective interventions at scale.
[i] Lipsey, M., Farran, D., Hofer, K., Dong, N., & Bilbrey, C. (March, 2012). Effects of the Tennessee Voluntary Pre-Kindergarten Program on school readiness. Presented at the Society for Research on Educational Effectiveness, Washington, DC.
[ii] Lipsey, M., Farran, D., & Hofer, K., (March, 2016). Effects of the Tennessee Voluntary Pre-Kindergarten Program on student outcomes through third grade. Presented at the Society for Research on Educational Effectiveness, Washington, DC.
[iii] Farran, D.C., & Lipsey, M.W. (October, 2015). Expectations of sustained effects from scaled up pre-K: Challenges from the Tennessee study. Brookings Institution: Evidence Speaks, 6. Downloaded on March 15, 2016 from http://www.brookings.edu/research/papers/2015/10/08-expectations-of-sustained-effects-from-scaled-up-prek-challenges-tennessee-study-farran-lipsey; Lipsey, M., Farran, D., & Hofer, K., (March, 2016). Effects of the Tennessee Voluntary Pre-Kindergarten Program on student outcomes through third grade. Presented at the Society for Research on Educational Effectiveness, Washington, DC.
[iv] Ricciuti, H.N. (1991). Malnutrition and cognitive development: Research-policy linkages and current research directs. In L. Okagaki & R.J. Sternberg (Eds.), Directors of Development: Influences on the Development of Children’s Thinking (pp. 59-80). Hillsdale, NJ: Lawrence Erlbaum Associates.
[v] Whitehurst, G.J. (February, 2014). Does Pre-k work? It depends how picky you are. Brookings Institution: The Brown Center Chalkboard Series, 56. Downloaded on March 15, 2016 from http://www.brookings.edu/research/papers/2014/02/26-does-prek-work-whitehurst.
Published by