Is Fountas and Pinnell's LLI Evidence Based?
Recently, Fountas and Pinnell’s literacy instruction program Leveled Literacy Instruction (LLI) has faced intense scrutiny for its efficacy. However, much of the criticism and support of LLI has been qualitative in nature. For example, the Fountas and Pinnell website, has a bibliography of studies on LLI and cites that this research has been positive, as proof of their efficacy. However, followers of my research know that I believe efficacy (in education) can only be determined via quantitative measures and meta-analysis and the Fountas and Pinnell website does not list any quantitative information, in support of their programs efficacy.
I looked for a meta-analysis of the topic; however, I could not find any current meta-analysis of the topic, despite the fact that Fountas and Pinnell’s program is the most widely used English Language program in the world and despite the fact that it has faced intense scrutiny. I decided to do my own meta-analysis of the topic, to try and do my best job of answering the question is LLI evidence-based. Now one caveat that I will make is that this meta-analysis is of course not peer reviewed, so please take my findings with a grain of salt.
On average, efficacy studies of LLI had an ES of .28, which is in the moderate to low range. Meaning that LLI does have a positive impact, but that impact is below the average impact of a literacy intervention and below the average impact of many other literacy programs. Phonics, for example, has an average ES (according to Hattie) of .60 and Synthetic Phonics has an ES of .45 compared to a control group of regular phonics, according to the NRP 2006 study. This suggests that on average synthetic phonics is far superior to LLI. Similarly RTI, morphology instruction, and repeated reading all show far greater effect sizes than LLI. With that being said, I think it is fair to say that LLI is better than no specialized literacy instruction; however, I do not think it is fair to say that LLI is a high yield strategy or even “evidence-based”.
While the mean ES of LLI is .28, there are several caveats that I feel I need to make. Firstly, all of these LLI intervention studies made use of small group instruction and small group instruction has been shown to have greater impacts than regular class instruction. I therefore think, the fact that LLI uses small group instruction likely inflates its results.
Additionally, a large number of the LLI studies were done by the same institute, the University of Memphis Center for Research Policy in Education (CRPE). Indeed, the LLI website only cites one study not done by this institute. These studies, as far as I could find, all showed effect sizes much more positive than all other studies done on the topic. Indeed one CRPE study found a mean ES of .57, which was a outlier compared to the rest of studies on the topic. I also found some strange statistical anomalies in the papers. For example, the 2015 paper by the institute, had 3 effect sizes based on identical standard deviations and results for both their control group, intervention group, and on their pre and post tests. The resulting effect sizes were also extremely high. For this reason, I excluded these 3 effect sizes from my calculations. On the other hand of the spectrum, the Gonzalez study also had an extreme outlier with an ES of -.22. However, this study had a very small sample size.
Ultimately I am concerned that the CRPE might have some research bias, in favour of LLI; however, their studies on the surface appeared very well done. They had large sample sizes, were randomized control trials, and posted their raw data. For these reasons I think the most reasonable ES to reflect the likely efficacy of LLI would be the Mean ES Controlling for Outlier Data, which is .38.
I searched the Education Source Database for all studies including the phrases: “Fountas and Pinnell”, “LLI”, and “Levelled Literacy Instruction”. I also consulted the bibliography on LLI efficacy studies, given on the LLI website. I also looked for additional studies, using Google and Google Scholar. I included all studies on LLI that either reported on an ES or included their raw statistical data. I excluded any studies that had too small of a sample size or did not include a control group. In total 10 studies were found on the topic; however, 2 were excluded. In order to standardize the data as much as possible, I tried to re-calculate as much data as I could, using Cohen's D effect size method for studies with sample sizes over 50 and Hedges G for studies under 50. However, the Gonzalez 2018 study study recorded their effect size, but did not record their raw data or how they calculated their effect sizes.
Final Grade: C+: The program is not research based, IE most of the principles are not supported by the meta-analysis data.
Qualitative Grade: 1/10
The program includes the following evidence based principles: comprehension instruction.
Written by Nate Joseph
Last Edited 2022-04-10
Center for Research Policy in Education. (2012). Efficacy of the Leveled Literacy Intervention System for K–2 Urban Students: An Empirical Evaluation of LLI in Denver Public Schools Study Dates: 2011–2012. University of Memphis. Retrieved from <https://www.fountasandpinnell.com/shared/resources/FP_LLI_Research_CREP-LLI-Efficacy-Full-Report-2012.pdf>.
Center for Research Policy in Education. (2010). Implementation of Effective Intervention: An Empirical Study to Evaluate the Efficacy of Fountas & Pinnell’s Leveled Literacy Intervention System (LLI) Retrieved from <https://www.fountasandpinnell.com/shared/resources/FP_LLI_Research_CREP-LLI-Efficacy-Full-Report-2010.pdf>.
Center for Research Policy in Education. (2016). The Efficacy of the Leveled Literacy Intervention System for Students in Grades 3–5: Data Summary Report for Abilene Independent School District 2015–2016. University of Memphis. Retrieved from <https://www.fountasandpinnell.com/shared/resources/LLI%203-5%202015_16-AISD_FULL_REPORT-web.pdf>.
Center for Research Policy in Education. (2015). The Efficacy of the Leveled Literacy Intervention System for Students in Grades 3–5: Data Summary Report for Denver Public Schools 2015–2016. University of Memphis. Retrieved from <https://www.fountasandpinnell.com/shared/resources/LLI%203-5%202015_16-DPS_FULL_REPORT-web.pdf>.
Center for Research Policy in Education. (2016). The Efficacy of the Leveled Literacy
Intervention System for Students in Grades 3–5: Data Summary Report for Sandwich Public Schools 2015–2016. University of Memphis. Retrieved from <https://www.fountasandpinnell.com/shared/resources/LLI%203-5%202015_16-SPS_FULL_REPORT-web.pdf>.
Heinmann. (2010). Levelled Literacy Intervention, Research and Data Collection Project. Fountas and Pinnell. Retrieved from <https://www.fountasandpinnell.com/shared/resources/FP_LLI_Research_Research-and-Data-Collection-Project-Report.pdf>.
Gonzalez, et al. (2018). Challenges in Adolescent Reading Intervention: Evidence from a Randomized Control Trial. Mathmatica Policy Research. Retrieved from <https://files.eric.ed.gov/fulltext/ED587404.pdf>.
Lang, Laura & Torgesen, Joseph & Vogel, William & Chanter, Carol & Lefsky, Evan & Petscher, Yaacov. (2009). Exploring the Relative Effectiveness of Reading Interventions for High School Students. Journal of Research on Educational Effectiveness. 2. 149-175. 10.1080/19345740802641535.
Metz, Tracie Jean, "A case study: Effects of using leveled literacy intervention on fourth and fifth grade students' reading achievement" (2014). ETD Collection for Fayetteville State University. AAI3581423.
J, Majewski. (2018). The effects of a leveled literacy inter acy intervention (LLI) on elementary age students reading below grade level. Rowan University. Retrieved from <https://rdw.rowan.edu/cgi/viewcontent.cgi?article=3574&context=etd>.
J, Hattie. (2021). Visible Learning Metax. Retrieved from <www.visiblelearningmetax.com>.
N, Hansford. (2021). Morphology. Pedagogy Non Grata. Retrieved from <https://nathanielhansford.wixsite.com/website/morphology>.
Linnea, et al. (2001). Systematic Phonics Instruction Helps Students Learn to Read: Evidence From the National Reading Panel’s Meta-Analysis. Ontario Institute for Studies in Education. Retrieved from <https://www.dyslexie.lu/JDI_02_02_04.pdf>.