Evidence Based Education
Does Phonics Instruction Help With Comprehension?
Abstract:
A meta-analysis was conducted comparing the effects of systematic phonics and balanced literacy reading instruction on reading comprehension assessments. 16 studies were located. Systematic phonics instruction showed a mean effect size of .30, 95% CI= [.15, .45] and balanced literacy showed a mean effect size of .22 95% CI =[-.68, 2.12]. Systematic phonics instruction also showed a higher effect size on standardized reading comprehension tests, when compared to vocabulary, strategy, content, or morphology instruction. There are many types of instruction and skill/knowledge sets that improve student reading comprehension. However, these results suggest that decoding instruction is an essential component for language programs seeking to improve reading comprehension outcomes, at least for young or struggling readers.
Introduction:
Recently, I completed a meta-analysis on reading comprehension instruction, of which I submitted for peer review and posted a summary of to my blog. In this meta-analysis, I analyzed the impact of different types of comprehension instruction on standardized measures of reading comprehension. If you want to read the full summary you can find it here: https://www.teachingbyscience.com/reading-comprehension
However, the core results of this meta-analysis can be seen in the below graph:
One question that still nagged at me though, was how did reading instruction, not specifically targeted at reading comprehension, affect comprehension results? A common criticism of the NRP meta-analysis is that it did not control for decoding, vs fluency, vs comprehension outcomes. Indeed, phonics critics will often argue that phonics instruction only benefits decoding and does not improve comprehension. I have even some phonics critics go far to say that students who learn from phonics are "word callers", able to read the words on the page, but unable to understand them. These critics often instead advocate for a balanced literacy or morphology focused approach. From a theoretical perspective, I can see the potential advantage of these approaches, on comprehension, over phonics based instruction. While the term Balanced Literacy is typically loosely defined, advocates of this type of instruction typically focus on passage comprehension over word recognition, and morphology is inherently linked to word understanding. That said, long time readers of this blog will know, I prefer to look at data, over theoretical hypotheses.
Recently (2023), Dr. Mat Burns, Dr. Kelly Cartwright, and Dr. Nell Duke conducted a secondary-meta-analysis that included an analysis, which looked at the impact of phonics instruction on reading comprehension. They identified 9 studies, on classroom instruction, which showed a median effect size of .19 (negligible) and 32 studies for struggling readers, which showed a median effect size of .48, for a mean overall effect size of .41. These results, appear to suggest that phonics instruction has a positive, but negligible effect on reading comprehension for average readers, but a moderately positive effect on reading comprehension for struggling readers.
Interestingly, Burns et al. (2023) also looked at morphology instruction. They found 3 studies on the effects of morphology for average readers, on reading comprehension, with a median effect size of .16. They found 1 study on the effects of morphology for average readers, on reading comprehension, with a mean effect size of .14. These results seem to suggest that the impact of morphology instruction on reading comprehension is negligible. These morphology results are also inline with my recent meta-analysis on reading comprehension, which also found negligible effects for reading comprehension. Moreover, these results are also in line with my secondary meta-analysis of morphology, which showed negligible results for reading comprehension (https://www.pedagogynongrata.com/morphology) . It does seem safe to say that there is strong empirical evidence showing that morphology has little measurable benefit on reading comprehension. Of course, this is not to say that there are no other benefits of morphology instruction, indeed, I believe there are many.
I couldn’t help but notice that Burns et al. (2023), received some criticism on social media for this meta-analysis, from a prominent morphology scholar, who seemed surprised by the high results found, for phonics. Due to my previously conducted meta-analyses on phonics, structured literacy, and my reviews of the empirical evidence for language programs, I now have a large data-base of studies for different programs in my google drive. I decided to conduct a meta-analysis, of my own, on the impact of systematic phonics programs and balanced literacy programs on reading comprehension, using this data-base. I wanted to see if I could both replicate the results of Burns et al. (2023), and to see if the results of phonics were better than the results of balanced literacy on reading comprehension, a question not previously systematically examined in the scientific literature.
Methodology:
Within this meta-analysis, I chose to only include systematic phonics studies and not all phonics studies. In my opinion, the real debate question is not, if some phonics improves reading outcomes, but rather if a structured/systematic phonics approach is better than a balanced literacy approach. The NRP defined a systematic phonics approach as one that included a scope and sequence, included explicit phonics instruction, and used decodable readers. Most researchers agree that some phonics instruction is necessary for younger readers. However, there are still some scholars who advocate against systematic phonics instruction. With this in mind, I felt including programs like, Words their Way, and Read 180, which include phonics, but are not easily coded as either balanced literacy or structured literacy could potentially obfuscate the results, rather than add clarity to this debate.
From my data-base of studies on programs, I identified 11 programs as structured literacy, which also had experimental or quasi-experimental studies: Jolly Phonics, Open Court, Lexia, Reading Simplified, Wonders, SPIRE, SIPPS, Corrective Reading, Reading Mastery, CKLA, and Wilson. However, only 6 of these programs had studies that specifically measured reading comprehension outcomes. Those programs were: Open Court, Lexia, Reading Simplified, Wonders, Corrective Reading, Reading Mastery, and Wilson. In general, it did seem that there was far less research on the impact of structured literacy on comprehension, then there was decoding or fluency. I was only able to find 13 studies in my data-base that fit this inclusion criteria.
Within my data-base of Balanced Literacy studies, I had 3 programs commonly agreed upon as Balanced Literacy based that had experimental or quasi-experimental studies: Reading Recovery, Leveled Literacy Instruction, and Units of Study. However, only Reading Recovery had studies specifically measuring comprehension. I found 3 RCTs on Reading Recovery that looked at Reading Comprehension.
Within my data-base, all effect sizes had been previously calculated. If the original authors had calculated their own effect size, it was accepted at face value. In the event that the effect size had not been calculated, Hedge’s g and Cohen’s d were used to measure the effect size (Hedge’s g if the study had a sample below 50 and Cohen’s d if the sample was above 50). All calculated effect sizes were previously verified by at least one other author.
Results:
A mean effect size of .30, 95% CI= [.15, .45] was found for the effect of systematic phonics instruction on reading comprehension outcomes. A mean effect size of .22 95% CI =[-.68, 2.12] was found for the effect of balanced literacy instruction on reading comprehension outcomes. Both results were small. However, the balanced literacy results were 27% smaller. Moreover, the small number of balanced literacy studies and wide range of confidence intervals, suggest that there is insufficient research to indicate that Balanced Literacy approaches can meaningfully improve reading comprehension results. However, as I have previously discussed in this blog assessments on reading comprehension that are not standardized, appear to show very inflated results. So I re-calculated these results, by excluding all non standardized assessments.
Surprisingly, the results for systematic phonics actually went up when we excluded, researcher designed assessments. These results would suggest that if assessment rigor is controlled for, systematic phonics instruction is not only better than balanced literacy instruction for reading comprehension, but also all types of comprehension instruction, except reciprocal teaching. This is especially meaningful for primary instruction, as reciprocal teaching has only been studied in intervention settings for older students and thus, would seem to suggest that the best way to improve young students' reading comprehension is to teach them how to decode.
Of course, effect sizes are contextual, so I decided to conduct a moderator analysis, using a random effects model, on the systematic phonics studies, as can be seen in the below chart.
Effect sizes are also quality dependent, so I conducted a regression analysis of the systematic phonics studies to better clarify the impact of quality on these results:
The regression results suggest that the three highest quality studies, showed on average, some of the highest effects. Suggesting that the found effect sizes were not primarily influenced by study quality.
Discussion:
I would have hypothesized that the impact of systematic phonics instruction on reading comprehension would have been slightly higher than that of balanced literacy or morphology instruction. However, I was very surprised that the impact of phonics instruction on standardized reading comprehension assessments was higher than most types of comprehension instruction. These phonics results were statistically similar to that of the Burns et al. (2023) study and when compared to my recent comprehension meta-analysis suggest that decoding instruction might have the largest impact on comprehension instruction in the early grades, especially for struggling readers., as can be seen in the below graphic, which synthesizes the standardized test results of both this study and my previous one.
Limitations:
This meta-analysis was conducted based on the Pedagogy Non Grata data-base of studies and was not based on a new systematic search. Because this meta-analysis was based on my own data-base of studies and not a systematic search, I have no intent of even trying to peer-review it. All individual study effect sizes were verified by at least one other author. However, the meta-analysis calculations were only calculated by one person (myself), which limits the reliability of this data. The total number of studies in this meta-analysis were very small, in part, because this question appears to be under-researched. There was an especially small number of studies on the impact of balanced literacy reading instruction, on reading comprehension. Programs that did not neatly fit into the coding of systematic phonics or balanced literacy were not included within the analysis. All systematic phonics instruction studies were on either primary aged students or on struggling readers. All balanced literacy studies were on grade 1 struggling readers.
Written by Nathaniel Hansford, Kathryn Garforth.
Contributed to by: Kristen McQuillan
Last Edited 2023-03-11
References: (A * indicates that the study was in the meta-analysis). Please click here: to download a copy of the google spreadsheet:
*-Amendum, S. J., Bratsch, H. M., & Vernon, F. L. (2018). Investigating the Efficacy of a Web‐Based Early Reading and Professional Development Intervention for Young English Learners. Reading Research Quarterly, 53(2), 155–174. https://doi-org.ezproxy.lakeheadu.ca/10.1002/rrq.188
*-Amendum, S. J., Vernon-Feagans, L., & Ginsberg, M. C. (2011). The Effectiveness of a Technologically Facilitated Classroom-Based Early Reading Intervention. Elementary School Journal, 112(1), 107–131.
*-Benner, G. J., Michael, E., Ralston, N. C., & Lee, E. O. (2022). The impact of supplemental word recognition strategies on students with reading difficulties. International Journal of Instruction, 15(1) 837-856. https://doi.org/10.29333/iji.2022.15148a.
*-Borman, G. D., Dowling, N. M., & Schneck, C. (2008). A Multisite Cluster Randomized Field Trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389–407. https://doi.org/10.3102/0162373708326283
*-Bratsch-Hines, M., Vernon-Feagans, L., Pedonti, S., & Varghese, C. (2020). Differential Effects of the Targeted Reading Intervention for Students With Low Phonological Awareness and/or Vocabulary. Learning Disability Quarterly, 43(4), 214–226. https://doi-org.ezproxy.lakeheadu.ca/10.1177/073194871985868
-Burns, Matthew & Duke, Nell & Cartwright, Kelly. (2023). Evaluating components of the active view of reading as intervention targets: Implications for social justice. School psychology (Washington, D.C.). 38. 30-41. 10.1037/spq0000519.
*-"Heidi K. Oglesbee. (2014). Does Employing the Wilson’s Fundations Program Impact the Reading Growth of First Grade Students. Masters Thesis. College of Bowling Green
-Hansford, N,. McGlynn, S & King, J. (2023) What's Better for Reading Comprehension: Strategy or Content Knowledge Instruction? Teaching by Science. https://www.teachingbyscience.com/reading-comprehension
-Hansford, N. (2022). A Secondary Meta-Analysis of Morphology. Teaching by Science.
https://www.pedagogynongrata.com/morphology
*-Lloyd, J., Cullinan, D., Heins, E. D., & Epstein, M. H. (1980). Direct Instruction: Effects on Oral and Written Language Comprehension. Learning Disability Quarterly, 3(4), 70–76. https://doi.org/10.2307/1510677
Long-Term Effects and Cost-Effectiveness. Under the Investing in Innovation (i3) Scale-Up. University of Deleware.
*-O'Connor, R. E., Jenkins, J. R., Cole, K. N., & Mills, P. E. (1993). Two approaches to reading instruction with children with disabilities: does program design make a difference?. Exceptional children, 59(4), 312–323. https://doi.org/10.1177/001440299305900404
*-Pinnell, G. S., DeFord, D. E., & Lyons, C. A. (1988). Reading Recovery: Early intervention for at-risk first graders (Educational Research Service Monograph). Arlington, VA: Educational Research Service.
*-Schechter, R., Macaruso., Kazakoff, E., & Brooke, E. (2015) Exploration of a Blended Learning Approach to Reading Instruction for Low SES Students in Early Elementary Grades, Computers in the Schools, 32:3-4, 183-200, DOI: 10.1080/07380569.2015.1100652
State University. Retrieved from <https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?accession=bgsu1398712529&disposition=inline&fbclid=IwAR2-IlwYz2bYzdyjWRXYxN7bkSrgSWii6MdbHyEb0RRZMdaiEgBQt6OGJNU>. "
*-Schmitt, M. C., & Gregory, A. E. (2005). The Impact of an Early Literacy Intervention: Where Are the Children Now? Literacy Teaching and Learning: An International Journal of Early Literacy, 10(1), 1-20.
*-Sirinides, P., Gray, A., & May, H. (2018). The Impacts of Reading Recovery at Scale: Results From the 4-Year i3 External Evaluation. Educational Evaluation and Policy Analysis, 40(3), 316–335. https://doi.org/10.3102/0162373718764828
*-Stockard, J. (2010). Promoting Reading Achievement and Countering the “Fourth-Grade Slump”: The Impact of Direct Instruction on Reading Achievement in Fifth Grade. Journal of Education for Students Placed at Risk, 15(3), 218–240. https://doi.org/10.1080/10824669.2010.495687
*-Stockard, Jean & Engelmann, Kurt. (2010). The development of early academic success: The impact of Direct Instruction's Reading Mastery.. Journal of Behavior Assessment and Intervention in Children. 1. 2-24. 10.1037/h0100357.
*-Vernon-Feagans, L., Kainz, K., Ginsberg, M., Hedrick, A., & Amendum, S. (2013). Live Webcam Coaching to Help Early Elementary Classroom Teachers Provide Effective Literacy Instruction for Struggling Readers: The Targeted Reading Intervention. Journal of Educational Psychology, 105(4), 1175–1187. https://doi-org.ezproxy.lakeheadu.ca/10.1037/a0032143
*-Wanzek, J., & Roberts, G. (2012). Reading interventions with varying instructional emphases
*-Yu, L., & Rachor, R. (2000, April). The two-year evaluation of the three-year Direct Instruction program, in an urban public school system. Presented at the annual meeting of the American Educational Research Association.