One question that still nagged at me though, was how did reading instruction, not specifically targeted at reading comprehension, affect comprehension results? A common criticism of the NRP meta-analysis is that it did not control for decoding, vs fluency, vs comprehension outcomes. Indeed, phonics critics will often argue that phonics instruction only benefits decoding and does not improve comprehension. I have even some phonics critics go far to say that students who learn from phonics are "word callers", able to read the words on the page, but unable to understand them. These critics often instead advocate for a balanced literacy or morphology focused approach. From a theoretical perspective, I can see the potential advantage of these approaches, on comprehension, over phonics based instruction. While the term Balanced Literacy is typically loosely defined, advocates of this type of instruction typically focus on passage comprehension over word recognition, and morphology is inherently linked to word understanding. That said, long time readers of this blog will know, I prefer to look at data, over theoretical hypotheses. 

​

Recently (2023), Dr. Mat Burns, Dr. Kelly Cartwright, and Dr. Nell Duke conducted a secondary-meta-analysis that included an analysis, which looked at the impact of phonics instruction on reading comprehension. They identified 9 studies, on classroom instruction, which showed a median effect size of .19 (negligible) and 32 studies for struggling readers, which showed a median effect size of .48, for a mean overall effect size of .41. These results, appear to suggest that phonics instruction has a positive, but negligible effect on reading comprehension for average readers, but a moderately positive effect on reading comprehension for struggling readers. 

​

Interestingly, Burns et al. (2023)  also looked at morphology instruction. They found 3 studies on the effects of morphology for average readers, on reading comprehension, with a median effect size of .16. They found 1 study on the effects of morphology for average readers, on reading comprehension, with a mean effect size of .14. These results seem to suggest that the impact of morphology instruction on reading comprehension is negligible. These morphology results are also inline with my recent meta-analysis on reading comprehension, which also found negligible effects for reading comprehension. Moreover, these results are also in line with my secondary meta-analysis of morphology, which showed negligible results for reading comprehension (https://www.pedagogynongrata.com/morphology) . It does seem safe to say that there is strong empirical evidence showing that morphology has little measurable benefit on reading comprehension. Of course, this is not to say that there are no other benefits of morphology instruction, indeed, I believe there are many. 

I couldn’t help but notice that Burns et al. (2023), received some criticism on social media for this meta-analysis, from a prominent morphology scholar, who seemed surprised by the high results found, for phonics. Due to my previously conducted meta-analyses on phonics, structured literacy, and my reviews of the empirical evidence for language programs, I now have a large data-base of studies for different programs in my google drive. I decided to conduct a meta-analysis, of my own, on the impact of systematic phonics programs and balanced literacy programs on reading comprehension, using this data-base. I wanted to see if I could both replicate the results of Burns et al. (2023), and to see if the results of phonics were better than the results of balanced literacy on reading comprehension, a question not previously systematically examined in the scientific literature. 

Methodology:

Within this meta-analysis, I chose to only include systematic phonics studies and not all phonics studies. In my opinion, the real debate question is not, if some phonics improves reading outcomes, but rather if a structured/systematic phonics approach is better than a balanced literacy approach. The NRP defined a systematic phonics approach as one that included a scope and sequence, included explicit phonics instruction, and used decodable readers. Most researchers agree that some phonics instruction is necessary for younger readers. However, there are still some scholars who advocate against systematic phonics instruction. With this in mind, I felt including programs like, Words their Way, and Read 180, which include phonics, but are not easily coded as either balanced literacy or structured literacy could potentially obfuscate the results, rather than add clarity to this debate.   

From my data-base of studies on programs, I identified 11 programs as structured literacy, which also had experimental or quasi-experimental studies: Jolly Phonics, Open Court, Lexia, Reading Simplified, Wonders, SPIRE, SIPPS, Corrective Reading, Reading Mastery, CKLA, and Wilson. However, only 6 of these programs had studies that specifically measured reading comprehension outcomes. Those programs were: Open Court, Lexia, Reading Simplified, Wonders, Corrective Reading, Reading Mastery, and Wilson. In general, it did seem that there was far less research on the impact of structured literacy on comprehension, then there was decoding or fluency. I was only able to find 13 studies in my data-base that fit this inclusion criteria. 

Within my data-base of Balanced Literacy studies, I had 3 programs commonly agreed upon as Balanced Literacy based that had experimental or quasi-experimental studies: Reading Recovery, Leveled Literacy Instruction, and Units of Study. However, only Reading Recovery had studies specifically measuring comprehension. I found 3 RCTs on Reading Recovery that looked at Reading Comprehension. 

Within my data-base, all effect sizes had been previously calculated. If the original authors had calculated their own effect size, it was accepted at face value. In the event that the effect size had not been calculated, Hedge’s g and Cohen’s d were used to measure the effect size (Hedge’s g if the study had a sample below 50 and Cohen’s d if the sample was above 50). All calculated effect sizes were previously verified by at least one other author.  

Results:

A mean effect size of .30, 95% CI= [.15, .45] was found for the effect of systematic phonics instruction on reading comprehension outcomes. A mean effect size of .22 95% CI =[-.68, 2.12] was found for the effect of balanced literacy instruction on reading comprehension outcomes. Both results were small. However, the balanced literacy results were 27% smaller. Moreover, the small number of balanced literacy studies and wide range of confidence intervals, suggest that there is insufficient research to indicate that Balanced Literacy approaches can meaningfully improve reading comprehension results. However, as I have previously discussed in this blog assessments on reading comprehension that are not standardized, appear to show very inflated results. So I re-calculated these results, by excluding all non standardized assessments.