Article Summary: 

This article continues on with our series on what percentage of students can reach grade level, as most typically defined by either the 30th percentile on a norm referenced assessment or the basic benchmark on the NAEP test. In previous articles we showed that 97.5% of students could achieve grade level standards if three conditions were met:

1. 82% of students were reading on grade level, based on core instruction alone 

2. Struggling readers received 80 hours or more of tier 3 instruction

3. Tier 3 instruction included systematic phonics instruction. 

(https://www.pedagogynongrata.com/tier-3-instruction) 

The biggest barrier here would seem to be that in most US schools only 63% of students are currently reading on grade level. This means that less than 95% of students would likely reach the basic benchmark, if struggling readers were provided sufficient tier 3 instruction. This leads to the question, can we get 82% of students reading based on just core instruction alone? In our last article (https://www.pedagogynongrata.com/core-instruction), we showed that with phonics instruction starting in kindergarten, this was at least possible, if not necessarily probable. 

For this article, we wanted to test whether or not state policies based on the science of reading were effectively improving the number of students reading at grade level. We matched 12 states that adopted science of reading laws prior to 2020 with states that did not and then compared their 2022 NAEP results. On average states that passed SOR laws showed a smaller covid slump and outperformed states that did not. However, these differences were very small and not statistically significant. That said, some of the SOR laws appeared to be more effective than others. 

Introduction: 

There is substantial experimental evidence that a systematic phonics approach is superior to an unsystematic phonics approach or a balanced literacy approach. In 2001 the NRP conducted the first meta-analysis of this topic and found a mean effect size of .44, for systematic phonics. Both Camilli 2003 and Stuebing 2008 re-analyzed the NRP studies again, to measure the fixed effect of systematic phonics vs unsystematic phonics. In both studies, the systematic phonics approach outperformed, if only by a smaller effect size. Since the NRP, 14 more meta-analyses have been conducted on phonics instruction (to the best of my knowledge) and on average showed a mean effect size of .43, if we exclude for outlier studies. If we include outliers, that effect size jumps to .54 (Hansford, 2022). 

In 2022, I conducted a  meta-analysis, for this blog, with Joshua King of phonics vs balanced literacy programs also showed that phonics based programs showed roughly double the learning impact as did balanced literacy based programs, this meta-analysis has since been updated and submitted for peer review. Similarly, earlier this month, I re-analyzed the 2009 Torgesen and 2002 Mathes literature reviews of the impact of tier 3 instruction. In this re-analysis, I consistently found that a systematic phonics approach outperformed an approach that focused on cueing/constructivist pedagogies. To date there are no peer-reviewed meta-analyses on the topic of balanced literacy. To the best of my knowledge, the only meta-analysis specifically, conducted on balanced literacy, was mine, and it showed a low impact. 

Over the last several years, there has been a growing recognition of the importance for systematic phonics instruction and consequently, many states have adopted so-called “science of reading laws”. These state policies typically mandate a series of pedagogical shifts, mainly replacing balanced literacy with structured literacy. However, critics of these laws have often tried to claim a lack of evidence for mandating systematic phonics instruction on a state level. Oftentimes, these critics will point to a lack of research demonstrating that systematic phonics at a state level improves the number of students at grade level. 

I am very critical of the idea that a lack of state level evidence is problematic. While on paper, wanting to see state level outcomes, seems meaningful, as we want to know that public policy works. However, I am skeptical of the experimental value. While a small scale study might have far fewer students, we also have a much better understanding of what happened. In a state level analysis, we really have no meaningful idea of what actually happened in the classroom. Afterall, curriculum books often just sit dusty on school shelves. It also needs to be pointed out that the number of students on grade level is not actually a particularly scientific metric, as it uses an arbitrary cut off mark that is inconsistent across populations, tests, and contexts. Using the number of students at benchmark does not account for students who are far above grade level, nor does it account for if students are only slightly above grade level. Conversely, an effect size is primarily used to measure the magnitude of effect that a treatment provides to one group of students over another. For example, we could have 100% of students in one class, on grade level and 0% in another, but the mean difference could still be only 1%. The outcome would look significant; however, in reality, this difference could be entirely statistically insignificant. 

That said, it is still necessary to examine the impact of state SOR laws on the percentage of students at grade level, for three reasons. Firstly, governments are spending millions of dollars implementing these laws and people want to know if it's working. Secondly, there is value in identifying how effectively scientifically validated pedagogy can be implemented on a state level. Lastly, while effect sizes and meta-analyses are better tools for identifying the scientifically demonstrable efficacy of pedagogies, they are far less well understood by the public. There is therefore a value in identifying the impact of state education “SOR” policies on the percentage of students at grade level. 

With these reasons in mind, we decided to analyze changes in the NAEP scores, for states who had passed “SOR” laws. We decided to focus on the following research questions:

1. Are SOR laws working?

2. What is the impact of SOR laws on the number of students at grade level?

3. What types of policy factors have the greatest impact?

Methods:

In July of 2022, Sarah Schwartz published a list of states that have passed SOR policy laws. She also recorded how comprehensive these laws were, based on a scale of 1-6. To get the highest score of 6 states had to include legislation that targeted professional development, teacher preparation, teacher licensing, assessment, materials, and instruction. I decided to use this data, combined with the NAEP data-base of testing scores, to try and measure if these laws were actually effectively improving student achievement. 12 of these states had passed SOR laws prior to 2020. I  used these states to form a treatment group. I then selected 11 states that had equivalent 2019 NAEP reading scores for grade 4, one control state was used twice, as there were no other states available with equivalent test scores. These combined scores were used to represent pre-intervention reading scores. I then compared the 24 states, based on the 2022 testing results, to measure the magnitude of effect for these SOR laws. Cohen’s d effects sizes were calculated by comparing the difference in raw scores and dividing them by the standardized deviation between states (opposed to students).