Evidence Based Education
What Doe the Scientific Instruction Tell Us About Primary Aged Math Instruction?
Long time readers will know that it is my opinion that efficacy for a pedagogical strategy or factor cannot be determined without meta-analysis. Generally speaking meta-analysis effect sizes are supposed to be viewed with the following interpretation.
However, research is always context specific. A single meta-analysis can find an effect size of .45 on a topic, but that does not necessarily mean that the importance of that effect is moderate. We need to be able to identify how those factors interrelate to other factors within the same context. For example, my recent non-peer reviewed meta-analysis on phonics found a mean ES of structured Literacy of .45, which is a moderate effect. However, it is significantly higher than the effect found for Balanced Literacy and Whole Language within the meta-analysis literature. As these are the most popular approaches to language instruction, we can say that Structured Literacy is the most evidence based approach of the three, despite having only a moderate effect size.
For this reason, it can be useful to conduct a secondary meta-analysis, where the results of multiple meta-analyses are put into a single meta-analysis, to better allow comparisons between different pedagogies, within the same context. Recently, I conducted a secondary meta-analysis on the topic of math instruction, which looked at 41 meta-analyses and studies, to attempt at ranking math pedagogies and factors based on their effect size. Although truthfully, I think the greatest weakness of this analysis lies in the fact that the impact of pedagogical factors tends to be age dependent, meaning that the pedagogies that work for one age do not work for another. For example, research tends to show that situational problems produce low results in younger students than older ones. To this end, I have endeavored to break down the same meta-analysis data, by age. In this article, I have compiled all the meta-analysis research I have, of primary aged students, into a single secondary meta-analysis.
Results:
Discussion:
Overall, we see the largest impacts for word problems, fluency, math facts, numeracy, and student centered instruction. However, I do want to add some caveats to this research. The number facts meta-analysis data point was from a meta-analysis of single case study research, which means the effect size is likely inflated. However, IRD effect sizes of .70 or higher are still supposed to indicate large to very large effects. That being said, I think the math facts effect size is one of the least reliable effect sizes on this list. I also have some concerns about the effect size for Student Centered Teaching, as the meta-analysis which it was sourced from, showed results that seemed divergent from the rest of the literature, something you can read more about here: ( https://www.teachingbyscience.com/student-centered-teaching )
I must admit I also found it quite surprising that word problems ranked so high and manipulatives ranked so low. I would have thought word problems would rank lower, because students might not have the language skills to solve them, or the concept/fluency knowledge, to properly apply their math understanding to solve them. However, one important caveat here, is that the meta-analysis I am drawing this effect size from found much higher results for single step problems, than multi-step problems. With this in mind, I think the scientific research would suggest that giving primary students single step word problems is very important. Although, I would imagine it is important to read these problems to students to prevent language skills from being a limiting factor in math development.
My best hypothesis as to why word problems are important for primary students is that it helps students to contextualize their math knowledge and realize the conceptual base behind the abstract concepts they are learning. However, it is always easier to determine the what, than it is the why. That being said, if my hypothesis were correct, you would think that manipulatives would do better, as the purpose of manipulatives is to help students better understand the concepts behind their math.
Overall, we see very limited meta-analysis research on math instruction that is specifically related to primary aged students. I think this is a weakness of the scientific literature in general for the subject. To better improve the contextual accuracy of effect sizes in future, I would personally like to see greater research in this regard. And while I do have some concerns about the reliability of this research, I think best practice is to base instruction on our current base of evidence and to adapt as new evidence comes out.
Factor Glossary.
Word Problems: This effect size was from the 2022 Myers meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/word-problems
Fluency and Numeracy: Within the scientific literature fluency is defined as accuracy and speed. Ergo, fluency instruction is instruction specifically meant to improve this. Some might refer to this instruction as “skill and drill”. “Numeracy skills may include mathematical-logical thinking, relational reasoning, and specific concepts foundational for number sense such as one-to-one correspondence” This effect size came from the Cason 2019 meta-analysis. For more information on this topic, you can read this article: https://www.teachingbyscience.com/math-fluency
Math Fact Fluency: Math fluency instruction specific to basic arithmetic, IE adding, subtraction, multiplying, and dividing. This effect size came from the 2012 Methe meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/math-fact-fluency
Student Centered Teaching: “learning experiences intended to address the distinct learning needs, interests, aspirations, or cultural backgrounds of individual students and groups of students. It focuses on the needs of students, involves modifications and adaptations, and often is premised on the notion that learners construct their own understanding of the world, and thus must be active participants in learning.” (Hattie, 2022). This effect size came from the 2021 Emanet meta-analysis. For more information you can read this article. https://www.teachingbyscience.com/student-centered-teaching
Direct Instruction Games: “Learning is linked with stimulus-response conditioning, rapid-pace drills, or structured lesson plans that generate student engagement through pacing and immediate feedback. Learning and instruction that entails rote memorization of facts and does not necessarily facilitate creative thought. The presentation of the game follows questions, answers, and feedback. Repetitive practice is offered.” (Kacmaz, 2022). This effect size came from the 2022 Kacmaz meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/game-based-learning-in-math
Manipulatives: Instruction that makes use of physical objects students can use to better understand the concepts behind math. This effect size came from the 2013 Carbonneau meta-analysis. For more information you can read the following article: https://www.teachingbyscience.com/manipulatives
Experiential Games: “Learning and teaching in games are based on learning by doing and solving real-life problems through experiencing and interacting with the environment. Learners gain understanding by engaging in simulated actions related to real-life experiences and learn by interacting with the objects in the game. The fundamental basis for experiential learning is the active role of the learner through interaction with the environment.” (Kacmaz, 2022). This effect size came from the 2022 Kacmaz meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/game-based-learning-in-math
Homework: This effect size is based on the 2018 Fan meta-analysis. For more information you read the following article: https://www.teachingbyscience.com/math-homework
Discovery Based Games: “Learning occurs as students discover concepts on their own through levels. Discovery learning builds on existing knowledge to discover new things, the learner applies inquiry-based reasoning, performs problem solving, makes the decision, and applies strategy. Students interact with games by exploring and manipulating objects or performing experiments”. (Kacmaz, 2022). This effect size came from the 2022 Kacmaz meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/game-based-learning-in-math
Cooperative Education: This effect size was specifically looking at cooperative vs competitive motivation. This effect size was based on the 1995 Qin meta-analysis. For more information you can read the following article: https://www.teachingbyscience.com/cooperative-vs-competitive-education
Game Based Learning: This effect size came from the 2022 Kacmaz meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/game-based-learning-in-math
Differentiation: Differentiation is an umbrella term that applies to multiple different pedagogies. Studies on differentiation often include research on ability grouping, teaching to learning styles, streaming, podding, enrichment, and individualized instruction. This effect size came from the 2018 Deunk meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/differentiation
Technology: This effect size is based on the 2022 Ran meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/technology-and-math
Calculators: This effect size is based on the 2003 Ellington meta-analysis. For more information you can read this article: https://www.teachingbyscience.com/calculator
Written by Nathaniel Hansford
Last Edited 2022-04-17
References:
Kacmaz. (2022). Examining pedagogical approaches and types of mathematics knowledge in educational games: A meta-analysis and critical review. Educational Research Review, 35, N.PAG.
Deunk. (2018). Effective differentiation Practices:A systematic review and meta-analysis of studies on the cognitive effects of differentiation practices in primary education. Educational Research Review, 24, 31–54.
E, Emanet, Et al.(2021). The Effects of Student-Centered Teaching Methods Used in Mathematics Courses on Mathematics Achievement, Attitude, and Anxiety: A Meta-Analysis Study. Participatory Educational Research. Vol 8(2), PP. 240-259.
Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380-400. doi:http://dx.doi.org/10.1037/a0031084
Qin, Z., Johnson, D. W., & Johnson, R. T. (1995). Cooperative Versus Competitive Efforts and Problem Solving. Review of Educational Research, 65(2), 129–143. https://doi.org/10.3102/00346543065002129
Fan. (2017). Homework and students’ achievement in math and science: A 30-year meta-analysis, 1986–2015. Educational Research Review, 20, 35–54.
Ran. (2022). A meta‐analysis on the effects of technology’s functions and roles on students’ mathematics achievement in K‐12 classrooms. Journal of Computer Assisted Learning., 38(1), 258–284.
Ellington, A.J. (2003). A Meta-Analysis of the Effects of Calculators on Students' Achievement and Attitude Levels in Precollege Mathematics Classes. Journal for Research in Mathematics Education, 34, 433.
Methe, S., Kilgus, S., Neiman, C., & Chris Riley-Tillman, T. (2012). Meta-Analysis of Interventions for Basic Mathematics Computation in Single-case Research. Journal of Behavioral Education, 21(3), 230–253. https://doi-org.ezproxy.lakeheadu.ca/10.1007/s10864-012-9161-1
Cason, M., Young, J., & Kuehnert, E. (2019). A meta-analysis of the effects of numerical competency development on achievement: Recommendations for mathematics educators. Investigations in Mathematics Learning, 11(2), 134–147. https://doi.org/10.1080/19477503.2018.1425591