"However, Hattie also distilled his synthesis by publishing an average effect size for over 100 different topics in education research."
After I read Visible Learning, I started to look for criticisms. There were very few. Some of the more persuasive ones were from mathematicians / statisticians / logicians, who said that the way he combined different studies was problematic (among other things). Didau summarizes those here:
Thanks for sharing! I hadn't seen that article from Didau. It's wild to me how often I hear Hattie referenced in regular PD sessions, and how clearly shallow his methodology is.
A few years ago, Ollie Lovell had Adrian Simpson on his podcast criticizing meta-analysis, which in Hattie's case is really the even less-defensible meta-meta-analysis. That was the first time I heard a criticism of Hattie in a rigorous way. Ollie had Hattie on the next month to defend his work: if anything, that episode convinced me even more that Simpson is on the right track with his critique.
Interesting. I'm not a big fan of podcasts but I'm very curious how Hattie defended himself. It's tricky because on the surface meta-analysis makes some sense. For me it's the endless oversimplifications I observe in practice that convinced me this stuff is not a good way to communicate about the science.
I’m not a big podcast guy either, but Ollie’s Education Research Reading Room had Dylan Wiliam one time when I had a long drive coming up and got me hooked on his and just a couple of others.
This is great! I thought that after grad school, I wouldn’t have to evaluate research studies anymore. Wrong! The longer I teach, the more I read, and the more research I evaluate in order to improve my pedagogical practice. Recently, I used much of what you’ve outlined here when evaluating Cognitive Mutualism research. I’m bookmarking this so that I can keep referring back to it.
Man, Hattie's website is almost unreadably self-promotional. I found a nice article from some McGill researchers about some of the statistical errors he's made: https://mje.mcgill.ca/article/view/9475/7229. It's pretty unfortunate given that he has a PhD in Statistics from U of Toronto as well.
Yup. I emphasize Hattie because I've had multiple people in multiple PD contexts cite his work uncritically. One note on that article: I believe it references errors in the first edition of Visible Learning that have since been corrected. Even without the statistical errors, I don't think Hattie's basic methodology is useful because it's so easy to share those numbers out of context.
That attention span thing is... upsetting. It is a problem that education research and mainstream cognitive psychology research are separate spheres; the intervening vacuum leaves a lot of room for nonsense.
Not that cognitive psychologists never say ridiculous things--but that *particular* thing they would not get away with saying!
I thought I was going to combust when it happened. I had beefed with that same consultant previously and felt pretty demeaned by that response so I didn't speak up about the attention span nonsense. My experience has honestly been pretty negative trying to address a lot of this in the moment.
Considering how mad I am just reading about this, I can only imagine how it felt to be there!
What is so frustrating is that the slightest bit of critical reflection on experience with actual human beings is sufficient to falsify these kinds of claims.
This is definitely an article I wish I had encountered during my teacher training at university, which often purported apparently infallible conclusions about constructivist teaching and learning. There is a lot more to consider and a lot more perspectives to learn about so thank you again for your work on this topic.
"However, Hattie also distilled his synthesis by publishing an average effect size for over 100 different topics in education research."
After I read Visible Learning, I started to look for criticisms. There were very few. Some of the more persuasive ones were from mathematicians / statisticians / logicians, who said that the way he combined different studies was problematic (among other things). Didau summarizes those here:
https://daviddidau.substack.com/p/visible-learning-invisible-errors
Thanks for sharing! I hadn't seen that article from Didau. It's wild to me how often I hear Hattie referenced in regular PD sessions, and how clearly shallow his methodology is.
A few years ago, Ollie Lovell had Adrian Simpson on his podcast criticizing meta-analysis, which in Hattie's case is really the even less-defensible meta-meta-analysis. That was the first time I heard a criticism of Hattie in a rigorous way. Ollie had Hattie on the next month to defend his work: if anything, that episode convinced me even more that Simpson is on the right track with his critique.
http://www.ollielovell.com/errr/adriansimpson/
Interesting. I'm not a big fan of podcasts but I'm very curious how Hattie defended himself. It's tricky because on the surface meta-analysis makes some sense. For me it's the endless oversimplifications I observe in practice that convinced me this stuff is not a good way to communicate about the science.
I’m not a big podcast guy either, but Ollie’s Education Research Reading Room had Dylan Wiliam one time when I had a long drive coming up and got me hooked on his and just a couple of others.
This is great! I thought that after grad school, I wouldn’t have to evaluate research studies anymore. Wrong! The longer I teach, the more I read, and the more research I evaluate in order to improve my pedagogical practice. Recently, I used much of what you’ve outlined here when evaluating Cognitive Mutualism research. I’m bookmarking this so that I can keep referring back to it.
Thanks Adrian!
Man, Hattie's website is almost unreadably self-promotional. I found a nice article from some McGill researchers about some of the statistical errors he's made: https://mje.mcgill.ca/article/view/9475/7229. It's pretty unfortunate given that he has a PhD in Statistics from U of Toronto as well.
Yup. I emphasize Hattie because I've had multiple people in multiple PD contexts cite his work uncritically. One note on that article: I believe it references errors in the first edition of Visible Learning that have since been corrected. Even without the statistical errors, I don't think Hattie's basic methodology is useful because it's so easy to share those numbers out of context.
That attention span thing is... upsetting. It is a problem that education research and mainstream cognitive psychology research are separate spheres; the intervening vacuum leaves a lot of room for nonsense.
Not that cognitive psychologists never say ridiculous things--but that *particular* thing they would not get away with saying!
I thought I was going to combust when it happened. I had beefed with that same consultant previously and felt pretty demeaned by that response so I didn't speak up about the attention span nonsense. My experience has honestly been pretty negative trying to address a lot of this in the moment.
Considering how mad I am just reading about this, I can only imagine how it felt to be there!
What is so frustrating is that the slightest bit of critical reflection on experience with actual human beings is sufficient to falsify these kinds of claims.
This is definitely an article I wish I had encountered during my teacher training at university, which often purported apparently infallible conclusions about constructivist teaching and learning. There is a lot more to consider and a lot more perspectives to learn about so thank you again for your work on this topic.
Thank you!