I taught high school for a long time and retired a couple years ago. The one indisputable result of the NCLB earthquake and subsequent aftershocks was the heightened tendency of various players to make big claims that, upon closer inspection, turned out to be essentially hype. But it's actually funny - in kind of a nice way - that 'direct instruction' is having a moment.
Very much appreciate the thoughts and discussion. I'm interested in how we are defining learning. What exactly are the kiddos in the video "showing off"? It feels like given a question asked in a particular way, a few of them can say the words they were taught to say. Is that learning? What would we be comfortable claiming the kiddos in the video have learned? At Alpha, some kiddos can score very highly on a thing called MAP. Alpha is equating learning with scoring highly on MAP. Is that ok?
I agree with you that both Direct Instruction and Alpha School are focused on relatively narrow outcomes. That's a criticism I've made of DI before. That said, in Project Follow Through, Direct Instruction had better results on both basic skills and problem-solving then all of the other curricula. Basic skills are an ingredient in higher-order thinking. Not the whole thing, but lots of students struggle with higher-order skills because they lack foundational knowledge. I would also emphasize here that those are kindergarten students. Rote or not, what they're doing is pretty impressive for kindergarteners and far beyond a typical kindergarten student.
What we measure is what ends up getting valued, so I definitely wonder if Alpha School is narrowing their focus on MAP in an unhealthy way. Still, Alpha School students are doing pretty cool stuff in the afternoons. Parents are choosing the school. If the goal is to get the basics down in the morning and then go practice competitive debate or chess or whatever in the afternoons, that sounds like an education many families would choose.
I just watched it without the sound as you recommend and I disagree. There is definitely some practiced call-and-response (that's a big part of DI) but I find this impressive, especially considering these are kindergarten students.
Engelmann himself makes clear at the start of the video that this is not an example of teaching, but rather students "showing off" what they have already learned. It’s also worth noting that the children shown do not appear to be at kindergarten level; they look closer to the end of the program, around Year 3.
In addition, official evaluations of Direct Instruction (DI) show considerable variation in outcomes. Richard Anderson, who directed the results analysis, concluded:
“The Direct Instruction Model achieved this heroic level of performance with only a minority of its groups. If educators do know how to close the achievement gap between disadvantaged children and the nation in general, Follow Through has not proved they can do it consistently.”
Given this context—and Engelmann’s background in marketing—it’s reasonable to suspect that the students featured were selected because they showcase the program at its very best.
Thanks for sharing! Good point about the "testing on healthiest patients first metaphor" though I agree with you that phase I is probably not necessary in education. There are certainly plenty of "experiments" similar to this one being conducted in lots of schools already.
"...and for those students the program accelerates their learning faster than what they would achieve in a typical school. Is it 20 percent? 50 percent?"
If it's not 100% it means that inequality in life outcomes for students will increase. And that will generate a lot of resistance to Alpha from some people.
The real danger is admins taking Alpha School's AI hype at face value, buying some blinkered program at an inflated price, and using it as a justification to slash staff.
I agree. I could imagine a nightmare scenario where admin get rid of teaching positions, hire paraprofessionals to supervise students on Timeback, most students spend the time screwing around or playing computer games, and it's all a complete disaster. The worst part is that I bet some students would still learn, and that could be used to justify keeping the whole program running.
Apologies for the vaguepost, as it's not my place to make any official statements or revelations -- that said, regarding your conclusion:
"If Alpha School is serious about being “the future of education,” the solution is simple. I’m seeing these allusions to Engelmann. Follow in his footsteps. Partner with a struggling school where outcomes are poor. Test the curriculum there. Refine and iterate with the students who are most likely to struggle."
I suspect you will be eating a bit of crow in less time than you think :)
I’m not an alpha booster, but don’t their results in Brownsville and Texas sports academy completely invalidate this thesis? I don’t think parents sending their kids to TSA (for free with vouchers) because they want to become d1 athletes are subject to particularly strong academic selection effects
My impression is that a lot of the low-income families in Brownsville left the school. I don't know many details about TSA but "free private school via voucher" is very different from "seek out students who are struggling academically." I would be curious to see those demographics and outcomes broken out separately from what Alpha has shared about the program as a whole. I think Brownsville especially could have been a great testing ground but my impression from the Wired story (unfortunately paywalled) is that a bunch of students had trouble keeping up academically and that led to departures.
In a fall Wired piece (https://www.wired.com/story/ai-teacher-inside-alpha-school/) Alpha School initially said they would share raw data but did not follow through. I haven't seen any independent analysis so I'm trusting what they say.
To answer your question: I don't know students are learning faster. Maybe Alpha School is lying, or cherrypicking, or something. I have seen a number of testimonials, and the data is plausible to me given how they run their program. I'm sharing some skepticism about their program assuming the results are real. Totally reasonable to doubt the results themselves.
Thanks. I'm not familiar with report cards. You can't just put the outcomes... you have to put the inputs..? Otherwise it's meaningless and just reflects your cohort and not the added value that you methods have conferred?
When I put something on Twitter the Principal said 'they do well on standardized tests'.
The question is:
- Do they do well on standardized test because they are at the Alpha School
or
- Are they at the Alpha School because they are the kind of student who does well on standardized test?
AND
Would they do well anyway independent of where they are... because they're in a family who can afford $40k a year, which itself confers a number of significant educational advantages?
It's also not a 'curriculum' per se, or certainly not as I understand it (26 years in the UK, 10 years in the UK). The model couldn't fly in the UK.
We dont need to hold our breath and wait to see what they do. Alpha already tried to use their model with other demographics and has failed miserably! They just don't advertise that. Out of the 30 students that started at Alpha Brownsville in 2022, of which a majority of them were Low SES students, only 2 remained to the end of the 3rd year. They also attempted to use their model in prisons and failed miserably as well. They blamed families for their 1x learning results in their homeschool program. I can keep going. There comes a point where it fails to be a startup and it starts being a failed method.
Even among the results they claim, I would love if Alpha had ANY third party review or peer review of their "research". Instead, they seem to be doing the opposite - chuck some NWEA pdfs into an LLM, copy/paste the output and call it a day. They've only publicly talked about the results from Alpha Austin as well, which they show kids come IN to the school close to the 99th percentile. So i'm not sure we've seen any public evidence that Alpha School works even for the population using it today.
Yeah I read in the Wired piece a while back that they initially agreed to share raw data, but later decided not to share it. I don't want to write a post that says "what if Alpha School is lying" because I have no evidence either way. Also, there's no way they do this "give deep insights" thing via LLM internally. I'm sure there's a ton more analysis they're doing.
You'd think - but if you read the central whitepaper on which they hang the whole 2x learning claim, you'll see that they get that conclusion by using the entirely wrong column from their NWEA report (2x is the average of the Growth Standard Error, not the growth itself). I notified them of this last July but they still have it up on their website.
I agree with this but I think we should also acknowledge that they share wildly more data than any other private school. Most private schools do not even make a claim of accelerated learning, as what they are selling is mainly peer effects.
Do you have strong feelings about loosely defined learning? As in, in year 4 maths there are concrete outcomes in the curriculum and that's what you're aiming for. There may be value in not doing things like that all day, so some time might be spent doing fuzzier things like exploring various careers by having a general discussion about how bananas get onto grocery shelves or something. I haven't explained this well. I think it's a comment/question about your statement "school should be about learning" and whether "the curriculum" should be tacked onto the end.
My current thought is that about 90-95% of school learning time should be pretty focused on the curriculum, and the remaining 5-10% should be spent doing fuzzy stuff. That's about where I am in my class right now. I could be convinced to move those numbers around a bit if a school is very successful teaching the core curriculum but the basic split feels right to me. I'm partial to what I wrote about a few months ago and called "glimpses": https://fivetwelvethirteen.substack.com/p/glimpses
That said, I think here having a diverse staff of teachers with different interests is good. Teachers should be accountable for teaching the curriculum, spend most of their time focused on that goal, but also be given 5-10% leeway to do fun stuff they feel passionate about. Glimpses are what I'm passionate about, but for someone else it will be personal finance or whatever and that's great.
I taught high school for a long time and retired a couple years ago. The one indisputable result of the NCLB earthquake and subsequent aftershocks was the heightened tendency of various players to make big claims that, upon closer inspection, turned out to be essentially hype. But it's actually funny - in kind of a nice way - that 'direct instruction' is having a moment.
Very much appreciate the thoughts and discussion. I'm interested in how we are defining learning. What exactly are the kiddos in the video "showing off"? It feels like given a question asked in a particular way, a few of them can say the words they were taught to say. Is that learning? What would we be comfortable claiming the kiddos in the video have learned? At Alpha, some kiddos can score very highly on a thing called MAP. Alpha is equating learning with scoring highly on MAP. Is that ok?
I agree with you that both Direct Instruction and Alpha School are focused on relatively narrow outcomes. That's a criticism I've made of DI before. That said, in Project Follow Through, Direct Instruction had better results on both basic skills and problem-solving then all of the other curricula. Basic skills are an ingredient in higher-order thinking. Not the whole thing, but lots of students struggle with higher-order skills because they lack foundational knowledge. I would also emphasize here that those are kindergarten students. Rote or not, what they're doing is pretty impressive for kindergarteners and far beyond a typical kindergarten student.
What we measure is what ends up getting valued, so I definitely wonder if Alpha School is narrowing their focus on MAP in an unhealthy way. Still, Alpha School students are doing pretty cool stuff in the afternoons. Parents are choosing the school. If the goal is to get the basics down in the morning and then go practice competitive debate or chess or whatever in the afternoons, that sounds like an education many families would choose.
The kids are following a script. I dare you to watch those interviews without audio and tell me what their body language is telling you.
I just watched it without the sound as you recommend and I disagree. There is definitely some practiced call-and-response (that's a big part of DI) but I find this impressive, especially considering these are kindergarten students.
Engelmann himself makes clear at the start of the video that this is not an example of teaching, but rather students "showing off" what they have already learned. It’s also worth noting that the children shown do not appear to be at kindergarten level; they look closer to the end of the program, around Year 3.
In addition, official evaluations of Direct Instruction (DI) show considerable variation in outcomes. Richard Anderson, who directed the results analysis, concluded:
“The Direct Instruction Model achieved this heroic level of performance with only a minority of its groups. If educators do know how to close the achievement gap between disadvantaged children and the nation in general, Follow Through has not proved they can do it consistently.”
Given this context—and Engelmann’s background in marketing—it’s reasonable to suspect that the students featured were selected because they showcase the program at its very best.
Engelmann did these kind of live demonstrations all the time. Nobody present ever mentions a script, except the teachers one.
excellent stuff, dylan
Thanks Adam!
it's kicking off on Twitter if you are interested...lots of people seem very invested in AS's success...here if you are interested https://x.com/adamboxer1/status/2024867703463268611?s=20
Thanks for sharing! Good point about the "testing on healthiest patients first metaphor" though I agree with you that phase I is probably not necessary in education. There are certainly plenty of "experiments" similar to this one being conducted in lots of schools already.
"...and for those students the program accelerates their learning faster than what they would achieve in a typical school. Is it 20 percent? 50 percent?"
If it's not 100% it means that inequality in life outcomes for students will increase. And that will generate a lot of resistance to Alpha from some people.
The real danger is admins taking Alpha School's AI hype at face value, buying some blinkered program at an inflated price, and using it as a justification to slash staff.
I agree. I could imagine a nightmare scenario where admin get rid of teaching positions, hire paraprofessionals to supervise students on Timeback, most students spend the time screwing around or playing computer games, and it's all a complete disaster. The worst part is that I bet some students would still learn, and that could be used to justify keeping the whole program running.
Apologies for the vaguepost, as it's not my place to make any official statements or revelations -- that said, regarding your conclusion:
"If Alpha School is serious about being “the future of education,” the solution is simple. I’m seeing these allusions to Engelmann. Follow in his footsteps. Partner with a struggling school where outcomes are poor. Test the curriculum there. Refine and iterate with the students who are most likely to struggle."
I suspect you will be eating a bit of crow in less time than you think :)
I look forward to the announcement!
I’m not an alpha booster, but don’t their results in Brownsville and Texas sports academy completely invalidate this thesis? I don’t think parents sending their kids to TSA (for free with vouchers) because they want to become d1 athletes are subject to particularly strong academic selection effects
My impression is that a lot of the low-income families in Brownsville left the school. I don't know many details about TSA but "free private school via voucher" is very different from "seek out students who are struggling academically." I would be curious to see those demographics and outcomes broken out separately from what Alpha has shared about the program as a whole. I think Brownsville especially could have been a great testing ground but my impression from the Wired story (unfortunately paywalled) is that a bunch of students had trouble keeping up academically and that led to departures.
How do you *know* students are learning faster?
Where are the data?
I've looked to no avail... I can't find the metrics they're using. Please can you expand on the basis for your assessment?
Thanks!
Here's where Alpha School shared their mid-year NWEA MAP data: https://futureofeducation.substack.com/p/alphas-mid-year-report-card-is-in
In a fall Wired piece (https://www.wired.com/story/ai-teacher-inside-alpha-school/) Alpha School initially said they would share raw data but did not follow through. I haven't seen any independent analysis so I'm trusting what they say.
To answer your question: I don't know students are learning faster. Maybe Alpha School is lying, or cherrypicking, or something. I have seen a number of testimonials, and the data is plausible to me given how they run their program. I'm sharing some skepticism about their program assuming the results are real. Totally reasonable to doubt the results themselves.
Thanks. I'm not familiar with report cards. You can't just put the outcomes... you have to put the inputs..? Otherwise it's meaningless and just reflects your cohort and not the added value that you methods have conferred?
When I put something on Twitter the Principal said 'they do well on standardized tests'.
The question is:
- Do they do well on standardized test because they are at the Alpha School
or
- Are they at the Alpha School because they are the kind of student who does well on standardized test?
AND
Would they do well anyway independent of where they are... because they're in a family who can afford $40k a year, which itself confers a number of significant educational advantages?
It's also not a 'curriculum' per se, or certainly not as I understand it (26 years in the UK, 10 years in the UK). The model couldn't fly in the UK.
Good post. There's something sad about advertising for Alpha Schools.
We dont need to hold our breath and wait to see what they do. Alpha already tried to use their model with other demographics and has failed miserably! They just don't advertise that. Out of the 30 students that started at Alpha Brownsville in 2022, of which a majority of them were Low SES students, only 2 remained to the end of the 3rd year. They also attempted to use their model in prisons and failed miserably as well. They blamed families for their 1x learning results in their homeschool program. I can keep going. There comes a point where it fails to be a startup and it starts being a failed method.
You can read my substack for a first hand account on the Brownsville experience.
I read it, thank you for sharing! Alpha School has a long way to go if they want to truly serve all students.
Even among the results they claim, I would love if Alpha had ANY third party review or peer review of their "research". Instead, they seem to be doing the opposite - chuck some NWEA pdfs into an LLM, copy/paste the output and call it a day. They've only publicly talked about the results from Alpha Austin as well, which they show kids come IN to the school close to the 99th percentile. So i'm not sure we've seen any public evidence that Alpha School works even for the population using it today.
Yeah I read in the Wired piece a while back that they initially agreed to share raw data, but later decided not to share it. I don't want to write a post that says "what if Alpha School is lying" because I have no evidence either way. Also, there's no way they do this "give deep insights" thing via LLM internally. I'm sure there's a ton more analysis they're doing.
You'd think - but if you read the central whitepaper on which they hang the whole 2x learning claim, you'll see that they get that conclusion by using the entirely wrong column from their NWEA report (2x is the average of the Growth Standard Error, not the growth itself). I notified them of this last July but they still have it up on their website.
Check out more details here:
https://www.theaiwhotaughtme.com/e/7-is-alpha-school-really
Makes me question that they have anyone reviewing their results with a critical eye, even internally...
I agree with this but I think we should also acknowledge that they share wildly more data than any other private school. Most private schools do not even make a claim of accelerated learning, as what they are selling is mainly peer effects.
I have thoroughly enjoyed this series, thank you!
Do you have strong feelings about loosely defined learning? As in, in year 4 maths there are concrete outcomes in the curriculum and that's what you're aiming for. There may be value in not doing things like that all day, so some time might be spent doing fuzzier things like exploring various careers by having a general discussion about how bananas get onto grocery shelves or something. I haven't explained this well. I think it's a comment/question about your statement "school should be about learning" and whether "the curriculum" should be tacked onto the end.
My current thought is that about 90-95% of school learning time should be pretty focused on the curriculum, and the remaining 5-10% should be spent doing fuzzy stuff. That's about where I am in my class right now. I could be convinced to move those numbers around a bit if a school is very successful teaching the core curriculum but the basic split feels right to me. I'm partial to what I wrote about a few months ago and called "glimpses": https://fivetwelvethirteen.substack.com/p/glimpses
That said, I think here having a diverse staff of teachers with different interests is good. Teachers should be accountable for teaching the curriculum, spend most of their time focused on that goal, but also be given 5-10% leeway to do fun stuff they feel passionate about. Glimpses are what I'm passionate about, but for someone else it will be personal finance or whatever and that's great.