AI School Promised Faster Learning. 7 Questions to Ask Before You Believe It

The New York Times ran a headline this fall that should be taped to the door of every AI-school open house: "Students at an A.I. School Were Supposed to Learn Faster. Many Aren't." The school promised acceleration through AI-driven personalization. According to the headline, many students are not learning faster.

We can't give you the school's name, the test scores or the sample size, because the reporting we were able to retrieve supplied only the headline, not the body text. Read the full story for the numbers. The headline alone tells us what matters: the promise and the result came apart.

"Faster" Is a Claim, Not a Result

Edtech has been selling speed for years. Inc. headlined a piece on edtech funding that promises you'll learn 2x faster, and the tagline is the same one you hear on school tours.

But "faster" is an empty word until someone tells you faster at what, measured by whom, compared with which students. Nobody selling an AI-accelerated program wants those three questions asked at full volume. You can ask them in a five-minute call.

Here is the list. Use it on any school, program or app that says AI makes learning quicker.

Eagle’s pro tip

Policy changes before anyone announces it

The rule that affects your kid is usually settled in a board meeting months before a letter goes home. I read those. Get it while it still matters.

Subscribe today

The Parent Question Kit

1. What is the comparison group?

Why it matters: "Our students gained a year in six months" means nothing without a control. Students improve over time on their own. A result only counts if similar students without the AI tool were tracked in parallel, ideally assigned by lottery.

• Good answer: "We compared enrolled students to a matched or randomly assigned group using the same curriculum without the AI component."
• Red flag: "We compared them to national averages" or "to where they started." That is a before-and-after photo, not an experiment.

2. Which test, and who administered it?

Why it matters: A vendor grading its own homework gets A's. Assessments built into the product measure whether students got good at the product. An independent, externally normed exam measures whether they learned the subject.

• Good answer: A named third-party assessment, given by an outside party, with results the school will share.
• Red flag: "Our internal assessments show…" or "students are mastering 140% of grade-level content." Ask who wrote the test.

3. Over what time frame, and was it measured after the novelty wore off?

Why it matters: New tools get a bump. Students are curious, teachers are energized, and the first eight weeks look great. The honest question is what happens in month eight.

• Good answer: A full academic year or more of data, with results reported by semester.
• Red flag: A "pilot" of a few weeks, or a testimonial from the launch month.

4. How many students, and how many left before testing?

Why it matters: If 200 students started and 120 are left at test time, the 80 who quit are missing from the average. Programs that lose their struggling students will always look like they work.

• Good answer: Enrollment numbers at start and at test, with attrition stated plainly.
• Red flag: "Our students" with no denominator. A sample of 30 handpicked students is a story, not a study.

5. What does "learned" mean: finished modules or demonstrated mastery?

Why it matters: This one is where the speed claims hide. Clicking through a unit at 2x pace is completion. Being able to solve a fresh problem three weeks later is mastery. A student can sprint through a platform and retain very little.

• Good answer: Mastery measured on new problems, after a delay, without the AI helping.
• Red flag: "Students complete grade-level content in half the time." Completion is a progress bar, not an education.

6. What does the human teacher do all day?

Why it matters: If the AI is the instructor, you are buying software with a supervisor. If the AI handles practice and the teacher handles explanation, motivation and catching students who are quietly lost, that is a different product. Ask for the daily schedule.

• Good answer: A specific description of live instruction, feedback on student work and one-on-one time, with the teacher's role defined by task.
• Red flag: "Guides" or "coaches" who monitor dashboards, plus a vague line about "supporting the learning journey."

7. What does it cost per student, compared with district per-pupil spending?

Why it matters: Premium pricing needs premium evidence. If the program costs far more than what your district spends per student, the gain has to be large enough to justify it. If it costs less, ask what was cut to get there.

• Good answer: A clear cost breakdown, including fees, devices, and what the program charges for if results fall short.
• Red flag: Price tied to the "transformation" rather than to a stated, measurable outcome. No refund or exit policy if the promised gains don't show.

What the Research Actually Looks Like

A useful AI-in-education evidence base does exist, and it is growing. Nature has published a meta-analysis of 35 experimental studies on ChatGPT's impact on student learning outcomes. We could not retrieve the paper's text, so we are not going to quote you effect sizes or a publication date. Check them before you cite it to anyone.

The structure is what you want to look for: experimental designs, pooled across dozens of studies, rather than one vendor's case study. When a school can point to independent randomized work, with study size and year stated, that is the strong end of the evidence ladder. When it can only point to its own dashboard, that is the weak end.

The Manhattan Institute has also published a framework titled "One Size Does Not Fit All," arguing for developmentally appropriate AI use in K-12. We could not retrieve its text either, so treat the title as a pointer to read, not a finding to repeat. The idea in the title is still worth carrying into your questions: what works for a 17-year-old may not suit an 8-year-old, and a school should be able to tell you where it draws that line.

It's Not Just K-12

If you think school administrators are the only ones unsure how to measure AI's classroom effect, look at higher ed. Chemical & Engineering News has reported that college STEM faculty face the same uncertainty about how AI is changing learning in their courses. We did not retrieve that article's text, so we won't characterize its specifics.

The parallel stands without them. The professors who teach the subject and grade the exams are still working out whether AI helps, hurts or just shifts the work around. If they don't have the answer, the sales team at a startup school does not have it either.

The Bottom Line

AI tutoring may well help with specific skills under specific conditions: practice with immediate feedback, for example, alongside a good human teacher. Nothing in the NYT headline says otherwise, and we are not anti-technology here.

But the burden of proof sits with the seller. "Faster" has to come with a comparison group, an independent test, a real time frame and honest attrition numbers. A school that welcomes those questions has probably done the work. A school that gets defensive has told you what you need to know.

You lose nothing by asking. If you're weighing a flexible, human-taught alternative, see how an online high school near you works, and pressure-test it with the same kit. Our guides on vetting online schools for student athletes and what "CTE works" actually means use the same method. And for a reality check on how far achievement has recovered, read why returning to 2019 test scores isn't good enough.

Source note: the New York Times, Inc., Nature, Manhattan Institute and C&EN items are cited by headline and publisher only. We could not retrieve their full texts, so we have not reported figures or findings from any of them.