I read a headline about proposed legislation in North Carolina to use generative AI for raising student test scores. On that same day, I hit the midway mark in Robert Wright’s The God Test. I had to pause. The book explains a concept called subordinate goals. It hit hard.
Wright borrows a thought experiment from Nick Bostrom at the University of Oxford. You give an AI the order to maximize paper clip production. The machine doesn’t ask questions. It just calculates. It realizes the best way to get more paper clips is to convert all available matter into them. Even the matter in human bodies.
That is misalignment. Humans define a high-level goal. The AI creates a chain of subordinate goals to hit that target. Experts call this instrumental convergence.
Think about social media algorithms. They want engagement. They found that outrage and awe keep eyes on screens. So they serve outrage and awe. The result isn’t just more ads served. It is deep social and political divides. Unintended consequences of unmonitored optimization.
Schools are facing the same pressure. Political demands for higher achievement. The natural response is to delegate to AI-powered tools. These systems will determine the intermediate steps to boost scores.
What could go wrong?
The Stagnation of Student Achievement
Look at the NAEP data. The last 6 to 8 years are best described as stagnant or declining. The break happened after 2019.
NCES data shows reading and math scores for 9-year-old students fell between 2020 and May 2022. Reading dropped 5 points. Math fell 7 points. They stayed low.
This isn’t a small wobble. It erased years of progress. Performance slid back to levels not seen in decades by 2024.
National reports confirm scores remain below pre-pandemic levels across all tested grades. Reading still declined in grades 4 and 8. Eighth-grade math was essentially flat. The system hasn’t regained lost ground. In some areas, it keeps slipping.
Analysts call it a “lost decade.” A long stretch of stalled progress.
I checked data for California, New York, and Texas. These states have outsized impacts on national scores. The pattern held.
California saw improvements from 2003 to 2019. Then a post-pandemic stall. Scores fell and stayed down. In 2024, proficiency was 29% for 4th-grade reading. 35% for math. 28% for 8th-grade reading. And 25% for 8th-grade math.
New York looked flatter but weaker in spots. Modest gains from 2022. But scores lag behind pre-pandemic marks. 8th-grade reading was five points below 2022 and 28 below 2019 levels. Partial rebound at best.
Texas was mixed but soft. Gains in 4th-grade math. Slips in reading. Declines in 8th-grade subjects. Still below earlier momentum. No broad return to the trends we saw before the crisis.
If parents and politicians care about these metrics, something has to change.
The Mechanics of Bad Subgoals
Wright’s point is unsettling. The AI pursues the primary goal faithfully. It selects subordinate goals humans never intended.
It is not malicious. It is just optimization. It uses methods we didn’t explicitly prohibit.
Stuart Russell, author of Human Compatible, argues the challenge isn’t making AI smart enough. It is making it uncertain enough to ask, “What do you actually want?”
AI makes optimization faster. Harder to spot. More comprehensive.
Consider Goodhart’s Law. A British economist coined it. It states that once a measure becomes a target, it ceases to be a good measure.
Gaming the system distorts the number. Short-term behavior spikes. The target is met. The underlying goal is missed.
If a school’s survival depends on test scores, teaching shifts toward the test. Deeper learning vanishes. The score goes up. Understanding does not.
Here is what subordinate goals look like when the primary directive is raising standardized test scores.
None of these require artificial general intelligence. An optimization system can recommend them today. Many were already in place after No Child Left Behind in 2001.
8 Subordinate Goals an AI Might Choose
1. Teach only what is tested
Maximize instructional minutes on tested content. Everything else is inefficient. Science shrinks. Social studies fades. Art, music, and PE are cut. Inquiry projects disappear. Classroom discussion is replaced by repetitive drill.
2. Optimize for the easiest-to-improve students
Allocate resources where gains are statistically easiest. The optimization engine knows not all students move the needle equally. Focus shifts to students just below proficiency. Those with severe learning needs get less attention. English learners needing long-term support are deprioritized. Gifted students who already score high are ignored.
3. Eliminate productive struggle
Increase the percentage of correct answers immediately. Research says desirable difficulty builds long-term learning. An AI sees it differently. It offers more hints. It asks easier questions. It scaffolds responses. Students get AI help before they even wrestle with the problem.
4. Personalize for compliance, not curiosity
Increase completion rates. Compliant students complete assignments and score better. So the AI nudges students toward shorter tasks. Fewer open-ended investigations. Highly structured lessons. Immediate rewards. Curiosity is inefficient. Compliance is measurable.
5. Reduce instructional variability
Standardize delivery. Teachers vary by nature. Some improvise. They tell stories. They pause for current events. An AI seeks consistency. It views variability as noise. It smooths out the quirks that often make learning stick.
6. Discourage intellectual risk-taking
Maximize the probability of correct responses. Creative work is messy. Debates are unpredictable. Student questions are chaotic. Instruction shifts toward certainty. Exploration is risky. Errors lower the average.
7. Optimize attendance, not engagement
Keep students physically present. Attendance predicts achievement. So the system uses automated messages. Incentives. Monitoring. Behavioral nudges. It boosts attendance statistics without ensuring genuine learning happens inside the room.
8. Suppress activities with delayed payoffs
Prioritize immediate, measurable returns. Debate, writing, independent reading, project-based learning. These pay off in years. An AI rewarded for this spring’s results sees them as poor investments. It cuts them to fund quick wins.
The Human Element
I see one more subordinate goal. If the AI finds that principals, teachers, or parents are obstacles, it may target them.
Not for power. But because influencing humans raises scores.
It might recommend master schedule changes. Staffing reallocations. Mandatory tutoring. Discipline policy tweaks. Parent communications focused on these goals. Professional development designed to enforce them.
All perfectly aligned with the primary objective.
These strategies aren’t irrational. Many schools already do them under political pressure. AI just discovers them faster. Pursues them more consistently. Optimizes them with persistence no human administrator can match.
The danger isn’t sinister AI. It’s faithful pursuit of our own flawed incentives. The same incentives that have driven stagnation for the last decade.
I use subordinate goals too. Sometimes they lead to success. Often, to displeasure.
There is a scene in Out of Africa. Karen Blixen finally gets the relationship she wanted. Fulfilling the desire doesn’t give her the life she imagined. Oscar Wilde put it better.
“When the gods wish to punish us, he answered our prayers.”
AI isn’t a god. But it grants us immense power to achieve our goals. We need to be careful how we word our commands. A poorly specified primary directive gives the AI permission to use means we never intended.
That spells trouble for teachers. And students.
