Anomalies Wanted: When Test Scores Don't Predict Everything

Clay Christensen was known for, among other things, a hand-crafted sign that hung outside his door. It read: “Anomalies Wanted.”

Clay loved anomalies—things that defied a prevailing theory or conventional wisdom. Rather than brush them aside, he saw anomalies as chances to learn and sharpen one’s understanding of how the world worked.

If a theory couldn’t explain an anomaly, then something about the theory must be incomplete. Maybe the categorization scheme wasn’t quite right. Maybe it misunderstood causality. Or perhaps the anomaly occurred in a different circumstance that should refine the boundaries of the theory to create a series of “if-then” statements (as in—if you’re in this circumstance, then the following holds; but if you’re in this one, then something else applies).

Either way, the discovery of an anomaly—what

Adam Grant

might call discovering that you were wrong or that you didn’t know something—was an opportunity to learn and improve the predictive power of a practical theory.

I thought of all that when I read Marty Lueken’s recent piece for

Informed Choice

, “Test Scores Matter—But Not How You Think.”

The piece opened by citing the recent article in Education Next, “The Predictive Power of Standardized Tests,” which “highlights the strong relationship between middle school test scores and long-run educational outcomes such as college attendance, degree attainment, and earnings.”

But then Lueken points out an anomaly—or a paradox: “several studies of private school choice programs have found something different: students in choice program sometimes score lower on standardized tests, but end up doing better in the long run.”

He notes that we might not understand causality—"just because early test scores predict later success doesn’t mean that raising test scores necessarily causes greater success”—and that we might not be categorizing the circumstances a child is in correctly. As in, test scores may matter more for a child in a public school than in a private school.

Why would that be?