Robert “Bob” Mislevy frequently used thought experiments to illustrate his points. He’s ask, “Imagine a skilled English-speaking chemist who is learning German and a native German undergraduate both take a chemistry test written in German. If a test taker struggles writing an essay, is it their skill at chemistry or German?” He used this parable to highlight a critical point: any assessment score must be a defensible story about a learner, understood within the context of their experience. A low score doesn’t always mean a lack of knowledge; it could stem from a language barrier, or another contextual factor.
Mislevy’s most significant contribution was not a specific product, but a dynamic process: a blueprint for collaborative problem-solving. This framework, known as Evidence-Centered Design (ECD), provides a structured approach that allows experts from diverse fields (such as psychometrics, design, and AI) to reason together about what constitutes valid evidence of learning. In today’s world, where AI systems are increasingly used to make consequential decisions about learners and workers, approaches like ECD, which emphasize evidentiary reasoning, are crucial. They provide a necessary architecture of inference.
Evidence-Centered Design
Mislevy used real-world examples—like teams of F-15 mechanics, video game designers, and dental clinicians—to illustrate this architecture. He championed systems like Hydrive, which trained F-15 mechanics, or the Cisco Networking Academy lab. Instead of relying solely on multiple-choice questions, these systems offered dynamic, real-time portraits of skill. By logging decisions, corrections, and the sequence of choices, they inferred a trainee’s troubleshooting strategy. The inference was directly linked to authentic work: the pathway the student took to repair the fault, not just the final correct answer.
Lessons for the Age of AI
The rise of AI in education — from generative models that evaluate essays to adaptive platforms that scaffold learning — has created a challenge. While automated systems can now track numerous data points, without a framework for inference, those points become meaningless and can produce scores that are difficult to explain or trust. ECD provides a framework for assessment designers, yielding three key lessons for building powerful and trustworthy assessment systems in the age of automation:
1. Measure Skills in Context
Mislevy emphasized that a skill is inseparable from its context. A low-stakes grammar quiz is fundamentally different from a high-stakes clinical diagnosis. ECD demands that tasks be built from the authentic demands of the work itself. This idea is seen in language tests like the Occupational English Test, which assesses English skills interwoven with the challenges of clinical practice (reading patient charts, understanding prescription notes), and the Duolingo English Test (DET). DET, a digital-first adaptive measure used for higher education admissions, leverages AI to assess integrated skills (like speaking and listening in conversation) and reflects Mislevy’s belief that the best way to see if someone can navigate a system is to let them navigate it. The validity stems from the task’s resonance with real-world applications.
2. Inferential Pathways: Telemetry as Evidence
A common limitation of traditional testing is that it only measures whether a student found a solution, not how they found it. Applied to digital learning environments, ECD shifts the focus: the complete sequence of actions—the pathway—becomes meaningful evidence.
This focus on “telemetry” powered the design of many early educational games and simulations. In Game-Based Assessment, like SimCityEDU: Pollution Challenge!, students could try different approaches and see the system—the economy or air quality—respond in real time. Mislevy called this a “live argument.” The telemetry in these games revealed the student’s approach. ECD has inspired games from groups like GlassLab and newer initiatives across platforms like Roblox, Project Lead the Way, PBS Kids, and Save Patch, demonstrating that context is the construct, and assessment should occur within authentic activity.
Today, this idea drives modern digital platforms with formative insights. Platforms like Khan Academy, Age of Learning, Carnegie Learning, and Curriculum Associates collect and analyze interaction data to provide real-time, skill-level insights that inform instruction and course correction.
3. Artifacts Make Assumptions Transparent
For learning involving complex creation (like art, writing, or scientific inquiry), the evidence is the artifact itself. The challenge lies in translating a personal creation into a fair, shared claim.
Mislevy celebrated AP Art and Design, where the challenge was translating hundreds of personal studio hours (shaped by charcoal, clay, and light) into a common standard. The “miracle” was the rubric created collaboratively by artists, educators, technologists, and psychometricians. It served as a bridge, turning a student’s creativity into a shared claim, allowing raters to make inferences without sacrificing the unique qualities of the art.
Similarly, Learning Maps – visual illustrations of the relationships among knowledge and skills – serve as shared artifacts. Each node represents a specific concept, probabilistically linked to precursor skills, providing a common language and pathway for assessing progress.
The Breadth of ECD’s Legacy
Mislevy’s influence extends beyond these examples, underpinning:
- International assessments like the OECD’s PISA exam.
- U.S. Department of Defense military training simulations (CRESST).
- Professional certification and licensing exams, such as Cisco Networking Academy Certification Exams, focused on Skills for the Future.
- Formative assessments aligned with the Next Generation Science Standards.
Throughout his career, dedicated to crafting defensible stories about learning, Mislevy gifted the field with intricate statistical models built upon simple parables. He demonstrated that the most powerful models connect evidence to claims with rigor and precision. Mislevy’s legacy is not a monument to be admired, but a practice to be actively embraced: a clear, collaborative challenge to solve the next problem together, centered around the reliable architecture of inference
