Blog Archives
GlassLabs releases research on Psychometric Considerations in Game-based Assessment
Press Release: GlassLab Publishes Research on Game-based Assessment
For Immediate Release
February 6, 2014
Digital Games Can Improve Measurement of Student Learning With Continuous Assessment, According to New Research From GlassLab
New white paper offers framework for integrating game design and educational assessment
Redwood City, Calif. – February 6, 2013 — Researchers have figured out a new way to give teachers a dynamic portrait of a student’s learning in action, using video games. In a white paper released today by Institute of Play project GlassLab (the Games, Learning and Assessment Lab), a team of assessment data scientists, learning designers and game developers describe a multidisciplinary approach to designing a new type of classroom game — a game-based assessment. Game-based assessments can provide a rich understanding of the different factors that affect educational achievement and predict how a student’s performance might change over time.
The white paper, “Psychometric Considerations in Game-Based Assessment,” answers the provocative questions that stand in the way of realizing the full potential of games to transform learning and assessment: How can scientists make sense of the endless stream of data generated by a digital game — the entire spectrum from wayward mouse clicks to strategic choices in gameplay? How can psychometric data help game designers build better challenges to improve learning outcomes? And how can experts in diverse fields come together to build and test new game-based assessments?
“Game-based assessments may hold the promise of a richer, multi-dimensional portrait of student learning, but they also present a new frontier in assessment design, ripe with challenges and opportunities for psychometricians and game designers to explore collaboratively,” says co-author Robert Mislevy, a psychometrics consultant for GlassLab, pioneer of evidence-centered assessment design, and Frederic M. Lord Chair in Measurement and Statistics at ETS. “This paper provides a framework for the continued exploration of this new frontier and proposes a design approach for developing and testing new game-based assessments.”
“Psychometric Considerations in Game-Based Assessment” is the first publication from GlassLab, and contains findings from the development of the Lab’s first game-based assessment product, SimCityEDU: Pollution Challenge!, which launched in November 2013. GlassLab’s research and development efforts are made possible by the generous support of the John D. and Catherine T. MacArthur Foundation and the Bill & Melinda Gates Foundation.
A project of the nonprofit Institute of Play, GlassLab is an interdisciplinary partnership between leaders in commercial games and experts in learning and assessment to develop next-generation educational games. Co-authors of “Psychometric Considerations in Game-Based Assessment” include researchers from Institute of Play, Educational Testing Service, Electronic Arts, and Pearson’s Center for Digital Data, Analytics and Adaptive Learning.
The 160-page white paper is available for free download today from Institute of Play. To download the full white paper and Executive Summary as a PDF e-book, or to explore print-on-demand options, please visit http://bit.ly/glasslab-research.
The next white paper from GlassLab, scheduled for publication in Fall/Winter 2014, will detail GlassLab’s Evidence-Centered Game Design process for developing game-based assessments.
Making Meaningfull Learning Games
John Flinchbaugh/Flickr
Katrina Schwartz writes that:
“Proving the effectiveness of a game is no simple proposition. Games are all different. When students play a game that focuses on a particular skill, often called “drill-and-kill” games, they often see improved test scores. But those games don’t necessarily get at the higher order thinking skills that games have the potential to help develop. But more complex games often don’t offer clearly correlated outcomes that a teacher can use to prove effectiveness.
The debate has led some game developers and their funders to focus on how the game itself can be used to assess learning — how the game can become a test. But White says that also changes the nature of the game. When a child sits down to play a game, he assumes it’s “failure agnostic” and that allows him to experiment and meander his way through the game freely. When assessment is involved, it changes his relationship to the game. White isn’t against using games for assessment, but he doesn’t think anyone has really figured out how to accurately use the data that games provide to assess in a meaningful way.
Glasslab’s SimCityEDU, funded by the Gates and MacArthur Foundations, is the highest profile example of gaming for assessment. White says if they can crack the code he’ll license it from them, but it’s a lot easier to pull analytics and metrics for a single game than it is to generalize those principles into something that could be applied to all educational games. And, when assessment is involved, the price of developing a game skyrockets.” To read the full article on Mindshift, click here;
It is good to read a discussion of Higher Order Thinking Skills in games and gaming for assessment. These are two important issues that need more attention and development. Unfortunately these games have too often targeted the lower order thinking skills of Blooms’ taxonomy. In 2007, John Rice conducted a study which assessed higher order thinking skills in video games with the aim of developing a tool for parents and educators to rate the level of higher order thinking skills in video games. He found that the educational efficacy of these games varies greatly with respect to higher order thinking skill. (Rice, 2007, p. 88).
Rice began his study by examining; “lower level learning in so-called edutainment products and concludes with an example of an advanced social studies simulation that fosters higher order thinking. [Rice] identifies characteristics of highly cognitive virtual interactive environments and offers a detailed index and scoring rubric as a tool for teachers and preservice teachers to use when evaluating the tendencies a video game demonstrates toward encouraging higher order thinking in its participants (Rice, 2007, p. 87).”
When educators are looking for a game that does more than drill facts in to the brains of students, Rice’s rubric can serve as a helpful tool in selecting a game that lifts students to the top of Blooms’ taxonomy (1956).
References:
Bloom, B. S. (1956). Taxonomy of educational objectives, handbook 1: Cognitive domain. New York: Longmans Green.
Rice, J. W. (2007). Assessing Higher Order Thinking in Video Games. Journal of Technology and Teacher Education, 15(1), 87-100.