Assessing Serious Games: The GRAND Assessment Framework

Domini Gee, Man-Wai Chu, Simeon Blimke, Geoffrey Rockwell, Sean Gouglas, David Holmes, Shannon Lucky

Abstract


The videogame industry is a considerable market: in 2012, the industry was worth over $86 billion USD and about seventy-two percent of American households play videogames. It is unsurprising, then, that commercial and educational developers and/or researchers have sought to capitalise on videogames. Games and simulation technologies have been used for educational purposes for thousands of years prior to the digital era (Gee 2007). Digital games, however, offer many new affordances including increased accessibility, reinforced automation (i.e., fair and consistent application of rules), embedded data-gathering for assessment, dynamic adaptation to student needs, the ability to simulate complex situations for student inquiry in a safe context, and reduced overall costs (Jin and Low 2011). However, it is difficult to assess the process of serious game development and effectiveness of educational play. Many serious games retrofit assessment late into the project, creating a gap between original intents and the game's current uses, limiting effectiveness of measuring and meeting the project's goals. As such, we propose an assessment framework that synthesises work from various fields (educational assessment, game design, usability, project management) that aims to guide researchers and game developers through a project from its inception to the end by presenting specific topics to address and questions to answer throughout the game design phase of the project. By building assessment into the game development from the get-go, original intents and a game's current uses can more closely align, allowing for stronger, purposeful games.

L’industrie du jeu vidéo est un marché appréciable. En 2012, elle dépassait 86 milliards de dollars US et environ soixante-douze pour cent des ménages américains jouent aux jeux vidéo. Il n’est pas surprenant alors que des concepteurs et des chercheurs de la sphère commerciale et éducationnelle ont cherché à tirer profit des jeux vidéo. Bien avant l’avènement de l’ère numérique, jeux et technologie de simulation étaient utilisés à des fins pédagogiques (Gee 2007). Le jeu vidéo offre toutefois de nouvelles affordances : accessibilité accrue, automatisation renforcée (c.-à-d. l’application juste et systématique des règles), collecte de données intégrée pour l’évaluation, adaptation dynamique aux besoins des élèves, possibilité de simuler des situations complexes pour l’expérience de recherche de l’élève dans un contexte sécuritaire et, finalement, réduction des coûts globaux (Jin and Low 2011). Il est toutefois difficile d’évaluer le processus de conception de jeux sérieux et l’efficacité du jeu éducationnel. De nombreux jeux sérieux intègrent l’évaluation tard dans le projet, créant un écart entre les intentions originelles et les utilisations actuelles du jeu, limitant ainsi l’efficacité à mesurer et respecter les objectifs du projet. À ce titre, nous proposons un cadre d’évaluation synthétisant les travaux dans divers domaines (évaluation pédagogique, conception de jeu, facilité d’utilisation, gestion de projet) qui guiderait chercheurs et concepteurs de jeux dans un projet, de son lancement à sa fin, en présentant des sujets précis à traiter et des questions à répondre pendant la phase conception du jeu du projet. L’incorporation dès le départ de l’évaluation dans la conception du jeu permettrait un meilleur alignement des intentions premières et des utilisations actuelles du jeu, permettant ainsi une expérience de jeux plus forte et plus significative.

References


Adams, E. 2009. "The Designer's Notebook: Sorting out the Genre Muddle." Gamasutra. Accessed August 19, 2013. http://www.gamasutra.com/view/feature/4074/the_designers_notebook_sorting_.php?page=2.

Annetta, L., and S. Bronack. 2010. Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds. Rotterdam: Sense Publisher.

ATCS21 (Assessment and Teaching of 21st Century Skills). 2009. What are 21st Century Skills?. Accessed August 19, 2013. http://atc21s.org/index.php/about/what-are-21st-century-skills/.

Beale, I.L., P.M. Kato, V.M. Marín-Bowling, N. Guthrie, S.W. Cole. 2009. "Improvement in Cancer-Related Knowledge Following Use of a Psychoeducational Videogame for Adolescents and Young Adults with Cancer." Journal of Adolescent Health 41.

Consalvo, Mia and Nathan Dutton. 2006. "Analysis: Developing a Methodological Toolkit for the Qualitative Study of Games." Game Studies 6 (2).

Chantam, R. 2011. "After the Revolution: Game-Informed Training in the U.S. Military." In Computer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher. Charlotte: Information Age Publishing.

Despain, Wendy and Keyvan Acousta. 2013.100 Principles of Game Design. Berkeley: New Riders.

DiCerbo, K. E, and J. T. Behrens. 2012. "Implications of the Digital Ocean on Current and Future Assessment." In Computers and Their Impact on State Assessment: Recent History and Predictions for the Future, edited by R. Lizzitz and H. Jiao, 273-306. Charlotte: Information Age Publishing.

ESA (Entertainment Software Association). 2011. 2011 Sales, Demographics and Usage Data: Essential Facts about the Computer and Videogame IndustryAccessed August 19, 2013.http://www.theesa.com/facts/pdfs/ESA_EF_2011.pdf.

ESA (Entertainment Software Association) 2012. 2012 Sales, Demographics and Usage Data: Essential Facts about the Computer and Videogame Industry.Accessed August 19, 2013.

http://www.theesa.com/facts/pdfs/ESA_EF_2012.pdf.

Fletcher, J.D. 2011. "Cost Analysis in Assessing Games for Learning." InComputer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher, 417-434. Charlotte: Information Age Publishing.

Gee, J. P.2007. Good Videogames + Good Learning: Collected Essayson Videogames, Learning and Literacy.New York: Peter Lang Publishing.

Gillispie, L., F. Martin, and M. Parker. 2009. "Effects of the Dimension-M 3D Video Gaming Experience on Middle School Student Achievement and Attitude in Mathematics." In Proceedings of Society for Information Technology and Teacher Education International Conference 2009, edited by I. Gibson et al., 1462-1469. Chesapeake: AACE.

Gierl, M. J., C. Alves, and R. Taylor-Majeau. 2010. "Using the Attribute Hierarchy Method to Make DiagnosticInferences about Examinees' Knowledge and Skills in Mathematics: An Operational Implementation of Cognitive Diagnostic Assessment." International Journal of Testing 10:318-341.

Hsu, T. Y. and G. F. Chiou. 2011. "Preservice Science Teachers' Prior Gameplay Experience and their Perceptions of Digital Game-Supported Learning." In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2011, edited by T. Bastiaens and M. Ebner, 3331-3339. Chesapeake: AACE.

Ito, Mizuko. 2009. Engineering Play: A Cultural History of Children's Software. Cambridge: The MIT Press.

Jin, P. and R. Low. 2011. "Implications of Game Use for Explicit Instruction." InComputer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher. Charlotte: Information Age Publishing.

Kirkle, Sonny E., Steve Tomblin, and Jamie Kirkley. 2005. "Instructional Design Authoring Support for the Development of Serious Games and Mixed Reality Training." In The Interservice/Industry Training, Simulation and Education Conference (I/ITSEC). Arlington: National Defense Industrial Association.

Knowledge Adventure. 2007. "About Knowledge Adventure." Math Blaster. Accessed August 19, 2013.

http://www.mathblaster.com/About.aspx.

Komoski, Ken. 1979. "Counterpoint: Learner Verification of Instructional Materials." Educational Evaluation and Policy Analysis 1 (3):101-103.

Komoski, Ken. 1984. "Formative Evaluation: The Empirical Improvement of Learning Materials." Performance and Instruction Journal 22 (5): 3-4.

Marsh, Herbert W. 1991. "Multidimensional Students' Evaluations of Teaching Effectiveness: A Test of Alternative Higher-Order Structures." Journal of Educational Psychology 83 (2): 285-296.

Media Smarts. 2012. "Educational Games." Media Smarts. Accessed August 19, 2013. http://mediasmarts.ca/digital-media-literacy/educational-games.

Messick, S. 1989. "Validity." In "Educational Measurement, 3rd Edition" ,edited by R. L. Linn, 13-103. New York: Macmillan.

Michael. David. 2003. The Indie Game Development Survival Guide. Hingham: Charles River Media.

Molka-Danielsen, J. 2009. "The New Learning and Teaching Environment." In Learning and Teaching in the Virtual World of Second Life, edited by J. Molka-Danielsen and M. Deutschmann, 13-26. Trondheim: Tapir Academic Press.

Pellegrino, J. W., N. Chudowsky, and R. Glaser. 2001. Knowing What Students Know: The Science and Design of Educational Assessment. Washington: National Academy Press.

Perry, David and Rusel DeMaria. 2009. David Perry on Game Design: A Brainstorming Toolbox. Boston: Charles River Media.

Play-Pr. 2012. "The Intelliphone Challenge: Fort Edmonton Park Geo-Locative Game." Accessed August 10, 2013. http://playpr.hexagram.ca/projects/.

Prensky, Marc. 2006."Don't Bother Me Mom, I'm Learning!": How Computer and Videogames Are Preparing Your Kids for Twenty-first Century Success and How You Can Help!.St. Paul: Paragon House.

Rice, John. W. 2007. "Assessing Higher Order Thinking in Videogames." Journal of Technology and Teacher Education 15 (1): 87-100.

Rollings, Andrew. 2003. Andrew Rollings and Ernest Adams on Game Design. Indianapolis: New Riders.

Rogers, Scott. 2010. Level Up!: The Guide to Great Videogame Design. Chichester: Wiley.

Saltzman, Marc. 1999. Game Design: Secrets of the Sages. Indianapolis: Macmillan Digital Pub.

Sauve, Louise. 2010. "Usability Guidelines for a Generic Educational Game Shell." In Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned,edited by David Kaufman and Louise Sauve, 416-434. Hershey: Information Science Reference.

Sauve, Louise, Lise Renaud, and Gabriela Hanca Elissaide. 2010. "Formative Evaluation of an Online Educational Game." In Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned, edited by David Kaufman and Louise Sauve, 416-434. Hershey: Information Science Reference.

Serious Games Initiatives. 2002. "The Serious Games Initiatives." Serious Games. Accessed August 19, 2013.http://www.seriousgames.org/.

Schell, Jesse. 2008. The Art of Game Design: A Book of Lenses. San Francisco: Morgan Kaufmann Publishers Inc.

Shute, Valerie. 2011. "Stealth Assessment in Computer-Based Games to Support Learning." In Computer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher, 503-524. Charlotte: Information Age Publishing.

Tobias, Sigmund, J.D. Fletcher, David Yun Dai, and Alexander P. Wind. 2011. "Review of Research on Computer Games." In Computer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher, 112-222. Charlotte, NC: Information Age Publishing.

Tobias, Sigmund, and J.D. Fletcher. "Computer Games, Present and Future." In Computer Games and Instruction, edited by Sigmund Tobias and J.D. Fletcher, 525-546. Charlotte, NC: Information Age Publishing.

Walker, Mark. 2003. Games That Sell!Plano: Wordware Pub.

Zapata-Rivera, D. 2012. Embedded Assessment of Informal and Afterschool Science Learning. Paper presented at the Summit on Assessment of Informal and After-School Science Learning.


Full Text: HTML

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.