[39] Sabourin,J. etal. [29] Fadel,C., M.Bialik and B. Committee on Defining Deeper Learning and 21st Century Skills., National Academies Press, Washington, D.C., http://dx.doi.org/10.17226/13398. 3, p.6, http://dx.doi.org/10.14507/epaa.v3n6.1995. Why game- or simulation-based assessment in education? Source: Games for change (n.d.[50]) ; Glasslab (n.d.[51]). The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions. 170-179, http://dx.doi.org/10.1016/j.chb.2015.07.045. While the field of educational measurement has made significant progress in this area in recent decades, extending frameworks like Universal Design for Learning (Rose, 2000[42]) to game-based assessment requires careful design, extensive testing, and, in some cases, the invention of new approaches and novel technologies such as haptic feedback technologies that enable the implementation of touch-based user interfaces allowing the assessment of visually-impaired students (Darrah, 2013[43]). [37] Deterding,S. etal. The results from the experiment revealed statistically significant differences in performance between boys and girls after they used the augmented reality platform. This chapter discusses how recent advancements in digital technology could lead to a new generation of game-based standardised assessments in education, providing education systems with assessments that can test more complex skills than traditional standardised tests can. In other words, the AI can be used to play all of the proposed variations of the GBA as means of increasing the likelihood that they are all comparable in difficulty before moving to expensive and time-consuming pilot testing with human test-takers. [38] Yang,F. etal. This includes patterns of choices, search behaviours, time-on-task behaviours, and, in some cases, eye movement or other biometric information. [29] Fadel,C., M.Bialik and B. These new models and others in development are intended to better address theories of cognition and learning and to capture complex latent ability structures. (2018), Education 4.0 - Artificial Intelligence Assisted Higher Education: Early recognition System with Machine Learning to support Students Success, 2018 IEEE 24th International Symposium for Design and Technology in Electronic Packaging (SIITME), http://dx.doi.org/10.1109/siitme.2018.8599203. [41] Bergner,Y. and A.von Davier (2018), Process Data in NAEP: Past, Present, and Future, Journal of Educational and Behavioral Statistics, Vol. (2012), Exploring different technological platforms for supporting co-located collaborative games in the classroom, Computers in Human Behavior, Vol. the data collected during the assessment game/simulation process). In Crisis in Space, ACTNext developed a pilot version of a GBA designed to assess the collaborative problem solving and related socioemotional skills of middle school (ISCED-2) students. 116/11. As an interim measure, the student can be assessed under more standardised simulation conditions to gauge progress toward summative goals. 15/3, pp. [9] Duncan,R. and C.Hmelo-Silver (2009), Learning progressions: Aligning curriculum, instruction, and assessment, Journal of Research in Science Teaching, Vol. 5-13, http://dx.doi.org/10.1111/j.1745-3992.2009.00149.x. 20/S1, pp. Success requires an interdisciplinary team with a broad range of skills, including game designers, software engineers ideally with a background in game, and cognitive scientists, as well as the test designers, content experts, educational researchers, and psychometricians usually needed to develop an assessment. [8] Sanders,W. and S.Horn (1995), Educational Assessment Reassessed. [2] Gong,B. [46] de la Torre,J. and J.Douglas (2004), Higher-order latent trait models for cognitive diagnosis. The test-taker engages with a modified version of the SimCity interface to solve various urban issues. 10, http://dx.doi.org/10.3389/fpsyg.2019.00853. Source: https://actnext.org/collaboration-assessment-online-games/; https://actnext.org/research-and-projects/cps-x-crisis-in-space/ (reproduced with permission). 21-34, http://dx.doi.org/10.1111/emip.12189. [31] Seelow,D. (2019), The Art of Assessment: Using Game Based Assessments to Disrupt, Innovate, Reform and Transform Testing.. [15] Chung,G. (2014), Toward the Relational Management of Educational Measurement Data.. In the long term, integrated assessment systems should rely on game- and simulation-based scenarios to evaluate how students integrate and apply knowledge, skills, and abilities. (2011), When Off-Task is On-Task: The Affective Role of Off-Task Behavior in Narrative-Centered Learning Environments, in Lecture Notes in Computer Science, Artificial Intelligence in Education, Springer Berlin Heidelberg, Berlin, Heidelberg, http://dx.doi.org/10.1007/978-3-642-21869-9_93. GBAs, on the other hand, aim to blur the line between traditional assessment and more engaging learning activities through the use of games and simulations designed to measure constructs in an environment that maximises flow and rewards students for demonstrating their cognitive processes in more engaging and authentic situations, not just their ability to memorise key facts (Shute etal., 2009[7]). Formative tests are also given during instruction but are closely linked to specific instruction and individual performance. 45-49, http://dx.doi.org/10.1177/016264340001500307. Educators, administrators, and policymakers should consider integrating GBA in their educational assessment systems, as it offers unique advantages over more traditional approaches. (2012). (2019), A Tricky Balance: The Challenges and Opportunities of Balanced Systems of Assessment., in. (2019), Game Design for Eliciting Distinguishable Behavior., Paper prepared for the 33rd Conference on Neural Information Processing Systems., https://papers.nips.cc/paper/8716-game-design-for-eliciting-distinguishable-behavior.pdf (accessed on 2January2020). [22] Ferrara,S. etal. Interim tests are given during the instructional period to evaluate progress toward summative goals and suggest instructional changes. (Sanders and Horn, 1995[8])) and how game-based assessment may ameliorate them: the need to apply modern psychological theory to assessment; insufficient alignment of assessment with curriculum and instruction (Duncan and Hmelo-Silver, 2009[9]); lack of integration of assessments for different purposes, including formative, interim, and summative (Perie, Marion and Gong, 2009[10]); inability of traditional assessment to measure some important and increasingly policy-relevant constructs (Darling-Hammond, 2006[11]), and; declines in student engagement and motivation (Nichols and Dawson, 2012[12]). [46] de la Torre,J. and J.Douglas (2004), Higher-order latent trait models for cognitive diagnosis, Psychometrika, Vol. 44/6, pp. One key design element that reduces risk of differential item functioning in game-based assessment is the design of effective tutorials within each game or simulation that quickly teach the necessary game mechanics to those test-takers possibly less familiar with the user interface. Source: Bundesministerium fr Bildung und Forschung, n.d.[26]. (2019), The Expanded Evidence-Centered Design (e-ECD) for Learning and Assessment Systems: A Framework for Incorporating Learning Goals and Processes Within Assessment Design. The seminal volume Knowing What Students Know (National Research Council, 2001[13]) brought cognitive theory into the assessment realm using a framework accessible to teachers and policy makers. [31] Seelow,D. (2019), The Art of Assessment: Using Game Based Assessments to Disrupt, Innovate, Reform and Transform Testing., Journal of Applied Testing Technology, Vol. [51] Glasslab (n.d.), SIMCITY edu: pollution challenge, https://s3-us-west-1.amazonaws.com/playfully-games/SC/brochures/SIMCITYbrochure_v3small.pdf (accessed on 30April2021). The third example game-based assessment, Project Education Ecosystem Placement (PEEP) is also in the pilot phase and is intended to measure problem solving in the ISCED-2 and 3 population via a game where test-takers are required to construct a viable food web or ecosystem and place it in the natural environment where it can thrive. For this reason, building GBAs is relatively expensive and is thus not always an efficient way to measure simple constructs. 57/3, pp. [8] Sanders,W. and S.Horn (1995), Educational Assessment Reassessed, education policy analysis archives, Vol. [18] Pellegrino,J. and M.Hilton (eds.) It called for an examination of mental functions involved in deep understanding, concepts that are difficult to assess with the sort of short, disconnected questions typical of standardised tests (Darling-Hammond etal., 2013[14]). [53] Bundesministerium fr Bildung und Forschung (n.d.), ASCOT+, https://www.ascot-vet.net (accessed on 30May2021). [3] DiCerbo,K. (2014), Game-Based Assessment of Persistence.. (2012), Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining.. (2009), Epistemic Network Analysis: A Prototype for 21st-Century Assessment of Learning, International Journal of Learning and Media, Vol. We now turn to a closer examination of the features of GBA and a brief discussion of how to design game-based tests. (2014), Psychometric Considerations in Game-Based Assessment., Glasslab Games, Redwood City, CA. 79/4, pp. [48] Ciolacu,M. etal. 1/2, pp. Note: Designed to be used as part of a longer assessment of problem solving, the PEEP task challenges students to build a viable ecosystem and place it in a natural environment where it can thrive. [10] Perie,M., S.Marion and B.Gong (2009), Moving Toward a Comprehensive Assessment System: A Framework for Considering Interim Assessments, Educational Measurement: Issues and Practice, Vol. 2, pp. An additional challenge to consider with GBAs is the need to make them accessible for students with disabilities. While traditional educational assessments are designed to generally meet standards of technical quality in areas like validity (does the assessment measure what it is supposed to measure? ), Improving Education Through Assessment, Innovation, and Evaluation., Cambridge, Mass. [23] Marion,S. etal. Games and simulations have become more central due to the potential for eliciting evidence of deeper understanding and cognitive processes. We briefly review each of the following specific criticisms of traditional standardised assessment (e.g. In addition, the need for GBA to meet assessments more stringent scientific criteria of validity, reliability, and fairness, means that the transferability of engagement and immersion may be somewhat limited or at least different in nature (Oranje etal., 2019[20]). 46/6, pp. Technologies used for measurement include eye-tracking and natural language processing. [52] Chopade,P. etal. [17] Martone,A. and S.Sireci (2009), Evaluating Alignment Between Curriculum, Assessment, and Instruction, Review of Educational Research, Vol. [21] Shepard,L., W.Penuel and J.Pellegrino (2018), Using Learning and Motivation Theories to Coherently Link Formative Assessment, Grading Practices, and Large-Scale Assessment, Educational Measurement: Issues and Practice, Vol. [35] Mislevy,R. (2018), Sociocognitive Foundations of Educational Measurement., Routledge, New York. [34] Shute,V. and M.Ventura (2013), Stealth Assessment: Measuring and Supporting Learning in Video Games., in John,D. and C.MacArthur (eds. [3] DiCerbo,K. (2014), Game-Based Assessment of Persistence., Educational Technology & Society, Vol. It will be launched (and legally recognised for exams in Germany) in 2022. 1170-1177. In the United States, for example, as theoretically-based, instructionally-relevant assessments have become more prominent, there have been widespread calls for coherence in the assessment enterprise across all levels (Gong, 2010[2]; Marion etal., 2019[23]). [42] Rose,D. (2000), Universal Design for Learning. Trilling (2015), Four-dimensional Education: the Competencies Learners Need to Succeed., Center for Curriculum Redesign, Cambridge, MA. For example, while the benefits of GBA have led several operational assessment programs, such as PISA and the U.S. National Assessment of Educational Progress (NAEP), to add game or simulation components, due to cost they have done so in a limited fashion as part of a hybrid approach combined with more traditional item types and assessment strategies (Bergner and von Davier, 2018[41]). As with any assessment, stakeholders should feel confident in what is being measured and how. (2010), Using balanced assessment systems to improve student learning and school capacity: An introduction., Council of Chief State School Officers, Washington, DC. 333-353, http://dx.doi.org/10.1007/bf02295640. Note: Introduced in 2013 by the now-defunct GlassLab, SimCityEDU: Pollution Challenge was a transformation of the popular SimCity videogame franchise into an assessment of middle school (primarily ISCED 2). This is an especially relevant critique in education for two reasons. ), reliability (does it do this consistently and with minimal error? For example, psychometricians have suggested new measurement models reflecting task complexity (Mislevy etal., 2000[44]; Bradshaw, 2016[45]; de la Torre and Douglas, 2004[46]). 606-609, http://dx.doi.org/10.1002/tea.20316. [18] Pellegrino,J. and M.Hilton (eds.) Beyond psychometric innovation, game- and simulation-based assessment also poses new opportunities for technical innovation based on recent developments in machine learning and artificial intelligence (Ciolacu etal., 2018[48]). ACTNext also has implemented advanced machine learning technology such as natural language processing (NLP) to process these data and score instances of collaboration as successful or unsuccessful (Chopade etal., 2019[52]). OECD iLibrary (2013), Criteria for High-quality Assessment.. For examples, they are more costly and difficult to develop than traditional standardised tests based on a simple succession of discrete questions or small tasks. (eds. (2016), Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning, Computers in Human Behavior, Vol. Using the webcam at the top of the screen, the system determines the location of each students astronaut by detecting the relative position of each student to the paper markers. (2015), Does agency matter? More recently, leaders in education policy, teaching and learning, and cognitive theory have come together to call for greater coherence among instruction, curriculum, and assessment and for a comprehensive assessment system that informs decisions from the statehouse to the schoolhouse (Gong, 2010[2]). For example, differential access to computers in the home or school environment as well as (possibly gendered) differences in familiarity with video game mechanics or user interface components could exacerbate existing achievement gaps or create new ones. measuring proficiency with academic content) and reserve the more expensive GBA scenarios and simulations for the measurement of more complex cognitive constructs. Moreover the use of GBA should not be limited to summative assessment alone but should instead be part of a coherent system of assessment throughout the academic year. 82, pp. As Mislevy (2018[35]) notes, "the worst way to design a game- or simulation-based assessment is to design what seems to be a great game or simulation, collect some observations to obtain whatever information the performances provide, then hand the data off to a psychometrician or a data scientist to figure out how to score it. While there are benefits to post hoc exploratory analyses, they should not be the driving mechanism for how one scores the assessment. [28] Vincent-Lancrin,S. etal. [26] Madaus,G. and M.Russell (2010), Paradoxes of High-Stakes Testing. Although this data mining should not replace the design process described above, experience suggests that computer-aided iteration here can improve the reliability and efficiency of game-based assessment by increasing the amount of useful information on test-taker performance available (Mislevy etal., 2014[49]). Thus a cost-effective and robust system of assessments might use game-based scenarios in combination with traditional and less costly discrete technology-enhanced assessment items. However, this is not to say that analysis of actual test-taker data in the GBA development process is not important. The use of games or simulations is a very promising way to assess these complex constructs either as part of a revised curricular framework or as a novel addition to the content covered by the usual standardised tests (Stecher and Hamilton, 2014[30]; Seelow, 2019[31]). Box10.1 highlights an example for assessing vocational skills in Germany. [42] Rose,D. (2000), Universal Design for Learning, Journal of Special Education Technology, Vol. The chapter is organised as follows: we first argue that game-based assessments address many of the critiques of traditional assessment and have the potential of being aligned more closely to teaching and learning in the classroom; we then explain how these assessments work, what kind of technology they use, what kind of data they draw on, and highlight the challenges in building them; we provide some examples of game-based standardised assessments, before reflecting on the role they could have in the future, and what national infrastructure may be required to deliver them at scale.
Sitemap 28