Serious Games for Healthcare Professional Training: A Systematic Review
2 Department of Computer Science, Air University, Multan, Pakistan
3 Mercy Corps, Islamabad, Pakistan
4 School of Information Management, Sun Yat-sen University, Guangzhou, Guangdong, China
Received Date: Feb 12, 2019 / Accepted Date: Mar 20, 2019 / Published Date: Mar 27, 2019
This open-access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (CC BY-NC) (http://creativecommons.org/licenses/by-nc/4.0/), which permits reuse, distribution and reproduction of the article, provided that the original work is properly cited and the reuse is restricted to noncommercial purposes. For commercial reuse, contact [email protected]
Objectives: This review aims to evaluate the performance of serious games as a training tool compared to other methods of continued professional development (CPD) and continued medical education (CME) for healthcare professionals.
Methods: PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), Cumulative Index of Nursing and Allied Health Literature (CINAHL), Web of Science, World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP), PLOS ONE, ClinicalTrials. gov, were searched for available randomized control trials (RCTs) up to June 2018. We used the CASP (Critical Appraisal Skills Programme) tool to evaluate the quality of RCTs.
Results: The search identified 1430 papers; among them, 119 were evaluated. Finally 17 RCTs involving 2978 participants were selected in this systematic review. The serious games (SGs) were classified into three broader categories: 1) specifically designed games to enhance training skills and learning gains, 2) game design elements to bolster the sense of competition for knowledge enhancement, 3) commercially available video games for training on medical procedures. Four studies found levels of satisfaction among participants of SGs to be high; none of the studies evaluated the impact of the games on beliefs or behaviors. Overall, the studies provided limited evidence to support a strong connection between the use of serious games and improved performance.
Conclusion: SGs can be an effective alternate/ complementary component of healthcare training curriculum. However, existing heterogeneous assessment methodologies are not accurately depicting the effectiveness of games. More robust RCTs/research designs are needed to evaluate the effectiveness of serious games.
Healthcare professional; Serious games; Learning gains; Surgery; Medical training
A recent World Health Organization’s (WHO) report estimated a global shortfall of 12.9 million healthcare workers by 2035 . One reason underlying this huge shortfall is the inaccessibly and lack of scalability of conventional training programs for healthcare providers (HCPs). To overcome this paucity, it is required to develop and implement interventions that can lead to an enhanced efficacy through efficient training . In addition, HCPs need to be updated with the latest advancements in their respective fields. Innovative teaching strategies, training and courses, related to continued professional development (CPD) and continued medical education (CME) are essential for creating a dynamic learning environment.
Computer-tailored interventions can be used to overcome these obstacles and to develop the essential cognitive skills . Moreover, patient safety concerns motivate the training of the healthcare personnel in simulated settings to replicate substantial aspects of the real world in a fully interactive manner. Such a trend has been emerging over the past few decades. Low-fidelity and high-fidelity simulation have been used for medical education and as an assessment tool to evaluate knowledge gaps. Simulation assisted learning is auspicious as it can render repeated practice as well as specific feedback [4, 5]. The ubiquity of videogames play has seen a push towards serious games (SGs), which have an explicit educational purpose and provide viable methods for training and developing skills.
SGs, a potential solution for interactive learning, are poised to take on a greater role in healthcare training. Bergeron defines serious games as interactive computer applications, with or without significant hardware components, designed for imparting knowledge or learning skills, which integrate scoring element, as well as challenging objectives and stimulating design . America’s Army, a gaming platform for first person shooter games developed by the U.S. Army, can be considered as the first well executed and popular serious game that gained total public awareness; this SG also proved that skill acquisition is possible through gaming . Increasing research interests towards the use of serious games in healthcare education is evident by a growing number of articles and systematic reviews [8, 9, 10, 11]. According to a report published by Global Market Insights, the healthcare gamification market size was over 16 billion USD in 2016, with an expected compound annual growth rate (CAGR) of over 12% from 2017 to 2024.
The aim of SGs in the field of medical professional education is to enhance intrinsic motivation for learning by providing the progress of the players towards a specific task and ultimately help in achieving long-term goals . Serious game design combines learning theory and empirical outcomes about boosting skill learning along with principles of game design elements and game types, thereby creating a distinctive intervention tool that can improve cognitive, social, and/or health-related skills beyond the context of the game [13, 14, 15].
Many authors have proposed classification for serious games based upon the design elements, features, and characteristics. Djaouti and colleagues  proposed a three-dimensional criterion– viz, the Gameplay, Purpose, Scope (G/P/S) criteria– for classifying serious games. In the context of this criterion, Gameplay refers to the rules and structure of the game; Purpose refers to the precise focus of the game apart from entertainment; and Scope refers to the intended market, audience, and scope of the game. Djaouti et al. used this criterion for classifying serious educational games to guide teachers about the games suitable for academics. In another work , the authors have proposed a three-dimensional taxonomy of serious games. The first dimension deals with the platform of digital game; the second considers the genre of the game; and the third dimension considers the engagement of players in the game. Another scheme based on a four- dimensional classification was proposed in , viz., (1) the deliverance of primary education content, (2) the learning principles, (3) the targeted players (e.g. their age), and (4) the platform on which the game is played.
Despite the exponential growth of SG industry and the promising claims about the efficacy of these games, systematic reviews are more cautious, and suggested that before incorporating serious games in the training and curriculum of medical professionals, extensive assessment and validation is essential. Graafland et al.  considered the games published between 1995 and 2012, and he found that instead of focusing on game effectiveness, developers are more devoted towards the commercialization of the games. The authors suggested that games used for blended learning need validation before amalgamation into surgical skill training. Wang et al.  classified the published serious games into 8 different game genres, i.e., adaptation, adventure, board game, management simulation, platform, puzzle, quiz, training simulation. The authors identified serious games in medical education as an emerging field which requires substantial evaluation and used qualitative criteria to analyze the authenticity of the evidence stated by game authors. Akl et al.  looked for the articles published before 2007 and found no substantial proof to approve or negate the effectiveness of serious games as a useful educational tool for medical students. A recent systematic review evaluated the pedagogical perspective of medical games. The pedagogical tools devised to analyze the educational effects of games were: behaviorist, cognitive, humanist, and constructivist perspectives .
The lack of coherence as to the effectiveness of serious games raises the question how influential is the SG intervention as compared to other e-learning and Virtual Reality (VR) based learning techniques? In order to address these questions we conducted a systematic review of Randomized Control Trials (RCTs). The major contributions of this review are summarized as follows:
• Identification and appraisal of a rapidly growing body of evidence on Serious Gaming and Gamification in various medical professional domains ranging from emergency medical training to neurosurgery.
• Proposal of multidimensional classification for serious games by looking into the features that are essential in their design and have the potential to make an effective serious game, which can eventually renders a significant improvement in the learning and performance skills of medical professionals.
• Evaluation of the impact and validity of intervention based on the four levels proposed by Kirkpatrick.
• Analysis as a thematic systematic review and categorization of the data into themes and sub-themes.
• Highlighted the positive aspects and limitations of these games for the knowledge acquisition and training of healthcare professionals.
There have already been systematic reviews for the evaluation of the effectiveness of serious games, however, we performed a thematic analysis for identifying, analyzing and reporting themes within the articles which compared SGs with the traditional and other contemporary didactic tools and thus, provided meaningful insights. We aimed to develop a comprehensive and unbiased evaluation of work around the topic to highlight the limitations, benefits, and future research areas.
A systematic search was performed in accordance with the Cochrane Collaboration guidelines . We defined serious games that are fun to engage while transferring skills and building awareness [24, 25]. We explicitly considered games that were designed for the training of healthcare providers. Serious gaming, e-learning, and virtual reality simulation tend to overlap and to strictly differentiate these interventions is quite challenging . In exploring the relationships between RCTs, this SR aims to answer the following question:
−What is the effectiveness of serious games compared to other types of virtual simulators or e-learning interventions in enhancing the skills, learning objective, satisfaction level, and professional attitude of healthcare professionals?
We developed the inclusion and exclusion criteria by using our PICO and research question:
• Healthcare professionals include doctors, clinicians, physicians, nurses, physiotherapists, paramedics undertaking postgraduate studies and/or CPD (continuing professional development) education and skills training activities and courses.
• RCTs discussing SG as an intervention will be considered relevant.
• Web or Internet-based interventions featuring distinct game elements that used game mechanics and design techniques are considered relevant.
• Traditional didactic curriculum, virtual reality based simulators and conventional training methods.
Primary & secondary outcomes
• Efficacy of serious games for improving the learning gains and skill enhancement for medical professionals.
• Change of behavioral attitude of healthcare personnel towards patients.
Setting or context(s)
• The healthcare setting will not always be specified though many of the papers with health topics will imply healthcare settings.
• The contexts may be online, “virtual” and electronic, or other medium on the Web.
Study type or methodology
• The study type or methodology will be systematic in nature, with systematic searches of the literature, including a qualitative or quantitative analysis.
Inclusion and Exclusion Criteria
• We included peer-reviewed RCTs that evaluated serious games for medical training, with no time restriction for the search of studies;
• Papers published in English and full text was available, were included;
• Studies in which participants were medical professionals. We excluded undergraduate students who were not yet licensed to practice;
• RCTs that examined gamification interventions, having distinct game elements or design techniques to engage and motivate participants to achieve their goals;
• Papers analysing effective aspects of SGs, including benefits and limitations were included;
• Papers scored high (≥ 8) using the CASP Checklist for Randomized Control Trials instrument;
• As this study’s focus was the performance evaluation of serious game interventions for medical training, we excluded the studies that discussed only VR based simulators and gamification without comparing with any serious game intervention for health training purpose.
PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), Cumulative Index of Nursing and Allied Health (CINAHL), Web of Science, WHO ICTRP, PLOS ONE, ClinicalTrials.gov, were queried using Medical Subject Headings (MeSH) terms. No lower limit for the publication date was applied; the last search date was June, 2018. Search terms were selected using an iterative process, we augmented our search strategy with keywords extracted from the well-known works on serious games and past systematic reviews [7, 19, 20, 27, 28]. To achieve optimal sensitivity, we used combination of words as given in Figure 1 and created search strategies with controlled or index terms given in the Appendix I and abbreviations are provided in Table 1. The researchers (AI and SMA) searched additional articles through citation and by snowballing. Any disagreement was adjudicated by the third reviewer (MNKB). Finally, RCTs focusing on the use of specific educational games and compared with traditional curriculum and other simulators for health professions were retrieved.
Table 1: List of abbreviations.
|RCTs||Randomized Control Trials|
|CASP||Critical Appraisal Skills Programme|
|CPD||Continued Professional Development|
|CME||Continued Medical Education|
|SP||Standardized Patient drill|
|SE game||Space Education game|
|IC||Informal face to face Consensus method|
|PCPs||Primary Care Physicians|
|START||Simple Triage and Rapid Treatment|
|OLT||Orthotopic Liver Transplantation|
|MIS||Minimally Invasive Surgery|
We screened the databases for the relevant studies based on the title and abstract screening. Each article was coded using NVivo12 , a qualitative analysis software which helps to find connections and understand underlying themes and patterns. For the collection of data an extraction form was developed in Microsoft Excel. The form has three categories: (1) study identification, (2) analysis of game design and strategies, (3) outcomes and results extractions. The first section contained information related to the study, for instance, name of the journal in which the article was published, country where the game was designed, and demographic details of participants. The second section contained several features and aspects which were helpful to create a relation between technology, and learning objectives. In the last section we compared the results of intervention and control groups. If the required information was not available in the published trial, the authors were contacted in order to evaluate the findings of the trial correctly. We performed a thematic analysis using methods as described by Braun et al. . Reasons for exclusion were recorded using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).
The 17 RCTs in this review were analyzed using a methodology of thematic analysis as described by Braun and Clarke . Thematic analysis is a dynamic tool for identifying, analyzing and reporting patterns (themes) within data. It helps to organize the data set to render meaningful insights. According to Thomas et al. , a thematic synthesis is comprised of three essential stages: coding of text ‘line-by-line’, development of ‘descriptive themes’ and a generation of ‘analytical themes’. While the development of descriptive themes is closely linked to the primary studies, the analytical themes represent a stage of interpretation where the reviewers ‘go beyond’ the primary studies and generate new interpretive constructs, explanations, and hypotheses. The quality of evidence regarding the overall effectiveness of SGs was assessed by three scales; 1) the CASP tool, 2) degree to which the SGs fulfilled the validation process , 3) Kirkpatrick’s hierarchy of educational outcomes. We analyzed how the selected interventions adopted Kirkpatrick four level model to measure the effectiveness construct of the outcomes . Among the four levels, reaction highlights perception of the participants about the intervention. Learning focuses on the knowledge and skill acquisition. Behavior evaluates the changes in behavior due to intervention. And results measures on the long term organizational benefits.
The systematic search identified 1430 articles. After removing 65 duplicate articles, title and abstract screening provided 506 papers for full article screening. A total of 441 articles did not meet the inclusion criteria and were excluded. Of the remaining 119 articles, 102 met exclusion criteria, were not RCTs or completed. A total of 17 articles were found to be relevant. Reasons for excluding articles were documented and reported in a PRISMA flow diagram, depicted in Figure 2. These games were divided into three categories containing (10, 3, and 4) games, respectively. More specifically, category 1 comprised 10 serious games developed specifically for the purpose of education and professional skill development; category 2 comprised 3 games involving multiple game elements (such as competitive environment, scoring leader boards, and avatars), finally category 3 comprised four commercially available games that were not designed particularly for learning gains but associated to elevate the performance of medical personnel in the field of surgery. The compiled list of included serious games is presented in Table 2. We examined the articles to validate SGs and evaluated for achievement of steps in the validation process, according to criteria regarded as best evidence [19, 33]. The selected studies are critically appraised using CASP. Two reviewers (SMA, YK) independently provided scores out of 11 for each paper by indicating “yes” to each of eleven items on the CASP checklist, presented in Table 3. Discrepancies were resolved by a third reviewer (AI).
Table 2: Assessment methodologies of serious game studies.
|Study||Classification||Platform||Male/Female||Drop outs||Results(IG/CG)||Findings||Educational goals|
|Knight et al. ||Category 1||Computer||61/26||4||Tagging accuracy [Chi2 = 13.126, p = 0.02]||IG was superior to CG||SG enhanced learning and accuracy of the triage process|
|Telner et al. ||Category 2||Physical one to one||19/16||4||1.90 [-1.44, 5.24]||CG was superior to IG||Game based group scored 0.3 lower than case based group|
|Andreatta et al. ||Category 1||Full-immersive virtual reality||N/A||0||-1.68 [-3.86, 0.50]||CG was slightly superior to IG||Results were in favor of the SP (control) group|
|Kerfoot et al. ||Category 1||Web-based||662/706||349||Median completion score 98% (IQR 25)||IG was superior to CG||SE game can improve knowledge|
|Heselmans et al. ||Category 2||Web-based||14/106||3||Medical imaging scenario (d=0.46, P=.37)||IG was superior to CG||Discussion and consensus help in guideline development when evidence are sparse|
|Kerfoot et al. ||Category 1||Web-based||38/73||0||(90% [SD, 8] versus 78% [SD,19], Cohen d 0.8, P$<$0.001)||IG was superior to CG||Online SE game provide significant reduction in the time to BP target|
|McGrath et al. ||Category 1||Full-immersive virtual reality||N/A||N/A||0.19 [-0.01, 0.39]||IG was similar in performance to CG||Moderate effect size favoring intervention|
|Clarke et al. ||Category 1||iPad||N/A||6||Total score (P < 0.0005), number of errors (P = 0.019)and time saved (P <0.0005)||IG was superior to CG||SG can improve performance of surgical instrument recognition|
|Scales et al. ||Category 2||Web-based||206/217||133||45% (SD 11) verses 41% (SD 12, P=0.127)||IG was superior to CG||Game mechanics can be an effective engaging tool for learning|
|Katz et al. ||Category 1||iPad||N/A||4||GG, 7.95 [3.65] verses CG, 4.8 [4.48]; P = 0.02)||IG was superior to CG||SG can improve the clinical performance of residents|
|Mohan et al. ||Category 1||iPad||236/101||175||Estimated difference 0.17, 0.09 to 0.25; P<0.001)||IG was slightly superior to CG||Video game improved triage decision making|
|Diehl et al. ||Category 1||Computer||69/65||36||28 [SD 14] verses 23 [SD 17], P=.06||IG was superior to CG||InsuOnline was highly effective for medical education|
|Graafland et al. ||Category 1||Mobile||14/10||7||Problem solved (59 vs. 33%, P = 0.029), problem recognized (67 vs. 42%, [P = 0.14])||IG was superior to CG||Resident underwent SG training responded better to equipment-related problems during surgery|
|Giannoti et al. ||Category 3||Gaming console||18/24||N/A||Wii group showed a significant improvement in performance [P = 0.05] for 13 of the 16 metrics||IG was superior||Video games can be a cost effective alternative for training laparoscopy|
|Adams et al. ||Category 3||Gaming console and gaming system||23/12||0||Xbox 17.7, Nintendo 11.8 and simulator 4.6 sec [P=-0.052]||IG was superior to CG||Playing the video games helped to ease stress|
|Rujin et al. ||Category 3||Gaming console||71/129||0||Wii: 62 (14), PS2: 40 (14)||IG was superior to CG||Wii and PS2 significantly improved laparoscopic skills|
|Plerholpes et al. ||Category3||Mobile||24/16||N/A||Fewer total errors 0.35 versus 1.25, [P = 0.002]||IG was superior to CG||Warm up mobile games can help to reduce errors in surgery|
Table 3: Quality assessment of the 17 RCTs.
|RCTs in this review||Focused issue||Randomization of participants||All participants accounted for conclusion||Participants blind to the intervention||Treatment of groups||Impact of treatment effect||Precision of estimate effect||Applicable to local population||Important outcomes considered||Benefits worth cost||I||II|
Almost all the authors of articles reported some positive impacts of serious games on learning or skill enhancement. However, the degree of these effects and validity of the evidence varied. We observed that most game projects (nine) were designed and developed in USA. Two games were designed in Canada; others were developed in UK, Netherlands, Italy, Belgium and Brazil, as shown in Figure 3. 82.4% of games were played by single players, while 17.6% were multiplayer games. 23.5% of the games used a web-based environment on computers/laptops, 29.4% were based on mobile/iPad apps (11.7%/17.6%), 11.7% SGs were comprised of immersive virtual 3D environment, and 11.7% used gaming consoles and gaming systems. The remaining 17.6% were computer based games and 6.1% were played using certain game mechanics. Other details of the selected studies are described in the Table 2 and Table 4.
Table 4: Developmental overview of included RCTs.
|Intervention||Control||Speciality||Game type||Sample size||Follow up||Assessment|
|Card sort exercise ||Triage trainer||Triage training||Training simulation||91||N/A||Through instructors|
|Board game (snakes and leader) ||Case based CME||N/A||Quiz||35||3 month post-test||N/A|
|Cave ||Standardized patient (SP)drill||Triage training||Training simulation||15||2 week post-test||Triage rating scale by one of the researchers|
|Space Education game ||Educational online posting||BP management||Quiz||111||52 weeks post-test||Online assessment|
|MCQs based CP Game ||&Informal face-to-face consensus (IC) method||Nursing and obstetrics||Multi players role playing||120||N/A||Online assessment|
|Space Education game ||Educational online posting||Urology||Quiz||1470||N/A||Intrinsic scoring game mechanism|
|Second life ||Traditional oral examination format||Emergency medicine||Role playing||35||N/A||Proctors|
|Instrument Trainer and PeriopSim ||N/A||Neurosurgery||Training simulation||18||N/A||Gamification scoring elements|
|Online team-based game mechanics ||Individual performance feedback||Multiple||Quiz||422||N/A||Program directors|
|Night shift ||Triage training||Training simulation||Didactic education apps||368||6 month post-test||Virtual simulation|
|OLT Trainer ||Educational materials and literature||Anesthesiology||Adventure, simulation||44||N/A||Grading rubrics and graders|
|InsuOnline ||Onsite lectures and cases discussion||Primary care physicians||Quiz, training simulation||134||3 months post-test||Self-assessed through web-based questionnaire|
|Dr. Game, Surgeon Trouble ||Regular curriculum for MIS||Endoscopy||Training simulation||31||N/A||Senior Surgeon and OSATS form|
|Nintendo Wii ||No training with the Nintendo Wii||Laparoscopy||Action||42||28 days post-test||Laparoscopic simulator|
|Xbox 360 and Nintendo ||Laparoscopic simulator||Laparoscopy||Action||31||N/A||Box training platform|
|Nintendo Wii and PlayStation2 ||N/A||Laparoscopy||Action||42||30 min post-test||Through proctor|
|Super Monkey Ball 2 ||Didactic education apps||Laparoscopy||Action||40||N/A||Using standard laparoscopic instruments|
Findings from the Thematic Analysis
The thematic analysis is presented as a conceptual map in Appendix II and each of the 17 RCTs are mapped in Table 5 to the specific relevant sub-themes it belongs to out of the total of 78 sub-themes. The conceptual map reveals the multi-dimensional nature of SGs, its effectiveness on healthcare professionals, potential benefits and challenges. All 17 papers were classified into 7 major themes:
Table 5: Assessment methodologies of serious game studies.
|RCT Name||Gaming platform||Comparator||Duration||Medical field||Measured Outcomes||Psychology/ Emotions||Learning gains|
|Knight et al. 2010 ||Triage trainer||Card sort exercise||10-15 mins||Triage training||Triaging accuracy. step accuracy||N/A||Enhanced knowledge and skills for triage|
|Telner et al. 2010 ||Board games||Case based CME||1 hour||Primary care physician||Total score||Want to experience again||N/A|
|Andreatta et al. 2010 ||CAVE||SP drill||N/A||Triage training||Time to complete task and accuracy rate||N/A||Enhanced knowledge and skills for triage|
|Kerfoot et al. 2012 ||SE game||Educational online posting||34 weeks||Primary care physician||Percentage of questions attempted||Want to experience again||Overcome regional difference among physicians|
|Heselmans et al. 2013 ||CP game app||IC method||N/A||Nursing and obstetrics||Amount of (dis)agreement among group||Enjoyable but a little dissatisfaction||Importance of consensus|
|Kerfoot et al. 2014 ||SE game||Educational online posting||52 weeks||Urology||Percentage of questions attempted||Enhanced sense of engagement||Learning retention for up to 2 years|
|McGrath et al. 2015 ||3D VR immersive environment||Traditional oral examination||N/A||Emergency medicine||Total scores in competence scale||Satisfaction||Alternative for oral examination|
|Clarke et al. 2016 ||PeriopSim instrument trainer and PeriopSim Burr Hole surgery||N/A||N/A||Neurosurgery||Time saved. number of errors||N/A||Better instruments knowledge and recognition|
|Scales et al. 2016 ||Team-based game mechanics||Individual performance feedback||10 weeks||Multiple||Percentage of questions attempted||Enhanced sense of engagement||Improved participation and motivation|
|Katz et al. 2017 ||OLT trainer||Educational materials||1 month||Liver transplantation||Total scores in competence scale||Very satisfied||Problem solving skills and attitude|
|Mohan et al. 2017 ||Night shift||Triage training||1.5 hours||Triage training||Triaging accuracy. step accuracy||Fun experience and user friendly||Enhanced knowledge and skills for triage|
|Diehl et al. 2017 ||InsuOnline||Triage training||4 hours||Primary care physician||Total score||Fun. pleasant and practice-changing||Improved clinical performance|
|Graafland et al. 2017 ||Dr Game Surgeon Trouble||Regular curriculum for MIS||1 hour||Laparoscopy||Problems recognized. solved and performance||N/A||Problem solving skills and attitude|
|Giannoti et al. 2013 ||Nintendo Wii||No training with the Nintendo Wii||1 hour||General Endoscopy||Time to complete task and accuracy rate||N/A||Improved laparoscopic skills|
|Adams et al. 2012 ||Xbox 360. Nintendo Ds. Call of duty 4||Laparoscopic simulator||6 weeks||Laparoscopy||Time to complete task and accuracy rate||Sense of cooperation||Improved laparoscopic skills|
|Rujin et al. 2012 ||Nintendo Wii. PlayStation2||N/A||30 mins||Laparoscopy||Total score to perform a task||Ease stress||Improved laparoscopic skills|
|Plerholpes et al. 2011 ||Super Monkey Ball 2||Didactic educational apps||10 mins||Laparoscopy||Path length. hand dominance. errors||Stress release||Improved laparoscopic skills|
1) Medical fields
2) Gaming platforms
3) Duration of intervention
4) Learning gains
5) Measured Outcomes
7) Psychology and Emotions
While the seven themes provide insights into the 17 RCTs, and aid in our categorization, we developed 78 sub-themes and categories for further detail.
The medical fields discussed in the papers are: laparoscopy [34, 35, 36, 37, 38], emergency medicine (EM) [39, 40, 41, 42], neurosurgery , general, vascular and endoscopic surgery, liver transplantation , primary care physicians and internal medicine [45, 46, 47, 48, 49] and urology [34, 50].
Duration of interventions was diverse ranging from 10 minute to 52 weeks. Prolonged time duration of an intervention helps to have an extensive and long lasting impact in learning gain, 52 weeks , 34 weeks , 10 weeks , 6 weeks , 1 month [35, 44], 4 hours , 1 hour [34, 40, 42, 46], 10 minutes [38, 40]. In three studies, time duration for the intervention was not mentioned [41, 43, 47].
Several commercial games as well as specifically designed games were studied to boost the learning and training abilities of professionals. Xbox 360, Nintendo DS, Call of duty 4 , CAVE  PeriopSim instrument trainer and PeriopSim Burr Hole surgery , InsuOnline , Nintendo Wii, PlayStation2 , , Dr. Game, Surgeon Trouble , CP game app , OLT trainer , Space education game [45, 50], Super Monkeyball 2 , Triage trainer  Night shift , Team-based game mechanics , Board games , and 3D VR immersive environment .
SGs were evaluated rigorously against the comparators, laparoscopic simulator [34, 36], live disaster drill , onsite learning session , card sort exercise , case based CME , standardized patient (SP) drill , informal face to face consensus (IC) method , mannequin based simulation session , Educational content in an online posting [45, 50], traditional oral examination , feedback regarding individual performance , apps based on traditional education , and case discussion regular curriculum for MIS .
Recurring themes of benefits were related to improved laparoscopic skills [34, 35, 36, 37], enhanced knowledge and skills for triage [39, 40, 42], better instrument knowledge and recognition [34, 43], problem solving skills and attitude, importance of consensus , improved clinical performance , learning retention for up to 2 years [45, 50], time saved, number of errors [34, 38, 40, 42, 43, 44], total scores in competence scale , accuracy rate [37, 39, 40, 42], amount of (dis)agreement , smoothness, and improved hand coordination .
The measured outcomes on the basis of which the SGs were compared with control groups were time to complete the task [34, 35, 36, 39, 40, 43, 45], number of errors occurred [39, 43]. Total scores to complete the procedure [37, 41, 43, 44, 46, 48], problem recognition , accuracy rate , disagreement and concordance of answers , percentage of questions attempted [49, 50] and triaging accuracy and step accuracy [39, 40, 42].
Health professionals expressed that by undergoing the intervention they experienced many positive emotions; pleasant, fun , stress release [36, 38], practice changing , satisfaction [34, 43, 47] sense of cooperation [36, 49], effective conflicts resolution in argument , dissatisfaction , enhanced sense of engagement [45, 49, 50], want to experience again [42, 46].
Positive Aspects and Limitations of Serious Games for Healthcare Professional Training
Positive effects and benefits
SGs have been used across a range of medical professionals from primary care provider to residents and surgeons with both positive and negative effects. One paper in our inclusion set is Knight et al. , a highly-cited paper from 2010. The authors found that it is difficult to evaluate the benefits of serious games and game design elements with most studies showing no significant effects. Although, every study mentioned some positive aspects of incorporation of games within the medical curriculum, CME, and training skills but also recommended better evaluation to determine the precise impact [35, 38, 41, 47, 50].
During laparoscopic surgery decision making, precision in movement in critical emergency situations is required. There are scenarios where residents have to deal with equipment failure problems and to make decisions that are crucial for the patient’s life. Games in [34, 43] were designed for improved recognition of instruments in minimum time to provide situation awareness and stress management. To ease out the surgeons prior to laparoscopic surgery a warm up game Supper Ball Monkey 2 was evaluated in , that helped to enhance the performance in terms of time, error reduction, and hand dominance. Commercial games like Xbox, Nintendo Wii, Call for Duty 4, ans PlayStation2 can influence positively for fast and improved task performance, depth precision, better hand-eye coordination with fewer errors [35, 36, 37].
Effective learning strategies for CME are required to help in knowledge retention and to support the continuous educational gain for better decision making and situation handling even for experienced healthcare personnel. Questions are sent to the participants through e-mail and points were awarded based on resident’s performance on the questions. To instill motivation and competition, marks of other participants were also shared. The games in [45, 46, 49, 50] suggested that increasing the participation level and evaluation of knowledge by using extensively designed MCQs, true-false questions and games elements like leader boards can foster the learning gains and also provide a way to compare the knowledge and understanding of medical professionals across the globe . InsuOnline was developed for CME which provided a platform to PCPs to practice and provide medical education on InsuOline therapy for diabetes mellitus (DM), this study showed that significant gains were observed in the competence level and attitude of PCPs .
In case of mass disaster for the physicians who do not have ample experience, accurate triaging becomes very difficult, thus risking the life of patients. Virtual reality based games can provide flexible, on demand training options by using a stable and repeatable platform essential for the development of assessment protocols. Several games were designed by keeping this aspect in mind [39, 40, 42]. We encourage continued evaluation of these alternative video games.
To ingrain the sense of cooperation and consensus, there were RCTs that showed instead of making critical decisions independently at early professional level, it is better to discuss with fellow colleagues and know their opinion for the same situation. In this way, novice professionals can seek the guidance using online games [36, 47, 49]. Four studies also found high levels of satisfaction among participants and they were eager to participate in such interventions [41, 47, 48, 49].
Second Life can be an effective alternative of oral examination to ensure transparency and can remove chances of examiner bias. Although, no significant difference in scores was observed in  but the examinee found VR based examination less intimidated. In addition, it can be a less expensive alternative of traditional medical exams with intelligently used resources.
train the medical professionals and to provide them training similar to real life, special care was taken to design games [34, 35, 36, 39, 40, 42, 43, 44, 48]. Complete medical procedure was designed in the games that can be very beneficial for the flexible training of less experienced healthcare professionals. Different levels in the game represented various medical scenarios which helped the professionals to get familiarize with various scenarios and the professionals can practice these levels multiple times to memorize the situations and to enhance their knowledge.Limitations of serious games
Most of included RCTs have more than one methodological limitation; however, the overall quality of evidence was acceptable. From the evidence, little is known about the sustainability or long-term effects of these SGs [36, 37, 38, 43, 44, 47, 49]. None of the studies assessed the long term outcomes of the games for changing behavior, only one study consider patient related outcomes . To testify the robustness of these games it was required to assess the implementations of games in different settings, which was missing in all the studies.
The studies were designed to evaluate only transient performance with no data on long-term skill retention. It is worth mentioning that most of these studies had duration of a few hours of training with SGs [34, 35, 37, 38, 40, 42, 46, 48], suggesting that for skill acquisition the health professionals are needed to do recursive practice with the motivation of achieving skills. Serious games have been compared with several traditional didactic techniques to evaluate whether SGs are an effective alternative of traditional curriculum at the same time provide better learning gains [45, 49, 50]. However, the settings for the interventions might be the cause why the participants show better level of engagement and involvement, i.e., the participants are reminded to complete the intervention periodically , which helped more participants of SG intervention to complete the game. In addition, attempting more number of questions than the control group did not imply that the learning gain of that group was greater than the control group. Moreover, even after sending reminders the rate of participation was observed to decline similar to the control group .
Despite of many prominent positive aspects of SGs, games provided no significant improvement in performance, learning gains and knowledge retention [35, 39, 47]. No difference in outcomes was observed in  only a moderate effect size favoring the intervention. From our systematic study of the existing studies, we can report the VR based games can add additional value by complementing the existing systems but the results demonstrated that these are not yet substitutes of the traditional examination system. Even after adding fun elements to increase the level of participation, the intervention group showed less scores compared to the control group .
Serious games have the potential of improving cognitive skills and can alter the existing market for training modalities in a variety of domains including education, advertising, corporate training, medicine and healthcare. However, not all SGs have proved to be effective, special consideration and rigorous evaluation should be paid during their design and development. Past systematic reviews reported diverse conclusions about the effectiveness of SGs for medical professionals. A possible explanation could be the variability in the design elements and features of the game. This review provided a number of insightful findings by classifying the serious games into themes and categories. The methodological quality of the included studies was heterogeneous, as were the associated study designs. Unlike other reviews, we also considered the RCTs which implemented several game design elements, for instance competition, leader board, avatars, challenging goals, hints etc. and observed their efficacy compared to traditional curriculum, to facilitate a wide range of learning objectives. We assessed data for meta-analysis but could not conduct it due to the heterogeneity of data.
One possible limitation of this review is the omissions of articles which included medical students as their participants. There are several studies showing positive results which substantiate the fact that serious games improve engagement and motivation, thus, can be incorporated in the curriculum to increase learning gains. Enochsson et al. , noted a positive correlation between playing computer games and performance in endoscopic surgery performed by medical students. Educational games in an obstetrics and gynecology were compared with standard lectures in . Third-year medical students in the intervention group express that games are helpful in retaining information, entertaining and engaging. The study  showed that game-based e-learning (GbEl) performed better compared to a script-based approach for the training of urinalysis for transferring cognitive skills.
Another limitation is that we have only included RCTs and have excluded many of the positive single arm studies because of the greater potential for bias. In order to correctly analyze the working of these games towards the training of medical personnel it is required to add the control groups in trials. This requirement is more needed in educational interventions as outcomes are usually measured pre and post intervention. However, it is also a fact that the investigators are interested to conduct single-arm trial and measure the easily evident increase of the score from pre to post intervention, as in this way it is convenient to show the significant difference between the baseline and post intervention knowledge.
Serious gaming has various noteworthy attributes like motivation, recurrence, association and the integration of multiple senses essential for the learning purpose. It is evident that the number of empirical studies in this domain is limited. The selected randomized controlled trials depicted that serious games help to relieve stress prior to surgical procedures, increase the technical performance, bolster decision making abilities regarding instrument selection in case of equipment malfunctioning and correct triaging under critical emergency situations. However, for future research, it is required to conduct longitudinal studies to observe the long-term effect of skill enhancement using serious games. Analyzing the RCTs lead us think that serious games can be an effective complementary tool for continued medical education, however, robust and exhaustive research design are required to measure the efficacy of serious game. In addition, before integration of serious games into continual medical education thorough validation is required.
While designing a serious game several aspects must be kept in mind; adaptive game design elements as players have different skill sets and experience. In order to acquire high credibility in the field of medical, reliable and meticulous standards must turn into a reality. Teams of expert game developers and specialized medical professionals are required to work in collaboration to devise the methods and techniques that are required to transform and process raw sensory data to design fully functional game worlds. Gaming simulation is gaining recognition as a training method in various domains, but its effectiveness has not been conclusively established.
Syed Mustafa Ali and Maged N. Kamel Boulos conceived the idea. Aneeqa Ijaz wrote the protocol. Aneeqa Ijaz and Syed Mustafa Ali collected and screened the articles. Aneeqa Ijaz and Muhammad Yasir Khan processed and extracted the data. Syed Mustafa Ali and Muhammad Yasir Khan performed the critical appraisal. Aneeqa Ijaz wrote the paper. Junaid Qadir and Maged N. Kamel Boulos provided comments and critically revised the final version of paper.
Appendix I: Search Terms
• (serious gam*) OR (videogam*) OR (video gam*) OR (gaming) AND ((educat*) OR (train*))
• ((serious games) AND video games) AND medical education
• ((serious games) AND simulat*) AND health
• ((((serious game) AND eLearning) AND health training) OR medical training) OR health education
• ((serious game) AND virtual reality) AND health training
• (((serious games) AND gamification AND Humans[Mesh] AND En- glish[lang])) AND health training
• ((serious games) AND medical training)
• ((serious games) AND medical education)
• ((serious games) AND medical education) OR health training
• ((serious games) AND health* training )
• ((serious games) AND health* education) AND health* training
• ((serious games) AND health* education) OR health* training
• ((((((serious gam*) OR videogame*) OR video gam*) OR gaming) AND medical education) OR educat*) OR training
• (((((serious gam*) AND videogame*) OR video gam*) OR gaming) AND medical educa- tion) OR medical training
• (((serious games) AND virtual reality) AND health education)
• (((games) AND medical training)) AND random* control trial
Appendix II: Conceptual Map
Appendix III: Summary of Studies
Triage Trainer was compared with a card-sort exercise among 91 medical preofessionals . The study was evaluated by integrating into MIMMS course. Assessors found a substantial increase in triage accuracy in terms of tagging and step accuracy for the Triage Trainer group in posttest cases, proving concurrent validity. However, no prominent time difference is observed to triage all casualties between the two groups. The learning enhancement by employing the intervention shows level 2 Kirkpatrick outcomes.
Triage training on the CAVE system was evaluated among emergency medicine (EM) residents using the Simple Triage and Rapid Treatment (START) algorithm. Prior to the drill residents were delivered 1 hour lecture then each resident was asked for triaging 14 victims during the disaster drill . Concurrent validity was not proven as the control group performed better on the posttest. Regardless of no performance improvement participants favor the use of CAVE system to practice, indicates level 1 Kirkpatrick outcomes.
The Blood Pressure (BP) Management Game was associated with increased trainee knowl- edge (Kirkpatrick level 2) and improved patient outcomes (Kirkpatrick level 4). Attending physicians were randomized to 1 of 2 groups: serious game, whereas the control group received an online posting of the same educational content . The intervention group scored significantly. The BP Management Game was associated with increased trainee knowledge thus verify content and concurrent validity. The efficacy of a space education game was examined in an RCT by Kerfoot and his colleagues in . The participants are divided into two cohorts: some receive two questions per email every two days others get four every four days. Baseline scores and completion scores favors that the use of game mechanics can elevate the participant’s interest and 67% enrollees show their interest to participate in future studies (Kirkpatrick level 1). The authors claim that the intervention help in learning outcomes (Kirkpatrick level 2), knowledge retention and improve clinical behavior (Kirkpatrick level 4).
Second life was used to construct a virtual emergency environment for the assessment of emergency medicine residents. 35 residents were randomized to simulated virtual examination format or the conventional oral examination . 79% of the participants preferred virtual examination (Kirkpatrick level 1). Based on the proctors scores there was no prominent difference between the control and intervention group, thus concurrent validity was not verified. However, the examinees stated that the simulation environment was realistic, objective and fair, proving the face validity.
PeriopSim Instrument Trainer and PeriopSim for Burr Hole Surgery were designed for neurosurgery residents, to apply the knowledge and for correct identification of instruments during a surgery procedure . Concurrent validity is proved as participants in intervention showed significant improvement in the recognition and utilization of simulated surgical instruments through posttest and their performance was compared with the experts exhibiting the construct validity. This game helps the resident to reduce the time for identify correct instruments by making fewer errors and thus provide leaning gains (Kirkpatrick level 2).
OLT trainer was designed for residents having liver transplant training for imparting learning or skills, while leveraging the elements of video games like engagement, self-motivation and repetition. The game is compared with traditional training methods in an RCT . According to the simulation instructors and grading rubrics, there is a prominent improvement in performance among both group but specifically in the game group, depicting concurrent validity. This study reports that 81% participants are satisfied by the intervention (Kirkpatrick Level 1), enhanced the trainee knowledge (Kirkpatrick Level 2) and behavior (Kirkpatrick Level 3).
Night Shift was developed to provide narrative engagement for physicians to replicate emer- gency department environment (face validity) and to provide a platform for pattern recognition to recognize moderate-severe injuries .The game was compared with traditional didactic procedures. A six months posttest showed that the video game improved triage decision making of physicians using exposure in a validated virtual simulation, exhibiting concurrent validity.
InsuOnline was developed for primary care physicians (PCPs) to alleviate clinical inertia and to improve decision making for enhancing quality of care for diabetes mellitus (DM) patients and compared with the control group . Interventions higher competence score depicted the influential learning aspects of game (Kirkpatrick level 2). 62% of the physicians expressed their interest and willingness to join similar activities in future as well (Kirkpatrick level 1). In the 3 months posttest the participants said that they are professionally more confident by changing their practicing method (Kirkpatrick level 3). Patient related attitudes were also improved both after the game Kirkpatrick level 4
Dr. Game, Surgeon Trouble was designed to train surgical trainees in recognizing to correct equipment during minimal invasive surgery and compared with regular curriculum . In the posttest, the intervention group show significant performance by solving more problems than the control group, showing concurrent validity. The intervention group was also better in situational awareness and recognition of problems, implying learning gains (Kirkpatrick level 2).
Three studies have integrated the game mechanics into the interventions and studied the impact of gaming elements towards the self-driving learning goals. A CPGame application was developed to compare the human computation and informal face to face consensus method . This app provides a platform to discuss the scenarios and point of view among the colleagues without any conflict and the study displayed positive outcomes as most of the participants of the intervention found this app as an efficient method of learning and discussion platform (Kirkpatrick level 2). The impact of team-based competition was examined in . Both control and intervention groups received similar questions from various medical fields but in intervention group the players were teamed up, which enhanced the participation and social collaboration as the team-based group has 4% better first correct response than the control group. The concept of snakes and leaders games is used along with multiple choice and true false questions to access the knowledge gain among family physicians . A prominent number of participants expressed their interest in taking part of such fun games for learning enhancement (Kirkpatrick level 1). However the game based group scored 1.6 points lower than the control group in the posttest, thus no concurrent validity is confirmed.
Not only the games specifically designed for training purpose, it is observed that video games can help in transferring positive outcomes and skill acquisition. We included four studies that considered commercially available games that helped in imparting laparoscopic psychomotor skills. The games include action games and adventure games on various platforms. Such games usually have a challenge or a task which is required to achieve and the performance is evaluated by intrinsic scoring methods. To test their concurrent validity, these games are compared with simulators. A clear evidence of improved performance in laparoscopic handling speed and errors by playing Super Monkey Balls game can be observed in . The player interacts with the game using Nintendo Wii. Xbox and PlayStation2 controllers that have been re-purposed as laparoscopic tools [43, 44]. The player must locate 10 balls and snap their photos using a 0 degree camera, second task required eye-hand coordination, to do so, and the player must perform a number of actions, which replicate laparoscopic actions in the operating room, such as grasping and cutting. However, the studies had insufficient design for drawing conclusions about validity for long lasting learning laparoscopic skills by solely relaying on the video games [36, 37]. Nonetheless, these games helped to improve the multidimensional movements and concentration towards a certain goals but just playing the games cannot serve the entire purpose of learning and training enhancement.
- World Health Organization. World health statistics 2016: monitoring health for the SDGs sustainable development goals. World Health Organization; 2016.
- Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. The Lancet. 2010; 376(9756): 1923-1958.
- Frehywot S, Vovides Y, Talib Z, Mikhail N, Ross H, Wohltjen H, et al. E-learning in medical education in resource constrained low-and middle-income countries. Hum Resour Health. 11(1): 4.
- Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011; 306(9): 978-988.
- Barry Issenberg S, Mcgaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005; 27(1): 10-28.
- Bergeron B. Developing serious games. Hingham: Charles River Media. 2006.
- Zyda M. From visual simulation to virtual reality to games. Computer. 2005; 38(9): 25-32.
- Graafland M, Dankbaar M, Mert A, Lagro J, De Wit-Zuurendonk L, Schuit S, et al. How to systematically assess serious games applied to health care. JMIR Serious Games. 2014; 2(2).
- De Ribaupierre S, Kapralos B, Haji F, Stroulia E, Dubrowski A, Eagleson R. Healthcare training enhancement through virtual reality and serious games. In: Ma M, Jain L, Anderson P, editors. Virtual, Augmented Reality and Serious Games for Healthcare 1. Berlin, Heidelberg: Springer; 2014. p. 9-27.
- Laamarti F, Eid M, Saddik AE. An overview of serious games. Int J Comp Game Technol. 2014; 2014: 358152.
- De Wit‐Zuurendonk LD, Oei SG. Serious gaming in women’s health care. BJOG. 2011; 118: 17-21.
- Thompson D, Baranowski T, Buday R, Baranowski J, Thompson V, Jago R, et al. Simul Gaming. 2010; 41(4): 587-606.
- Shute VJ, Ventura M, Bauer M, Zapata-Rivera D. Melding the power of serious games and embedded assessment to monitor and foster learning. Serious games: Mechanisms and Effects. 2009; 2: 295-321.
- Giunti G, Baum A, Giunta D, Plazzotta F, Benitez SE, Gómez AR, et al. Serious games: A concise overview on what they are and their potential applications to healthcare. Stud Health Technol Inform. 2015; 216: 386-390.
- Grace L. Game type and game genre. Retrv Feb. 2005; 22(2009): 8.
- Felicia P, editor. Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches: Multidisciplinary Approaches. IGI Global; 2011.
- Jantke KP, Gaudl S. Taxonomic contributions to digital games science. In: 2010 2nd International IEEE Consumer Electronics Society's Games Innovations Conference; 2010 Dec 21-23; Hong Kong, China. IEEE; 2010. p. 1-8.
- Ritterfeld U, Cody M, Vorderer P, editors. Serious games: Mechanisms and effects. Routledge; 2009.
- Graafland M, Schraagen JM, Schijven MP. A Systematic review of serious games for medical education and surgical skills training. Br J Surg. 2012; 99(10):1322-1330.
- Wang R, DeMaria S Jr, Goldberg A, Katz D. A systematic review of serious games in training health care professionals. Simul Healthc. 2016; 11(1): 41-51.
- Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z, et al. The effect of educational games on medical students' learning outcomes: a systematic review: BEME Guide No 14. Med Teach. 2010; 32(1): 16-27.
- Gorbanev I, Agudelo-Londoño S, González RA, Cortes A, Pomares A, Delgadillo V, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online. 2018; 23(1): 1438718.
- Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration. 2005.
- Stokes BG. Videogames have changed: time to consider ‘serious games’? Dev Edu J. 2005; 11(3): 12.
- Hamari J, Shernoff DJ, Rowe E, Coller B, Asbell-Clarke J, Edwards T. Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Comput Hum Behav. 2016; 54: 170-179.
- Susi T, Johannesson M, Backlund P. Serious games–An overview [Technical Report HS-IKI-TR-07-001]. Skövde, Sweden: University of Skövde School of Humanities and Informatics. 2007.
- Kapp KM. The gamification of learning and instruction: game-based methods and strategies for training and education. John Wiley & Sons; 2012.
- Blakely G, Skirton H, Cooper S, Allum P, Nelmes P. Educational gaming in the health sciences: systematic review. J Adv Nurs. 2009; 65(2): 259-269.
- Welsh E. Dealing with data: Using NVivo in the qualitative data analysis process. In: Forum: qualitative social research. 2002.
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006; 3(2): 77-101.
- Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008; 8: 45.
- Kirkpatrick DL. Implementing the Four Levels: a practical guide for effective evaluation of training programs: easyread super large 24pt edition. ReadHowYouWant. Com. 2009.
- Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc. 2003; 17(10): 1525-1529.
- Graafland M, Bemelman WA, Schijven MP. Game-based training improves the surgeon's situational awareness in the operation room: A randomized controlled trial. Surg Endosc. 2017; 31(10): 4093-4101.
- Giannotti D, Patrizi G, Di Rocco G, Vestri AR, Semproni CP, Fiengo L, et al. Play to become a surgeon: Impact of nintendo wii training on laparoscopic skills. PLoS One. 2013; 8(2): e57372.
- Adams BJ, Margaron F, Kaplan BJ. Comparing video games and laparoscopic simulators in the development of laparoscopic skills in surgical residents. J Surg Educ. 2012; 69(6): 714-717.
- Ju R, Chang PL, Buckley AP, Wang KC. Comparison of nintendo wii and playstation-2 for enhancing laparoscopic skills. JSLS. 2012 ;16(4): 612-618.
- Plerhoples TA, Zak Y, Hernandez-Boussard T, Lau J. Another use of the mobile device: warm-up for laparoscopic surgery. J Surg Res. 2011 ; 170(2): 185-188.
- Andreatta PB, Maslowski E, Petty S, Shim W, Marsh M, Hall T, et al. Virtual reality triage training provides a viable solution for disaster-preparedness. Acad Emerg Med. 2010 ; 17(8): 870-876.
- Knight JF, Carley S, Tregunna B, Jarvis S, Smithies R, De Freitas S, et al. Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation. 2010 ;81(9): 1175-1179.
- McGrath J, Kman N, Danforth D, Bahner DP, Khandelwal S, Martin DR, et al. Virtual alternative to the oral examination for emergency medicine residents. West J Emerg Med. 2015; 16(2): 336-343.
- Mohan D, Farris C, Fischhoff B, Rosengart MR, Angus DC, Yealy DM, et al. Efficacy of educational video game versus traditional educational apps at improving physician decision making in trauma triage: randomized controlled trial. BMJ. 2017; 359: j5416.
- Clarke DB, Kureshi N, Hong M, Sadeghi M, D'Arcy RC. Simulation-based training for burr hole surgery instrument recognition. BMC Med Educ. 2016; 16: 153.
- Katz D, Zerillo J, Kim S, Hill B, Wang R, Goldberg A, et al. Serious gaming for orthotopic liver transplant anesthesiology: A randomized control trial. Liver Transpl. 2017; 23(4): 430-439.
- Kerfoot BP, Turchin A, Breydo E, Gagnon D, Conlin PR. An online spaced-education game among clinicians improves their patients' time to blood pressure control: A randomized controlled trial. Circ Cardiovasc Qual Outcomes. 2014; 7(3): 468-474.
- Telner D, Bujas-Bobanovic M, Chan D, Chester B, Marlow B, Meuser J, et al. Game-based versus traditional case-based learning: comparing effectiveness in stroke continuing medical education. Can Fam Physician. 2010; 56(9): e345-e351.
- Heselmans A, Aertgeerts B, Donceel P, Van de Velde S, Vanbrabant P, Ramaekers D. Human computation as a new method for evidence-based knowledge transfer in Web-based guideline development groups: proof of concept randomized controlled trial. J Med Internet Res. 2013; 15(1): e8.
- Diehl LA, Souza RM, Gordan PA, Esteves RZ, Coelho IC. InsuOnline, an electronic game for medical education on insulin therapy: A randomized controlled trial with primary care physicians. J Med Internet Res. 2017; 19(3): e72.
- Scales CD Jr, Moin T, Fink A, Berry SH, Afsar-Manesh N, Mangione CM, et al. A randomized, controlled trial of team-based competition to increase learner participation in quality-improvement education. Int J Qual Health Care. 2016; 28(2): 227-232.
- Kerfoot BP, Baker H. An online spaced-education game for global continuing medical education: a randomized trial. Ann Surg. 2012; 256(1): 33-38.
- Schlickum MK, Hedman L, Enochsson L, Kjellin A, Felländer-Tsai L. A Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: A prospective randomized study. World J Surg. 2009; 33(11): 2360-2367.
- O'Leary S, Diepenhorst L, Churley-Strom R, Magrane D. Educational games in an obstetrics and gynecology core curriculum. Am J Obstet Gynecol. 2005; 193(5): 1848-1951.
- Boeker M, Andel P, Vach W, Frankenschmidt A. Game-based e-learning is more effective than a conventional instructional method: a randomized controlled trial with third-year medical students. PLoS One. 2013; 8(12): e82328.