Authors Andy South Charlotte Hemingway Claire Dormann Edward K. Thomsen Kirsten A. Duda Marlize Coleman Michael Coleman Robert Farmer
Malaria Journal Thomsen et al. Malar J (2018) 17:422 https://doi.org/10.1186/s12936-018-2572-2 CASE STUDY Open Access ResistanceSim: development and acceptability study of a serious game to improve understanding of insecticide resistance management in vector control programmes Edward K. Thomsen1, Charlotte Hemingway1, Andy South1, Kirsten A. Duda1, Claire Dormann1, Robert Farmer2, Michael Coleman1* and Marlize Coleman1,3 Abstract The use of insecticides is the cornerstone of effective malaria vector control. However, the last two decades has seen the ubiquitous use of insecticides, predominantly pyrethroids, causing widespread insecticide resistance and compro- mising the effectiveness of vector control. Considerable efforts to develop new active ingredients and interventions are underway. However, it is essential to deploy strategies to mitigate the impact of insecticide resistance now, both to maintain the efficacy of currently available tools as well as to ensure the sustainability of new tools as they come to market. Although the World Health Organization disseminated best practice guidelines for insecticide resistance man- agement (IRM), Rollback Malaria’s Vector Control Working Group identified the lack of practical knowledge of IRM as the primary gap in the translation of evidence into policy. ResistanceSim is a capacity strengthening tool designed to address this gap. The development process involved frequent stakeholder consultation, including two separate work- shops. These workshops defined the learning objectives, target audience, and the role of mathematical models in the game. Software development phases were interspersed with frequent user testing, resulting in an iterative design process. User feedback was evaluated via questionnaires with Likert-scale and open-ended questions. The game was regularly evaluated by subject-area experts through meetings of an external advisory panel. Through these processes, a series of learning domains were identified and a set of specific learning objectives for each domain were defined to be communicated to vector control programme personnel. A simple “game model” was proposed that produces realistic outputs based on player strategy and also runs in real-time. Early testing sessions revealed numerous usability issues that prevented adequate player engagement. After extensive revisions, later testing sessions indicated that the tool would be a valuable addition to IRM training. Keywords: Serious games, Insecticide resistance management, Insecticide resistance, Vector control, Capacity building, Training *Correspondence: email@example.com 1 Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK Full list of author information is available at the end of the article © The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/ publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Thomsen et al. Malar J (2018) 17:422 Page 2 of 15 Background theory suggests that effective learning is accomplished In 2000, with the signing of the Abuja Declaration, lead- through active involvement of the learner, a self-directed ers from malaria-endemic countries across sub-Saharan approach, and working with realistic scenarios . All of Africa committed themselves to decrease the burden of these criteria are central to a simulation game. In addi- malaria . This increase in political will was rapidly fol- tion, social cognitive theory is based on the idea that lowed by greater financial support from global partners. behaviour is driven by the understanding of the world in As a result, after three decades of stagnation since the which a person lives, including the positive and negative close of the World Health Organization (WHO) Global outcomes witnessed as a result of choices made  and Malaria Elimination Programme in 1969, the last 18 years beliefs in personal efficacy; games influence the player’s has seen a rapid scale-up of malaria control interven- understanding of the world around them by enabling tions. Insecticide-based vector control lies at the heart of them to explore complex problems in a safe setting, the global strategy. allowing them to make mistakes and learn from them Pyrethroids, with their low mammalian toxicity, long without real-world consequences. residual life, and relatively low production cost, became The value of serious games has seen increased atten- the dominant insecticide class of choice during the scale- tion from many industries since 2002 , most notably up. At the time of the declaration, resistance to pyre- the healthcare sector. Games have been used to improve throids was almost negligible, with just a few populations adherence to self-medication among cancer patients , of vectors exhibiting resistance on the African continent relieve symptoms of depression , and train medical . Now, with the extensive use of these insecticides for and surgical personnel , among many other applica- both insecticide-treated nets (ITNs) and indoor residual tions. However, games have not yet been used among spraying (IRS), not a single country in sub-Saharan Africa the implementers of public health programmes, where is free from pyrethroid resistance [3, 4]. Resistance to all relatively few individuals are responsible for engaging other classes of public health insecticides is ubiquitous in complex decision-making processes that ultimately as well. Consequently, the ability to control the vectors impact the health of tens of thousands of people. In addi- responsible for transmitting the disease is compromised. tion, there have been relatively few applications of seri- The path to this situation is characterized by an insuf- ous games in low and middle income countries where ficient safeguarding of the available insecticide products. increased computer literacy is producing a generation Despite proven strategies to curb resistance , vec- that may be particularly receptive to digital gaming tor control programmes around the world have relied solutions. exclusively on monotherapies, mainly pyrethroids, for Here a serious game was developed to improve under- years. Cross-resistance between this and other insecti- standing and adoption of strategies to manage insecticide cide classes limits the number of alternatives, resulting resistance among vector control programmes in malaria- in development of further resistance to these products endemic countries. Here, the game development and the as well. These practices result in some countries without process of developing it, is presented with results from viable vector control strategies. preliminary acceptability studies. If there is not a culture change surrounding public health insecticide use, there is a risk that the effectiveness ResistanceSim of existing and new insecticides will be compromised by Open simulation resistance. Recognizing the gravity of the current situa- ResistanceSim is a management simulation game that tion, the WHO published the Global Plan for Insecticide immerses players in a fictional sub-Saharan African Resistance Management (GPIRM) , which provides country. The player can interact with several environ- technical recommendations for national control and ments (Fig. 1). At the province level, the player sees four elimination programmes to sustainably manage resist- districts. By clicking on one of the district labels, the ance. However, the operational implementation of these game zooms into the district level, where the player can recommendations is lacking, and innovative solutions are interact with several villages or towns. At the district level required to communicate the principles and implications view, the player can rotate, pan, and zoom the camera to of insecticide resistance management (IRM). investigate their environment. At any time, the player can ‘Serious games’ are games designed for purposes access the national capital, where they can interact with beyond mere entertainment. They blend the engaging, various stakeholders. Each geographical location in the fun, and challenging components of gaming with the game has different characteristics: the mosquito species goal of supplying the player with skills and knowledge present, their behaviour, their insecticide resistance pro- useful in real-life situations, ultimately supporting atti- file, and the malaria transmission season and intensity all tude and behaviour change. Modern instructional design vary from place to place. There are a total of 12 locales Thomsen et al. Malar J (2018) 17:422 Page 3 of 15 (See figure on next page.) Fig. 1 The three different map levels of ResistanceSim. a Shows the district level, which allows players to perform actions in three locales per district. b Shows the province level, which allows players to perform actions in four districts. c Shows the national capital, which allows players to interact with stakeholders that are defined as a village or town (three locales in four decrease. The model takes into account seasonal popu- districts) that players can interact with. lation fluctuations, frequency of resistance, intensity of The actions that the player can perform at each map resistance, resistance mechanisms, mosquito behaviour, level are different. At the district map level, the player can intervention quality, intervention coverage, and com- initiate entomological surveillance activities at any of the munity engagement, among other factors, in producing district locales (Fig. 1a). These activities involve collect- the outputs. In this model, a handful of parameters can ing mosquitoes to monitor transmission intensity, vector be changed to generate various scenarios. The parameter behaviour, or insecticide resistance. Players can choose values themselves are stored in an editable spreadsheet how they identify their mosquitoes (by morphology or in the cloud, which allows the behaviour of the game to PCR), which collection methods they use, and which be changed without the game code itself being modi- assays they will use to characterize resistance. Any deci- fied. The game model, which comprises approximately 20 sions made here will impact the data that is available to lines of code, was originally written in R and can be found them later. At the province map level (Fig. 1b), the player here: https://github.com/AndySouth/resistanceGame. can initiate interventions including the distribution of ResistanceSim includes several indicators of player pro- long-lasting insecticidal nets (LLINs) or IRS using vari- gress so that the user understands how they are doing. ous insecticides. In addition, they can perform commu- First, there are a series of stoplight symbols above each of nity engagement, training, or intervention monitoring the district labels in the province map view (Fig. 3). These activities. At the national capital, the player can interact icons can either be pink, amber, or green, and provide a with stakeholders in various ways, including fundrais- quick indication of whether the player has collected the ing, sharing data, and participating in planning meet- recommended type of data in that district. Second, there ings (Fig. 1c). The player is in control of time, so they can are the district and province health bars. These bars indi- queue up any number of actions across all geographical cate the relative health of that particular district or the levels before advancing time. Once they do advance time, province as a whole. The value displayed in these bars is the game moves forward 1 month and any actions they directly related to the transmission, and therefore pro- have put in the queue will be completed. Each action is vides an indication of how well vector control is working. associated with a cost, and the appropriate amount of Lastly, after each advance of time, the player is presented money will be deducted from the player’s budget as they with a summary of the training, community engagement, perform actions. and health levels of each district, as well as an indica- The player can view data that they collect from either tion of whether each of these levels is going up or down the district level (Fig. 2a) or the province level (Fig. 2b). (Fig. 4). The data that appears in the data visualization screens is determined by what actions the player has performed. Roadmap For example, if a player completes transmission monitor- To provide the player with some direction as they are ing activities in months 6–12 of year 1, but not months first learning how the simulation works, they can play 1–5, they will only see the data for the second half of through the Roadmap (Fig. 5). The Roadmap consists of a the year. The game model (described below) generates series of missions, each with its own goal, learning objec- the underlying values for all the data visualization com- tives, and decisions that need to be made. The missions ponents. These values are influenced by the player’s follow a logical progression: engaging with stakeholders, decisions. collecting baseline data, followed by missions describing A very simple mathematical model (a few core lines the data visualization components and how to interpret of code) was developed to get the mosquito populations that data, and finally some missions on how to deploy in ResistanceSim to react in realistic ways, in terms of interventions. The Roadmap continues so that players both abundance and resistance levels, to player inputs. can then monitor their intervention, evaluate the data Therefore, if a player deploys an effective intervention, after the first year, and plan for another intervention the they will see the mosquito population go down. Con- following year. versely, if they deploy an intervention that the mosquito Each mission begins with a start screen that describes population is resistant to, they will see a less dramatic what the goal of the mission is and what the player is Thomsen et al. Malar J (2018) 17:422 Page 4 of 15 Thomsen et al. Malar J (2018) 17:422 Page 5 of 15 Fig. 2 The data visualization components at the a district level and b province level. Players can collect and visualize data on vector species composition, behaviour, and density, malaria transmission, insecticide susceptibility, resistance intensity, resistance mechanisms, intervention quality, and residual efficacy expected to learn (Fig. 6). Once the player presses describes why players received their particular star-rat- “Start Mission”, they are guided through the various ing, and provides hints on how to get more stars. steps required to complete their goal. Depending on the decisions they make during the mission, players IRM course can receive various star-ratings on the feedback screen The Roadmap and the open simulation described upon mission completion (Fig. 7), with good decisions above were incorporated into a gaming-enhanced earning players more stars. The feedback screen also insecticide resistance management training course. Thomsen et al. Malar J (2018) 17:422 Page 6 of 15 Fig. 3 A simple stoplight visual to indicate whether the player has collected the recommended types of data. Clicking on the lights reveal hints for changing the colour of the light (shown on left) Fig. 4 The feedback window that appears every time the player chooses to advance time. It gives a quick snapshot of how health, community engagement, and training levels are changing in each district This course lasts between 2 and 3 days, depending how The mini lecture on a particular topic is given just many modules the course facilitator wants to include. before students play through the corresponding mis- In its most condensed form, the course begins on the sion, so they have the opportunity to apply their learn- first day with a series of mini lectures interspersed ing immediately. with short bursts of gameplay in the Roadmap. This The second day comprises group work and gameplay allows students of all backgrounds to begin playing in the open simulation. Students are given one of sev- the game with the same foundational knowledge. Mini eral IRM strategies to employ in the open simulation. lecture topics include: mosquito collection methods, They are then given the opportunity to implement this vector control tools, insecticide resistance and how strategy for several hours. At the end, each student or to measure it, intervention monitoring strategies, etc. group presents the results of their strategy to the rest Thomsen et al. Malar J (2018) 17:422 Page 7 of 15 Fig. 5 The Roadmap is a series of missions designed to provide structure to the simulation. The player starts with missions on stakeholder engagement and baseline data collection (shown in figure), and continues to play missions related to selecting interventions and monitoring the impact of those interventions Fig. 6 The mission start screen indicates to the player the learning objectives for this particular mission, and what the goal of the mission is Thomsen et al. Malar J (2018) 17:422 Page 8 of 15 Fig. 7 The mission feedback screen provides immediate feedback on the player’s decisions in the mission, assigning an overall star-rating for all the decisions that were made. It also provides hints one how to improve the star-rating. Clicking on “More Info” will provide the player with additional in-depth feedback on each of the decisions they made during the level, indicating why the decision was good or bad of the class, so that all students can benefit from each interspersed with frequent user testing and external advi- other’s experience. sory committee meetings. These processes are detailed below. Platform ResistanceSim was produced using the Unity game Stakeholder workshops engine for use on Windows and Mac-based PCs, as well Two workshops were held early in the ResistanceSim as android tablets. The complexity of the user interface development process. The first was held over 2 days in prevented the adaptation of the game for smartphones May 2015 with the primary aims to discuss and deter- due to the average size of screens. It can be used with or mine (1) the learning objectives that would be incor- without an internet connection. porated into the game specification, (2) the value and potential use of current disease control mathematical Development process models to support learning objectives, and (3) game The development process for ResistanceSim continued design and scenario options to best support the learning for just over 2 years from May 2015 to September 2017 objectives. Participants at this workshop included repre- (Fig. 8). It generally followed the ADDIE instructional sentatives from malaria control programmes in sub-Saha- design framework, which organizes the development of ran Africa and Southeast Asia, mathematical modellers, instructional materials into analysis, design, develop- potential funding partners, members of the Engaging ment, implementation, and evaluation procedures . Tools for Communication in Health (ETCH) team at the This manuscript highlights the analysis, design, and Liverpool School of Tropical Medicine (LSTM), and the development processes. WHO. The first step involved convening stakeholders to The objectives were achieved through a guided analyse the need for such a tool, define the learning brainstorming session using a modified Charrette objectives and target audience, clarify the role of math- procedure , followed by group discussions. Prior ematical models in the game, and identify delivery to the workshop, the organizers identified four major strategies. Software developers were then engaged and categories of activities related to IRM where vector learning objectives were mapped to game elements in a control programmes currently face challenges: plan- living game design document. Development sprints were ning and implementation of IRM strategies, resistance Thomsen et al. Malar J (2018) 17:422 Page 9 of 15 Fig. 8 The processes involved in the development of ResistanceSim. Ongoing activities are indicated in the three boxes at the top monitoring, current and new tools (for surveillance, team, WHO Global Malaria Programme (GMP), and control, quality assurance, etc.), and the biology of Abt Associates, the implementers of the President’s resistance. There was also an “other” topic to capture Malaria Initiative (PMI) Africa Indoor Residual Spray challenges that did not fit easily into a single category. (AIRS) project. Opinions of the workshop participants Workshop participants were placed in groups of 4–5 on various aspects of the rollout strategy were gathered individuals, and each group spent 10 min brainstorm- through interactive polling (Turning Technologies). ing challenges faced by vector control programmes related to a single topic. They rotated until all groups Advisory committee meetings visited all topics. Challenges were summarized by the Quarterly advisory meetings were held with an external workshop leaders and re-phrased into potential learn- panel. Panel members had expertise in insecticide resist- ing objectives. The mathematical modeller partici- ance, pedagogy, and public health. The advisory commit- pants provided their expert opinion on whether/how tee provided direction across several different aspects of each learning objective could be supported by the use the development project, including the technical accu- of existing mathematical models. These discussions racy of ResistanceSim, the teaching strategies embedded allowed the workshop organizers to produce a living in the tool, and the methods used to evaluate it. They also document that defined the game’s learning objectives provided recommendations on synergies with existing and the role of mathematical models in supporting research or vector control implementation projects. these objectives. The second workshop was held over 2 days in Janu- Playability testing ary 2016. The objective of this workshop was to define Routine testing was conducted throughout the develop- the preferred rollout strategy for the game, including ment of ResistanceSim by the ETCH team. Playability how to make the game available and how it should be testing with external users was performed four times used. Participants in this workshop included repre- coinciding with major development milestones. The pri- sentatives from malaria control programmes in sub- mary objective of these testing sessions was to identify Saharan Africa, potential funding partners, the ETCH bugs and usability issues. However, if the testers were Thomsen et al. Malar J (2018) 17:422 Page 10 of 15 members of the target audience, a secondary objective (Protocol 16-016) and the Medical Research Council of was to assess acceptability as a learning tool. Zimbabwe (Protocol MRCZ/E/140). In May 2016, 26 users were recruited from LSTM Results from the Likert-scale survey were summarized and stakeholder organizations to test the first beta ver- with standard statistical measures of mean and standard sion of ResistanceSim. This version had all the required error. Any comparisons between pre- and post-question- functionality but had not been tested to ensure it was naires were made using paired t-tests. The audio from free of defects. Users were given a copy of the software the workshop was transcribed and analysed inductively. with instructions on how to install it on their personal Illustrative quotes for each theme were documented. laptops. They were also given a structured spread- Results from both testing sessions described above sheet that allowed them to capture usability issues as were fed back into another large development sprint they were playing, and were asked to complete a short which lasted for approximately 9 months. Major changes survey rating their experience playing the game. They were made during this time to improve usability of the answered questions about their engagement, the ease in tool. The third playability testing session was held at which they learned how to play the game, and how easy LSTM in April of 2017, and included six users purpose- it was to understand the various components. Users fully selected with expertise in education or operational were then allowed to play the game in their own time vector control. These users were given a brief introduc- over the course of 8 days, and their responses were col- tion to the tool, and were allowed to play through the lected afterwards via email. All bugs identified during game for 3 h, documenting any bugs or usability issues this beta testing were fixed prior to further user testing. in a similar format to the first testing session. Pedagogi- The second major testing session occurred in Zim- cal feedback on the delivery of the tool was particularly babwe in July 2016. During this time, the AIRS project useful at this time, and was used to shape the develop- was conducting a regional entomological training ses- ment of a more comprehensive facilitated session. This sion. It included 30 participants representing malaria facilitated session, which included gameplay, directed vector control programmes from 11 countries in sub- activities using the game, group work, and mini lectures Saharan Africa. For this session, the game was tested on was finally tested with a group of 20 individuals at LSTM the final day of the week-long training course. Partici- in July 2017. The users included individuals well-versed pants were asked to complete a short pre-game survey with vector control and insecticide resistance, as well as to capture demographic information and awareness of those less familiar in order to gauge the response of a IRM training resources. The survey also included Lik- diverse audience. Bugs and usability issues were docu- ert-scale questions to evaluate participants’ perceptions mented in a similar manner. of demand for IRM training tools, of their own IRM knowledge, and of games and people who play games. Results It also included an open-answer question asking them Refining learning objectives and rollout to describe the steps involved in IRM. Then, partici- The original list of learning objectives generated from the pants were given a brief introduction to the game and first workshop included 21 items across the topics of vec- could play on their personal laptops for approximately tors, resistance, disease epidemiology, chemical-based 3 h while a facilitator circulated around the room to interventions, intervention monitoring and impact evalu- answer any questions. After the play session, partici- ation, finances, stakeholders, and unforeseen challenges. pants were placed in groups and provided with discus- Over the course of designing, developing and testing sion questions in one of three topics: positive aspects the game, these learning objectives were further refined of the game, barriers to a positive user experience, or (Table 1). It was also recognized that certain learning barriers to sustainable implementation. After 20 min, objectives may take longer to achieve through gameplay the groups rotated in a Charrette procedure (described than others, such as evaluating the cost-effectiveness of above) so that all groups contributed to all topics. At various intervention strategies. the end of this workshop, the facilitator led a discussion Workshop participants identified that malaria trans- about each topic and asked groups to explain or expand mission models, including OpenMalaria and EMOD, on certain aspects. Audio from the discussion was were more detailed than necessary to support the recorded. Participants were asked to complete a post- learning objectives and at that time had little considera- game survey that included many of the same questions tion of insecticide resistance. Even if they were thought as the pre-game survey, but in addition asked them for suitable, it would be impossible to get these models their perceptions on individual game elements, as well to run in the background of the game due to a lack of as the value of the game as a whole. This research was computing power. Population genetic models to pre- approved by institutional review boards at the LSTM dict the evolution of insecticide resistance  also Thomsen et al. Malar J (2018) 17:422 Page 11 of 15 Table 1 Complete list of learning objectives addressed in ResistanceSim Topic Learning objective Stakeholders Identify which stakeholders to involve in insecticide resistance management planning Vectors Compare the data obtained from various mosquito collection methods Compare the data obtained from different species identification methods Identify which collection method is required to determine transmission intensity Explain why it is important to use consistent collection sites Explain how vector bionomics influence intervention choices Resistance Describe the process of generating insecticide susceptibility data Identify the collection and test methods available to determine insecticide susceptibility, resistance intensity, and resistance mechanisms Describe the data required to construct a resistance profile Explain the importance of species identification in constructing a resistance profile and interpreting resistance data Illustrate the effect of continuously using insecticides with one mode of action Evaluate the different insecticide resistance management strategies available Apply this evaluation to make an appropriate resistance management plan Evidence-based decisions Explain why it is important to look at data before making an intervention decision Evaluate what insecticide class to use based on the resistance data Assess when to deploy an intervention based on vector density and transmission data Intervention monitoring Explain why it is important to use consistent methodology for routine monitoring Identify the information that different intervention monitoring tools provide Explain how quality assuring interventions contributes to insecticide resistance management Compare the information gathered from different monitoring tools Explain why it is important to monitor transmission Explain why it is important to monitor insecticide susceptibility, resistance intensity, and resistance mechanisms Demonstrate how to improve the quality and coverage of an intervention Finances Evaluate the cost-effectiveness of various intervention strategies These learning objectives were first identified during stakeholder workshops, and further revised during the game development process contain more detail than is necessary to support the impact. To encourage the uptake of the tool, it was sug- learning objectives. Since ResistanceSim is designed to gested that a comprehensive curriculum and course be a learning tool, and not a decision-support tool, it structure were created and distributed with the game was decided that an extensive validated model was not itself. This would serve as a facilitator’s guide, and make needed. All that was needed was something that would it easier for country vector control programmes to adopt generate outputs to the players within game scenarios the tool. that would support individual learning objectives. This, therefore, led to the development of the ResistanceSim Beta testing 1 game model (described above). In order to develop and Results from the first beta testing session held at the test the model outputs, a web application was devel- LSTM and remotely with other stakeholders produced oped using the Shiny package (RStudio Inc.) to allow a list of 32 bugs. Usability issues were numerous, and the development team to manipulate model parameters included confusion about the tutorial section, how data and test various scenarios quickly and easily. Simulta- is presented in the game, and whether players’ decisions neously, the game developers transferred the code to were good or bad and why. Players’ opinions of the game C#, the language used by game development platform at this time were neutral, neither agreeing nor disagree- Unity, so that the mosquito populations in the game ing with many of the survey questions (Fig. 9). After dis- reacted as expected. cussing these issues with the beta testers and amongst Participants in the rollout workshop felt that the game the ETCH team, a list of 79 change requests were pro- should be incorporated into existing IRM training activi- duced to help address some of the issues with data visual- ties, rather than being played individually or in a sepa- ization, the tutorial section, and player feedback. Prior to rate session. In addition, it was decided that playing the the next testing session in Zimbabwe, all bugs were fixed, game as part of a facilitated session would have the most Thomsen et al. Malar J (2018) 17:422 Page 12 of 15 5 4.5 Strongly disagree - strongly agree 4 3.5 3 2.5 2 1.5 1 The game It was easy I want to The The The scoring The I found The game was fun to to learn play the entomolgy intervenon in the game graphics in playing the was too play. how to play game again. data in the data in the was easy to the game game difficult. the game. game was game was understand. were frustrang. easy to easy to appropriate. understand. understand. Fig. 9 User perceptions (n = 8) of the first beta version of ResistanceSim. Error bars represent the standard error of the mean and change requests were prioritized to focus on the clar- head trying to work out how to navigate around and fig- ity of the tutorial section and data visualization. ure out how to play the game, I feel like you need to be able to jump in a lot more quickly.” In addition, users Beta testing 2 expressed frustration in the way the instructions are User experiences in Zimbabwe were more positive than presented: “We are not engaging with the game. The in the first testing session. Users generally felt that the actual reason (for this) is that the instructions are not game improved their understanding of various topics clear.” They were also disappointed by the lack of direc- related to vector control (Fig. 10a), and that the data pre- tion: “I can see the provincial health bar going up and sented in the game was easy to understand. The tutorial down, but there is no specific goal,” “As a player, you section was still difficult for users to work through, and should be able to monitor independently how you are this was reflected in both the survey answers (Fig. 10b) doing as far as your learning,” “It should have different as well as the progress that most people made during the levels.” test session—only 4 out of 30 participants were able to Despite these difficulties, 90% of participants indi- make it past the tutorial during the 3-h play session. cated that they need more support related to IRM, and Feedback during the workshop discussion shed more they agreed that the tool would be a valuable addition light on the positive and negative aspects of the game. to the training currently available for IRM and vector Players enjoyed the fact that their own actions in the control (Fig. 10b). game influenced the outcomes: “We also like the inter- All of the feedback from the LSTM and Zimba- pretation of data where you could see the impact of IRS bwe testing sessions were consolidated and solutions on vector density.” They also expressed satisfaction in were proposed to address most of the usability issues. the complexity of the topics covered in the game: “We The solutions fell in two categories: tutorial and inter- liked how the game instructed you to start your activi- face improvements. The tutorial section was reworked ties at national level, then move to the province, to the replacing it with a guided, mission-oriented “Roadmap.” district, down to the village … this makes you aware of This guides players through the various aspects of the the need to involve all levels in terms of implementa- game itself, while slowly introducing the complexity of tion and planning.” However, it also became clear that the content. The Roadmap provides immediate feed- while complexity in the topics covered was desirable, back on player decisions, so that they know what they complexity in the user interface was preventing users are doing well and why. The second category of game from interacting meaningfully with content: “Rather changes involved simplifying the user interface of the than spend maybe an hour or 2 h just cracking your open simulation while retaining the complexity of the Thomsen et al. Malar J (2018) 17:422 Page 13 of 15 a 7 6 Strongly disagree - strongly agree 5 4 3 2 1 how to how to decide how to how how to how to how to monitor vector which vector manage intervenons monitor interpret construct an populaons control inseccide influence intervenon various types inseccide intervenons resistance inseccide efficacy of resistance resistance to choose resistance data management plan b 7 6 Strongly disagree - strongly agree 5 4 3 2 1 The tutorial session The tutorial session The tutorial session It was easy to It was easy to figure at the beginning of at the beginning of at the beginning of understand what out how complete the game was... the game was... the game was... was happening in acons in the game [easy to understand] [informave] [relevant to the rest the game of the game] Fig. 10 User perceptions (n = 28) during the second beta testing session in Zimbabwe of a the degree to which ResistanceSim improved their understanding of various topics and b the ease of use of the tutorial section. Error bars represent the standard error of the mean content it covered. Changes in this category included Usability testing reworking how data is displayed, removing extrane- Results from usability testing in April 2017 indicated a ous aspects of the user interface, and providing regu- vast improvement in the game. All participants (n = 6) lar updates to the player about how their decisions are felt that ResistanceSim would be a valuable addition to an impacting game outcomes. All of these changes were IRM course. In contrast to the first beta testing session completed over a software development sprint lasting conducted in May 2016 (Fig. 9), all participants indicated approximately 9 months. that they wanted to play the game again. However, most Thomsen et al. Malar J (2018) 17:422 Page 14 of 15 also felt that in order to get the most out of the game, and external advisory boards, and frequent user test- players needed to spend more time with it: “…by the time ing focused on playability and perceived usefulness. The you get through the missions (Roadmap), I felt then pre- results from this work are promising, in that the final pared to go into the game. But it’s almost like you need user-led product has been deemed a valuable potential some thinking time…it requires time to get the most out of addition to IRM training activities. As serious games it.” In addition, one participant who did not have a back- have been shown to have positive impacts on knowledge ground in vector control found it difficult to understand and motivation , an important next step will be to what they were doing because of unfamiliarity with some evaluate ResistanceSim for its effect on knowledge acqui- of the terminology used. It was suggested that additional sition, self-efficacy, and decision-making behaviours in learning material be provided that allowed all users to vector control programmes that have used the game as a start with the same level of knowledge. A total of 22 bugs training tool. and 14 usability issues were identified and documented Serious games have been used extensively in the health in both the Roadmap and the open simulation. Most of field, particularly aimed at training health profession- the usability issues related to the transition between als [12, 17] or changing behaviour of patients to improve the Roadmap and the open simulation, where users are their health outcomes [10, 11, 18]. However, to our introduced to some new functionality that is not explic- knowledge, there are no serious games that target public itly described. Feedback from this session resulted in two health policy implementers, whose decisions have a mas- major developments. First, a series of short tutorial vid- sive impact on the health of many individuals. In addi- eos were created to ease the transition from the Roadmap tion, are only few examples of games being used in low to the open simulation. Second, a structured lesson plan and middle income countries or focused on diseases of and additional teaching resources were created (exer- poverty [19–23]. With computer use ubiquitous across cises, discussion topics, and slide sets) so that Resistanc- multiple sectors in sub-Saharan Africa, and continuing eSim was integrated into a facilitated course on IRM. to increase , this presents a significant opportunity to utilize technology as a capacity strengthening tool. Training course Previous literature reviews highlighted the necessity The facilitated ResistanceSim training course was tested of iterative evaluation of instructional elements, game- with 20 individuals at LSTM in July 2017. The course play mechanics, and user interface  when design- lasted from 0900 to 1600 h with time for breaks. Feed- ing serious games. The results from this study reiterate back was gathered through a simple open questionnaire this recommendation. Despite the early involvement of that asked participants what they liked about the course subject experts, game designers, and regular reviews and what could be improved. In general, participants from an external advisory committee, the first beta test- were enthusiastic about the tool, and expressed satisfac- ing revealed that users simply did not enjoy playing the tion with the complimentary course material: “I liked the game. It was only after additional revisions and testing linkage/balance between course instruction and activities,” that a product was produced that struck the right balance “[the additional components] added considerable value to between engagement and instruction that motivated the ResistanceSim game itself.” The value of the Roadmap users to keep playing. was recognized as a way to slowly introduce complicated The iterative nature of the development process also concepts: “I liked the look and atmosphere of the applica- allowed the elucidation of potential implementation tion and the way that the structure built up as you got fur- strategies, since users indicated that the game should be ther into the modules and I started to make linkages and used as part of a structured course. This allowed us to test adopt reinforced behaviours etc.” In addition, some sug- the game in this context during the final stage of develop- gestions were made to improve the exercises and group ment. The instructional resources are available for open work that were completed as part of the course so that all use (at etch.lstmed.ac.uk), so that potential Resistanc- participants can equally benefit from the ResistanceSim eSim course facilitators have guidance on the curriculum tool itself. and structure of the course. Discussion Conclusions A ‘serious game’ was developed aimed at improving In order to ensure the sustainability of public health understanding of insecticide resistance management insecticides, they must be used judiciously and intel- strategies among vector control programme personnel, ligently. Strengthening the capacity of malaria vector with the ultimate goal of influencing decision-making control programmes to manage insecticide resistance is processes. Over the course of 2 years, the game was eval- a critical component of this, but training resources are uated for its validity through consultation with experts limited. ResistanceSim, developed here, is a management Thomsen et al. Malar J (2018) 17:422 Page 15 of 15 simulation game that immerses the player in a fictitious References 1. Global Partnership to Roll Back Malaria. The Abuja declaration on roll back vector control programme, to fill this gap. Early and malaria in Africa. Geneva; 2000. repeat testing with target users and involvement of stake- 2. Hemingway J, Ranson H. Insecticide resistance in insect vectors of human holders was vital in the development of the tool. This disease. Annu Rev Entomol. 2000;45:371–91. 3. Hemingway J. The role of vector control in stopping the transmis- process has enabled us to improve user experience and sion of malaria: threats and opportunities. Philos Trans R Soc B. provide a viable environment for learning. The potential 2014;369:20130431. for this serious game to be useful in training has been 4. Ranson H, N’guessan R, Lines J, Moiroux N, Nkuni Z, Corbel V. Pyrethroid resistance in African anopheline mosquitoes: what are the implications demonstrated, and its utility in operational settings is for malaria control? Trends Parasitol. 2011;27:91–8. currently being tested. 5. REX consortium. Heterogeneity of selection and the evolution of resist- ance. Trends Ecol Evol. 2013;28:110–8. Authors’ contributions 6. WHO. Global plan for insecticide resistance management in malaria vec- EKT designed the tool, designed the study, collected the data, analysed the tors. Geneva: World Health Organization; 2012. data, wrote the manuscript. CH designed the tool, designed the study, col- 7. Reigeluth CM. Instructional-design theories and models: a new paradigm lected the data, analysed the data, reviewed the manuscript. AS designed the of instructional theory. Routledge; 2013. tool, designed the game model, reviewed the manuscript. KAD, CD collected 8. Bandura A. Health promotion by social cognitive means. Health Educ the data, reviewed the manuscript. MiC supported the development of the Behav. 2004;31:143–64. tool, reviewed the manuscript. RF designed the tool, reviewed the manuscript. 9. Djaouti D, Alvarez J, Jessel J, Rampnoux O. Origins of serious games. Seri- MaC conceived of the tool, designed the tool, reviewed the manuscript. All ous games edutainment applications. London: Springer; 2011. p. 25–43. authors read and approved the final manuscript. 10. Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves behavioral outcomes in adolescents and young adults with cancer: a Author details randomized trial. Pediatrics. 2008;122:e305–17. 1 Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK. 11. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Mathijs F. The 2 Extra Mile Studios, 142 West Nile St, Glasgow G1 2RQ, UK. 3 Present Address: effectiveness of SPARX, a computerised self help intervention for adoles- IVCC, Pembroke Pl, Liverpool L3 5QA, UK. cents seeking help for depression : randomised controlled non-inferiority trial. BMJ. 2012;344:2598. Acknowledgements 12. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious We would like to acknowledge the IVCC for support and guidance during the games for medical education and surgical skills training. Br J Surg. game development process, as well as Abt Associates for agreeing to host one 2012;99:1322–30. of the testing sessions. 13. Branson R, Rayner GT, Cox JL, Furman JP, King J. Interservice procedures for instructional systems development. Executive Summary and Model. Competing interests 1975. BF is the CEO of Extra Mile Studios, the developer of ResistanceSim. However, 14. Cramer HL, Wehking RJ. Charretting the planning process. Washington, as the tool is free to use and distribute, the company does not stand to gain D.C.; 1973. anything from the publication of this manuscript. 15. Levick B, South A, Hastings IM. A two-locus model of the evolution of MaC currently works for IVCC, the primary funder of this tool. However, insecticide resistance to inform and optimise public health insecticide at the time of development and testing of ResistanceSim, MaC was affiliated deployment strategies. PLoS Comput Biol. 2017;13:e1005327. with the Liverpool School of Tropical Medicine. 16. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on computer games and serious games. Comput Educ. 2012;59:661–86. Availability of data and materials 17. Wang R, DeMaria S, Goldberg A, Katz D. A systematic review of serious The datasets generated and/or analysed during the current study are available games in training health care professionals. Simul Healthc. 2016;11:41–51. from the corresponding author on reasonable request. 18. DeSmet A, Van Ryckeghem D, Compernolle S, Baranowski T, Thompson D, Crombez G, et al. A meta-analysis of serious digital games for healthy Consent for publication lifestyle promotion. Prev Med (Baltim). 2014;69:95–107. Not applicable. 19. Luz S, Masoodian M, Cesario RR, Cesario M. Using a serious game to pro- mote community-based awareness and prevention of neglected tropical Ethics approval and consent to participate diseases. Entertain Comput. 2015;15:43–55. https://doi.org/10.1016/j. This research was approved by institutional review boards at the LSTM entcom.2015.11.001. (Protocol 16-016) and the Medical Research Council of Zimbabwe (Protocol 20. Buchinger D, da Hounsell M, Um S. Jogo Sério Colaborativo Para MRCZ/E/140). Aprender Sobre a Doença da Dengue. Informática Educ Teor Prática. 2015;18:67–84. Funding 21. Lima T, Barbosa B, Niquini C, Araujo C, Lana R. Playing against dengue Development of ResistanceSim was provided by the Bill and Melinda Gates design and development of a serious game to help tackling dengue. In: Foundations through the IVCC (OPP1148615) and by the Wellcome Trust 2017 IEEE 5th Int. Conf. Serious Games Appl. Heal. SeGAH. 2017. Institutional Strategic Support Fund. 22. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Dis- tributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS ONE. 2012;7:e37245. Publisher’s Note 23. Kam M, Agarwal A, Kumar A, Lal S. Designing e-learning games for rural Springer Nature remains neutral with regard to jurisdictional claims in pub- children in India: a format for balancing learning with fun. In: Proc. 7th lished maps and institutional affiliations. ACM Conf. Des. Interact. Syst. 2008. p. 58–67. 24. Cirera X, Lage F, Sabetti L. ICT use, innovation, and productivity: evidence Received: 23 February 2018 Accepted: 9 November 2018 from Sub-Saharan Africa. Policy Res. Work. Pap. 2016. 25. Papastergiou M. Exploring the potential of computer and video games for health and physical education: a literature review. Comput Educ. 2009;53:603–22.