DOKK Library

ResistanceSim: development and acceptability study of a serious game to improve understanding of insecticide resistance management in vector control programmes

Authors Andy South Charlotte Hemingway Claire Dormann Edward K. Thomsen Kirsten A. Duda Marlize Coleman Michael Coleman Robert Farmer

License CC-BY-4.0

                                                                                                                                       Malaria Journal
Thomsen et al. Malar J    (2018) 17:422

 CASE STUDY                                                                                                                                      Open Access

ResistanceSim: development
and acceptability study of a serious game
to improve understanding of insecticide
resistance management in vector control
Edward K. Thomsen1, Charlotte Hemingway1, Andy South1, Kirsten A. Duda1, Claire Dormann1, Robert Farmer2,
Michael Coleman1* and Marlize Coleman1,3

  The use of insecticides is the cornerstone of effective malaria vector control. However, the last two decades has seen
  the ubiquitous use of insecticides, predominantly pyrethroids, causing widespread insecticide resistance and compro-
  mising the effectiveness of vector control. Considerable efforts to develop new active ingredients and interventions
  are underway. However, it is essential to deploy strategies to mitigate the impact of insecticide resistance now, both
  to maintain the efficacy of currently available tools as well as to ensure the sustainability of new tools as they come to
  market. Although the World Health Organization disseminated best practice guidelines for insecticide resistance man-
  agement (IRM), Rollback Malaria’s Vector Control Working Group identified the lack of practical knowledge of IRM as
  the primary gap in the translation of evidence into policy. ResistanceSim is a capacity strengthening tool designed to
  address this gap. The development process involved frequent stakeholder consultation, including two separate work-
  shops. These workshops defined the learning objectives, target audience, and the role of mathematical models in the
  game. Software development phases were interspersed with frequent user testing, resulting in an iterative design
  process. User feedback was evaluated via questionnaires with Likert-scale and open-ended questions. The game was
  regularly evaluated by subject-area experts through meetings of an external advisory panel. Through these processes,
  a series of learning domains were identified and a set of specific learning objectives for each domain were defined
  to be communicated to vector control programme personnel. A simple “game model” was proposed that produces
  realistic outputs based on player strategy and also runs in real-time. Early testing sessions revealed numerous usability
  issues that prevented adequate player engagement. After extensive revisions, later testing sessions indicated that the
  tool would be a valuable addition to IRM training.
  Keywords: Serious games, Insecticide resistance management, Insecticide resistance, Vector control, Capacity
  building, Training

  Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3
Full list of author information is available at the end of the article

                                          © The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
                                          (http://creat​iveco​mmons​.org/licen​ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
                                          provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license,
                                          and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creat​iveco​mmons​.org/
                                          publi​cdoma​in/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Thomsen et al. Malar J   (2018) 17:422                                                                              Page 2 of 15

Background                                                       theory suggests that effective learning is accomplished
In 2000, with the signing of the Abuja Declaration, lead-        through active involvement of the learner, a self-directed
ers from malaria-endemic countries across sub-Saharan            approach, and working with realistic scenarios [7]. All of
Africa committed themselves to decrease the burden of            these criteria are central to a simulation game. In addi-
malaria [1]. This increase in political will was rapidly fol-    tion, social cognitive theory is based on the idea that
lowed by greater financial support from global partners.         behaviour is driven by the understanding of the world in
As a result, after three decades of stagnation since the         which a person lives, including the positive and negative
close of the World Health Organization (WHO) Global              outcomes witnessed as a result of choices made [8] and
Malaria Elimination Programme in 1969, the last 18 years         beliefs in personal efficacy; games influence the player’s
has seen a rapid scale-up of malaria control interven-           understanding of the world around them by enabling
tions. Insecticide-based vector control lies at the heart of     them to explore complex problems in a safe setting,
the global strategy.                                             allowing them to make mistakes and learn from them
   Pyrethroids, with their low mammalian toxicity, long          without real-world consequences.
residual life, and relatively low production cost, became          The value of serious games has seen increased atten-
the dominant insecticide class of choice during the scale-       tion from many industries since 2002 [9], most notably
up. At the time of the declaration, resistance to pyre-          the healthcare sector. Games have been used to improve
throids was almost negligible, with just a few populations       adherence to self-medication among cancer patients [10],
of vectors exhibiting resistance on the African continent        relieve symptoms of depression [11], and train medical
[2]. Now, with the extensive use of these insecticides for       and surgical personnel [12], among many other applica-
both insecticide-treated nets (ITNs) and indoor residual         tions. However, games have not yet been used among
spraying (IRS), not a single country in sub-Saharan Africa       the implementers of public health programmes, where
is free from pyrethroid resistance [3, 4]. Resistance to all     relatively few individuals are responsible for engaging
other classes of public health insecticides is ubiquitous        in complex decision-making processes that ultimately
as well. Consequently, the ability to control the vectors        impact the health of tens of thousands of people. In addi-
responsible for transmitting the disease is compromised.         tion, there have been relatively few applications of seri-
   The path to this situation is characterized by an insuf-      ous games in low and middle income countries where
ficient safeguarding of the available insecticide products.      increased computer literacy is producing a generation
Despite proven strategies to curb resistance [5], vec-           that may be particularly receptive to digital gaming
tor control programmes around the world have relied              solutions.
exclusively on monotherapies, mainly pyrethroids, for              Here a serious game was developed to improve under-
years. Cross-resistance between this and other insecti-          standing and adoption of strategies to manage insecticide
cide classes limits the number of alternatives, resulting        resistance among vector control programmes in malaria-
in development of further resistance to these products           endemic countries. Here, the game development and the
as well. These practices result in some countries without        process of developing it, is presented with results from
viable vector control strategies.                                preliminary acceptability studies.
   If there is not a culture change surrounding public
health insecticide use, there is a risk that the effectiveness   ResistanceSim
of existing and new insecticides will be compromised by          Open simulation
resistance. Recognizing the gravity of the current situa-        ResistanceSim is a management simulation game that
tion, the WHO published the Global Plan for Insecticide          immerses players in a fictional sub-Saharan African
Resistance Management (GPIRM) [6], which provides                country. The player can interact with several environ-
technical recommendations for national control and               ments (Fig. 1). At the province level, the player sees four
elimination programmes to sustainably manage resist-             districts. By clicking on one of the district labels, the
ance. However, the operational implementation of these           game zooms into the district level, where the player can
recommendations is lacking, and innovative solutions are         interact with several villages or towns. At the district level
required to communicate the principles and implications          view, the player can rotate, pan, and zoom the camera to
of insecticide resistance management (IRM).                      investigate their environment. At any time, the player can
   ‘Serious games’ are games designed for purposes               access the national capital, where they can interact with
beyond mere entertainment. They blend the engaging,              various stakeholders. Each geographical location in the
fun, and challenging components of gaming with the               game has different characteristics: the mosquito species
goal of supplying the player with skills and knowledge           present, their behaviour, their insecticide resistance pro-
useful in real-life situations, ultimately supporting atti-      file, and the malaria transmission season and intensity all
tude and behaviour change. Modern instructional design           vary from place to place. There are a total of 12 locales
Thomsen et al. Malar J     (2018) 17:422                                                                                                       Page 3 of 15

 (See figure on next page.)
 Fig. 1 The three different map levels of ResistanceSim. a Shows the district level, which allows players to perform actions in three locales per
 district. b Shows the province level, which allows players to perform actions in four districts. c Shows the national capital, which allows players to
 interact with stakeholders

that are defined as a village or town (three locales in four                    decrease. The model takes into account seasonal popu-
districts) that players can interact with.                                      lation fluctuations, frequency of resistance, intensity of
  The actions that the player can perform at each map                           resistance, resistance mechanisms, mosquito behaviour,
level are different. At the district map level, the player can                  intervention quality, intervention coverage, and com-
initiate entomological surveillance activities at any of the                    munity engagement, among other factors, in producing
district locales (Fig. 1a). These activities involve collect-                   the outputs. In this model, a handful of parameters can
ing mosquitoes to monitor transmission intensity, vector                        be changed to generate various scenarios. The parameter
behaviour, or insecticide resistance. Players can choose                        values themselves are stored in an editable spreadsheet
how they identify their mosquitoes (by morphology or                            in the cloud, which allows the behaviour of the game to
PCR), which collection methods they use, and which                              be changed without the game code itself being modi-
assays they will use to characterize resistance. Any deci-                      fied. The game model, which comprises approximately 20
sions made here will impact the data that is available to                       lines of code, was originally written in R and can be found
them later. At the province map level (Fig. 1b), the player                     here: https​://githu​​outh/resis​tance​Game.
can initiate interventions including the distribution of                           ResistanceSim includes several indicators of player pro-
long-lasting insecticidal nets (LLINs) or IRS using vari-                       gress so that the user understands how they are doing.
ous insecticides. In addition, they can perform commu-                          First, there are a series of stoplight symbols above each of
nity engagement, training, or intervention monitoring                           the district labels in the province map view (Fig. 3). These
activities. At the national capital, the player can interact                    icons can either be pink, amber, or green, and provide a
with stakeholders in various ways, including fundrais-                          quick indication of whether the player has collected the
ing, sharing data, and participating in planning meet-                          recommended type of data in that district. Second, there
ings (Fig. 1c). The player is in control of time, so they can                   are the district and province health bars. These bars indi-
queue up any number of actions across all geographical                          cate the relative health of that particular district or the
levels before advancing time. Once they do advance time,                        province as a whole. The value displayed in these bars is
the game moves forward 1 month and any actions they                             directly related to the transmission, and therefore pro-
have put in the queue will be completed. Each action is                         vides an indication of how well vector control is working.
associated with a cost, and the appropriate amount of                           Lastly, after each advance of time, the player is presented
money will be deducted from the player’s budget as they                         with a summary of the training, community engagement,
perform actions.                                                                and health levels of each district, as well as an indica-
  The player can view data that they collect from either                        tion of whether each of these levels is going up or down
the district level (Fig. 2a) or the province level (Fig. 2b).                   (Fig. 4).
The data that appears in the data visualization screens is
determined by what actions the player has performed.                            Roadmap
For example, if a player completes transmission monitor-                        To provide the player with some direction as they are
ing activities in months 6–12 of year 1, but not months                         first learning how the simulation works, they can play
1–5, they will only see the data for the second half of                         through the Roadmap (Fig. 5). The Roadmap consists of a
the year. The game model (described below) generates                            series of missions, each with its own goal, learning objec-
the underlying values for all the data visualization com-                       tives, and decisions that need to be made. The missions
ponents. These values are influenced by the player’s                            follow a logical progression: engaging with stakeholders,
decisions.                                                                      collecting baseline data, followed by missions describing
  A very simple mathematical model (a few core lines                            the data visualization components and how to interpret
of code) was developed to get the mosquito populations                          that data, and finally some missions on how to deploy
in ResistanceSim to react in realistic ways, in terms of                        interventions. The Roadmap continues so that players
both abundance and resistance levels, to player inputs.                         can then monitor their intervention, evaluate the data
Therefore, if a player deploys an effective intervention,                       after the first year, and plan for another intervention the
they will see the mosquito population go down. Con-                             following year.
versely, if they deploy an intervention that the mosquito                          Each mission begins with a start screen that describes
population is resistant to, they will see a less dramatic                       what the goal of the mission is and what the player is
Thomsen et al. Malar J   (2018) 17:422   Page 4 of 15
Thomsen et al. Malar J     (2018) 17:422                                                                                                     Page 5 of 15

 Fig. 2 The data visualization components at the a district level and b province level. Players can collect and visualize data on vector species
 composition, behaviour, and density, malaria transmission, insecticide susceptibility, resistance intensity, resistance mechanisms, intervention
 quality, and residual efficacy

expected to learn (Fig. 6). Once the player presses                           describes why players received their particular star-rat-
“Start Mission”, they are guided through the various                          ing, and provides hints on how to get more stars.
steps required to complete their goal. Depending on
the decisions they make during the mission, players                           IRM course
can receive various star-ratings on the feedback screen                       The Roadmap and the open simulation described
upon mission completion (Fig. 7), with good decisions                         above were incorporated into a gaming-enhanced
earning players more stars. The feedback screen also                          insecticide resistance management training course.
Thomsen et al. Malar J    (2018) 17:422                                                                                                    Page 6 of 15

 Fig. 3 A simple stoplight visual to indicate whether the player has collected the recommended types of data. Clicking on the lights reveal hints for
 changing the colour of the light (shown on left)

 Fig. 4 The feedback window that appears every time the player chooses to advance time. It gives a quick snapshot of how health, community
 engagement, and training levels are changing in each district

This course lasts between 2 and 3 days, depending how                        The mini lecture on a particular topic is given just
many modules the course facilitator wants to include.                        before students play through the corresponding mis-
In its most condensed form, the course begins on the                         sion, so they have the opportunity to apply their learn-
first day with a series of mini lectures interspersed                        ing immediately.
with short bursts of gameplay in the Roadmap. This                             The second day comprises group work and gameplay
allows students of all backgrounds to begin playing                          in the open simulation. Students are given one of sev-
the game with the same foundational knowledge. Mini                          eral IRM strategies to employ in the open simulation.
lecture topics include: mosquito collection methods,                         They are then given the opportunity to implement this
vector control tools, insecticide resistance and how                         strategy for several hours. At the end, each student or
to measure it, intervention monitoring strategies, etc.                      group presents the results of their strategy to the rest
Thomsen et al. Malar J     (2018) 17:422                                                                                                      Page 7 of 15

 Fig. 5 The Roadmap is a series of missions designed to provide structure to the simulation. The player starts with missions on stakeholder
 engagement and baseline data collection (shown in figure), and continues to play missions related to selecting interventions and monitoring the
 impact of those interventions

 Fig. 6 The mission start screen indicates to the player the learning objectives for this particular mission, and what the goal of the mission is
Thomsen et al. Malar J    (2018) 17:422                                                                                                    Page 8 of 15

 Fig. 7 The mission feedback screen provides immediate feedback on the player’s decisions in the mission, assigning an overall star-rating for all the
 decisions that were made. It also provides hints one how to improve the star-rating. Clicking on “More Info” will provide the player with additional
 in-depth feedback on each of the decisions they made during the level, indicating why the decision was good or bad

of the class, so that all students can benefit from each                      interspersed with frequent user testing and external advi-
other’s experience.                                                           sory committee meetings. These processes are detailed
ResistanceSim was produced using the Unity game                               Stakeholder workshops
engine for use on Windows and Mac-based PCs, as well                          Two workshops were held early in the ResistanceSim
as android tablets. The complexity of the user interface                      development process. The first was held over 2 days in
prevented the adaptation of the game for smartphones                          May 2015 with the primary aims to discuss and deter-
due to the average size of screens. It can be used with or                    mine (1) the learning objectives that would be incor-
without an internet connection.                                               porated into the game specification, (2) the value and
                                                                              potential use of current disease control mathematical
Development process                                                           models to support learning objectives, and (3) game
The development process for ResistanceSim continued                           design and scenario options to best support the learning
for just over 2 years from May 2015 to September 2017                         objectives. Participants at this workshop included repre-
(Fig. 8). It generally followed the ADDIE instructional                       sentatives from malaria control programmes in sub-Saha-
design framework, which organizes the development of                          ran Africa and Southeast Asia, mathematical modellers,
instructional materials into analysis, design, develop-                       potential funding partners, members of the Engaging
ment, implementation, and evaluation procedures [13].                         Tools for Communication in Health (ETCH) team at the
This manuscript highlights the analysis, design, and                          Liverpool School of Tropical Medicine (LSTM), and the
development processes.                                                        WHO.
   The first step involved convening stakeholders to                            The objectives were achieved through a guided
analyse the need for such a tool, define the learning                         brainstorming session using a modified Charrette
objectives and target audience, clarify the role of math-                     procedure [14], followed by group discussions. Prior
ematical models in the game, and identify delivery                            to the workshop, the organizers identified four major
strategies. Software developers were then engaged and                         categories of activities related to IRM where vector
learning objectives were mapped to game elements in a                         control programmes currently face challenges: plan-
living game design document. Development sprints were                         ning and implementation of IRM strategies, resistance
Thomsen et al. Malar J    (2018) 17:422                                                                                              Page 9 of 15

 Fig. 8 The processes involved in the development of ResistanceSim. Ongoing activities are indicated in the three boxes at the top

monitoring, current and new tools (for surveillance,                        team, WHO Global Malaria Programme (GMP), and
control, quality assurance, etc.), and the biology of                       Abt Associates, the implementers of the President’s
resistance. There was also an “other” topic to capture                      Malaria Initiative (PMI) Africa Indoor Residual Spray
challenges that did not fit easily into a single category.                  (AIRS) project. Opinions of the workshop participants
Workshop participants were placed in groups of 4–5                          on various aspects of the rollout strategy were gathered
individuals, and each group spent 10 min brainstorm-                        through interactive polling (Turning Technologies).
ing challenges faced by vector control programmes
related to a single topic. They rotated until all groups                    Advisory committee meetings
visited all topics. Challenges were summarized by the                       Quarterly advisory meetings were held with an external
workshop leaders and re-phrased into potential learn-                       panel. Panel members had expertise in insecticide resist-
ing objectives. The mathematical modeller partici-                          ance, pedagogy, and public health. The advisory commit-
pants provided their expert opinion on whether/how                          tee provided direction across several different aspects of
each learning objective could be supported by the use                       the development project, including the technical accu-
of existing mathematical models. These discussions                          racy of ResistanceSim, the teaching strategies embedded
allowed the workshop organizers to produce a living                         in the tool, and the methods used to evaluate it. They also
document that defined the game’s learning objectives                        provided recommendations on synergies with existing
and the role of mathematical models in supporting                           research or vector control implementation projects.
these objectives.
  The second workshop was held over 2 days in Janu-                         Playability testing
ary 2016. The objective of this workshop was to define                      Routine testing was conducted throughout the develop-
the preferred rollout strategy for the game, including                      ment of ResistanceSim by the ETCH team. Playability
how to make the game available and how it should be                         testing with external users was performed four times
used. Participants in this workshop included repre-                         coinciding with major development milestones. The pri-
sentatives from malaria control programmes in sub-                          mary objective of these testing sessions was to identify
Saharan Africa, potential funding partners, the ETCH                        bugs and usability issues. However, if the testers were
Thomsen et al. Malar J   (2018) 17:422                                                                       Page 10 of 15

members of the target audience, a secondary objective         (Protocol 16-016) and the Medical Research Council of
was to assess acceptability as a learning tool.               Zimbabwe (Protocol MRCZ/E/140).
  In May 2016, 26 users were recruited from LSTM                 Results from the Likert-scale survey were summarized
and stakeholder organizations to test the first beta ver-     with standard statistical measures of mean and standard
sion of ResistanceSim. This version had all the required      error. Any comparisons between pre- and post-question-
functionality but had not been tested to ensure it was        naires were made using paired t-tests. The audio from
free of defects. Users were given a copy of the software      the workshop was transcribed and analysed inductively.
with instructions on how to install it on their personal      Illustrative quotes for each theme were documented.
laptops. They were also given a structured spread-               Results from both testing sessions described above
sheet that allowed them to capture usability issues as        were fed back into another large development sprint
they were playing, and were asked to complete a short         which lasted for approximately 9 months. Major changes
survey rating their experience playing the game. They         were made during this time to improve usability of the
answered questions about their engagement, the ease in        tool. The third playability testing session was held at
which they learned how to play the game, and how easy         LSTM in April of 2017, and included six users purpose-
it was to understand the various components. Users            fully selected with expertise in education or operational
were then allowed to play the game in their own time          vector control. These users were given a brief introduc-
over the course of 8 days, and their responses were col-      tion to the tool, and were allowed to play through the
lected afterwards via email. All bugs identified during       game for 3 h, documenting any bugs or usability issues
this beta testing were fixed prior to further user testing.   in a similar format to the first testing session. Pedagogi-
  The second major testing session occurred in Zim-           cal feedback on the delivery of the tool was particularly
babwe in July 2016. During this time, the AIRS project        useful at this time, and was used to shape the develop-
was conducting a regional entomological training ses-         ment of a more comprehensive facilitated session. This
sion. It included 30 participants representing malaria        facilitated session, which included gameplay, directed
vector control programmes from 11 countries in sub-           activities using the game, group work, and mini lectures
Saharan Africa. For this session, the game was tested on      was finally tested with a group of 20 individuals at LSTM
the final day of the week-long training course. Partici-      in July 2017. The users included individuals well-versed
pants were asked to complete a short pre-game survey          with vector control and insecticide resistance, as well as
to capture demographic information and awareness of           those less familiar in order to gauge the response of a
IRM training resources. The survey also included Lik-         diverse audience. Bugs and usability issues were docu-
ert-scale questions to evaluate participants’ perceptions     mented in a similar manner.
of demand for IRM training tools, of their own IRM
knowledge, and of games and people who play games.            Results
It also included an open-answer question asking them          Refining learning objectives and rollout
to describe the steps involved in IRM. Then, partici-         The original list of learning objectives generated from the
pants were given a brief introduction to the game and         first workshop included 21 items across the topics of vec-
could play on their personal laptops for approximately        tors, resistance, disease epidemiology, chemical-based
3 h while a facilitator circulated around the room to         interventions, intervention monitoring and impact evalu-
answer any questions. After the play session, partici-        ation, finances, stakeholders, and unforeseen challenges.
pants were placed in groups and provided with discus-         Over the course of designing, developing and testing
sion questions in one of three topics: positive aspects       the game, these learning objectives were further refined
of the game, barriers to a positive user experience, or       (Table 1). It was also recognized that certain learning
barriers to sustainable implementation. After 20 min,         objectives may take longer to achieve through gameplay
the groups rotated in a Charrette procedure (described        than others, such as evaluating the cost-effectiveness of
above) so that all groups contributed to all topics. At       various intervention strategies.
the end of this workshop, the facilitator led a discussion       Workshop participants identified that malaria trans-
about each topic and asked groups to explain or expand        mission models, including OpenMalaria and EMOD,
on certain aspects. Audio from the discussion was             were more detailed than necessary to support the
recorded. Participants were asked to complete a post-         learning objectives and at that time had little considera-
game survey that included many of the same questions          tion of insecticide resistance. Even if they were thought
as the pre-game survey, but in addition asked them for        suitable, it would be impossible to get these models
their perceptions on individual game elements, as well        to run in the background of the game due to a lack of
as the value of the game as a whole. This research was        computing power. Population genetic models to pre-
approved by institutional review boards at the LSTM           dict the evolution of insecticide resistance [15] also
Thomsen et al. Malar J       (2018) 17:422                                                                                                     Page 11 of 15

Table 1 Complete list of learning objectives addressed in ResistanceSim
Topic                                  Learning objective

Stakeholders                           Identify which stakeholders to involve in insecticide resistance management planning
Vectors                                Compare the data obtained from various mosquito collection methods
                                       Compare the data obtained from different species identification methods
                                       Identify which collection method is required to determine transmission intensity
                                       Explain why it is important to use consistent collection sites
                                       Explain how vector bionomics influence intervention choices
Resistance                             Describe the process of generating insecticide susceptibility data
                                       Identify the collection and test methods available to determine insecticide susceptibility, resistance intensity, and
                                         resistance mechanisms
                                       Describe the data required to construct a resistance profile
                                       Explain the importance of species identification in constructing a resistance profile and interpreting resistance data
                                       Illustrate the effect of continuously using insecticides with one mode of action
                                       Evaluate the different insecticide resistance management strategies available
                                       Apply this evaluation to make an appropriate resistance management plan
Evidence-based decisions               Explain why it is important to look at data before making an intervention decision
                                       Evaluate what insecticide class to use based on the resistance data
                                       Assess when to deploy an intervention based on vector density and transmission data
Intervention monitoring                Explain why it is important to use consistent methodology for routine monitoring
                                       Identify the information that different intervention monitoring tools provide
                                       Explain how quality assuring interventions contributes to insecticide resistance management
                                       Compare the information gathered from different monitoring tools
                                       Explain why it is important to monitor transmission
                                       Explain why it is important to monitor insecticide susceptibility, resistance intensity, and resistance mechanisms
                                       Demonstrate how to improve the quality and coverage of an intervention
Finances                               Evaluate the cost-effectiveness of various intervention strategies
These learning objectives were first identified during stakeholder workshops, and further revised during the game development process

contain more detail than is necessary to support the                               impact. To encourage the uptake of the tool, it was sug-
learning objectives. Since ResistanceSim is designed to                            gested that a comprehensive curriculum and course
be a learning tool, and not a decision-support tool, it                            structure were created and distributed with the game
was decided that an extensive validated model was not                              itself. This would serve as a facilitator’s guide, and make
needed. All that was needed was something that would                               it easier for country vector control programmes to adopt
generate outputs to the players within game scenarios                              the tool.
that would support individual learning objectives. This,
therefore, led to the development of the ResistanceSim                             Beta testing 1
game model (described above). In order to develop and                              Results from the first beta testing session held at the
test the model outputs, a web application was devel-                               LSTM and remotely with other stakeholders produced
oped using the Shiny package (RStudio Inc.) to allow                               a list of 32 bugs. Usability issues were numerous, and
the development team to manipulate model parameters                                included confusion about the tutorial section, how data
and test various scenarios quickly and easily. Simulta-                            is presented in the game, and whether players’ decisions
neously, the game developers transferred the code to                               were good or bad and why. Players’ opinions of the game
C#, the language used by game development platform                                 at this time were neutral, neither agreeing nor disagree-
Unity, so that the mosquito populations in the game                                ing with many of the survey questions (Fig. 9). After dis-
reacted as expected.                                                               cussing these issues with the beta testers and amongst
  Participants in the rollout workshop felt that the game                          the ETCH team, a list of 79 change requests were pro-
should be incorporated into existing IRM training activi-                          duced to help address some of the issues with data visual-
ties, rather than being played individually or in a sepa-                          ization, the tutorial section, and player feedback. Prior to
rate session. In addition, it was decided that playing the                         the next testing session in Zimbabwe, all bugs were fixed,
game as part of a facilitated session would have the most
Thomsen et al. Malar J                                     (2018) 17:422                                                                                              Page 12 of 15


              Strongly disagree - strongly agree







                                                         The game It was easy I want to       The         The      The scoring     The        I found   The game
                                                         was fun to to learn     play the entomolgy intervenon in the game graphics in playing the was too
                                                           play.   how to play game again. data in the data in the was easy to the game        game      difficult.
                                                                    the game.              game was game was understand.          were      frustrang.
                                                                                            easy to     easy to                appropriate.
                                                                                          understand. understand.
 Fig. 9 User perceptions (n = 8) of the first beta version of ResistanceSim. Error bars represent the standard error of the mean

and change requests were prioritized to focus on the clar-                                                  head trying to work out how to navigate around and fig-
ity of the tutorial section and data visualization.                                                         ure out how to play the game, I feel like you need to be
                                                                                                            able to jump in a lot more quickly.” In addition, users
Beta testing 2                                                                                              expressed frustration in the way the instructions are
User experiences in Zimbabwe were more positive than                                                        presented: “We are not engaging with the game. The
in the first testing session. Users generally felt that the                                                 actual reason (for this) is that the instructions are not
game improved their understanding of various topics                                                         clear.” They were also disappointed by the lack of direc-
related to vector control (Fig. 10a), and that the data pre-                                                tion: “I can see the provincial health bar going up and
sented in the game was easy to understand. The tutorial                                                     down, but there is no specific goal,” “As a player, you
section was still difficult for users to work through, and                                                  should be able to monitor independently how you are
this was reflected in both the survey answers (Fig. 10b)                                                    doing as far as your learning,” “It should have different
as well as the progress that most people made during the                                                    levels.”
test session—only 4 out of 30 participants were able to                                                       Despite these difficulties, 90% of participants indi-
make it past the tutorial during the 3-h play session.                                                      cated that they need more support related to IRM, and
   Feedback during the workshop discussion shed more                                                        they agreed that the tool would be a valuable addition
light on the positive and negative aspects of the game.                                                     to the training currently available for IRM and vector
Players enjoyed the fact that their own actions in the                                                      control (Fig. 10b).
game influenced the outcomes: “We also like the inter-                                                        All of the feedback from the LSTM and Zimba-
pretation of data where you could see the impact of IRS                                                     bwe testing sessions were consolidated and solutions
on vector density.” They also expressed satisfaction in                                                     were proposed to address most of the usability issues.
the complexity of the topics covered in the game: “We                                                       The solutions fell in two categories: tutorial and inter-
liked how the game instructed you to start your activi-                                                     face improvements. The tutorial section was reworked
ties at national level, then move to the province, to the                                                   replacing it with a guided, mission-oriented “Roadmap.”
district, down to the village … this makes you aware of                                                     This guides players through the various aspects of the
the need to involve all levels in terms of implementa-                                                      game itself, while slowly introducing the complexity of
tion and planning.” However, it also became clear that                                                      the content. The Roadmap provides immediate feed-
while complexity in the topics covered was desirable,                                                       back on player decisions, so that they know what they
complexity in the user interface was preventing users                                                       are doing well and why. The second category of game
from interacting meaningfully with content: “Rather                                                         changes involved simplifying the user interface of the
than spend maybe an hour or 2 h just cracking your                                                          open simulation while retaining the complexity of the
Thomsen et al. Malar J                                    (2018) 17:422                                                                                              Page 13 of 15

                            a 7

              Strongly disagree - strongly agree





                                                          how to     how to decide       how to             how       how to      how to        how to
                                                       monitor vector which vector       manage       intervenons   monitor     interpret   construct an
                                                        populaons       control       inseccide        influence intervenon various types inseccide
                                                                      intervenons     resistance       inseccide    efficacy of resistance  resistance
                                                                        to choose                       resistance                  data     management

                               b 7

              Strongly disagree - strongly agree





                                                        The tutorial session The tutorial session The tutorial session     It was easy to    It was easy to figure
                                                        at the beginning of at the beginning of at the beginning of       understand what     out how complete
                                                          the game was...      the game was...      the game was...       was happening in   acons in the game
                                                       [easy to understand]     [informave]      [relevant to the rest       the game
                                                                                                      of the game]
 Fig. 10 User perceptions (n = 28) during the second beta testing session in Zimbabwe of a the degree to which ResistanceSim improved their
 understanding of various topics and b the ease of use of the tutorial section. Error bars represent the standard error of the mean

content it covered. Changes in this category included                                                      Usability testing
reworking how data is displayed, removing extrane-                                                         Results from usability testing in April 2017 indicated a
ous aspects of the user interface, and providing regu-                                                     vast improvement in the game. All participants (n = 6)
lar updates to the player about how their decisions are                                                    felt that ResistanceSim would be a valuable addition to an
impacting game outcomes. All of these changes were                                                         IRM course. In contrast to the first beta testing session
completed over a software development sprint lasting                                                       conducted in May 2016 (Fig. 9), all participants indicated
approximately 9 months.                                                                                    that they wanted to play the game again. However, most
Thomsen et al. Malar J   (2018) 17:422                                                                         Page 14 of 15

also felt that in order to get the most out of the game,       and external advisory boards, and frequent user test-
players needed to spend more time with it: “…by the time       ing focused on playability and perceived usefulness. The
you get through the missions (Roadmap), I felt then pre-       results from this work are promising, in that the final
pared to go into the game. But it’s almost like you need       user-led product has been deemed a valuable potential
some thinking time…it requires time to get the most out of     addition to IRM training activities. As serious games
it.” In addition, one participant who did not have a back-     have been shown to have positive impacts on knowledge
ground in vector control found it difficult to understand      and motivation [16], an important next step will be to
what they were doing because of unfamiliarity with some        evaluate ResistanceSim for its effect on knowledge acqui-
of the terminology used. It was suggested that additional      sition, self-efficacy, and decision-making behaviours in
learning material be provided that allowed all users to        vector control programmes that have used the game as a
start with the same level of knowledge. A total of 22 bugs     training tool.
and 14 usability issues were identified and documented            Serious games have been used extensively in the health
in both the Roadmap and the open simulation. Most of           field, particularly aimed at training health profession-
the usability issues related to the transition between         als [12, 17] or changing behaviour of patients to improve
the Roadmap and the open simulation, where users are           their health outcomes [10, 11, 18]. However, to our
introduced to some new functionality that is not explic-       knowledge, there are no serious games that target public
itly described. Feedback from this session resulted in two     health policy implementers, whose decisions have a mas-
major developments. First, a series of short tutorial vid-     sive impact on the health of many individuals. In addi-
eos were created to ease the transition from the Roadmap       tion, are only few examples of games being used in low
to the open simulation. Second, a structured lesson plan       and middle income countries or focused on diseases of
and additional teaching resources were created (exer-          poverty [19–23]. With computer use ubiquitous across
cises, discussion topics, and slide sets) so that Resistanc-   multiple sectors in sub-Saharan Africa, and continuing
eSim was integrated into a facilitated course on IRM.          to increase [24], this presents a significant opportunity to
                                                               utilize technology as a capacity strengthening tool.
Training course                                                   Previous literature reviews highlighted the necessity
The facilitated ResistanceSim training course was tested       of iterative evaluation of instructional elements, game-
with 20 individuals at LSTM in July 2017. The course           play mechanics, and user interface [25] when design-
lasted from 0900 to 1600 h with time for breaks. Feed-         ing serious games. The results from this study reiterate
back was gathered through a simple open questionnaire          this recommendation. Despite the early involvement of
that asked participants what they liked about the course       subject experts, game designers, and regular reviews
and what could be improved. In general, participants           from an external advisory committee, the first beta test-
were enthusiastic about the tool, and expressed satisfac-      ing revealed that users simply did not enjoy playing the
tion with the complimentary course material: “I liked the      game. It was only after additional revisions and testing
linkage/balance between course instruction and activities,”    that a product was produced that struck the right balance
“[the additional components] added considerable value to       between engagement and instruction that motivated
the ResistanceSim game itself.” The value of the Roadmap       users to keep playing.
was recognized as a way to slowly introduce complicated           The iterative nature of the development process also
concepts: “I liked the look and atmosphere of the applica-     allowed the elucidation of potential implementation
tion and the way that the structure built up as you got fur-   strategies, since users indicated that the game should be
ther into the modules and I started to make linkages and       used as part of a structured course. This allowed us to test
adopt reinforced behaviours etc.” In addition, some sug-       the game in this context during the final stage of develop-
gestions were made to improve the exercises and group          ment. The instructional resources are available for open
work that were completed as part of the course so that all     use (at, so that potential Resistanc-
participants can equally benefit from the ResistanceSim        eSim course facilitators have guidance on the curriculum
tool itself.                                                   and structure of the course.

Discussion                                                     Conclusions
A ‘serious game’ was developed aimed at improving              In order to ensure the sustainability of public health
understanding of insecticide resistance management             insecticides, they must be used judiciously and intel-
strategies among vector control programme personnel,           ligently. Strengthening the capacity of malaria vector
with the ultimate goal of influencing decision-making          control programmes to manage insecticide resistance is
processes. Over the course of 2 years, the game was eval-      a critical component of this, but training resources are
uated for its validity through consultation with experts       limited. ResistanceSim, developed here, is a management
Thomsen et al. Malar J       (2018) 17:422                                                                                                       Page 15 of 15

simulation game that immerses the player in a fictitious                         References
                                                                                 1. Global Partnership to Roll Back Malaria. The Abuja declaration on roll back
vector control programme, to fill this gap. Early and                                malaria in Africa. Geneva; 2000.
repeat testing with target users and involvement of stake-                       2. Hemingway J, Ranson H. Insecticide resistance in insect vectors of human
holders was vital in the development of the tool. This                               disease. Annu Rev Entomol. 2000;45:371–91.
                                                                                 3. Hemingway J. The role of vector control in stopping the transmis-
process has enabled us to improve user experience and                                sion of malaria: threats and opportunities. Philos Trans R Soc B.
provide a viable environment for learning. The potential                             2014;369:20130431.
for this serious game to be useful in training has been                          4. Ranson H, N’guessan R, Lines J, Moiroux N, Nkuni Z, Corbel V. Pyrethroid
                                                                                     resistance in African anopheline mosquitoes: what are the implications
demonstrated, and its utility in operational settings is                             for malaria control? Trends Parasitol. 2011;27:91–8.
currently being tested.                                                          5. REX consortium. Heterogeneity of selection and the evolution of resist-
                                                                                     ance. Trends Ecol Evol. 2013;28:110–8.
Authors’ contributions                                                           6. WHO. Global plan for insecticide resistance management in malaria vec-
EKT designed the tool, designed the study, collected the data, analysed the          tors. Geneva: World Health Organization; 2012.
data, wrote the manuscript. CH designed the tool, designed the study, col-       7. Reigeluth CM. Instructional-design theories and models: a new paradigm
lected the data, analysed the data, reviewed the manuscript. AS designed the         of instructional theory. Routledge; 2013.
tool, designed the game model, reviewed the manuscript. KAD, CD collected        8. Bandura A. Health promotion by social cognitive means. Health Educ
the data, reviewed the manuscript. MiC supported the development of the              Behav. 2004;31:143–64.
tool, reviewed the manuscript. RF designed the tool, reviewed the manuscript.    9. Djaouti D, Alvarez J, Jessel J, Rampnoux O. Origins of serious games. Seri-
MaC conceived of the tool, designed the tool, reviewed the manuscript. All           ous games edutainment applications. London: Springer; 2011. p. 25–43.
authors read and approved the final manuscript.                                  10. Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves
                                                                                     behavioral outcomes in adolescents and young adults with cancer: a
Author details                                                                       randomized trial. Pediatrics. 2008;122:e305–17.
  Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK.   11. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Mathijs F. The
  Extra Mile Studios, 142 West Nile St, Glasgow G1 2RQ, UK. 3 Present Address:       effectiveness of SPARX, a computerised self help intervention for adoles-
IVCC, Pembroke Pl, Liverpool L3 5QA, UK.                                             cents seeking help for depression : randomised controlled non-inferiority
                                                                                     trial. BMJ. 2012;344:2598.
Acknowledgements                                                                 12. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious
We would like to acknowledge the IVCC for support and guidance during the            games for medical education and surgical skills training. Br J Surg.
game development process, as well as Abt Associates for agreeing to host one         2012;99:1322–30.
of the testing sessions.                                                         13. Branson R, Rayner GT, Cox JL, Furman JP, King J. Interservice procedures
                                                                                     for instructional systems development. Executive Summary and Model.
Competing interests                                                                  1975.
BF is the CEO of Extra Mile Studios, the developer of ResistanceSim. However,    14. Cramer HL, Wehking RJ. Charretting the planning process. Washington,
as the tool is free to use and distribute, the company does not stand to gain        D.C.; 1973.
anything from the publication of this manuscript.                                15. Levick B, South A, Hastings IM. A two-locus model of the evolution of
    MaC currently works for IVCC, the primary funder of this tool. However,          insecticide resistance to inform and optimise public health insecticide
at the time of development and testing of ResistanceSim, MaC was affiliated          deployment strategies. PLoS Comput Biol. 2017;13:e1005327.
with the Liverpool School of Tropical Medicine.                                  16. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic
                                                                                     literature review of empirical evidence on computer games and serious
                                                                                     games. Comput Educ. 2012;59:661–86.
Availability of data and materials                                               17. Wang R, DeMaria S, Goldberg A, Katz D. A systematic review of serious
The datasets generated and/or analysed during the current study are available        games in training health care professionals. Simul Healthc. 2016;11:41–51.
from the corresponding author on reasonable request.                             18. DeSmet A, Van Ryckeghem D, Compernolle S, Baranowski T, Thompson
                                                                                     D, Crombez G, et al. A meta-analysis of serious digital games for healthy
Consent for publication                                                              lifestyle promotion. Prev Med (Baltim). 2014;69:95–107.
Not applicable.                                                                  19. Luz S, Masoodian M, Cesario RR, Cesario M. Using a serious game to pro-
                                                                                     mote community-based awareness and prevention of neglected tropical
Ethics approval and consent to participate                                           diseases. Entertain Comput. 2015;15:43–55. https​://
This research was approved by institutional review boards at the LSTM                entco​m.2015.11.001.
(Protocol 16-016) and the Medical Research Council of Zimbabwe (Protocol         20. Buchinger D, da Hounsell M, Um S. Jogo Sério Colaborativo Para
MRCZ/E/140).                                                                         Aprender Sobre a Doença da Dengue. Informática Educ Teor Prática.
Funding                                                                          21. Lima T, Barbosa B, Niquini C, Araujo C, Lana R. Playing against dengue
Development of ResistanceSim was provided by the Bill and Melinda Gates              design and development of a serious game to help tackling dengue. In:
Foundations through the IVCC (OPP1148615) and by the Wellcome Trust                  2017 IEEE 5th Int. Conf. Serious Games Appl. Heal. SeGAH. 2017.
Institutional Strategic Support Fund.                                            22. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Dis-
                                                                                     tributed medical image analysis and diagnosis through crowd-sourced
                                                                                     games: a malaria case study. PLoS ONE. 2012;7:e37245.
Publisher’s Note                                                                 23. Kam M, Agarwal A, Kumar A, Lal S. Designing e-learning games for rural
Springer Nature remains neutral with regard to jurisdictional claims in pub-
                                                                                     children in India: a format for balancing learning with fun. In: Proc. 7th
lished maps and institutional affiliations.
                                                                                     ACM Conf. Des. Interact. Syst. 2008. p. 58–67.
                                                                                 24. Cirera X, Lage F, Sabetti L. ICT use, innovation, and productivity: evidence
Received: 23 February 2018 Accepted: 9 November 2018
                                                                                     from Sub-Saharan Africa. Policy Res. Work. Pap. 2016.
                                                                                 25. Papastergiou M. Exploring the potential of computer and video games
                                                                                     for health and physical education: a literature review. Comput Educ.