Authors Andy South Charlotte Hemingway Claire Dormann Edward K. Thomsen Kirsten A. Duda Marlize Coleman Michael Coleman Robert Farmer
License CC-BY-4.0
Malaria Journal
Thomsen et al. Malar J (2018) 17:422
https://doi.org/10.1186/s12936-018-2572-2
CASE STUDY Open Access
ResistanceSim: development
and acceptability study of a serious game
to improve understanding of insecticide
resistance management in vector control
programmes
Edward K. Thomsen1, Charlotte Hemingway1, Andy South1, Kirsten A. Duda1, Claire Dormann1, Robert Farmer2,
Michael Coleman1* and Marlize Coleman1,3
Abstract
The use of insecticides is the cornerstone of effective malaria vector control. However, the last two decades has seen
the ubiquitous use of insecticides, predominantly pyrethroids, causing widespread insecticide resistance and compro-
mising the effectiveness of vector control. Considerable efforts to develop new active ingredients and interventions
are underway. However, it is essential to deploy strategies to mitigate the impact of insecticide resistance now, both
to maintain the efficacy of currently available tools as well as to ensure the sustainability of new tools as they come to
market. Although the World Health Organization disseminated best practice guidelines for insecticide resistance man-
agement (IRM), Rollback Malaria’s Vector Control Working Group identified the lack of practical knowledge of IRM as
the primary gap in the translation of evidence into policy. ResistanceSim is a capacity strengthening tool designed to
address this gap. The development process involved frequent stakeholder consultation, including two separate work-
shops. These workshops defined the learning objectives, target audience, and the role of mathematical models in the
game. Software development phases were interspersed with frequent user testing, resulting in an iterative design
process. User feedback was evaluated via questionnaires with Likert-scale and open-ended questions. The game was
regularly evaluated by subject-area experts through meetings of an external advisory panel. Through these processes,
a series of learning domains were identified and a set of specific learning objectives for each domain were defined
to be communicated to vector control programme personnel. A simple “game model” was proposed that produces
realistic outputs based on player strategy and also runs in real-time. Early testing sessions revealed numerous usability
issues that prevented adequate player engagement. After extensive revisions, later testing sessions indicated that the
tool would be a valuable addition to IRM training.
Keywords: Serious games, Insecticide resistance management, Insecticide resistance, Vector control, Capacity
building, Training
*Correspondence: michael.coleman@lstmed.ac.uk
1
Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3
5QA, UK
Full list of author information is available at the end of the article
© The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license,
and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/
publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Thomsen et al. Malar J (2018) 17:422 Page 2 of 15
Background theory suggests that effective learning is accomplished
In 2000, with the signing of the Abuja Declaration, lead- through active involvement of the learner, a self-directed
ers from malaria-endemic countries across sub-Saharan approach, and working with realistic scenarios [7]. All of
Africa committed themselves to decrease the burden of these criteria are central to a simulation game. In addi-
malaria [1]. This increase in political will was rapidly fol- tion, social cognitive theory is based on the idea that
lowed by greater financial support from global partners. behaviour is driven by the understanding of the world in
As a result, after three decades of stagnation since the which a person lives, including the positive and negative
close of the World Health Organization (WHO) Global outcomes witnessed as a result of choices made [8] and
Malaria Elimination Programme in 1969, the last 18 years beliefs in personal efficacy; games influence the player’s
has seen a rapid scale-up of malaria control interven- understanding of the world around them by enabling
tions. Insecticide-based vector control lies at the heart of them to explore complex problems in a safe setting,
the global strategy. allowing them to make mistakes and learn from them
Pyrethroids, with their low mammalian toxicity, long without real-world consequences.
residual life, and relatively low production cost, became The value of serious games has seen increased atten-
the dominant insecticide class of choice during the scale- tion from many industries since 2002 [9], most notably
up. At the time of the declaration, resistance to pyre- the healthcare sector. Games have been used to improve
throids was almost negligible, with just a few populations adherence to self-medication among cancer patients [10],
of vectors exhibiting resistance on the African continent relieve symptoms of depression [11], and train medical
[2]. Now, with the extensive use of these insecticides for and surgical personnel [12], among many other applica-
both insecticide-treated nets (ITNs) and indoor residual tions. However, games have not yet been used among
spraying (IRS), not a single country in sub-Saharan Africa the implementers of public health programmes, where
is free from pyrethroid resistance [3, 4]. Resistance to all relatively few individuals are responsible for engaging
other classes of public health insecticides is ubiquitous in complex decision-making processes that ultimately
as well. Consequently, the ability to control the vectors impact the health of tens of thousands of people. In addi-
responsible for transmitting the disease is compromised. tion, there have been relatively few applications of seri-
The path to this situation is characterized by an insuf- ous games in low and middle income countries where
ficient safeguarding of the available insecticide products. increased computer literacy is producing a generation
Despite proven strategies to curb resistance [5], vec- that may be particularly receptive to digital gaming
tor control programmes around the world have relied solutions.
exclusively on monotherapies, mainly pyrethroids, for Here a serious game was developed to improve under-
years. Cross-resistance between this and other insecti- standing and adoption of strategies to manage insecticide
cide classes limits the number of alternatives, resulting resistance among vector control programmes in malaria-
in development of further resistance to these products endemic countries. Here, the game development and the
as well. These practices result in some countries without process of developing it, is presented with results from
viable vector control strategies. preliminary acceptability studies.
If there is not a culture change surrounding public
health insecticide use, there is a risk that the effectiveness ResistanceSim
of existing and new insecticides will be compromised by Open simulation
resistance. Recognizing the gravity of the current situa- ResistanceSim is a management simulation game that
tion, the WHO published the Global Plan for Insecticide immerses players in a fictional sub-Saharan African
Resistance Management (GPIRM) [6], which provides country. The player can interact with several environ-
technical recommendations for national control and ments (Fig. 1). At the province level, the player sees four
elimination programmes to sustainably manage resist- districts. By clicking on one of the district labels, the
ance. However, the operational implementation of these game zooms into the district level, where the player can
recommendations is lacking, and innovative solutions are interact with several villages or towns. At the district level
required to communicate the principles and implications view, the player can rotate, pan, and zoom the camera to
of insecticide resistance management (IRM). investigate their environment. At any time, the player can
‘Serious games’ are games designed for purposes access the national capital, where they can interact with
beyond mere entertainment. They blend the engaging, various stakeholders. Each geographical location in the
fun, and challenging components of gaming with the game has different characteristics: the mosquito species
goal of supplying the player with skills and knowledge present, their behaviour, their insecticide resistance pro-
useful in real-life situations, ultimately supporting atti- file, and the malaria transmission season and intensity all
tude and behaviour change. Modern instructional design vary from place to place. There are a total of 12 locales
Thomsen et al. Malar J (2018) 17:422 Page 3 of 15
(See figure on next page.)
Fig. 1 The three different map levels of ResistanceSim. a Shows the district level, which allows players to perform actions in three locales per
district. b Shows the province level, which allows players to perform actions in four districts. c Shows the national capital, which allows players to
interact with stakeholders
that are defined as a village or town (three locales in four decrease. The model takes into account seasonal popu-
districts) that players can interact with. lation fluctuations, frequency of resistance, intensity of
The actions that the player can perform at each map resistance, resistance mechanisms, mosquito behaviour,
level are different. At the district map level, the player can intervention quality, intervention coverage, and com-
initiate entomological surveillance activities at any of the munity engagement, among other factors, in producing
district locales (Fig. 1a). These activities involve collect- the outputs. In this model, a handful of parameters can
ing mosquitoes to monitor transmission intensity, vector be changed to generate various scenarios. The parameter
behaviour, or insecticide resistance. Players can choose values themselves are stored in an editable spreadsheet
how they identify their mosquitoes (by morphology or in the cloud, which allows the behaviour of the game to
PCR), which collection methods they use, and which be changed without the game code itself being modi-
assays they will use to characterize resistance. Any deci- fied. The game model, which comprises approximately 20
sions made here will impact the data that is available to lines of code, was originally written in R and can be found
them later. At the province map level (Fig. 1b), the player here: https://github.com/AndySouth/resistanceGame.
can initiate interventions including the distribution of ResistanceSim includes several indicators of player pro-
long-lasting insecticidal nets (LLINs) or IRS using vari- gress so that the user understands how they are doing.
ous insecticides. In addition, they can perform commu- First, there are a series of stoplight symbols above each of
nity engagement, training, or intervention monitoring the district labels in the province map view (Fig. 3). These
activities. At the national capital, the player can interact icons can either be pink, amber, or green, and provide a
with stakeholders in various ways, including fundrais- quick indication of whether the player has collected the
ing, sharing data, and participating in planning meet- recommended type of data in that district. Second, there
ings (Fig. 1c). The player is in control of time, so they can are the district and province health bars. These bars indi-
queue up any number of actions across all geographical cate the relative health of that particular district or the
levels before advancing time. Once they do advance time, province as a whole. The value displayed in these bars is
the game moves forward 1 month and any actions they directly related to the transmission, and therefore pro-
have put in the queue will be completed. Each action is vides an indication of how well vector control is working.
associated with a cost, and the appropriate amount of Lastly, after each advance of time, the player is presented
money will be deducted from the player’s budget as they with a summary of the training, community engagement,
perform actions. and health levels of each district, as well as an indica-
The player can view data that they collect from either tion of whether each of these levels is going up or down
the district level (Fig. 2a) or the province level (Fig. 2b). (Fig. 4).
The data that appears in the data visualization screens is
determined by what actions the player has performed. Roadmap
For example, if a player completes transmission monitor- To provide the player with some direction as they are
ing activities in months 6–12 of year 1, but not months first learning how the simulation works, they can play
1–5, they will only see the data for the second half of through the Roadmap (Fig. 5). The Roadmap consists of a
the year. The game model (described below) generates series of missions, each with its own goal, learning objec-
the underlying values for all the data visualization com- tives, and decisions that need to be made. The missions
ponents. These values are influenced by the player’s follow a logical progression: engaging with stakeholders,
decisions. collecting baseline data, followed by missions describing
A very simple mathematical model (a few core lines the data visualization components and how to interpret
of code) was developed to get the mosquito populations that data, and finally some missions on how to deploy
in ResistanceSim to react in realistic ways, in terms of interventions. The Roadmap continues so that players
both abundance and resistance levels, to player inputs. can then monitor their intervention, evaluate the data
Therefore, if a player deploys an effective intervention, after the first year, and plan for another intervention the
they will see the mosquito population go down. Con- following year.
versely, if they deploy an intervention that the mosquito Each mission begins with a start screen that describes
population is resistant to, they will see a less dramatic what the goal of the mission is and what the player is
Thomsen et al. Malar J (2018) 17:422 Page 4 of 15
Thomsen et al. Malar J (2018) 17:422 Page 5 of 15
Fig. 2 The data visualization components at the a district level and b province level. Players can collect and visualize data on vector species
composition, behaviour, and density, malaria transmission, insecticide susceptibility, resistance intensity, resistance mechanisms, intervention
quality, and residual efficacy
expected to learn (Fig. 6). Once the player presses describes why players received their particular star-rat-
“Start Mission”, they are guided through the various ing, and provides hints on how to get more stars.
steps required to complete their goal. Depending on
the decisions they make during the mission, players IRM course
can receive various star-ratings on the feedback screen The Roadmap and the open simulation described
upon mission completion (Fig. 7), with good decisions above were incorporated into a gaming-enhanced
earning players more stars. The feedback screen also insecticide resistance management training course.
Thomsen et al. Malar J (2018) 17:422 Page 6 of 15
Fig. 3 A simple stoplight visual to indicate whether the player has collected the recommended types of data. Clicking on the lights reveal hints for
changing the colour of the light (shown on left)
Fig. 4 The feedback window that appears every time the player chooses to advance time. It gives a quick snapshot of how health, community
engagement, and training levels are changing in each district
This course lasts between 2 and 3 days, depending how The mini lecture on a particular topic is given just
many modules the course facilitator wants to include. before students play through the corresponding mis-
In its most condensed form, the course begins on the sion, so they have the opportunity to apply their learn-
first day with a series of mini lectures interspersed ing immediately.
with short bursts of gameplay in the Roadmap. This The second day comprises group work and gameplay
allows students of all backgrounds to begin playing in the open simulation. Students are given one of sev-
the game with the same foundational knowledge. Mini eral IRM strategies to employ in the open simulation.
lecture topics include: mosquito collection methods, They are then given the opportunity to implement this
vector control tools, insecticide resistance and how strategy for several hours. At the end, each student or
to measure it, intervention monitoring strategies, etc. group presents the results of their strategy to the rest
Thomsen et al. Malar J (2018) 17:422 Page 7 of 15
Fig. 5 The Roadmap is a series of missions designed to provide structure to the simulation. The player starts with missions on stakeholder
engagement and baseline data collection (shown in figure), and continues to play missions related to selecting interventions and monitoring the
impact of those interventions
Fig. 6 The mission start screen indicates to the player the learning objectives for this particular mission, and what the goal of the mission is
Thomsen et al. Malar J (2018) 17:422 Page 8 of 15
Fig. 7 The mission feedback screen provides immediate feedback on the player’s decisions in the mission, assigning an overall star-rating for all the
decisions that were made. It also provides hints one how to improve the star-rating. Clicking on “More Info” will provide the player with additional
in-depth feedback on each of the decisions they made during the level, indicating why the decision was good or bad
of the class, so that all students can benefit from each interspersed with frequent user testing and external advi-
other’s experience. sory committee meetings. These processes are detailed
below.
Platform
ResistanceSim was produced using the Unity game Stakeholder workshops
engine for use on Windows and Mac-based PCs, as well Two workshops were held early in the ResistanceSim
as android tablets. The complexity of the user interface development process. The first was held over 2 days in
prevented the adaptation of the game for smartphones May 2015 with the primary aims to discuss and deter-
due to the average size of screens. It can be used with or mine (1) the learning objectives that would be incor-
without an internet connection. porated into the game specification, (2) the value and
potential use of current disease control mathematical
Development process models to support learning objectives, and (3) game
The development process for ResistanceSim continued design and scenario options to best support the learning
for just over 2 years from May 2015 to September 2017 objectives. Participants at this workshop included repre-
(Fig. 8). It generally followed the ADDIE instructional sentatives from malaria control programmes in sub-Saha-
design framework, which organizes the development of ran Africa and Southeast Asia, mathematical modellers,
instructional materials into analysis, design, develop- potential funding partners, members of the Engaging
ment, implementation, and evaluation procedures [13]. Tools for Communication in Health (ETCH) team at the
This manuscript highlights the analysis, design, and Liverpool School of Tropical Medicine (LSTM), and the
development processes. WHO.
The first step involved convening stakeholders to The objectives were achieved through a guided
analyse the need for such a tool, define the learning brainstorming session using a modified Charrette
objectives and target audience, clarify the role of math- procedure [14], followed by group discussions. Prior
ematical models in the game, and identify delivery to the workshop, the organizers identified four major
strategies. Software developers were then engaged and categories of activities related to IRM where vector
learning objectives were mapped to game elements in a control programmes currently face challenges: plan-
living game design document. Development sprints were ning and implementation of IRM strategies, resistance
Thomsen et al. Malar J (2018) 17:422 Page 9 of 15
Fig. 8 The processes involved in the development of ResistanceSim. Ongoing activities are indicated in the three boxes at the top
monitoring, current and new tools (for surveillance, team, WHO Global Malaria Programme (GMP), and
control, quality assurance, etc.), and the biology of Abt Associates, the implementers of the President’s
resistance. There was also an “other” topic to capture Malaria Initiative (PMI) Africa Indoor Residual Spray
challenges that did not fit easily into a single category. (AIRS) project. Opinions of the workshop participants
Workshop participants were placed in groups of 4–5 on various aspects of the rollout strategy were gathered
individuals, and each group spent 10 min brainstorm- through interactive polling (Turning Technologies).
ing challenges faced by vector control programmes
related to a single topic. They rotated until all groups Advisory committee meetings
visited all topics. Challenges were summarized by the Quarterly advisory meetings were held with an external
workshop leaders and re-phrased into potential learn- panel. Panel members had expertise in insecticide resist-
ing objectives. The mathematical modeller partici- ance, pedagogy, and public health. The advisory commit-
pants provided their expert opinion on whether/how tee provided direction across several different aspects of
each learning objective could be supported by the use the development project, including the technical accu-
of existing mathematical models. These discussions racy of ResistanceSim, the teaching strategies embedded
allowed the workshop organizers to produce a living in the tool, and the methods used to evaluate it. They also
document that defined the game’s learning objectives provided recommendations on synergies with existing
and the role of mathematical models in supporting research or vector control implementation projects.
these objectives.
The second workshop was held over 2 days in Janu- Playability testing
ary 2016. The objective of this workshop was to define Routine testing was conducted throughout the develop-
the preferred rollout strategy for the game, including ment of ResistanceSim by the ETCH team. Playability
how to make the game available and how it should be testing with external users was performed four times
used. Participants in this workshop included repre- coinciding with major development milestones. The pri-
sentatives from malaria control programmes in sub- mary objective of these testing sessions was to identify
Saharan Africa, potential funding partners, the ETCH bugs and usability issues. However, if the testers were
Thomsen et al. Malar J (2018) 17:422 Page 10 of 15
members of the target audience, a secondary objective (Protocol 16-016) and the Medical Research Council of
was to assess acceptability as a learning tool. Zimbabwe (Protocol MRCZ/E/140).
In May 2016, 26 users were recruited from LSTM Results from the Likert-scale survey were summarized
and stakeholder organizations to test the first beta ver- with standard statistical measures of mean and standard
sion of ResistanceSim. This version had all the required error. Any comparisons between pre- and post-question-
functionality but had not been tested to ensure it was naires were made using paired t-tests. The audio from
free of defects. Users were given a copy of the software the workshop was transcribed and analysed inductively.
with instructions on how to install it on their personal Illustrative quotes for each theme were documented.
laptops. They were also given a structured spread- Results from both testing sessions described above
sheet that allowed them to capture usability issues as were fed back into another large development sprint
they were playing, and were asked to complete a short which lasted for approximately 9 months. Major changes
survey rating their experience playing the game. They were made during this time to improve usability of the
answered questions about their engagement, the ease in tool. The third playability testing session was held at
which they learned how to play the game, and how easy LSTM in April of 2017, and included six users purpose-
it was to understand the various components. Users fully selected with expertise in education or operational
were then allowed to play the game in their own time vector control. These users were given a brief introduc-
over the course of 8 days, and their responses were col- tion to the tool, and were allowed to play through the
lected afterwards via email. All bugs identified during game for 3 h, documenting any bugs or usability issues
this beta testing were fixed prior to further user testing. in a similar format to the first testing session. Pedagogi-
The second major testing session occurred in Zim- cal feedback on the delivery of the tool was particularly
babwe in July 2016. During this time, the AIRS project useful at this time, and was used to shape the develop-
was conducting a regional entomological training ses- ment of a more comprehensive facilitated session. This
sion. It included 30 participants representing malaria facilitated session, which included gameplay, directed
vector control programmes from 11 countries in sub- activities using the game, group work, and mini lectures
Saharan Africa. For this session, the game was tested on was finally tested with a group of 20 individuals at LSTM
the final day of the week-long training course. Partici- in July 2017. The users included individuals well-versed
pants were asked to complete a short pre-game survey with vector control and insecticide resistance, as well as
to capture demographic information and awareness of those less familiar in order to gauge the response of a
IRM training resources. The survey also included Lik- diverse audience. Bugs and usability issues were docu-
ert-scale questions to evaluate participants’ perceptions mented in a similar manner.
of demand for IRM training tools, of their own IRM
knowledge, and of games and people who play games. Results
It also included an open-answer question asking them Refining learning objectives and rollout
to describe the steps involved in IRM. Then, partici- The original list of learning objectives generated from the
pants were given a brief introduction to the game and first workshop included 21 items across the topics of vec-
could play on their personal laptops for approximately tors, resistance, disease epidemiology, chemical-based
3 h while a facilitator circulated around the room to interventions, intervention monitoring and impact evalu-
answer any questions. After the play session, partici- ation, finances, stakeholders, and unforeseen challenges.
pants were placed in groups and provided with discus- Over the course of designing, developing and testing
sion questions in one of three topics: positive aspects the game, these learning objectives were further refined
of the game, barriers to a positive user experience, or (Table 1). It was also recognized that certain learning
barriers to sustainable implementation. After 20 min, objectives may take longer to achieve through gameplay
the groups rotated in a Charrette procedure (described than others, such as evaluating the cost-effectiveness of
above) so that all groups contributed to all topics. At various intervention strategies.
the end of this workshop, the facilitator led a discussion Workshop participants identified that malaria trans-
about each topic and asked groups to explain or expand mission models, including OpenMalaria and EMOD,
on certain aspects. Audio from the discussion was were more detailed than necessary to support the
recorded. Participants were asked to complete a post- learning objectives and at that time had little considera-
game survey that included many of the same questions tion of insecticide resistance. Even if they were thought
as the pre-game survey, but in addition asked them for suitable, it would be impossible to get these models
their perceptions on individual game elements, as well to run in the background of the game due to a lack of
as the value of the game as a whole. This research was computing power. Population genetic models to pre-
approved by institutional review boards at the LSTM dict the evolution of insecticide resistance [15] also
Thomsen et al. Malar J (2018) 17:422 Page 11 of 15
Table 1 Complete list of learning objectives addressed in ResistanceSim
Topic Learning objective
Stakeholders Identify which stakeholders to involve in insecticide resistance management planning
Vectors Compare the data obtained from various mosquito collection methods
Compare the data obtained from different species identification methods
Identify which collection method is required to determine transmission intensity
Explain why it is important to use consistent collection sites
Explain how vector bionomics influence intervention choices
Resistance Describe the process of generating insecticide susceptibility data
Identify the collection and test methods available to determine insecticide susceptibility, resistance intensity, and
resistance mechanisms
Describe the data required to construct a resistance profile
Explain the importance of species identification in constructing a resistance profile and interpreting resistance data
Illustrate the effect of continuously using insecticides with one mode of action
Evaluate the different insecticide resistance management strategies available
Apply this evaluation to make an appropriate resistance management plan
Evidence-based decisions Explain why it is important to look at data before making an intervention decision
Evaluate what insecticide class to use based on the resistance data
Assess when to deploy an intervention based on vector density and transmission data
Intervention monitoring Explain why it is important to use consistent methodology for routine monitoring
Identify the information that different intervention monitoring tools provide
Explain how quality assuring interventions contributes to insecticide resistance management
Compare the information gathered from different monitoring tools
Explain why it is important to monitor transmission
Explain why it is important to monitor insecticide susceptibility, resistance intensity, and resistance mechanisms
Demonstrate how to improve the quality and coverage of an intervention
Finances Evaluate the cost-effectiveness of various intervention strategies
These learning objectives were first identified during stakeholder workshops, and further revised during the game development process
contain more detail than is necessary to support the impact. To encourage the uptake of the tool, it was sug-
learning objectives. Since ResistanceSim is designed to gested that a comprehensive curriculum and course
be a learning tool, and not a decision-support tool, it structure were created and distributed with the game
was decided that an extensive validated model was not itself. This would serve as a facilitator’s guide, and make
needed. All that was needed was something that would it easier for country vector control programmes to adopt
generate outputs to the players within game scenarios the tool.
that would support individual learning objectives. This,
therefore, led to the development of the ResistanceSim Beta testing 1
game model (described above). In order to develop and Results from the first beta testing session held at the
test the model outputs, a web application was devel- LSTM and remotely with other stakeholders produced
oped using the Shiny package (RStudio Inc.) to allow a list of 32 bugs. Usability issues were numerous, and
the development team to manipulate model parameters included confusion about the tutorial section, how data
and test various scenarios quickly and easily. Simulta- is presented in the game, and whether players’ decisions
neously, the game developers transferred the code to were good or bad and why. Players’ opinions of the game
C#, the language used by game development platform at this time were neutral, neither agreeing nor disagree-
Unity, so that the mosquito populations in the game ing with many of the survey questions (Fig. 9). After dis-
reacted as expected. cussing these issues with the beta testers and amongst
Participants in the rollout workshop felt that the game the ETCH team, a list of 79 change requests were pro-
should be incorporated into existing IRM training activi- duced to help address some of the issues with data visual-
ties, rather than being played individually or in a sepa- ization, the tutorial section, and player feedback. Prior to
rate session. In addition, it was decided that playing the the next testing session in Zimbabwe, all bugs were fixed,
game as part of a facilitated session would have the most
Thomsen et al. Malar J (2018) 17:422 Page 12 of 15
5
4.5
Strongly disagree - strongly agree
4
3.5
3
2.5
2
1.5
1
The game It was easy I want to The The The scoring The I found The game
was fun to to learn play the entomolgy interven on in the game graphics in playing the was too
play. how to play game again. data in the data in the was easy to the game game difficult.
the game. game was game was understand. were frustra ng.
easy to easy to appropriate.
understand. understand.
Fig. 9 User perceptions (n = 8) of the first beta version of ResistanceSim. Error bars represent the standard error of the mean
and change requests were prioritized to focus on the clar- head trying to work out how to navigate around and fig-
ity of the tutorial section and data visualization. ure out how to play the game, I feel like you need to be
able to jump in a lot more quickly.” In addition, users
Beta testing 2 expressed frustration in the way the instructions are
User experiences in Zimbabwe were more positive than presented: “We are not engaging with the game. The
in the first testing session. Users generally felt that the actual reason (for this) is that the instructions are not
game improved their understanding of various topics clear.” They were also disappointed by the lack of direc-
related to vector control (Fig. 10a), and that the data pre- tion: “I can see the provincial health bar going up and
sented in the game was easy to understand. The tutorial down, but there is no specific goal,” “As a player, you
section was still difficult for users to work through, and should be able to monitor independently how you are
this was reflected in both the survey answers (Fig. 10b) doing as far as your learning,” “It should have different
as well as the progress that most people made during the levels.”
test session—only 4 out of 30 participants were able to Despite these difficulties, 90% of participants indi-
make it past the tutorial during the 3-h play session. cated that they need more support related to IRM, and
Feedback during the workshop discussion shed more they agreed that the tool would be a valuable addition
light on the positive and negative aspects of the game. to the training currently available for IRM and vector
Players enjoyed the fact that their own actions in the control (Fig. 10b).
game influenced the outcomes: “We also like the inter- All of the feedback from the LSTM and Zimba-
pretation of data where you could see the impact of IRS bwe testing sessions were consolidated and solutions
on vector density.” They also expressed satisfaction in were proposed to address most of the usability issues.
the complexity of the topics covered in the game: “We The solutions fell in two categories: tutorial and inter-
liked how the game instructed you to start your activi- face improvements. The tutorial section was reworked
ties at national level, then move to the province, to the replacing it with a guided, mission-oriented “Roadmap.”
district, down to the village … this makes you aware of This guides players through the various aspects of the
the need to involve all levels in terms of implementa- game itself, while slowly introducing the complexity of
tion and planning.” However, it also became clear that the content. The Roadmap provides immediate feed-
while complexity in the topics covered was desirable, back on player decisions, so that they know what they
complexity in the user interface was preventing users are doing well and why. The second category of game
from interacting meaningfully with content: “Rather changes involved simplifying the user interface of the
than spend maybe an hour or 2 h just cracking your open simulation while retaining the complexity of the
Thomsen et al. Malar J (2018) 17:422 Page 13 of 15
a 7
6
Strongly disagree - strongly agree
5
4
3
2
1
how to how to decide how to how how to how to how to
monitor vector which vector manage interven ons monitor interpret construct an
popula ons control insec cide influence interven on various types insec cide
interven ons resistance insec cide efficacy of resistance resistance
to choose resistance data management
plan
b 7
6
Strongly disagree - strongly agree
5
4
3
2
1
The tutorial session The tutorial session The tutorial session It was easy to It was easy to figure
at the beginning of at the beginning of at the beginning of understand what out how complete
the game was... the game was... the game was... was happening in ac ons in the game
[easy to understand] [informa ve] [relevant to the rest the game
of the game]
Fig. 10 User perceptions (n = 28) during the second beta testing session in Zimbabwe of a the degree to which ResistanceSim improved their
understanding of various topics and b the ease of use of the tutorial section. Error bars represent the standard error of the mean
content it covered. Changes in this category included Usability testing
reworking how data is displayed, removing extrane- Results from usability testing in April 2017 indicated a
ous aspects of the user interface, and providing regu- vast improvement in the game. All participants (n = 6)
lar updates to the player about how their decisions are felt that ResistanceSim would be a valuable addition to an
impacting game outcomes. All of these changes were IRM course. In contrast to the first beta testing session
completed over a software development sprint lasting conducted in May 2016 (Fig. 9), all participants indicated
approximately 9 months. that they wanted to play the game again. However, most
Thomsen et al. Malar J (2018) 17:422 Page 14 of 15
also felt that in order to get the most out of the game, and external advisory boards, and frequent user test-
players needed to spend more time with it: “…by the time ing focused on playability and perceived usefulness. The
you get through the missions (Roadmap), I felt then pre- results from this work are promising, in that the final
pared to go into the game. But it’s almost like you need user-led product has been deemed a valuable potential
some thinking time…it requires time to get the most out of addition to IRM training activities. As serious games
it.” In addition, one participant who did not have a back- have been shown to have positive impacts on knowledge
ground in vector control found it difficult to understand and motivation [16], an important next step will be to
what they were doing because of unfamiliarity with some evaluate ResistanceSim for its effect on knowledge acqui-
of the terminology used. It was suggested that additional sition, self-efficacy, and decision-making behaviours in
learning material be provided that allowed all users to vector control programmes that have used the game as a
start with the same level of knowledge. A total of 22 bugs training tool.
and 14 usability issues were identified and documented Serious games have been used extensively in the health
in both the Roadmap and the open simulation. Most of field, particularly aimed at training health profession-
the usability issues related to the transition between als [12, 17] or changing behaviour of patients to improve
the Roadmap and the open simulation, where users are their health outcomes [10, 11, 18]. However, to our
introduced to some new functionality that is not explic- knowledge, there are no serious games that target public
itly described. Feedback from this session resulted in two health policy implementers, whose decisions have a mas-
major developments. First, a series of short tutorial vid- sive impact on the health of many individuals. In addi-
eos were created to ease the transition from the Roadmap tion, are only few examples of games being used in low
to the open simulation. Second, a structured lesson plan and middle income countries or focused on diseases of
and additional teaching resources were created (exer- poverty [19–23]. With computer use ubiquitous across
cises, discussion topics, and slide sets) so that Resistanc- multiple sectors in sub-Saharan Africa, and continuing
eSim was integrated into a facilitated course on IRM. to increase [24], this presents a significant opportunity to
utilize technology as a capacity strengthening tool.
Training course Previous literature reviews highlighted the necessity
The facilitated ResistanceSim training course was tested of iterative evaluation of instructional elements, game-
with 20 individuals at LSTM in July 2017. The course play mechanics, and user interface [25] when design-
lasted from 0900 to 1600 h with time for breaks. Feed- ing serious games. The results from this study reiterate
back was gathered through a simple open questionnaire this recommendation. Despite the early involvement of
that asked participants what they liked about the course subject experts, game designers, and regular reviews
and what could be improved. In general, participants from an external advisory committee, the first beta test-
were enthusiastic about the tool, and expressed satisfac- ing revealed that users simply did not enjoy playing the
tion with the complimentary course material: “I liked the game. It was only after additional revisions and testing
linkage/balance between course instruction and activities,” that a product was produced that struck the right balance
“[the additional components] added considerable value to between engagement and instruction that motivated
the ResistanceSim game itself.” The value of the Roadmap users to keep playing.
was recognized as a way to slowly introduce complicated The iterative nature of the development process also
concepts: “I liked the look and atmosphere of the applica- allowed the elucidation of potential implementation
tion and the way that the structure built up as you got fur- strategies, since users indicated that the game should be
ther into the modules and I started to make linkages and used as part of a structured course. This allowed us to test
adopt reinforced behaviours etc.” In addition, some sug- the game in this context during the final stage of develop-
gestions were made to improve the exercises and group ment. The instructional resources are available for open
work that were completed as part of the course so that all use (at etch.lstmed.ac.uk), so that potential Resistanc-
participants can equally benefit from the ResistanceSim eSim course facilitators have guidance on the curriculum
tool itself. and structure of the course.
Discussion Conclusions
A ‘serious game’ was developed aimed at improving In order to ensure the sustainability of public health
understanding of insecticide resistance management insecticides, they must be used judiciously and intel-
strategies among vector control programme personnel, ligently. Strengthening the capacity of malaria vector
with the ultimate goal of influencing decision-making control programmes to manage insecticide resistance is
processes. Over the course of 2 years, the game was eval- a critical component of this, but training resources are
uated for its validity through consultation with experts limited. ResistanceSim, developed here, is a management
Thomsen et al. Malar J (2018) 17:422 Page 15 of 15
simulation game that immerses the player in a fictitious References
1. Global Partnership to Roll Back Malaria. The Abuja declaration on roll back
vector control programme, to fill this gap. Early and malaria in Africa. Geneva; 2000.
repeat testing with target users and involvement of stake- 2. Hemingway J, Ranson H. Insecticide resistance in insect vectors of human
holders was vital in the development of the tool. This disease. Annu Rev Entomol. 2000;45:371–91.
3. Hemingway J. The role of vector control in stopping the transmis-
process has enabled us to improve user experience and sion of malaria: threats and opportunities. Philos Trans R Soc B.
provide a viable environment for learning. The potential 2014;369:20130431.
for this serious game to be useful in training has been 4. Ranson H, N’guessan R, Lines J, Moiroux N, Nkuni Z, Corbel V. Pyrethroid
resistance in African anopheline mosquitoes: what are the implications
demonstrated, and its utility in operational settings is for malaria control? Trends Parasitol. 2011;27:91–8.
currently being tested. 5. REX consortium. Heterogeneity of selection and the evolution of resist-
ance. Trends Ecol Evol. 2013;28:110–8.
Authors’ contributions 6. WHO. Global plan for insecticide resistance management in malaria vec-
EKT designed the tool, designed the study, collected the data, analysed the tors. Geneva: World Health Organization; 2012.
data, wrote the manuscript. CH designed the tool, designed the study, col- 7. Reigeluth CM. Instructional-design theories and models: a new paradigm
lected the data, analysed the data, reviewed the manuscript. AS designed the of instructional theory. Routledge; 2013.
tool, designed the game model, reviewed the manuscript. KAD, CD collected 8. Bandura A. Health promotion by social cognitive means. Health Educ
the data, reviewed the manuscript. MiC supported the development of the Behav. 2004;31:143–64.
tool, reviewed the manuscript. RF designed the tool, reviewed the manuscript. 9. Djaouti D, Alvarez J, Jessel J, Rampnoux O. Origins of serious games. Seri-
MaC conceived of the tool, designed the tool, reviewed the manuscript. All ous games edutainment applications. London: Springer; 2011. p. 25–43.
authors read and approved the final manuscript. 10. Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves
behavioral outcomes in adolescents and young adults with cancer: a
Author details randomized trial. Pediatrics. 2008;122:e305–17.
1
Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK. 11. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Mathijs F. The
2
Extra Mile Studios, 142 West Nile St, Glasgow G1 2RQ, UK. 3 Present Address: effectiveness of SPARX, a computerised self help intervention for adoles-
IVCC, Pembroke Pl, Liverpool L3 5QA, UK. cents seeking help for depression : randomised controlled non-inferiority
trial. BMJ. 2012;344:2598.
Acknowledgements 12. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious
We would like to acknowledge the IVCC for support and guidance during the games for medical education and surgical skills training. Br J Surg.
game development process, as well as Abt Associates for agreeing to host one 2012;99:1322–30.
of the testing sessions. 13. Branson R, Rayner GT, Cox JL, Furman JP, King J. Interservice procedures
for instructional systems development. Executive Summary and Model.
Competing interests 1975.
BF is the CEO of Extra Mile Studios, the developer of ResistanceSim. However, 14. Cramer HL, Wehking RJ. Charretting the planning process. Washington,
as the tool is free to use and distribute, the company does not stand to gain D.C.; 1973.
anything from the publication of this manuscript. 15. Levick B, South A, Hastings IM. A two-locus model of the evolution of
MaC currently works for IVCC, the primary funder of this tool. However, insecticide resistance to inform and optimise public health insecticide
at the time of development and testing of ResistanceSim, MaC was affiliated deployment strategies. PLoS Comput Biol. 2017;13:e1005327.
with the Liverpool School of Tropical Medicine. 16. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic
literature review of empirical evidence on computer games and serious
games. Comput Educ. 2012;59:661–86.
Availability of data and materials 17. Wang R, DeMaria S, Goldberg A, Katz D. A systematic review of serious
The datasets generated and/or analysed during the current study are available games in training health care professionals. Simul Healthc. 2016;11:41–51.
from the corresponding author on reasonable request. 18. DeSmet A, Van Ryckeghem D, Compernolle S, Baranowski T, Thompson
D, Crombez G, et al. A meta-analysis of serious digital games for healthy
Consent for publication lifestyle promotion. Prev Med (Baltim). 2014;69:95–107.
Not applicable. 19. Luz S, Masoodian M, Cesario RR, Cesario M. Using a serious game to pro-
mote community-based awareness and prevention of neglected tropical
Ethics approval and consent to participate diseases. Entertain Comput. 2015;15:43–55. https://doi.org/10.1016/j.
This research was approved by institutional review boards at the LSTM entcom.2015.11.001.
(Protocol 16-016) and the Medical Research Council of Zimbabwe (Protocol 20. Buchinger D, da Hounsell M, Um S. Jogo Sério Colaborativo Para
MRCZ/E/140). Aprender Sobre a Doença da Dengue. Informática Educ Teor Prática.
2015;18:67–84.
Funding 21. Lima T, Barbosa B, Niquini C, Araujo C, Lana R. Playing against dengue
Development of ResistanceSim was provided by the Bill and Melinda Gates design and development of a serious game to help tackling dengue. In:
Foundations through the IVCC (OPP1148615) and by the Wellcome Trust 2017 IEEE 5th Int. Conf. Serious Games Appl. Heal. SeGAH. 2017.
Institutional Strategic Support Fund. 22. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Dis-
tributed medical image analysis and diagnosis through crowd-sourced
games: a malaria case study. PLoS ONE. 2012;7:e37245.
Publisher’s Note 23. Kam M, Agarwal A, Kumar A, Lal S. Designing e-learning games for rural
Springer Nature remains neutral with regard to jurisdictional claims in pub-
children in India: a format for balancing learning with fun. In: Proc. 7th
lished maps and institutional affiliations.
ACM Conf. Des. Interact. Syst. 2008. p. 58–67.
24. Cirera X, Lage F, Sabetti L. ICT use, innovation, and productivity: evidence
Received: 23 February 2018 Accepted: 9 November 2018
from Sub-Saharan Africa. Policy Res. Work. Pap. 2016.
25. Papastergiou M. Exploring the potential of computer and video games
for health and physical education: a literature review. Comput Educ.
2009;53:603–22.