Plaintext
Scrum.org
The Evidence-Based Management
Guide
Measuring Value to Enable Improvement and Agility
September 2020
Purpose of the EBM Guide
Evidence-Based Management (EBM) is an empirical approach that helps organizations to
continuously improve customer outcomes, organizational capabilities, and business results
under conditions of uncertainty. It provides a framework for organizations to improve their
ability to deliver value in an uncertain world, seeking a path toward strategic goals. Using
intentional experimentation and evidence (measures), EBM enables organizations to
systematically improve their performance over time and refine their goals based on better
information.
By measuring current conditions, setting performance goals, forming small experiments for
improvement that can be run quickly, measuring the effect of the experiment, and inspecting
and adapting goals and next steps, EBM helps organizations to take into account the best
available evidence to help them make decisions on ways to improve.
This Guide defines EBM, its concepts, and its application.
© 2020 Scrum.org
This publication is offered for license under the Attribution Share-Alike license of Creative Commons,
accessible at http://creativecommons.org/licenses/by-sa/4.0/legalcode and also described in
summary form at http://creativecommons.org/licenses/by-sa/4.0/. By utilizing this EBM Guide, you
acknowledge and agree that you have read and agree to be bound by the terms of the Attribution
Share-Alike license of Creative Commons.
2
EBM Helps Organizations Seek toward Their Goals
in a Complex World Using Empiricism
Complex problems defy easy solutions, but instead require organizations seek toward their
goals in a series of small steps, inspecting the results of each step, and adapting their next
actions based on feedback (see Figure 1).1
This model has several key elements:
A Strategic Goal, which is something important that the organization would like to achieve.
This goal is so big and far away, with many uncertainties along the journey that the
organization must use empiricism. Because the Strategic Goal is aspirational and the
path to it is uncertain, the organization needs a series of practical targets, like
Intermediate Goals, achievements of which will indicate that the organization is on the path
to its Strategic Goal. The path to the Intermediate Goal is often still somewhat uncertain,
but not completely unknown.
Immediate Tactical Goals, critical near-term objectives toward which a team or group of
teams will work help toward Intermediate Goals.
A Starting State, which is where the organization is relative to the Strategic Goal when it
starts its journey.
A Current State, which is where the organization is relative to the Strategic Goal at the
present time.
In order to progress toward the Strategic Goal, organizations run experiments which involve
forming hypotheses that are intended to advance the organization toward their current
Intermediate Goal. As they run these experiments and gather results, they use the evidence
they obtain to evaluate their goals and determine their next steps to advance toward these
goals.
1
For more on complexity, see the Scrum Theory section of the Scrum Guide at
https://www.scrumguides.org/scrum-guide.html
3
Figure 1: Reaching strategic goals requires experimenting, inspecting, and adapting2
Setting Goals
When setting goals, organizations must define specific measures that will indicate that the goal
is achieved. Goals, measures, and experiments should be made transparent in order to
encourage organizational alignment.
Consider the case of the response to an infectious disease:
● The Strategic Goal is to eradicate the effects of the disease, as measured by the
number of people who fall ill and suffer significant illness. Measurement is
important; in this example, the goal is focused on the effects of the disease, and
not on the means for achieving the desired impact. For example, the goal is not
to vaccinate a certain percentage of the population against the disease; that may
be an activity necessary to achieving the Strategic Goal, but it is not the Strategic
Goal.
2
Figure adapted from Mike Rother’s Improvement Kata (http://www-
personal.umich.edu/~mrother/The_Improvement_Kata.html)
4
● An example of an Intermediate Goal is the successful completion of a trial of a
vaccine against the disease. This is still ambitious and measurable, and
achieving it may require the completion of many different activities, but it is seen
as a necessary step on the path to achieving the Strategic Goal.
● Examples of immediate tactical goals may include activities like isolating
symptoms, evaluating a therapy, sequencing the DNA of a virus or bacterium,
and so forth.
The Strategic Goal is usually focused on achieving a highly desirable but unrealized outcome
for a specific group of people that results in improved happiness, safety, security, or well-being
of the recipients of some product or service. In EBM, we refer to this as Unrealized Value, which
is the satisfaction gap between a beneficiary’s desired outcome and their current experience.
Unrealized Value is described in greater detail below, in the Key Value Areas section.
Understanding What Is Valuable
Organizations measure many different kinds of things. Broadly speaking, measures fall into
three categories:
● Activities. These are things that people in the organization do, such as perform work,
go to meetings, have discussions, write code, create reports, attend conferences, and so
forth.
● Outputs. These are things that the organization produces, such as product releases
(including features), reports, defect reports, product reviews, and so on.
● Outcomes. These are desirable things that a customer or user of a product
experiences. They represent some new or improved capability that the customer or user
was not able to achieve before. Examples include being able to travel to a destination
faster than before, or being able to earn or save more money than before. Outcomes
can also be negative, as in the case where the value a customer or user experiences
declines from previous experiences, for example when a service they previously relied
upon is no longer available.
The problem most organizations face, which is often reflected in the things they measure, is that
measuring activities and outputs is easy, while measuring outcomes is difficult. Organizations
may gather a lot of data with insufficient information about their ability to deliver value. However,
delivering valuable outcomes to customers is essential if organizations are to reach their goals.
For example, working more hours (activities) and delivering more features (outputs) does not
necessarily lead to improved customer experiences (outcomes).
EBM Focuses on Four Key Value Areas
In addition to using hypotheses and experiments to move toward goals, EBM provides a set of
perspectives on value and the organization’s ability to deliver value. These perspectives are
called Key Value Areas (KVAs). These areas examine the goals of the organization (Unrealized
Value), the current state of the organization relative to those goals (Current Value), the
5
responsiveness of the organization in delivering value (Time-to-Market), and the effectiveness
of the organization in delivering value (Ability-to-Innovate). Focusing on these four dimensions
enables organizations to better understand where they are and where they need to go (see
Figure 2).
Figure 2: EBM focuses on four Key Value Areas (KVAs).
Each KVA focuses on a different aspect of either value, or the ability of the organization to
deliver value. Delivering business value (Current Value) is important, but organizations must
also show that they can respond to change (Time-to-Market) while being able to sustain
innovation over time (Ability-to-Innovate). And they must be able to continually make progress
toward their long-term goals (Unrealized Value) or they risk succumbing to stagnation and
complacency.
Example Key Value Measures (KVMs) for each KVA are described in the Appendix.
Current Value (CV)
The value that the product delivers today
The purpose of looking at CV is to understand the value that an organization delivers to
customers and stakeholders at the present time; it considers only what exists right now, not the
value that might exist in the future. Questions that organizations need to continually re-evaluate
for current value are:
1. How happy are users and customers today? Is their happiness improving or declining?
6
2. How happy are your employees today? Is their happiness improving or declining?
3. How happy are your investors and other stakeholders today? Is their happiness
improving or declining?
Considering CV helps an organization understand the value that their customers or users
experience today.
Example: While profit, one way to measure investor happiness, will tell you the
economic impact of the value that you deliver, knowing whether customers are happy
with their purchase will tell you more about where you may need to improve to keep
those customers. If your customers have few alternatives to your product, you may have
high profit even though customer satisfaction is low. Considering CV from several
perspectives will give you a better understanding of your challenges and opportunities.
Customer happiness and investor happiness also do not tell the whole story about your
ability to deliver value. Considering employee attitudes recognizes that employees are
ultimately the producers of value. Engaged employees that know how to maintain,
sustain and enhance the product are one of the most significant assets of an
organization, and happy employees are more engaged and productive.
Unrealized Value (UV)
The potential future value that could be realized if the organization met the
needs of all potential customers or users
Looking at Unrealized Value helps an organization to maximize the value that it realizes from a
product or service over time. When customers, users, or clients experience a gap between their
current experience and the experience that they would like to have, the difference between the
two represents an opportunity; this opportunity is measured by Unrealized Value.
Questions that organizations need to continually re-evaluate for UV are:
1. Can any additional value be created by our organization in this market or other markets?
2. Is it worth the effort and risk to pursue these untapped opportunities?
3. Should further investments be made to capture additional Unrealized Value?
The consideration of both CV and UV provides organizations with a way to balance present and
possible future benefits. Strategic Goals are formed from some satisfaction gap and an
opportunity for an organization to decrease UV by increasing CV.
Example: A product may have low CV, because it is an early version being used to test
the market, but very high UV, indicating that there is great market potential. Investing in
the product to try to boost CV is probably warranted, given the potential returns, even
though the product is not currently producing high CV.
7
Conversely, a product with very high CV, large market share, no near competitors, and
very satisfied customers may not warrant much new investment; this is the classic cash
cow product that is very profitable but nearing the end of its product investment cycle
with low UV.
Time-to-Market (T2M)
The organization’s ability to quickly deliver new capabilities, services, or
products
The reason for looking at T2M is to minimize the amount of time it takes for the organization to
deliver value. Without actively managing T2M, the ability to sustainably deliver value in the
future is unknown. Questions that organizations need to continually re-evaluate for T2M are:
1. How fast can the organization learn from new experiments and information?
2. How fast can you adapt based on the information?
3. How fast can you test new ideas with customers?
Improving T2M helps improve the frequency at which an organization can potentially change
CV.
Example: Reducing the number of features in a product release can dramatically
improve T2M; the smallest release possible is one that delivers at least some
incremental improvement in value to some subset of the customers/users of the product.
Many organizations also focus on removing non value-added activities from the product
development and delivery process to improve their T2M.
Ability to Innovate (A2I)
The effectiveness of an organization to deliver new capabilities that might
better meet customer needs
The goal of looking at the A2I is to maximize the organization’s ability to deliver new capabilities
and innovative solutions. Organizations should continually re-evaluate their A2I by asking:
1. What prevents the organization from delivering new value?
2. What prevents customers or users from benefiting from that innovation?
Improving A2I helps an organization become more effective in ensuring that the work that it
does improves the value that its products or services deliver to customers or users.
Example: A variety of things can impede an organization from being able to deliver new
capabilities and value: spending too much time remedying poor product quality, needing
to maintain multiple variations of a product due to lack of operational excellence, lack of
8
decentralized decision-making, inability to hire and inspire talented, passionate team-
members, and so on.
As low-value features and systemic impediments accumulate, more budget and time is
consumed maintaining the product or overcoming impediments, reducing its available
capacity to innovate. In addition, anything that prevents users or customers from
benefiting from innovation, such as hard to assemble/install products or new versions of
products, will also reduce A2I.
Progress toward Goals in A Series of Small Steps
The first step in the journey toward a Strategic Goal is understanding your Current State. If your
focus is to achieve a Strategic Goal related to Unrealized Value (UV), as is typically the case,
then measuring the Current Value (CV) your product or service delivers is where you should
start (of course, if your product or service is new then its CV will be zero). To understand where
you need to improve, you may also need to understand your effectiveness (A2I), and your
responsiveness (T2M).
The Experiment Loop (shown in Figure 1) helps organizations move from their Current State
toward their Next Target Goal, and ultimately their Strategic Goal, by taking small, measured
steps, called experiments, using explicit hypotheses.3 This loop consists of:
● Forming a hypothesis for improvement. Based on experience, form an idea of
something you think will help you move toward your Next Target Goal, and decide how
you will know whether this experiment succeeded based on measurement.
● Running your experiments. Make the change you think will help you to improve and
gather data to support or refute your hypothesis.
● Inspecting your results. Did the change you made improve your results based on the
measurements you have made? Not all changes do; some changes actually make things
worse.
● Adapting your goals or your approach based on what you learned. Both your goals
and your improvement experiments will likely evolve as you learn more about customers,
competitors, and your organization's capabilities. Goals can change because of outside
events, and your tactics to reach your goals may need to be reconsidered and revised.
Was the Intermediate Goal the right goal? Is the Strategic Goal still relevant? If you
achieved the Intermediate Goal, you will need to choose a new Intermediate Goal. If
you did not achieve it, you will need to decide whether you need to persevere, stop, or
pivot toward something new. If your Strategic Goal is no longer relevant, you will need to
either adapt it, or replace it.
3
The Experiment Loop is a variation on the Shewhart Cycle, popularized by W. Edwards Deming, also
sometimes called the PDCA (Plan-Do-Check-Act) cycle; see https://en.wikipedia.org/wiki/PDCA.
9
Hypotheses, Experiments, Features, and Requirements
Features are “distinguishing characteristics of a product”4, while a requirement is, practically
speaking, something that someone thinks would be desirable in a product. A feature description
is one kind of requirement.
Organizations can spend a lot of money implementing features and other requirements in
products, only to find that customers don’t share the company’s opinion on their value; beliefs in
what is valuable are merely assumptions until they are validated by customers. This is where
hypotheses and experiments are useful.
In simplified terms, a hypothesis is a proposed explanation for some observation that has not
yet been proven (or disproven). In the context of requirements, it is a belief that doing something
will lead to something else, such as delivering feature X will lead to outcome Y. An experiment
is a test that is designed to prove or reject some hypothesis.
Every feature and every requirement really represent a hypothesis about value. One of the
goals of an empirical approach is to make these hypotheses explicit and to consciously design
experiments that explicitly test the value of the features and requirements. The entire feature or
requirement need not actually be built to determine whether it is valuable; it may be sufficient for
a team to simply build enough of it to validate critical assumptions that would prove or disprove
its value.
Explicitly forming hypotheses, measuring results, and inspecting and adapting goals based on
those results are implicit parts of an agile approach. Making this work explicit and transparent is
what EBM adds to the organizational improvement process.
End Note
Evidence-Based Management is free and offered in this Guide. Although implementing only
parts of EBM is possible, the result is not Evidence-Based Management.
Acknowledgements
Evidence-Based Management was collaboratively developed by Scrum.org, the Professional
Scrum Trainer Community, Ken Schwaber and Christina Schwaber.
4
Adapted from the IEEE 829 specification
10
Appendix: Example Key Value Measures
To encourage adaptability, EBM defines no specific Key Value Measures (KVMs). KVMs listed
below are presented to show the kinds of measures that might help an organization to
understand its current state, desired future state, and factors that influence its ability to improve.
Current Value (CV)
KVM Measuring:
Revenue per Employee The ratio (gross revenue / # of employees) is a key competitive
indicator within an industry. This varies significantly by industry.
Product Cost Ratio Total expenses and costs for the product(s)/system(s) being
measured, including operational costs compared to revenue.
Employee Satisfaction Some form of sentiment analysis to help gauge employee
engagement, energy, and enthusiasm.
Customer Satisfaction Some form of sentiment analysis to help gauge customer
engagement and happiness with the product.
Customer Usage Index Measurement of usage, by feature, to help infer the degree to
which customers find the product useful and whether actual usage
meets expectations on how long users should be taking with a
feature.
Unrealized Value (UV)
KVM Measuring:
Market Share The relative percentage of the market not controlled by the
product; the potential market share that the product might achieve
if it better met customer needs.
Customer or User The difference between a customer or user’s desired experience
Satisfaction Gap and their current experience.
Desired Customer A measure that indicates the experience that the customer would
Experience or like to have
satisfaction
11
Time-to-Market (T2M)
KVM Measuring:
Build and Integration The number of integrated and tested builds per time period. For a
Frequency team that is releasing frequently or continuously, this measure is
superseded by actual release measures.
Release Frequency The number of releases per time period, e.g. continuously, daily,
weekly, monthly, quarterly, etc. This helps reflect the time needed
to satisfy the customer with new and competitive products.
Release Stabilization The time spent correcting product problems between the point the
Period developers say it is ready to release and the point where it is
actually released to customers. This helps represent the impact of
poor development practices and underlying design and code
base.
Mean Time to Repair The average amount of time it takes from when an error is
detected and when it is fixed. This helps reveal the efficiency of
an organization to fix an error.
Customer Cycle Time The amount of time from when work starts on a release until the
point where it is actually released. This measure helps reflect an
organization’s ability to reach its customer.
Lead Time The amount of time from when an idea is proposed, or a
hypothesis is formed until a customer can benefit from that idea.
This measure may vary based on customer and product. It is a
contributing factor for customer satisfaction.
Lead Time for Changes The amount of time to go from code-committed to code
successfully running in production. For more information, see the
DORA 2019 report.
Deployment Frequency The number of times that the organization deployed (released) a
new version of the product to customers/users. For more
information, see the DORA 2019 report.
Time to Restore Service The amount of time between the start of a service outage and the
restoration of full availability of the service. For more information,
see the DORA 2019 report.
Time-to-Learn The total time needed to sketch an idea or improvement, build it,
deliver it to users, and learn from their usage.
12
Time to remove The average amount of time from when an impediment is raised
Impediment until when it is resolved. It is a contributing factor to lead time and
employee satisfaction.
Time to Pivot A measure of true business agility that presents the elapsed time
between when an organization receives feedback or new
information and when it responds to that feedback; for example,
the time between when it finds out that a competitor has delivered
a new market-winning feature to when the organization responds
with matching or exceeding new capabilities that measurably
improve customer experience.
Ability to Innovate (A2I)
KVM Measuring:
Innovation Rate The percentage of effort or cost spent on new product capabilities, divided
by total product effort or cost. This provides insight into the capacity of the
organization to deliver new product capabilities.
Defect Trends Measurement of change in defects since last measurement. A defect is
anything that reduces the value of the product to a customer, user, or to the
organization itself. Defects are generally things that don’t work as intended.
On-Product Index The percentage of time teams spend working on product and value.
Installed Version The number of versions of a product that are currently being supported.
Index This reflects the effort the organization spends supporting and maintaining
older versions of software.
Technical Debt A concept in programming that reflects the extra development and testing
work that arises when “quick and dirty” solutions result in later remediation.
It creates an undesirable impact on the delivery of value and an avoidable
increase in waste and risk.
Production The number of times in a given period that the Development Team was
Incident Count interrupted to fix a problem in an installed product. The number and
frequency of Production Incidents can help indicate the stability of the
product.
13
Active Product The number of different versions (or variants) of a product or service.
(Code) Branches Provides insight into the potential impact of change and the resulting
complexity of work.
Time Spent The amount of time spent applying changes across different versions of a
Merging Code product or service. Provides insight into the potential impact of change and
Between the resulting complexity of work.
Branches
Time Spent Examples include time lost to interruptions caused by meetings or calls,
Context-Switching time spent switching between tasks, and time lost when team members are
interrupted to help people outside the team can give simple insight into the
magnitude of the problem.
Change Failure The percentage of released product changes that result in degraded
Rate service and require remediation (e.g. hotfix, rollback, patch). For more
information, see the DORA 2019 report.
14