Validated learning: How to rollout small change experiments iteratively

Validated learning: How to rollout small change experiments iteratively

Rollout small change experiments iteratively, using a cycle that supports a mentality of learning and experimentation.

Profile image of Joao Gama
Oct 22, 2019 • 2 min read

After multiple pitches and (convincing) conversations, we had finally achieved a milestone: we had gotten the go-ahead on experimenting. We had been allowed to conduct the change effort not as a change plan, but as a learn-and-go process. And after so long fighting for this possibility, we had a natural question:

Now what???

We were working on a medical services company, which had multiple clinics all over the country. Our task was to improve customer satisfaction when using our medical services in-person (not through the app), as more and more people had been migrating online for help. But our in-person service was the differentiating factor, and they wanted to improve the experience as we prepared to offer a "seamless" medical product with online and physical elements.

But now that we had our shot to manage our project in an agile way, the question was: where to get started and where to focus first? As we brainstormed possible experiments, our possible options quickly got to the dozens. So then, how could we prioritize which one to start with? What composed a good experiment, and how to make sure each experiment maximizes the effort? I will share with you what I have learned from this experience.

First, you should not start by brainstorming experiments. This is too broad a search, and there is no way to really decide which ones are best. The way to start is by defining the business goals. The more you understand the goals, the easier it will be to define a criteria for your experiments.

Second, you should select the most important business goal and break it into its "elements". As you start breaking it into elements, you will quickly notice that the elements are easier to prioritize - even though some of them seem to oppose each other.

Third, you only start brainstorming experiments when the business element has been defined and agreed to by leadership. Now you know you are on the right track, and almost any experiment will produce relevant lessons. On the brainstorming, try to be creating instead of focusing on what can "realistically" be implemented. Remember, brainstorming is about thinking outside of the box - and nothing proposed is bad.

Fourth, you can pick and merge the best ideas of the brainstorming session. Usually, the multiple crazy ideas that come out can be incorporated into a few really good experiments, but only a (very) few will stand out. Those are the ones to start with.

Fifth, select as many experiments as you team can manage. For us this number was usually one concurrent experiment at a time. But you could have a larger team and could do multiple experiments at the same time. Keep in mind that the results of experiments are necessary to prioritize the next experimentation cycle. So concurrent lines of experimentations should focus on non-conflictive elements of the strategy.

So there you have it. 5 steps to define an experiment that is small enough yet relevant for your effort. 

2 minute read
Revision #3
507 Words
Created on Dec 12, 2019 22:27,
last edited on Oct 22, 2019 12:55