Lesson: Product Growth with Gustaf Alströmer

Step #9 Iterate: Your rate of experiments indicates how fast you will grow

In terms of how long you would build experiments, you want to try as long as you have some confidence that there’s significance around the data. If there’s a 0.1% change, it probably doesn’t matter, but if there’s 5% change, you want to make sure that that change is significant. There’s a fairly easy way to figure that out just looking at the math and the numbers.

Once you have that, you should probably finish that experiment and go onto the next one. Your rate of experiments is another great source to how you grow it faster. If you have a few experiments and you have some small ones, it’s kind of natural thinking that if you have more experiments with some number of small ones, the more you will grow, and if all these experiments are focused around conversion optimization or compounding growth, then they all kind of pay off over time.

If you do a 2% experiment and if you’re like, “Oh, that wasn’t that much,” you can always plot that experiment and say, “Okay, what if I had 1,000 more users or 1,000 times more users, what would a 2% gain look like?” and you plot those curves next to each other and say, “If I have 2% more growth and that’s compounding over time, it becomes a lot of users.”

In the early days, small things really matter, especially if you do a lot of small things. Not everything you do will have a 10%, 20%, 30%, 40% improvement on your growth rate but if you do many small things that might compound and actually if you apply it over time, it looks really good.

I think you should be iterating as fast as you can. I think it's rare that people launch too often and not too seldom. I think it's almost always that case that you are launching too rare and not too often. Most people should just speed up their rate of experiments, probably.

Some companies have been really good at building platform infrastructure to just do a lot of experiments like many experiments running at the same time, or the rate of experimentation is really fast. In some ways that becomes difficult if you're doing mobile apps or something like that then that slows down the rate of experiments.

The amount of users that you have slows down the rate of experiments. If you have Voxer, you have millions of daily users, so it's fairly easy to do some experiments to prove some significance. If you only have a few thousand users, it might take you a lot longer to prove that point. I think you should do it as fast as you can. The more users you have, the faster you can iterate.