Lesson: Data-Driven Decisions with Mike Greenfield

Step #6 Sample Size: Getting statistically meaningful

You said we had large samples sizes with six million users. Obviously these things are all relative and we felt like we had small sample sizes relative to Facebook or Twitter or LinkedIn. Certainly with millions of users you can do things that you can't do with tens or hundreds or thousands of users. Generally speaking, you want to test things where the desired action is taken by the least few hundred people, and it depends on how big of a change you really need.

There's certain cases where a 1% difference is going to make a material impact on a business, and in other cases, your product is fundamentally failing and if you have a 1% change, that's not going to go from failing to not failing — you need a 50% change or a doubling or that sort of thing. So a question I would ask is, “How big of a change do you need for it to make a real impact on your business?” and, “Do you have the numbers to support that?”

If 3% of people are doing something and you need a 50% change for that to happen, do you have a large enough sample to measure that 50% difference? Ballpark numbers, going from 20 to 30, it's going to be hard. If you have 20 people doing it and your test might result in 30 people doing it, it's tough to tell that difference, whereas 200 to 300, it's easy to tell that difference. However, a 1% difference, going from 200 to 202, that's not statistically meaningful. You need a larger sample.