Lesson: Data-Driven Decisions with Mike Greenfield

Step #3 Testing: Finding the right fit

We did a lot of A/B testing throughout with Circle of Moms. Early on it was more around the invitation process and how do we get users into the system. Then later on, when those Facebook invites weren't working as well, it was more about how do we reengage our users, how do we find something that's going to be interesting for them.

We would have 20 different piece of content that we might send somebody in a given week. We had an editorial team writing 20 things each week. Each of those 20 things would go out to, say, 10,000 users, so a total of 200,000 users out of 6 million getting one of those. From that, we could get a gauge on is this an e-mail that people are going to click on, yes or no, and to some extent who's going to click on it.

Once we had that initial sense, we would take a bunch of those 20 pieces of content and say, "You know what? These are not winners. They're not going to go out to anybody else." Then we would take the better performing ones and we would send them to more people and we would figure out both what's the best subject line and what's the best way to present this specific blog post, but then also what is the type of user that's going to click on this e-mail.

We used both content from our community and content from our team and they complimented each other in certain ways. What we generally found was that it was easier to get really good content that we wrote because our team learned over time and then figured out what worked. But I think it is certainly is possible to have good community content. I think that Core does a very good job with their e-mail system. I think the average quality of their answers and their questions is a little bit higher than what we saw at Circle of Moms. If you take any random question, it's more likely that it's well-worded then something similar with Circle of Moms.

I don't know all the dynamics of how they built their e-mail system, but that makes it a lot easier to say, "Okay. Let's take community generated content." We did some things with Circle of Moms where we would take a community post and we'd stick it in an e-mail. We had a handful of successes, but the success rate was lower than it was for the things that we were writing ourselves.