AB testing and pressing questions

Before you realize it, its been more than a year since i last posted. Its been mostly Twitter activity (@shiv17674) so its not like i’ve been completely holding back my thoughts.

Anyway, I find myself getting excited about my recent escapades with AB testing that i felt it would make for a nice post. Lately I’ve been experimenting with Optimizely a lot and its been eye opening to say the least. From small UI changes to complex functionality changes it sure looks like we have a come a long way since relying on simple user tests and surveys to tell us what a customer/user would prefer.

It all started after reading Avinash Kaushik‘s book Web Analytics 2.0 and the Always be Testing book (on Amazon). Shall we just say I have read about the top of the mountain and I’d like to go there. If you want a crash course on AB testing, I highly recommend this article from Smashing Magazine.

So when the opportunity presented itself, I jumped on it and now I’m like a kid in a candy store. (Do i go with Candy A or Candy B). Of course nerdy ecstasy aside, there are some really important organizational business decisions that will present themselves when we find the results of our experiments.

– Once we have the capability in place, everybody’s going to want to run these experiments. Organizational changes will have to be in place without overly burdening with red tape.  

– How do you prioritize which tests to run first?

– How do you make sure one test does not interfere with another?

– What if we can’t test everything? Will some changes remain untested?

– How long do we need to run a test and how does that influence our planning?

– I assume there are always things that just aren’t testable. Some things are just emotional and visual elements on a page. They are meant to make the customer comfortable within your product or add credibility. Whether to place those elements on the top or bottom or left or right will probably not be measurable. Or is there a way to measure that?

– What if the test reveals that your popular feature driving a sizeable chunk of your KPIs is actually hindering your customers ability to fully engage with your product?  

All these questions do have answers or at least I believe there are. But they’ll need to be evaluated and revised over time based on the experience we collect. I will try and post these thoughts on here when possible along with lessons learned. Certainly exciting times.