If You Don’t Learn, You Might As Well Not Test
This is a guest post by our friend Richard de Vries.
The most popular way of optimizing a website is by A/B testing it. For some reason the bigger the website gets, the harder it becomes to test. If you are dealing with optimization in a big company there is a fair chance you will recognize this. Putting up with rules and regulations, trying to get everyone involved and not getting people to agree upon what to test are just a few of the many things you will run into. The easiest way to solve this is to avoid working for big companies and only work for startups. However if you change your approach to testing, you might be able to create some magic, even in a big company.
The difference between a startup and a big company
According to the book The Lean startup by Eric Ries the difference between a startup and a mature company is the ability to learn. The key to success for a startup lies in how fast and how effective it can learn. For some reason, the more mature a company gets, the less effective it becomes at learning. I think this alone is the reason why bigger companies seem so stupid at times.
Why learning is so important
“On Monday Jack goes to the pub and drinks 5 whiskey and a glass of water. On Tuesday he wakes up with a huge headache. At night he goes to the pub again and drinks 5 whiskey and glass of sprite. On Wednesday he wakes up… again a huge headache, so that night he drinks 5 whiskey and a glass of coke. When Jack wakes up on Thursday with yet another hangover, he says to himself ‘I really shouldn’t drink that last drink anymore!’.”
This is a story told by Paul Hughes on the design for conversion conference. With this story Paul tried to explain the danger of testing the wrong thing in AB testing. I think this is the absolute best way to explain why you should learn from your tests.
Build – Measure – Learn
Once you have established that learning is a vital part of optimisation, you need to add this to your ‘process’. I put process in quotes because it can be anything from a strict company process to a bunch of people more or less doing things in the same order.
I’m sure you have built and measured things before (probably even through some sort of process). So basically, all we do is adding a learning phase to that process. In the lean startup build and measure does not have to differ very much from the way you already work, it’s just all agile.
If you are building or designing websites you are very familiar with this phase. Build is a little bit confusing as it is both the design and development phase. What should be important to remember in this phase is that you want to build something that is measurable, not something that is perfect.
Once you have established that something built and designed to a level that can be tested you can measure it. This does not have to be an A/B test by definition. You can also measure in some (guerilla) usability testing. What is important from the object of the measure phase is that you receive much data. In my experience, the more data the better, but one should not jump to conclusions. Most of the times, conclusions that are drawn in this phase are rushed and therefore simply not true.
Conclusions in the measure phase are not true, simply because data is not a conclusion and a conclusion is not data. In order to get to a conclusion you must ask why. In fact you must ask why five times
Say we test a piece of copy: “Best value” vs. “Most purchased” and “Most purchased” won. Apart from relying on existing knowledge such as the psychology of persuasion, you can also go back to your case. Apparently your users were triggered more by the fact that most people bought this versus it being well priced. So we ask “Why are users triggered more by most purchased than best value?”
(possible) answer “Because people are more concerned about getting the right package than getting the best deal for their money?” – “Why?”
(possible) answer “Because they fear that when buying the wrong package they can no longer switch” – “Why?”
(possible) answer “Because it is not clear how to switch between packages on the conversion page” – “Why?”
(possible) answer “Because there is no text, link or graphic explaining this”
Exactly this answer is a hypothesis to test again. The smart reader here notices that we only needed 4 why’s to get to this. Excellent!
Use what you learn
Every big organisation stepped in the pitfall of not learning anything at all because they got stuck in finding a way to capture that knowledge. Wiki’s, templates, books, sessions, presentation are all great ways of sharing knowledge and they should all definitely be used in some way. However the main purpose of learning in our context is using it again. By defining a new hypothesis from our test is the ultimate way to do this.
And now it all begins
Optimizing in this cycle is very much like an engine (or any other cyclic process) the first cycle will be tough and clunky. And again much like an engine, the first cycle will be clunkier and tougher the bigger your website or company is. However once you get it started and you manage to get through the first cycle you will find that it will go smoother and smoother and you might have created your own startup magic in a big company.