Google Website Optimizer Hint: Beware of Small Numbers, Don't Rush to Judgment
Science takes time.
Whenever we're about to launch a new experiment, I always give clients the same speech. It goes something like this:
"You probably won't be able to resist logging into GWO and checking the results every day. But always bear in mind that experiments cannot yield reliable results until they have run long enough to gather sufficient data. Early results are often surprising -- even alarming -- but you must resist jumping to conclusions. Let the experiment run its course."
Now I have a confession to make: I sometimes have a very hard timing taking my own advice. A recent example comes to mind.
We were running an A/B test of a lead-generation form for a Financial Institution. Our revised ("B") version quickly took the lead over the original. Within a week or so, it was showing an improvement of about 200%! We were astonished, and of course we started congratulating ourselves on how clever we were.
Luckily, we didn't share our internal boasting with the client. Because over the next few weeks, we watched in dismay as our lead fell to 150%... to 100%... to 50%... and eventually settled at around 20%. Still respectable, but imagine how bad we'd have looked if we had told the clients we'd given them a 200% lift!
In other cases, we've seen GWO declare a winner, only to "undeclared" it the next day. You can't rely entirely on what the tool is telling you.
So when can you confidently say the experiment has run its course? I'd suggest a good rule of thumb is, when all three of these criteria are met:
- The experiment has run for at least two weeks (preferably longer)
- Each version of your page has 100 or more conversions, and
- Google declares a winner
Don't trust what small numbers tell you. Be patient... difficult as it is.
If you like this post then please consider subscribing to our RSS feed.
My point is just that even if GWO does declare a winner quickly, it's best to keep the test running for a while longer. Don't rely on small numbers, regardless of what GWO says.
Of course, it's just a rule of thumb and there may be exceptions. Especially with regards to the "minimum 100 conversions per version" guideline. If you're running a multivariate test with dozens (or even hundreds) of combinations, it may be impractical to wait for 100 conversions per version. But the principle remains the same: don't jump to conclusions and stop the test at the first indication that you've found a winner. The situation could change over time.