A/B testing with Orchard?

Topics: General
May 27, 2011 at 11:48 PM

A common practice for firms wishing to optimize their websites is to engage in A/B testing ( http://en.wikipedia.org/wiki/A/B_testing ).

Here is an example of using it for websites ( http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/ ).

Has anyone thought about how this might be done with Orchard? I am also curious to know if this is something you are doing with other websites and how important you find it to be?

May 28, 2011 at 1:14 AM

From that wiki article, "A/B testing, split testing or bucket testing is a method of marketing testing" - I'm not sure how a marketing testing practise would apply to web optimization?

May 28, 2011 at 3:31 AM
Edited May 28, 2011 at 3:32 AM
randompete wrote:

From that wiki article, "A/B testing, split testing or bucket testing is a method of marketing testing" - I'm not sure how a marketing testing practise would apply to web optimization?

Perhaps an example might help. Suppose you were running a travel website where you wanted people to register. Now suppose you showed (based on random selection) half the new visitors a picture of European beaches on the home page and the other half a picture of the Grand Canyon. Further suppose that you checked your weblogs and found that visitors from North America who were shown the photo of Europe were twice as likely to register as those shown the photo of the Grand Canyon. Further suppose that the exact opposite were true for visitors from Europe.

Once you had that data, you might conclude that the best strategy for attracting new people to register on your site would be to show the Grand Canyon to visitors from Europe and European Beaches to people from America.

Big sites like Google and Bing do this sort of thing all the time, delivering content tailored to what they know (such as IP address or time of day) about a web visitor.

May 28, 2011 at 10:44 AM

So it's basically a case of collecting detailed statistics about usage of your website, tracking someone from the moment they arrive to the moment they leave. Then interpreting that data. To be honest something like Google Analytics is probably the best and most readily available tool for something like that; statistical processing of that kind is hard and very database intensive. Remember big sites like Google have ridiculous amounts of servers to be able to do this stuff.

May 28, 2011 at 12:14 PM
randompete wrote:

So it's basically a case of collecting detailed statistics about usage of your website, tracking someone from the moment they arrive to the moment they leave. Then interpreting that data. To be honest something like Google Analytics is probably the best and most readily available tool for something like that; statistical processing of that kind is hard and very database intensive. Remember big sites like Google have ridiculous amounts of servers to be able to do this stuff.

Actually the Google tool for this is Website Optimizer ( http://www.google.com/websiteoptimizer/tour.html ). Here is a video with an overview of the various places Google wants specific javascript inserted into a page: http://www.youtube.com/watch?v=1yTjj9MnzRY

With static HTML pages this is comparatively straight forward to do what Google requires. With dynamic page composition, as in Orchard, there are additional considerations. As a simple example, consider A/B testing for a single widget where the test you wish to perform is two different colors of fonts for just that widget. Normally font color would be determined by the CSS sheet, layout (for injection of the required javascript) by the placementinfo file etc. It is not clear to me how to do what Google would require with Orchard.