A-B Testing

This is old, but I’m blogging it so I can remember it:

ContentBiz 2005: Motley Fool and Optimost “Landing Page Tests Case Study” contentbiz05.jpg (by David Eckoff, guest blogger): I am attending the 5th Annual Selling Subscriptions to Internet Content Summit in New York this week, privileged to be co-blogging on-location with Dorian Benkoil. This is the first in a series of my reports covering the more interesting presentations from the conference.
Tuesday morning Greg Martz of Motley Fool and Mark Wachen of Optimost presented a case study on “Landing Page Tests – What worked and Failed Out of Dozens of Design and Copy Elements“.
Martz talked about how Motley Fool had been performing standard A-B tests on its landing pages to optimize the design and copy for maximum conversion. However, the A-B tests were very time consuming and limited in scope.
Seeking a better solution, Motley Fool turned to an approach called multivariable testing. What’s that about? It’s a method of testing multiple variables each having multiple values.
Working with Optimost, Motley Fool tested 13 variables, including page heading, headline, order of copy blocks, copy, offer presentation, submit buttons, guarantee language, etc. In total, 88 values were tested, including page headings, 16 headlines, 6 order layouts, etc.
Think you could test that manually? Think again. According to Optimost, with that number of variables and values, there are over 1 BILLION possible permutations!
The result of the initiative: 39.5% increase in clicks to the order page, and 36.4% increase in subscriptions. Whammy!
What advice did Martz and Wachen give the audience?
* Test everything, and everything against everything else. (Copy, offers, submit buttons, images, ordering, etc.)
* In the long term, multivariable testing is much more cost effective than traditional A-B testing.
* Need buy-in of tech and design teams, this is crucial to success. The key is to help the teams develop a culture of testing, instead of relying on gut instinct or a laundry list of best practices.
* Testing instructions: rinse and repeat. Not just a matter of keeping copy or design fresh. Until you get 100% conversion, you can do better!
In my next report, I’ll uncover the details presented by Stephen Wynkoop and Dr. Flint McLaughlin in “How An Optimized Subscription Path Increased Paid Subscribers 175%”.
de.thumbnail.photo.jpg Guest blogger DAVID ECKOFF is senior director at RealNetworks, where he leads business development for Real SuperPass. The opinions expressed in his guest blogging from the Subscription Summit are his own and not those of Real. You can reach David Eckoff by e-mailing davideckoff …at… gmail.com

Author: David Churbuck

Cape Codder with an itch to write

0 thoughts on “A-B Testing”

  1. A few others too:


    There are other cheaper products as well.

    Sitespect is cool in the way it’s minimally invasive. You don’t need to add js all over the place, nor do they keep talking about the Taguchi Method.

    Another great ab test for testing ab testing companies is to ask for the greek that represents variance. That separates the cream from the curdle when they start the stats speak.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: