Split Testing

It's nice to have many people visit your Web site. It's even better when people don't just come to your site, but also enjoy your content. But, best of all is when visitors to your site do what you would like them to do—sign up for your newsletter, register for your SaaS application or buy one of your products.

The rate at which visitors become customers is called the "conversion rate", and it's probably the top priority for Web-based businesses. If you can convert 10% of your visitors into customers, you're doing twice as well as the person who can convert 5% of visitors into customers.

What leads people to convert more often? That's a question to which I'm still learning the answer, as is an entire industry of Internet marketers and "conversion optimization" experts. The thing is, it's often difficult to know what will work. Should you use a green button or a red one? Should your headline be a question or a statement?

One of the most popular, and effective, ways to check the effectiveness of your copy is to do "split testing", sometimes known as "A/B testing". Although I heard about A/B testing years ago, I have only recently actually started to apply it.

In this article, I describe a simple way to run split tests on your Web pages. The examples I show here are written in Ruby, but there are A/B testing libraries for many different languages. There also are third-party businesses that can help you with A/B testing, either on your own Web site or on your mailing lists, lead pages and other products.

It's All about Conversion

The most important thing to keep in mind when you're doing split testing is that you're trying to get users to do something. What that something is depends on your site. The goal at Amazon is to get you to buy things. The goal at Google is to get you to click on ads. And the goal for a mailing-list subscription form is to get people to sign up.

Once visitors have achieved your goal, they have been "converted" into customers. The goal is to increase the number of such conversions—giving you more customers, subscribers or users of your system. The key insight with split testing is that by changing the text, graphics and even layout of your page, the number of conversions will change as well. The question is, which of your various ideas will work best?

Split testing uses the scientific method, backed by simple statistics, to try to answer that question. In science, you can test a hypothesis by using a control and an experiment. Split testing works the same way. You take your existing text and an alternate text, and display them to your users. One of them probably will work better than the other. You analyze the numbers to understand which worked better, and then you use the best one. Then you start all over again, trying to find another improvement via split testing.

In order for split testing to work, you need to do several things:

  • Define what counts as a conversion. This often is described in terms of the user arriving at a particular page on the site, such as a "thank you" for shopping that appears after a successful sale.

  • Define the control and alternate texts.

  • Wait for enough users to see both.

  • Analyze the numbers.

Introducing Split

Regular readers of this column know that I tend to use Ruby on Rails for most of my Web development work. Rails isn't the only Web framework written in Ruby, although it certainly is the most widely used. All modern Ruby-based Web frameworks use "Rack" to communicate with the HTTP server software that invokes them. This is analogous to the Python world's WSGI. The idea in both of these cases is that the HTTP server doesn't need to know much about the Ruby or Python application (or framework) and vice versa. This means libraries that know how to work with Rack can operate with any framework, not just with Rails. In the specific case of Ruby, Rack-sensitive gems thus can work with Rails, Sinatra or anything else that follows the Rack API.

The Split gem, written by Andrew Nesbitt (with a number of contributors), is one such package of code, able to provide split-testing execution and analysis to any Rack-compatible application server. It is based on a number of earlier split-testing gems, including the well-known A/Bingo gem by Patrick Mackenzie. The idea behind Split is you identify your conversion target, automatically try two or more variations on users, collect statistics on which of them actually managed to achieve more conversions, and then start to test things again.

Split takes care of much of this work for you, allowing you to concentrate on the portion of your site you want to change, rather than the mechanics of setting up experiments and reporting their results. Split also implements a number of features that ensure the statistical power of your results. For example, it will involve each participant in only a single experiment at a time, and it will compare results only after at least 30 people have seen it. These options can be changed, of course, but the defaults are more than good enough for most basic tests that you'll want to run.

You install split as you would any other Ruby gem:

gem install split -V

Note that I use rvm, the Ruby version manager, which allows me to work with multiple versions of Ruby at any given time. One side effect of using rvm is that gems are installed in my own home directory, rather than on the system. As a result, I don't need to preface the gem command with sudo; depending on your system configuration, you might need to do so.

After installing the gem, you also will need to install and start Redis, an extremely fast and popular key-value store. I have used Redis in a number of projects through the years, and I never cease to be amazed by its utility for caching. In the case of Split, Redis is used to keep track of the experiments you are running. You can start Redis by executing:


Running a Split Test

Now that you have installed the gem and Redis, you're ready to perform some experiments. Let's create a simple Web site using Sinatra, in which the page displays a link. While Sinatra has a reputation for being simple and for allowing you to create a single-page application, I've generally gone for a configuration that uses several files: the application itself, a Rack configuration file (setup.ru) and a Gemfile to list all the Ruby gems I want to use (via the "Bundler" management gem). These are shown in Listings 1, 2 and 3.

Listing 1. linksplit.rb, the Main Sinatra Application

#!/usr/bin/env ruby

class LinkSplit < Sinatra::Base
  enable :sessions
  helpers Split::Helper

  get '/foo' do
    erb :foo

  get '/bar' do
    erb :bar

Listing 2. config.ru, the Rack Configuration File for LinkSplit


require './linksplit.rb'
require 'split/dashboard'

run Rack::URLMap.new("/" => LinkSplit.new,
                     "/split" => Split::Dashboard.new)

Reuven M. Lerner, Linux Journal Senior Columnist, a longtime Web developer, consultant and trainer, is completing his PhD in learning sciences at Northwestern University.