Split Testing

Listing 3. Gemfile, Used by Bundler

source 'https://rubygems.org'

gem "sinatra"
gem 'split', github: 'andrew/split'

Let's review them briefly. The setup.ru file is the way that Rack runs your application. If you want to run the application without any external HTTP server, you can use the rackup command:

rackup config.ru

Rack then will run an HTTP server (defaulting to the built-in WEBrick server that comes with Ruby), acting as the glue between the server and your application. The config.ru basically bootstraps your application, loading the gems listed in the Gemfile (thanks to Bundler), then loading your application and the special code needed to run the Split dashboard.

Then you do something interesting. Rather than simply running your application, you tell Rack that it should route requests to different places. Anything starting with a / should go to LinkSplit.new, meaning a new instance of the Sinatra application. But, anything starting with /split will be routed to a completely separate Sinatra application, Split::Dashboard.new, part of the Split gem.

The Sinatra application defines two different URLs, both of which are available via HTTP GET requests. The first, /foo, displays the contents of the foo.erb file (located in views/foo.erb), as shown in Listing 4. While this file seems, on the surface, to be no different from any other ERb (embedded Ruby) document, it contains the telltale sign of a split test:

<% ab_test("click_text", "Click on me!", 
 ↪"Click here!") do |click_text| %>
    <%= click_text %>
<% end %>

The ab_test method, loaded by the line:

helpers Split::Helper

in linksplit.rb does several things at once: it defines a split test (the first parameter), and it provides the two alternatives that you are interested in testing. In this example, you can see that you're testing whether "Click on me!" (the control) converts more frequently than "Click here!" (the experiment). You then pass a block to the ab_test method. The block can contain any text you want, and it can be as long or short as you want. The key thing to realize is that the block parameter (click_text in this case) will contain one of the two text strings.

Listing 4. views/foo.erb

      <a href="/bar">
        <% ab_test("click_text", "Click on me!", 
         ↪"Click here!") do |click_text| %>
          <%= click_text %>
        <% end %>

The above split test, thus, will compare the efficacy of two different links. But, you easily could switch the class of an HTML tag (thus giving different styles, including colors and fonts), a different location, the addition of a picture or anything else.

Once this is in place, the Split gem will produce an experiment, displaying one of these text strings to each of your users. Typically, you'll want to show them equally. There are ways to change the ratios, so that you're showing your experimental text to only a small proportion of your users.

However, showing these different texts isn't quite enough. You also need to be able to tell Split when visitors have "converted"—that is, when they have achieved the goal that you set out. In this particular case, you want to know which text is more effective at getting users to click on the link. So, you should report a conversion back to Split when the user has clicked on the link. In other words, in the Sinatra application's handler for the "/bar" method, you invoke the "finished" method, passing it the name of the experiment you want to indicate was finished:


Once you have put a split test in place, you can sit back and wait for users to come to your site. Some will get the first test, and some will get the second. How do you know which was more effective? By using the Split dashboard, which you connected to /split when you set up routing for Rack. It will show the percentage of users who responded to each of your experimental texts, so you can see which was more effective. Split also will tell you the confidence that it has in the results of this test. As a general rule, the more people you test, the more confident you can be in the results. But, statistics also show that even a (surprisingly) small sample size can give you interesting and meaningful results.

The dashboard is useful in several ways. First, it allows you to look at your various experiments, seeing how many people are viewing each of your experimental texts, and how many did and didn't complete the test. It also allows you, after examining the data, to use one of the two experimental texts permanently. This is useful in several ways. It means that a nonprogrammer can control and decide on each text. But, it also means that you don't have to go back into the code and remove your experiment when it is over. You can take some time, letting the Split system handle it for you. You also can reset the statistics, allowing you to try new ideas or throw out old ones.


Split testing is a powerful tool for helping ensure that your goals are met—whether those goals are selling something on-line or just getting people to sign up for your mailing list. The Split gem works with any Rack-powered Web site, and it makes it extremely easy to implement and then act on your optimization experiments. I encourage you to try some experiments on your own Web applications and see if you can find ways to get your users to convert into customers. If so, congratulations! You've discovered that modern, on-line marketing requires an understanding of programming as much as an understanding of the customer.


The Split gem described here is at https://github.com/andrew/split. As indicated in this article, Split works with any Rack-based Web application, which basically means Ruby on Rails and Sinatra.

You can read more about Sinatra at http://sinatrarb.com.

Patrick Mackenzie, the author of the A/Bingo gem, has written extensively on the subject of conversion optimization. You can read his articles on the subject at http://www.kalzumeus.com. In particular, look for the "conversion optimization" heading on his "greatest hits" page.


Reuven M. Lerner, Linux Journal Senior Columnist, a longtime Web developer, consultant and trainer, is completing his PhD in learning sciences at Northwestern University.