A/B testing can turn skeptics into believers. Jamie, a designer at 37signals, recently shared a great case study of A/B testing Highrise.
So, what are the mechanics of actually setting up an A/B test in a Rails app?
One approach we’re trying right now on
Trajectory is using the split gem with
KISSMetrics. For the first few months of Trajectory’s life, we felt it was
necessary to introduce Trajectory and explain why we made it in the face of
existing tools.
Now, it’s time to explain its benefits on its own merits and see how well that resonates with potential users versus the old copy. This recipe will show that example. We’ll also be testing more dramatic layouts, which this combination can also handle.
Setup
Gemfile:
gem "split"
config/initializers/split.rb:
Split.redis = ENV['REDISTOGO_URL'] || 'redis://localhost:6379'
Split.redis.namespace = "split:trajectory"
Red
Let’s use the Cucumber directory convention.
features/visitor/views_homepage.feature:
Scenario: Original landing page
When I go to the home page with the "original" alternative for the "landing_page" experiment
Then I should see "Over the past 8 years, we've used many tools"
And KISSmetrics receives the following properties:
| property | value |
| landing_page | original |
Scenario: New copy on landing page
When I go to the home page with the "new_copy" alternative for the "landing_page" experiment
Then I should see "One gorgeous tool that everyone actually LIKES to use"
And KISSmetrics receives the following properties:
| property | value |
| landing_page | new_copy |
The split gem’s documentation provides a way to override the alternatives.
features/support/paths.rb:
when /^the home page with the "([^"]+)" alternative for the "([^"]+)" experiment/
"/?#{$2}=#{$1}"
We can figure out the expected Javascript from the KISSMetrics API documentation.
features/step_definitions/kissmetrics_steps.rb:
Then /^KISSmetrics receives the following properties:$/ do |table|
table.hashes.each do |hash|
property = hash['property']
value = hash['value']
expected_javascript = %Q{_kmq.push(["set","#{property}","#{value}"]);}
page.should have_content(expected_javascript)
end
end
Green
In this example, we set up two partials that simply contain a translation.
The translation uses the Rails i18n API and is backed by Copycopter in order to make changes to it without re-deploying.
app/views/homes/_new_copy.html.erb:
<%= t(".letter-new-copy", :default => %{
<p><strong>Less frustration, more joy: there's a better way to build software.</strong></p>
<p>You know that your software planning tools aren't perfect. It's not clear what to do next.</p>
<p>We had the same problems. We’re thoughtbot, a web design and development agency, and that's why we made Trajectory.</p>
<p>Imagine if you could build better software, faster. Imagine your teammates not waiting on each other, having a clear sense of what to do next.</p>
<p>It's super easy to try out for free. Hit the ground running by importing your Pivotal Tracker project or invite your current team members.</p>
}) %>
app/views/homes/_original.html.erb:
<p><strong>Hi, we’re thoughtbot, a web design and development agency.</strong>
<%= t(".letter", default: %{
<p>Over the past 8 years, we've used many tools for project communication and planning.</p>
<p>Basecamp was great for discussion and communication. Pivotal Tracker was great for user stories and emergent planning.</p>
<p>We've grown tired of having one tool that designers love, one tool that developers love, and no tool that clients love.</p>
<p>We created Trajectory to solve our own problems. We now use it on all of our projects. Maybe it can solve your problems too.</p>
}) %>
Using split is pretty simple.
app/views/homes/show.html.erb:
<%= render partial: ab_test("landing_page", "original", "new_copy") %>
split provides a web interface but we have all our funnel metrics in
KISSMetrics so we want to send the data there.
app/views/shared/_javascript.html.erb:
<script type="text/javascript">
<% Split::Experiment.all.each do |experiment| %>
_kmq.push(['set', { '<%= experiment.name %>': '<%= ab_test(experiment.name, *experiment.alternative_names) %>' }]);
<% end %>
</script>
Putting A/B testing in context
This is a recipe for “how” to A/B test but if you’re interested in “when” to test, see our A/B testing page in our playbook.