---
title: 'Missed Connections: I Saw You Testing My Website'
teaser:
tags: design
author: Chad Mazzola
published_on: 2010-07-21
---

![''](http://media.tumblr.com/tumblr_l5wz5vA2qI1qa5kd1.jpg)

Usability testing is like flossing: everyone knows it's important, but they
don't do it nearly as much as they should. Because let's face it, making sites
is a lot more exciting than testing them. Well, this past week we ran our first
round of in-house usability tests and wanted to share how we did it, what we
learned, and what we need to improve.

[This presentation](http://www.slideshare.net/glusman/lean-usability) by Meetup
on Lean Usability was a big help in putting together our initial plans. It does
a good job of laying out the pros and cons of several approaches and points out
what you should focus on at the beginning, before you have any experience to
guide you. Read it.

## How we did it

* We chose to test an important form that we were in process of redesigning for
  a client. With differing opinions about the approach to certain aspects of the
  redesign, it was a prime candidate for usability testing.
* We decided to use [Silverback](http://silverbackapp.com/) to capture the test
  sessions. We liked it because of the ability to record audio and video of both
  the user and the screen, giving us an objective record of what happened during
  the session that could be reviewed later.
* We placed an ad in the "volunteers" section on craigslist. The message gave a
  quick summary of what we needed the volunteers to do, who we were, and
  directed them to a [Wufoo](http://wufoo.com/) form where they provided us with
  their information. We offered $30 for about 30 minutes of their time and ended
  up having almost 70 responses in less than 24 hours.
* We decided to schedule 5 sessions, one right after the other, during one
  afternoon. We picked 5 people whose schedules worked with ours and emailed
  them to let them know we had scheduled a session for them. We also emailed
  everyone we didn't select, thanking them for contacting us.
* Before the sessions, we wrote up scenarios that would be given to the user. We
  decided on three tasks, each more complex than the last, that would focus on a
  key point of the clients' form we were redesigning. We also prepared a release
  form that let testers know that we'd be recording audio and video of their
  session, that this would be viewed by thoughtbot and our client, but would not
  be used for any other purpose.

## What we learned

* We sent a confirmation message the morning of the testing to all the testers,
  and ended up having 4 out of 5 show up.
* The scenarios/instructions are important. Our first tester was completely
  stumped by the first scenario, and we realized that we had not been as clear
  as we could have been. We quickly modified the instructions before the next
  session.
* Silverback has drawbacks. Each of our sessions was between 22 - 30 minutes and
  takes up more than a GB of hard drive space. Additionally, exporting video of
  the sessions takes a long time. On a MacBook with an dual core processor and 2
  GB of RAM, it takes 2 to 3 hours. (It still takes over an hour on a maxed-out
  MacBook Pro.)
* It takes time. Recruiting testers, writing scenarios, running sessions,
  exporting video, sending the results -- all of these things take time. Make a
  realistic estimate of how much time you'll actually be investing in doing
  usability testing.

## What to improve

* The users we recruited for this round of testing didn't need to have any
  special qualifications. As we do more testing, we'll likely encounter a
  situation where we'll need users with specific knowledge related to the site
  they'll be testing. (Running usability tests on
  [Hoptoad](http://hoptoadapp.com) would present this problem.) It's unclear if
  craigslist will work in these situations.
* While it's impossible to completely remove any confusion that the instructions
  may introduce, we'll need to do a better job of writing scenarios that are
  clear.