Usability tests are the most effective way to test a
product's user experience. Continuous testing verifies verifies that
the product and team are focused on solving people's
real problems and creating a great user experience for the product.
We've found that having a testing plan early is the best way to
consistently run user interviews and usability tests during the project.
A starting point for the project's interviews and testing should be biweekly.
This sets the expectation that interviews and testing are needed and the team
should discuss if that plan will work for their individual project.
Testing should always accommodate the users and project needs.
We're testing the software, not you.
Usability is the measure of how easy it is for a user to reach an outcome.
We think about usability testing similarly to test-driven development: writing
falsifiable outcomes for users. Our outcomes are written in the form of a
script that's written in the same way as a jobs story.
When I arrive at work
I want to review the team's status updates
so I can help any blocked teammates.
Once the script is written, we find testers.
The most representative candidates are going to be sourced from our existing
userbase. Send out a tweet, add a banner to our site, or add a link to our
newsletter. Even pre-launched projects have a mailing list to use.
When we have trouble sourcing or aren't interested in existing users,
craigslist can be effective to find candidates. Our office manager puts a
posting on craigslist, schedules them to come into our office, and pays them
$30 for their time after the test.
We have a simple template for finding people on craigslist.
After the tester has arrived, we introduce ourselves and explain the process.
There's no need to be strictly formal, we want them to be at ease. To have a
relaxed user test, it's important to remind the user of a few things:
- "We are looking for your honest feedback."
- "None of what you're about to see was made by me. There's no way you can hurt
- "We're testing the software, not you. It's not user testing, it's usability
If we are filming them, we ask them to sign a simple consent form.
Once underway, we ask them to say out loud what they're thinking as they're
using the software, which can feel unnatural but is important. While running the
session, the only reasons to speak are to get them to talk out loud again, ask
questions, and provide them with outcomes to achieve.
We should avoid leading questions such as "Was this task difficult for you?" but
should ask follow-up questions. For example, if someone says "That's awesome!",
we shouldn't silently pat ourselves on the back. We should say "Why is it
awesome for you?" We are seeking understanding.
After the session, we look back at our notes to identify the top handful of
problems and fix them before the next round of usability tests. If there's a
problem revealed with the navigation, there's a temptation to totally re-do
it. We try to resist that urge and come up with less drastic changes.