We test products. Not people.
Running user research and usability tests early and often is critical to product success.
We even test sketches to get a feel for flows and mental models. The earlier the stage, the more we're testing the problem/solution fit and gathering research on potential users. The later the stage, the more we're testing actual usability of the product. How easy it is for a user to reach an outcome?
How we test
User interviews and usability tests are the most effective way to test a product viability and usability. By continually testing it verifies that the product and team are focused on solving people's real problems and creating a great user experience for the product. We've found that having a testing plan early is the best way to consistently run user interviews and usability tests during the project. A starting point for the project's interviews and testing should be biweekly. This sets the expectation that interviews and testing are needed and the team should discuss if that plan will work for their individual project. Testing should always accommodate the users and project needs.
First we create outcomes
We think about usability testing similarly to test-driven development: writing falsifiable outcomes for users. Our outcomes are written in the form of a script that's written in the same way as a jobs story.
an example outcome to test
When I arrive at work I want to review the team's status updates so I can help any blocked teammates.
Then we find testers
The most representative candidates are going to be sourced from our existing userbase. Send out a tweet, add a banner to our site, or add a link to our newsletter. Even pre-launched projects have a mailing list to use.
When we have trouble sourcing or aren't interested in existing users, craigslist can be effective to find candidates. Our office manager puts a posting on craigslist, schedules them to come into our office, and pays them $30 for their time after the test.
After the tester has arrived, we introduce ourselves and explain the process. There's no need to be strictly formal, we want them to be at ease. If we are filming them, we ask them to sign a simple consent form.
Don't Worry …
To have a relaxed user test, it's important to remind the user of a few things
None of what you're about to see was made by me. There's no way you can hurt my feelings.
We're testing the software, not you. It's not user testing, it's usability testing.
We are looking for your honest feedback.
We ask questions
Once underway, we ask them to say out loud what they're thinking as they're using the software, which can feel unnatural but is important. While running the session, the only reasons to speak are to get them to talk out loud again, ask questions, and provide them with outcomes to achieve.
We should avoid leading questions such as “Was this task difficult for you?” but should ask follow-up questions. For example, if someone says “That's awesome!”, we shouldn't silently pat ourselves on the back. We should say “Why is it awesome for you?” We are seeking understanding.
We learn and apply
After the session, we look back at our notes to identify the top handful of problems and fix them before the next round of usability tests. If there's a problem revealed with the navigation, there's a temptation to totally re-do it. We try to resist that urge and come up with less drastic changes.
Then we do it all over again.
Hire us to help you build excellent software
Our best work gets done when we can work face-to-face with you. Chat with a designer or developer at one of our studios near you.Let’s make something great together