User Testing Gone Wild: A Guide to Course Correction

Jaclyn Perrone

You have a usability test this afternoon and you thought of everything. Your prototype is built, your script is written, you gargled, flossed, did some vocal warm-ups. Excellent. You are so ready for this.

You and your team are traveling to a clothing boutique to talk with the store owner. You will show her your new inventory app and will collect her unfiltered, honest feedback about its interface and interactions.

Right?

Most of the time, yes. But sometimes the unexpected happens. No matter how much you prepare for an interview or usability test, you still need to keep on your toes and quell some hiccups along the way. These hiccups can come from you, your team, your materials, your environment, the users themselves, etc. Some are obvious, and some are not so obvious. The following are ways to address these common problems, in the hopes of avoiding them altogether.

Your user is clamming up and growing quiet

They’re sweaty, their arms are crossed, and their hands shake when they touch the tablet. Or worse, they start to back away from you…slowly. Body language is the most important indicator of a person’s comfort. If they slowly roll their chair to the other end of the table as you speak to them, you need to rethink your approach.

How to avoid this:

  • Build a rapport before you transition into testing. If you are in their store, allow them to greet you and show you around. If they traveled to your office, ask them how their trip was and if they need any refreshments. Shifting the focus onto them shows that you care about their needs and what they have to say.
  • Learn how to read body language to empathize with your user. Once you become aware of what their body is telling you, you can start to adjust yours accordingly.
  • Know that a usability test can be distressing for them. They might not be as familiar with this process as you are. They can easily feel like they’re being watched, tested, and judged by a panel of professionals. Not a fun feeling. It is your job to make them feel as comfortable as possible, so that you can gain their trust. And having one person facilitate will help them open up and ultimately feel comfortable making mistakes.

During the session:

  • Check your posture. Are you crowding them or leaning in too much? Loosen up and give them more breathing room.
  • Check where you are sitting. If you are sitting across from them, swing around and sit next to them. This shows that you are with them, not against them.
  • Ask them if they need a break.
  • Make them laugh! There’s no better way to ease the tension.

Too many people talking at once

Your team has gone rogue. Everyone started out on their best behavior. The facilitator was facilitating, the observer observing, and the note-taker taking notes. But there’s been a shift and everyone is participating now. Your allies are talking over each other, asking questions out of sync and in a rapid fire motion. But wait! You were the one leading this.

How to avoid this:

  • A usability test should have no more than two people from your team. Any more than that, and things can get uncomfortable for you and the user.
  • Make sure everyone knows their role before testing begins. One person facilitates and talks to the user while the other observes and takes notes. Anyone else is a quiet observer.
  • If you are an observer and have a question you’d like to ask, write it on a post-it and pass it to the facilitator. Having only one person talk to the user will help them forget about the other people in the room. This will in turn make them more comfortable and more likely to open up.
  • Debrief after each session. When you finish talking about what you learned from the user, switch to team execution and dynamics. Use this time to call out specific instances of interruptions and missed opportunities. Communicating frequently and openly makes these conversations feel less accusatory and more constructive.

During the session:

  • Maintain eye contact with your team. The second you stop looking at each other to give and read cues, you’re disconnected and are more likely to act as individuals rather than as a unit.
  • If a teammate is overexcited and talking more than the user, find a non-verbal way to tell them to ease up a little. A gentle arm squeeze and smile can go a long way.
  • After the test is over, remind your team that the user should be doing 99% of the talking. We are merely there to listen and learn.

You ran out of questions

You went through that script way faster than intended. The user performed wonderfully and wasn’t stuck on anything. But you have a full 20 minutes left before the session is over. You start to get a little nervous and now you’re asking one-off rhetorical questions that you know won’t really tell you much.

How to avoid this:

  • Loosen up your script a bit. Write down topics you’d like to cover in addition to the questions you want to ask. Having topics allows you to go wherever the conversation leads, rather than sticking to questions that might be rendered irrelevant after the first couple of minutes.
  • Incorporate a bonus section into your script that covers some nice-to-have questions. These are things that could go unanswered, but would be interesting to ask. For example, “If you had a magic wand that could do anything to this app, what would you do?”

During the session:

  • Turn to the user’s phone. Ask them to show you their favorite apps. The ones they used on the way to the session, the ones they find simple or complicated, the ones they can’t live without. Have them perform tasks and explain to you what happens at each step. See if they have one that’s similar to what you’re testing and learn about what works and what doesn’t.
  • They don’t have a smart phone? Then ask them to tell you some stories. Think about the tasks they will perform in your app, and ask how they do them now. Even asking about what they did yesterday could be helpful for your research.
  • If you are truly done with the test and feel like your questions have been answered, end early.

Your prototype is busted

But it worked in the office! We know. But if it depends on Wi-Fi, and you are in a location where it’s not available, then you’ll have a problem.

How to avoid this:

  • Host all of your files locally so you don’t need Wi-Fi. This includes scripts, images, libraries, etc.
  • Test it yourself before you present it to someone else. Go through all the tasks to make sure it’s not broken.
  • Bring the device you have been testing on. Testing on a new device can throw all sorts of unexpected wrenches.
  • Bring print-outs of your screens as a backup plan. Have the user tap on elements and ask what they would expect to happen next.

During the session:

  • Turn to paper. If you don’t have print-outs with you, quickly sketch out the parts of the prototype that you’d like to test and ask questions about them.
  • If you don’t have the time to draw anything yourself, have the user do it. Ask them to draw from memory whatever they tool they are using now to get the job done. Just seeing what they remember will give so much insight to whatever you are building. You can also ask them to draw the ideal interface they would want to use.
  • As mentioned in the previous scenario, asking them to show you apps on their phone and to tell you stories can really help if your prototype is unsalvageable.

Your facilitator is leading the witness

They started out just fine. But they gradually spiraled into the land of leading questions with yes or no answers. Now they can’t stop starting questions with: “Wouldn’t it be great if…?”, “Do you like it when…?”, “Would you use this feature…?” These types of questions stifle conversation by making it difficult to ask follow-up questions.

How to avoid this:

  • Spend time with your script and turn any leading questions into open-ended ones. “When was the last time you…?”, “What do you think will happen when you…”, “How often do you…?”.
  • Conduct some practice interviews with your teammates. Two of you can take turns asking each other questions, while a moderator points out examples of open-ended or leading questions.
  • It takes time to get into the habit of asking for stories rather than opinions. The more interviews you do, the better you’ll get!

During the session:

  • Ask the user “why?” after some of their answers. This can help open the conversation a bit, and may naturally get your partner back on track.
  • Communicate to your facilitator via a post-it note. Scribble a gentle reminder asking for “open-ended” and “stories”.
  • Write down some examples of their leading questions, so you can give them pointers after the interview. Having concrete examples makes it easier to give constructive feedback.

The user is stuck on fake data

In your prototype’s world, $2.00 + $5.00 = $10.00, and the user is having a hard time believing it.

How to avoid this:

  • Be diligent about the fake data you put in. If you are showing a shopping cart, make sure the prices add up. If you are trying to show a history of orders, make sure that all the dates are not the same. If you don’t want to use any values, draw some scribbles and tell the user that it’s data.
  • Don’t use lorem ipsum. Use real names and real(ish) descriptions. Making your data coherent can save you a lot of time during the test and prevent any confusion. It can also bring up things you didn’t realize were issues, like character length.
  • Don’t use the pictures or names of people they may know, like celebrities, yourself, or your team members. They have the potential to create unwelcome distractions and assumptions about your product.

During the session:

  • Take a moment to explain that because this is a prototype, the data is neither real nor relevant for today’s test. Be sure to mention this in an intro at the beginning to alleviate any questions or confusion.
  • Keep redirecting their focus to the task at hand.
  • If they really can’t see past the fake data, ask them why. Maybe it’s not about whether or not 2 + 5 = 10, but that they were expecting something else entirely.

When all else fails…

…take a deep breath and remember that you’re all here with the same purpose: to make your product a success. The fact that you are doing usability research shows that you care about your users and want them to love your product as much as you do. So relax! And your users will, too.